You Only Need Adversarial Supervision for Semantic Image Synthesis
Authors: Edgar Schönfeld
Vadim Sushko
Dan Zhang
Juergen Gall
Bernt Schiele
Anna Khoreva
Venue: ICLR 2021 Poster (2021)
Keywords: Semantic Image Synthesis
GANs
Image Generation
Deep Learning

Abstract: Despite their recent successes, GAN models for semantic image synthesis still suffer from poor image quality when trained with only adversarial supervision. Historically, additionally employing the VGG-based perceptual loss has helped to overcome this issue, significantly improving the synthesis quality, but at the same time limiting the progress of GAN models for semantic image synthesis. In this work, we propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We re-design the discriminator as a semantic segmentation network, directly using the given semantic label maps as the ground truth for training. By providing stronger supervision to the discriminator as well as to the generator through spatially- and semantically-aware discriminator feedback, we are able to synthesize images of higher fidelity with better alignment to their input label maps, making the use of the perceptual loss superfluous. Moreover, we enable high-quality multi-modal image synthesis through global and local sampling of a 3D noise tensor injected into the generator, which allows complete or partial image change. We show that images synthesized by our model are more diverse and follow the color and texture distributions of real images more closely. We achieve an average improvement of $6$ FID and $5$ mIoU points over the state of the art across different datasets using only adversarial supervision.

Published as a conference paper at ICLR 2021
YOU ONLY NEED ADVERSARIAL SUPERVISION FOR SEMANTIC IMAGE S SYNTHESIS
Edgar Schönfeld
Vadim Sushko
Bosch Center for Artificial Intelligence
Bosch Center for Artificial Intelligence
Dan Zhang
Jürgen Gall University of Bonn
Bosch Center for Artificial Intelligence
Bernt Schiele
Anna Khoreva Bosch Center for Artificial Intelligence
Max Planck Institute for Informatics
ABSTRACT
Despite their recent successes, GAN models for semantic image synthesis still suffer from poor image quality when trained with only adversarial supervision. Historically, additionally employing the VGG-based perceptual loss has helped to overcome this issue, significantly improving the synthesis quality, but at the same time limiting the progress of GAN models for semantic image synthesis. In this work, we propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We re-design the discriminator mantic segmentation network, directly using the given semantic label maps as the ground truth for training. By providing stronger supervision to the discriminator as well as to the generator through spatially- and semantically- feedback, we are able to synthesize images of higher fidelity with better align- ment to their input label maps, making the use of the perceptual loss superfluous. Moreover, we enable high-quality multi-modal image synthesis through global and local sampling of 3D noise tensor injected into the generator, which allows complete or partial image change. We show that images synthesized by our model are more diverse and follow the color and texture distributions of real images more closely. We achieve an average improvement of FID and 5 mIoU points over the state of the across different datasets using only adversarial supervision.
se-
discriminator
Semantic label map
SPADE Park et al.
2019 w/O VGG
Our model (OASIS), sampled with different noise
with VGG
w/o VGG
Existing semantic image synthesis models heavily rely on the VGG-based perceptual loss improve the quality of generated images. contrast, our model can synthesize diverse and high-quality images while only using an adversarial loss, without any external supervision.
Figure 1:
"Equal contribution. Correspondence fedgar. schoenfeld, vadim m. sushko (ushka)@boscaa @bosch. com.

---

Published as a conference paper at ICLR 2021
1 INTRODUCTION
Conditional generative adversarial networks (GANs) (Mirza& & Osindero 2014) synthesize images conditioned on class labels (Zhang et al.] 2019. Brock et al. 2019), text (Reed et al. 2016 Zhang
2018), or semantic label maps (Wang al. 2018 Park et al., 2019). In this work, we focus on the latter, addressing semantic image syn- thesis. Semantic image synthesis enables rendering of realistic images from user-specified layouts, without the use of an intricate graphic engine. Therefore, its applications range widely from con- tent creation and image editing to generating training data that needs to adhere to specific semantic requirements (Wang et al.] 2018]] Chen & Koltun, (0017). Despite the recent progress stabilizing GANs (Gulrajani et al. [0117] 7 Miyato et al.| [018 Zhang & Khoreva 2019) and developing their architectures (Zhang et al. [0199] Karras al.] 2019), state-of-the-art GAN-based semantic image
al., 20), other images (Isola et al. |017]] Huang et al.
synthesis models (Park et al. 2019; Liu et al. 2019) still greatly suffer from training instabilities and poor image quality when trained only with adversarial supervision (see Fig. IT. An established practice to overcome this issue is to employ a perceptual loss [Wang et al. 2018) to train the genera- tor, in addition the discriminator loss. The perceptual loss aims to match intermediate features of synthetic and real images, that are estimated via an external perception network. A popular choice for such a network is VGG (Simonyan & Zisserman 2015), pre-trained on ImageNet (Deng et al.) 2009). Although the perceptual loss substantially improves the accuracy of previous methods, it comes with the computational overhead introduced by utilizing an extra network for training. More- over, it usually dominates over the adversarial loss during training, which can have a negative impact on the diversity and quality of generated images, as we show in our experiments. Therefore, in this work we propose a novel, simplified model that achieves state-of-the-ar results without requiring perceptual loss.
A fundamental question for GAN-based semantic image synthesis models is how to design the discriminator to efficiently utilize information from the given semantic label maps. Conventional methods (Park et al., [2019; Wang et 20188 Liu et al.) [019]] Isola et al. (017)) adopt multi-scale classification network, taking the label map as input along with the image, and making global image-level real/fake decision. Such a discriminator has limited representation power, as it is not incentivized to learn high-fidelity pixel-level details of the images and their precise alignment with the input semantic label maps. To mitigate this issue, we propose an alternative architecture for the discriminator, re-designing it as an encoder-decoder semantic segmentation network (Ronneberger et al. 2015), and directly exploiting the given semantic label maps as ground truth via (Nlass cross-entropy loss (see F. This new diseriminator provides semantically-awart pixel-level feed- back the generator, partitioning the image into segments belonging to one of the N real semantic classes or the fake class. Enabled by the discriminator per-pixel response, we further introduce La- belMix regularization, which fosters the discriminator to focus more on the semantic and structural differences of real and synthetic images. The proposed changes lead to a much stronger discrimina- tor, that maintains a powerful semantic representation of objects, giving more meaningful feedback to the generator, and thus making the perceptual loss supervision superfluous (see Fig. D
Isola
Next, we propose to enable multi-modal synthesis of the generator via 3D noise sampling. Pre- viously, directly using ID noise as input was not successful for semantic image synthesis, as the generator tended to mostly ignore it or synthesized images of poor quality (Isola et al. 2017 Wang et al. 2018). Thus, prior work (Wang et al., 2018 Park et al. 2019) resorted to using an image encoder to produce multi-modal outputs. In this work, propose lighter solution. Empowered by our stronger discriminator, the generator can effectively synthesize different images by simply re-sampling 3D noise tensor, which is used not only as the input but also combined with interme- diate features via conditional normalization at every layer. Such noise is spatially sensitive. so we can re-sample it both globally (channel-wise) and locally (pixel-wise), allowing change not only the appearance of the whole scene, but also of specific semantic classes or any chosen areas (see Fig. 2). We call our model OASIS, as it needs only adversarial supervision for semantic image synthesis.
In summary, our main contributions are: (1) We propose a novel segmentation- bbase discriminator architecture, that gives more powerful feedback to the generator and eliminates the necessity of the perceptual loss supervision.
(2) We present a simple 3D noise sampling scheme, notably increasing the diversity of multi-modaa synthesis and enabling complete image. (3) With the OASIS model, we achieve high quality results on the ADE20K, Cityscapes and COCO-stuff datasets, on average improving the state of the art by 6 FID and 5 mloU points, while
partial change of the generated
2

---

Published as a conference paper at ICLR 2021
Label map
OASIS
OASIS 2
OASIS
OASIS
OASIS
Global (the whole scene)
Local (bod)
Local (shape)
Figure 2: OASIS multi-mode synthesis results. The 3D noise can be sampled globally (first 2 rows), changing the whole scene, or locally (last 2 rows), partially changing the image. For the latter, sample different noise per region, like the bed segment (in red) or arbitrary areas defined by shapes.
relying only on adversarial supervision. We show that images synthesized by OASIS exhibit much higher diversity and more closely follow the color and texture distributions of real images. Our code and pretrained models are available at https: //tp://ggtttt//// hwtp://osocchheeeaacc///aaaaaaa
2 RELATED WORK
Semantic image synthesis. Pix2pix (Isola et al. 2017) first proposed to use conditional GANs (Mirza & Osindero 2014) for semantic image synthesis, adopting an encoder-decoder gen- erator which takes semantic label maps as input, and employing PatchGAN discriminator. Since then, various generator and discriminator modifications have been introduced (Wang et al.] 2018 Park al. 20199 Liu et al., 2019 Tang et al.) 2020c b. Ntavelis al. (020). Besides GANs, Chen & Koltun 2017) proposed to use a cascaded refinement network (CRN) for high high-resolution semantic image synthesis, and SIMS (Qi et et al. 2018) extended with a non non-oonnnmaeeee parametriaee component, serving as a memory bank of source material to assist the synthesis. Further, Li et al. (2019) employed implicit maximum likelihood estimation (Li & Malik 2018) increase the variety of the CRN model. How- ever, these approaches still underperform in comparison to state-of-the-ae GAN models. Therefore, next we focus on the recent GAN architectures for semantic image synthesis.
Discriminator architectures. Pix2pix (Isola et al. 2017), Pix2pixHD (Wang et al. 2018 and SPADE (Park al. 2019) all employed a multi-scale PatchGAN discriminator, that takes im- age and its semantic label map as input. CC-FPSE (Liu et al. 2019) proposed a ceature-pyramee discriminator, embedding both images and label maps into joint feature map. and then consec- utively upsampling it in order to classify it as real/fake at multiple scales. LGGAN (Tang et all 2020ch introduced classification-based feature learning module to learn more discriminative and class-specific features. In this work, we propose to use a pixel-wise semantic segmentation network as discriminator instead of multi-scale image classifiers in the above approaches, and directly exploit the semantic label maps for its supervision. Segmentation-bant based discriminators have been shown to improve semantic segmentation (Souly et al. 2017) and unconditional image synthesis (Schönfeld et al. (220)). but to the best of our know knowledge have not been explored for semantic image synthesis and our work is the first to apply adversarial semantic segmentation loss for this task.
Generator architectures. Conventionally, the semantic label map is provided to the image genera- tion pipeline via encoder (Isola al. 2017: Wang et al. 2018, Tang et al. 2020c by Ntavelis et al.) 2020). However, it is shown to be suboptimal at preserving the semantic information until the later stages of image generation. Therefore, SPADE introduced spatially-adaptive normalization layer
3

---

Published as a conference paper at ICLR 2021
SPADE
OASTS
VGG
Fille
Real Iasus
Laze: Xap
Imsuee
Label Map
Perorptual Lose
Imsag
Beal Fadko
Segmentation Board Diecriminatur
Roal/ Pake
Enooder
Generator
Pale buogg
Multi Scale
Genetator
Fale Traage
Dicrininator
SPADE (left) vs. OASIS (right). OASIS outperforms SPADE, while being simpler and lighter: it uses only adversarial loss supervision and single segmentation-baser discriminator, without relying on heavy external networks. Furthermore, OASIS learns to synthesize multi-modal outputs by directly re-sampling the 3D noise tensor, instead of using an image encoder as in SPADE.
Figure 3:
that directly modulates the label map onto the generator's 's hidden layer outputs at various scales. Alternatively, CC-FPSE proposed to use spatially-varying convolution kernels conditioned on the label map. Struggling with generating diverse images from noise, both Pix2pixHD and SPADE re- sorted to having an image encoder in the generator design to enable multi-mode synthesis. The generator then combines the extracted image style with the label map to reconstruct the original im- age. By alternating the style vector, one can generate multiple outputs conditioned on the same label map. However, using image encoder is a resource demanding solution. In this work, we enable multi-modal synthesis directly through sampling of 3D noise tensor injected at every layer of the network. Differently from structured noise injection Allarri & Wonka (2020) and class-specific latent codes of /hu et al. (2020) we inject the 3D noise along with label maps and adjust it to image resolution, also enabling re-sampling of selected semantic segments (see Fig. 2
Perceptual losses. Gatys et (2015); Gatys et al. (016); Johnson al. (2016) and Bruna et al. (016)) were pioneers at exploiting perceptual losses to produce high-quality images for super- resolution and style transfer using convolutional networks. For semantic image synthesis, the VGG- based perceptual loss was first introduced by CRN, and later adopted by Pix2pixHD. Since then, has become a default for training the generator (Park et al., 2019) [Liu et al.] 2019 TTa et al.] 2020 Tang et al.) 2020a). As the perceptual loss is based on a VGG network pre-trained on Ima- geNet (Deng et al., 2009), methods relying on it are constrained by the ImageNet domain and the representational power of VGG. With the recent progress on GAN training, c.g. by architecture designs and regularization techniques, the actual necessity of the perceptual loss requires a reassess- ment. We experimentally show that such loss imposes unnecessary constraints on the generator, significantly limiting sample diversity. While our model, trained without the VGG loss, achieves improved image diversity while not compromising image quality.
3 OASIS MODEL
In this section, we present our OASIS model, which, in contrast to other semantic image synthesis methods, needs only adversarial supervision for generator training. Using SPADE as a starting point (Sec. J.I], we first propose to re-design the discriminator as semantic segmentation network, directly using the given semantic label maps as ground truth (Sec. 3.2). Empowered by spatially- and semantically-awarr feedback of the new discriminator, we next re-design the SPADE generator, enabling its effective multi-modal synthesis via 3D noise sampling (Sec. 3.3)
3.1 THE SPADE BASELINE
We choose SPADE as our baseline as it is a sfate-of-the-art model and relatively simple represen- tative of conventional semantic image synthesis models. As depicted in Fig. 3 the discriminator of SPADE largely follows the PatchGAN multi-scale discriminator (Isola et al., 2017). adopting two image classification networks operating at different resolutions. Both of them take the channel-wise concatenation of the semantic label map and the real/synthesized image input, and produce true/- fake classification scores. On the generator side, SPADE adopts maatially-adaptie layers effectively integrate the semantic label map into the synthesis process from low to high scales. Additionally, the image encoder is used to extract the style vector from the reference image and then combine it with 1D noise vector for multi-modal synthesis. The training loss of SPADE
normalization
4

---

Published as a conference paper at ICLR 2021
consists of three terms, namely, an adversarial loss, a feature matching loss and the VGG-based perceptual loss: L = maxç minp Ladv + Atm + Cvgg Overall, SPADE is a resource de- manding model at both training and test time, ie.. with two PatchGAN discriminators, an image encoder in addition to the generator, and the VGG loss. In the following, we revisit its architecture and introduce simpler and more efficient model that offers better performance with less complexity.
3.2 OASIS DISCRIMINATOR
For the generator to learn to synthesize images that are well aligned with the input semantic label maps, we need a powerful discriminator that coherently captures discriminative semantic features at different image scales. While classification-base discriminators, such as PatchGAN, take label maps as input concatenated to images, they can afford to ignore them and make the decision solely on image patch realism. Thus, we propose to cast the discriminator task as a multi-class semantic segmentation problem to directly utilize label maps for supervision, and accordingly alter its archi- tecture to an encoder-decoder segmentation network (see Fig. 3 Encoder-decoder networks have proven to be effective for semantic segmentation (Badrinarayanan et al. 2016 Chen et al.) 2018). Thus, we build our discriminator architecture upon U-Net (Ronneberger et al. 2015), which con- sists of the encoder and decoder connected by skip connections. This discriminator architecture is multi-scale through its design, integrating information over up- and down-sampling pathways and through the encoder-decoder skip connections. For details on the architecture see C.p.
The segmentation task of the discriminator is formulated to predict the per-pixel class label of the real images, using the given semantic label maps as ground truth. In addition to the N semantic classes from the label maps, all pixels of the fake images are categorized as one extra class. Overall, we have N + classes in the semantic segmentation problem, and thus propose use a (N+1)-class cross-entropy loss for training. Considering that the N semantic classes are usually imbalanced and that the per-pixel size of objects varies for different semantic classes, we weight each class by its inverse per- per-pixel frequency, giving rare semantic classes more weight. In doing so, the contributions of each semantic class are equally balanced, and, thus, the generator is also encouraged to adequately synthesize less-represented classes. Mathematically, the new discriminator loss is expressed as:
N
HxW
HxW
Ln = -I
OC
Σ
14 log D(x) - E(e,e)
Σ 2
log D(G(2,0))s
(1)
c=l
where x denotes the real image; (z, 1) is the noise-label map pair used by the generator to synthe- size a fake image; and the discriminator D maps the real or fake image into a per-pixel (-+lssclass prediction probability. The ground truth label map has three dimensions, where the first two cor- respond to the spatial position (i,j) E H x W, and the third one is one-hot vector encoding the class c E [1,.,,N+1], The class balancing weight a, is the inverse of the per- per-pixel class frequency H x W
ag
HxW Eij
Fx (1 [1.ja- = 1]
LabelMix regularization. In order to encourage our discriminator to focus on differences in con- tent and structure between the fake and the real classes, we propose a LabelMix regularization. Based on the semantic layout, we generate a binary mask M to mix a pair (a, f) of real and fake images conditioned on the same label map: LabelMix(r, 2, M)- = M O + (1 M) @ &, as visu- alized in Fig. 4 Given the mixed image, we further train the discriminator to be equivariant under the LabelMix operation. This is achieved by adding consistency loss term Leons to Eq. T
Leons = || Progite (LabelMix((; à, M)) LabelMix (Diogits Piogica (6), M
(3)
where Diogits are the logits attained before the last softmax activation layer, and || norm. This consistency loss compares the output of the discriminator on the LabelMix image with the LabelMix of its outputs, penalizing the discriminator for inconsistent predictions. LabelMix is different to CutMix (Yun et al.] 2019). which randomly samples the binary mask M. A random mask will introduce inconsistency between the pixel-level classes and the scene layout provided by the label map. For object with the semantic class c, it will contain pixels from both real and fake images, resulting in two labels, i.e. c and N +1. To avoid such inconsistency. the mask of LabelMix is generated according to the label map. providing natural borders between semantic regions, Fig. [ (Mask M). Under LabelMix regularization, the generator is encouraged to respect the natural semantic boundaries, improving pixel-level realism while also considering the class segment shapes.
the L2
5

---

Published as a conference paper at ICLR 2021
Label map
Real image
Fake image di
Mask M
(rabelMix (a.))
LabelMixx
Figure 4: LabelMix regularization. Real I and fake å images are mixed using a binary mask M. sampled based on the label map, resulting in LabelMix The consistency regularization then minimizes the L2 distance between the logits of DLabelMix, a; and LabelMix(D,.D.)) In this visualization, black corresponds the fake class in the N+1 segmentation output.
Other variants. Besides the proposed (N+1)-class cross entropy loss, there are other ways to train the eegmentation-asee discriminator with the label map. One can concatenate the label map to the input image, analogous to SPADE. Another option is to use projection, by taking the inner product between the last linear layer output and the embedded label map, analogous class-label conditional GANs (Miyato & Koyama 2018). For both alternatives, the training loss is pixel-Ievel real/fake binary cross-entropy (Schönfeld al., 2020). From the label map encoding perspective, these two variants use labels map as input (concatenated to image or at last linear layer), propagating forward through the network. The (N+1)-setting uses the label map for loss computation, so is propagated backward via gradient updates. Backward propagation ensures that the discriminator learns semantic-aware features, contrast to forward propagation, where the label map alignment is not as strongly enforced. Performance comparison of the label map encodings is shown in Table 5
3.3 OASIS GENERATOR
To stay in line with the OASIS discriminator design, the training loss for the generator changed to
N
HxW
LG = -
Σ
Σ tijj log D(G(2, ())
(4)
which direct outcome of the non-saturation trick (Goodfellow et al. 2014) to Eq. We next re-design the generator to enable multi-modal synthesis through noise sampling. SPADE is deter- ministic in its default setup, but can be trained with an extra image encoder to generate multi-modal outputs. We introduce simpler version, that enables synthesis of diverse outputs directly from input noise. For this, we construct a noise tensor of size 64x H x W, matching the spatial dimensions of the label map HxW. Channel-wise concatenation of the noise and label map forms 3D tensor used as input to the generator and also as conditioning at every spatially-adaptive normalization layer. In doing so, intermediate feature maps are conditioned noise (see Fig. 33.. With such a design, the generator produces diverse, noise-dependent images. As the 3D noise channel- and pixel-wise sensitive, at test time, one can sample the noise globally, per-channel, and locally, per-segment or per-pixel, for controlled synthesis of the whole scene or of specific semantic objects. For example, when generating a scene of a bedroom, one can re- re-sample the noise locally and change the appearance of the bed alone (see Fig. ( Note that for simplicity during training we sample the 3D noise tensor globally, i.c. per-channel, replicating each channel value spatially along the height and width of the tensor. We analyse alternative ways of sampling 3D noise during training in App. A.7 Using image styles via an encoder, as in SPADE, is also possible in our setting, by replacing noise with encoder features. Lastly, to further reduce the complexity, we remove the first residual block in the generator, reducing the number of parameters from 96M to 72M (see App. C.2 without noticeable performance loss (see Table 3).
both the semantic labels and the

---

Published as a conference paper at ICLR 2021
Label map
Ground truth
2ix2piXHD
SPADE
CC-FPSE
OASIS
Figure 5: Qualitative comparison of OASIS with other methods on ADE20K. Trained with only adversarial supervision, our model generates images with better perceptual quality and structure.
4 EXPERIMENTS
We conduct experiments on three challenging datasets: ADE20K (Zhou et al. 2017). COCO-stuff (Caesar et al. 2018) and Cityscapes (Cordts et al. 2016). Following Qi et al. 2018), 8), we also evaluate OASIS on the experimental setting of Park al. al. (2019). We did not use the GAN feature matching loss for OASIS, as we did not observe any improvement with it (see App. A.5)) and used the VGG loss only for ablations with Avag 10. We did not experience any training instabilities and, thus, did not employ any extra stabilization techniques. All our models use an exponential moving average (EMA) of the generator weights with 0.9999 decay. For further training details refer to App. C.3
a subset of ADE20K containing outdoor scenes. We follow
Following prior work (Isola al. 2017: Wang al. 2018 Park et al. 2019 Liu al., 2019
we evaluate models quantitatively on the validation set using the Fréchet Inception Distance (FID) (Heusel et al. (017) and mean pnvessectionovee-Unnee both quality and diversity and has been shown to be well aligned with human judgement (Heusel jet al. 2017). We show additional evaluation of quality and diversity with "improved precision and recall" in App. [A.9 Mean IoU is used to assess the alignment of the generated image with the ground truth label map, computed via a pre-trained semantic segmentation network. We use Uper- Net101 (Xiao al., (018)) for ADE20K, multi-scale DRN-D-105 (Yu al. 2017) for Cityscapes, and V2 (Chen et al.] 2013) for COCO-Stuff. We additionally propose to compare color and texture statistics between generated and real images on ADE20K to better understand how the perceptual loss influences performance. For this, we compute color histograms in LAB space and measure the earth mover's distance between the real and generated sets (Rubner al. (000)). We measure the texture similarity to the real data as the >-disttacc between Local Binary Patterns his- tograms (Ojala et al. 1996). As different classes have different color and texture distributions, we aggregate histogram distances separately per class and then take the mean. Lower values for the texture and color distances indicate closer similarity to real data.
(mloU). FID is known to be sensitive to
4.1
MAIN RESULTS
We use SPADE as our baseline, using the authors' implementatior For a fair comparison, we train this model without the feature matching loss and using EMA (Yaz et al. 2018) at test phase, which
g www./abbs/PPD

---

Published as a conference paper at ICLR 2021
Table 1: Comparison with other methods across datasets. Bold denotes the best performance.
ADE20K
ADE-outd. FID4
Cityscapes mloUt
COCO-stuff
Method
# param
VGG
FID
mloU1
mIoUt
FID
FID4
mloUt
CRN
84M
73.3
22.4
99.0
16.5
104.7
52.4
70.4
23.7
SIMS
56M
n/a
n/a
67.7
13.1
49.7
47.2
n/a
n/a
Pix2pixHD
183M
81.8
20.3
97.8
17.4
95.0
58.3
111.5
14.6
LGGAN
n/a
31.6
41.6
n/a
n/a
57.7
68.4
n/a
n/a
CC-FPSE
131M
31.7
43.7
n/a
n/a
54.3
65.5
19.2
41.6
SPADE
102M
33.9
38.5
63.3
30.8
71.8
62.3
22.6
37.4
32.9
42.5
51.1
32.1
47.8
64.0
21.7
38.8
SPADE+
102M
60.7
21.0
65.4
22.7
61.4
47.6
99.1
16.1
OASIS
94M
28.3
48.8
48.6
40.4
47.7
69.3
17.0
44.1
Table 2: Multi-modal synthesis evaluation on ADE20K. Bold and red denote the best and the worst performance, respectively.
Texture
Color
3.4
3.2
with VGG
Method
Multi-mod.
VGG
M
LPIPST
FID
FID|
1.7
VGG
1.8
2.1
2.2
2.4
SPADE+
Encoder
0.85
0.16
33.4
40.2
x 0.35
0.50
58.4
18.7
SPADE+
3D noise
/ 0.53
0.36
34.4
36.2
OASIS
SPADE+
OASIS
SPADE+
x 0.65
0.35
28.3
48.8
OASIS
3D noise
/
0.88
0.15
31.6
50.8
Figure 6: Histogram distances real data.
further refer SPADE+. We found that the feature matching loss has negligible impact (see App. A.5). while EMA significantly incrassss the performance for all metrics (see Table
OASIS outperforms the current state of the art on all datasets with an average improvement of 6 FID and 5 mloU points (Table D. Importantly, OASIS achieved the improvement via adversarial supervision alone. On the contrary, SPADE+ does not produce images of high visual quality without the perceptual loss, and struggles to learn the color and texture distribution real images (Fig. (6). A strong discriminator is the key factor for good performance: without rich training signal from the discriminator, the SPADE+ generator has to learn through minimizing the VGG loss. With the stronger OASIS discriminator, the perceptual loss does not overtake the generator supervision (see App. A.2 allowing to produce images with the color and texture distribution closer to the real data.
Fig. shows a qualitative comparison of our results to previous models. Our approach noticeably improves image quality, synthesizing finer textures and more natural colors. With the powerful feedback from the discriminator, OASIS is able to learn the appearance of small or rarely occurring semantic classes (which is reflected in the per-class IoU scores presented in App. plausible results even for complex scenes with rare classes and reducing unnatural artifacts.
producing
Multi-modal image synthesis. In contrast to previous work, OASIS produce diverse images by directly re-sampling input 3D noise. As 3D noise modulates features directly at every layer of the generator different scales, matching their resolution, it affects both global and local characteristics of the image. Thus, the noise can be sampled globally, varying the whole image, or locally, resulting in the selected object change while preserving the rest of the scene (see Fig.
To measure the variation in the multi-modal generation, and LPIPS (Zhang et al. 20185) between images generated from the same label map. We generate 20 images and compute the mean pairwise scores, and then average over all label maps. The lower the MS-SSIM and the higher the LPIPS scores, the more diverse the generated images are. To assess the effect of the perceptual loss and the noise sampling on diversity, we train SPADE+ with 3D noise or the image encoder, and with or without the perceptual loss. Table 2 shows that OASIS, without perceptual loss, improves over SPADE+ with the image encoder, both in terms of image diversity (MS-SSIM, LPIPS) and quality (mean FID, mloU across 20 realizations). Using 3D noise further increases diversity for SPADE+. However, strong quality-diversity tradeoff exists for SPADE+: 3D noise improves diversity the cost of quality, and the perceptual loss improves quality the cost of diversity. For OASIS, the VGG loss also reduces diversity but does not noticeably affect quality. Note that in our experiments Max does not notably affect diversity (see App. A.1
evaluate MS-SSIM (Wag et 2003
8

---

Published as a conference paper at ICLR 2021
4.2 ABLATIONS
We conduct ablations on ADE20K evaluate our proposed changes. The main ablation shows the impact of our new discriminator, lighter generator, LabelMix and 3D noise. Further ablations are concerned with architecture changes and the label map encodings in the discriminator, where for fair comparison use no 3D noise and LabelMix.
Main ablation. Tablee[showw that SPADE+ scores low on the image quality metrics without the percep- tual loss. Replacing the SPADE+ discriminator with the OASIS discriminator, while keeping the gener- ator fixed, improves FID and mIoU by more than 30 points. Changing the SPADE+ generator to the lighter OASIS generator leads to a negligible degra- dation of 0.3 in FID and 0.5 in mloU. With La- belMix FID improves further by point (more ab- lations LabelMix in App. A.44) Adding 3D noise improves FID but degrades mIoU, as diversity complicates the task of the pre-trained semantic seg- mentation network used to compute the score. For OASIS the perceptual loss deteriorates FID by more than 2 points, but improves mloU. Overall, without the perceptual loss the new discriminator the key the performance boost over SPADE+.
Table 3: OASIS ablation on ADE20K. Bold denotes the best performance.
G
D
VGG
LabelMix
FID
mloU
SPADE+
SPADE+
X
X
60.7
21.0
SPADE+
OASIS
29.0
52.1
x
x
29.3
51.6
OASIS
OASIS
x
28.4
50.6
OASIS
28.3
48.8
noise
OASIS
31.6
50.8
Ablation the discrininator architecture. We train the OASIS generator with three alternative dis- criminators: the original multi-scale PatchGAN con- sisting of two networks, single-scale PatchGAN,
Table 4: Ablation on the architecture.
Bold denotes the best performance, red high- lights collapsed runs.
and ResNet-based discriminator, corresponding to the encoder of the U-Net shaped OASIS discrimi- nator. Table shows that the alternative discrim- inators only perform well with perceptual supervi- sion, while the OASIS discriminator achieves supe- rior performance independent of it. The single-scale discriminators even collapse without the perceptual loss (highlighted in red in Table
D architecture
w/o VGG
with VGG
FID
mloU
FID, mloUt
MS- PatchGAN (2x)
60.7
21.0
32.9
42.5
PatchGAN
197
0.62
34.2
42.2
ResNet-PatchGAN
147
0.42
32.4
45.1
OASIS
29.3
51.6
29.2
51.1
Ablation on the label map encoding. We study four different label map encodings: input concate- nation, as in SPADE, projection conditioned on the label map (Myyato & Koyama| 2018), employing la- bel maps as ground truth for the N+1 segmentation loss, or for the class-balanced N+1 loss (see Sec. 3.2) As shown in Table input concatenation is not sufficient without additional perceptual loss supervi- sion, leading to training collapse. Without percep- tual loss, the N+1 loss outperforms the input con- catenation and the projection in both the FID and mloU metrics. The class balancing noticeably improves mloU due to better supervision for rarely occurring semantic classes. More ablations can
Table 5: Ablation the label map encod- ing. Bold denotes the best performance, red highlights collapsed runs.
w/o VGG
with VGG
Label encoding
FID4 mloUf
FID, mloUt
Input concatenation Projection N+1 loss
280
0.02
30.0
43.9
32.4
44.9
28.0
46.9
28.3
47.2
28.6
49.8
Balanced N+1 loss
29.3
51.6
29.2
51.1
be found in App. A
5 CONCLUSION
In this work we propose OASIS, a semantic image synthesis model that only relies on adversar- ial supervision to achieve high fidelity image synthesis. In contrast to previous work, our model eliminates the need for perceptual loss, which often imposes extra constraints on image quality and diversity. This is achieved via detailed spatial and semantic-aware supervision from our novel segmentation-based discriminator, which uses semantic label maps ground truth for training. With this powerful discriminator, OASIS can easily generate diverse multi-modal outputs by re-sampling the 3D noise, both globally and locally, allowing to change the appearance of the whole scene and of individual objects. OASIS significantly improves over the state of the art in terms image quality and diversity, while being simpler and more lightweight than previous methods.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENT
Jürgen Gall has been supported by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) under Germany's Excellence Strategy EXC 2070 -390732324.
REFERENCES
Yazeed Alharbi and Peter Wonka. Disentangled image generation through structured noise injection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. Transactions on Pattern Analysis and Machine Intelligence, 2016.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations (ICLR), 2019.
Joan Bruna, Pablo Sprechmann, and Yann LeCun. Super-resolution with deep convolutional suffi- cient statistics. In International Conference on Learning Representations (ICLR), 2016.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
Liang-Chich Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Se- mantic image segmentation with deep convolutional nets and fully connected crfs. International Conference on Learning Representations (ICLR), 2015.
Liang-Chie Ching Chen, Yukun Zhu, George Papandreou. Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In European Con- ference on Computer Vision (ECCV), 2018.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In International Conference on Computer Vision (ICCV) 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
L. A. Gatys, A. S. Ecker, and M. Bethge. Image style transfer using convolutional neural networks. In Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Leon Gatys, Alexander S Ecker, and Matthias Bethge. Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems (NeurlPs). 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of Wasserstein GANs. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In European Conference on Computer Vision (ECCV), 2018.
10

---

Published as a conference paper at ICLR 2021
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to -mage translation with conditional adversarial networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference Computer Vision (ECCV), 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, pp. 3927-3936, 2019.
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation. OrXiv:1809.09000 2018.
arXiv preprint
Ke Li, Tianhao Zhang, and Jitendra Malik. Diverse image synthesis from semantic layouts via conditional imle. In Proceedings of the IEEE International Conference Computer Vision, pp- 0220, 2019.
Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, el. Learning predict layout-to-image condi- tional convolutions for semantic image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv:1411.1784 2014.
Takeru Miyato and Masanori Koyama, cGANS with projection discriminator. In International Conference on Learning Representations (ICLR), 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
Evangelos Ntavelis, Andrés Romero, Iason Kastanis, Luc Van Gool, and Radu Timofte. Sesame: Semantic editing scenes by adding, manipulating or erasing objects. 2rXiv:2004.04977, 2020.
Timo Ojala, Matti Pietikäinen, and David Harwood. A comparative study of texture measures with classification based on featured distributions. Pattern recognition, 29(1):51-59, 1996.
Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Xiaojuan Qi, Qifeng Chen, Jiaya Jia, and Vladlen Koltun. Semi-parametris image synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models.
In Advances Neural Information Processing Systems, pp. 12268-12279, 2019.
Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. International Conference on Machine learning (ICML), 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. The earth mover's distance as metric for image retrieval. International Journal of Computer Vision (IJC), 2000.
11

---

Published as a conference paper at ICLR 2021
Mchdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assess- ing generative models via precision and recall. Advances Neural Information Processing Systems, pp. 5228-52231 2018.
Edgar Schönfeld, Bernt Schiele, and Anna Khoreva. A u-net based discriminator for generative adversarial networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari. How good is my gan? In Proceedings of the European Conference on Computer Vision (ECCV), pp. 213-229, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations (ICLR), 2015.
Nasim Souly, Concetto Spampinato, and Mubarak Shah. Semi supervised semantic segmentation using generative adversarial network. In International Conference on Computer Vision (ICCV), 2017.
Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, and Nenghai Yu. Rethinking ppatiall-adaapiiv normalization. arXiv:2004. 200.02000 2020.
Hao Tang, Song Bai, and Nicu Sebe. Dual attention gans for semantic image synthesis. In Proceed- ings of the 28th ACM International Conference on Multimedia, pp. 1994-2002, 2020a.
Hao Tang, Xiaojuan Qi, Dan Xu, Philip HS Torr, and Nicu Sebe. Edge guided gans with semantic preserving for semantic image synthesis. arXiv:2003. 13898, 2020b.
Hao Tang, Dan Xu, Yan Yan, Philip HS Torr, and Nicu Sebe. Local class-specific and global image- level generative adversarial networks for semantic-guided scene generation. In Conference Computer Vision and Pattern Recognition (CVPR), 2020c.
Ting- MingCChunWV Wang, Ming- Yu Liu, Jun-Yar Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional GANs. Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Zhou Wang, Eero P Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In Asilomar Conference on Signals, Systems & Computers, 2003.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision (ECCV), 2018.
Yasin Yaz, Chuan Chuan-Sheng Foo, Stefan Winkler, Kim-Hur Yap, Georgios Piliouras, Vijay Chan- drasekhar, et al. The unusual effectiveness of averaging in gan training. In International Confer- ence on Learning Representations, 2018.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Interna- tional Conference on Computer Vision (ICCV), 2019.
Dan Zhang and Anna Khoreva. PA-GAN: Improving GAN training by progressive augmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
H. Zhang. T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas. StackGAN++: Realistic image synthesis with stacked generative adversarial networks. Transactions on Pattern Analysis and Machine Intelligence, 2018a.
Han Zhang, Ian J. Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International Conference on Machine learning (ICML), 2019.
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al. Resnest: Split-attention networks. arXiv preprint arXiv:2004.0880 08955 2020.
12

---

Published as a conference paper at ICLR 2021
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. Conference on Computer Vision and Pattern Recognition (CVPR), 2018b.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Zhen Zhu, Zhiliang Xu, Ansheng You, and Xiang Bai. Semantically multi-modal image synthesis. In Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
13

---

Published as a conference paper at ICLR 2021
APPENDIX
This supplementary material the main paper structured as follows:
A
Additional quantitative results.
A.
Main ablation study on two datasets.
A.2
The influence of the perceptual loss training dynamics.
Per-class IoU scores across different datasets.
A
Comparing LabelMix and CutMix for consistency regularization.
A.5
Ablation on the Feature Matching loss.
A.6
Ablation on using multiple OASIS discriminators.
A.7
Ablation on noise sampling strategies during training.
Additional experiments on COCO-stuff.
A.9
Additional image quality metrics.
B
Additional qualitative results.
B.1
Visual comparison of OASIS to other works.
B.2
Multi-modal synthesis results for different label maps.
B.3
Interpolations between multi-modal images in latent space.
B.4
Application to unlabelled data.
B.5
Additional visual LabelMix examples.
A detailed description the OASIS architecture and its training details.
Discriminator architecture.
Generator architecture.
Learning objective and training details.
A QUANTITATIVE RESULTS
A.1
SUMMARIZED MAIN ABLATION OVER TWO DATASETS
Table A: Summarized ablation on two datasets. Bold denotes the best performance. Red denotes the worst performance among experiments with 3D noise. Green denotes the major performance gains that caused by the proposed OASIS discriminator and LabelMix.
Cityscapes mloUt
ADE20K
Method
FID
MS-SSIM
FID
mloU1
MS-SSIM
SPADE+
61.4
47.6
1.0
60.7
21.0
1.0
+ OASIS D, G
54.1
67.6
1.0
29.3
51.6
1.0
3D noise
51.5
66.3
0.62
28.9
47.3
0.63
Mix
47.7
69.3
0.64
28.3
48.8
0.65
+ VGG
46.1
72.0
0.84
31.6
50.8
0.88
In Table A present summarized version of ablations for the ADE20K and Cityscapes
dataset. The following observations can be made:
(1) Looking at the 2nd row Table A we see that the main performance gain comes from the discriminator design (major) (OASIS D,G). The OASIS generator is lighter version of the SPADE generator, which does not result in performance improvement (Table 3). but has significantly less parameters. second source of improvement is LabelMix.
(2) The mloU can drop when 3D noise is added, as diversity complicates the task of the pre- pre-trained semantic segmentation network that is used to compute the mloU score. Note that the purpose of noise is not to improve the image quality (FID) but to improve diversity (MS-SSIM).
(3) The perceptual loss can hurt performance and diversity by biasing the generator towards Ima- geNet, as in this the perceptual loss encourages generating images with more standard semantic features This facili- tates the task of external pretrained segmenters, and consequently helps raise the mloU metrie.
the target distribution is more difficult to recreate fully. By punishing diversity,
14

---

Published as a conference paper at ICLR 2021
A.2 THE INFLUENCE OF VGG ON TRAINING DYNAMICS
Table and Figure illustrate that performance of SPADE+ strongly depends on the perceptual loss. In contrast, OASIS achieves high quality without this loss (Table We find the explanation in the fact, that the SPADE+ Patch-GAN discriminator does not provide a strong training signal for the generator. At the absence of strong supervision from the discriminator, the generator resorts to learning mostly from the VGG loss. The loss curves Fig. support this finding: throughout the training the SPADE+ model focuses on minimizing the VGG loss, keeping the adversarial generator loss more or less constant. In contrast, OASIS significantly improves adversarial generator loss during training, learning fool the segmentation- based OASIS discriminator. That indicates better adversarial balance, when the generator learns semantically meaningful features that the segmenter judges as real. The difference in scales of G loss for models comes from different objectives, since SPADE+ optimizes binary cross entropy, and OASIS minimizes multi-class cross entropy with N+1 classes.
10.4
SPADE+ OASIS
3.5
SPADE+
DASIS
10.2
G aaversarial loss
3.0
VGG loss
2.5
10.0
2.0
9.8
mmww
9.6
1.0
0.5
9.4
50
100
150
200
50
100
150
200
Training epochs
Training epochs
Figure A: VGG and adversarial G losses for SPADE and OASIS, trained with the perceptual loss
A.3 PER-CLASS IOU SCORES
As seen Table T the main paper, OASIS significantly outperforms previous approaches in mloU. We found that the improvement comes mainly from the better IoU scores achieved for less- represented semantic classes. To illustrate the gain, we report per-class IoU scores on ADE20K, COCO-Stuff and Cityscapes in Tables B and D. For visualization purposes, we sorted the seman- tie classes of all datasets, ordering by their pixe frequency in the training images.
Taking ADE20k as an example, Table B highlights that the relative gain in mloU is especially high for the group of less-represented semantic classes, that cover less than 3% of all the images. For these rare classes the relative gain over the baseline exceeds 40%. We found that the gain majorly comes from the per-class balancing applied the OASIS loss function. In order to illustrate this effect, we train OASIS without the proposed balancing. Table B reveals, this baseline reaches bit higher score for frequent classes, but shows worse performance for the rarely occurring ones. This is expected, as the balancing down-weights the objects met frequently while up-weights infrequent classes. We thus conclude that the balancing draws the attention of the discriminator to rarely occurring semantic classes, which results in much higher quality of the generation.
A.4 ABLATION ON LABELMIX
Consistency regularization for the segmentation output of the discriminator requires a method of generating binary masks. Therefore, we compare the effectiveness of CutMix (Yun et al. 2019) and our proposed LabelMix. Both methods produce binary masks, but only LabelMix respects the boundaries between semantic classes in the label map. Table|E]compare the FID and mloU scores of OASIS trained with both methods the Cityscapes dataset. It can be seen that LabelMix improves both FID (51.5 vs. 47.7) and mloU (66.3 vs. 69.3), in comparison to OASIS without consistency regularization. CutMix-based consistency regularization only improves the mloU (66.3 67.4),
15

---

Published as a conference paper at ICLR 2021
Table B: Per-class IoU scores on ADE20K. Bold denotes the best performance.
mIoU
Classes IDs
Occupied area
SPADE+ (with VGG)
OASIS without per-class balancing
OASIS
(without VGG)
(without VGG)
0 29
86.4%
63.7
69.1
68.8
30 59
7.2%
47.4
52.4
56.6
60 89
3.5%
45.3
47.0
51.5
90 119
1.8%
29.3
36.2
41.5
120 149
1.0%
26.2
31.2
39.7
0-149
100%
42.4
47.2
51.6
(all classes)
Table C: Per-class IoU scores on COCO-Stuff. Bold denotes the best performance.
Table D: Per-class IoU scores Cityscapes. Bold denotes the best performance.
mloU
mlou
Classes IDs
Area
Classes IDs
Area
SPADE+
OASIS
SPADE+
OASIS
-
69.3%
51.1
59.0
75.6%
91.6
89.6
36 69
15.9%
43.9
50.3
18.3%
75.7
74.9
70 - 103
8.7%
40.5
40.9
7 10
3.9%
60.0
66.9
104 137
4.5%
35.9
36.6
14
1.4%
60.3
66.0
138 171
1.4%
22.1
40.6
15 18
0.6%
38.1
55.1
()171
0-18
100%
38.8
45.5
100%
63.8
69.3
(all classes)
(all classes)
but not much as LabelMix (69.3). We suspect that since the images are already partitioned through the label map, an additional partition through CutMix results in dense patchwork of areas that differ by semantic class and real-fake class identity. This may introduce additional label noise during training for the discriminator. To avoid such inconsistency between semantic classes and real-fake identity, the mask of LabelMix is generated according to the label map, providing natural borders between semantic regions, that the real and fake objects are placed side-by-side without interfering each other. Under LabelMix regularization, the generator is encouraged to respect the natural semantic class boundaries, improving pixel-level realism while also considering the class segment shapes.
A.5 ABLATION ON FEATURE MATCHING LOSS
We measure the effect of the feature matching loss (FM) in the absence and presence of the per- ceptual loss (VGG). Table and G present the results for OASIS on Cityscapes and SPADE+ on ADE20K. For both SPADE+ and OASIS we observe that the feature matching loss does only affect the FID notably when no perceptual loss used.
In the case where no perceptual loss used, we observe that the feature matching prolongs the time until SPADE+ collapses, resulting in better FID also improves. Hence, the role of the FM loss in the training of SPADE+ is to stabilize the training through additional self-supervision. This observation is in line with the general observation that SPADE and other semantic image synthesis models require the help of additional losses because the adversarial supervision through the discriminator is strong enough. In comparison, observe any training collapses in OASIS, despite not using any extra losses. For OASIS, the feature matching loss results worse FID (by 0.8 points) in the absence of the perceptual loss. We also observe degradation of 1.1 mloU points through the FM loss, in the case where the perceptual supervision is present. This indicates that the FM loss negatively affects the strong supervision from the semantic segmentation adversarial loss of OASIS.
(49.7 60.7). Consequently, the mloU
did not
16

---

Published as a conference paper at ICLR 2021
Table E: Ablation study on the impact of LabelMix and CutMix for consistency regularization (CR) in OASIS on Cityscapes. Bold denotes the best performance.
Transformation
FID
mloU 1
No CR
51.5
66.3
CutMix
52.1
67.4
LabelMix
47.7
69.3
Table F: OASIS on Cityscapes. Bold denotes the best performance.
Table G: SPADE+ on ADE20K.
Bold denotes the best performance.
VGG
FM
FID
mloUt
VGG
FM
FID
mloU1
X
47.7
69.3
X
X
60.7
21.0
X
/
48.5
69.1
/
49.7
32.5
J
X
46.1
72.0
X
32.9
42.5
46.5
70.9
32.6
42.9
A.6 ABLATION ON USING MORE THAN ONE OASIS DISCRIMINATOR
A major difference between SPADE and OASIS is that OASIS employs only one discriminator, while SPADE uses two PatchGAN discriminators at different scales. Naturally, the question arises how OASIS performs with two discriminators at different scales, as in SPADE. For this, Table H presents the FID and mIoU performance of OASIS with two discriminators operating at scales and 0.5 on Cityscapes. One can see that an additional discriminator at scale 0.5 does not improve per- formance, but slightly worsens it. The reason that no performance gain is visible is that the OASIS discriminator already encodes multi-scal information through its U-Net structure: skip connections between encoder, decoder and individual blocks integrate information at all scales. In contrast, SPADE requires two discriminators to capture information at different scales.
Table H: Comparison of using and 2 discriminators at different scales for OASIS on Cityscapes. Bold denotes the best performance.
# of OASIS
FID L 47.7
mloUt
discriminator
69.3
discriminators at different scales (1 and 0.5)
48.7
68.8
A.7 ABLATION ON NOISE SAMPLING STRATEGIES DURING TRAINING
Our 3D noise can contain the same sampled vector for each pixel, different vectors for different regions. This allows for different noise sampling schemes during training. Table [ shows the effect of using different methods of sampling 3D noise for different locations during training: Image-level sampling creates one global ID noise vector and replicates along the height and width of the label map to create a 3D noise tensor. Region-level sampling relies generating one ID noise vector per label, and stacking them 3D to match the height and width of the label map. Pixel-level sampling creates different noise for every spatial position, with replication taking place. Mix switches between image-level and region-level sampling via a coin flip decision at every training step. With no obvious winner in performance, choose the simplest scheme (image-level) for our experiments.
By choosing image-level sampling for training, we thus generate a single ID latent noise vector of size 64, broadcast it to 64xHxW W and concatenate with the label map (NxHxW), This new composite tensor is used as input to the 1st generator layer and at all SPADE-norm layers. The noise is not ignored for the following reasons:
(1) The noise modulates the activations directly at every layer, so is very hard to ignore. Here, it is important to emphasize how the noise is used: For SPADE it was observed that label maps are paid more attention to if they used for location-sensitive conditional batch normalization (CBN). Analogously, we observe that the noise also paid more attention to when it is injected via CBN. Like label maps, which are 3D tensors of stacked one-hot vectors, stack noise vectors into 3D
17

---

Published as a conference paper at ICLR 2021
tensor of the same dimensions. Thus, in the same way that SPADE is spatially sensitive to labels, OASIS spatially sensitive to both labels and noise.
(2) The 3D broadcasting strategy provides spatially uniform signal making it easy embed mantic meaning into the latent code (see interpolations, Fig. I [ As noise modulates features at different scales in the generator, matching their resolution, it affects both global and local character- istics. This is why a generator trained with image-level noise can perform region-level manipulation inference (Fig. H). However, more evolved spatial noise sampling schemes can explored in the future.
Table I: Different noise sampling strategies during training. Bold denotes the best performance.
Cityscapes mloUt
ADE20K mloUt
Sampling
FID
MS-SSIM
FID
MS-SSIM,
image-level
47.7
69.3
0.64
28.3
48.8
0.65
region-level
48.1
69.7
0.62
28.8
48.1
0.58
pixel-level
50.9
65.5
0.84
28.6
34.0
0.68
mix
46.4
70.9
0.68
28.5
47.6
0.66
A.8 ADDITIONAL EXPERIMENTS ON COCO-STUFF
We performed all our extensive ablations on ADE20K and Cityscapes, due to their shorter train- ing time. Training on ADE20K and Cityscapes takes circa 10 days on 4 Tesla V100 GPUs while training weeks. Therefore, only executed essential experiments on COCO- stuff. We compare the results of these experiments in Table For SPADE+, it can be seen that without the external perceptual supervision of VGG, training collapses (with FID 99.1 at the best checkpoint be- fore collapse). In contrast, for OASIS image quality is better without VGG (16.7 vs 18.0 FID).
Table J: Performance on COCO-stuff. Bold
denotes the best perfromance.
Model
VGG 3D noise
FID,
mloU MS-SSIM
on COCO-stuff can stretch
SPADE
22.6
37.4
1.0
SPADE=
99.1
16.1
1.0
SPADE-
1
x
21.7
38.8
1.0
OASIS
16.7
45.5
1.0
OASIS
18.0
44.2
1.0
OASIS
17.0
44.1
0.61
When 3D noise is added to OASIS, sampling of multi-modal images is enabled (0.61 1.0 MS-
SSIM), with very similar performance in synthesis quality (17.0 vs 16.7 FID) and slightly worse mloU (44.1 45.5 mloU) due the increased variation of generated samples, as the semantic segmentation task of the pre-trained segmentation network becomes harder.
A.9 ADDITIONAL EVALUATION METRICS
Currently, the FID score is the most widely adopted metric for quantifying image quality of GAN models. However, is often argued that the FID score does not adequately disentangle synthesis quality and diversity Kynkäänniemi al. 2019). Recently, series of metrics have been proposed to address this issue by measuring scores related to the concepts of precision and recall (Ravuri & Vinyals 2019 Shmelkov al. 2018 Sajjadi al. 2018 Kynkaänniemi al. (019). Here, we have a closer look at the "improved precision and recall" score proposed by (Kykkäänneem et al.) 2019), where precision is the probability that a generated image falls into the estimated support of the real image distribution, and recall is the probability that real image falls into the estimated support of the generator distribution. Precision and recall can be interpreted as sample quality and diversity. Table K preseents comparison of precision (P) and recall R) between SPADE+ and OASIS. It can be seen that OASIS outperforms SPADE+ both in terms of image quality and variety.
Table K: Comparison precision recall metric between SPADE+ and OASIS. Bold denotes the best performance.
ADE20K
ADE-outd. Pt
Cityscapes Pt
COCO-Stuff
Model
Pt
R
R
R
Pt
R
SPADE+
0.71
0.52
0.62
0.51
0.54
0.34
0.63
0.56
OASIS
0.77
0.57
0.77
0.56
0.58
0.55
0.67
0.59
18

---

Published as a conference paper at ICLR 2021
B QUALITATIVE RESULTS
B.1 COMPARISON TO OTHER METHODS
In this section we present a visual comparison between OASIS and other semantic image synthesis methods. Firstly, show images generated by SPADE Park et al. 2019). CC-FPSE (Liu et al. 2019) and OASIS on ADE20k, COCO-Stuff, and Cityscapes (in Figures B and respectively). A further comparison for SPADE, SPADE+ and OASIS is presented in Figure We observed that OASIS often produces more visually plausible images than the previous methods. Our method commonly produces finer textures, especially for complex and large semantic objects, e.g building facades, mountains, water.
We also note that OASIS usually generates brighter and more diverse colors, compared to other methods. As we showed in Section f in the main paper, the diversity in colors partially comes from the fact that the feature space of the OASIS generator is not constrained by the VGG loss. We observed that images, generated by SPADE and CC-FPSE, typically have closer colors, while OASIS frequently generates objects with completely different color tones. This also forms one of the failure modes of our approach, when the colors of objects fall out of distribution and seem unnatural (see Figure G
B.2 MOLTI-MODAL IMAGE SYNTHESSS
Multi-modal image synthesis for given label map is easy for OASIS: we simply re-sample noise like in a conventional unconditional GAN model. Since OASIS employs a 3D noise tensor (64- channels x height width), the noise can be re-sampled entirely ('globally") or only for specific re- gions in the 2D image plane ("locally"). For our visualizations, we replicate single 64-dimensional noise vector along the spatial dimensions for global sampling. For local sampling, we re-sample a new noise vector and use it to replace the global noise vector every spatial position within a stricted area of interest. The results are shown in Figure F The generated images are diverse and of high quality. We observe different degrees of variety for different object classes. For example, buildings change drastically in appearance and often change their spatial orientation with respect to the road. On the other side, many common objects (like tables) vary in color, texture, and illumina- tion, but do not change shapes as they are restricted by the fine details of the region that is outlined for them in the label map.
Local noise re-sampling does not have to be restricted only semantic class areas: in Figure H we sample a different noise vector for the left and right half of the image, well as for arbitrarily shaped regions. In effect, the two areas can differ substantially. However, often a bridging element is found between two areas, such as clouds extending partly from one region to the other region of the image.
B.3 LATENT SPACE INTERPOLATIONS
In Figure I we present images that are the results of linear interpolations in the latent space (see Fig. [D) using OASIS model trained the ADE20K dataset. To generate the images, sample two noise vectors E R&4 and interpolate them with three intermediate points. The images are synthesized for these five different noise inputs while the label map is held fixed. Note that in Figure I we only vary the noise globally, not locally (See Section 3.3 in the main paper). In contrast, Figure T shows local interpolations. For this, we only ra-sam the 3D noise in the area corresponding to single semantic class. The effect is that only the appearance of the selected semantic class varies while the rest of the image remains fixed. It can be observed that strong changes in a local area can slightly affect the surroundings if the local area is also very big. As such, the clouds are slightly different the first and last panel of the mountain row and tree row Figure
We see from Figure I and J that the trajectories in the latent space are smooth and semantically meaningful. For example, we observe transitions from winter to summer, day to night, green trees to leafless trees, shiny parquet to matt carpet, as well as smooth transitions between buildings with different architectural styles.
19

---

Published as a conference paper at ICLR 2021
B.4 APPLICATION TO UNLABELLED DATA
OASIS has a unique property that its discriminator trained an image segmenter. We observed that it shows good performance on this task, reaching the mloU of 40.0 on ADE20K validation set. For comparison, current state of the art on ADE20K mloU of 46.91, achieved by ResNeST (Zhang et al., 2020). Such good segmentation performance allows OASIS to be applied to unla- belled images: given an unseen image without ground truth annotation, OASIS can predict label map via the discriminator. Subsequently feeding this prediction to the generator allows to synthe- size a scene with the same layout but different style. This property shown in Fig. K Due to the good segmentation performance, the recreated scenes closely follow the ground truth label map of the original image. The high sensitivity of OASIS to the 3D noise enforces good variability, so the recreations different from each other. We believe that creating multiple versions of one image while retaining the layout can be useful for data-augmentation.
B.5 LABELMIX
Figure shows additional visual examples of LabelMix regularization, as described in Section 3.2 in the main paper. It can be seen that the discriminator prediction the mixed images often dif- fers from the mix of individual predictions on real and fake images. In particular, regions that are classified real the latter are classified as fake when the images are mixed. This means that the discriminator takes the global context into account for local predictions and thereby often bases the prediction on arbitrary details that should not affect the real-fake class identity. return, the consistency regularization helps to minimize the difference between these two predictions.
20

---

Published as a conference paper at ICLR 2021
Label map
Ground truth
SPADE
CC-FPSE
OASIS
Figure B: Qualitative comparison of OASIS with other methods on ADE20K.
21

---

Published as a conference paper at ICLR 2021
Label map
Ground truth
SPADE
CC-FPSE
OASIS
Figure C: Qualitative comparison of OASIS with other methods on COCO-Stuff.
22

---

Published as a conference paper at ICLR 2021
Label map
Ground truth
Pi Wang al. 2018
SPADE Park et al. (2019)
CC-FPSE Liu et al. (2019)
OASIS
Label map
Ground truth
Pix2pixHID Wang et al. 2018
SPADE Park et al. (2019
CC-FPSE Liu et al. 2019)
OASIS
Label map
Ground truth
Pix2pixHD Wang al. 2018
SPADE Park et al. (2019)
CC-FPSE Liu et al. 2019)
OASIS
Label map
Ground truth
Pi Wang et al. 2018
SPADE Park et al. (2019)
CC-FPSE Liu et al. 2019)
OASIS
Figure D: Qualitative comparison of OASIS with other methods Cityscapes.
23

---

Published as a conference paper at ICLR 2021
Label map
Ground truth
SPADE
SPADE+
OASIS
Figure E: Qualitative comparison of OASIS with SPADE and SPADE+ using ADE20K (row 1-3), COCO-stuff (row 4-6) and Cityscapes (row 7-9).
24

---

Published as a conference paper at ICLR 2021
Label map
OASIS
OASIS 2
OASIS
OASIS 4
OASIS
Global
Local
Global
Local
EAEAS
Global
Local
Global
Local
Figure F: Images generated by OASIS on ADE20K with 256 x 256 resolution using different 3D noise inputs. For each label map the noise re-sampled globally (first row) or locally in the areas marked in red (second row). Note that the images not stitched together but generated in single forward passes.
25

---

Published as a conference paper at ICLR 2021
Label map
Ground truth
SPADE
CC-FPSE
OASIS
Figure G: Failure mode of OASIS. Our model generates diverse images, sometimes producing
object with outlier colors and textures. We compare to Park et al. (2019) and Liu et al. (2019).
Figure H: Images generated by OASIS in one forward pass (no collage), with different noise vectors for different image regions.
26

---

Published as a conference paper at ICLR 2021
Figure Global latent space interpolations between images generated by OASIS for various outdoor and indoor scenes the ADE20K dataset at resolution 256 x 256.
27

---

Published as a conference paper at ICLR 2021
Mountain
Sky
Tree
Water
Building
Floor
Wall
Figure J: Latent space interpolations in local regions of the 3D noise, corresponding to a single semantic class. The noise is only changed within the restricted area. Trained on the ADE20K dataset resolution 256 256.
28

---

Published as a conference paper at ICLR 2021
GT label map
Input image
Segmentation
Recreation
Recreation 2
Recreation
Figure K: After training, the OASIS discriminator can be used to segment images. Columns 1- 3 show the ground truth label map, real image, and segmentation of the discriminator. Using the predicted label map the generator can produce multiple versions of the original image by resampling noise (Recreations 1- 1-3). Note that this alleviates the need of ground truth maps during inference.
29

---

Published as a conference paper at ICLR 2021
Label map
Real image
Fake image
Mask
LabelMixxx..))
DtabelMix
LabelMix
(x,4)
Figure L: Visual examples of LabelMix regularization. Real x and fake & images are mixed using binary mask M, sampled based on the label map, resulting in LabelMix The consistency reg- ularization then minimizes the distance between the logits of DrabelMix, and pa. this visualization, black corresponds to fake class the N+1 segmentation output.
30

---

Published as a conference paper at ICLR 2021
C ARCHITECTURAL AND TRAINING DETAILS
The architecture of OASIS builds upon SPADE Park et al. (2019). the following, we describe in detail our proposed changes the discriminator and the generator.
C.I DISCRIMINATOR ARCHITECTURE
This OASIS discriminator follows a U-Net architecture and is built from ResNet blocks, inspired in their design by Brock et al. (2019). The architecture of the OASIS discriminator is outlined in Table L It has in total 22M learnable parameters and is bigger than the multi-scale PatchGAN discriminator (5.5M) used by SPADE Park et al. 2019). The increased capacity of the OASIS dis- criminator allows to learn more powerful representation and provide more informative feedback to the generator.
Table L: The OASIS discriminator. N refers to the number of semantic classes.
Operation
Input
Size
Output down
Size
ResBlock-Down
image
(3,256,256)
(128.128, 128)
ResBlock-Done Down
down.1
(128,128,
down_2
(128,64.64)
ResBlock-Dowe
down_2
(128,64,64)
down_3
(256,32,32)
ResBlock-Done
down_3
(256,32,32)
down_4
(256,16,16)
ResBlock-Dove -own
down_4
(256,16,16)
down_5
(512,8,8)
ResBlock-Down
down_5
(512,8,8)
down_6
(512,4,4)
ResBlock-Upp
down.
(512,4,4)
up
(512,8,8)
ResBlock-Up
cat (up-1, down.5)
(1024,8,8)
up.2
(256,16,16)
ResBlock-UP Up
cat (up-2, down_4)
(612,16,16)
up_3
(256,32,32)
ResBlock- Up
cat (up_3, down_
(512,32,32)
up_4
(128,64,64)
ResBlock- Up
cat (up-4, down_2)
(256,64.64)
up_5
(128,128, 128)
ResBlock-Up
cat (up_5, down_1)
1256,188,129
up_6
(64,256,256)
Conv2D
up_6
(64,256,256)
out
(N+1,256,256)
C.2 GENERATOR ARCHITECTURE
The generator architecture is built from SPADE ResNet blocks and includes concatenation of 3D noise with the label map along the channel dimension as an option. The generator can be either trained directly the label maps or with 3D noise concatenated to the label maps. The latter option shown M
OASIS generator drops the first residual block used in Park et al. 2019). which decreases the number of learnable parameters from 96M to 72M. The optional 3D noise injection brings additionally 2M parameters. This sampling scheme is five times lighter than the image encoder used by SPADE (10M).
C.3 LEARNING OBJECTIVE TRAINING DETAILS
Learning objective. We train our model with -+lasslass cross entropy as an adversarial loss. Ad- ditionally, the discriminator is regularized with the proposed LabelMix consistency regularization. The full OASIS learning objective thus takes the following form:
HxW
COSSIS = - (E(x.)
Σ
OC
Σ
lig. D(G(2, 0)ss
HxW Σ
HxW
CD
Σ ac
ta log D(x)s.j.c
E(2,))
Σ
log D(G(z, (C)(2)))),s.=444) +
b2
+ ALM
Diogits
LabelMix(x, 3, M) LabelMix (Drogies (2), D Diogits (5), M
where x denotes the real image and (z, t) the noise-label map.
31

---

Published as a conference paper at ICLR 2021
Table M: The OASIS generator. N refers to the number of semantic classes, z is noise sampled from unit Gaussian, y is the label map, interp interpolates given input to the appropriate spatial dimensions of the current layer.
Operation
Input
Size
Output
Size
z_3D
(64,256,256)
Concatenate
(N,256,256)
z-Y
(64+N,256,256)
Conv2D
interp (z.y)
(64+-N,8,8)
(1024,8,8)
(1024,8,8)
SPADE-ResBlock
up_ up_1
(1024,16,16)
interp (z.y)
(64+N,8,8)
up_1
(1024.16.16)
SPADE-ResBlock
up 2
(512,32,32)
interp (z y)
(64+N,16,16)
up_2
(512,32,32)
SPADE-ResBlock
up_3
(256,64,64)
interp (2-y)
(64+NN33233)
up_3
(256,64,64)
SPADE-ResBlock
up_4
(128,128,11)
interp (z-y)
(64+N,64,64)
up_4
(128,128,12)
SPADE-ResBlock
up_5
(64,256,256)
interp (z_y)
(64+N,128 128)
Conv2D, LeakyRelu, TanH
up_5
(64,256,256)
(3,256,256)
Our objective function is different from SPADE. Their model uses hinge adversarial loss and adds the VGG perceptual loss and feature matching loss to train the generator. For an easier comparison, we provide the objective function of SPADE:
SPADE LG
((,)) [tt G(2, + drM (2,,,.)) Σ
(t, - D
N (2,6,, Σ
1 (x) F (G(2
CD ESPADE (r,,)) [min(0.-1) -1 D(t, x)) E(,,) [min(0, -1 -1 log D(t, G(z, t)),
where the pre-trained VGG network.
Training details. We follow experimental setting (Park et al. 2019). The image resolution set to 256x256 for ADE20K and COCO-Stuff and 256x512 for Cityscapes. The Adam (Kingmal & Ba 2015) optimizer was used with momentums 3 = (0, 0.9999 and constant learning rates (0.0001, 0.0004) for G and D. We did not apply the GAN feature matching loss, and used the VGG perceptual loss only for ablations with AvGa 10. The coefficient for LabelMix ÄLM was set to 5 for ADE20k and Cityscapes, and 10 for COCO-Stuff. All our models use exponential moving average (EMA) of the generator weights with 0.9999 decay (Brock et al. 2019). All the experiments were run 4 Tesla V100 GPUs, with a batch size of 20 for Cityscapes, and 32 for ADE20K and COCO-Stuff. The training epochs are 200 on ADE20K and Cityscapes, and 100 for the larger COCO-S Stufff dataset. On average, complete on Ade20k takes around 0.95ms per training image.
pass with batch size 32
32


Reviews:
[{"review_id": "xAEqnGGAJC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-rebuttal:\n\nI acknowledge reading the rebuttal as well as other reviewers comments. I'm satisfied with the rebuttal, I think that the authors have addressed many of my initial comments and I'm happy to increase the score of the paper to 7. If the paper gets accepted to the conference I would encourage the authors to include and expand a discussion about method limitations in the main body of the paper.\n----------------------------------------------------------------\nPre-rebuttal:\n\nThe paper deals with semantic image synthesis and proposes a model that can generate images from segmentation masks. The main methodological contribution of the paper is a segmentation-based discriminator. By using such discriminator, the model is able to learn high quality image syntheses. The proposed pipeline is evaluated on multiple datasets (ADE20K, Cityscapes and COCO-stuff). The qualitative and quantitative results highlight good image quality of the proposed pipeline.\n\n\nPros:\n\nSolid model evaluation including large number of ablations.\nSimple and intuitive pipeline design that is shown to produce high quality image syntheses.\nWell written paper and well structured appendix.\n\n\nCons:\n\nAlthough the synthesized image quality is improved it seems to come at the expense of image diversity.\nThe terms used in the papers are sometimes confusing and could be improved.\nAlthough the paper uses standard evaluation metrics for image generation (such as FID), their use for conditional image synthesis is is not fully justified.\n\n\nDetailed review and comments:\n\nAbstract:\n-> \"semantic image synthesis GAN\" - I'm not convinced with this wording used in the paper. It mixes task (semantic image synthesis) with a type of generative model (GAN). This introduces some confusions in the text, e.g. \"semantic image synthesis GAN models still greatly suffer from poor image quality when trained with only adversarial supervision\" that might be a bit unclear since GANs are always trained with only adversarial supervision. I would recommend to  simply use \"semantic image synthesis\".\n\nIntroduction:\n-> Conditional GANs references: Goodfellow et al 2014 is a good reference for unconditional GAN, not conditional GAN setup. Currently, this reference is linked with other references to conditional reconstruction.\n\n-> Since this is an application oriented paper I think that the introduction would benefit from motivating a bit the task that is under study in this paper. e.g. why semantic image synthesis is of interest to ICLR community?\n\n-> \"Although the perceptual loss substantially improves the accuracy of previous methods, it comes with the computational overhead introduced by utilizing an extra network for training. Moreover, it usually dominates over the adversarial loss during training, introducing a bias towards ImageNet, which can have a negative impact on the diversity and quality of generated images, as we show in our experiments.\" - I'm not convinced if the diversity reduction when using the perceptual loss is due to the bias towards ImageNet. The reduction of diversity is rather an effect of perceptual loss overall. Could the authors comment on this?\n\nPrior work references:\n-> This reference (https://arxiv.org/pdf/2004.03590.pdf) seems relevant. Could the authors comment on this reference in the paper? Adding comparison to this work would make the paper even stronger.\n\nMethodology:\n-> I find the wording of \"3D noise sampling\" a bit unclear. Is it just a resized version of 1D tensor? Is there any importance on how the 3D noise sampling is structured? Could the authors clarify the idea of 3D noise sampling in the main body of the paper?\n\nExperiments:\n-> Could the authors comment why FID is a good metric for semantic image synthesis? Justifying the choice of metrics would make the evaluation section stronger.  From the discussion of the results, (diversity quality tradeoffs) it seems that the paper would benefit from reporting precision-recall like metrics for GANs (e.g. https://arxiv.org/abs/1807.09499, https://arxiv.org/abs/1905.10887, https://arxiv.org/abs/1904.06991). Adding additional metrics would further improve the paper.\n\n-> Table 2. It is unclear to me what is the meaning of red color numbers. Could the authors add clarifications about it to the caption?\n\n-> Table 2. It is bit surprising to see that the model trained only with adversarial loss has relatively low diversity. I would expect that the lack of perceptual loss would lead also to increased diversity in synthesized images. Could the authors comment on what is the limiting factor to increase sample diversity? Could the diversity be affected by the proportion of mixing (real vs. fake)  in LabelMix?\n\n-> Would it be possible to add qualitative results for SPADE+? This model seems to have high diversity scores and it would be interesting to see model samples.\n\n-> The paper lacks discussion on the limitations of the method. Could the authors comment on the potential limitations of the suggested approach?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "sWlHES_DBvN", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors approach the problem of conditional image generation via generative adversarial networks. To this end, they propose an approach that utilizes only semantic segmentation annotations and adversarial loss. No perceptual loss is required. Their discriminator leverages semantic labels to improve the image generations. They evaluate their approach on a variety of datasets including ADE20K, COCO, and CityScapes. They demonstrate substantial quantitative and qualitative performance over baselines and perform an ablation analysis.\n\nPros:\n1. The problem of mask conditioned image generation has immediate applications in computational photography and computer graphics.\n\n2. The paper is well written and easy to understand.\n\n3. Performance is impressive and experimental evaluation is thorough. The authors perform an ablation analysis.\n\n4. The proposed techniques (Segment based discriminator, 3d noise vector) are useful insights into tuning GANs to perform well.\n\nCons:\n1. Technical novelty is limited. The contributions ( segment discriminator and 3D noise) are useful engineering increments but not particularly large insights into the workings of image generation.\n\n2. While the need for VGG based perceptual loss may not be needed here, it seems like the perceptual loss has just been shifted to the spatial realm. There is still a dependence on strong semantic constraints. Instead of using image class labels to train features, the per pixel class labels are used. \n\nIn summary I believe the approach shows promise and important tweaks for image generation networks. Performance is good. However, the technical novelty is limited.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "edh39Rikp7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Post Rebuttal:\n\nThank you for the detailed rebuttal. The comment made about being able to use this in a semi-supervised setting is an exciting direction and I encourage the authors to pursue it on larger less-labeled datasets mentioned in the review in a future work/final submission. I am glad that removing VGG improved the results on COCO.\n\nUltimately, I am keeping my score at 7, accept.\n\n# Pre-Rebuttal: \n\nSummary \n=============================================================================================================\n\nThis paper provides an improvement to the SPADE+ architecture that removes the need for a perceptual loss mechanism while also quantitatively improving the outputs with respect to FID/mmIOU. The paper achieves this while reducing the number of params and removing the ImageNet bias of the original perceptual VGG architecture. The authors evaluate their architecture on 3 datasets: ADE20K, Cityscapes and COCO-stuff. They do while using only adversarial supervision. They achieve this by proposing label mix to train the network by having the discriminator output class labels including an additional \"fake\" label in addition to a variety of other optimizations.\n\nOverall, I vote for accepting this paper. It significantly improves on the SPADE+ architecture by removing a pre-trained component and reducing the number of params. The technique seems to improve upon prior work qualitatively, quantitatively, and performance-wise. \n\nPros\n==============================================================================================================\n1. The paper is very well written.\n1. The paper does provide a very thorough comparison with prior works.\n1. This paper improves on the results of prior methods while reducing the # parameters and the need for imagenet pretraining.\n\nCons: \n==============================================================================================================\n1. Pixel level semantic segmentations are difficult and expensive to generate. This limits the number of datasets that can be used for these work. In this sense, the data is highly supervised. \n1. That being said, I would love to the results on a more challenging dataset like LVIS and/or Open Images. For Open Images, it would be interesting to see if the performance scales with 350+ classes. For the former, it would be interesting to see how well it does on objects that appear very rarely. (I don't expect the results on LVIS to be good, but I would expect them to be better than SPADE+).\n1. Architectural ablations only cover the first two datasets, any reasons why there are none for COCO-stuff?  One of the main claims of the paper, (that this technique does better without VGG), is not evaluated on COCO-stuff.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received 3 reviews with positive ratings: 7,6,7. The reviewers appreciated overall quality of the manuscript, thoroughness of the evaluation, and practical importance of this work (mentioning though that the technical novelty is still not high). They also acknowledged impressive empirical performance. The authors provided detailed responses to each of the reviews separately, which seemed to have resolved the remaining concerns.
As a result, the final recommendation is to accept this work for presentation at ICLR as a poster.
Decision: Accept (Poster)