Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis
Authors: Hao Tang
XIAOJUAN QI
Guolei Sun
Dan Xu
Nicu Sebe
Radu Timofte
Luc Van Gool
Venue: ICLR 2023 poster (2023)
Keywords: Semantic image synthesis
contrastive learning
GANs
edge

Abstract: We propose a novel \underline{e}dge guided \underline{g}enerative \underline{a}dversarial \underline{n}etwork with \underline{c}ontrastive learning (ECGAN) for the challenging semantic image synthesis task. Although considerable improvement has been achieved, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it difficult to synthesize local details and structures. 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results (e.g., missing small objects). 3) Existing semantic image synthesis methods focus on modeling ``local'' semantic information from a single input semantic layout. However, they ignore ``global'' semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. Edge information is produced by a convolutional generator and introduces detailed structure information. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. Doing so can capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts. Experiments on three challenging datasets show that our ECGAN achieves significantly better results than state-of-the-art methods.

Published as a conference paper at ICLR 2023
EDGE GUIDED GANS WITH CONTRASTIVE LEARNING FOR SEMANTIC IMAGE SYNTHESIS
Hao Tang', Xiaojuan Qi' Guolei Sun', Dan Xu², Nicu Sebe*, Radu Timofte Luc Van Gool 'ETH Zurich, University of Hong Kong, "HKUST, *University of Trento, UUniversity of Wurzburg
ABSTRACT
We propose a novel edge guided generative adversarial network with contrastive learning (ECGAN) for the challenging semantic image synthesis task. Although considerable improvement has been achieved, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it difficult to synthe- size local details and structures. 2) The widely adopted CNN operations such as convolution, down-sampling and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, Icading to semantically inconsistent results (e.g., missing small objects). 3) Existing seman- tic image synthesis methods focus on modeling "local" semantic information from single input semantic layout. However, they ignore "global" semantic infor- mation of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose use edge as an in- termediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. Edge information is produced by a convolutional generator and introduces detailed structure information. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose novel contrastive learning method, which aims to enforce pixel em- beddings belonging to the same semantic class to generate more similar image content than those from different classes. Doing so can capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple in- put semantic layouts. Experiments on three challenging datasets show that our ECGAN achieves significantly better results than state-of-the-ae art methods.
INTRODUCTION
Semantic image synthesis refers to generating photo-realistic images conditioned on pixel-level se- mantic labels. This task has wide range of applications such image editing and content genera- tion (Chen & Koltun, 2017; Isola et al., 2017; Guo et al., 2022; Gu et al., 2019; Bau et al., 20;9;;;; Liu et al., 2019; Qi et al., 2018; Jiang et al., 2020). Although existing methods conducted inter- esting explorations, we still observe unsatisfactory aspects, mainly in the generated local structures and details, as well small-scale objects, which believe are mainly due three reasons: 1) Conventional methods (Park et al., 2019; Wang et al., 2018; Liu et al., 2019) generally take the semantic label map as input directly. However, the input label map provides only structural infor- mation between different semantic-class regions and does not contain any structural information within each semantic-class region, making it difficult to synthese rich local structures within each class. Taking label map S in Figure 1 as an example, the generator does not have enough structural guidance to produce a realistic bed, window, and curtain from only the input label (S). 2) The classic deep network architectures tion, non- non-linearity, and up- sa-samnl layers, which will cause the problem of spatial resolution losses of the input semantic labels. 3) Existing methods for this task are typically based on global image- level generation. In other words, they accept a semantic layout containing several object classes and aim generate the appearance of each one using the same network. In this way, all the classes
constructed by stacking convolutional, down- wwwn-sampling normaliza-

---

Published as a conference paper at ICLR 2023
I
S"
Edge Generator G.
Semantic Preserving Module
G2
4
Similarity Loss
Attention Guided Edge Transfer Module G1
Sharing Encoder
S Hoea
9 Fale
8
5 Fageo
E
Image Generator
Modal-Modaity Discriminator
Real or Fake?
Figure 1: Overview of the proposed ECGAN. It consists of a parameter-sharing encoder F,, an edge generator Ge, an image generator G,, an attention guided edge transfer module G,, a label generator G, a similarity loss module, a contrastive learning module G. (not shown for brevity), and a multi- modality discriminator D. Ge and G; are connected by G. from two levels, i.e., edge feature-level and content-level, to generate realistic images. G, is proposed to preserve the semantic information of the input semantic labels. Gi aims transfer the generated image back to the label for calculating the similarity loss. G. tries to capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts. D aims distinguish the outputs from two modalities, i.e., edge and image. The symbol @ denotes channel-wise concatenation.
treated equally. However, because different semantic classes have distinct properties, using specified network learning for each would intuitively facilitate the complex generation of multiple classes.
To address these three issues, in this paper, we propose novel edge guided generative adversarial network with contrastive learning (ECGAN) for semantic image synthesis. The overall framework of the proposed ECGAN is shown in Figure 1.
To tackle 1), we first propose an edge generator to produce the edge features and edge maps. Then the generated edge features and edge maps are selectively transferred to the image generator and improve the quality of the synthesized image by using our attention guided edge transfer module.
To tackle 2), we propose an effective semantic preserving module, which aims at selectively high- lighting class-dependent feature maps according to the original semantic layout. We also propose a new similarity loss to model the relationship between semantic categories. Specifically, given generated label S" and corresponding ground truth S, similarity loss constructs a similarity map to supervise the learning.
To tackle 3), straightforward solution would be to model the generation of different image classes individually. By so doing, each class could have its own generation network structure or parameters, thus greatly avoiding the learning of biased generation space. However, there is fatal disad- vantage to this. That is, the number of parameters of the network will increase linearly with the number of semantie classes N, which will cause memory overflow and make impossible train the model. If we use Pe and Pa to denote the number of parameters of the encoder and decoder, respectively, then the total number of the network parameter should be + xpa since we need new decoder for each class. To further address this limitation, we introduce a pixel-wise contrastive pixel-wise method. By
learning approach that elevates the current image-wise training method leveraging the global semantic similarities present in labeled training layouts, this method leads to the development of well-structured feature space. In this case, the total number of the network pa- rameter only is PPa Moreover, we explore image generation from class-specific context, which beneficial for generating richer details compared to the existing image-level generation methods. A new class-specific pixel generation strategy is proposed for this purpose. It can effectively han- dle the generation of small objects and details, which are common difficulties encountered by the global-based generation.
With the proposed ECGAN, we achieve new state-of of-te-of-the-an results on Cityscapes (Cordts et al., 2016), ADE20K (Zhou al., 2017), and COCO-Stufi (Caesar et al., 2018) datasets, demonstrating the effectiveness of our approach in generating images with complex scenes, and showing signifi- cantly better results compared with state-of-the-ar methods.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Edge Guided Image Generation. Edge maps are usually adopted in image inpainting (Ren et al,, 2019; Nazeri al., 2019a; Li et al., 2019) and image super-resolution (Nazeri et al., 2019b) tasks to reconstruct the missing structure information of the inputs. For example, Nazeri et al. (2019a) proposed an edge generator to hallucinate edges in the missing regions given edges, which can be re- garded as an edge completion problem. Using edge images as the structural guidance. EdgeConnect (Nazeri et al., 2019at 19a) achieves good results even for some highly structured scenes. Unlike previ- ous works, including EdgeConnect, semantic label-to-edge translation. To the best of our knowledge, we are the first to generate edge maps from semantic labels. Then the generated edge maps, with more local structure information, can be used to improve the quality of the image results.
propose a novel edge generator to perform a new task, i.e.,
Semantic Image Synthesis aims to generate a photo-realistic image from a semantic label map (Chen & Koltun, 2017; Park et al., 2019; Liu et al., 2019; Bansal et al., 2019; Zhu et al., 2020a; Ntavelis et al., 2020; Zhu al., 2020b; Sushko et al., 2021; Tan al., 2021b;a; Zhu et al., 2020b). With semantic information as guidance. existing methods have achieved promising performance. However, we can still observe unsatisfying aspects, especially on the generation of the small-scale objects, which we believe is mainly due to the problem of spatial resolution losses associated with deep network operations such as convolution, normalization, down- www-ssamlinng,.. etc. To solve this problem, Park et al. (2019) proposed GauGAN, which uses the input semantic labels to modulate the activations normalization layers through a spatially-adaptive transformation. However, the spatial resolution losses caused by other operations, such as convolution and down- www--samplnng have not been resolved. Moreover, we observe that the input label map has only a few semantic classes in the entire dataset. Thus the generator should focus more on learning these existing semantic classes rather than all the semantic classes. To tackle both limitations, we propose a novel semantic preserving module, which aims to selectively highlight class-dependent feature maps according to the input labels for generating semantically consistent images. We also propose a new similarity loss model the intra-class and inter-class semantic dependencies.
Contrastive Learning, Recently, the most compelling methods for learning representations without labels have been unsupervised contrastive learning (Oord et al., 2018; Hjelm et al., 2018; Pan al., 2021: Wu et al., 2018; Chen et al., 2020), which significantly outperform other pretext task- bask-based al- ternatives (Gidaris et al., 2018; Doersch et al., 2015; Noroozi & Favaro, 2016). Contrastive learning aims to learn the general features of unlabeled data by teaching and guiding the model which data points are different or similar. For example, Hu et al. (2021) designed a region-aware contrastive learning to explore semantic relations for the specific semantic segmentation problem. Zhao et al. (2021) and Wang et al. (2021) also proposed two new contrastive learning- based strategies for the semantic segmentation task. However, for semantic image synthesis in this paper. The semantic image synthesis task is very different from the semantic segmentation task, which requires us to tailor the network structure and loss function to our generation task. Specifically, we propose a new training protocol that explores global pixel relations labeled layouts for regularizing the generation embedding space.
propose novel pixel-wise contrastive learning method
EDGE GUIDED GANS WITH CONTRASTIVE LEARNING
Framework Overview. Figure I shows the overall structure of ECGAN for semantic image synthe- sis, which consists of a semantic and edge guided generator G and multi-modality discriminator D. The generator G consists of eight components: (1) a parameter-sharing convolutional encoder E is proposed produce deep feature maps F; (2) an edge generator Ge adopted to generate edge maps I' taking as input deep features from the encoder; (3) an image generator G; is used to produce intermediate images I': (4) an attention guided edge transfer module G. designed to forward useful structure information from the edge generator to the image generator; (5) the se- mantic preserving module G, is developed to selectively highlight class-dependent feature maps according to the input label for generating semantically consistent images I"; (6) label genera- tor Gj is employed to produce the label from I"; (7) the similarity loss is proposed to calculate the intra-class and inter-class relationships. (8) the contrastive learning module G. aims to model global semantic relations between training pixels, guiding pixel embeddings towards cross-image category- representations that eventually improve the generation performance.
3

---

Published as a conference paper at ICLR 2023
X
CONW
COW
COW
GON.
Edge Generator
F
Attention Guided Edge Transfer Module
CxHxW
DONV
CONV
CONY
CONY
Image Generator
x
Figure 2: Structure of the proposed edge generator Ge. attention guided edge transfer module Gt, and image generator Gz. Ge selectively transfers useful local structure information to G, using the proposed attention guided transfer module G,. The symbols @, 8. and @ denote element-wise addition, element-wise multiplication, and Sigmoid activation function, respectively.
Meanwhile, to effectively train the network, we propose multi multi-modality discriminator D that dis- tinguishes the outputs from both modalities, i.e., edge and image.
3.1 EDGE GUIDED SEMANTIC IMAGE SYNTHESIS
Parameter-Sharing Encoder. The backbone encoder E can employ any deep network architecture, e.g., the commonly used AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2015), and ResNet (He et al., 2016). We directly utilize the feature maps from the last convolutional layer as deep feature representations, i.e., F=E((S,, where E represents the encoder; SER N X is the input label, with H and W width and height of the input semantic labels, and N the total number of semantic classes. Optionally, one can always combine multiple intermediate feature maps enhance the feature representation. The encoder is shared by the edge generator and the image generator. Then, the gradients from the two generators all contribute to updating the parameters of the encoder. This compact design can potentially enhance the deep representations as the encoder can simultaneously learn structure representations from the edge generation branch and appearance representations from the image generation branch.
Edge Guided Image Generation. As discussed, the lack of detailed structure or geometry guidance makes extremely difficult for the generator to produce realistic local structures and details. To overcome this limitation, propose to adopt the edge guidance. A novel edge generator Ge designed to directly generate the edge maps from the input semantic labels. This also facilitates the shared encoder learn more local structures of the targeted images. Meanwhile, the image generator G; aims to generate photo-realistic images from the input labels. In this way, the encoder boosted learn the appearance information of the targeted images.
Previous works (Park et al., 2019; Liu et al., 2019; Qi et al., 2018; Chen & Koltun, 2017; Wang al., 2018) directly use deep networks generate the target image, which is challenging since the network needs to simultaneously learn appearance and structure information from the input labels. In contrast, the proposed method learns structure and appearance separately via the proposed edge generator and image generator. Moreover, the explicit guidance from the ground truth edge maps can also facilitate the training of the encoder. The framework of both edge and image generators is illustrated in Figure 2. Given the feature maps from the last convolutional layer of the encoder, i... C'x x where and W are the width and height of the features, and is the number of channels, the edge generator produces edge features and edge maps which are further utilized to guide the image generator to generate the intermediate image I'. The edge generator G2 contains n convolution layers and correspondingly produces intermediate feature maps F-=(Fİ) 1 After that, another convolution layer with Tanh non-linear activation is utilized to generate the edge map I xx Meanwhile, the feature maps F is also fed into the image generator G; to generate
4

---

Published as a conference paper at ICLR 2023
Figure 3: Left: Overview of the proposed semantic preserving module Ge. which aims at capturing the semantic information and predicts scaling factors conditioned on the combined feature maps F. These learned factors selectively highlight class-dependent feature maps, which are visualized in different colors. The symbols @, 2. and @ denote element- -wise addition, element-wise multiplica- tion, and Sigmoid activation function, respectively. Right: Visualization of three different feature channels in F' on Cityscapes, i.e., road, car, and vegetation.
intermediate feature maps P Then another convolution operation with Tanh non-linear activation is adopted to produce the intermediate image I/ER3*H> In addition, the intermediate edge feature maps F. and the edge map I' are utilized to guide the generation of the image feature maps F; and the intermediate image I' via the Attention Guided Edge Transfer as detailed below.
Attention Guided Edge Transfer. We further propose a novel attention guided edge transfer mod- ule G2 explicitly employ the edge structure information to refine the intermediate image repre- sentations. The architecture of the proposed transfer module G. is illustrated in Figure 2. To trans- fer useful structure information from edge feature maps F to the image feature maps Fi-(F)) the edge feature maps are firstly processed by a Sigmoid activation function to gen- erate the corresponding attention maps F, (=Sigmoii((F.=((RRF - The attention aims provide structural information (which cannot be provided by the input label map) within each semantic class. Then, we multiply the generated attention maps with the corresponding image feature maps obtaín the refined maps, which incorporate local structures and details. Finally, the edge refined features are element-wisse wisely summed with the original image features to produce the final edge refined features, which are further fed to the next convolution layer as =iigmodddd(FF) F +F (j=1..... nn) In this way, the image feature maps also contain the local structure information provided by the edge feature maps. Similarly, to directly employ the structure information from the generated edge map I. for image generation, we adopt the attention guided edge transfer module to refine the generated image directly with edge information as
I' Sigmoid(T)) x + +
where I -i-Sigo (X) is the generated attention map. We also visualize the results in Figure 10.
3.2 SEMANTIC PRESERVING IMAGE ENHANCEMENT
Semantic Preserving Module. Due to the spatial resolution loss caused by convolution, normal- ization, and down-sampling layers, existing models (Wang et al., 2018; Park et al., 2019; Qi et al., 2018; Chen & Koltun, 2017) cannot fully preserve the semantic information of the input labels as illustrated in Figure 7. For instance, the small "pole" is missing, and the large "fence" is incom- plete. To tackle this problem, we propose a novel semantic preserving module, which aims to select class-dependent feature maps and further enhance them through the guidance of the original seman- tic layout. An overview of the proposed semantic preserving module G, is shown in Figure 3(eft). Specifically, the input of the module denoted as F, is the concatenation of the input label S, the generated intermediate edge map I' and image I', and the deep feature F produced from the shared encoder E. Then, we apply a convolution operation on F to produce a new feature map F. with the number of channels equal to the number of semantic categories, where each channel corresponds to a specific semantic category (a similar conclusion can be found in Fu et al. (2019)). Next, we on F. to obtain the global information of each class, followed by a Sigmoid activation function to derive scaling factors 7 as in y (avgmoid(Avgooo(((( where each value represents the importance of the corresponding class. Then, the scaling factor y is adopted to reweight the feature map F. and highlight corresponding class-dependent feature maps. The reweighted feature map is further added with the original feature F. compensate for information loss due to multiplication, and produces F==F. X +J., where N
apply the averaging pooling operation on
After that, we perform another convolution operation on Ft to obtain the feature map I' (CN+++++))) to enhance the representative capability of the feature. In addition, F
5

---

Published as a conference paper at ICLR 2023
Pixel-Wisse Contrastive Learning
C Sasss-Si SpecsSeecffffe. Pixel Generation
Porti Push
Figure 4: Current semantic image synthesis methods learn map pixels an embedding space but ignore intrinsic structures of labeled training data (i.e., inter-layout relations among pixels from the same class, marked with the same color). Our proposed approach, pixel-wise contrastive learning, fosters a new training strategy by explicitly addressing intra-class compactness and inter-class dis- persion. By pulling pixels of the same class closer and pushing pixels from different classes apart, our method can create a better-structured embedding space, which leads to the same class generating more similar image content and improves the performance of semantic image synthesis.
has the same size as the original input one F, which makes the module flexible and can be plugged into other existing architectures without modifications of other parts to refine the output. In Fig- 3rright), visualize three channels in F' on Cityscapes, i.e., road, car, and vegetation. We can easily observe that each channel learns well the class-level deep representations.
Finally, the feature map F' is fed into convolution layer followed by a Tanh non-linear activation layer to obtain the final result 7". Our semantic preserving module enhances the representational power of the model by adaptively recalibrating semantic class-dependent feature maps and shares similar spirits with style transfer (Huang & Belongie, 2017), and SENet (Hu et al., 2018) and EncNet (Zhang al., 20182) One intuitive example of the utility of the module is for the generation small object classes: these classes are easily missed in the generation results due to spatial resolution loss, while our scaling factor can put an emphasis on small objects and help preserve them.
Similarity LOss. Preserving semantic information from isolated pixels is very challenging for deep networks. To explicitly enforce the network to capture the relationship between semantic categories, a new similarity loss is introduced. This loss forces the network to consider both intra-class and inter- class pixels for each pixel in the label. Specifically, state-of-the-art pretrained model (i.c., Seg- Former (Xie et al., 2021)) is used to transfer the generated image I" back to label S" ER x x where N is the total number of semantic classes, and H and W represent the width and height of the image, respectively. A conventional method uses the cross entropy loss between S" and S to address this problem. However, such a loss only considers the isolated pixel while ignoring the semantic correlation with other pixels.
To address this limitation, we construct a similarity map from NxWW Firstly, we reshape S to NERNXM xM where M-HWW Next, we perform a matrix multiplication to obtain a similarity map A= eR MM This similarity map encodes which pixels belong to the same category, meaning that if the j-th pixel and the i-th pixel belong to the same category, then the value of the j-th row and the i-th column in A is 1; otherwise, it is O. Similarly, we can obtain a similarity map A" from the label S" Finally, we caleulate the binary cross entropy loss between the two similarity maps {am EA, mE[1, ] and {a% EA",
, m1[,, }
M
1
Esim 5")
Σ
(am log a² + (1 am) log(1 a%)))-
M
This similarity loss explicitly captures intra-class and inter-class semantic correlation, leading to better generation results.
3.3 CONTRASTIVE LEARNING FOR SEMANTIC IMAGE SYNTHESIS
Pixel-Wite Contrastive Learning. Existing semantic image synthesis models use deep networks to map labeled pixels non-linear embedding space. However, these models often only take

---

Published as a conference paper at ICLR 2023
into account the "local" context of pixel samples within an individual input semantic layout, and fail to consider the "global" context of the entire dataset, which includes the semantic relationships between pixels across different input layouts. This oversight raises important question: what should the ideal semantic image synthesis embedding space look like? Ideally, such a space should not only enable accurate categorization of individual pixel embeddings, but also exhibit well- structured organization that promotes intra-class similarity and inter-class difference. That is, pixels from the same class should generate more similar image content than those from different classes in the embedding space. Previous approaches to representation learning propose that incorporating the inherent structure of training data can enhance feature discriminativeness. Hence, we conjecture that despite the impressive performance of existing algorithms, there is potential to create a more well-structured pixel embedding space by integrating both the local and global context.
The objective of unsupervised representation learning is to train an encoder that maps each training semantic layout S to a feature vector ==B(S), where B represents the backhone encoder network. The resulting vector should be accurate representation of S. To accomplish this task, contrastive learning approaches use a training method that distinguishes a positive from multiple negatives, based the similarity principle between samples. The InfoNCE (Oord et al., 2018; Gutmann & Hyvärinen, 2010) loss function, a popular choice for contrastive learning, can be expressed as
exp(w (00(u-24/1 (+/7)
Ls = log exp(v 1.00-u4/10 v+/T) + Ev_ ENS EN, exp(v U_/T)²
(3)
where U4 represents an embedding of positive for S, and Ng includes embeddings of negatives. The symbol "." refers to the inner (dot) product, and T>00 is temperature hyper-parameter. It is worth noting that the embeddings used the loss function are normalized using the I.2 method.
One limitation of this training objective design is that it only penalizes pixel-wise predictions inde- pendently, without considering the relationno between pixels. To overcome this limitation, we take inspiration from (Wang et al., 2021; Khosla et al., 2020) and propose contrastive learning method that operates at the pixel level and is intended to regularize the embedding space while also investigating the global structures present in the training data (see Figure 4). Specifically, our con- trastive loss computation uses training semantic layout pixels as data samples. For a given pixel i with its ground-trute truth semantic label c, the positive samples consist of other pixels that belong to the same class c, while the negative samples include pixels belonging to other classes C\\c. As a result, the proposed pixel-wise contrastive learning loss is defined follows
1 P./
exp(i- ii4/T)
L
log exp(i exp(i-i4/7) 4/7)
(4)
ΣL_EN. exp(i-i_/7))
L.P.
For each pixel i, we use P; and N; to represent the pixel embedding collections of positive and negative samples, respectively. Importantly, the positive and negative samples and the anchor i are not required to come from the same layout. The goal of this pixel-wise contrastive learning approach is to create an embedding space in which same-class pixels are pulled closer together, and different- class pixels are pushed further apart. The result of this process is that pixels with the same class generate image contents that are more similar, which can lead to superior generation performance.
Class-Specific Pixel Generation. To overcome the challenges posed by training data imbalance between different classes and size discrepancies between different semantic objects, we introduce approach that is specifically designed to generate small object classes and fine details. Our proposed method is a class-specific pixel generation approach that focuses on generating image content for each semantic class. By doing so, classes during joint optimization, and each subgeneration branch can concentrate on a specific class generation, which in turn results in similar generation quality for different classes and yields richer local image details.
avoid the interference from large object
An overview of the class-specific pixel generation method is provided in Figure After the pro- posed pixel-wise contrastive learning, we obtain a class-specific feature map for each pixel. Then, the feature map is fed into a decoder for the corresponding semantic class, which generates an output image 7, Since we have the proposed contrastive learning loss, we can use the parameter-shared decoder to generate all classes. To better learn each class, we also utilize pixel-wise L1 recon- struction loss, which can be expressed LL, DA B,,, [I] [.]]]. The final output I, from
7

---

Published as a conference paper at ICLR 2023
Input Layour
SOTA Mathod
Curs
Ground Truth
Input Layout
SOTA Method
Oura
Ground Truth
Figure Existing SOTA method (i.e., OASIS) vs. our proposed ECGAN on three datasets. Cityscapes: left; ADE20K: top right two; COCO-Stuff: bottom right two.
Table I: User study Cityscapes, ADE20K, and COCO-Stuff. The numbers indicate the percent-
age of users who favor the results of the proposed method over the competing methods.
AMT f
Cityscapes 3.4
ADE20K
COCO-Stuff
Ours vs. CRN (Chen Koltun, 2017)
94.8 + 2.7
95.3 2.1
Ours vs. Pix2pixHD (Wang et al., 2018)
87.2 8.8
93.6 + 3.1
93.9 + 2.4
Ours vs. SIMS (Qi et al., 2018)
Ours vs. GauGAN (Park et al., 2019)
84.7
86.2 + 8.74
90.8 + 2.5
Ours vs. DAGAN (Tang et al., 2020a)
8.9
Ours vs. CC-FPSE (Liu al., 2019)
79.5 4.2
+ 3.9
86.7 2.8
Ours vs. LGGAN (Tang et al., 2020b)
78.4
82.7 + 4.5
Ours vs. OASIS (Sushko et al., 2021)
76.7 4.8
80.6 + 4.5
82.5
the pixel generation network can be obtained by performing element-wise addition of all the class-specific outputs: I- @ Tgp @ + IgN-
EXPERIMENTS
Datasets and Evaluation Metrics. We follow GauGAN (Park et al., 2019) and conduct experiments on three datasets, i.e., Cityscapes (Cordts et al., 2016), ADE20K (Zhou et al., 2017), and COCO- Stuff (Caesar et al., 2018). We employ the mean Intersection-ove--Unioo (mloU), Pixel Accuracy (Acc), and Fréchet Inception Distance (FID) (Heusel al., 2017) the evaluation metrics.
State-of-the-Ar Comparisons. We adopt GauGAN as the encoder F. to validate the effectiveness of the proposed method. Visual comparison results all three datasets with the SOTA method are shown in Figure 5. ECGAN achieves visually better results with fewer visual artifacts than the SOTA method. For instance, scene datasets such Cityscapes and ADE20K, ECGAN generates sharper images than the baseline, especially at local structures and details.
Moreover, we follow the same evaluation protocol as GauGAN and conduct a user study. Specif- ically, we provide the participants with an input layout and two generated images from different models and ask them to choose the generated image that looks more like a corresponding image of the layout. The users are given unlimited time to make the decision. For each comparison, we randomly generate 400 questions for each dataset, and each question is answered by different par- ticipants. For other methods, we use the public code and pretrained models provided by the authors to generate images. As shown Table 1, users favor our synthesized results on all three datasets compared with other competing methods, further validating that the generated images by ECGAN are more natural. Although the user study is more suitable for evaluating the quality of the generated images, also follow previous works and use mIoU, Acc, and FID for quantitative evaluation. The
8

---

Published as a conference paper at ICLR 2023
Table 2: Quantitative comparison of different methods on Cityscapes, ADE20K, and COCO-Stuff.
Citssapes
ADEZOK
COCO-Stuf"
Method
mIoU
Acc
mloUt
Ace 1
FID L
mIoU 1
Acc f
FID 1
CRN (Chen Koltun, 2017)
77.1
104.7
22.4
68.8
73.3
23.7
40.4
70.4
SIMS (Qi al., 2018)
75.5
Pix2pixHD (Wang el. 2018)
58.3
81.4
95.0
20.3
69.2
81.8
14.6
45.8
111.5
GauGAN (Park et al, 2019)
62.3
81.9
71.8
38.5
79.9
33.9
37.4
67.9
22.6
DPGAN (Tang Sebe, 2021)
65.2
82.6
53.0
39.2
80.4
31.7
DAGAN (Tang
2020a)
66.1
31.9
SelectionGAN (Tang al., 2019)
83.8
82.4
65.2
40.1
81.2
33.1
SeloctionGANN+ (Tang et al., 20226)
64.5
82.7
63.4
41.7
81.5
32.2
LGGAN (Tang al., 2020b)
68.4
83.0
57.7
41.6
81.8
31.6
LGGAN+ (Tang al., 2022n)
67.7
41.4
81.5
30.5
CC-FPSE (Liu et al., 2019)
65.5
82.3
54.3
43.7
82.9
31.7
41.6
70.7
19.2
OASIS (Sushko et al., 2021)
69.3
47.7
48.8
28.3
44.1
17.0
RESAIL (Shi et al., 2022)
69.7
83.2
45.5
49.3
30.2
44.7
73.1
18.3
SAFM (Lvv 2022)
70.4
83.1
49.5
86.6
32.8
43.3
73.4
24.6
ECGAN (Ours)
72.2
83.1
44.5
50.6
83,1
25.8
46.3
70.5
15.7
Table Ablation study of the proposed ECGAN on Cityscapes, ADE20K, and COCO-Stuff.
Cityscapes
ADE20K
COCO-Stuff
Setting
mloU1
Acc f
FID
mluU
Acc 1
FID L
mloU
Acc
FID 4
B1
E+G.
58.6
81.4
65.7
36.9
78.5
38.2
36.8
65.1
24.5
B2
E+G.+G.
60.2
81.7
61.0
38.7
79.2
36.3
37.5
66.3
22,9
B3
E+G.+Ge4G;
61.5
82.0
59.0
40.6
80.3
34.6
39.1
67.0
21.7
EAG,4G
57.1
42.0
32.4
41.4
68.2
19.8
EAG.AGAAGAAGAAG
66.8
82.7
52.2
45.8
82.4
29.9
43.7
17.6
B6
BGG.GG.AGGGGGGGHH
72.2
83.1
44.5
50.6
83.1
25.8
46.3
70.5
15.7
results of the three datasets are shown in Table The proposed ECGAN outperforms other leading methods by a large margin on all three datasets, validating the effectiveness of the proposed method.
Ablation Study. We conduct extensive ablation studies three datasets evaluate different com- ponents of the proposed ECGAN. Our method has six baselines (i.e., B1, B2, B3, B4, B5, B6) as shown in Table 3: B1 means only using the encoder E and the proposed image generator G, to synthesize the targeted images. B2 means adopting the proposed image generator G; and edge gen- erator G. to simultaneously produce both edge maps and images. B3 connects the image generator G₁ and the edge generator Ge by using the proposed attention guided edge transfer module G.. B4 employs the proposed semantic preserving module G, to further improve the quality of the final results. B5 uses the proposed label generator G1 to produce the label from the generated image and then calculate the similarity loss between the generated label and the real one. B6 is our full model and uses the proposed pixel-wis contrastive learning and class-specific pixel generation methods to capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts. As shown in Table 3, each proposed module improves the performance all three metrics, validating the effectiveness. We also provide more ablation analysis in Appendix.
5 CONCLUSION
We propose a novel ECGAN for semantic image synthesis. It introduces four core components: edge guided image generation strategy, attention guided edge transfer module, semantic preserving module, and contrastive learning module. The first one is employed to generate edge maps from input semantic labels. The second one is used to selectively transfer the useful structure information from the edge branch to the image branch. The third one is adopted to alleviate the problem of spatial resolution losses caused by different operations in the deep nets. The last one utilized to investigate global semantic relations between training pixels, guiding pixel embeddings towards cross-ine pross-image category-disee dispryiiiiirimiativeee representations. Extensive experiments on three datasets show that ECGAN achieves significantly better results than existing models.
6 ACKNOWLEDGMENT
This work was partly supported by ETH General Fund, the Alexander von Humboldt Foundation, and the EU H2020 project A14Media under Grant 951911.

---

Published as a conference paper at ICLR 2023
REFERENCES
Aayush Bansal, Yaser Sheikh, and Deva Ramanan. Shapes and context: in-the-wild image synthesis & manipulation. In CVPR, 2019.
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with generative image prior. ACM TOG, 38(4):1-11, 2019a.
David Bau, Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial net- works. In ICLR, 2019b.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018. 2,
John Canny, computational approach edge detection. IEEE TPAMI, (6):679-698, 1986.
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In ICCV, 2017. 3, 4, 5, 8,
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020. 3
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016. 2, 8
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In ICCV, 2015.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 5
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014. 14
Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, and Lu Yuan. Mask-guided portrait editing with conditional gans. In CVPR, 2019.
Jiaxian Guo, Jiachen Li, Huan Fu, Mingming Gong, Kun Zhang, and Dacheng Tao. Alleviating se-
manties distortion in unsupervised low-level image-to-image translation via structure consistency constraint. In CVPR, 2022.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In AISTATS, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017. 8
Devon Hjelm, Alex Fedorov, Samuel Marchilon,, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018. 3
Hanzhe Hu, Jinshi Cui, and Liwei Wang. Region-aware contrastive learning for semantic segmen- tation. In ICCV, 2021.
Jie Hu, Li Shen, and Gang Sun. Squeezz-and-exxciatioo networks. In CVPR, 2018.
10

---

Published as a conference paper at ICLR 2023
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In ICCV, 2017. 6
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with
conditional adversarial networks. In CVPR, 2017. 1
Liming Jiang, Changxu Zhang, Mingyang Huang, Chunxiao Liu, Jianping Shi, and Chen Change Loy. Tsit: A simple and versatile framework for image-to-image translation. In ECCV, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NeurlPS, 2020. 7
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 14
Alex Krizhevsky, Iya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012. 4
Jingyuan Li, Fengxiang He, Lefei Zhang, Bo Du, and Dacheng Tao. Progressive reconstruction of visual structure for image inpainting. In ICCV, 2019. 3
Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, et al. Learning to predict sorouttto-imagee
conditional convolutions for semantic image synthesis. In NeurlPS, 2019. 4,
Zhengyao Lv, Xiaoming Li, Zhenxing Niu, Bing Cao, and Wangmeng Zuo. Semantic-shape adaptive feature modulation for semantic image synthesis. In CVPR, 2022.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018. 13
Kamyar Nazeri, Eric Ng. Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Struc- ture guided image inpainting using edge prediction. In ICCV Workshops, 2019a. 3
Kamyar Nazeri, Harrish Thasarathan, and Mehran Ebrahimi. Edge-informed single image super- resolution. In ICCV Workshops, 2019b.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV. 2016. 3
Evangelos Ntavelis, Andrés Romero, Iason Kastanis, Luc Van Gool, and Radu Timofte. Sesame: Semantic editing of scenes by adding, manipulating erasing objects. In ECCV, 2020. 3
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arxiv preprint arXiv: 2018. 3,
Tian Pan, Yibing Song. Tianyu Yang, Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. CVPR, 2021.
Taesung Park, Ming-Yi Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu, Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019. 1, 4, 5, 9, 14, 15
Xiaojuan Qi, Qifeng Chen, Jiaya Jia, and Vladlen Koltun. Semi Semi-parametrri image synthesis. In CVPR, 2018. 1, 5, 9,
Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li. Shan Liu, and Ge Li. Structureflow: Image inpainting via structure-aware appearance flow. In ICCV, 2019. 3
Yupeng Shi, Xiao Liu, Yuxiang Wei, Zhongqin Wu, and Wangmeng Zuo. Retrieval-basee spatially adaptive normalization for semantic image synthesis. In CVPR, 2022. 9
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Vadim Sushko, Edgar Schönfeld, Dan Zhang, Juergen Gall, Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In ICLR, 2021. 9, 15

---

Published as a conference paper at ICLR 2023
Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Bin Liu, Gang Hua, and Nenghai Yu. Diverse semantic image synthesis via probability distribution modeling. In CVPR, 2021a.
Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He, Lu Yuan, Gang Hua, and Nenghai Yu. Efficient semantic image synthesis via class-adaptive normalization. IEEE TPAMI, 2021b. 3
Hao Tang and Nicu Sehe. Layout-to-image translation with double pooling generative adversarial networks. IEEE TTP, 30:7903-7913, 2021. 9
Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J Corso, and Yan Yan. Multi-channel attention selection gan with cascaded semantic guidance for cross- viess image translation. In CVPR, 2019. 9, 13
Hao Tang, Song Bai, and Nicu Sebe. Dual attention gans for semantic image synthesis. In ACM MM, 2020a. 8,
Hao Tang, Dan Xu, Yan Yan, Philip HS Tom, and Nicu Sebe. Local class-specific and global image- level generative adversarial networks for semantic-guided scene generation. In CVPR, 2020b. 8,
Hao Tang, Ling Shao, Philip HS Torr, and Nicu Sebe. Local and global gans with semantic-aware upsampling for image generation. IEEE TPAMI, 45(1):768-784, 2022a. 9
Hao Tang, Philip HS Torr, and Nicu Sebe. Multi-channe attention selection gans for guided image- to-image translation. IEEE TPAMI, (01):1-16, 2022b. 9
Ting- Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. CVPR, 2018. 1, 4,
Wenguan Wang, Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu, and Luc Van Gool. Explor- ing cross-image pixel contrast for semantic segmentation. In ICCV, 2021. 3,
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, 2018.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, 2018. 14
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. NeurlPS, 2021.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In ICCV, 2015. 17
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In CVPR, 2017. 14
Hang Zhang, Kristin Dana, Jianping Shi, Zhongyue Zhang, Xiaogang Wang, Ambrish Tyagi, and Amit Agrawal. Context encoding for semantic segmentation. In CVPR, 2018a.
Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018b. 15
Xiangyun Zhao, Raviteja Vemulapalli, Philip Andrew Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and Ying Wu. Contrastive learning for label efficient semantic segmentation. In ICCV, 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 2, 8
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region-adaptive normalization. In CVPR, 2020a. 3
Zhen Zhu, Zhiliang Xu, Ansheng You, and Xiang Bai. Semantically multi-modas image synthesis. In CVPR, 2020b.
12

---

Published as a conference paper at ICLR 2023
Appendix
A MODEL TRAINING
Multi-Modality Discriminator. To facilitate the training of the proposed method for high-quality edge and image generation, a novel multi-modality discriminator is developed simultaneously distinguish outputs from two modality spaces, i.c., edge and image. Since the edges and RGB images share the same structure, they can be learned using the multi-modality discriminator. In the preliminary experiment, also tried and an image discriminator). but no performance improvement was observed while increasing the model complexity. Thus, multi-modality discriminator is shown in Figure I, which is capable of discriminating both real/fake images and edges. To discriminate real/fake edges, the discriminator loss considering the semantic label S and the generated edge I' (or the real edge Ie) is as
two discriminators (i.e., edge discriminator
the proposed multi- monllityodaltyy discriminator. The framework of the
LCGAN(Ge, D) = IEs.I. [log D(S, + E.. [log(1 D(S,
(5)
which guides the model distinguish real edges from fake generated edges. Further, discriminate real/fake images, the discriminator loss regarding semantic label S and the generated images I', 1" (or the real image T) as Eq. 6, which guides the model to discriminate real/fake images,
LOGAN (G,, G., D) (A L) [log DSS, + AEss,r" [log(1 - D(S, ["))]
Es.r [log(1 - D(S, 1'))]
(6)
where À controls the losses of the two generated images. The inclusion of I' and I" is a cascaded coarse-to-fim generation strategy (Tang et al., 2019), i.c., I' is the coarse result, while I" is the refined result. The intuition is that I" will be better generated based on I', so we provide I' to the discriminator ensure that I' is also realistic.
Optimization Objective. Equipped with the multi-modality discriminator, we elaborate on the training objective for the proposed method follows. Five different losses, i.e., the multi-modas modality adversarial loss, the similarity loss, the contrastive learning loss, the discriminator feature matching loss Ly, and the perceptual loss C, are used to optimize the proposed ECGAN,
min max C do (CoGGANGGe D) (G,, Gs, +No Loim (sim(s Csim (S,S")
1)
Multi-Modality Adversarial Loss
Similarity Loss
A
Li + Liy
+X9 (Cy 1y) (T, (T,(T,
Contrastive Learning Loss
Discriminator Feature Matching Loss
Xp (Cp(Tes (T. (T,
Perceptual Loss
where Ae As A₁. Ay. and Ap are the parameters of the corresponding loss that contributes to the total loss L; where Ly matches the discriminator intermediate features between the generated im- ages/edges and the real images/edges; where Lp matches the VGG extracted features between the generated images/edges and the real images/edges. By maximizing the discriminator loss, the gen- erator is promoted to simultaneously generate reasonable edge maps that can capture the local-aware structure information and generate realistic images semantically aligned with the input labels.
B IMPLEMENTATION DETAILS
For both the image generator G; and edge generator Ge, the kernel size and padding size of con- volution layers are all 3x3 and 1 for preserving the feature map size. We set n=3 for generators G. G.. and G, The channel size of feature F is set to C=64. For the semantic preserving module G., we adopt an adaptive average pooling operation. Spectral normalization (Miyato et al., 2018) 018)& is applied to all the layers in both the generator and discriminator. We adopt the Canny edge detector (Canny, 1986) extract edge maps for training. During the testing phase, do not need to additional data. Thus the comparison with existing methods is fair. When calculating the contrastive learning loss, found that the more layouts used, the better the performance is. When more
13

---

Published as a conference paper at ICLR 2023
than 8 layouts used as imput, the performance improvement not obvious, but the training of the whole model will become slow. Therefore, considering the balance between performance and time, finally choose 8 layouts imput calculate the contrastive learning loss.
Also, we follow the training procedures of GANs (Goodfellow ct al., 2014) and alternatively train the generator G and discriminator D, i.e., gradient descent step the discriminator and generator alternately. We use the Adam solver (Kingma & Ba, 2015) and set [1 =0 32=0.999. Ae. A. A y and Ap in Eq. is set to 1, 10, and 10, respectively. All in both Eq. and are 2. We conduct the experiments on an NVIDIA DGXI with V100 GPUs.
C ADDITIONAL RESULTS
Input Label
Attention Map
Edge Map
Ground Truth
Our Result
Ground Truth
Figure 6: Edge and attention maps generated by the proposed method.
Visualization of Edge and Attention Maps. We also visualize the generated edge and attention maps in Figure 6. We observe that the proposed method can generate reasonable edge maps accord- ing to the input labels. Thus the generated edge maps can be used to provide more local structure information for generating more photo-realistic images.
Visualization of Segmentation Maps. We follow GauGAN and apply pre-trained segmentation networks (Yu et al., 2017; Xiao et al., 2018) on the generated images to produce segmentation maps. Results compared with the baseline method are shown in Figure 7. We observe that the proposed method consistently generates better semantic labels than the baseline on both datasets.
Baseline
Label by Ours
Label by Curs
Figure 7: Segmentation labels generated by the baseline and our proposed method. "Ours I" and "Ours II" stand for I' and I", respectively.
Multi-Modal Image and Edge Synthesis. We follow GauGAN (Park et al., 2019) and apply a style encoder and KL-Divergence loss with loss weight of 0.05 enable modtil image and edge synthesis. As shown in Figure 8, our model generates different edges and images from the same input layout, which we believe will benefit other tasks, e.g., image inpainting and super-resolution.
14

---

Published as a conference paper at ICLR 2023
Input Label
Generated Edges
Generated Images
Figure 8: Results generated by the proposed method for multi-modal image and edge synthesis.
Table 4: Multi-modal synthesis evaluation on ADE20K.
Method
Multi-Modal
LPIPS 1
GauGAN+ (Park et al., 2019)
Encoder
0.16
GauGAN+ (Park et al., 2019)
3D Noise
0.50
OASIS (Sushko et al., 2021)
3D Noise
0.35
ECGAN (Ours)
Encoder
0.18
ECGAN (Ours)
3D Noise
0.52
Table 5: mloU of small objects on Cityscapes.
mloU 1
Pole
Light
Sign
Rider
Mbike
Bike
Overall
OASIS (Sushko et al., 2021)
23.4
32.6
14.9
27.3
31.2
26.6
26.0
ECGAN (Ours)
26.2
36.7
17.4
30.2
33.5
28.7
28.8
Moreover, we follow OASIS (Sushko et al., 2021) and use LPIPS (Zhang et al., 2018b) to evaluate the variation in the multi-model image synthesis the ADE20K dataset. Following in OASIS, generate 20 images and compute the mean pairwise scores, and then average over all label maps. The higher the LPIPS scores, the more diverse the generated images are. We follow OASIS and GauGAN, and employ two settings (i.e., encoder and 3D noise) to evaluate multi-modal image syn- thesis. Table shows that the proposed ECGAN achieves better results than OASIS and GauGAN in both settings. Note that existing methods (e.g., OASIS (Sushko et al., 2021) and GauGAN (Park al., 2019)) can only achieve multi-modal image synthesis.
Evaluation Focused on Small Objects. We report mloU on six small object categories of Cityscapes (i.e., pole, light, sign, rider, mbike, and bike) in Table Our ECGAN generates better mloU than the SOTA method (i.e., OASIS (Sushko et al., 2021)) on all these small object classes. We also show visualization results in Figure 9, clearly confirming that the proposed method highly capable of preserving small objects in the output.
SOTA Method
ECGAN (Ours)
SOTA Method
ECGAN (Ours)
Ground Truth
Figure 9: Visualization of small objects Cityscapes.
D ADDITIONAL ABLATION STUDY
The ablation study results shown in Table 3.
15

---

Published as a conference paper at ICLR 2023
Input Layout
I
r
Figure 10: Visualization of the differences after the edge-guuided refinement in Eq. 1.
Effect of Edge Guided Generation Strategy. When using the edge generator Ge to produce the corresponding edge map from the input label, performance all evaluation metrics is improved. We also provide several visualization results of the differences (see Eq. 1) after the edge-guided refinement in Figure 10.
Effect of Attention Guided Edge Transfer Module. We observe that the implicitly learned edge structure information by the "E+G,+G," baseline is not enough for such challenging task. Thus we further adopt the transfer module G, to transfer useful edge structure information from the edge generation branch the image generation branch. We observe that performance gains obtained on the mloU, Acc, and FID metrics in all three datasets. This means that Ge indeed learns rich feature representations with more convincing structure cues and details and then transfers them from the generator Ge to the generator G;-
Effect of Semantic Preserving Module. By adding G., the overall performance is further boosted on all the three datasets. This means G, indeed learns and highlights class-specific semantic feature maps, leading to better generation results. In Figure 7, we show some samples of the generated semantic maps. We observe that the semantic maps produced by the results with G, (i.e., "Label by Ours Figure 7) are more accurate than those without using G, ("Label by Ours 1" in Figure 7). Moreover, visualize three channels in F' Cityscapes in Figure 3(right), i.e., road, car, and vegetation. Each channel learns well the class-level deep representations.
Effect of Similarity Loss. By adding the proposed label generator G1 and similarity loss, the overall performance is further boosted on all three metrics. This means the proposed similarity loss indeed captures more intra-class and inter-class semantic dependencies, leading better semantic layouts in the generated images.
Effect of Contrastive Learning. When adopting the proposed pixel-wise contrastive learning and class-specific pixel generation methods to produce the results, the performance is significantly im- proved on all three datasets on all three evaluation metrics. This means that the model does indeed learn more discriminative class-specific feature representation, confirming the superiority of our design.
Input Label
HED Edge
Canny Edge
HED Result
Canny Result
Ground Truth
Figure 11: HED vs. Canny edge extraction.
16

---

aim Published as a conference paper at ICLR 2023
Effect of Edge Extraction Methods. We also conduct experiments Cityscapes with HED (Xie & Tu, 2015), leading to the following results: 56.7 (FID), 64.5 (mloU), and 82.3 (Acc), which are slightly worse than the results of Canny in Table 3. The reason is that the edges from HED are very thick and cannot accurately represent the edge of objects. It also ignores some local details since it focuses extracting the contours of objects. Thus, HED is unsuitable for our setting to generate more local details/structures. Moreover, we see that the generated HED edges contain artifacts, as indicated in the red boxes in Figure 11, which makes the generated images tend have blurred edges.
17


Reviews:
[{"review_id": "rlxmLogqVZK", "reviewer": "Reviewer_3rGQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To address semantic image synthesis, this paper mainly proposes to tackle three issues of lack of details from semantic lables, spatial resolution loss from CNN operations, and ignoring 'global' semantic information from a single input semantic layout, with the design of edge guided generative adversarial network (GAN), semantic perserving module, and pixel-wise contrastive learning, respectively. This work seems complicated combining some existing techniques with some new changes.", "summary_of_the_review": "Considering the weaknesses of limited novelty and insufficient evaluation, as well as the issues of clarity and reproducibility, I lean towards rejecting.\n\n------\nAfter discussion with AC and the other reviewers, I agree to marginally accept the paper.", "strength_and_weaknesses": "Strenghts:\n  - The writing is good.\n  - The raised three issues make some sense for semantic image synthesis.\n\nWeaknesses:\n  - The design is complicated and the novelty is limited. The proposed method, namely ECGAN, contains a lot of CNN-based modules with different given functions, for example, Edge Generator for producing edge maps from semantic maps, Image Generator for generating intermediate images from semantic maps, as well as Semantic Preserving Module and Multi-Modality Discriminator, etc., but actually, it's not sure or clear how these modules perform as the given definitions. More importantly, for the three solutions dealing with the three issues, the contributions are limited, for instance, edge guided semantic image synthesis is not new (\"Edge Guided GANs with Semantic Preserving for Semantic Image Synthesis\", arXiv:2003.13898), and the similar idea on pixel-wise contrastive learning has been explored on semantic segmentation (\"Contrastive Learning for Label Efficient Semantic Segmentation\", ICCV 2021; \"Region-Aware Contrastive Learning for Semantic Segmentation\", ICCV 2021). \n  - The evaluation is not sufficient without considering the state-of-the-art methods. The state-of-the-art methods about semantic image synthesis include the works published on the recent CVPR 2022, like \"Retrieval-based Spatially Adaptive Normalization for Semantic Image Synthesis\", \"Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesis\", as well as \"Alleviating Semantics Distortion in Unsupervised Low-Level Image-to-Image Translation via Structure Consistency Constraint\". Moreover, the ablation study in the Appendix is somehow insufficient to validate the efficacy of the design, since it's better to show that each component indeed performs as given definitions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "r_dbGPnJBL", "reviewer": "Reviewer_UoWk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a ECGAN to advance semantic image synthesis. Three ideas were proposed in this paper to jointly boost the performance: the edge map with attention modules to guide image generation, the semantic preserving module together with the similarity loss on the semantic layout, and a pixel-wise contrastive learning mechanism. Experiments were done on commonly used datasets such as Cityscapes, ADE20K, and COCO-Stuff and output quality is measured by mIoU, Acc, FID and user studies. As shown in the paper, the proposed approach consistently outperforms existing benchmarks. ", "summary_of_the_review": "Overall, this paper presented a new SOTA for semantic image synthesis. The ideas are interning and the proposed system is novel. The evaluation is comprehensive.  ", "strength_and_weaknesses": "Strengths:\n- The proposed ideas leveraged additional labels such as edges and maximized the semantic layout label usage, which are interesting ideas. Jointly using various types of labels and proving those labels are helpful to boost performance are important contributions to peers and the community.\n- Benchmark results consistently improve upon existing ones.\n- Ablation studies are done comprehensively\n- The motivation and approach are presented clearly and the paper is easy to read.\n\nWeaknesses:\n- The proposed approach leverages additional information, so it might not be fair comparison with existing methods. Given the scale of the training dataset, overfitting might exist. It would be interesting if authors could test the model on images in the wild or at least beyond the Cityscapes, ADE20K, and COCO-Stuff.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "j4U3wbPtxSa", "reviewer": "Reviewer_AiR8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a semantic image synthesis method by generating edge as an intermediate state to guide the following generation. Meanwhile, a semantic preserving module is designed to selectively choose class-dependent feature maps. One contrastive learning strategy by considering different layout inputs is proposed to encourage generating similar content for same semantic class. The experiments show convincing qualitative and quantitative results to demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "This is an encouraging work that pushes the direction of more complex scene generation. The idea of contrastive learning by taking multiple layout into account is of some values. ", "strength_and_weaknesses": "+ Focus on more detailed local structures which are missing in the global layout input\n+ SOTA results compared with existing counterparts, especially on the challenging COCO dataset\n+ A novel semantic preservation module to reduce spatial resolution loss caused by generic CNNs\n\nI just have a few following minor concerns:\n\nHow is the edge generator learned? Is it trained by some edge GT data or totally learned in an unsupervised way? More details about losses or designs are needed as it is one of the major contributions in this work\n\nHow many layout inputs are considered together when introducing the contrastive learning loss? How does this hyper-parameter affect the performance? Will the contrastive learning reduce diversity for a certain class that is by itself able to have multiple possible outputs? For example, for person class, it is likely to synthesize person in different clothing textures.\n\nWhich method in Figure 5 is the SOTA method?\n\nLooks the user study in Table 1 is A/B test. Better to also report with the confidence intervals.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eCxo4PV1Nig", "reviewer": "Reviewer_QAjR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel semantic image synthesis model (semantic segmentation map in, RGB image out) based on adversarial training. \nThe main idea is to use edge maps as a structure to guide the image synthesis process. \nTo this end a special edge-generating branch is used to generate an edge map from the semantic input map.\nThe image generating branch leverages the generated edge map (and intermediate features) via an ``attention'' mechanism to update the image synthesis features. \nTraining is based on conditional discriminators for edge maps and RGB images (canny edge maps are used for ground truth). \nAdditionally, a binary similarity loss is used to encourages pixels that should belong to the same class according to the input maps to take the same label according a pre-trained segmentation network.\nFinally, pixel-wise contrastive learning is employed to improve results, which consists in encouraging pixels of the same class to have similar features.\n\n", "summary_of_the_review": "\nThe paper introduces some novel ideas, and shows quantitative evaluation results improving over state-of-the-art methods, as well as user preference in a user study. \nSome parts of the paper lack clarity, and analysis of generation diversity is not included. \n\n### Post-rebuttal comment ###\nBased on the author rebuttal and other reviews, I'm upgrading my recommendation of the paper to a weak accept. \nIn particular the authors have added a number of clarification and additional experimental results that strengthen the paper compared to the original submission. \n", "strength_and_weaknesses": "\n# Strengths \n\n* Edge information is used as a guiding signal to improve semantic image synthesis\n\n* Contrastive learning is used to improve feature learning in the semantic synthesis pipeline\n\n* Experimental results improve over a collection of strong baseline methods.\n\n\n# Weaknesses \n\n* Not all components of the method are clearly explained in the paper. In particular  the class-specific pixel generation approach was unclear. \n\n* The paper is lacking detail regarding some technical material and experiments, see below. \n\n* The paper is unclear in how it introduces diversity among the generated images, and results are not analysed quantitatively or qualitatively for diversity. It would  strengthen the paper to clarify this and include such an evaluation, eg using the LPIPS metric following OASIS.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster