Semantic Image Synthesis with Unconditional Generator
Authors: JungWoo Chae
Hyunin Cho
Sooyeon Go
Kyungmook Choi
Youngjung Uh
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative model

Abstract: Semantic image synthesis (SIS) aims to generate realistic images according to semantic masks given by a user. Although recent methods produce high quality results with fine spatial control, SIS requires expensive pixel-level annotation of the training images. On the other hand, manipulating intermediate feature maps in a pretrained unconditional generator such as StyleGAN supports coarse spatial control without heavy annotation. In this paper, we introduce a new approach, for reflecting user's detailed guiding masks on a pretrained unconditional generator. Our method converts a user's guiding mask to a proxy mask through a semantic mapper. Then the proxy mask conditions the resulting image through a rearranging network based on cross-attention mechanism. The proxy mask is simple clustering of intermediate feature maps in the generator. The semantic mapper and the rearranging network are easy to train (less than half an hour). Our method is useful for many tasks: semantic image synthesis, spatially editing real images, and unaligned local transplantation. Last but not least, it is generally applicable to various datasets such as human faces, animal faces, and churches.

Semantic Image Synthesis with Unconditional Generator
Jung Woo Chae Hyunin Cho
Sooyeon Go
Kyungmook Choi'
Youngjung Uh'!
Yonsei Unviersity, Scoul, South Korea
LG CNS AI Research, Seoul, South Korea
(ajwyonsei, hyunin9528, sccyeon8655, kyungmook. cht.soooo.ccii,,,.ooo..o choi, yj.uh) Mhyossei.ac.. (cjwoolgens) @lgens. com
Abstract
Semantic image synthesis (SIS) aims to generate realistic images that match given semantic masks. Despite recent advances allowing high-quality results and precise spatial control, they require a massive semantic segmentation dataset for training the models. Instead, we propose to employ a pre-trained unconditional generator and rearrange its feature maps according to proxy masks. The proxy masks are prepared from the feature maps of random samples in the generator by simple clustering. The feature rearranger learns to rearrange original feature maps to match the shape of the proxy masks that are either from the original sample itself from random samples. Then we introduce semantic mapper that produces the proxy masks from various input conditions including semantic masks. Our method is versatile across various applications such free-form spatial editing of real images, sketch-to-phot to-photo, and even scribble-to-phie -hoto... Experiments validate advantages our method on a range of datasets: human faces, animal faces, and buildings. Code will be available online soon.
Segmentation Mask
Sketch
Free-form Mask
Segmentation Mask
Mapper
Conditional
Unconditional
pair
Proxy Mask
Proxy Mask
Proxy Mask
Train
Gonerated Image
Image
Inference
Generated Image
Image
Image
Conditional GAN
Ours
Figure 1: Conditional GANs vs Ours (Left) Conditional GANs train a generator using expensive pairs of semantic masks and images. (Right) On the other hand, our method does not require large number of image-mask pairs in training process and uses a pre-trained unconditional generator for semantic image synthesis. Furthermore, it accommodates various types of inputs such as sketches even simpler scribbles.
"qqual contribution. Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Semantic image synthesis (SIS) aims to synthesize images according to given semantic masks. Its ability to reflect the spatial arrangement of the semantic masks makes it powerful tool for various applications such as scene generation 21 25 and photorealistic image editing (4). However, it requires abundant pairs of images and per-pixel class annotations which are laborious and error prror-prone. If we want to change the granularity of control, we need to re-annotate the images and re-train the model. E.g.. a network pretrained on human faces and their semantic segmentation masks cannot be used for generating human faces from simple scribbles and requires re-training on scribble annotation.
few studies 20 have attempted to bypass this issue by employing pretrained unconditional generators for image manipulation without the need for annotations. These approaches, however, struggle to allow detailed manipulation, presenting challenges for users desiring to customize gener- ated images. Furthermore, users have to manually identify relevant clusters and tweak feature maps, leading to burdensome manipulation process.
In response to these challenges, we introduce a novel method for semantic image synthesis: rear- ranging feature maps of a pretrained inconditional generator according to given semantic masks. We break down our problem into two sub-problems to accomplish our objectives: rearranging and preparing supervision for the rearrangement. First, we design a rearranger that produces feature maps rearranged according to inputs by aggregating features through attention mechanism. Second, we prepare randomly generated samples, their feature maps, and spatial clustering. Note that the samples are produced online during training thus it does not require additional storage. This approach assumes that we have two random samples and their intermediate feature maps from the generator. The rearranger learns to rearrange the feature maps of one sample to match the semantic spatial arrangement of the other sample.
However, a semantic gap may arise when the generator receives an input mask. As shown in Figure Tright), this gap is manifested in the discrepancy between the input segmentation mask and the proxy mask. The proxy masks are the arrangement of feature maps which are derived from the feature maps in the generator through simple clustering. We solve this perception difference by training a semantic mapper that can convert an input mask to the proxy mask.
Our proposed method offers several key advantages. Not only does it significantly reduce the training time and the burden of manual annotation, but it also leverages pretrained GANs to yield high-quality synthesized images. Additionally, once we train the rearranger, the need only train the semantic mapper, even when the segmentation configuration changes, makes our approach computationally efficient compared to methods requiring retraining of the entire model. Moreover, it is compatible with other latent editing techniques such as StyleCLIP
As our method operates through feature maps rearrangement, it enables free-form mask spatial editing with fewer constraints on the mask's shape. Furthermore, our semantic mapper accepts different types of input conditions such as HED (37 and Canny Edges. Therefore, approach be applied to various applications like sketch-to-photo and scribble-to-photo transformations.
We demonstrate pixel-level content creation with a pretrained generator several datasets including CelebA.Mask-He LSUN Church, and L.SUN Bedroom. We also present real image editing examples where the source image is edited to conform to the shape of the target image target mask. Lastly, our method achieves noticeably higher mIOU scores and significantly lower FID than prior works. Furthermore, our method outperforms the baselines in both qualitative and quantitative measures.
2 Related Work
Semantic image synthesis Semantic Image Synthesis (SIS) is specialized type of conditional
image generation that can accurately mirror user intentions at pixel level 7
33 40
17
27 33
21 47 25 26). Over time, the field has advanced significantly, improving controllability and thus
enabling the production of diverse, high-quality results. These advancements also allow for the adjustment of the style of individual class components based on semantic masks. Like preceding GAN methods, the application of diffusion in SIS also facilitates the alteration of image style under specific conditions (34. More recently, diffusion-basee baaee conditional image generation has taken this further, creating high-quality images using semantic masks and other inputs like edge maps and user

---

Despite their benefits, SIS methods are resource- resource-intensive, requiring many image-label pairs during learning. Moreover, any changes in annotations necessitate complete retraining of the model, leading to inefficiencies. Unlike these methods, our approach uses pairs of generator feature maps and the corresponding generated images as data, effectively eliminating the need for expensive image-label pair datasets. As a result, our method provides significant cost and time savings compared to previous methods.
scribbles 66 31 32
24 42
Feature map manipulation for semantic part editing Manipulating feature maps of generator directly influences the resulting synthesized image, and the clustering results of these feature maps yield semantic part representations [4. Prior works have leveraged these characteristics of feature maps to conduct local editing tasks such as enlarging the eyes or changing the shape of ears without manual annotations [20]. However, these methods are inconvenient when reflect the intention of user during the image editing process, presenting a considerable challenge. To address this, we propose a novel approach that transforms the feature maps into the shape of given mask, allowing more intuitive and precise image generation. This approach significantly enhances the user experience by providing an efficient and convenient means of generating images that closely align with the user's intention.
Mapping between feature maps and semantics The utility of feature maps extends beyond image generation to tasks such as few-shot semantic segmentation, offering rich semantic information from 2]. Techniques such as linear transformation or a simple CNN network can generate high-quality annotations from minimal set of label-image pairs (38 22]] While the studies mentioned above utilize feature maps to generate segmentation maps, our research goal takes reverse approach. Inspired by [4. ours create proxy masks based the information provided by segmentation maps or other conditions so that the model can rearrange feature maps based on the transferred clusters. Notably, LinearGAN[3R excels in enabling few-shot segmentation and achieving semantic image synthesis by finding the optimized latent codes. However, optimization in the latent space in LinearGAN lacks detailed pixel-level control and makes it challenging apply user-desired styles properly. Our method conducts semantic image synthesis at the feature map level, significantly enhancing pixel-level control and allowing users apply various styles from references in their preferred manner.
a pretrained generator [I 45
38 29
Exemplar guided image synthesis Utilizing exemplar images rather than varying domain con- ditions could provide broader controllability range. Most exemplar image generation problems aim to stylize the given condition with the style of the exemplar image [4] 46 16 9. However, we focus more on reflecting the shape of the exemplar image. Techniques such as SNI DATIK], TransEditor (39], and CoordGAN use dual latent spaces for image editing, which helps to disen- tangle the structure and style of image. By incorporating an encoder extract the structural latent codes of the exemplar image, it is possible to generate images that retain the shape of the exemplar while exhibiting range of stylistic variations. However, these approaches predominantly rely on latent codes, making it challenging to replicate the structure of exemplar images precisely. While similarly employing latent codes, method enhances spatial manipulation accuracy by rearranging the feature maps.
Foundation model for segmentation The Segment Anything Model (2 is a foundational vision model trained on over billion masks. With SAM, zero-shot segmentation for general images becomes possible. The method allows the creation of segmentation masks for entire image datasets, which can then be used to train conventional SIS models. However, SAM does not cover every comain--peccfic semantics. Additionally, utilizing previous SIS models with SAM still requires the time-consuming process of generating mask pairs and training SIS models from scratch. Our method addresses situations where foundational models like SAM may not suitable. can be applied to user-specific custom datasets, offering more efficient and adaptable solution.
3 Method
In this section, we explain our problem formulation and its components. Figure compares the conventional semantic image synthesis and our method. The conventional semantic image synthesis methods require pixel-wise semantic label maps for all images. Conversely, goal is to provide
3

---

G1
22
G1
G2
Leeff
Clustering
Clustering
Feuture Maps
Rearranged
ft
Feature Maps
Rearranger
Feature Maps
Rearrunged Fenture Maps
Rearranger
Rs
Proxy
Lmmash
Proxy
Proxy
(a) Mask Loss
(b) Self-reconstruction Loss
Figure 2: Rearranger achieves self-supervised learning by incorporating self-reconse refnosecontttitionnn loss and mask loss, reducing the reliance on a large image-mas dataset. G1 is earlier layers of the generator and G2 is later layers after the proxy mask resolution.
spatial control pretrained unconditional generator using an input mask. We formulate the problem as rearranging intermediate features in the generator according to the input mask. As the means for providing spatial control, we introduce a proxy mask, which is defined by unsupervised clustering of the features. Then, we train a rearranger for rearranging the features according to the proxy mask. We introduced semantic mapper that maps the input condition to the proxy mask to bridge the perception gap between the proxy mask and the input mask.
3.1 Self-supervised learning of spatial control
Our primary objective is to spatially rearrange the feature maps in the unconditional generator to generate appropriate images according the specific conditions. To this end, we introduce proxy mask that specifies the target arrangement and rearranger, network that utilizes attention mechanism. Since attention a weighted sum of values based on the similarity of the query and the key, has proven effective for condition- based input modifications to rearrange the feature maps according to the given proxy mask.
Given a proxy mask m, and original feature maps f; = (z, where z is a random latent code, denotes an index of the sample and G2 the front part of the model up and including the layer corresponding to the feature maps used to generate the proxy mask, rearranger produces
[QKT V. Here, Q denotes the embedding of the proxy mask m, which is non-linear transformation of the proxy mask., K and V denotes the embedding of the original feature maps f, d denotes the dimension of the embedding space. Formally,
feature maps through a cross- atteetitoo mechanism: Attention( Q, K, V) = softmax
Va
Q Q=QQ- m. K WK fy, V
Wy f,.
(1)
where W() Rexdr are learnable embedding matrices 8 30 For simplicity, we use rearranged feature maps f' Attention(m,, fy, fy) instead of Attention K, V). With this attention mechanism, the image rendering can be represented as Xtiee G2(T))) where (i. j) is random pair of latent codes, Xjse denotes the synthesized image and G2 is the rest part of generator consists of later layers of generator. Further, we define proxy mask m; in terms of f, via Clustering which is K K-means clustering algorithm. expressed as m, Clustering(f)).
As shown in Figure 2 to make output of rearranger
accurately reflect the mask given as condition, we encourage the network to reconstruct the proxy mask from the rearranged feature maps by
Lmask CrossEntropy(mm', m.)
Where m Clustering((I,)) denotes the proxy mask of rearranged feature maps.
However, maintain the style source latent, we encourage the network self-reconstruct when proxy mask created from the same latent, is used the query for attention:

---

S Σ [ -f: f:l12
Our overall learning pipeline operates follows. We first prepare the original feature maps f, from random latent code z; We then apply K-means clustering the feature maps to derive segmentation- like masks. We apply the loss as mentioned above functions using these masks and feature maps, which are created from different latents. These processes work in a self-suuprvised manner, enabling the rearranger to learn effectively and autonomously by leveraging the representation of the pretrained generator and unannotated dataset.
Finally, the overall training term, denoted as Lrotal. is composed of the non-saturating adversarial loss with R -regullarizaion 5 18 self-reconstructionn loss, and mask reconstruction loss.
Lital frady + Aretth + mask frmusk + ARI LRI
3.2 Semantic guide
In the earlier sections, we have introduced an ap- proach to provide spatial control for a pretrained unconditional GAN through proxy masks gen- erated via unsupervised feature maps cluster- ing. However, the process lacks a direct connec- tion from the input mask the proxy mask. To bridge this gap, we propose semantic mapper to convert the user given condition including input mask into proxy mask, as illustrated in Figure This conversion can be represented as Mapper(c), c), where c is the input condition.
One-shot
Segmentation
Network
Mapper
Proxy Mask
Guiding
Lrec
K-meuns Prediction
Feature Maps
Proxy Mask
Figure 3: Semantic Mapper transforms input masks that the generator cannot comprehend into proxy masks, enabling the generator understand and synthesize the corresponding output. When training the semantic mapper, one-shot segmen- tation network and reconstruction loss are used.
We begin the process by generating images and their corresponding proxy masks from random latent codes. Next, we employ one-shot seg- mentation network, as suggested in [29], to pre- pare set of mask and proxy mask pairs. These pairs are used to train our semantic mapper, allowing it to generate proxy mask m; from the input mask m SegNet(f.) by
Crec = CrossEntropy(mm/. m.)
(5)
Here, SegNet denotes the one-shot segmentation network and the concatenation of several feature maps for the SegNet input. Through this training, establish a connection that bridges the perception gap between human and the generator, enabling the generator to better understand and integrate the semantic content intended by the user.
Furthermore, have also incorporated an adversarial training strategy in the semantic mapper's training process. We utilize a discriminator identical to the one used in StyleGAN2 [II], which discriminates between real and fake images generated with proxy masks. The overall training term for the semantic mapper comprises adversarial loss, reconstruction loss, and R regularization.
Liotal
Aadv Ladv + Cree + ARI Cri
Through this Semantic Guide, we have enabled our generator to capture and integrate the desired semantic information effectively. Consequently, the synthesized images produced by our method closely align with the user's intended semantic content.
4 Experiments
We have conducted range of experiments to highlight the adaptability of our approach across diverse datasets, including HelcbAMask-HO [14), LSUN Church, LSUN Bedroom 41, FFHQ (10),
5

---

FFHQ
ISUN
LiBerGAN
Ours
LineirAAN
Ours
Figure 4: Comparison with LinearGAN when receiving the same mask. LinearGAN is an indirectly optimized method that adjusts the mask through optimization, while our approach is designed to create images that better fit the given mask. Our method performs better by producing more accurate images that align with the masks. Both LinearGAN and Ours use single mask to train each model.
and AFHQ 33] Moreover, we used these datasets for conditional image synthesis under various conditions, including segmentation masks, sketches, and user scribbles. Regarding model architecture, we employed the generator and discriminator of StyleGAN2 [1] as pretrained model. Furthermore, used architecture similar SPADE [21 and simple UNet architecture for segmentation masks and different conditions, respectively. For more detailed information about these experiments and settings, please refer to the Appendix. In addition, include ablation study in the Appendix that shows the validity of our proposed loss function and the appropriate number of clusterings.
mapper architecture
4.1 Semantic image synthesis
4.1.1
Comparison to few-shot semantic image synthesis method
We present comparison with LinearGAN 38 method specifically tailored for few-shot semantic image synthesis utilizing a pretrained generator. LincarGAN optimizes the latent vector, which most closely matches the shape of the mask, using few-shot semantic segmentation for semantic image synthesis. Our method surpasses LinearGAN in semantic image synthesis performance with only single imput mask. To provide accurate comparison with LinearGAN, also conducted comparisons in few-shot setting.
Figure illlustrate the difficulty of LinearGAN in accurately reflecting the overall mask structure. While LinearGAN can generate images that align with the input mask for datasets with less shape diversity, such human faces, it struggles with LSUN church or LSSNN bedroom because of the diverse shape of the datasets. This challenge arises as LinearGAN often finds difficult to identify latent closely resembling the shape. Consequently, it fails to reflect intricate shapes effectively. In contrast, our method adjusts the feature maps to correspond with the shape of the input mask, demonstrating better results with detail.
The quantitative comparison, as seen Table I utilizes the mloU between the input mask and the mask generated by pretrained segmentation network. This metric evaluates the accuracy of the model reflecting the mask. Our method, surpassing LinearGAN across all datasets, shows its precision in reflecting the input mask, even with limited data. Remarkably, this superiority is maintained irrespective of the number of input masks. This advantage is both qualitatively and quantitatively evident, validating the effectiveness of our approach in few-shot semantic image synthesis.

---

N
mloU
Method
Church
Bedroom
FFHQ
LinearGAN
16 1.4
17.5 2.0
37.2 0.8
Ours
23.8 4.4
35.6 4.0
49.8 1.0
4
LinearGAN
18 1.3
21.6 0.9
39.1 0.5
Ours
2..6
44.3 2.4
54.3 1.18
8
LinearGAN
19.6
21.7
39.4 0.9 54.4 1.7
Ours
28.7 1.23
44.7 3.0
16
LinearGAN
20.4 0.6 28.8
22.3
40 0.2
Ours
45.7 0.9
56.7 = 1.19
Table mloU results of LinearGAN and Ours on various datasets. N means number of mask images used in few-shot setting.
Ours (One-shot)
OASIS (One-shot)
INADE (One-shot)
OASIS (Full-shot)
INADE (Full-shot)
Image
Figure 5: The comparison results between method and supervised models using the mask (Including one-shot setting). Compared to existing SIS methods, Ours produces results that are closer the ground truth. Additionally, Our result shows the most natural-looking output.
4.1.2 Comparison to full-shot method
Furthermore, we carried out both quantitative and qualitative assessments our proposed method and the baseline approaches with CelebAMask-HO [4], LSUN Church, LSUN Bedroom [41] and FFHQ [1]]
As illustrated in Figure 5 we adopted the RepurposeGAN 29 approach to generate one-shot data pairs for comparison under one-shot setting. While the results from competitors frequently display unnatural artifacts, our method yields more natural and closer-to-ground truth outcomes. Also, as depicted in Table 2 our method has higher mIoU than existing techniques with one-shot training data pairs. The reason is that our approach excels in accurately segmenting large mask regions. Our method shows lower mloU than the conventional models trained with full-shot 28K image- annotation pairs because IoU decreases for fine fine-grained grained classes that are not included in the one-shot setting, such as accessories with low frequencies. However, when it comes to the FID, our work significantly outperforms the one-shot settings. It is typically superior to the full-shot methods, even with just single annotation. In addition, we have included results for other metrics, such I.IPS and Precision and Recall, in the Appendix.
7

---

FID
mloU(vs gt mask)
Acc(vs mask)
SEAN
28.1
75.9
92.3
OASIS
20.2
74.0
91.5
INADE
21.5
74.1
93.2
SDM
18.8
77.0
n/a
OASIS (1-shot)
25.5
45.5
82,7
INADE (1-shot)
28.1
44.3
83,8
Ours (1 (1-shot)
18.5
53.1
88.2
Table 2: Quantitative comparison with SEAN, OAShS(I-shot/ / full-shot), INADE(1-shot full-shot), SDM. mIOU and Ace are measured by comparing with the ground truth mask.
Latent-rdited (Yellow hair)
Latent-edited
Original
Original
(Perm hair)
Free-form Mask
Ours
LinearGAN
(Full-shot)
Figure 6: Result of image synthesis with free-form mask. Images in the first inverted images in StyleGAN and their edited images with the HairCLIP (36). Images are converted to the spatial shapes of both masks and images in the first column while preserving styles and identities. We also show comparison with LinearGAN (38) and OASIS (25) in the last three columns. For comparison sample the images with random noise for all models
4.2 Free-form image manipulation
In this subsection, we show the flexibility our proposed method in facilitating free-form image manipulation. Our approach underscores its capability of handling masks that deviate substantially from typical training distributions, such as oversized human cars and big oval eyes. Figure iillustraees the image synthesis results when such unconventional masks the masks deviated from the typical human form, our method successfully synthesized results that corresponded appropriately to the provided masks. In contrast, when LinearGAN (38) and OASIS (25 receive a mask that falls outside the typical human range as shape guide, they fail to generate results that align with the shape indicated by the mask.
F
used as inputs. Even in cases where
Moreover, the versatility of our method further emphasized by its compatibility with latent editing techniques such as HairCLIP [36] as shown in Figure 6 While latent editing changes the hairstyle to a yellow color or perm style, our method preserves the shape of images corresponding to given masks. See the Appendix G for additional samples of generated images conditioned with free-form masks.

---

LSUN Cats
LSUN Church
FFHQ
Seribbles
Sketch
PSS
Ours
Ours
(a) Scribbles
(b) Sketch
Figure 7: Image synthesis results when diverse types of input masks entered, such scribbles and sketches
Target Shape Images
Target Shape Images
Style
Style
Ours
DAY Transdditr
Figure 8: Image generation results with real image guide. The top row consists of images that serve as the style input, while the left column contains target image that determine the desired shape.
4.3 Conditional image synthesis using variety of conditions.
This part broadens the scope of semantic image synthesis to include just segmentation masks and diverse conditions like sketches and scribbles. Figure 7[a) demonstrates the outcome when simple scribbles are used as an input mask. Remarkably, even with a scribble mask created with just single line, our method can generate image synthesis that is well-suited the indicated shape. Moreover, this user-friendly aspect further enhances the usability of our approach. Additionally, we visualize our results and compare them qualitatively with pSp (23) GAN inversion approach demonstrating its ability to invert various inputs into latent codes, not solely real images. As illustrated in Figure (b)) our comparison with pSp, especially in semantic image synthesis using sketch inputs, reveals significant difference. pSp can generate coarse structures effectively with pSp encoder and style mixing, but it struggles to capture and reproduce the fine details of the input conditions. On the other hand, our method surpasses pSp with more corresponding results with fewer restrictions of matching latent code because of flexible spatial control of our model using feature maps rearranging. In the Appendix E, we further demonstrate the proficiency of our method in generating images based on various input conditions with implementation details. This additional evidence underscores the versatility of our method and superior performance in conditional image synthesis.

---

4.4 Exemplar-guided image generation
Beyond the specified conditions, our model produces an output image that has a similar structure to provided sample. The results of image generation are displayed in Figure g The latent codes are derived from real images by utilizing the e4e inversion technique [28] that was trained with the CelebAMask-HO training set. This inverted latent code enables us to obtain proxy mask of the sample, thereby aligning the generated images with the real image structure.
In Figure 8 it is evident that our generated results better preserve shape in comparison to other exemplar-guides image generations. The results of Transeditor [39] and DAT [13] do preserve the original shape as effectively, with DAT even showing disparity in style. Although competitors use dual latent spaces portray exemplar structure more accurately, the structural latent code does not always reflect the shape correctly. In contrast, our method more accurately mirrors the exemplar shape because the Rearranger directly forms the feature maps with the proxy mask.
Conclusion
Semantic image synthesis has received significant attention for its ability to reflect intentions images at a very detailed level, but achieving this requires substantial data. Therefore, we proposed our method as solution to address this issue. Our method rearranges the features of pretrained generator to enable detailed control over images, and all elements of our approach are self-supervised, except for single annotation mapped to human perception. With just annotation, we achieve precise image control, as demonstrated through experiments in semantic image synthesis with various input conditions. Our method also showed data efficiency in situations where the pixel-level annotation changes. Overall, our approach demonstrated a high level of detail control without the dependence on annotations.
Limitations Our one-shot results outperform other competitors, but slight performance gap remains when compared full-shot outcomes. Therefore, future work will make it possible to outperform to beat full-shot performance. Also, our approach operates at the feature map level, which, while enabling the natural incorporation masks, needs achieve complete pixel-level control. For more details, please refer to the Appendix D.
Acknowledgement
This work was supported by the National Research Foundation of Korea(NRF) grant (No. 202211F1410724111) funded by the Korea government (MSIT).
References
[1] Yazeed Alharbi and Peter Wonka. Disentangled image generation through structured noise injection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5134-5142, 2020.
[2] Dmiiryy Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint arXiv:21 2021.
[3] Yunjey Choi. Youngjung Uh. Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEFF/CVF conference computer vision and pattern recognition, pages 8188-8181 18888197, 2020.
Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncovering the local semanties of gans. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5770 5779. IEEE Computer Society, 2020.
[5] Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):
139-144, 2020.
[6] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 01866800-855) 2020.
10

---

[7] Phillip Isola, Jun Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-imagg translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and patiern recognition. pages 1125-1134, 2017.
[8] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, pages 4651- 4664. PMIR, 2021.
[9] Chang Jiang. Fei Gao, Biao Ma, Yuhao Lin, Nannan Wang, and Gang Xu. Masked and adaptive transformer for exemplar based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22418 22427,22222222 2023.
[10] Tero Karras, Samuli Laine, and Timo Aila. A style-baset generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[11] Tero Karras, Samuli Laine. Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedingy of the IEEE/CVF conference on computer vision and puttern recognition, pages 81 $110-8115 81,9, 2020.
[12] Alexander Kirillov, Frie Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:23344002643 2023.
[13] Gihyun Kwon and Jong Chul Ye. Diagonal attention and style-based gan for content-style disentanglement in image generation and translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[14] Cheng- Han Lec, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings the IEEE/CVF Conference Computer Vision and Pottern Recognition, pages 5549-5558, 2020.
[15] Daiqing Li. Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8300-8311, 2021.
[16] Songhua Liu, Jingwen Ye, Sucheng Ren, and Xinchao Wang. Dynast: Dynamic sparse transformer for exemplar-guided image generation. In European Conference on Computer Vision, pages 72-90. Springer, 2022.
[17] Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, et al. Learning to predict layout-to-image conditional convolutions for semantic image synthesis. Advances in Neural Information Processing Systems, 32, 2019.
[18] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning, pages 3481-3490. PMLR, 2018
[19] Jiteng Mu, Shalini De Mello, Zhiding Yu, Nuno Vasconcelos, Xiaolong Wang, Jan Kautz, and Sifei Liu. Coordgan: Self-supervised dense correspondences emerge from gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10011-10020. 2022.
[20] James Oldfield, Christos Tzelepis, Yannis Panagakis, Mihalis A Nicolaou, and loannis Patras. Panda: Un- supervised learning of parts and appearances the feature maps of gans, arXiv preprint arXiv:2206. 00048. 2022.
[21] Taesung Park, Ming- Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2337-2346, 2019.
[22] Patashnik, Zongze Wu. Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2085-2099 2021.
[23] Flad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: stylegan encoder for image-to-image translation. In Proceedings of the IEFF/CVF conference compuder vision and puttern recognition, pages 2287-2296. 2021.
[24] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffuuson models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages
June 2022.
11

---

[25] Edgar Schönfeld, Vadim Sushko, Dan Zhang, Juergen Gall. Bernt Schiele, and Anna Khoreva. You only need adversarial supervision for semantic image synthesis. In International Conference on Learning Representations.
[26] Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Bin Liu, Gang Hua, and Nenghai Yu. Diverse semantic image synthesis via prohability distribution modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[27] Hao Tang, Song Bai, and Nicu Sebe. Dual attention gans for semantic image synthesis. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1994-2002, 2020.
[28] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen- Or. Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4): 14, 2021.
[29] Nontawat Tritrong, Pitchapor Revatboworwwong, and Supasorn Suwajanakorn. Repurposing gans for one-shot semantic part segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 4475-4485, 2021.
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[31] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or. Sketch-guided text-to-image diffusion models. arXiv preprint arXiv:2211.33777 2022.
[32] Tengfei Wang, Ting Zhang. Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining is all you need for imaeeto--image translation, 2022.
[33] Ting- Chun Wang. Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference computer vision and pattern recognition, pages
2018.
[34] Weilun Wang, Jianmin Bao, Wengang Zhou, Dongdong Chen, Dong Chen, Lu Yuan, and Houqiang Li. Semantic image synthesis via diffusion models. arXiv preprint arXiv:2207.00050, 2022.
[35] Yi Wang, La Qi, Ying-Cong Chen, Xiangyu Zhang, and Jiaya Jia. Image synthesis via semantic composition. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 13749-13758, 2021.
[36] Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Zhentao Tan, Lu Yuan, Weiming Zhang, and Nenghai Yu. Hairelip: Design your hair by text and reference image. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18072-18081, 2022.
[37] Saining Xie and Zhuowen Tu. Holistically nested edge detection. In Proceedings of the IEEE international conference on computer vision, pages 3395-1403 2015.
[38] Jianjin Xu and Changxi Zheng, Linear semantics in generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9351-9366 2021.
[39] Yanbo Xu, Yueqin Yin, Liming Jiang, Qianyi Wu, Chengyao Zheng, Chen Change Loy, Bo Dai, and Wayne Wu. Transeditor: transformer-based dual-space gan for highly controllable facial editing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7683-7692, 2022.
[40] Dingdong Yang, Seunghoon Hong, Yunscok Jang, Tianchen Zhao, and Honglak Lee. Diversity-sensstive conditional generative adversarial networks. In International Conference on Learning Representations.
[41] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint (rXiv:1506.000 2015.
[42] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. 2023.
[43] Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen. Cross-domain correspondence learning for exemplar- based image translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5143-5153, 2020.
[44] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for sho-shot semantic segmentation. IEEE transactions on cybernetics, 50(9):3855-334 2020.
12

---

[45] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin. Jean-Francois Lafleche, Adela Barriuso. Antonio Torralba, and Sanja Fidler. Dataselgan: Efficient labeled data factory with minimal human effort. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10145- 10155, 2021.
[46] Xingran Zhou, Zhang, Ting Zhang, Pan Zhang, Jiammin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. Cocosnet v2: Full full-resolution correspondence learning for image translation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1465-11475, 2021.
[47] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region- adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3104-5113, 2020.
13


Reviews:
[{"review_id": "FgDNfuOHiZ", "reviewer": "Reviewer_Lq4s", "summary": "This paper proposes a new method for semantic image synthesis that can generate images from given semantic segmentation maps, particularly in few-shot settings with different forms of inputs. The method trains a model to transform the input semantic segmentation maps into feature maps and cluster the output to outline the discriminativeness and boundaries in the resulting features. It then learns a conditional GAN to generate realistic images conditioned on random noise and the processed feature maps transformed from the corresponding semantic layouts. These designs enable few-shot capability and handle various forms of inputs (such as sketches and scribbles). The proposed method is extensively tested in experiments, which validate its effectiveness.", "questions": "1. It would be interesting to investigate whether the semantic mapper could be initialized by pretrained semantic segmentation models or how its design could be improved. This could potentially improve the final performance and is worth exploring further.\n2. Since diffusion-based methods can now be used for semantic segmentation, it raises the question of whether this framework could be extended to diffusion-based image synthesis. This is an intriguing possibility to explore, although there may be potential difficulties to overcome, such as the complexity of the model and the computational resources required.", "limitations": "To explore the method's limitations, it would be beneficial to discuss some failure cases. This could help identify situations where the proposed method may not perform well and provide insight into potential areas for improvement.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes an interesting method design that enables few-shot image synthesis by predicting a proxy mask from intermediate features predicted from the input. This approach also enables the method to handle inputs in different forms beyond just segmentation maps.\n2. The experiments conducted on CelebAMask-HQ, LSUN Church, and LSUN Bedroom datasets validate the effectiveness of the proposed method. The one-shot performance on FID, mIoU, and accuracy metrics is impressive.", "weaknesses": "1. The paper should discuss why certain method designs were not ablated, such as the use of k-means to generate the proxy mask from predicted features. The impact of this design choice on the effectiveness of the proposed method should also be measured and discussed.\n2. The paper should include a discussion of the capacity and efficiency of the proposed model. Specifically, it should analyze how scaling the model impacts the quantitative results.\n3. To provide a more comprehensive understanding of the proposed method, the paper should include an analysis of failure cases and discuss their limitations. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our positive aspects: Ours has an interesting method design and showcases impressive one-shot performance across FID, mIoU, and accuracy metrics.\n\n### Ablation\n>The paper should discuss why certain method designs were not ablated, such as the use of k-means to generate the proxy mask from predicted features. The impact of this design choice on the effectiveness of the proposed method should also be measured and discussed\n\nThe key components of our method are rearranger, semantic mapper, and the training scheme. We follow existing works for peripheral components as follows. We use k-means because it is a common choice for clustering [A]. We also chose cross attention, which is a technique commonly used to incorporate conditions into inputs, such as transforming references into the form of the target [B,C].\n\n[A] Collins et al. Editing in Style: Uncovering the Local Semantics of GANs. CVPR 2020.  \n[B] Liu et al. DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation. ECCV 2022.  \n[C] Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022.\n\n\n#### Capacity, efficiency, and scaling\n>The paper should include a discussion of the capacity and efficiency of the proposed model. Specifically, it should analyze how scaling the model impacts the quantitative results\n\nWe performed the scaling experiment of the proposed module indirectly by reducing the original CNN layer by half. For the semantic mapper, we have not experimented with further scaling, as it is identical to the generator of OASIS except that it does not use 3D noise, and we will experiment further if there are questions about this part. The details of the structure are described in Appendix A.1. The results of the experiments are shown in the table below. We report the number of parameters (capacity), training time (efficiency), and FID & mIoU over different scales (scaling). The results show that our method is time efficient to train and achieves high image quality. The efficiency of our module comes from the fact that it only learns to rearrange the representation, unlike existing SIS methods. \n\n|                                                          \t| number of params \t| training time \t| FID   \t| mIoU \t|\n|----------------------------------------------------------|------------------|---------------|-------|------|\n| Rearranging network (Number of CNN layers halved) \t| 30M           \t| 8 hours        \t|  8.71 \t| n/a  \t|\n| Rearranging network                                  \t| 54M           \t| 18 hours       \t|  7.48 \t| n/a  \t|\n| Semantic Mapper                                      \t| 69M           \t| 12 hours       \t| 18.53 \t| 53.10 |\n| OASIS                                                 | 93M           \t| 77 hours       \t| 20.23 \t| 74.01 |\n\n\n\n\n\n\n#### Failure cases\n\n>To provide a more comprehensive understanding of the proposed method, the paper should include an analysis of failure cases and discuss their limitations.\n\n\nWe show the failure case in Figure R1. The result shows that the mask does not reflect well in some areas, and there are two reasons for this. First, as mentioned in Limitation, the proxy mask has a lower resolution of $64^2$ than the actual resolution of $256^2$. Therefore, some semantics are lost when mapping to the proxy mask. The next case of failure is a mislabeled annotation. Since the semantic mapper is completely dependent on a single mask, we could observe cases where, for example, if a strand of hair is not correctly annotated, the effect is reflected in the result, and the hair becomes the background. Furthermore, these cases improve with increasing annotation, as shown quantitatively in Table 2.\n\n\n\n\n#### Weight initialization using pretrained segmentation models\n>It would be interesting to investigate whether the semantic mapper could be initialized by pretrained semantic segmentation models or how its design could be improved. This could potentially improve the final performance and is worth exploring further \n\n\nIt is indeed an interesting idea. However, semantic mapper receives semantic masks, while semantic segmentation models receive RGB images. Initializing weights from a pretrained model usually replaces last layers. On the other hand, adapting a pretrained segmentation model for semantic mapper requires replacing the input layer.  Thank you for initiating great discussion.\n\n\n#### Extension to the diffusion models\n>Since diffusion-based methods can now be used for semantic segmentation, it raises the question of whether this framework could be extended to diffusion-based image synthesis. This is an intriguing possibility to explore, although there may be potential difficulties to overcome, such as the complexity of the model and the computational resources required.\n\n\nWe agree that our method is applicable for diffusion models. We first confirmed that the proxy mask can be obtained by clustering the latents of the U-net of stable diffusion, and there is also related work on clustering with latents [D]. Manually modifying the latents in a single timestep successfully edits the resulting image. We have edited facial photos to have three eyes, and the result is attached as figure R2 in the PDF. We have tried to add a Rearranging Network and a Semantic Mapper to the model. However, it was not straightforward to make it work. This makes it a plausible next research item.\n\n[D] Xu et al. ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. CVPR 2023."}, {"review_id": "e22T9HGPeQ", "reviewer": "Reviewer_swNj", "summary": "The paper introduces a method to adapt unconditional GAN generators to pixel-level-conditioned tasks, such as semantic image synthesis. The method is based on the observation that the intermediate generator's features contain rich knowledge about the semantics of generated objects. This knowledge is extracted to build a proxy mask via clustering, which can be manipulated by the introduced Reanranging Network. The manipulation to a user-specified conditioning mask is performed by a separate component called Semantic Mapper. Overall, the method allows to achieve high-quality, highly conrollable SIS without large paired image-mask datasets. The experiments also demonstrate a range of applications, including image manipulation, style-based SIS, and sketch-to-image synthesis.", "questions": "Please answer the concerns in Weaknesses. \n\n", "limitations": "No, more discussion is needed. I suggest adding the limitation points from Weaknesses if they remain relevant after the rebuttal. There is no discussion on societal impact at all. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "\n\n[Contribution] - The main strength, in my opinion, is itself the successful SIS from unconditional GANs. The literature on SIS GANs has long been somewhat detached from unconditional GANs, with its own tricks and training methods (e.g., pix2ix architectures, perceptual losses) and shortcomings (e.g., limited diversity). Apart from the data efficiency (claimed by authors), applying unconditional GANs to SIS brings the potential to improve SIS with all the recent advancements from other GAN literature. \n\n[Efficiency] - The proposed method uses a pre-trained GAN generator and one manual annotation (proxy mask -> real mask). This is a very cheap method to achieve SIS compared to conventional SIS GAN models. This can enable SIS on datasets that do not have good segmentation datasets, which can reduce costs. As the authors claimed, if the segmentation style changes, this model does not need to be re-trained, just the Semantic Mapper. \n\n[Performace] - Unconditional synthesis is naturally more difficult than SIS with paired datasets, so the quality of the model does not reach all prior SIS GANs trained on full datasets. But it is impressively not far, both in FID and mIoU. \n\n[Applications] - Some applications enabled by the model seem to extend the prior capabilities of both unconditional GANs and SIS GANs.", "weaknesses": "[Limitation] - Enforcing each generated image to have K classes via K-means clustering seems rather restrictive. This works for FFHQ and Churches, but would not work for more classic SIS datasets like Ade20K, Cityscapes, or Coco-Stuff, where each image has only a subset of semantic classes. Solving this requires, probably, nontrivial extensions. \n\n[Evaluation] - One point of using unconditional GANs instead of SIS could be their better conditioning and diversity. I think it would be interesting to see more analysis on unconditional GANs vs SIS GANs, to see the trade-offs. For example, is LPIPS diversity of synthesis higher with StyleGANv2 compared to, say, OASIS? Other metrics like PPL, Precision&Recall would be also interesting. \n\n[Semantic Mapper] - It is a bit unclear how Semantic Mapper is trained. Is it correct that for a single given proxy mask, the user has to draw a real mask himself? In this case, it is also somehow a limitation, as it introduces overhead.  More research is needed to find proxy-real mask alignment more automatically. \n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough recognition of the strengths in our paper: Our method offers the advantage of applying unconditional GANs to SIS. Furthermore, when the segmentation style changes, only retraining the Semantic Mapper is required, showcasing another strength. The model enables new applications that appear to expand the capabilities of both unconditional GANs and SIS.\n\n#### Explanation of feature maps clustering\n>Enforcing each generated image to have K classes via K-means clustering seems rather restrictive. This works for FFHQ and Churches, but would not work for more classic SIS datasets like Ade20K, Cityscapes, or Coco-Stuff, where each image has only a subset of semantic classes. Solving this requires, probably, nontrivial extensions.\n\nWe cluster the set of features from *512 samples* rather than each sample to K clusters. Formally,  $\\mathbf{f}_i = G_1(z_i)$ where $z_i$ is a random latent code and $\\mathbf{f}_i \\in \\mathcal{R}^{CHW}$  are feature maps. We run K-means over a set of features $\\mathbf{f}_i(x,y)$ to prepare K cluster centroids where $x$ and $y$ are coordinates of feature maps $\\mathbf{f}_i$. Hence, our method is applicable to such datasets if their unconditional generators are publicly available. We emphasize that our method enriches the boundary of SIS to datasets without semantic masks.\n\n\n#### Diversity of proposed method\n\n>One point of using unconditional GANs instead of SIS could be their better conditioning and diversity. I think it would be interesting to see more analysis on unconditional GANs vs SIS GANs, to see the trade-offs. For example, is \ndiversity of synthesis higher with StyleGANv2 compared to, say, OASIS? Other metrics like PPL, Precision&Recall would be also interesting.\n\nThank you for the insightful comment. Yes, the diversity of our method measured by LPIPS is higher than INADE and OASIS. Consulting precision and recall together does not pick a definite winner. The table below summarizes the measures. For measuring LPIPS, we follow INADE [E].\n|           | Ours | INADE | OASIS | \n|-----------|------|-------|-------|\n| LPIPS     | 0.45 | 0.36  | 0.30  | \n| Precision | 0.76 | 0.86  | 0.69  | \n| Recall    | 0.22 | 0.19  | 0.36  | \n\n\n[E] Tan, Zhentao, et al. Diverse semantic image synthesis via probability distribution modeling. CVPR 2021.\n\n#### Training method of the Semantic Mapper and Advantages of self-made Mask\n\n>It is a bit unclear how Semantic Mapper is trained. Is it correct that for a single given proxy mask, the user has to draw a real mask himself? In this case, it is also somehow a limitation, as it introduces overhead. More research is needed to find proxy-real mask alignment more automatically\n\n\n\nThe semantic mapper is a function that receives a semantic mask and produces a proxy mask. It requires pairs of semantic masks and proxy masks for training. For a random fake image and its proxy mask produced by $G_2(G_1(z))$ and $G_1(z)$, respectively, a user draws a semantic mask for the fake image (yes it is correct). By letting the user draw the mask, our method allows different types of inputs ranging from the custom number of semantic classes to binary contour scribbles according to the intention of user."}, {"review_id": "KeRIQ1MemW", "reviewer": "Reviewer_pArX", "summary": "This paper proposed a self-supervised semantic image generation method. It adopts a pretrained generator, and rearranges the intermediate features to make the generated images align with the given input. Extensive experiments show the method achieve comparable performance with previous GAN based methods.", "questions": "Are G1 and G2 in Fig. 2 the same model?", "limitations": "see weakness.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The training is self-supervised and does not require paired data collection.\n\nThe writing is clear.\n\nResults on diverse datasets, including faces and buildings.", "weaknesses": "- The key motivation of this work is that it is difficult to collect large-scale paired data for semantic generation models. But this is not challenging for the cases shown by authors, i.e., mask-to-image, sketch-to-image. For example, we can use well-trained semantic segmentation models, e.g., SAM, to perform semantic segmentation, and use the sketch extraction method, e.g., HED, Canny, to extract sketches. In this way, we can easily obtain large-scale paired training data. Also, there are also some methods, e.g., PITI that use pretraining to alleviate the data issue.\n- The visual results do not show clear improvement than previous methods in Fig. 4, 5. For sketch to image generation in Fig. 7, the generated images do not align with the given inputs.\n- For k-means, how do you decide the cluster numbers? How would the noisy clustered mask affect the results? As the proxy mask is noisy, how would it achieve fine control of generated results?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your comment and for acknowledging our strengths. Our method is self-supervised and doesn't need paired mask-image collection. \n\n#### Comparison with full dataset generation using semantic segmentation models(SAM)\n\n>The key motivation of this work is that it is difficult to collect large-scale paired data for semantic generation models. But this is not challenging for the cases shown by authors, i.e., mask-to-image, sketch-to-image. For example, we can use well-trained semantic segmentation models, e.g., SAM, to perform semantic segmentation, and use the sketch extraction method, e.g., HED, Canny, to extract sketches. In this way, we can easily obtain large-scale paired training data. Also, there are also some methods, e.g., PITI that use pretraining to alleviate the data issue.\n\n\nAs mentioned in general response 2, There are still advantages to our method when the desired semantics are outside the semantic boundaries of SAM, or when a custom mask is desired.  Also, the use of SAM can address data-related challenges. However, using SAM requires training from scratch. In contrast, our approach uses a pre-trained model. This demonstrates not only annotation efficiency but also time efficiency compared to training a semantic image synthesis model from scratch. Furthermore, existing SIS models do not present results for the diverse range of conditions presented in our approach.\n\n\n#### Quality of the results\n\n>The visual results do not show clear improvement than previous methods in Fig. 4, 5. For sketch to image generation in Fig. 7, the generated images do not align with the given inputs\n\nIn Fig. 4, OASIS (one-shot) produces faces with components from multiple people e.g., ears and clothes. However, our approach does not show such characteristics. In Fig. 5, our approach aligns better than LinearGAN. For example, compared to LinearGAN, our work aligns the detailed shape of the hair better. In addition, our work also presents a more natural appearance of the face. This is because ours can give users desired styles with the control of latents. However, since LinearGAN uses random initialization of latents and latent optimization process with cross entropy loss, it is difficult to give user preferences. Furthermore, our approach shows lower FID scores compared to previous work. In Fig. 7, when comparing the shape of the hairstyle across all result images, our approach aligns with the sketch better than pSp.\n\n#### How to determine the number of clusters\n\n>For k-means, how do you decide the cluster numbers?\n\nIncreasing the number of clusters improves image quality and fine control. However, employing excessively many numbers of clusters reduces consistency between the proxy mask and human understanding, making semantic mapper hard to train. Hence, we opt for 25 clusters as an optimal choice. Appendix B.2 provides the above experiments.\n\n\n#### The proxy mask is noisy\n>  How would the noisy clustered mask affect the results? As the proxy mask is noisy, how would it achieve fine control of generated results?\n\nWe interpreted the term \"noisy clustered mask\" to mean that the proxy mask contains some noise. Even if the proxy mask is somewhat noisy, adversarial training favors the output image to be in the domain. However, the model also learns to accurately reflect the proxy mask in the image through $\\mathcal{L}_{\\text{mask}}$. Therefore, if the noise is too strong, it may corrupt the image. If our interpretation of the term \"noisy clustered mask\" is incorrect, we would appreciate additional context to clarify its meaning.\n\n#### G1 and G2\n>Are G1 and G2 in Fig. 2 the same model?\n\nFullGenerator($z$) = $G_2(G_1(z))$, where FullGenerator denotes the entire generator of StyleGAN2. In other words, $G_1$ is the composite of layers earlier than the rearranger. $G_2$ is the rest."}, {"review_id": "fql3B4owPY", "reviewer": "Reviewer_iJE3", "summary": "In this paper, authors propose to use well pre-trained StyleGANv2 for conditional image synthesis, e.g., semantic image synthesis. To achieve conditional control, a rearranging mapping network and a semantic mapper module are proposed to connect the semantic input and middle spatial feature control within the StyleGANv2 framework. Experiments show that the proposed method can generate images under various input conditions.", "questions": "  My questions are listed in the weaknesses.", "limitations": "Authors have discussed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well organized  and easy to understanding.\n2. The proposed method is flexible to various conditional inputs, such as semantic mask and sketch.", "weaknesses": "1. As mentioned in the limitation section of the paper, the quality of generated image is not very good. For example, images (FFHQ) in Figure 5 show that the shape of hair is not well aligned with the input mask. In Figure 7, the background of cat shows artifacts and the structure of hair is not match the sketch (third column). It seems that the proposed method still can not control the spatial conditions well, although it is better than previous method, LinearGAN.\n2. The ablation study can be further improved. \n    a. Why self reconstruction loss and mask loss are used different format (i.e., MSE loss and CE loss)? \n    b. What is the main factor for the performance of the method (self-supervised learning stage, or semantic guide stage)?  If we use full-shot segmentation network, can we further benefit the proposed method?\n    c. It seems that given a semantic mask, it is first transferred to a proxy mask, and then mapped to features ($F_{arr}$) along with the $F_{ori}$ (generated with random noise). An intuitive feeling is that when the randomly generated $F_{ori}$ is similar to the condition (proxy mask), the synthesis result will be better. So is the synthesis result sensitive to random noise?\n3. Some expressions are not very clear. Authors claim that existing supervised methods require pixel-level annotations and are need to be re-trained if the number of classes changes. But the proposed method still need to re-train the semantic mapper if we want to add new semantic classes. So, I'm not sure what the author wants to emphasize.\n4. With the development of large-scale vision fundament models, like SAM, we can now easily get the pixel-level annotations of images. And the results in the paper show that the performance of the proposed method is not good as full-supervised methods (OASIS). Thus, I suggest to provide more detailed analysis of the advance and potential applications of the proposed method. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our advantages: Our method is flexible to diverse conditions.\n\n#### Quality of the results\n\n>As mentioned in the limitation section of the paper, the quality of generated image is not very good. For example, images (FFHQ) in Figure 5 show that the shape of hair is not well aligned with the input mask. In Figure 7, the background of cat shows artifacts and the structure of hair is not match the sketch (third column). It seems that the proposed method still can not control the spatial conditions well, although it is better than previous method, LinearGAN.\n\nWe use masks at $64^2$ resolution and the images are $1024^2$ resolution. The gap between mask and image inevitably leads to limited precision in control such as fine hair. These are mentioned in the limitation. Similarly, the sketch in Figure 7 also becomes a proxy mask that is low resolution to generate the image. Therefore, it does not reflect a perfect alignment of hair details. Nevertheless, ours achieve much better alignment than LinearGAN and pSp. Artifacts in the background of the image occur in only a handful samples. As shown in Figure R4 in the PDF, most of the images generated with styles and scribbles as conditions still show a strong ability to reflect the given conditions and maintain high quality without artifacts.\n\n#### Ablation\n>a. Why self reconstruction loss and mask loss are used different format (i.e., MSE loss and CE loss)? \n>b. What is the main factor for the performance of the method (self-supervised learning stage, or semantic guide stage)? If we use full-shot segmentation network, can we further benefit the proposed method? \n>c. It seems that given a semantic mask, it is first transferred to a proxy mask, and then mapped to features (F_arr) along with the F_ori (generated with random noise). An intuitive feeling is that when the randomly generated is similar to the condition (proxy mask), the synthesis result will be better. So is the synthesis result sensitive to random noise?\n\n**a. Design choice of loss functions** We used cross-entropy to align pixel classes in predicted and target masks, a common loss function for semantic segmentation and image synthesis tasks involving mask prediction. However, for feature maps, it is a regression problem rather than a pixel-by-pixel classification. Therefore, we used the L2 loss accordingly.\n\n**b-1. Main factor of performance**  The main goal and advantage of our work is to perform semantic image synthesis using the pre-trained unconditional generator, rather than performance improvement. Therefore, the Rearranger, the Semantic Mapper and the training strategies are all the main factors.\n\n**b-2. Full-shot segmentation network** Table 2 also shows that performance increases with increasing annotation. Additionally, we observed a better result in FID (15.37) when using the pre-trained segmentation network with CelebA-MaskHQ instead of the one-shot annotation. When we use the mIoU metric also increased from 53.10 to 60.92. \n\n\n**c. Independence between the input and condition** \nIn Appendix D. Figures S9 and S10, the diagonal direction shows images generated with feature maps and proxy masks from the same latent. The results show that the reproduction is almost identical to the original. However, when we looked at other images, we found that using feature maps and proxy masks generated from different latents did not affect the image quality. Note that this is different from giving a custom mask as a condition. For a quantitative comparison, we measured the FID and found that the images generated from random latent pairs had an FID of 7.48, while the images generated from the same latent had an FID of 9.90.\n\n\n\n\n#### Flexibility to changing conditions\n\n>Some expressions are not very clear. Authors claim that existing supervised methods require pixel-level annotations and are need to be re-trained if the number of classes changes. But the proposed method still need to re-train the semantic mapper if we want to add new semantic classes. So, I'm not sure what the author wants to emphasize.\n\n\nOur method also needs to be retrained when the number of classes changes. However, our method consists of two modules, the Rearranging Network and the Semantic Mapper. The training time for this is only **12 hours** for CelebA-MaskHQ ($256^2$), since only the latter needs to be retrained if the input conditions or the number of classes change. On the other hand, given that the SIS competitor takes 76 hours, it is more advantageous to respond to changes in input conditions, such as changes in the number of classes. (see results in Figure R3 in the PDF)\n\n\n\n#### Efficiencies of the proposed method\n\n>With the development of large-scale vision fundament models, like SAM, we can now easily get the pixel-level annotations of images. And the results in the paper show that the performance of the proposed method is not good as full-supervised methods (OASIS). Thus, I suggest to provide more detailed analysis of the advance and potential applications of the proposed method.\n\n\nAs mentioned in general response 2, There are still advantages to our method when the desired semantics are outside the semantic boundaries of SAM, or when a custom mask is desired. Also, the advantage of our method lies not only in its data efficiency but also in the time efficiency for training compared to training an SIS model from scratch. Furthermore, unlike supervised SIS methods, our method is also versatile enough to generate images that reflect various types of input conditions, such as sketches."}, {"review_id": "iP25XI3a02", "reviewer": "Reviewer_4iJ7", "summary": "This work proposes a novel method for semantic image synthesis from an unconditional generator. Specifically, a proxy mask is used to provide semantic guidance to reararange the middle layer's feature. A segmentation module is learned to map user input to the proxy mask in a few-shot manner. Experiments over different datasets show proposed method's effectiveness.", "questions": "The image quality is not as good as previous methods for full-shot setting. Why does this happen? I hope the authors could provide some analysis and hypotheses, and it would be even better if they could conduct experiments to support their claims.", "limitations": "The authors have discussed the limitations in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Semantic image synthesis typically requires learning from a large amount of paired data. However, this work relaxes this requirement to just a single pair. This approach is innovative and has the potential to unlock new applications that enable the control of image synthesis through customized signals.\n\n2. The authors have implemented thorough experiments over different dataset and different kinds of control signals (seg, sketch, scribble, etc.).", "weaknesses": "1. The formulation is messy. For example, $m_{proxy}$ and $K$ (in Equation 2) are undefined when they are first mentioned in the paper. \n\n2. There are many loss terms proposed. However, the authors do not provide ablation study results over them.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the advantages of our paper: While conventional methods require a large amount of paired data for training, ours is innovative in that it requires only a single pair to enable training. Furthermore, our approach has the potential to be applied to facilitate the control image synthesis.\n\n#### Expressions\n>The formulation is messy. For example, m_proxy and K (in Equation 2) are undefined when they are first mentioned in the paper.\n\n\nThanks for pointing out the formulation, we will fix them in our revised version.\nm_proxy denotes proxy mask, $K$ denotes K-means clustering algorithm, $G_1$ is the earlier layers of the StyleGAN up to the layer corresponding to the feature maps used to generate the proxy mask. $G_2$ is the rest of the layers of the model.\n\n>There are many loss terms proposed. However, the authors do not provide ablation study results over them.\n\nThe ablation study of using loss terms to train a Rearranging Network is shown in Appendix B.1 of our paper. Appendix Fig. S3 shows that reconstruction loss and mask loss play a crucial role in the training. There is an additional ablation study for Semantic Mapper loss terms below. Training with the adversarial loss while training Semantic Mapper helps to get better quality with FID 18.53, although we can get proper images without using adversarial loss. The quantitative results are in the table below.\n|        | Semantic Mapper with adv loss | Semantic Mapper without adv loss |\n|--------|----------------------------------|-------------------------------|\n| FID    | 18.53                            | 21.30                         |\n| mIoU   | 53.10                            | 55.12                         |\n\n\n#### Quality of the results\n\n>The image quality is not as good as previous methods for full-shot setting. Why does this happen? I hope the authors could provide some analysisma and hypotheses, and it would be even better if they could conduct experiments to support their claims.\n\n\nThe general response 1. shows that our method performs well in one-shot training. The main goal and advantage of our method is to use pretrained generator without many annotation pairs.  The performance of one-shot is highly dependent on the annotation of a single image, so pixel-level mislabeling affects the image quality. Figure R1 in the PDF shows the corresponding failure cases. In addition, as mentioned in the general response 1, the performance increases when pretrained segmentation model is added instead of one-shot segmentation model. (FID 15.37)"}]

Author rebuttal:
# Global Response
We thank all the reviewers for their insightful reviews. We first summarize the strengths of our paper recognized by the reviewers.


1. Our method can apply unconditional GANs to SIS, which has the potential to leverage recent advancements from other GAN literature.
2. The results demonstrate the flexibility of our approach to handle various conditional inputs and its applicability to diverse datasets. 
3. Training process is self-supervised, eliminating the need for paired data collection. 
4. Ours have been conducted thorough experiments over different datasets and control signals, further validating the effectiveness of our approach. 
5. One-shot performance on FID, mIoU, and accuracy metrics is impressive, showcasing its effectiveness in various evaluation aspects.


Next, we aim to address common concerns raised by reviewers.

**1. Comparison with supervised method in image quality**(pArX, iJE3, 4iJ7)
First, our method focuses on one-shot learning, so it is not fair to compare it with a fully annotated supervised method. For our targeted one-shot learning, our method outperforms competitors, including supervised methods. However, due to extreme annotation limitations, the efficacy of the mapper might be constrained. 
Therefore, we conducted additional experiments to confirm whether increased supervision improves performance. For the experiments, we utilized a pretrained segmentation network for CelebA-MaskHQ instead of one-shot segmentation network. All other models are the same. In this setting, as in the case of using sketches as an input condition, the output of the generated image passed through Unet and the proxy mask made from the feature maps corresponding to the image were used as a training pair to train the semantic mapper. Consequently, the FID improved to 15.37, indicating enhanced the image quality.

**2. Pretrained vision models addressing annotation problem** (pArX, iJE3)
There have been opinions on how pretrained segmentation models like SAM can conveniently address annotation issues. 
Although SAM covers many domains, it cannot cover all the semantics in different target domains. On the other hand, our method allows the use of a customized mask that contains the desired semantic information from an example image.   In addition, there is a gap with human drawings such as HED and Canny, which can be overcome by using drawings as annotations. In this way, we can use directly the user drawings as conditions when generating images.
Despite the limitation of SAM, utilizing it can facilitate the acquisition of a large paired dataset. However, conventional methods using lots of mask-image pair datasets need to require training models from scratch using the acquired dataset, leading to computation challenges. In contrast, our approach eliminates the need for from-scratch training, providing not only data efficiency but also computational efficiency. Further detailed comparisons between from-scratch and our method can be found below.



Meta-review: Overall, the paper received positive feedbacks. The writing is clear and the proposed framework enables SIS with unconditional GAN with the help of proxy masks prediction. Authors provided additional ablations and evaluations to resolve most of the concerns raised by reviewers. Please revise the final manuscript accordingly upon acceptance.
Decision: Accept (poster)