BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation
Authors: Mingcong Liu
Qiang Li
Zekui Qin
Guoxin Zhang
Pengfei Wan
Wen Zheng
Venue: NeurIPS 2021 Poster (2021)
Keywords: GAN
image generation
stylized face synthesis
arbitrary style transfer

Abstract: Generative Adversarial Networks (GANs) have made a dramatic leap in high-fidelity image synthesis and stylized face generation. Recently, a layer-swapping mechanism has been developed to improve the stylization performance. However, this method is incapable of fitting arbitrary styles in a single model and requires hundreds of style-consistent training images for each style. To address the above issues, we propose BlendGAN for arbitrary stylized face generation by leveraging a flexible blending strategy and a generic artistic dataset. Specifically, we first train a self-supervised style encoder on the generic artistic dataset to extract the representations of arbitrary styles. In addition, a weighted blending module (WBM) is proposed to blend face and style representations implicitly and control the arbitrary stylization effect. By doing so, BlendGAN can gracefully fit arbitrary styles in a unified model while avoiding case-by-case preparation of style-consistent training images. To this end, we also present a novel large-scale artistic face dataset AAHQ. Extensive experiments demonstrate that BlendGAN outperforms state-of-the-art methods in terms of visual quality and style diversity for both latent-guided and reference-guided stylized face synthesis.

BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation
Mingcong Liu Y-tech, Kuaishou Technology Luumingong@@@kuuishhoo....oo com
Qiang Li Y-tech, Kuaishou Technology liqi iigiang03@kuiishou com
Zekui Qin
Guoxin Zhang Y-tech, Kuaishou Technology zhangguoxinkkuaishou. com
Y-tech, Kuaishou Technology
qinzekui00@kuuaishouu com
Pengfei Wan Y-tech, Kuaishou Technology banpengfel wwt.rifehiuuuuusu...omoooo com
Wen Zheng
Y-tech, Kuaishou Technology zhengven@kuaishou. com
Abstract
Generative Adversarial Networks (GANs) have made dramatic leap in high- fidelity image synthesis and stylized face generation. Recently, layer-swapping mechanism has been developed to improve the stylization performance. However, this method incapable of fitting arbitrary styles single model and requires hundreds of style-consistent training images for each style. To address the above issues, we propose BlendGAN for arbitrary stylized face generation by leveraging flexible blending strategy and generic artistic dataset. Specifically, train self-supervised style encoder on the generic artistic dataset to extract the representations of arbitrary styles. In addition, weighted blending module (WBM) is proposed to blend face and style representations implicitly and control the arbitrary stylization effect. By doing so, BlendGAN
first
gracefully fit arbitrary styles in unified model while avoiding case-by-card preparation of style-consistent training images. To this end, also present novel large-scal scale artistic face dataset AAHQ. Extensive experiments demonstrate that BlendGAN outperforms state-of- the-art methods in terms of visual quality and style diversity for both latent-guided and reference-guided stylized face synthesis. Our project wehpage is httpss:/onionn liu iithub.io/BlenGGAN
Introduction
Generative adversarial networks (GANs) have achieved impressive performance in various tasks such as image generation 3 4 5 6 9. image super-resolution [T], and image translation In recent years, GAN has also been widely used for face stylization such as portrait drawing 20 caricature 22 manga 23], and anime (24.. With the deepening of
12
13
14
15
16.
17
18
19
GAN research, the community has further paid attention the quality of generated images and the disentangiement ability of latent space.
To achieve high-quality generation and latent disentanglement, StyleGAN [8] introduces the adap- tative instance normalization layer (AdaIN) 25 and proposes a new generator architecture, which
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Reference
Source
Figure 1: Illustration of some reference efference-guided synthesis results. Our framework generate stylized face images with high quality.
achieves pretty good performance in generating face images. StyleGAN2 19 explores the causes of droplet-like artifacts and further improves the quality of generated images by redesigning generator normalization. StyleGANZ-ada [26] proposes an adaptive discriminator augmentation mechanism which reduces the required number of images several thousands. In the experiment of StyleGAN2- ada, the model is trained on MetFaces dataset [26] to generate artistic faces, however, the style of image is uncontrollable, and the corresponding original face cannot be obtained.
Recently, a layer-swapping mechanism (27 has been proposed to generate high-quality natural and stylized face image pairs, and equipped with additional encoder for unsupervised image translation [28]. In particular, these methods first finetune StyleGAN from a pretrained model with target-style faces and then swap some convolutional layers with the original model. Given randomly sampled latent codes, the original model generates natural face images, while the layer-swapped model outputs stylized face images correspondingly. Though effective large, these methods have train models in case-by- base-by-case flavor, thus a single model can only generate images with a specific style. Besides, these methods still require hundreds of target-style images for finetuning, and one has carefully select the number of training iterations, as well as the swapped layers.
In this paper, we present BlendGAN for arbitrary stylized face generation BlendGAN resorts to flexible blending strategy and a generic artistic dataset to fit arbitrary styles without relying on style- consistent training images for each style. In particular, of two latent parts: a face code (controlling the face attributes) and a style code (controlling the artistic appearance). Firstly, self-supervised style encoder is trained via instance discrimination objective to extract the style representation from artistic face images. Secondly, a weighted blending module (WBM) is proposed to blend the face and style latent codes into final code which is then fed into a generator. By controlling the indicator in WBM, we are able to decide which parts of the face and style latent codes to be blended thus controlling the stylization effect. By combining the
analyze stylized face image composed
Our framework can also cooperate with GAN inversion 29
30 31
32
33
or StyleGAN distillation 34
methods to enable end-to-end style transfer image translation. We will show the results in the appendix.

---

Generator
Discriminator
Real/
Disce
Fake
Pugge
3,
MAP.
Real/
WBM
Dayle
Fake
2/ 1)0.)
w/
Real/
Desie latent
Fake
2,
Figure Overview of the proposed framework. The style encoder Estyle extracts the style latent code Z, of reference style image. The face latent code y is randomly sampled from the standard Gaussian distribution. Two MLPs transform face and style latent codes into their W spaces separately, then they are combined by the weighted blending module (WBM) and fed into generator G to synthesise natural and stylized face images. Three discriminators are used our method. The face discriminator D face distinguishes between real and fake natural-face images, the style discriminator Datyle distinguishes between real and fake stylized-face images, and the style latent discriminator Dstyle_latent predicts whether the stylized-face image is consistent with the style latent code
style encoder and the generator with WBM, our framework can generate natural and stylized face image pairs with either randomly sampled style or reference style (see Figure I.
As for the generic artistic data, we present a novel large-scale dataset of high-quality artistic face images,
(AAHQ), which covers wide variety of painting styles, color tones, and face attributes. Experiments show that compared state-of-the-ang methods, our framework can generate stylized face images with higher visual quality and style diversity for both latent-guided and dnferrece-guided synthesis.
Related Work
Generative Adversarial Networks The recent years have witnessed rapid advances of generative adversarial networks (GANs) I for image generation. The key to the success of GANs is the adversarial training between the generator and discriminator. Various improvements to GANs have been proposed to stabilize GAN training and improve the image quality. WGAN 3 uses the Wasserstein distance as new cost function train the network. WGAN-GP [4] and SNGAN improve the stability of GAN by introducing gradient penalty and spectral normalization separately make the training satisfy the i-Lipschitz constraint. SAGAN (35 enlarges the receptive field using the self-attention mechanism. BigGAN [6] shows the significant improvement of image quality when training GANs at large scales. StyleGAN [8] 9 redesigns the generator architecture with AdaIN layers, making it better disentangle the latent factors of variation. In this work, we employ GANs to generate high-quality stylized face images.
Image-to-Image Translation With the good performance of GANs, many GAN-based image-to- image translation techniques have been explored in recent years 12 13
16
17
36
[8]
37 19
38 For image translation in two domains, Pix2pix [12] proposes the first unified image-to-imant framework to translate images to another domain. CycleGAN [3] proposes cycle-consistency loss, making the network train with unpair data. UNIT 14 maps images in source and target domains to a shared latent space to perform unsupervised image translation. UGATIT 24 proposes an adaptive layer-instanee normalization (AdaLIN) layer control shapes during translation. For image translation in multi-domains, MUNIT [6 extends UNIT to multi-mode contexts by decomposing images into content and style spaces. FUNIT 17 leverages few-shot strategy to translate images using a few reference images from target domain. DRIT++ [18 also proposes multi-modal and multi-domain model using discrete domain encoding. Training with both latent codes and reference images, StarGANv2 [19 could provide both -giided and reference- reference-guided synthesis. Park et.al.

---

Style Encoder
matrix & concat
Z5
angmennt
VGG-/ 19
MLP
MLP
GNT-Xent
(fixed)
$12-D
projection
610,304-D
Figure 3: Architecture of the proposed style encoder, which consists a pretrained VGG-19 /9 44 network, Gram matrix and concal module and MLP predictor. More detailed notations described in the contexts Sec.
utilize the autoencoder structure to swap the style and content of two given images, and it can synthesis method. Although these methods could translate images to multiple target domains, the diversity of target domains is still limited and these methods require hundreds of training images for each domain. Besides, the above methods could only operate at resolution of at most 256x256 when applied to human faces.
38
also be regarded as a reference-guiee
Neural Style Transfer The arbitrary stylized face generation task could also be regarded kind of the neural style transfer [39 40 25). Gatys et al. [39], for the first time, proposes a neural style transfer method based on optimization strategy. Johnson al. [40] trains feed-forwart network to avoid the time consumption of optimization. Luan et al.
[4] proposes a photorealism regularization term to preserve the structure of source images. The above methods could only transfer source images to a single style with a model, many works extend the style flexibility and achieved multi-style or arbitrary style transfer such as MSG-Net [4]], StyleRemix 43 AdaIN (55) WCT [44] and SANet
45] Although arbitrary style transfer methods could transfer images to arbitrary styles, they only
transfer the reference style source images globally without considering local semantic styles, which leads to global texture artifacts in the outputs.
3 Method
Our goal is to train generator
8, = Gizy, Zo. Zo,
(I)
that generate image pair (natural face image ê and its artistic counterpart ..) with pair of given latent code (zy, 2.) and a blending indicator The face latent code z controls the face identity, and the style latent code z, controls the style of t.. The blending indicator i decides which parts of y and z, are blended to generate plausible images. Figure illustrates an overview of our framework.
3.1 Self-supervised Style Encoder
The style encoder Estyle is independently trained extract the style representation from an artistic face image while discarding the content information such as face identity. Many style transfer methods [39 40 4T obtain the style representation by caleulating the Gram matrices of feature maps extracted by pretrained VGG [46] network. However, if simply concatenating the Gram matrices of all the selected feature maps, the resulting style latent code ('style embedding" ) would have very high dimension. For example, if we choose layers relul_2 2_,, relu2 2, relu3 4, relu4 4 and relu5_4, the style embedding would be 610,300-444 domeen-dimmesionalll vector (64 + 128 + 256 5122 2). Higher latent code dimension leads to sparser distribution, which makes the generator harder train and perform worse in style disentanglement. Hence, following the self-supervised method SimCLR 47 48 we train the style encoder to reduce the dimension of style embeddings.

---

Figure 3] illustrates the architecture of our style encoder. The style image is augmented and then sent to a pretrained VGG [46] network. We calculate the Gram matrices of selected feature maps. flatten and concatenate them into long vector (610,304-I) in our case). Then the vector is passed into a 4-layer MLP module (predictor) to reduce the dimension and generate the style embedding 2, of the input style image. Following 22, another 2-layer MLP performs as a projection head for contrastive learning. The augmentations in the original SimCLR include affine transformation and color transformation, following the assumption that affine and color transformations do not change the classes of objects. However, for style encoding, the image style is strongly related to color, so we only use affine transformations in the augmentation step. During training the style encoder, only parameters of two MLP modules are optimized while VGG-19 is fixed to perform as a feature extractor. The network is trained using NT-Xent loss 47)
exp(sim(zz,
Xent = log
Ek=1 k=1 1 [1-4]] exp(sim(z; (z;, =x)/T)²
22
where (i.j) is pair of positive examples (augmented from the same image), z is the projection result, -)) is cosine similarity between two vectors, and T is temperature scalar.
3.2 Generator
The generator G is similar to StyleGAN2 [9] but its input is a combination of two latent codes face latent code y and style latent code gg- In particular, 2 controls the face identity of the generated images, and is randomly sampled from the standard Gaussian distribution N(0, 1). 2, controls the style of the generated stylized image, it could be either randomly sampled from NO, 1) or encoded by the style encoder Estyle from a reference artistic image. Two MLPs transform 2 and z, into their W spaces as w and w, separately. For 1024 1024 output, the dimension of w and w, are all 18 x 512. Controlled by the blending indicator i, the weighted blending module (WBM) combines w, and w into w, which is then fed into the generator as the final code.
WBM Different resolution layers in the StyleGAN model are responsible for different features in the generated image [8] (e.g. low resolution layers control face shape and hair style, high resolution layers control color and lighting). Therefore, the blending weights of w, and w should not be consistent at different layers. We propose WBM to blend the two codes, which can be described as:
W = W. ② à + w @ (1 (1-a)) a),
(3)
à = m m(i),
(4)
j < i
(i: [mo, m1, m2....,, m,], m,
0 j = 8 (0,
(5)
j >
where @ denotes the broadcasting element-wise product, Was W E R18x512 are the input codes, E IR 18 a learnable vector to balance the blending weights for w, and wy in different layers. m(i; ()) has the same dimension as a, it controls which layer should be blended. If i = 0, w w, C a + w @ (1 - a). the generator outputs stylized face image A.; if = 18, w w, the generator outputs natural face image žy. When 0 < i < 18, some low resolution layers are not influenced by style codes, which ensures the T. to keep the same face identity as é (see Figure 2 is used for finer adjustment.
3.3 Discriminator
There are three discriminators in our framework(see the right column in Figure 2) The face dis- criminator Dace and the style discriminator Datyle have the same architecture as in StyleGAN2 19]. D jace distinguishes between real and fake natural-face images, it only receives images sampled from the natural-face dataset (FFHQ) or generated by G' when i = 18. In the contrast, Datgle distinguishes between real and fake stylized- -iizeed-fceeeee images and it only receives images sampled from the artistic face dataset (AAHQ) or generated by G when i = O. The style latent discriminator D Dstyle_lace latent has the same architecture as the projection discriminator [49], which has two inputs a generated stylized-face image ã., and a style latent code z,, it predicts whether i. is consistent with z,, During training the network, we use an embedding queue to storage the style latent codes of the previously sampled style images embedded by E Estyle, and randomly sample as the fake Zg.
5

---

3.4 Training objectives
Given a natural-face image zy E Naturral a stylized-face image Vs Xstyle and a randomly sampled face latent code z ~ N(0, 1), we train our framework using the following adversarial objectives.
For the face discriminator, the blending indicator i = 18. G only takes 2j as the input and learns to generate natural-face image f (zz) via an adversarial loss
L face = Ex, [log D face (x } + E,, log (1 Doce Gi=is(Zy)],
(6)
where Dace (.) denotes the output of the face discriminator.
For the style discriminator, the blending indicator i 0. G takes 2 and inputs, and generate stylized face image Ve which has the same style as 2.. The loss is described as
styye E., llog D (It)) Ez, [log (1 Dstyle (Giuu Estyle 1))]
where the style encoder Estyle extracts the style latent code of x, and D (+tyle()) denotes the output of the style discriminator.
For the style latent discriminator, we denote z, = Extyle( (I.) as the style latent code of Is, and randomly sample another style latent code n from the embedding queue as a negative sample, then the loss could be described as
Sstyle_ Ezj.e, [log D Datyle laaele_lltetttt____ 2.)l + Eg [log (1 D Datyle_latenteeeeeeeeeee Jatent 2,
(8)
Consequently, we combine all the above loss functions as our full objective as follows:
LG = & face + Leayle + Latyle latents Lstylellatents
(9)
LD -GG.
(10)
4 Experiments
Our proposed method is able to generate arbitrary stylized face images with high quality and diversity. In this section, describe evaluation setups and test method both qualitatively and quantitatively on a large amount of images spanning a large range of face and style varieties.
Datasets As described in Sec. 3 for arbitrary stylized face generation, we need a natural-face dataset and an artistic face dataset to train the networks. We use FFHQ ( as the natural-face dataset, which includes 70,000 high-quality face image In addition, we build new dataset of artistic- images, Artstation-Artistic-- Artistic-face-HQ (AAHQ), consisting of 33, 33,245 high-quality artistic faces at 1024 resolution (Figure 4p. The dataset covers a wide variety in terms of painting styles, color tones, and face attributes. The artistic images are collected from Artstation (thus inheriting all the biases of that website) and automatically aligned and cropped as FFHQ Finally, we manually remove images without faces or with low quality. More details of this dataset can be found in the appendix.
Baselines We compare our model with several leading baselines on diverse image synthesis includ- ing AdaIN (25). MUNIT (16), FUNIT [17), DRIT+ [18), and StarGANv2 [19]. All of these methods could synthesise images in different modalities through the control of latent codes or reference im- ages. All the baselines are trained on FFHQ and AAHQ using the open-source implementations
provided by the authors, and our BlendGAN code is based on an unofficial PyTorch implementation of StyleGANT
Evaluation metrics
To evaluate the quality of our results, we use Frechet inception distance (FID) metric [50] to measure the discrepancy between the generated images and AAHQ dataset. A lower FID score indicates that the distribution of generated images are more similar to that of AAHQ dataset and the generated images are more plausible as real stylized-face images. In addition, we
"The FFHQ dataset is under Creative Commons BY-NC-SA 4.0 license.
hhtps://www.rrtstaiioo
https//githbbbbommmoooosiatttttt

---

Figure 4: The AAHQ dataset offers a lot of variety in terms of painting styles, color tones and face attributes.
Table I: FID and LPIPS comparison with different blending indicators.
Indicator
2
12
14
16
FID +
8.97
12.45
12.75
23.17
42.45
57.34
68.31
76.91
78.25
76.73
latent
LPIPS 1
0.581
0.571
0.568
0.515
0.459
0.367
0.304
0.207
0.145
0.159
FID 4
3.79
6.39
6.82
15.08
34.33
51.49
63.73
76.00
77.11
76.97
reference
LPIPS
0.661
0.651
0.650
0.599
0.540
0.450
0.377
0.237
0.191
0.160
adopt the learned perceptual image patch similarity (LPIPS) metric [1 to measure the style diversity of generated images. Following [9]. as for face latent code (corresponding to a natural-face image), we randomly sample 10 style latent codes or reference style images to generate 10 outputs and evaluate the LPIPS scores between every 2 outputs. We randomly sample 1000 face latent codes, repeat the above process and average all of the scores as the final LPIPS score. A higher LPIPS score indicates that the model could generate images with larger style diversity.
4.1 Blending indicator
As described in Sec. 3.2. the indicator in WBM controls in which layers the weights of w, and w should be blended. In StyleGAN generator, low-resolution layers are responsible for face shape and high-resolution layers control color and lighting. Hence, i gets larger, less low-resolution layers influenced by style codes, and the face shapes of stylized outputs are more similar to the natural-face images. As shown in Figure 5 when as the most different face identities from the corresponding natural-face images. When 18, the generated images completely lose the style information and become natural-face images. Table [ also shows that the style quality and diversity of the generated images decrease with the increase of i. We notice that when 6, the generated images have good balance between stylization strength and face shapes consistency with the natural-face images. To this end, we set i = in the qualitative experiments, while we use both 0 and in the quantitative experiments.
O, the generated images have the strongest style well
4.2 Comparison on arbitrary stylized face generation
Our framework can generate stylized-face images with two mechanisms: latent-guided generation and reference-guided generation.
Latent-guided generation. For latent sutedt- generation, the style latent code z, is randomly sampled from the standard Gaussian distribution. ( qualitative comparison of the competing methods. The results of AdaIN (55 are not shown because it is not able to take latent codes as the style inputs. Since FUNIT is designed for few-shot image-to-image translation, could not generate face images when inputting randomly sampled style latent codes. The results of MUNIT and DRIT++ have severe visible artifacts and they cannot reserve face identities in some images. StarGANy2 has the best performance among the baselines but the generated images still have some
7

---

Reference
i=
i= 2
i=4
i=6
i=8
==10
i=12
i=14
i=16
i=18
Figure 5:
Reference-guin guided stylized-face images with different blending indicators.
Table 2: Quantitative comparison latent-
Table 3: Quantitative comparison on reference-
guided stylized-face generation.
guided stylized-face generation.
Method
FID |
LPIPS 1
Method FID 4
LPIPS
AdaIN
37.23
0.345
MUNIT 29.69
0.394
MUNIT
103.61
0.192
FUNIT
176.96
0.500
FUNIT
87.71
0.327
DRIT+
22.53
0.448
DRIT+
31.71
0.241
StarGANv2
50.20
0.312
StarGANv2
50.03
0.307
BlendGAN (i 23.17
0.515
BlendGAN 6)
15.08
0.599
BlendGAN (i 0) 8.97
0.581
BlendGAN (i 0)
3.79
0.661
subtle artifacts and are not as natural as our results. We observe that the generated stylized-face images of our method have the highest visual quality, and well preserve the face identities of the source images. Quantitative comparison shown Table Although when
the FID score of our method is slightly higher than DRIT++, the qualitative comparison in Figure shows that our stylized images have less artifacts and higher visual quality. When O, our method has the lowest FID score and the highest LPIPS, indicating that our model could generate stylized images with the best visual quality and style diversity.
Reference-guided generation. For reference-guides generation, the style latent code z, embed- ded by the style encoder Estyle from a reference artistic face image. Figure ( iillustrates qualitative comparison results. The results show that the style transfer method AdaIN style information and introduces severe visible artifacts in their generated stylized faces. FUNIT and DRIT+ also lead visible artifacts in some images, as shown the first and fourth rows. Although MUNIT allows to input reference image at the testing stage. it only transfers the average color of the reference images, and the results do not have similar textures the references. StarGANv> has the best performance among the baselines, however, the style of some generate images is not consistent with their references, as shown in the fourth and fifth rows. As comparison, our BlendGAN generates stylized-face images with the highest quality and the style of generated images is the most consistent with the references thanks to our well-designed training strategy. Quantitative comparison is shown in Table 3 For = 0 and outperform all the previous leading methods by large margin. This implies that images generated by our model are the most similar to real stylized-far facc images. The LPIPS of our method is also
not consider semantic
6, our method achieves FID of 3.79 and 15.08, which

---

MUNIT
TUNIT
SarGANd
AJAIN
SUIGIN
Figure 6: Comparison of latent-guided generation. The source images gener- ated by our model with indicator = 18, and the style latents are randomly sam- pled from (0, 1). From left to right: the source image, the results of MUNIT (16), FUNIT I7]. DRIT+ (18), StarGANv2 19 and our BlendGAN. Digital zoom-in recommended.
Figure 7: Comparison of reference guided generation. The source images are generated by model with in- dicator = 18, and the reference images are sampled from the AAHQ dataset. From left to right: the refer- ence artistic-face image. the source natural-face image, the results of AdaIN [25], MUNIT [66. FUNIT [I7], DRIT+ [18), StarGANv2 [99 and our BlendGAN, Dig- ital zoom-in recommended.
the highest, indicating that our model can produce faces with the most diverse style considering the reference images. Besides, it is worth noting that method with i = 0 performs better than i 6 because low-resolution layers are not affected by indicator
Conclusions
In this paper, propose novel framework for arbitrary stylized face generation. Training with FFHQ and new large-scale artistic face dataset (AAHQ), our framework can generate infinite arbitrary stylized faces well their unstylized counterparts, and the generation could be either latent-guided or reference-guided. Specifically, we propose a self-supervised style encoder to extract the style representation from reference images, as well as a weighted blending module to implicitly combine the face and style latent codes for controllable generation. The experimental results show that our proposed model could generate images with high quality and style diversity, and the generated images have good style consistency with reference images, remarkably outperforming the previous leading methods. Our experiments are currently only performed on face datasets, future work includes extending this framework to other domains, such stylized landscape generation.
Broader Impact
This work would be of great interest to the general community. It shows the great potential of applying GAN models to stylized image generation, which could be further used for other tasks, such as style transfer, style manipulation, and new style creation.
Although the goal of this work is to generate faces for stylistic purposes, it still has some ethical well documented that common face datasets lack diversity [52]. As our model is trained using the FFHQ g dataset, it inherits the biases of this
challenges like other face generation methods.
Specifically, similar to those discussed in the previous works 54 53 for natural-face images, the model generates light-skinned faces more frequently than darker-skinned faces, and this concern can be mitigated by enlarging the racial diversity of the natural-face dataset. For stylized-face images, our AAHQ dataset already covers
dataset like other StyleGAN- based methods 27 28 53

---

wide diversity of faces (see Figure 4 and Section B of the supplementary materials), In addition, since our framework only extracts the style representation of images in the AAHQ dataset and does not extract information related to face identity, the ethical biases (if any) of this dataset will not be transferred to the generated images. Therefore, the stylized-face images generated by our model have equally good performance for both light-skinned and darker-skinned faces (as shown in Figures and 7). which further indicates the superiority of our framework. Nonetheless, our model still has the same challenges as PULSE [53] (i.e., input images with darker skin are now stylized as lighter-skinned faces in the output), and these challenges are worthy of further research.
[
Besides, though not the purpose of this work, the natural-face images generated by our model could also be misused in the "deepfakes" relevant applications, such as fake portraits in social media [55]. Wang et al. [56] showed that classifier trained to classify between real images and synthetic images generated by ProGAN 77). was able to detect fakes produced by other generators, among them, StyleGAN 8 and StyleGAN2 19. This finding can partially mitigate the above concern.
Last but not least, faces in the FFHQ and our AAHQ datasets are biometric data and thus need to be sourced and distributed more carefully, with attention to potential privacy, consent and copyright issues.
Acknowledgments and Disclosure of Funding
We sincerely thank all the reviewers for their comments. We also thank Xin Miao for helpful discussions as well as the implementation (57] of the projection discriminator [49]. Besides, we
thank Zhenyu Guo for help in preparing the comparison to StarGANv2 [9]. No external funding was received for this work.
References
[1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014,
[]
[2] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv: 2014. 1
[3] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[4] Ishaan Gulrajani. Faruk Ahmed, Martin Arjovsky. Vincent Dumoulin, and Aaron Courville. Improved training of wasserstein gans. In Proceedings the 31st International Conference Neural Information Processing Systems, pages 5769-5779, 2017.
[5] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018
[6] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference Learning Representations, 2018
[7] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lchtinen. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations, 2018. 10
[8] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4401-4410, 2019.
10
[9] Tero Karras, Samuli Laine, Miika Aittlla, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 81 10-8119, 2020.
10
[10] Christian Ledig. Lucas Theis, Ferene Huszár, Jose Caballero. Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super- resolution using generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681-4690, 2017.
10

---

[11] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European
Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018. T
[12] Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexci A Efros. Image-to-image translation with conditional adversarial networks. Proceedings of the IEFF conference on computer vision and pattern recognition, pages 1125-11344 134, 2017.
[13] Jun-Yan Zhu, Tacsung Park, Phillip Isola, and Alexci Efros. Unpaired image-to- to to-image translation using consisetn adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages
2017. 33
[14] Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Proceedings of the 31st International Conference Neural Information Processing Systems, pages 700-708, 2017.
[15] Taesung Park, Ming- Yu Liu, Ting- Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2337-2346, 2019.
[16] Xun Huang, Ming- Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European conference on computer vision (ECCV), pages 172-189, 2018.
6 9
[17] Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few- shot unsupervised image-to-image translation. In Proceedings of the IEEF/CVF International Conference
on Computer Vision, pages 10551-10560, 2019.
9
[18] Hsin Ying Lec, Hung- Yu Tseng, Mao, Jia Bin Huang. Yu-Ding Lu, Maneesh Kumar Singh, and Ming- Hsuan Yang. Drit++: Diverse image-to-ime pongeett-imman -imaa translation viadisentangled representations. International Journal of Computer Vision, pages 1-16, 2020.
3 6
[19] Yunjey Choi, Youngjung Uh. Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Pruceedings of the IEEE Conference on Computer Vision and Puttern Recognition,
2020. I 3 6
7
10
[20] Ran Yi. Yong- Jin Liu, Yu-Kun Lai, and Paul L Rosin. Apdrawinggan: Generating artistic portrait drawings from face photos with hierarchical gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10743-10752, 2019.
[21] Kaidi Cao, Jing Liao, and Yuan. Carigans: unpaired photo-to-caricature translation. ACM Transactions on Graphics (TOG), 37(6): 14, 2018.
[22] Yichun Shi, Debayan Deh, and Anil K Jain. Warpgan: Automatic caricature generation, Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10762-10771, 2012 D
[23] Hao Su, Jianwei Niu, Xuefeng Liu, Qingleng Li, Jiahe Cui, and Ji Wan. Unpaired photo-to-mang: translation based on the methodology of manga drawing. arXiv preprint arXiv:2004.16644 2020.
[24] Junho Kim, Minjae Kim, Hyconwoo Kang, and Kwanghee Lee. U-gat-it: unsupervised generative attentional networks with adaptive layer-instance normalization for image-to-image translation. arXiv preprint 1rXiv:1907.108.. 2019. T
[25] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE International Conference on Computer Vision, pages 1501 1510, 2017,
[26] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen. and Timo Aila. Training generative adversarial networks with limited data. In Proc. NewrIPS, 2020. 2
[27] Justin NM Pinkney and Doron Adler Resolution dependant gan interpolation for controllable image synthesis between domains. arXiv preprint arXiv:2010.05334. 2020.
[28] Sam Kwong, Jialu Huang, and Jing Liao. Unsupervised image-to-image translation via pre-trained stylegan2 network. IEEE Transactions on Multimedia, 2021.
[29] Weihao Xia, Yulun Zhang, Yujju Yang. Jing-Har Xue, Bolei Zhou, and Ming Hsuan Yang. Gan inversion:
A survey, arXiv preprint arXiv:2101.00 05278, 2021. 2

---

[30] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4432-4441, 2019.
[31] Yujun Shen, Jinjin Gu, Xiaoou Tang. and Bolei Zhou, Interpreting the latent space of gans for semantic face editing. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 9243-9252, 2020. 2
[32] Stanislav Pidhorskyi. Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14104-
14113, 2020. 2
[33] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Danicl Cohen- Or. Encoding in style: stylegan encoder for image-to-image translation. arXiv preprint 20X8v:2008.000
2020. 2
[34] Yuri Viazovetskyi, Vladimir Ivashkin, and Evgeny Kashin. Stylegan2 distillation for feed-forward image
manipulation. arXiv preprint 2rX0v:2003.0388/ 2020. 2
[35] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pages 7354-7363. PMLR, 2019.
[36] Hsin- -ing Lee, Hung- Yu Tseng, Jia-Bin Huang. Maneesh Kumar Singh, and Ming-Hsuan Yang. Diverse image-to-image translation via disentangled representations. In European Conference on Computer Vision, 2018. 3
[37] Yunjey Choi, Minje Choi, Munyoung Kim, Jung- Woo Ha. Sunghun Kim, and Jaegul Choo. Stargan: Unified generative adversarial networks for multi-domain image-to-ine to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018. 3
[38] Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei A. Efros,, and Richard Zhang. Swapping autoencoder for deep image manipulation. In Advances Neural Information Processing Systems, 2020. 3
[39] Leon Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414 2414-242,,222 2016. 4
[40] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, 2016. 4
[41] Hang Zhang and Kristin Dana. Multi-style generative network real-time transfer. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018. 4
[42] Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In Proceedings of the IEEE conference on compuder vision and pattern recognition, pages 4990-4998, 2017. I
[43] Hongmin Xu, Qiang Li, Wenbo Zhang, and Wen Zheng. Styleremix: An interpretable representation for neural image style transfer. arXiv preprint teXiv:1902. 10425, 2019.
[44] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming Hsuan Yang. Universal style transfer via feature transforms. Advances in Neural Information Processing Systems, 2017:386-396, 2017.
[45] Dae Young Park and Kwang Hee Lee. Arbitrary style transfer with style-attentional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5880-5888,
2019. 4
[46] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
[47] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geolfrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020. 5
[48] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020.
4
[49] Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In International Conference Learning Representations, 2018 5 10
12

---

[50] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. arXiv preprint urXiv:1706.0883 085000 2017.
[51] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechuman, and Oliver Wang. The unreasonable effectiveness deep features perceptual metric. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018. 7
[52] Michele Merler, Nalini Ratha, Rogerio S Feris, John R Smith. Diversity in faces. arXiv preprint arXiv:1901. 10436, 2019. 9
[53] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, Cynthia Rudin. Pulse: Self-super photo upsampling via latent space exploration of generative models. In Proceedings of the iecelerf conference on computer vision and pattern recognition, pages 2437-2445, 2020. 1
[54] Joni Salminen, Soon-gyo Jung. Shammur Chowdhury, and Bernard Jansen. Analyzing demographic bias in artificially generated facial pictures. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-8, 2020.
[55] K Hill and White. Designed to deceive: Do these people look real to you. New York Times, 2020. T0
[56] Sheng- Yu Wang. Oliver Wang, Richard Zhang, Andrew Owens, and Alexei Efros. Can-generated images surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8695-870000 5774, 2020. 10
[57] Xin Miao, Huayan Wang. Jun Fu, Jiayi Liu, Shen Wang, and Zhenyu Liao. Fine-grained control of artistic styles in image generation. arXiv preprint arXiv:21 2021. I0
13


Reviews:
[{"review_id": "sv3Zcz8_j9", "reviewer": "Reviewer_Jfp6", "summary": "The paper proposed a method for face generation across a wide range of artistic styles. The key contributions are:\n\n1) a style encoder that maps the gram matrix of the style image to a latent space that can be consumed by the StyleGAN decoder. \n\n2) a blending module that accomplishes style transfer with linear interpolation between the style vector and the face vector, and a leant control vector that decides the “blending factor” for different layers. \n\nThe paper also introduced a new dataset for artistic face images and demonstrated the method’s effectiveness in learning those artistic styles by a single network. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The work is novel because it studies how to use the idea of blending the feature maps of two networks for artistic face generation with “arbitrary styles”. As a downstream application of StyleGAN for faces, the generated stylized images are in general of higher quality in comparison to the early style transfer methods that work on general subjects. In addition, unlike the recent StyleGAN based methods such as toonify networks, which have to be trained for each new style, the proposed method trained a single network that can be applied to a wide variety of styles. The limitation of out-of-distribution style transfer is studied and addressed by one-shot learning that fine-tunes the networks on a single out-of-distribution reference style image for 1000 iterations.\n\nThe training objectives seem intuitive, and there are sufficient technical details for reproduction. The proposed AAHQ dataset will be very interesting to the research community. I hope the authors can open it to the public. \n\nThe results are encouraging, though I have a few comments in the following section. \n\nOverall the paper proposed a novel idea, and the method and results are solid. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In general, the quality of the results is encouraging, especially considering the method is trained to work with many different styles and can be quickly adapted to new styles that haven't been trained with. However, it is also clear that the identity in the style reference image can leak into the results. This is especially obvious in Figure 7 (supplementary). It seems the gender of the source images can often be overwritten by the reference style images (rows 3 and 4).\n\nI wonder why the i=18 results in Figure 4 (supplementary) are different between the Gram matrix approach and the style encoder approach? Did they use different face latent codes, or is there some differences in these two approaches (the values in the leaned alpha maybe?). Clarifying this would help the reader judge the likeness of the face identity in the results.\n\nIt would be helpful if the paper elaborates on the learned 18-dim alpha vector. What are the learned values in this vector? Is there any statistical bias that favors larger/smaller values in the certain layers for style transfer? Also, some comparisons between using the learned alpha and the results of using fixed alpha (such as 0.5 for midpoint interpolation) would be helpful.\n\nLast but not least, it is not clear to me why the method restricts itself to reference style images that contain only portraits. The proposed style encoder first computes the Gram matrix, which has been shown to work well across arbitrary subjects and will drop most of the spatial content information of those subjects. So to me, it seems non-portrait reference style images probably will also work fine. On the other hand, if the objective is to make the reference style images and the face images better aligned in the latent space for blending, then a PSPNet fashion style encoder will make more sense. I would like to hear comments from the authors.\n\nMinor: Line 142-143 says that the generator outputs stylized images when i=0 and natural images when i=18, but the rest of the paper (line 150-153 and eq. 6) seems to suggest the opposite.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "sQzV1zUyOYO", "reviewer": "Reviewer_xcb2", "summary": "The paper introduces a styleGAN based stylization approach that allows the transfer of arbitrary style using a single model. The contribution includes: (1) the large-scale AAHQ dataset, (2) design of the style encoder using self-supervised representation learning, (3) the general idea of separating the latent code into a face code and a style code and using a weighted blending module to control how to blend the codes together to control the amount of style that get transferred into the result. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper contains novel ideas and demonstrates compelling results. Though not the first paper to support the transfer of arbitrary style, compared to the previous feed-forward or optimization-based stylization approaches, the result quality has visibly improved. Therefore, I am in favor of accepting this paper but I do have the following suggestions and questions.\n\nThe results at a first glimpse is great. But with close examination, there are following issues. \nIn the supplementary material, some results indicate that identity is sometimes not well-maintained. For example, in Figure 5 bottom right corner some results show visible gender change. In figure 7, the rightmost column, women are getting beard from the style reference. These reflect that style and identity is not completely disentangled during the training. Some of the identity aspect is carried over from the style reference. Figure 6 shows that, the stylization results sometimes have different gaze of facial expression from the input (for example, the middle two results in the rightmost column have the eyes closed). Figure 7 shows that style is sometimes not preserved very well. For example, the lip color in the results is different from the style reference in the rightmost two columns and the second column from the left. I think this reflects the fundamental limitation of this type of stylization approach which is that it lacks semantic correspondence between the style reference and the result. I think the main paper should discuss these limitations. \n\nIt will be great to see ablation studies to analyze the effect of the style latent discriminator. What is the architecture of the style latent discriminator?  Several of the decision choices in the paper lack motivation and analysis. For example, the exact design of the two MLPs seems quite arbitrary. How do you decide how many layer to use?\n\nIn Line 224, the paper says “StarGANv2 … have some subtle artifacts and are not as natural as our results”. can you be more specific and point out the artifacts in StarGAN2 results? Also since the styles of StarGAN2 and the current paper are different, it is a bit hard to directly compare the results. \n\nI think the WBM paragraph in Section 3.2 can use some higher level description and motivation before diving into the equations. \n\nThe supp mentions 128M images are used to train the style encoder. How did you obtain the images? They are augmented from the AAHQ dataset or is it a new dataset? Why do you need so many images ? Have you tried encoder training with less?  Later on, the supp mentions 1.6M images used to train the generator and discriminators. How did you obtain these images through augmenting FFHQ + AAHQ? \n\nFigure 2 and Figure 3 in the supp mention that the source images are sampled from FFHQ dataset and the reference image from AAHQ. Are these images held off from training or are they part of the training set?\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "qM8BfvJQOm", "reviewer": "Reviewer_hywe", "summary": "This work proposes a method to synthesize faces with arbitrary artistic styles, where the style can be obtained from random samples or a reference style of a real face image. To achieve this, the authors introduce a style encoder to predict the style latent of a reference image. A face latent which controls the structure of the face is randomly sampled. The style latent and face latent are mixed in a fashion of style-mixing to generate the final output, where the degree of stylization is controlled by the number of layers blended with the style latent. The model is trained on FFHQ and a stylized face dataset the authors introduced simultaneously, where the face latent correspond to ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work provides a clear explanation of their method, and the writing is easy to follow. Also, the proposed dataset (AAHQ) will be useful to the community. Overall, the proposed idea of allowing arbitrary style input and style blending is interesting. However, both ideas have been explored by prior work. Also, while most prior work uses real images as sources, the proposed method only shows results on synthesized source images. It will be more convincing to provide real image editing results, so this work can be applied in a more realistic scenario. In my opinion, these issues will need to be addressed to make this work strong. More details are provided in the limitation section.\n\nUpdate: I read the feedback and all of my previous comments have been addressed. Also, after reading other reviewers' comments and authors' feedback, I agree that this work provides a novel way to combine stylization and generation, while also improving upon prior works. I will change my ratings to a 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Prior work by Park et.al. [1] also provides a solution to the proposed problem but is not compared or discussed in this work. Park et.al. show capabilities of arbitrary stylization in multiple domains (e.g. LSUN church, FFHQ, AFHQ, etc.), while the proposed method conducted experiments on only the face domain. Also, Park et.al. allows structure and style separation by training on a single domain, while the proposed method requires collecting a real and stylized domain separately. It appears that Park et.al. provides a stronger yet more elegant solution to this problem.\n\n2. No actual real image editing is shown in this work. Although real stylized images are used as a reference for the style, it is also important to show the capability of taking a real source image and stylize it.\n\n3. In evaluation, all the source images are face images generated by the proposed method without stylization. These source images are then used as the inputs for the baselines. Since the baselines are image-to-image translation methods, the comparison is fairer if actual real images are used as the source images.\n\n4. I'm confused why the completely stylized image ($i=0$) shares the same structure as the other faces ($i>0$). Does any of the face latent $w_f$ got blended into the final output? My guess is that the $\\theta$ term in Eq. 5 allows this preservation, but will be great to verify this. \n\n[1] Park et.al. Swapping Autoencoder for Deep Image Manipulation.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "MelmV-dqPDY", "reviewer": "Reviewer_19Vx", "summary": "The authors propose BlendGAN to perform arbitrary multi modal face stylization. They do so by pretraining a Style Encoder on an artistic dataset, then using weighted blending module (WBM) to blend face and style representations into an image using a StyleGAN-based decoder. They outperform previous methods qualitatively and quantitatively.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "One advantage of BlendGAN is its flexibility in using either randomly sampled style or a reference style to produce multimodal images. This allows arbitrary generation and unlike Toonify [1], does not require a new model for each new style.\n\nI think the main novelty comes from learning two different distributions within a single StyleGAN. The style and face latent codes have its own separate distribution and when fed into the generator, with appropriate I, the generator can produce either real or artistic face. The use of WBM simply interpolates these 2 codes to generate a hybrid face.\n\nIn L143, it says i=0 corresponds to stylized face but in L151 natural faces are paired with i=0? Should this not be i=18?\n\nAlthough Toonify is not multimodal, this is the closest baseline and I think it should be compared to. Otherwise the comparisons and quantitative results are sufficient and significant.\n\nQualitatively, BlendGAN captures textures well, but not global structures. For example, in Figure 1, all the face shapes are the same. This is a limitation due to the usage of layer swapping to main semantics. In Figure 5 if layer swapping is not used, it is closer to the style of the reference image. However, the semantics (such as expressions) are slightly off. Layer swapping is used to capture the semantics, at the cost of style similarity. The trade off seen in BlendGAN is the same as in Toonify. In this regard, there are no improvements over it. \n\nOverall, idea is not novel and is a combination of several previous work combining AdaIN [2], Toonify [1], and SimCLR [3]. However, there are some novel concepts in the implementation of the generator. The results are overall better than previous approaches, but BlendGAN still suffers similar problems faced by those approaches. \n\n[1] Pinkney, Justin NM, and Doron Adler. \"Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains.\" arXiv preprint arXiv:2010.05334 (2020).\n\n[2] Huang, Xun, and Serge Belongie. \"Arbitrary style transfer in real-time with adaptive instance normalization.\" In Proceedings of the IEEE International Conference on Computer Vision, pp. 1501-1510. 2017.\n\n[3] Chen, Ting, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. \"A simple framework for contrastive learning of visual representations.\" In International conference on machine learning, pp. 1597-1607. PMLR, 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: UPDATE: The revision of this paper has been reviewed and the paper has been accepted.  However, the following additional minor changes are required for the camera-ready:
- Acknowledge in the main text the issue that the new model has the same challenges as PULSE (i.e., input images with darker skin are now stylized as lighter skinned faces in the output).
- Update the "Broader Impacts" section to mention the fact that face data is biometric data and thus needs to be sourced and distributed more carefully, with attention to potential privacy, consent and copyright issues. 

----

It came to the attention of the program chairs and ethics review chairs very late in the review process that this paper deals with face generation, a sensitive application, but was not flagged for ethics review.  An emergency ethics review was obtained and the program chairs and ethics review chairs then discussed the paper in detail.

Based on this discussion, we have decided to conditionally accept the paper, given the ethical concerns related to face generation more broadly. In order for this paper to be fully accepted, authors need to address the following concerns:

- Provide examples beyond light-skinned faces and other potentially biased results or demonstrate some mitigation and reflection on any biased outcomes of the model. Include an analysis or disaggregated evaluation acknowledging any limitations in how the model handles faces of differing demographics. 
- Communicate any face data distribution restrictions or limitations to model distribution in light of privacy or malicious use concerns.
- Acknowledge any potential harmful applications or depictions that could arise from the use of this technology in an expanded “broader impacts and ethical considerations” section. 

We appreciate the cooperation of the authors in this process, and we hope that these adjustments and further reflection will improve the overall quality of the work. We hope authors can commit to these improvements as a requirement for inclusion at this conference. 

The original meta-review from the AC follows.

----

This paper proposes a model for synthesizing faces with diverse styles. The model can generate arbitrary stylized faces and their natural photo versions at the same time. In general, many reviewers find the idea interesting and the results encouraging. There are concerns regarding the technical novelty and comparisons against Swapping Autoencoder and Toonify. The rebuttal addressed most of the concerns and clarified the difference between the proposed work vs. Toonify and Swapping Autoencoder. The AC agreed with the reviewers’ consensus and recommended accepting the paper. 

Decision: Accept (Poster)