BlendGAN: Implicitly GAN Blending for Arbitrary Stylized Face Generation

1 Introduction . Generative adversarial networks ( GANs ) [ 1 ] have achieved impressive performance in various tasks such as image generation [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] , image super-resolution [ 10 , 11 ] , and image translation [ 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ] . In recent years , GAN has also been widely used for face stylization such as portrait drawing [ 20 ] , caricature [ 21 , 22 ] , manga [ 23 ] , and anime [ 24 ] . With the deepening of GAN research , the community has further paid attention to the quality of generated images and the disentanglement ability of latent space . To achieve high-quality generation and latent disentanglement , StyleGAN [ 8 ] introduces the adaptative instance normalization layer ( AdaIN ) [ 25 ] and proposes a new generator architecture , which ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . achieves pretty good performance in generating face images . StyleGAN2 [ 9 ] explores the causes of droplet-like artifacts and further improves the quality of generated images by redesigning generator normalization . StyleGAN2-ada [ 26 ] proposes an adaptive discriminator augmentation mechanism which reduces the required number of images to several thousands . In the experiment of StyleGAN2ada , the model is trained on MetFaces dataset [ 26 ] to generate artistic faces , however , the style of image is uncontrollable , and the corresponding original face can not be obtained . Recently , a layer-swapping mechanism [ 27 ] has been proposed to generate high-quality natural and stylized face image pairs , and equipped with an additional encoder for unsupervised image translation [ 28 ] . In particular , these methods first finetune StyleGAN from a pretrained model with target-style faces and then swap some convolutional layers with the original model . Given randomly sampled latent codes , the original model generates natural face images , while the layer-swapped model outputs stylized face images correspondingly . Though effective at large , these methods have to train models in a case-by-case flavor , thus a single model can only generate images with a specific style . Besides , these methods still require hundreds of target-style images for finetuning , and one has to carefully select the number of training iterations , as well as the swapped layers . In this paper , we present BlendGAN for arbitrary stylized face generation.2 BlendGAN resorts to a flexible blending strategy and a generic artistic dataset to fit arbitrary styles without relying on styleconsistent training images for each style . In particular , we analyze a stylized face image as composed of two latent parts : a face code ( controlling the face attributes ) and a style code ( controlling the artistic appearance ) . Firstly , a self-supervised style encoder is trained via an instance discrimination objective to extract the style representation from artistic face images . Secondly , a weighted blending module ( WBM ) is proposed to blend the face and style latent codes into a final code which is then fed into a generator . By controlling the indicator in WBM , we are able to decide which parts of the face and style latent codes to be blended thus controlling the stylization effect . By combining the 2Our framework can also cooperate with GAN inversion [ 29 , 30 , 31 , 32 , 33 ] or StyleGAN distillation [ 34 ] methods to enable end-to-end style transfer or image translation . We will show the results in the appendix . style encoder and the generator with WBM , our framework can generate natural and stylized face image pairs with either a randomly sampled style or a reference style ( see Figure 1 ) . As for the generic artistic data , we present a novel large-scale dataset of high-quality artistic face images , Artstation-Artistic-face-HQ ( AAHQ ) , which covers a wide variety of painting styles , color tones , and face attributes . Experiments show that compared to state-of-the-art methods , our framework can generate stylized face images with higher visual quality and style diversity for both latent-guided and reference-guided synthesis . 2 Related Work . Generative Adversarial Networks The recent years have witnessed rapid advances of generative adversarial networks ( GANs ) [ 1 ] for image generation . The key to the success of GANs is the adversarial training between the generator and discriminator . Various improvements to GANs have been proposed to stabilize GAN training and improve the image quality . WGAN [ 3 ] uses the Wasserstein distance as a new cost function to train the network . WGAN-GP [ 4 ] and SNGAN [ 5 ] improve the stability of GAN by introducing gradient penalty and spectral normalization separately to make the training satisfy the 1-Lipschitz constraint . SAGAN [ 35 ] enlarges the receptive field using the self-attention mechanism . BigGAN [ 6 ] shows the significant improvement of image quality when training GANs at large scales . StyleGAN [ 8 , 9 ] redesigns the generator architecture with AdaIN layers , making it better disentangle the latent factors of variation . In this work , we employ GANs to generate high-quality stylized face images . Image-to-Image Translation With the good performance of GANs , many GAN-based image-toimage translation techniques have been explored in recent years [ 12 , 13 , 14 , 16 , 17 , 36 , 18 , 37 , 19 , 38 ] . For image translation in two domains , Pix2pix [ 12 ] proposes the first unified image-to-image framework to translate images to another domain . CycleGAN [ 13 ] proposes a cycle-consistency loss , making the network train with unpair data . UNIT [ 14 ] maps images in source and target domains to a shared latent space to perform unsupervised image translation . UGATIT [ 24 ] proposes an adaptive layer-instance normalization ( AdaLIN ) layer to control shapes during translation . For image translation in multi-domains , MUNIT [ 16 ] extends UNIT to multi-modal contexts by decomposing images into content and style spaces . FUNIT [ 17 ] leverages a few-shot strategy to translate images using a few reference images from a target domain . DRIT++ [ 18 ] also proposes a multi-modal and multi-domain model using a discrete domain encoding . Training with both latent codes and reference images , StarGANv2 [ 19 ] could provide both latent-guided and reference-guided synthesis . Park et.al . [ 38 ] utilize the autoencoder structure to swap the style and content of two given images , and it can also be regarded as a reference-guided synthesis method . Although these methods could translate images to multiple target domains , the diversity of target domains is still limited and these methods require hundreds of training images for each domain . Besides , the above methods could only operate at a resolution of at most 256x256 when applied to human faces . Neural Style Transfer The arbitrary stylized face generation task could also be regarded as a kind of the neural style transfer [ 39 , 40 , 41 , 25 ] . Gatys et al . [ 39 ] , for the first time , proposes a neural style transfer method based on optimization strategy . Johnson et al . [ 40 ] trains a feed-forward network to avoid the time consumption of optimization . Luan et al . [ 42 ] proposes a photorealism regularization term to preserve the structure of source images . The above methods could only transfer source images to a single style with a model , many works extend the style flexibility and achieved multi-style or arbitrary style transfer such as MSG-Net [ 41 ] , StyleRemix [ 43 ] , AdaIN [ 25 ] , WCT [ 44 ] and SANet [ 45 ] . Although arbitrary style transfer methods could transfer images to arbitrary styles , they only transfer the reference style to source images globally without considering local semantic styles , which leads to global texture artifacts in the outputs . 3 Method . Our goal is to train a generator x̂f , x̂s = G ( zf , zs , i ) , ( 1 ) that can generate a image pair ( natural face image x̂f and its artistic counterpart x̂s ) with a pair of given latent code ( zf , zs ) and a blending indicator i . The face latent code zf controls the face identity , and the style latent code zs controls the style of x̂s . The blending indicator i decides which parts of zf and zs are blended to generate plausible images . Figure 2 illustrates an overview of our framework . 3.1 Self-supervised Style Encoder . The style encoder Estyle is independently trained to extract the style representation from an artistic face image while discarding the content information such as face identity . Many style transfer methods [ 39 , 40 , 41 ] obtain the style representation by calculating the Gram matrices of feature maps extracted by a pretrained VGG [ 46 ] network . However , if simply concatenating the Gram matrices of all the selected feature maps , the resulting style latent code ( ‘ style embedding ’ ) would have a very high dimension . For example , if we choose layers relu1_2 , relu2_2 , relu3_4 , relu4_4 and relu5_4 , the style embedding would be a 610,304-dimensional vector ( 642+1282+2562+5122 ∗2 ) . Higher latent code dimension leads to sparser distribution , which makes the generator harder to train and perform worse in style disentanglement . Hence , following the self-supervised method SimCLR [ 47 , 48 ] , we train the style encoder to reduce the dimension of style embeddings . Figure 3 illustrates the architecture of our style encoder . The style image is augmented and then sent to a pretrained VGG [ 46 ] network . We calculate the Gram matrices of selected feature maps , flatten and concatenate them into a long vector ( 610,304-D in our case ) . Then the vector is passed into a 4-layer MLP module ( predictor ) to reduce the dimension and generate the style embedding zs of the input style image . Following zs , another 2-layer MLP performs as a projection head for contrastive learning . The augmentations in the original SimCLR include affine transformation and color transformation , following the assumption that affine and color transformations do not change the classes of objects . However , for style encoding , the image style is strongly related to color , so we only use affine transformations in the augmentation step . During training the style encoder , only parameters of two MLP modules are optimized while VGG-19 is fixed to perform as a feature extractor . The network is trained using NT-Xent loss [ 47 ] : ` NT−Xenti , j = − log exp ( sim ( zi , zj ) /τ ) ∑2N k=1 1 [ k 6=i ] exp ( sim ( zi , zk ) /τ ) , ( 2 ) where ( i , j ) is a pair of positive examples ( augmented from the same image ) , z is the projection result , sim ( · , · ) is cosine similarity between two vectors , and τ is a temperature scalar . 3.2 Generator . The generator G is similar to StyleGAN2 [ 9 ] , but its input is a combination of two latent codes - face latent code zf and style latent code zs . In particular , zf controls the face identity of the generated images , and is randomly sampled from the standard Gaussian distribution N ( 0 , 1 ) . zs controls the style of the generated stylized image , it could be either randomly sampled from N ( 0 , 1 ) or encoded by the style encoder Estyle from a reference artistic image . Two MLPs transform zf and zs into their W spaces as wf and ws separately . For 1024× 1024 output , the dimension of wf and ws are all 18× 512 . Controlled by the blending indicator i , the weighted blending module ( WBM ) combines ws and wf into w , which is then fed into the generator as the final code . WBM Different resolution layers in the StyleGAN model are responsible for different features in the generated image [ 8 ] ( e.g . low resolution layers control face shape and hair style , high resolution layers control color and lighting ) . Therefore , the blending weights of ws and wf should not be consistent at different layers . We propose WBM to blend the two codes , which can be described as : w = ws α̂+wf ( 1− α̂ ) , ( 3 ) α̂ = α m ( i ) , ( 4 ) m ( i ; θ ) = [ m0 , m1 , m2 , . . . , mn ] , mj = { 0 j < i θ j = i 1 j > i , θ ∈ ( 0 , 1 ) , ( 5 ) where denotes the broadcasting element-wise product , ws , wf ∈ R18×512 are the input codes , α ∈ R18 is a learnable vector to balance the blending weights for ws and wf in different layers . m ( i ; θ ) has the same dimension as α , it controls which layer should be blended . If i = 0 , w = ws α + wf ( 1 − α ) , the generator outputs stylized face image x̂s ; if i = 18 , w = wf , the generator outputs natural face image x̂f . When 0 < i < 18 , some low resolution layers are not influenced by style codes , which ensures the x̂s to keep the same face identity as x̂f ( see Figure 2 ) . θ is used for finer adjustment .

Summary: This work proposes a method to synthesize faces with arbitrary artistic styles, where the style can be obtained from random samples or a reference style of a real face image. To achieve this, the authors introduce a style encoder to predict the style latent of a reference image. A face latent which controls the structure of the face is randomly sampled. The style latent and face latent are mixed in a fashion of style-mixing to generate the final output, where the degree of stylization is controlled by the number of layers blended with the style latent. The model is trained on FFHQ and a stylized face dataset the authors introduced simultaneously, where the face latent correspond to