StyleAlign: Analysis and Applications of Aligned StyleGAN Models

1 INTRODUCTION . Transfer Learning ( TL ) refers to the process in which a parent model , pretrained for some source domain/task , is used to improve the performance of a child model on a different target domain and/or task ( Pan & Yang , 2009 ) . The assumption underlying TL is that some knowledge learnt by the parent model is transferable to the new domain or task ( Pan & Yang , 2009 ; Torrey & Shavlik , 2010 ; Yosinski et al. , 2014 ) . The most common TL approach is fine-tuning , where the parent ’ s parameters are used to initialize those of the child . Next , the child ’ s parameters , or sometimes just a subset of them , are trained on the target domain/task . Once TL is completed , the child posseses some of the parent ’ s knowledge , despite the fact that the model parameters may have changed . Existing TL literature typically examines the performance of the child model , e.g. , in terms of classification accuracy ( He et al. , 2019 ) , or FID score ( Karras et al. , 2020a ) , without paying much attention to the relationship between parent and child models , induced by the transfer process . Typically , the child model is simply applied to the task it was trained on , while the parent model is no longer used , having fulfilled its purpose . In this work , we provide a complementary perspective , which focuses on analyzing and leveraging the shared knowledge between the two models . Specifically , we consider the case where the TL is performed by fine-tuning the same architecture . We refer to models obtained in this manner as aligned models . Several recent works ( Pinkney & Adler , 2020 ; bryandlee , 2020 ; Kwong et al. , 2021 ; Song et al. , 2021 ; Gal et al. , 2021 ) use aligned models in a novel manner . In all cases , an unconditional GAN , specifically StyleGAN2 ( Karras et al. , 2020b ) is fine-tuned from domain A to domain B . However , instead of applying the child model as an unconditional generator , it is used in conjunction with the parent model to form an image translation pipeline . First , an image from domain A is embedded into the latent space of the parent StyleGAN2 model . The resulting latent code is then fed either into the child model ( bryandlee , 2020 ; Song et al. , 2021 ; Gal et al. , 2021 ) , or into a hybrid model created by layer swapping , i.e. , by combining layers from the parent and the child ( Pinkney & Adler , 2020 ; Kwong et al. , 2021 ) . These methods have achieved great results in image-to-image translation between several domains . Most notably , translating real human face images to a variety of styles such as cartoons and oil paintings ( Pinkney & Adler , 2020 ; Kwong et al. , 2021 ; Song et al. , 2021 ) , but also translating humans to dogs and cats to wildlife ( bryandlee , 2020 ; Gal et al. , 2021 ) . However , they focus on a specific application ( image-to-image translation ) and do not explore or leverage aligned models further . As a result , many questions arise but remain unanswered . For example , which parts of the network change in the TL process , and which knowledge is inherited by the child from its parent ? To which degree do the answers to these questions depend on the similarity between the parent and child domains ? And , is knowledge not used by the child model completely lost or could it be recovered ? Finally , what further applications , besides image-to-image translation , can be solved using aligned models ? In this work , we delve deeper into model alignment . In light of previous works , we specifically focus on the state-of-the-art unconditional GAN architecture , StyleGAN2 ( Karras et al. , 2020b ) . The process of obtaining aligned models is incredibly simple : we start with a parent StyleGAN2 model trained on domain A and fine-tune it fully for domain B , yielding an aligned child model . We divide the investigation of model alignment into two parts . First , in Section 3 , we perform the first empirical analysis of the phenomenon , answering the questions posed above , as well as others . This analysis provides some surprising and novel insights that shed light on aligned StyleGAN2 models . For example , we discover that when fine tuning to a similar target domain , the parts of the model that change the most are the feature convolution weights in the synthesis network . In contrast , the changes in the mapping network and affine layers are negligible ( see Figure 1 ) . This crucially implies that the learned latent spacesW and S are barely affected by the fine-tuning . This explains our next discovery – that semantically meaningful directions in the latent space of the parent model , retain the same ( or similar ) semantics in the child model ( see Figure 2 ) . As the data domains become more distant , the mapping network and affine layers become more affected , which results in a weaker degree of semantic alignment . However , even in extreme cases , such as human faces and churches , some semantic alignment occurs . Another surprising discovery is that the semantic latent controls that seemingly disappear after transfer to the child model , are in fact merely hidden , rather than forgotten , and reappear if the child is retrained back to the parent ’ s domain . Second , in Section 4 , we use aligned models to solve several popular Computer Vision and Computer Graphics tasks . We start with the aforementioned image-to-image translation task ( Section 4.1 ) , examine a number of alternatives , and show that aligned models obtain state-of-the-art results for a variety of scenarios . This is especially impactful , as using aligned models for image translation is incredibly simple compared to dedicated methods for the same task , each devising its custom architecture and losses . Next , we explore additional tasks , for which aligned models have not been used before . In Section 4.2 we describe a simple method for fully automatic image morphing between fairly dissimilar domains , such as human to dog faces , which previously necessitated sophisticated methods ( Aberman et al. , 2018 ; Fish et al. , 2020 ) . Examples of smooth morphs are included in the accompanying video . In Section 4.3 , we use aligned models to solve zero-shot classification and regression tasks in domain B , where the supervision is available strictly in domain A . Conceptually , our method reduces a task in a zero-shot or few-shot setting to the same task in a different data domain where supervision is plentiful . In summary , while several previous works took advantage of aligned models implicitly , ours is the first work to conduct a thorough empirical study of this phenomenon . Our study reveals various interesting properties that we then use to further leverage aligned models for a variety of applications , almost effortlessly achieving state-of-the-art performance . 2 RELATED WORK . Latent Space of GANs : With the rapid evolution of GANs ( Goodfellow et al. , 2014 ) in recent years , understanding and controlling their latent representation has attracted considerable attention . Specifically , it has been shown that the intermediate latent space of StyleGAN ( Karras et al. , 2019 ; 2020b ; a ) possesses appealing properties , such as being semantically rich , disentangled and smooth . Many recent works have proposed methods to interpret the semantics encoded in that space and its extensions and apply them to image editing ( Jahanian et al. , 2019 ; Shen et al. , 2020a ; Härkönen et al. , 2020 ; Tewari et al. , 2020 ; Abdal et al. , 2020 ; Wu et al. , 2020 ; Patashnik et al. , 2021 ) . In order to benefit from these properties in real images , it is necessary to obtain the latent code from which a pretrained GAN can reconstruct the original input image . This task , commonly referred to as GAN Inversion , has been tackled by numerous recent works , either by using : ( i ) optimization ( Abdal et al. , 2019 ; Karras et al. , 2020b ) ; or ( ii ) an encoder ( Guan et al. , 2020 ; Pidhorskyi et al. , 2020 ; Richardson et al. , 2021 ; Tov et al. , 2021 ) ; or ( iii ) a hybrid approach using both ( Zhu et al. , 2016 ; Baylies , 2019 ; Zhu et al. , 2020a ) . See Xia et al . ( 2021 ) for a more thorough review . Image-to-Image translation : The seminal pix2pix work by Isola et al . ( 2017 ) , first introduced the use of conditional GANs to solve various supervised image-to-image translation tasks . Since then , their work has been extended to allow image synthesis in various different settings : high-resolution ( Wang et al. , 2018a ) , semantic image ( Park et al. , 2019 ; Zhu et al. , 2020b ; Liu et al. , 2019b ) , multidomain ( Choi et al. , 2018 ) , multimodal ( Zhu et al. , 2017b ) , and using a pre-trained generator ( Nitzan et al. , 2020 ; Richardson et al. , 2021 ; Luo et al. , 2020 ) . Another scenario that has received significant attention is unsupervised image-to-image translation ( Liu et al. , 2017 ; Zhu et al. , 2017a ; Kim et al. , 2017 ; Choi et al. , 2020 ; Lee et al. , 2020b ) , where no paired data samples are given . Regardless of the setting , all of the aforementioned works train an neural network , designed explicitly for the translation task . Recently , several works ( Pinkney & Adler , 2020 ; bryandlee , 2020 ; Kwong et al. , 2021 ; Song et al. , 2021 ; Gal et al. , 2021 ) have taken a different approach towards image-to-image translation . They observe that significant correspondence between generated images in different domains exists when an unconditional generator , such as StyleGAN2 ( Karras et al. , 2020b ) , is fine-tuned between the two domains . Accordingly , these works take a two-step approach towards image-to-image translation . First , they invert a given image into the latent space of StyleGAN in domain A and then forward the output latent code through a StyleGAN model for domain B . The latter model is obtained either by directly fine-tuning from the former model ( bryandlee , 2020 ; Song et al. , 2021 ; Gal et al. , 2021 ) or by layer swapping ( Pinkney & Adler , 2020 ; Kwong et al. , 2021 ) , i.e. , forming a model whose layers are partially those of the fine-tuned model and partially those of the model for domain A . In this work , we delve deeper into this phenomenon , which we refer to as model alignment , and go beyond the image-to-image translation task . For example , we demonstrate that the alignment property goes beyond high-level properties , such as pose , and that multiple fine-grained latent semantics are also aligned . We leverage this property for tasks such as morphing and zero-shot classification . Fine-tuning and Catastrophic Forgetting : Fine-tuning was proven advantageous across fields , settings and tasks and therefore became a standard practice in the deep learning literature . Prominent advantages of fine-tuning are enabling few-shot tasks such as classification ( Chen et al. , 2019 ) and unconditional generation ( Wang et al. , 2018b ; Mo et al. , 2020 ; Wang et al. , 2020 ; Li et al. , 2020 ; Ojha et al. , 2021 ) , improved performance in a wide variety of tasks ( Devlin et al. , 2018 ; Radford et al. , 2018 ; He et al. , 2020 ) and faster training convergence ( Wang et al. , 2018b ; He et al. , 2019 ) . While fine-tuning can be an effective technique for solving a new task , it has been well known for over 30 years ( McCloskey & Cohen , 1989 ) that in the process the model “ forgets ” how to solve the original task , a phenomenon referred to as Catastrophic Forgetting ( CF ) . For example , once a GAN for a certain domain A , is fine-tuned to another domain B , the resulting model can only generate images in domain B ( Seff et al. , 2017 ; Zhai et al. , 2019 ) . In settings such as continual learning and multi-task learning , CF is undesirable . In recent years , there has been progress in mitigating it using dedicated methods ( Kirkpatrick et al. , 2017 ; Kemker et al. , 2018 ) . CF has been also studied in the context of GANs ( Liang et al. , 2018 ; Li et al. , 2020 ; Thanh-Tung & Tran , 2020 ) . Aforementioned previous works devised methods to obtain a better child model using fine-tuning . From a fine-tuning perspective , this means the model would perform better on the new task . From a CF perspective , this means the model ’ s performance on the previous task should not be impaired . We differ from these works significantly , as we make no deliberate effort to affect what happens during training of the child model . Instead , we investigate the relationship between the parent and child models after naı̈ve fine-tuning , and then use it to solve a variety of applications .

Summary: This paper provides an in-depth study of the properties and applications of the semantic alignment between the original parent StyleGAN model and its finetuned child model on another dataset. Specifically, the paper empirically demonstrates the semantical alignment of the two models. Then, based on the properties, the paper solves serval tasks like image translation, image morphing, zero-shot image editing and attribute classification.