StyleAlign: Analysis and Applications of Aligned StyleGAN Models

1 INTRODUCTION . Transfer Learning ( TL ) refers to the process in which a parent model , pretrained for some source domain/task , is used to improve the performance of a child model on a different target domain and/or task ( Pan & Yang , 2009 ) . The assumption underlying TL is that some knowledge learnt by the parent model is transferable to the new domain or task ( Pan & Yang , 2009 ; Torrey & Shavlik , 2010 ; Yosinski et al. , 2014 ) . The most common TL approach is fine-tuning , where the parent ’ s parameters are used to initialize those of the child . Next , the child ’ s parameters , or sometimes just a subset of them , are trained on the target domain/task . Once TL is completed , the child posseses some of the parent ’ s knowledge , despite the fact that the model parameters may have changed . Existing TL literature typically examines the performance of the child model , e.g. , in terms of classification accuracy ( He et al. , 2019 ) , or FID score ( Karras et al. , 2020a ) , without paying much attention to the relationship between parent and child models , induced by the transfer process . Typically , the child model is simply applied to the task it was trained on , while the parent model is no longer used , having fulfilled its purpose . In this work , we provide a complementary perspective , which focuses on analyzing and leveraging the shared knowledge between the two models . Specifically , we consider the case where the TL is performed by fine-tuning the same architecture . We refer to models obtained in this manner as aligned models . Several recent works ( Pinkney & Adler , 2020 ; bryandlee , 2020 ; Kwong et al. , 2021 ; Song et al. , 2021 ; Gal et al. , 2021 ) use aligned models in a novel manner . In all cases , an unconditional GAN , specifically StyleGAN2 ( Karras et al. , 2020b ) is fine-tuned from domain A to domain B . However , instead of applying the child model as an unconditional generator , it is used in conjunction with the parent model to form an image translation pipeline . First , an image from domain A is embedded into the latent space of the parent StyleGAN2 model . The resulting latent code is then fed either into the child model ( bryandlee , 2020 ; Song et al. , 2021 ; Gal et al. , 2021 ) , or into a hybrid model created by layer swapping , i.e. , by combining layers from the parent and the child ( Pinkney & Adler , 2020 ; Kwong et al. , 2021 ) . These methods have achieved great results in image-to-image translation between several domains . Most notably , translating real human face images to a variety of styles such as cartoons and oil paintings ( Pinkney & Adler , 2020 ; Kwong et al. , 2021 ; Song et al. , 2021 ) , but also translating humans to dogs and cats to wildlife ( bryandlee , 2020 ; Gal et al. , 2021 ) . However , they focus on a specific application ( image-to-image translation ) and do not explore or leverage aligned models further . As a result , many questions arise but remain unanswered . For example , which parts of the network change in the TL process , and which knowledge is inherited by the child from its parent ? To which degree do the answers to these questions depend on the similarity between the parent and child domains ? And , is knowledge not used by the child model completely lost or could it be recovered ? Finally , what further applications , besides image-to-image translation , can be solved using aligned models ? In this work , we delve deeper into model alignment . In light of previous works , we specifically focus on the state-of-the-art unconditional GAN architecture , StyleGAN2 ( Karras et al. , 2020b ) . The process of obtaining aligned models is incredibly simple : we start with a parent StyleGAN2 model trained on domain A and fine-tune it fully for domain B , yielding an aligned child model . We divide the investigation of model alignment into two parts . First , in Section 3 , we perform the first empirical analysis of the phenomenon , answering the questions posed above , as well as others . This analysis provides some surprising and novel insights that shed light on aligned StyleGAN2 models . For example , we discover that when fine tuning to a similar target domain , the parts of the model that change the most are the feature convolution weights in the synthesis network . In contrast , the changes in the mapping network and affine layers are negligible ( see Figure 1 ) . This crucially implies that the learned latent spacesW and S are barely affected by the fine-tuning . This explains our next discovery – that semantically meaningful directions in the latent space of the parent model , retain the same ( or similar ) semantics in the child model ( see Figure 2 ) . As the data domains become more distant , the mapping network and affine layers become more affected , which results in a weaker degree of semantic alignment . However , even in extreme cases , such as human faces and churches , some semantic alignment occurs . Another surprising discovery is that the semantic latent controls that seemingly disappear after transfer to the child model , are in fact merely hidden , rather than forgotten , and reappear if the child is retrained back to the parent ’ s domain . Second , in Section 4 , we use aligned models to solve several popular Computer Vision and Computer Graphics tasks . We start with the aforementioned image-to-image translation task ( Section 4.1 ) , examine a number of alternatives , and show that aligned models obtain state-of-the-art results for a variety of scenarios . This is especially impactful , as using aligned models for image translation is incredibly simple compared to dedicated methods for the same task , each devising its custom architecture and losses . Next , we explore additional tasks , for which aligned models have not been used before . In Section 4.2 we describe a simple method for fully automatic image morphing between fairly dissimilar domains , such as human to dog faces , which previously necessitated sophisticated methods ( Aberman et al. , 2018 ; Fish et al. , 2020 ) . Examples of smooth morphs are included in the accompanying video . In Section 4.3 , we use aligned models to solve zero-shot classification and regression tasks in domain B , where the supervision is available strictly in domain A . Conceptually , our method reduces a task in a zero-shot or few-shot setting to the same task in a different data domain where supervision is plentiful . In summary , while several previous works took advantage of aligned models implicitly , ours is the first work to conduct a thorough empirical study of this phenomenon . Our study reveals various interesting properties that we then use to further leverage aligned models for a variety of applications , almost effortlessly achieving state-of-the-art performance . 2 RELATED WORK . Latent Space of GANs : With the rapid evolution of GANs ( Goodfellow et al. , 2014 ) in recent years , understanding and controlling their latent representation has attracted considerable attention . Specifically , it has been shown that the intermediate latent space of StyleGAN ( Karras et al. , 2019 ; 2020b ; a ) possesses appealing properties , such as being semantically rich , disentangled and smooth . Many recent works have proposed methods to interpret the semantics encoded in that space and its extensions and apply them to image editing ( Jahanian et al. , 2019 ; Shen et al. , 2020a ; Härkönen et al. , 2020 ; Tewari et al. , 2020 ; Abdal et al. , 2020 ; Wu et al. , 2020 ; Patashnik et al. , 2021 ) . In order to benefit from these properties in real images , it is necessary to obtain the latent code from which a pretrained GAN can reconstruct the original input image . This task , commonly referred to as GAN Inversion , has been tackled by numerous recent works , either by using : ( i ) optimization ( Abdal et al. , 2019 ; Karras et al. , 2020b ) ; or ( ii ) an encoder ( Guan et al. , 2020 ; Pidhorskyi et al. , 2020 ; Richardson et al. , 2021 ; Tov et al. , 2021 ) ; or ( iii ) a hybrid approach using both ( Zhu et al. , 2016 ; Baylies , 2019 ; Zhu et al. , 2020a ) . See Xia et al . ( 2021 ) for a more thorough review . Image-to-Image translation : The seminal pix2pix work by Isola et al . ( 2017 ) , first introduced the use of conditional GANs to solve various supervised image-to-image translation tasks . Since then , their work has been extended to allow image synthesis in various different settings : high-resolution ( Wang et al. , 2018a ) , semantic image ( Park et al. , 2019 ; Zhu et al. , 2020b ; Liu et al. , 2019b ) , multidomain ( Choi et al. , 2018 ) , multimodal ( Zhu et al. , 2017b ) , and using a pre-trained generator ( Nitzan et al. , 2020 ; Richardson et al. , 2021 ; Luo et al. , 2020 ) . Another scenario that has received significant attention is unsupervised image-to-image translation ( Liu et al. , 2017 ; Zhu et al. , 2017a ; Kim et al. , 2017 ; Choi et al. , 2020 ; Lee et al. , 2020b ) , where no paired data samples are given . Regardless of the setting , all of the aforementioned works train an neural network , designed explicitly for the translation task . Recently , several works ( Pinkney & Adler , 2020 ; bryandlee , 2020 ; Kwong et al. , 2021 ; Song et al. , 2021 ; Gal et al. , 2021 ) have taken a different approach towards image-to-image translation . They observe that significant correspondence between generated images in different domains exists when an unconditional generator , such as StyleGAN2 ( Karras et al. , 2020b ) , is fine-tuned between the two domains . Accordingly , these works take a two-step approach towards image-to-image translation . First , they invert a given image into the latent space of StyleGAN in domain A and then forward the output latent code through a StyleGAN model for domain B . The latter model is obtained either by directly fine-tuning from the former model ( bryandlee , 2020 ; Song et al. , 2021 ; Gal et al. , 2021 ) or by layer swapping ( Pinkney & Adler , 2020 ; Kwong et al. , 2021 ) , i.e. , forming a model whose layers are partially those of the fine-tuned model and partially those of the model for domain A . In this work , we delve deeper into this phenomenon , which we refer to as model alignment , and go beyond the image-to-image translation task . For example , we demonstrate that the alignment property goes beyond high-level properties , such as pose , and that multiple fine-grained latent semantics are also aligned . We leverage this property for tasks such as morphing and zero-shot classification . Fine-tuning and Catastrophic Forgetting : Fine-tuning was proven advantageous across fields , settings and tasks and therefore became a standard practice in the deep learning literature . Prominent advantages of fine-tuning are enabling few-shot tasks such as classification ( Chen et al. , 2019 ) and unconditional generation ( Wang et al. , 2018b ; Mo et al. , 2020 ; Wang et al. , 2020 ; Li et al. , 2020 ; Ojha et al. , 2021 ) , improved performance in a wide variety of tasks ( Devlin et al. , 2018 ; Radford et al. , 2018 ; He et al. , 2020 ) and faster training convergence ( Wang et al. , 2018b ; He et al. , 2019 ) . While fine-tuning can be an effective technique for solving a new task , it has been well known for over 30 years ( McCloskey & Cohen , 1989 ) that in the process the model “ forgets ” how to solve the original task , a phenomenon referred to as Catastrophic Forgetting ( CF ) . For example , once a GAN for a certain domain A , is fine-tuned to another domain B , the resulting model can only generate images in domain B ( Seff et al. , 2017 ; Zhai et al. , 2019 ) . In settings such as continual learning and multi-task learning , CF is undesirable . In recent years , there has been progress in mitigating it using dedicated methods ( Kirkpatrick et al. , 2017 ; Kemker et al. , 2018 ) . CF has been also studied in the context of GANs ( Liang et al. , 2018 ; Li et al. , 2020 ; Thanh-Tung & Tran , 2020 ) . Aforementioned previous works devised methods to obtain a better child model using fine-tuning . From a fine-tuning perspective , this means the model would perform better on the new task . From a CF perspective , this means the model ’ s performance on the previous task should not be impaired . We differ from these works significantly , as we make no deliberate effort to affect what happens during training of the child model . Instead , we investigate the relationship between the parent and child models after naı̈ve fine-tuning , and then use it to solve a variety of applications .

Summary: The paper undertakes extensive experimentation in the adaptability of latent space modifications from a base model, and a fine-tuned model for a secondary dataset. They demonstrate which areas of the models change most, and which information remains trained in the parameters, despite re-training. They go on to use their findings in downstream experiments, reaching state-of-the-art quality.