StyleAlign: Analysis and Applications of Aligned StyleGAN Models
Authors: Zongze Wu
Yotam Nitzan
Eli Shechtman
Dani Lischinski
Venue: ICLR 2022 Oral (2022)
Keywords: StyleGAN
transfer learning
fine tuning
model alignment
image-to-image translation
image morphing

Abstract: In this paper, we perform an in-depth study of the properties and applications of aligned generative models.
We refer to two models as aligned if they share the same architecture, and one of them (the child) is obtained from the other (the parent) via fine-tuning to another domain, a common practice in transfer learning. Several works already utilize some basic properties of aligned StyleGAN models to perform image-to-image translation. Here, we perform the first detailed exploration of model alignment, also focusing on StyleGAN. First, we empirically analyze aligned models and provide answers to important questions regarding their nature. In particular, we find that the child model's latent spaces are semantically aligned with those of the parent, inheriting incredibly rich semantics, even for distant data domains such as human faces and churches. Second, equipped with this better understanding, we leverage aligned models to solve a diverse set of tasks. In addition to image translation, we demonstrate fully automatic cross-domain image morphing. We further show that zero-shot vision tasks may be performed in the child domain, while relying exclusively on supervision in the parent domain. We demonstrate qualitatively and quantitatively that our approach yields state-of-the-art results, while requiring only simple fine-tuning and inversion. 

Published as a conference paper at ICLR 2022
STYLEALIGN: ANALYSIS AND APPLICATIONS OF ALIGNED STYLEGAN MODELS
Zongze Wu The Hebrew University
Yotam Nitzan Tel-Aviv University
Eli Shechtman
Dani Lischinski The Hebrew University
Adobe Research
ABSTRACT
In this paper, we perform an in-depth study of the properties and applications of aligned generative models. We refer two models aligned they share the same architecture, and one of them (the child) is obtained from the other (the par- ent) via fine-tuning to another domain, a common practice in transfer learning. Several works already utilize some basic properties of aligned StyleGAN models to perform image-to-imane ration of model alignment, also focusing on StyleGAN. First, empirically alyze aligned models and provide answers to important questions regarding their nature. In particular, we find that the child model's latent spaces are semantically aligned with those of the parent, inheriting incredibly rich semantics, even for dis- tant data domains such human faces and churches. Second, equipped with this better understanding, we leverage aligned models to solve diverse set of tasks. In addition to image translation, we demonstrate fully automatic cross-domain image morphing. We further show that zero-shot vision tasks may be performed in the child domain, while relying exclusively on supervision in the parent do- main. We demonstrate qualitatively and quantitatively that our approach yields state-of-the- results, while requiring only simple fine-tuning and inversion.
translation. Here, we perform the first detailed explo-
INTRODUCTION
Transfer Learning (TL) refers the process in which parent model, pretrained for some source domain/task, is used to improve the performance of a child model a different target domain and/or task (Pan & Yang, 2009). The assumption underlying TI. that some knowledge learnt by the parent model is transferable to the new domain or task (Pan & Yang, 2009; Torrey & Shavlik, 2010; Yosinski et al., 2014). The most common TL approach is fine-tuning, where the parent's s parameters used to initialize those of the child. Next, the child's parameters, or sometimes just a subset of them, are trained on the target domain/task. Once TL is completed, the child posseses some of the parent's knowledge, despite the fact that the model parameters may have changed.
Existing TL literature typically examines the performance of the child model, e.g., in terms of classi- fication accuracy (He et al., 2019), or FID score (Karras al., 2020a), without paying much attention to the relationship between parent and child models, induced by the transfer process. Typically, the child model simply applied the task it was trained on, while the parent model is longer used, having fulfilled its purpose. In this work, we provide a complementary perspective, which focuses on analyzing and leveraging the shared knowledge between the two models. Specifically, we con- sider the case where the TL is performed by fine-tuning the same architecture. We refer to models obtained this manner as aligned models.
Several recent works (Pinkney & Adler, 2020: bryandlee, 2020: Kwong et al., 2021; Song et al., 2021; Gal al., 2021) use aligned models in a novel manner. In all cases, an unconditional GAN, specifically StyleGAN2 (Karras et al., 2020b) is fine-tuned from domain A to domain B. However, instead of applying the child model as an unconditional generator, it is used in conjunction with the parent model to form an image translation pipeline. First, an image from domain A is embedded into the latent space of the parent StyleGAN2 model. The resulting latent code then fed either into the child model (bryandlee, 2020; Song et al., 2021; Gal et al., 2021), or into hybrid model created by layer swapping. i.c., by combining layers from the parent and the child (Pinkney & Adler, 2020; K Kwong et al., 2021).

---

Published as a conference paper at ICLR 2022
These methods have achieved great results in image-to-ines translation between several domains. Most notably, translating real human face images to a variety of styles such as cartoons and oil paint- ings (Pinkney & Adler, 2020; Kwong et al., 2021; Song et al., 2021). but also translating humans to dogs and cats to wildlife (bryandlee, 2020; Gal et al., 2021). However, they focus on a specific application (image-to-imanl
translation) and do not explore or leverage aligned models further. As a result, many questions arise but remain unanswered. For example, which parts of the network change in the TL process, and which knowledge inherited by the child from its parent? To which degree do the answers to these questions depend on the similarity between the parent and child do- mains? And, is knowledge not used by the child model completely lost or could it be recovered? Finally, what further applications, besides image-to-image translation, can be solved using aligned models?
In this work, we delve deeper into model alignment. In light of previous works, we specifically focus on the state-of-the-an unconditional GAN architecture, StyleGAN2 (Karras et al., 2020b). The process of obtaining aligned models is incredibly simple: we start with parent StyleGAN2 model trained on domain A and fine-tune it fully for domain B, yielding an aligned child model.
We divide the investigation of model alignment into two parts. First, in Section 3, we perform the first empirical analysis of the phenomenon, answering the questions posed above, as well as others. This analysis provides some surprising and novel insights that shed light on aligned StyleGAN2 models. For example, we discover that when fine tuning to a similar target domain, the parts of the model that change the most are the feature convolution weights in the synthesis network. In contrast, the changes in the mapping network and affine layers are negligible (see Figure 1). This crucially implies that the learned latent spaces W and S are barely affected by the fine-tuning. This explains our next discovery - that semantically meaningful directions in the latent space of the parent model, retain the same (or similar) semantics in the child model (see Figure 2). As the data domains become more distant, the mapping network and affine layers become more affected, which results in a weaker degree of semantic alignmens. However, even extreme cases, such as human faces and churches, some semantic alignment occurs. Another surprising discovery is that the semantic latent controls that seemingly disappear after transfer to the child model, are in fact merely hidden, rather than forgotten, and reappear if the child is retrained back to the parent's 's domain.
Second, in Section 4, we use aligned models to solve several popular Computer Vision and Computer Graphics tasks. We start with the aforementioned image-to to-image translation task (Section 4.1), ex- amine a number of alternatives, and show that aligned models obtain state-of-the-ar results for a variety of scenarios. This is especially impactful, as using aligned models for image translation is incredibly simple compared to dedicated methods for the same task, each devising its custom archi- tecture and losses. Next, we explore additional tasks, for which aligned models have not been used before. In Section 4.2 we describe a simple method for fully automatic image morphing between fairly dissimilar domains, such as human to dog faces, which previously necessitated sophisticated methods (Aberman et al., 2018; Fish et al., 2020). Examples of smooth morphs are included in the accompanying video. In Section 4.3, we use aligned models to solve zero-shot classification and regression tasks in domain B, where the supervision is available strictly in domain A. Conceptually, our method reduces task in a zero-shot or few-shot setting to the same task in a different data domain where supervision is plentiful.
In summary, while several previous works took advantage of aligned models implicitly, ours is the first work to conduct a thorough empirical study of this phenomenon. Our study reveals various in- teresting properties that we then use to further leverage aligned models for variety of applications, almost effortlessly achieving state-of-te oof-the-ar performance.
2 RELATED WORK
Latent Space of GANs:
With the rapid evolution of GANs (Goodfellow et al., 2014) in recent years, understanding and controlling their latent representation has attracted considerable attention. Specifically, it has been shown that the intermediate latent space of StyleGAN (Karras et al., 2019; 2020b;a) possesses appealing properties, such as being semantically rich, disentangled and smooth. Many recent works have proposed methods to interpret the semantics encoded in that space and its extensions and apply them to image editing (Jahanian et al., 2019; Shen et al., 2020a: Härkönen al., 2020; Tewari et al., 2020; Abdal et al., 2020; Wu et al., 2020; Patashnik al., 2021).
2

---

Published as a conference paper at ICLR 2022
In order to benefit from these properties in real images, it is necessary to obtain the latent code from which a pretrained GAN can reconstruct the original input image. This task, commonly referred to as GAN Inversion, has been tackled by numerous recent works, either by using: (i) optimization (Abdal et al., 2019; Karras et al., 2020b); or (ii) an encoder (Guan et al., 2020; Pidhorskyi et al., 2020; Richardson et al., 2021; Tov et al., 2021); or (iii) hybrid approach using both (Zhu al., 2016; Baylies, 2019; Zhu et al., 2020a). See Xia et al. (2021) for a more thorough review.
Image-to-Image translation: The seminal pix2pix work by Isola et al. (2017), first introduced the use of conditional GANs to solve various supervised image-to-image translation tasks. Since then, their work has been extended to allow image synthesis in various different settings: high-resolution (Wang et al., 201 8a), semantic image (Park et al., 2019; Zhu et al., 2020b; Liu et al., 2019b), multi- domain (Choi et al., 2018), 8), multimodal (Zhu et al., 2017b), and using a pre-trained generator (Nitzan et al., 2020; Richardson et al., 2021; Luo et al., 2020). Another scenario that has received significant attention is unsupervised image-to-ing to-iaae translation (Liu al., 2017; Zhu al., 201 7a; Kim al., 2017: Choi et al., 2020; Lee et al., 2020b), where no paired data samples are given.
Regardless of the setting, all of the aforementioned works train an neural network, designed ex- plicitly for the translation task. Recently, several works (Pinkney & Adler, 2020; bryandlee, 2020; Kwong et al., 2021; Song et al., 2021; Gal et al., 2021) have taken a different approach towards -mmseeet--mmee translation. They observe that significant correspondence between generated im- ages in different domains exists when an unconditional generator, such as StyleGAN2 (Karras et al., 2020b), is fine-tuned between the two domains. Accordingly, these works take two-step approach towards image-to-image translation. First, they invert a given image into the latent space of Style- GAN in domain and then forward the output latent code through a StyleGAN model for domain B. The latter model is obtained either by directly fine-tuning from the former model (bryandlee, 2020: Song et al., 2021; Gal et al., 2021) or by layer swapping (Pinkney & Adler, 2020; Kwong et al., 2021), i.e., forming a model whose layers are partially those of the fine-tuned model and partially those of the model for domain A.
In this work, we delve deeper into this phenomenon, which we refer to as model alignment, and go beyond the smage-to--image translation task. For example, we demonstrate that the alignment prop- erty goes beyond high-level properties, such as pose, and that multiple fine-grained latent semantics are also aligned. We leverage this property for tasks such as morphing and zero-shot classification.
Fine-tuning and Catastrophic Forgetting: Fine-tuning was proven advantageous across fields, settings and tasks and therefore became a standard practice in the deep learning literature. Prominent advantages of fine-tuning are enabling few-shot tasks such classification (Chen et al., 2019) and unconditional generation (Wang et al., 2018b; Mo et al., 2020; Wang et al., 2020; Li et al., 2020; Ojha et al., 2021), improved performance in wide variety of tasks (Devlin et al., 2018; Radford et al., 2018; He et al., 2020) and faster training convergence (Wang et al., 2018b; He et al., 2019).
While fine-tuning over 30 years (McCloskey & Cohen, 1989) that in the process the model "forgets" how to solve the original task, a phenomenon referred to Catastrophic Forgetting (CF). For example, once a GAN for a certain domain A, is fine-tuned to another domain B, the resulting model can only generate images in domain B (Seff et al., 2017: Zhai et al., 2019). In settings such as continual learning and multi-task learning, CF is undesirable. In recent years, there has been progress in mitigating it using dedicated methods (Kirkpatrick al., 2017: Kemker et al., 2018). CF has been also studied in the context of GANs (Liang et al., 2018; Li et al., 2020; Thanh-Tung & Tran, 2020).
be an effective technique for solving new task, it has been well known for
Aforementioned previous works devised methods to obtain a better child model using fine-tuning. From a fine-tuning perspective, this means the model would perform better on the new task. From a CF perspective, this means the model's performance on the previous task should not be impaired. We differ from these works significantly, as we make no deliberate effort to affect what happens during training of the child model. Instead, we investigate the relationship between the parent and child models after naive fine-tuning, and then use it to solve a variety of applications.
ANALYSIS OF ALIGNED STYLEGAN MODELS
As explained earlier, several previous works observed that a significant correspondence exists be- tween images in different domains, generated from the same latent code by a parent StyleGAN2
3

---

Published as a conference paper at ICLR 2022
full transfer
reset mapping
reset affine
reset tRGB
reset feat. conv
Figure 1: Effect of reset- ting the weights of different components in child models (Mega, Dog) to their initial values, which come from the parent model (FFHQ). Reset- ting the feature convolution weights causes the most dras- tic changes. Also see Figure 8.
Mega
Dogg
model and a child model obtained from it via fine tuning. Our goal is to further understand the relation between the parent and the child models. Below, we explore several aspects.
Which parts of the network change during transfer? Recall that the StyleGAN2 model is com- posed of a mapping function (3 to W), affine transformations (W to S), feature convolution layers, and tRGB convolution layers that transform feature maps to RGB images. We transfer parent StyleGAN2 model pretrained on FFHQ to the Mega cartoon dataset (Pinkney & Adler, 2020) and to AFHQ dog faces dataset (Choi et al., 2020), using ADA (Karras et al., 2020a). After the transfer, reset the weights of each of the above components in the child models (Mega, Dog) to their initial values in the parent model. The results of this experiment are shown in Figure 1. We observe that the greatest effect on the generated results is caused by resetting the feature convolution lay- ers, which changes the content and structure. Resetting the weights of other components, results in milder changes in both children. This implies that feature convolution layers change the most during transfer. The results also suggest that for the dog model, the affine and tRGB layers have changed significantly more than for the cartoon model. We attribute this difference to the distance between the data domains, and additional experiments in the appendix (Figure 8) support this hypothesis.
While resetting the mapping network has stronger effect on the dog model, note that the changes are fairly subtle in both datasets, implying that the mapping network changes very little. Effectively, this means that the same
Z is mapped to similar codes in the W spaces of the parent and the child; in other words, the two W spaces are point-wise aligned. This is a crucial observation as explains the success of previous works (Pinkney Adler, 2020; bryandlee, 2020; Kwong et al., 2021) in performing image translation based on aligned models. Simply put, the two latent spaces may be viewed as a single shared latent space. Thus, inversion serves as an encoder from the source domain this latent space, and the generator is decoder to the target domain. Viewed in this light, alignment-
image translation resembles several previous image translation approaches (Liu et al., 2017; Huang al., 2018; Liu et al., 2019a), which are based on shared latent spaces.
Semantic aligmment for similar domains. In addition to point-wise alignment, we find that the W and S latent spaces of the child model are also semantically aligned with those of the parent model. By semantic alignment, we refer to the property that latent space controls that affect various semantic attributes of images generated by the parent, have the same (or analogous) effect in the child model. This phenomenon is demonstrated below, both qualitatively and quantitatively.
We demonstrate alignment on closely related domains, by first fine-tuning a parent pretrained on FFHQ (Karras et al., 2019) to the Mega cartoon face dataset (Pinkney & Adler, 2020) and the Metface portrait dataset (Karras et al., 2020a). Next, apply a variety of latent semantic controls leamt by the parent to the child models. The controls are either individual channels in StyleSpace S, identified by Wu et al. (2020), or directions in W space, from InterFaceGAN (Shen al., 2020b). We manipulate images using these controls "as is" in the parent and child models. The initial latent code obtained by inverting a real image with an e4e encoder (Tov et al., 2021). As may be seen in Figures 2 and 9, regardless of the edited property, or the latent space used, the semantic controls affect the parent and the child models in exactly the same manner. Also see Figures 10 and 11.
To perform quantitative evaluation we measure the alignment by calculating the overlap between semantic controls found independently in the parent and child models. Since latent directions in W affinely related to channels in S, we only examine overlap between style channels. Concretely, we follow Wu et al. (2020) to discover localized channels in both models, and report the number of localized channels for each semantic region in Table ((a). As can be seen, there is consistently large
4

---

discuss Published as a conference paper at ICLR 2022
Original
Bangs
Smile
Gaze
Pose
Age
Gender
Figure 2: Semantic controls discovered for parent FFHQ model retain their function in the chil- dren models (Mega and Metface). This holds for individ- ual channels in S (bangs, smile, gaze), as well as for directions in W (pose, age, gender).
HHQQ
Mega
Metaace
3.169
6.501
9. 9.409
eyebrow eye ear nose mouth neck cloth hair
19
5
41
21
32
46
34
62
eychrow
35
15
eyebrow eye mouth neck cloth hair
eye
5 41
32
46
34
62
ear 19
16
torso 25
4
nose 39
14
eye 3
mouth 14
6
car 142
13
neck
24
18
nosc
10
cloth
27
13
(b) FFHQ2Dog
hair 10
10
(a) FFHQ2MeiFace
Table 1: Number localized StyleSpace channels for various semantic regions. Each column corresponds to semantic region in parent model, and each row to semantic region in child model. The number of localized channels that are shared between parent and child are in the center (an empty space denotes 0). (a) After transferring from natural face to portrait, number of localized channels retain their functions in the same areas (large values on the diagonal), rather than changing their function to other areas (all zeros except diagonal). (b) Even when transferring between more distant domains (human to dog face), we can see that multiple channels retain their function in the same areas (nose, ear), or shift to semantically corresponding areas (from human clothes to dog's torso, from human hair to dog's ears). Note that the dog face segmentation have no mouth region.
amount of overlapping channels the same semantic region. We verify that this overlap is not coin- cidental: performing the same experiment for two unaligned FFHQ models (trained from different random initializations) shows that they have much fewer overlapping channels (see Table 4).
To the best of knowledge, tie alignment phenomenon. Our experiments indicate that aligned models for related domains are indeed strongly semantically aligned. This phenomenon enables many applications based trans- ferring knowledge and supervision between aligned models. E.g., zero-shot editing as demonstrated Figure 2 and zero-shot clasifiicatonn/regession discussed in Section 4.3.
the first quantitatively measure the fine fne-grained seman-
Semantic alignment for more distant domains. To examine the degree of semantic alignment across a wider domain gap. we consider StyleGAN2 models transferred from FFHQ AFHQ dog faces (Choi et al., 2020). Figure 3 demonstrates that, even in this case, there are still multiple single-channel controls that retain their semantic meaning (e.g.. big eyes, black hair, short hair). Furthermore, there are also multi-channel editing directions in latent space that exhibit the same behavior, such as curly hair or small face (from StyleCIIP (Patashnik et al., 2021)), as well as pose (from InterFaceGAN (Shen et al., 2020b)). This appears to be the case for visual attributes that common to both domains, while controls for attributes that are not present the target domain (such glasses, lipstick, or beard) seem have effect the child model. However, later, the relevant knowledge is not lost; rather, it is only hidden.
The retained controls reflect some interesting analogies between the domains: for example, controls for hair color and curliness in humans, control fur color and curliness dogs, while hair length translates length of dog ears. Interestingly, psychologists have also observed correlation between
5

---

Published as a conference paper at ICLR 2022
Figure 3: Semantic alignment between single-channel and multi-channel con- trols for more distant domains (humans and dogs). See also Figure 12 and supp. videos.
Original
Big Eyes
Black Uliir
Short Hair
Curly Hair
Small Face
Pose
FFHQ
Dog
56 Figure 4: A smooth transition in images generated from the same latent code E 2 during fine- tuning. The epoch number ap- pears above each column. The most significant visual changes occur in early epochs (0-16). Also see Figure 13 and supple- mentary videos.
4
8
12
16
Dog2Cat FFHQ2DO
the hair length in women and the ear shape of their preferred dog breeds (Coren, 1999), which is consistent with the folk belief that people look like their dogs. The gradual emergence of some of these analogies is clearly revealed when examining the samples generated by the model as it evolves during the transfer process. Figure 4 shows images obtained for the each row), as the training progresses. In the top row, we can see how human hair gradually evolves into dog cars, and the human nose and mouth gradually evolve into the dog's nose and muzzle, while the pose remains mostly unchanged. A similarly smooth transition may be observed when transferring from AFHQ dogs to cats, as shown the bottom row.
latent vector z E 2 (in
Using the same quantitative evaluation method as before, we further quantify the alignment between a parent FFHQ model and child AFHQ dogs model. Results are displayed in Table 1(b). As can be seen, smaller number of channels preserve their semantics when transferred to AFHQ dog as compared to MetFace. Nevertheless, we still observe semantic alignment, albeit weaker, as human cars and hair overlap with dog cars, human cloth control the dog torso, etc.
We next experiment with even farther domains, with barely any similarity between parent and child, such human faces and churches, which were also examined by Ojha et al. (2021). Despite lack of commonality, the latent direction that controls face pose in the parent still controls the church pose in the child model (see Figure 14). We further examine double transfer, with FFHQ as parent, AFHQ dog as child and LSUN bedroom as grandchild. The pose direction in FFHQ still controls the pose in the grandchild bedroom model, shown in Figure 14.
Are latent semantics forgotten or hidden? As shown in Table 1(b), when transferring between distant domains, only small portion of the localized controls retain similar semantic function. An interesting question that arises is: are the remaining controls completely "forgotten" during the transfer learning, do they simply become inactive? To examine this, retrain the child AFHQ dog model back to the FFHQ domain, thereby obtaining grandchild model, and report the alignment between the original parent and the grandchild models (both for FFHQ) in Table may be seen that the effect of many of the localized controls are restored. For example, out of the 41 channels that control the ears in FFHQ, only 2 retain similar function in AFHQ dogs, but 20 regain their function in the grandchild model. This implies that these channels were merely hidden, but not forgotten, during the first transfer learning stage. It should be emphasized that there barely any such alignment between two unrelated models, even when they trained the same dataset, as shown in Table 4. Thus, the significant alignment between parent and grandchild (in Table 3) cannot be attributed to re-learning when fine tuning from the child to the grandchild.
Locality bias in semantics transfer. We explore this aspect and conclude that only some of the semantic alignment be attributed to locality bias (see the discussion in appendix Section A.3).

---

Published as a conference paper at ICLR 2022
Input
Ours
CUT
F-LSeSim
Input
Ours
CUT
F-LSeSim
CUT
F-LSeSim
Ours (2opt)
CUT
F-LSeSim
Ours (2opt)
cat2dog 74.9
73.8
34.2
44.5
36.3
7.36
dog2wild
25.4
37.6
10.9
10.0
17.6
2.22
(a) FID
(b) KIDx10
Figure 5: Comparison of 121 translation (cat2dog and dog2wild in the AFHQ dataset) with two state- of-the-art methods. Our method generates realistic target domain images that capture the pose from the source image. In contrast, both CUT and F-LSeSim fail to generate realistic images since they follow the shape of the source domain image too closely. A quantitative comparison in the table below indicates method is superior by wide margin, in both FID and KID.
4 APPLICATIONS
We next apply aligned models to solve three kinds tasks: image-to-imase translation (Sec. 4.1), cross-domain image morphing (Sec. 4.2) and zero-shot classification and regression (Section 4.3). Efficient training of generators for different resolutions is described in the appendix (Sec. A.4).
4.1 CROSS-DOMAIN IMAGE TRANSLATION
As demonstrated earlier, aligned models generate images with similar high-level semantic attributes, given the same latent code. This makes it trivial to translate images between the domains of the parent and the child models, even when these domains are more distant than realistic faces and cartoons paintings of human faces. For example, is easy to translate between faces of different species, which typically involves significant changes in both structure and appearance. Furthermore, there's 's need for task-specific training losses; all that is needed is pair of aligned models and an inversion method to embed real images into the latent space of the source domain StyleGAN.
In Section A.5 perform systematic study of which inversion methods (encoder latent opti- mization), and which latent spaces (W/W+/2/2+), are most effective for image translation. Some previous works (Pinkney & Adler, 2020; Kwong al., 2021) that considered only similar domains have used the W/4+ spaces. We find that Z space yields same level of results for similar domains, but superior results for distant domains, qualitatively and quantitatively. This could be directly ex- plained with a previous observation. For both settings the Z space is trivially shared, as it is non-learned space. However, only for similar domains are the W/W+ spaces aligned and shared.
In Figure 5 we compare our 12I results to two of the-ar art methods, CUT (Park et al., 2020) and F-LSeSim (Zheng et 2021). It may be seen that our method produces realistic and natural looking results, while these two previous methods exhibit severe artifacts, and attempt to follow the shape in the source image too closely, yielding unrealistic results. The table in Figure 5 provides quantitative support for our qualitative observations, yielding significantly lower FID and KID scores for both cat2dog and dog2wild translations. Figure 21 demonstrates our method's ability to perform image translation between dissimilar domains.
In addition to the 121 scenario examined above, aligned models are also able to perform reference- based image translation, where the resulting image combines the content of source image with the style from a second (reference) image (Huang et al., 2018; Choi et al., 2020). StyleGAN inherently supports content and style disentanglement through style mixing. Specifically, we combine the

---

Published as a conference paper at ICLR 2022
Source
Reference
Ours
StarGAN2
OverLORD
Source
Reference
Ours
StarGAN2
OverLORD
StarGAN22
OverL.ORD
Oars ( (2opt)
StaGAN2
OverLORD
Ours (Z
dog2cat
15.6
22.1
9.22
12.7
19.2
3.44
wild2dog
37.4
32.7
27.4
21.5
17.1
14.8
cat2dog
44.0
32.2
30.4
25.6
16.8
17.1
dog2wild
17.2
10.5
9.65
11.8
5.1
3.64
(a) FID
(b) KIDx10*
Figure 6: Comparison of reference-based image translation with StarGAN2 and OverLORD. Our method generates realistic target domain images that combine pose and structure from the source image with texture and color from the reference. StarGAN2 follows the source shape closely, resulting in non-realistic animals (1st example in dog2cat, all examples in wild2dog). OverL results preserve the appearance of the reference well, but sometimes fail to capture the pose and structure (e.g., ear shape) from the source image (2nd and 3rd examples in wild2dog). A quantitative comparison in the table below indicates superior performance of our method in both FID and KID.
early latent code (below 32 x 32 resolution) from a source image, with the late latent code (above equal to 32 32 resolution) from target domain reference image, and feed it the target model Figure 22. Figure 23 and Table 6 show that here, as well as for 121, inversion via Zopt works better than other inversions/spaces for multi-modal image translation. Figure 6 demonstrates that our results are better than those of current state-of-the-an methods, StarGAN-v2 (Choi al., 2020) and OverLORD (Gabbay & Hoshen, 2021).
to generate the result, as demonstrated
4.2 CROSS-DOMAIN IMAGE MORPHING
Image morphing is popular visual effect of smoothly transitioning between a pair of input images (Wolberg, 1998), which typically requires either manual automatic correspondences, in order to define a warp field. Cross-domain morphing, where the two images are from different domains, A and B, is particularly challenging (Aberman et al., 2018; Fish et al., 2020). However, using pair of aligned StyleGAN models for the two domains, it is possible to perform cross-domain image morphing automatically without the need for correspondences, or any other input! The two input images are first embedded into the W+ space of the corresponding generators, using e4e encoders (Tov et al., 2021). Next, smooth transition obtained by linearly interpolating between the resulting latent codes, while also interpolating between the model weights. Wang et al. (2019) previously proposed interpolating model weights in order to obtain a smooth transition between the "effects" of two different networks. We note that they do not discuss morphing real images, which a slightly different setting, and requires also interpolating latent codes as we propose here.
Layer swapping, proposed by Pinkney & Adler (2020), is alternative approach morph between domains. We discuss the differences between the two approaches in Section A.7. Concisely, our pro- posed method ensures continuous smooth transition, while layer swapping performs the transition as a series of discrete steps, rather than continuously.
We demonstrate automatic morphing between dog and cat faces in Figures 24 and 25, and dog and human faces in Figures 26 and 27. Interpolating the model weights (along each column) yields smooth transition between domains (different species, but the same pose and fur color), while interpolating the Wt latent codes (along each row) smoothly transitions inside each domain (same species, varying pose and fur color). In fact, any trajectory in this 2D interpolation space yields a smooth morph sequence between two input images. We simultancously interpolate along both dimensions to create the sequences shown in the supplementary video.
8

---

Published as a conference paper at ICLR 2022
Positive
Neutral
Negative
Black Fur
Cary Fur
Down Eat
Figure 7: Zero-shot dog attribute classification using aligned models (FFHQ and AFHQ dogs). In
the top row human "black hair" classifier becomes a "black fur" classifier, a "curly hair" classifier is able to classify "curly fur", and a "long hair" classifier becomes "down-pointing ears" classifier. The neutral columns correspond to images whose prediction scores are close to the cutoff value.
4.3 KNOWLEDGE TRANSFER FROM PARENT CHILD DOMAIN
Vision tasks human faces have been researched for years. Consequently, numerous datasets with detailed annotations exist. For example, images in the CelebA dataset (Liu et al., 2015) are labeled with 40 attributes such as "Young", "Curly hair", "Smiling", etc. Such annotations are not available for almost any other domain, such as animal faces, severely limiting the range of tasks that can be solved. As discussed earlier, this issue is prominent motivation for transfer learning. However, common transfer learning approaches are not applicable in the "zero-shot" setting, where there is abundant labeled data in the source domain, but strictly unlabeled data in the target domain.
We next show that this problem can be solved effectively for directly comparable attributes across domains by leveraging aligned models. Consider the case of head pose (specifically, yaw): clear and comparable attribute for both humans and dogs, however for humans there is abundant labeled data and for dogs there none. As demonstrated earlier, the latent pose semanties are aligned in the two models, and the parent's yaw editing direction continues to edit yaw in the child. As shown ear- lier, this holds for additional attributes. Thus, despite a major gap between the two domains image space, the gap in the latent space is considerably smaller, enabling transfer of knowledge between these domains. While naïvely applying model trained on the source images to the target images would fail, this approach works well when applied on the latent representation. To demonstrate this approach, we solve several zero-shot classification and regression tasks using models trained in the latent space of the parent StyleGAN model.
For regression tasks, we use LARGE (Nitzan et al., 2021), which demonstrated that the distance in W+ space to the decision hyperplane associated with semantic property, gauges the degree of that attribute in image space. See appendix (Section A.6) for more details. Zero-shot yaw regression results are depicted in Figures 28 and 29. As evident, the estimated yaw not only captures the correct tendency, but also produces a value that qualitatively seems reasonably close to actual yaw degree.
For classification tasks take similar approach. We simply replace the linear regression model with a logistic regression model and use a cutoff value of 0.5. As shown in Figure 7, our method can turn classifiers for human faces to classifiers for dog faces. These results also demonstrate that the attributes are not required to be exactly identical (pose to pose) but could be comparable in a more broad sense (long hair in humans to down wwwn-pointing ears in dogs).
CONCLUSION
In this work, we performed the first extensive investigation of the properties of aligned generative models. We initially answered several open questions, crucial for their understanding. The find- ings demonstrated impressive and surprising properties, such as semantic alignment across distant domains and knowledge being "hidden" instead of being forgotten. We then leveraged our new insights to apply aligned models for multitude of tasks. Interestingly. results for those tasks with a single, simple fine-tuning based method. We hope that our work can inspire others to consider aligned models simple paradigm for solving wide range of tasks.
obtain tfite-of-the-na

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
This work performs an extensive study of the properties of aligned generative models and applies such models for several computer vision tasks. In general, generative models and learning-based algorithms raise several concerns. Notably, generative models may be used to produce deceiving or offending content, e.g. deepfakes (Wikipedia, 2021), and data-driven algorithms may perpetuate biases exiting in their training sets. However, these concerns are general to the entire fields and are not amplified by this work.
7 REPRODUCIBILITY
Throughout the paper we provide detailed information facilitating reproduction of our results. For example, in each experiment specify the choices of latent space, specific layer and inversion method (e.g., Sections 4.1, 4.2 and A.5). Similarly, when applying latent editing directions we spec- ify with which method were they identified and in what space (e.g., Section 3, A.3). Additionally, we provide in the appendix (Section A.8) the information required to reproduce the child models. We expect these be sufficient for independent replication of our main findings. Separately, source code and pretrained models have been made available in the project's 's repository.
8 ACKNOWLEDGMENTS
We thank Daniel Cohen- Or for helpful discussions and encouragement and the anonymous reviewers for their comments. This work was supported in part by a gift from Adobe, by the Israel Science Foundation (grant no. 2492/20), and the Joint NSFC-ISF Research Grant Program (361 1/21).
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the StyleGAN latent space? In Proceedings the pp. 4432-4441, 2019.
the IEEE international conference on computer vision,
Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. StyleFlow: attribute-condtiooned ex- ploration of BoollGGN--geerrttee images using conditional continuous normalizing flows. arXiv preprint arXiv:2008. 02401, 2020.
Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, and Daniel Cohen-Or. Neural best-buddies: Sparse cross-domain correspondence. ACM Transactions on Graphics (TOG), 37 (4):69, 2018.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network Dissection: quantifying interpretability of deep visual representations. In Proc. CVPR, pp. 6541-6549, 2017.
Peter Baylies. http://////o/o////o/ooo////////a 2019. Accessed: January 2021.
bryandlee. FreezeG. https: / http://gghhh..cc////////eeeeee/e 2020. 2021.
2020. Accessed: May
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classification. arXiv preprint urXivv1994..44. 2019.
Yunjey Choi, Minje Choi, Munyoung Kim, lung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- GAN: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8789-87977 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8188-8197, 2020.
10

---

Published as a conference paper at ICLR 2022
Stanley Coren. Do people look like their dogs? Anthrozoös, 12(2):111-114, 1999.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint /.X0000810004800 2018.
N. Fish. R. Zhang, L. Perry, D. Cohen-Or, E. Shechtman, and C. Barnes. Image morphing with perceptual constraints and STN alignment. Computer Graphies Forum, 3966)303--313, 2020.
Aviv Gabbay and Yedid Hoshen. Scaling-up disentanglement for image translation. arXiv preprint arXiv:2103. 14017, 2021.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. StyleGAN-NADA: CLIP-guided domain adaptation of image generators. arXiv preprint arXiv:2108. 00946, 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2672-2680, 2014.
Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, and Xiaokang Yang. Collaborative learning for faster StyleGAN embedding. arXiv preprint arXiv:2007.01758 2020.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. GANSpace: Discovering interpretable GAN controls. arXiv preprint arXiv:2004.02546 2020.
Kaiming He, Ross Girshick, and Piotr Dollár. Rethinking imagenet pre- pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4918-4927,
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Xun Huang, Ming-Yo Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to- image translation. In ECCV, 2018
Phillip Isola, Jun Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1125-1134, 2017.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. arXiv preprini arXiv: (T0.::1997.071111 2019.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. CVPR, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. NewrIPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proc. CVPR, pp. 8110-8119 2020b.
Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Measur- ing catastrophic forgetting in neural networks. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, pp. 3390-3398, 2018.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In International Conference on Ma- chine Learning, pp. 1857-1865. PMLR, 2017.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinsker et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3511-3522 2017.
Sam Kwong, Jialu Huang, and Jing Liao. Unsupervised amage-to-image translation via pre-trained StyleGAN2 network. IEEE Transactions on Multimedia, 2021.

---

Published as a conference paper at ICLR 2022
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5549-5558, 2020a.
Hsin- Hsin-Ying Lee, Hung- Yu Tseng, Qi Mao, Jia-Bir Bin Huang, Yu-Ding Lu, Maneesh Kumar Singh, and Ming-Hsuan Yang. DRIT++: Diverse image-to-image translation via disentangled representa- tions. International Journal of Computer Vision, pp. 1-16, 2020b.
Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. arXiv preprint arXiv:2012.02788, 2020.
Kevin Liang, Chunyuan Li, Guoyin Wang, and Lawrence Carin. Generative adversarial network training is a continual learning problem. arXiv preprint arXiv:1811.11083 2018.
Ming- Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-imale translation networks. In Advances in neural information processing systems, pp. 700-708, 2017.
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In IEEE International Conference on Com- puter Vision (ICCV), 2019a.
Xihui Liu, Guojun Yin, Jing Shao, Xiaogang Wang, al. Learning to predict layout-to-image condi- tional convolutions for semantic image synthesis. In Advances Neural Information Processing Systems, pp. 570-580, 2019b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild, 2015.
Xuan Luo, Xuaner Zhang, Paul Yoo, Ricardo Martin-Brualla, Jason Lawrence, and Steven M. Scitz. Time-travel rephotography. arXiv preprint arXiv:2012.12261, 2020.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower (ed.), Psychology of Learning and Motivation, volume 24, pp. 109-165. Academic Press, 1989.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine- tuning GANs. arXiv preprint arXiv:2002. 10964, 2020.
Yotam Nitzan, Amit Bermano, Yangyan Li, and Daniel Cohen-Or. Face identity disentanglement via latent space mapping. ACM Trans. Graph., 39(6), November 2020. ISSN 0730-0301. doi: 10.1145/34 URL httph:tt///om /tw./oo.ooooooo http:///oo..or///....oe.../ w1t.///....11111411111
341 7826.
Yotam Nitzan, Rinon Gal, Ofir Brenner, and Daniel Cohen-Or. LARGE: Latent-based regression through GAN semanties. arXiv preprint arXiv:2107.11186. 2021.
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. arXiv preprint arXiv:2104.06820 2021.
Sinno Jialin Pan and Qiang Yang. survey on transfer learning. IEEE Transactions on knowledge and data engineering, (345):1345-1359 2009.
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun- Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2337-2346, 2019.
Taesung Park, Alexei A Efros, Richard Zhang, and Jun Man Zhu. Contrastive learning for unpaired smage-to-image translation. In European Conference on Computer Vision, pp. 319-345. Springer, 2020.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. StyleCLIP: Text- driven manipulation of StyleGAN imagery. arXiv preprint arXiv:2103. /7249, 2021.
12

---

Published as a conference paper at ICLR 2022
Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. Adversarial latent autoencoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 14104-141 14113, 2020.
Justin NM Pinkney Doron Adler. Resolution dependant GAN interpolation for controllable image synthesis between domains. arXiv preprini arXiv:2010.05334 2020..
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training preprint, 2018.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a StyleGAN encoder for image-to-image translation. In Proc. IEEE/CVF CVPR, pp. 2287-2296, 2021.
Ari Seff, Alex Beatson, Daniel Suo, and Han Liu. Continual learning in generative adversarial nets. arXiv preprini arXiv: (rXxv:/705.0839) 2017.
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou Interpreting the latent space of GANs for semantic face editing. In Proc. CVPR, pp. 9243 9252, 2020a.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. InterFaceGAN: interpreting the disentan- gled face representation learned by GANs. arXiv preprint arXiv:2005.09633 2020b.
Guoxian Song, Linjie Luo, Jing Liu, Wan-Chun Ma. Chumpong Lai, Chuanxia Zheng, and Tat-Jen Cham. AgileGAN: stylizing portraits by inversion-consistent transfer learning, ACM Transactions on Graphics (TOG), 40(4): 2021.
Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj. Florian Bernard, Hans-Peter Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt. StyleRig: Rigging StyleGAN for 3d control over portrait images. arXiv preprint arXiv:2004.00121, 2020.
Hoang Thanh-Tung and Truyen Tran. Catastrophic forgetting and mode collapse in GANs. In 2020 International Joint Conference on Neural Networks (LJCNN), pp. 1-10. IEEE, 2020.
Lisa Torrey and Jude Shavlik. Transfer learning. In Handbook of research machine learning applications and trends: algorithms, methods, and techniques, pp. 242-264. IGI global, 2010.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for StyleGAN image manipulation. arXiv preprint arXiv:2102.0277 100.02666 2021.
Ting- Chug Wang, Ming- Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional GANs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8798-8807, 2018a.
Xintao Wang, Ke Yu, Chao Dong, Xiaoou Tang, and Chen Change Loy. Deep network interpola- tion for continuous imagery effect transition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1692-1701, 2019.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez- Gariia, and Bogdan Raducanu. Transferring GANs: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 218-234, 2018b.
Yaxing Wang, Abel Konzalez-Garcia David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. MineGAN: effective knowledge transfer from GANs to target domains with few images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 9332-9341, 2020.
Wikipedia. Deepfake. http::///en..kkkkaaaaaa..o///o/p
2021.
George Wolberg. Image morphing: survey. The Visual Computer, 14(8):360-372 1998.
Zongze Wu, Dani Lischinski, and Eli Shechtman. StyleSpace analysis: Disentangled controls for StyleGAN image generation. 1rXiv:2011.12799 2020.
13

---

Published as a conference paper at ICLR 2022
Weihao Xia, Yulun Zhang, Yujiu Yang, Bing-Hao Xue, Bolei Zhou, and Ming Hsuan Yang. GAN inversion: survey, 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proc. ECCV, pp. 418-434, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27, 2014.
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. BiSeNet: Bilat- eral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 325-341, 2018.
Mengyao Zhai, Lei Chen, Frederick Tung, Jiawei He, Megha Nawhal, and Greg Mori. Lifelong GAN: Continual learning for conditional image generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2019.
Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. The spatially loss for various image translation tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16407-16417, 2021.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain GAN inversion for real image editing, arXiv preprini arXiv:2004.00049 2020a.
Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros. Generative visual manipula- tion on the natural image manifold. In Proceedings of European Conference on Computer Vision (ECCV), 2016.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proc. IEEE ICCV, pp. 2223-2232, 2017a.
Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Darrell, Alexei A Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In Advances in Neural Information Processing Systems, 2017b.
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. SEAN: Image synthesis with semantic region-adaptive normalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 5104-5113, 2020b.
A APPENDIX
A. EFFECT OF TUNING ON INDIVIDUAL FEATURE CONVOLUTION LAYERS
As part of our investigation of which parts of the network change during fine-tuning, we also exam- ine in more detail the effect of resetting the weights of individual feature convolution layers on the generated images. We reset one layer at time and measure the perceptual change in image using LPIPS. Results are displayed in Table 2. As may be seen, each layer has effect on the image, but resetting the middle resolution layers (32, 64, 128) induces the greatest L.IPS change.
A.2 FURTHER SEMANTIC ALIGNMENT ANALYSIS
Since many semantic attributes cannot be controlled by single style channel, nor by single ma- nipulation direction in W, we also compare the effect of different semantic manipulation directions in StyleSpace, which discovered for the parent model using CLIP (Patashnik et al., 2021). Fig- ures 10 and demonstrate that these compound manipulations, e.g., expressions and hair styles, also retain their semantics in the child models.
14

---

Published as a conference paper at ICLR 2022
A.3 LOCALITY BIAS IN SEMANTICS TRANSFER.
We have demonstrated that a variety of localized controls retain their function during transfer learn- ing between FFH and AFHQ. However, since the faces in these two datasets are roughly aligned, it is interesting to examine whether this occurs due to overlap between the corresponding seman- tic regions. To examine this, we perform transfer learning from model pretrained on FFHQ (at 256 resolution) to three different versions of the same dataset: (i) shifted 60 pixels to the right, (ii) shifted 60 pixels downward, and (iii) flipped upside down. Figure 15 shows that the shifts, and particularly the flip, affect the identity/appearance of the images generated from the same latent codes in Z, however other high-level characteristics, such as gender, age, or hair length, remain similar. We also show the effect of manipulating five different style channels across different layers and different semantic regions. For the horizontally shifted dataset, all five channels retain their function. For the vertical shift, four out of the five channels retain their function (channel 15_45 that controls lipstick loses its effect). For the upside down flip, two out of five channels (9 409 for gaze and 12 479 for blond hair) retain their function. In summary, for 11 out of 15 cases, the function of a channel was transferred despite a significant change in the locality. Thus, locality bias cannot explain all of the alignment that occurs.
There are, however, some interesting examples of strong locality bias. Channel 6_501 controls smiling in the parent FFHQ model, but after an upside-down flip it controls receding hairline, this implies locality bias does contribute to channel-wist semantics transfer, since the forehead of the flipped faces overlaps the mouth location in the original images.
A.4 SEMANTIC ALIGNMENT BETWEEN DIFFERENT RESOLUTIONS
Given a high resolution StyleGAN2 model, an aligned lower resolution model may be easily ob- tained by simply removing the high resolution layers, and fine-tuning to convergence. The fine- tuning is necessary, as without it the model generates low-contrast images, as shown in Figure 30. This works well because StyleGAN2 inherently supports multi-resolution synthesis, with the gener- ator containing ToRGB layers and the discriminator containing corresponding FromRGB layers that directly operate in image space for different resolutions. Assuming a high resolution (1024 x 1024) model is already available, creating low-resolution model (512 x 512) in this way is computation- ally efficient, requiring less than 2 days of fine-tuning on a single GTX 1080Ti GPU, compared to more than one month of training from scratch. Figure 30 shows that the resulting low-resolution model is highly aligned with the original: the same latent code
Z generates nearly the same image, and the semantic controls in the parent model have the same effect in the child model. One of the important consequences of such alignment is that there's 's no need to spend weeks of GPU time to re-discover the semantic StyleSpace controls (Wu et al., 2020). Furthermore, given an inversion model (Tov et al., 2021) for the parent model, it may be fine-tuned for the child model within a few GPU hours, instead 2-3 days of training from scratch.
A.5 METHODS AND SPACES FOR IMAGE TRANSLATION
To determine which latent space and inversion method (encoder optimization) is best suited for translation of real images, we explore a number of alternatives. We modify the pSp encoder (Richardson et al., 2021) embed images into W, Z, and Z+ (Song et al., 2021) spaces. For W+ we use the e4e encoder (Tov et al., 2021), which is based on pSp, but generates W+ codes with better alignment with the latent manifold. We also modify the latent optimization method from the official StyleGAN2 implementation (Karras et al., 2020b) to embed into these different spaces. For z/Z+, is crucial to use the truncation trick for both image inversion and generation, otherwise the translation results might exhibit strong artifacts (we use a truncation coefficient of 0.7). Inversion results corresponding to these different methods are shown in Figure 16 for AFHQ dogs and cats.
Examples of 12I translation (dog2wild and cat2dog) using these different inversion methods are shown in Figure 17. While inversion of source domain images to W+ yields arguably the best reconstructions, when translating to the target domain via W or W+, the color palette of the results seems wrong, especially for the dog2wild translation. We attribute this to the fact that the mapping function (from 2 to W) changes when fine tuning the parent to the child, which affects the color palette, and translating using W/W+ latent codes ignores this change. Translations via Z+ or 2 opt inversion also suffer from occasional color artifacts (mainly in the dog2wild examples).
15

---

Published as a conference paper at ICLR 2022
Both 2 and Zopt, the other hand, yield satisfactory translation results. We prefer Zopt because it tends to produce a vivid color palette and to maintain a stronger resemblance of the source images, in terms of pose, shape, and colors. Quantitatively, translating via Zopt inversion achieves best FID and KID over the other plausible alternatiees, as reported in Table 5. Therefore we use translation via opp our preferred method.
We perform similar study for translation between nearby domains (FFHQ and cartoon) in Figure 18, 19 and 20. In our subjective opinion, translation via Zopt still achieves the most cartoonish look. However, translations via W bear closer resemblance to the input portrait, while still achieving a satisfactory cartoonish look. As discussed in the text, this may be attributed to the fact that, for similar domains, the mapping function changes little during fine-tuning, resulting in pointwise alignment of the W spaces of the parent and child models.
A.6 ZERO-SHOT REGRESSION
To leverage aligned models for regression tasks, we use LARGE (Nitzan et al., 2021), which demon- strated that the distance in W++ space to the decision hyperplane associated with semantic property, gauges the degree of that attribute in image space. As their method is designed for a few-shot setting, simplify it slightly for our setting where the training data in the source domain is abundant. Con- cisely, we simply use the distances calculated in specific layers known to control certain attributes as the input features for the regression model. We demonstrate this approach for head pose regression and use the first four layers, which are known to control the pose in StyleGAN (Karras et al., 2019; Nitzan et al., 2021). At inference time, we use e4e (Tov al., 2021) to encode images of the target domain into the W+ space of the child model, compute the distances of the first four layers from the decision hyperplane, and input them to the human face yaw estimation model.
The zero-shot yaw regression results for AFHQ dogs and cats are depicted in Figures 28 and 29. As can be seen, the estimated yaw not only captures the correct tendency, but also produces value that qualitatively seems reasonably close to actual yaw degree.
A.7 METHODS TO BLEND ALIGNED MODELS
Layer swapping was introduced by Pinkney & Adler (2020) as a method to generate images of new domain by "blending" together two existing data domains. It does that by creating a hybrid model contains layers from two aligned models. Specifically, the first (coarse) layers are taken from one model and the last (fine) layers are taken from another. We note that this method blends the two data domains in specific manner. Thanks to the hierarchical structure of StyleGAN (Karras et al., 2019), the created model inherits the structure (coarse layers) from one model and texture from another (fine layers). Pinkney & Adler (2020) also mentioned that the fine layers could be interpolated between the models, however this idea wasn't 't applied in practice.
The layer swapping method was shown produce visually pleasing results on the task of stylizing human portraits (Pinkney& & Adler, 2020; Song et al., 1). However, there are few disadvantages to this method. First, when domains are more distant (e.g. faces of humans and dogs), the results obtained by this approach are less intuitive and visually pleasing (see Figures 31 to 33). This coin- cides well with our observation from Figure Since the feature convolution layers change much more significantly when transferring to distant domain, the layers of a layer-swapped model are more alien to each other. Second, the number of intermediate steps is limited by the number of con- volution layers in the generator, which is at most 18. This prevents the application of layer swapping for creating a smooth transition between images from different domains.
In our morphing application (Section 4.2), we present alternative method to blend two aligned models. There we propose to perform a simple linear interpolation of all model weights to achieve a gradual transition. We compare the results of this approach with layer swapping in Figures 31 to 33. Please note that our proposed method is able to obtain "blended" images that seem more smooth and natural.
16

---

Published as a conference paper at ICLR 2022
A.8 FINE-TUNING IMPLEMENTATION DETAILS
Given a model pretrained on the parent domain, we fine-tune it on the child domain. Specifically, model config-f and the default hyper-parameters from the official Nvidia StyleGAN2 and StyleGAN2-ADA implementations in tensorflow. We use the augmentations of StyleGAN2-ADA only when the child domain is AFHQ or Metface.
Note that StyleGAN2-ADA implementation chooses the config based on input image resolution. uses config-f for image resolution above 512 512, and config-e for other resolutions. To config-f without worrying about image resolution, one can specify the flag --cfg stylegan2 when using tran.py, and change line 179 train.py from spec. fmaps 1 if res >= 512 else 0.5 5 to spec. fmaps 1.
A.9 DETECTING LOCALIZED CHANNELS
We follow Wu et al. (2020) to discover localized channels the StyleGAN model. To reduce noise, only consider channels to be localized if they have the strongest gradient in the same semantic region over 75% of sampling images, rather than 50% used in the original paper. For FFHQ and Metface models, we use the semantic segmentation maps from BiSeNet (Yu al., 2018) pretrained on CelebAMask-HH HQ (Lee et al., 2020a). For AFHQ dogs, we using the semantic segmentation maps from unified parsing network (Xiao al., 2018) pretrained on Broden+ (Bau al., 2017).
17

---

Published as a conference paper at ICLR 2022
full transfer
reset mapping
reset aftine
reset IRGB
reset feat. code
Mega
Dog
Figure 8: We reset the weights of different components in child models (Mega, dog, church) to their initial values, which come from the parent model (FFHQ). When resetting the weights in feature convolution layers, the output images change more drastically (content, structure), while resetting the weights of other components causes milder effects. This implies feature convolution layers contain most of new learned knowledge.
18

---

Published as a conference paper at ICLR 2022
Resolution
16
32
64
128
256
512
LPIPS
0.156
0.385
0.390
0.432
0.440
0.405
0.369
0.355
Table 2: We measure the extent to which the change in each feature convolution layer (during fine- tuning) affects the generated images. Given parent FFHQ model and child AFHQ dog model, we reset the feature convolution weights for each resolution of the child model to their original values in the parent model, and measure the LPIPS distance between the images generated by child model before and after resetting the weights. A higher LPIPS score indicates image space, It may be seen that the greatest change is caused by resetting the middle resolution layers (32, 64, 128).
significant change in
eyebrow
eye
car
nose
mouth
neck
cloth
hair
19
41
21
32
46
34
62
eyebrow
29
8
eye
ear
45
20
nose
23
8
mouth
55
11
neck
15
cloth
65
19
hair 70
33
Table 3: The number of localized StyleSpace controls for various semantic regions for an FFHQ parent model and an FFHQ grandchild model, with training flow from FFHQ (parent) to AFHQ dog (child) then back to FFHQ (grandchild). Each column corresponds to semantic region for parent and each row a semantic region for grandchild. The number of localized channels shared between two models is indicated for each pair of semantic regions.
eyebrow
eye
ear
nose
mouth
neck
cloth
hair
19
5
41
21
32
46
34
62
cyebrow
22
eye
5
1
ear
44
1
1
nose
19
mouth
28
1
1
neck
43
1
cloth
32
2
1
hair 85
2
2
Table 4: The number of localized StyleSpace controls for various semantic regions for two randomly initialized FFHQ models. Each column corresponds to a semantic region in one model and each row semantic region in the other model. The number of localized channels shared between two models is indicated for each pair of semantic regions. It is evident that the two models only have small number of overlap channels across unrelated semantic regions (for example, hair and eye). This experiment serves as negative control to show that large number of overlap channels only occurs when the models have parent and child relation, as the case in Table
19

---

Published as a conference paper at ICLR 2022
Original
Bangs
Smile
Gaze
Pose
Age
Gender
FFHQ
Mega
Metface
3 169
6 501
9 409
Figure 9: Semantic alignment: semantic controls discovered for the parent model (FFHQ) retain their function in the children models (Mega and Metface). This holds for individual channels in S (bangs, smile, gaze), where the layer and channel number is indicated under each column. Semantic alignment is also observed for manipulation directions W (pose, age, gender).
Original
Angry
Skinny
Bald
Bob Cut
Hi-top Fade
FHHQ
Mega
Metface
Figure 10: Semantic alignment of multiple channels: semantically meaningful directions in StyleSpace discovered in the parent model (FFHQ). detected using StyleCLIP (Patashnik et al., 2021), still control the same attributes in children models (Mega and Metface).
20

---

Published as a conference paper at ICLR 2022
Original
Angry
Skinny
Bald
Bob Cut
Hi-top Fade
FFHQ
Megaa
Metface
Figure 11:
Semantic alignment of multiple channels: semantically meaningful directions in StyleSpace discovered in the parent model (FFHQ). detected using StyleCLIP (Patashnik et al., 2021), still control the same attributes in children models (Mega and Metface).
Original
Black Hair
Short Hair
Curly Hair
Small Face
Big Eyes
Posc
FFHQ
Dog
11.72
3.161
Figure 12: Examples of semantic aligmment between single-channel, as well multi-channel con- trols discovered for the parent model (StyleGAN2 trained on FFHQ) and a child model (AFHQ dogs). While the analogy between hair humans and fur dogs seems intuitive, there are also some less obvious analogies, such as hair length and ear length.
8
12
16
56
FFHQ2Dog
Dog2Cat
Figure 13: During transfer learning between domains, we can observe smooth transition in images generated from the same latent code Z. The top row demonstrates this for transfer from FFHQ to AFHQ dogs, while the bottom rows shows this for transfer from AFHQ dogs to cats. The number of epochs indicated above each column. The most significant visual changes occur early epochs (0-16), while later epochs mainly improve image quality and realism without significant changes in semantic attributes.
21

---

Published as a conference paper at ICLR 2022
Church
Church
Bedroom
Bedroom
Figure 14: Some degree of semantic alignment is present even when the source and target domains are very dissimilar. In the top two rows, we show that the latent direction that controls pose in the parent FFHQ model still controls pose in the child LSUN church model. In the bottom two rows, we examine a double transfer, with FFHQ as parent, AFHQ dog as child and LSUN bedroom as grandchild. The pose direction in FFHQ still controls the pose in the grandchild bedroom model.
22

---

Published as a conference paper at ICLR 2022
Original
3 169
6. 6.501
9.409
12 479
15. 45
FFHQ
Shift Down
Shift Right
up-down flip
FFHQ
Shift Down
Shift Right
up up-down flip
Figure 15: To understand whether locality bias contributes semantics transfer, fine-tune pretrained FFHQ model 256x 256 resolution, to (i) a FFHQ dataset shifted 60 pixels to the right, (i) a FFHQ dataset shifted 60 pixels downward. and (iii) FFHQ dataset flipped upside-down. We examine the semantics transfer for 5 channels across different layers and different semantic regions. For the shift right case, 5 channels retain their function. For the shift down case, out of channels retain their function (channel 15.45 loses its function for lipstick). For the upside-down (9.409 gaze and 12.479 blond hair) retain their function. In summary, for 11 out of 15 cases, the semantic function of channels is transferred even if we break the locality bias. These results imply that the transfer semantics cannot be fully attributed locality bias.
flip. 2 out
23

---

Published as a conference paper at ICLR 2022
Original
W
W+
Z
2+
Zopt
Z opt
Figure 16: To invert real images of animal faces to different latent spaces, we examine both encoders and latent optimization based methods. We use the pSp encoder (Richardson et al., 2021) as backbone and modify it to embed into W. 2, and Z+ (Song et al., 2021) spaces. For the W4 space, we use e4c (Tov et al., 2021), which also uses pSp (Richardson et al., 2021) as backbone. For optimization based inversion, 2020b) to E or Z+ space (two rightmost columns). All of the inversion methods yield reasonably faithful reconstructions, with occasional artifacts in the Z and Z +oo reconstructions. Note that, as we show below, that better reconstruction does not necessarily yield the best image translation.
modify the optimization code from StyleGAN2 (Karras al.,
24

---

Published as a conference paper at ICLR 2022
Original
W
W+
2
2+
Zopt
2 2+ opt
Figure 17: Comparison of 121 results (dog2wild in the top four rows, cat2dog in the four bottom ones) for the different inversions shown in Figure 16. The color palette appears to be wrong for both W+ and W encoding, especially for the dog wildlife translation. This is not surprising. since the mapping function changes during fine tuning (see Figure 1), affecting the color palette, and inverting into the W or W+ spaces ignores the difference between the mapping functions the parent and child. Translations via Z+ or oop inversion also suffer from occasional color artifacts (mainly in the dog2wild examples). Translations via either Z or Zopt provide satisfactory results. We prefer Zop because it typically yields more vivid color palette, while slightly better capturing the characteristics of the source images (especially the dog2wild examples).
25

---

Published as a conference paper at ICLR 2022
Original
W
W+
Wapt
W
Zopt
Z opt
Figure 18: To invert real images of human faces to different latent spaces, we examine both encoders and latent optimization based methods. We use the pSp encoder (Richardson et al., 2021) backbone and modify to embed into W space. For the W4 space, we use e4e (Tov et al., 2021). which also uses pSp (Richardson et al., 2021) backbone. We also experimented with using the pSp encoder to Z, and Z+ (Song et al., 2021) spaces, but training does not converge and results are unrealistic. For optimization-hased inversion, (Karras et al., 2020b) to W, W+, 2 or Z+ spaces. terms reconstruction quality alone, W+ typically yields the best inversions; however, as we show below, better reconstruction does necessarily yield the best image translation.
modify the optimization code from StyleGAN2
26

---

Published as a conference paper at ICLR 2022
Original
W
W+
Wo oppt
W W+opt opt
opt
2 opt
Figure 19: Comparison of 12I results (for real faces to cartoon-like, using FFHQ parent and Mega child) for the different inversions shown in Figure 18. Translation results via Wopt and W+opt contain strong artifacts. In our subjective opinion, translation via Zopt achieves the most cartoonish look. However, translations via W bear closer resemblance to the input portrait, while still achieving satisfactory cartoonish look. As discussed in the text, this may be attributed to the fact that, for similar domains, the mapping function changes little during fine-tuning, resulting in pointwise alignment the W spaces of the parent and child models.
27

---

Published as a conference paper at ICLR 2022
Zenc
enc
Zopt
Zenc
Z+ene
Zopt
cat2dog
48.8
68.5
34.2
16.1
34.8
7.36
dog2wild
22.1
24.8
10.9
10.9
12.2
2.19
wild2dog
60.0
62.5
34.7
15.5
22.7
5.98
dog2cat 30.4
21.1
17.9
14.5
5.49
3.79
(a) FID
(b) KIDx 10
Table 5: A quantitative comparison of 121 translation via different latent spaces and inversion meth- ods. Based the qualitative results shown in Figure 17, we consider encoder-based inversion for Z and Z+ spaces, and latent optimization method for Z space, to be promising methods and fur- ther examine them using FID and KID scores. Our results indicate that inversion Z using latent optimization achieves the best FID and KID for 12I translation tasks.
Original
Translated
Original
Translated
Original
Translated
Original
Translated
Figure 20: Image Toonification using our Zopt method.
Original
Translated
Original
Translated
Original
Translated
Original
Translated
WAITIN
Figure 21: Aligned models enable effective image translation between dissimilar domains (human face and dog face). Some interesting analogies emerge in these translations. For example, as the human hair becomes longer, so does the dog's fur, while the dog's s ears change from "candle flame" ears, "bat" ears, and finally to folded ('down-pointing' ears. The fur color mainly determined by the human hair color, and the dog pose mimics that of the human.
28

---

Published as a conference paper at ICLR 2022
Source Reference
3
6
12
15
18
21
23
Reference-based image translation. Given a real dog image as source and a real cat image reference, we aim obtain cat image that keeps the content (mainly pose) from the source and the style (fur texture and color) from the reference. We first invert the input real images latent space of StyleGAN, then take style codes for all layers below n (low resolution) from the source, and style codes for layers above or equal to n (high resolution) from the reference. The layer index r indicated above each column. Thus, index represents the inverted reference, and index 23 represents the translation of the source the target domain (cats), while the other indices correspond to standard style mixing in StyleGAN. We can see that when n is around 6, the images combine the pose of the source with the style of the reference.
Figure 22:
w+ enc
Z+ enc
Z enc
z.opts
w+ enc
Z+ enc
enc
Z. z.opts
dog2cat
10.3
11.2
13.7
9.22
4.87
4.85
6.56
3.43
wild2dog
44.5
37.6
36.6
27.4
27.2
21.1
18.8
14.8
cat2dog
42.1
44.3
40.7
30.4
27.6
28.2
27.0
17.0
dog2wild 37.9
28.3
18.5
9.65
21.5
16.2
12.5
3.64
(a) FID
(b) KIDx10
Table 6: A quantitative comparison of reference-based image translation using different inversion methods and latent spaces. It may be seen that the latent optimization method for Z achieves the best FID and KID for such translation tasks.
29

---

Published as a conference paper at ICLR 2022
Source
Reference
W++
2+
Zopt
Figure 23: A qualitative comparison of siereene-bbasd image translation for different methods and spaces. Since here the colors are determined by the higher layers of the generator, whose style parameters come from the inversion of the reference image. the translation via W4 does not suffer from color palette issues. Thus, both translations via W4 and via Zopt look satisfactory.
30

---

Published as a conference paper at ICLR 2022
f =
f1 = 0.2
f = 0.4
4 = 0.6
f = 0.8
0
ta
ta = 0.2
ta = 0.4
ba = 0.6
0.8
II
Figure 24: Given a pair of real images from domain A (top-left) and B (bottom-right), we smoothly transition between them by interpolating their latent codes W+, as well as model weights. I the interpolation coefficient for the latent codes, while to is the coefficient for the model weights. In the same column (fixed f1), we obtain a smooth transition between the domains (different species, but the same pose and fur color). In the same row (fixed f.2). we have smooth transition inside the same domain (same species, varying pose and fur color). Any trajectory between the top-left and bottom-right corners yields smooth morph sequence between two input images. See the accompa- nying video, which progresses along the diagonal t = l2.
31

---

Published as a conference paper at ICLR 2022
f1 = 0.2
f = 0.4
4 = 0.6
= 0.8
ta 0.2
0.4
b
0.6
0,5
S
|I
Figure 25: Given a pair of real images from domain A (top-left) and B (bottom-right), we smoothly transition between them by interpolating their latent codes W+, as well as model weights. I the interpolation coefficient for the latent codes, while t2 is the coefficient for the model weights. In the same column (fixed f1), we obtain a smooth transition between the domains (different species, but the same pose and fur color). In the same row (fixed f.2), we have smooth transition inside the same domain (same species, varying pose and fur color). Any trajectory between the top-left and bottom-right corners yields smooth morph sequence between two input images. See the accompa- nying video, which progresses along the diagonal t = l2.
32

---

Published as a conference paper at ICLR 2022
f
f1 = 0.2
f1 = 0.4
f = 0.6
4 = 0.8
f1
0.2
= 0.4
0.5
0.3
II
S
II
Figure 26: Given pair of real images from domain A (top-left) and B (bottom-right), we smoothly transition between them by interpolating their latent codes W+, as well as model weights. I the interpolation coefficient for the latent codes, while to is the coefficient for the model weights. In the same column (fixed t1), we obtain a smooth transition between the domains (different species, but the same pose and similar fur/hair color). In the same row (fixed fz2), have smooth transition inside the same domain (same species, varying pose and color). Any trajectory between the top- left and bottom-right corners yields smooth morph sequence between two input images. See the accompanying video, which progresses along the diagonal t 12.
33

---

Published as a conference paper at ICLR 2022
f1 = 0.2
f = 0.4
4 = 0.6
= 0.8
0
f2
0.2
ta =
0.4
by =
0.6
0.8
S
|I
Figure 27: Given pair of real images from domain A (top-left) and B (bottom-right), smoothly transition between them by interpolating their latent codes W+, as well as model weights. I the interpolation coefficient for the latent codes, while to is the coefficient for the model weights. In the same column (fixed t1), we obtain a smooth transition between the domains (different species, but the same pose and similar fur/hair color). In the same row (fixed fz2), have smooth transition inside the same domain (same species, varying pose and color). Any trajectory between the top- left and bottom-right corners yields smooth morph sequence between two input images. See the accompanying video, which progresses along the diagonal t 12.
34

---

Published as a conference paper at ICLR 2022
-30
-20
-10
10
30
Figure 28: Demonstration of our zero-shot dog yaw regression model. The images are from AFHQ dog dataset, split into several bins (rows), based on the regressed yaw values. The images shown are randomly picked from each bin (no cherry picking). The estimated yaw values capture the correct tendency (right facing to left facing), and in most cases appear to be close the actual yaw degree.
-20
-10
10
Figure 29: Demonstration of our zero-shot yaw regression model. The images from AFHQ cat dataset, split into several bins (rows), based on the regressed yaw values. The images shown are randomly picked from each bin (no cherry picking). The estimated yaw values capture the correct tendency (right facing to left facing), and in most cases appear to be close to the actual yaw degree.
35

---

Published as a conference paper at ICLR 2022
Original
Before FT
Smile
Gaze
Blond Hair
Lipstick
parent 1024
child 512
child 256
6.501
9.409
12.479
15.45
Figure 30: Starting from a pretrained StyleGAN2 model for FFHQ 1024 x 1024 resolution parent, we use its weights to initialize models for 512 x 512 or 256 256 resolution. Before fine tuning (FT), it only generates low contrast images. After fine tuning ("Original" column), similar images with the same attributes (identity, hair length, gender, etc.) as parent model are generated given the same code E Z. Note that the generated images are not pixel-wise identical, but the different style channels retain their semantic function, as demonstrated by the four rightmost columns.
36

---

Published as a conference paper at ICLR 2022
256
128
64
32
16
8
B
Layer Swapping AB
A
4
8
16
32
64
128
256
B
Layer Swapping: BA
0.125
0.25
0.375
0.5
0.625
0.75
0.875
B
Weight Interpolation
256
128
64
32
16
8
B
Layer Swapping: AB
A
4
8
16
32
64
128
256
B
Layer Swapping:
0.125
0.25
0.375
0.5
0.625
0.75
0.875
B
Weight Interpolation lation
256
128
64
32
16
8
B
Layer Swagyingg: AB
A
4
8
16
32
64
128
256
B
Layer Swapping:
0.125
0.25
0.375
0.5
0.625
0.75
0.875
B
Weight Interpolation
Figure 31: A comparison between layer swapping and model weight interpolation. We demonstrate transitioning between FFHQ and Mega using three different ways. Layer swapping:AB means using a hybrid model whose low resolution layers come from model A, and high resolution layers from model B, while Layer swapping: BA means the opposite roles (low from B, high from A). The reso- lution at which the switching occurs shown above each result. The swapping resolution used by Toonify (Pinkney & Adler, 2020) is either 16 x 16 32 32. Weight interpolation instead linearly interpolates the weights of all layers between model A and B. The interpolation ratio is shown shown above each result.
37

---

Published as a conference paper at ICLR 2022
256
128
64
32
16
8
B
Layer Swapping:AR
4
16
32
64
128
256
B
Layer Swapping BA
0.125
0.25
0.375
0.5
0.625
0.75
0.875
B
Weight Interpolation
256
128
64
32
16
8
B
Layer Swapping:AB
4
16
32
64
128
256
Layer Swapping: BA
0. 1.5
0.25
0.375
0.5
0.625
0.75
0.875
B
Weight Interpolation
256
128
64
32
16
8
4
B
Layer Swappingg AB
4
16
32
64
128
256
Layer Swapping: BA
A
0.125
0.25
0,375
0.5
0.625
0.75
0.875
B
Weight Interpolation ptoiioooon
Figure 32: A comparison between layer swapping and model weight interpolation. Here we demon- strate transitioning between AFHQ dog and cat. Refer to Figure 31 for more details.
38

---

Published as a conference paper at ICLR 2022
A
256
128
64
32
16
8
4
B
Layer Swapping:AR
4
16
32
64
128
256
B
Layer Swappingg BA
A
0.125
0.25
0.375
0.5
0.625
0.75
0.875
B
Weights Interpolation
A
256
128
64
32
16
8
B
Layer Swapping
4
16
32
64
128
256
B
Layer Swapping: BA
0.125
0.25
0.375
0.5
0.625
0.75
0.875
B
Weights Interpotatin
256
128
64
32
16
8
4
B
Layer Swapping AB
4
16
32
64
128
256
Layer Swapping: BA
A
0.125
0.25
0.375
0.5
0.625
0.75
0.875
B
Weights Interpolation
Figure 33: A comparison between layer swapping and model weight interpolation. Here we demon- strate transitioning between FFHQ and AFHQ dog. Refer to Figure for more details.
39

---

Published as a conference paper at ICLR 2022
FFHQ
Church
gFFHQ
FFHQ
Church
gFFHQ
Figure 34: Generative image translation from parent FFHQ model to child LSUN church model and grandchild FFHQ using the same latent code Since the domain gap between FFHQ and LSUN church is too large, we can barely see any correspondence. But the parent FFHQ model and grandchild FFHQ models generate faces with highly similar attributes and identity. This implies that knowledge that was not transferred from task A (FFHQ generation) to task B (LSUN church generation), is only hidden in the latter model's 's latent space, rather than forgotten.
40

---

Published as a conference paper at ICLR 2022
Original
Beard
Black Hair
Figure 35: Applying the "Beard" and "Black hair" manipulation directions from parent FFHQ model to child LSUN church model. The manipulation directions are discovered by StyleCLIP (Patashnik et al., 2021). Most manipulation directions from the FFHQ parent do not change anything in the child church model. Surprisingly, the beard direction from FFHQ appear to control the amount of trees in the church model to some extent, and the black hair direction from FFHQ makes the church building darker in the child model.
41

---

Published as a conference paper at ICLR 2022
Original
Smile
Gaze
Blond Hair
Lipstick
Parent
Grandchild
Parent
Grandchild
Parent
Grandehild
Parent
Grandchild
6. 501
9.409
12.479
15.45
Figure 36: Semantic alignment between parent and grandchild model. We first train a StyleGAN model on FFHQ (parent), then fine tune LSUN church (child), and finally fine tune back to FFHQ (grandchild). We can see that the same channel still controls the same attribute between parent and grandchild model. Interestingly, channel 15 5-45 controls lipstick in the parent model, but makes the face slightly pink in the grandchild model. Although the exact function has changed after fine tuning, it is still semantically related the original function.
42

---

Published as a conference paper at ICLR 2022
JUJJJ
Figure 37: We demonstrate the ability of our method to perform 12I tasks that only change texture, while preserving the structure. Although this dataset has paired edge maps and shoe images, the pairing information is not used by our method. We slightly blur the edge maps to make the images more continuous. We first train StyleGAN model on the shoes dataset (parent), then fine tune on edge maps dataset (child). Since edge maps mostly represent the structure of objects, and do not contain color or texture, we train an e4e encoder W+ from whose output we only use the parts that control generator resolutions below 32 32 (same as was done for multi-modes image translation). The parts that control higher-resolution layers are sampled, yielding multiple possible shoe images (sharing the same structure) for each edge map.
43

---

Published as a conference paper at ICLR 2022
Mega
Metface
Dog
Cat
Wild
Unrelated FFHQ
LI in W
0.033
0.057
0.162
0.172
0.141
0.391
Table 7: Average I.1 distance between w W vectors mapped from the same latent code E for different pairs of models. Using a pretrained FFHQ model as parent, it fine-tuned on different datasets separately. We sample 100K random vectors and compute the corresponding for each model. The mean change (per coordinate of w) is reported for each child model. It may be clearly seen that in models fine-tuned to nearby domains (Mega, Metface) the change w is much smaller than to more distant domains (Dog, Cat, Wild), and order of magnitude smaller than the difference another FFHQ model, trained independently. These results quantitatively demonstrate that the change in the mapping function is very small for similar domains, larger for more distant domains, even for distant domains the mapping functions are more closely related than those of two separately trained models.
44


Reviews:
[{"review_id": "r5jhnw3S5o9", "reviewer": "Reviewer_fnkZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n+ There are many works using aligned StyleGAN models but without deep analyses on the properties. This paper provides an in-depth study of the properties, which provides some insights on the aligned models and might inspire following more complex researches. \n+ The organization of the paper is good and the analyses are comprehensive. Several observations are studied with quantitative/visual analyses to prove key properties, followed by corresponding applications utilizing the properties, making the paper easy to follow and concrete.\n\n**Weaknesses:**\n+ While the part of the property study is comprehensive and interesting and provides insights, the part of the application is less insightful. The application involves the image translation, image morphing and zero-shot vision tasks. \n  - Image translation and image morphing are intuitive and well-studied in previous studies, as also pointed out in the paper, which might provide limited insights. Image morphing by interpolating aligned models has been studies in `[R1]`.\n  - The zero-shot vision tasks involve utilizing the label of the parent domain to edit/classify the child domain, which gives some novel ideas but alone are not very thorough to me.  \n  - I think it will be valuable to investigate the applications in terms of the specific properties. For example, only the conv parts of network change greatly for close domains, which supports translations. Can we just finetune the conv part and fix all other parts for better translations? (which is also discussed in `[R3]`) The paper discusses that the latent semantics are hidden rather than forgotten during finetune. It will be valuable to also discuss the potential application of this property.\n\n**Some small issues:**\n+ In Fig. 14, the authors demonstrate the semantic alignment between the human face and the church. However, only the church images are shown. The corresponding face images using the same latent z are suggested to be added to help the readers better find their correspondences.\n+ In Sec. 4.2, the paper claims that the layer swapping performs the transition as a series of discrete steps, rather than continuously. In the implementation of Pinkney `[R2]`, the layers can be smoothly swapped through the parameter ` blend_width` instead of hard swap, which might be able to perform the transition continuously.\n\n`[R1] 2019 CVPR Deep Network Interpolation for Continuous Imagery Effect Transition`\n\n`[R2] https://github.com/justinpinkney/toonify/blob/master/StyleGAN-blending-example.ipynb`\n\n`[R3] 2021 TMM Unsupervised Image-to-Image Translation via Pre-trained StyleGAN2 Network`\n", "summary_of_the_paper": "This paper provides an in-depth study of the properties and applications of the semantic alignment between the original parent StyleGAN model and its finetuned child model on another dataset. Specifically, the paper empirically demonstrates the semantical alignment of the two models. Then, based on the properties, the paper solves serval tasks like image translation, image morphing, zero-shot image editing and attribute classification. ", "summary_of_the_review": "This paper investigates an important topic of the semantic alignment of the parent model and finetuned child model. Despite the application part could be further enriched to better match the analyzed properties, the comprehensive analyses on the properties provides many insights and might arouse following researches. Therefore, I am positive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fzXOqXvBfJ8", "reviewer": "Reviewer_x6dT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The shared latent space for \"child\" and \"parent\" networks is interesting, and matches the learning representation goal. \n\n2. The overall idea and the proposed fine-tuning method towards analyzing the shared latent space is reasonable. The StyleGAN-based architecture is a powerful structure, which consists of a mapping network, an affine translation, and a generator. This is exactly suitable for exploring feature attributes at different levels.  \n\n3. The paper provides many interesting visual results, such as in Figures 1, 2, 3, and 4, to show the effectiveness of shared information in different networks. These interesting conclusions may contribute to the generation community to design better generator works. \n\n4. The wide applications are being met by subtly using the shared semantic information between different domains. \n\nCons:\n\nWhile the impressive results are achieved, I believe some parts need more clarification (even after considering the supplementary material):\n\n1. The key limitation of this work is the novelty. If I understand correctly, the authors just fine-tuning the different parts of the StyleGAN. This is very similar to the existing works AgileGAN and the shared attributes have also been mentioned in prior MUNIT. It takes me a hard time to buy the novelty of models, techniques, and theoretical insights. \n\n2. The authors claimed the \"two W spaces are point-wise aligned\" on page 4. While they showed some shared attributes in Figure 2 and Table 1, and they also demonstrated some attributes are hidden, I do not believe these attributes are \"point-wise\" aligned. Is this conclusion correct? How to demonstrate it?\n\n3. A quantitative and qualitative comparison with the latest CUT and F-LSeSim is given in Figure 5. While the improved results are provided, the original CUT and F-LSeSim work for aligned shapes. Why not provide some aligned examples, for example, horse2zebra, night2day, apple2orange, every two domains share a similar shape, but different appearances? This would be more robust to demonstrate the effectiveness of aligned style on both aligned examples (Figure 5) and unaligned examples (Figure 6).\n\n4. In Figure 6, the StarGAN2 seems to be able to provide more consistent results to the source image, such as the smiling mouth in the first row, and the same head poses for the third row. Except for the FID score, it seems the StratGAN2 provides better translated results (better shared content, such as pose and expression). \n\n5. The authors show abundant results in Figures 23, 24, 25, and 26. Similar results are also being met in the StyleGAN-based method. I do not fully understand the key challenge in such a situation.  ", "summary_of_the_paper": "This work is about the task of transfer learning to tame a new \"child\" network using a pre-trained \"parent\" network. While the model and fine-tuning technology lack novelty, the shared semantic information in the generation network are interesting. Finally, the authors applied the proposed aligned model to multiple tasks, including image-to-image translation, cross-domain image morphing, and zero-shot classification and regression. The impressive results with shared semantic information are achieved. ", "summary_of_the_review": "As can be inferred from the balance in strengths/weaknesses, my preliminary rating for this submission is borderline accept. While the interesting results are shown in the paper and supplementary material, the key contribution for models, techniques or theoretical insights is unclear. Furthermore, some comparisons seem unfair in the paper. The authors should clearly interpret them during the rebuttal. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OsMFaWtSNHb", "reviewer": "Reviewer_nPWN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Weaknesses\n\n\tThe main weakness for this paper is likely the limited novelty in the technical contributions, given that transfer learning is a fundamental, established technique. However, this paper is more of a study of this technique, applied to the particular domain of image GANs, and how their editability changes during the process.\n\n\tThe qualitative results do seem high quality, but it is sometimes quite difficult to see where the resemblance to the reference images are (especially with Figure 6, for the dogs dataset). Perhaps the translation is not focused strongly enough on the reference images. Despite more artefacts being present, I'd say OverLORD images are a better transfer, for both cats and dogs.\n\n\n\nStrengths\n\n\tThe primary contribution of the paper is the extensive experimentation, around transfer learning in GANs, and they perform this very well. Their approach is very well documented, is very thorough, and convincing. The supplementary videos are helpful in further displaying their results.\n\n\n\tAlthough there are aspects that are both better and worse, the samples from their models compared to other models, are mostly good. The benefits arise mostly from the editability of the output, given their findings that the channels maintain their editing effects from previous datasets with more labels.", "summary_of_the_paper": "The paper undertakes extensive experimentation in the adaptability of latent space modifications from a base model, and a fine-tuned model for a secondary dataset. They demonstrate which areas of the models change most, and which information remains trained in the parameters, despite re-training. They go on to use their findings in downstream experiments, reaching state-of-the-art quality.", "summary_of_the_review": "The paper presents some novel analysis in a popular field of research, with implications that can help drive the field further, with minimal changes.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "DyuxSQNQhO", "reviewer": "Reviewer_2pb8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##########################################################################\n\nPros:  \n1. The paper performs the first detailed exploration of model alignment based on StyleGANs. Considering the wide-range applications of StyleGAN, I think the analysis is timely, insightful, and interesting!\n2. To analyze the model alignment during transfer learning from the source domain to similar/distant target domains, the paper uses weight resetting and quantitative channel alignment measurement. These techniques probably inspire others to analyze their GAN models.\n3. The paper demonstrates impressive and promising experimental results on image translation and image morphing between different domains. Besides, it demonstrates the benefits of the shared latent space by zero-shot recognition/regression. \n \n##########################################################################\n\nCons: \n1. I’m confused about the example of “Age” in Figure 2. Why does the “Age” change the identity, background, and color largely? It looks like another totally different face.\n2. Many details of the results are missing. Take Figure 2 as an example, does the semantic control apply in overlap channels (e.g., 15 for eyebrow in FFHQ2MetFace) or all the distinct channels of the child model (e.g., eyebrow 35)? Are they in the same layer? How do the authors transfer the pose control from face to church and bedrooms in Figure 14? Please check all the results and provide the necessary details. \n3. Does the claim of “the hidden latent semantics” hold for the case of (parent FFHQ, child Church, grandchild FFHQ)?\n4. I think most of the successful results can be attributed to the well-structured properties of the human face, animal face, and mega face. It would be helpful to provide further discussion on the “real distant domain”, e.g., face VS church. For example, is it helpful to transfer learning from face to church? Are there any other shared semantics between face and church except pose?\n", "summary_of_the_paper": " \nThe paper provides interesting analysis and leveraging of GAN’s model alignment (i.e., transfer learning). Without custom architectures and losses, it demonstrates impressive performance in a diverse set of tasks (image translation and image morphing). It also demonstrates promising results for zero-shot image recognition by leveraging the shared latent space of aligned models. \n", "summary_of_the_review": " \nOverall, I vote for accepting. I like the analysis of aligned GANs’ models. The paper also demonstrates impressive experimental results by leveraging the properties of aligned models for image translation, image morphing, and zero-shot classification. My major concern is about the clarity of the paper and some missing details. Hopefully, the authors can address my concern in the rebuttal period. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an interesting analysis of aligned GAN models. The paper shows that when a model is obtained (fine-tuned) from another, then the corresponding hidden semantic spaces are aligned. The paper uses this property to show that without any additional architecture or training, the models can perform diverse tasks such as image translation and morphing. The paper also demonstrates that zero-shot tasks can be performed by learning in the parent domain and transferring to the child domain.

All reviewers agree that the paper presents an interesting analysis and findings and will make a valuable contribution to the field. The reviewers raised some particular concerns, which were addressed by the authors in their response.
Decision: Accept (Oral)