Year: 2021

Abstract:
Conditional image synthesis is an integral part of many X2I translation systems, including image-to-image, text-to-image and audio-to-image translation systems. Training these large systems generally requires huge amounts of training data. Therefore, we investigate knowledge distillation to transfer knowledge from a high-quality unconditioned generative model (e.g., StyleGAN) to a conditioned synthetic image generation modules in a variety of systems. To initialize the conditional and reference branch (from a unconditional GAN) we exploit the style mixing characteristics of high-quality GANs to generate an infinite supply of style-mixed triplets to perform the knowledge distillation. Extensive experimental results in a number of image generation tasks (i.e., image-to-image, semantic segmentation-to-image, text-to-image and audio-to-image) demonstrate qualitatively and quantitatively that our method successfully transfers knowledge to the synthetic image generation modules, resulting in more realistic images than previous methods as confirmed by a significant drop in the FID.

heading: 1 INTRODUCTION
text: Conditional image synthesis, also X2I translation, maps from an input domain (e.g. text, audio, segmentation maps, etc.) to the image domain. Benefiting from GANs (Goodfellow et al., 2014) and its follow-up improved versions (Gulrajani et al., 2017; Kang & Park, 2020; Salimans et al., 2016), they obtain remarkable performance on a wide variety of image synthesis tasks: image to image (I2I) (Lee et al., 2018; Zhu et al., 2017), audio to image (Chen et al., 2017; Wang et al., 2020a), text to image (Hu et al., 2021; Li et al., 2019; 2020; Radford et al., 2021; Zhang et al., 2017a) and semantic segmentation map to image (Isola et al., 2017; Wang et al., 2018). Despite impressive leaps forward for a variety of image synthesis tasks, there are still important challenges. Specifically, to obtain good results, existing works rely on large labelled datasets. Labeling these datasets is both laborious and time-consuming, considerably reducing the practical impact of these methods. It is noteworthy to observe that many of these models (Zhang et al., 2017a) apply transfer learning to the text and audio encoders (e.g. using a pretrained LSTM (Reed et al., 2016) model for text and pretrained GRU (Merkx et al., 2019) model for audio), however they train the image synthesis decoder from scratch. This happens because there are no established methods to transfer pretrained GANs to conditional image decoders; an omission which we aim to address in this paper.
In this paper, we investigate knowledge transfer for a variety of conditional image synthesis tasks. Traditional knowledge transfer for conditional image synthesis is often not possible, because there might not be a pretrained network available for the desired translation task (e.g. at the moment no high-quality pretrained network for segmentation map-to-image translation is available). It would therefore be preferable if the wide variety of high-quality GANs available for image generation could be exploited for X2I. Recent works (Wang et al., 2021; 2020b) leveraged a pretrained GAN to initialize an I2I translation model, managing to transfer knowledge to different image synthesis tasks. These methods, however, suffer from three problems: (I) They can only be used for I2I translation and do not generalize to other conditional image synthesis tasks. (II) They are GAN architecture-specific approaches, requiring the GAN architecture within the X2I system to be exactly the same as that of the pretrained GAN. This limits transfer to current state-of-the-art X2I systems for which no similar GAN architecture exists (like for example StarGANv2 (Choi et al., 2020) for I2I). (III) X2I systems are based on a conditional GAN, however existing methods for knowledge transfer for I2I do not initialize the conditional branch during the transfer, and therefore this has to
be learned from scratch during the finetuning on the target dataset. Learning this on small target datasets can be problematic.
To address the aforementioned problems, we propose several improvements for knowledge transfer to X2I systems: (I) we are the first to investigate knowledge transfer for X2I translation. Therefore, we propose a novel, unified transfer learning method, which can be used for varying kinds of conditional image synthesis tasks (Figure 1) which is based on generated images and therefore does not require any real data. (II) The student generator does not need to have the same architecture as the pretrained GAN. Therefore, we can use well-devised specific image synthesis architectures (e.g., SPADE (Park et al., 2019) and StarGANv2 (Choi et al., 2020)) by distilling knowledge from the pretrained teacher GAN (e.g., StyleGAN) to the task-specific student GAN. (III) We use the style mixing characteristic of StyleGAN to create style-mixed triplet data, which are used to transfer the knowledge efficiently to both I2I and X2I translation models. Furthermore, we propose a semantic diversity loss based on the style-mixed triplet, which contributes to learn the semantic information of the output image.
We perform experiments on a wide variety of image synthesis tasks, including text-to-image, audioto-image, segmentation map-to-image and I2I translations. We demonstrate the efficiency of the proposed knowledge distillation method, providing qualitative and quantitative results. We prove that the single pretrained GAN model can be universally used in varying specific task model. Additionally, leveraging the style mixing character of StyleGAN, further improves I2I translation performance.

heading: 2 RELATED WORK
text: GAN-based Conditional Image Synthesis. Benefiting from the advances in GANs and its variants in recent years, conditional image synthesis (also called X2I translation) research has developed rapidly. Two typical approaches have been investigated for GAN-based image synthesis, namely, unsupervised (Kim et al., 2017; Yi et al., 2017) and supervised image generation (Park et al., 2019; Zhang et al., 2017b). The latter inputs conditional information (e.g. text, audio, image, segmentation map etc.) to synthesize images which contain the corresponding semantic information (i.e. the conditional information). Specifically, text-to-image translation (Hu et al., 2021; Li et al., 2020; Zhang et al., 2017a) aims to synthesize high-realistic images which are semantically consistent with the text descriptions. Recent work (Hu et al., 2021) introduces semantic-spatial batch normalization to better exploit the text information. Similar to text-to-image translation, both audio-to-image translation (Chen et al., 2017; Wang et al., 2020a) and segmentation map-to-image translation (Bau et al., 2020; Park et al., 2019) aim to learn a mapping from the audio/segmentation map to the output image. Different to the above image synthesis tasks, image-to-image translation (Park et al., 2020; Zhu et al., 2017) performs projection from the source to the target image domain. In this paper, we explore transfer learning from GANs to a variation of conditional image synthesis tasks.
Transfer learning. A considerable research effort has investigated transferring knowledge for both discriminative (Donahue et al., 2014; Hinton et al., 2014) and generative tasks (Noguchi & Harada, 2019; Zhao et al., 2020). There also exist several approaches (Goetschalckx et al., 2019; Jahanian et al., 2020) which focus on the image manipulation based on the pretrained GAN. Given a target
semantic attribute they aim to manipulate the output image of a pretrained GAN. However, these methods do not focus on transfer learning for target data. Some other methods (Abdal et al., 2019; Zhu et al., 2020a) reverse the given image into the input latent space of the pretrained GAN (e.g., StyleGAN), and manage to restructure the target image by optimization of the latent representation. Recent work (Shocher et al., 2020; Wang et al., 2021; 2020b) performed knowledge transfer from a pretrained classification model (e.g., VGG (Simonyan & Zisserman, 2014)) or the discriminator (BigGAN (Brock et al., 2019)) for I2I translation. However, both DeepI2I (Wang et al., 2020b) and TransferI2I (Wang et al., 2021) require that the GAN architecture is identical with the generator used in the I2I architecture. As a consequence, these methods cannot be applied to well-designed I2I translation architectures (like starGANv2 (Choi et al., 2020)) since they do not use a standard GAN architecture. The proposed method could address these problems.

heading: 3 KNOWLEDGE TRANSFER FOR X2I
text: Problem setting. Our goal is to transfer knowledge from a pretrained high-quality unconditional GAN to an X2I translation system for the case when training data is limited. The proposed method consist of two stages. In the first stage we perform the data-free knowledge transfer method explained in Sec. 3.2 and 3.3. This stage does not require any target data. In the second stage, we apply a standard finetuning of this distilled model on the target dataset. Distillation can be used to perform knowledge transfer to other architectures, however, there is little target data available. Fortunately, the pretrained GAN can generate infinite data for data distillation, meaning that we do not require access to any real data (i.e., data-free). Also, we transfer knowledge from an unconditional GAN (e.g., StyleGAN) to a conditional GAN (e.g., I2I translation); requiring us to propose techniques to initialize the conditional branch. Exploiting the style-mixed characteristic (Sec. 3.1) of StyleGAN we propose two solutions: one for I2I (Sec. 3.2) and another one for X2I (Sec. 3.3). Our method consists of two stages:(1) transfer leaning without any real data (i.e., style-mixed triplets in Sec. 3.1, I2I translation in Sec. 3.2 and X2I translation in Sec. 3.3 ) and (2) finetunning with the real target data.
3.1 STYLE-MIXED TRIPLETS
Benefiting from the style mixing ((Figure 2 (a))) characteristic of StyleGAN, we are able to create an infinite amount of triplet images from any of the two domains. Let zt ∈ RZ indicates the input noise of the pretrained generator GT (teacher). given the input noises z1t and z 2 t , the mapping network MT of the StyleGAN generator encodes them to a style vector s1t = MT (z 1 t ) and s 2 t = MT (z 2 t ). We define GT (z) = G′T (MT (z)) where G ′ T takes styles vectors as an input to each of its layers. We further feed these style vectors to the teacher generator G′T to output x 1 t = G ′ T (s 1 t ) and x 2 t = G′T (s 2 t ) (where x 1 t ,x 2 t ∈ X and image domain X = RH×W×3) respectively. Here the function Φ(s1t , s 2 t ) selects to forward s 1 t to the layers of G ′ T important for the content of the generated image (the bottom layers), and s2t to those important for the style (the top layers) 1. Interestingly, a new image yt = G′T (Φ(s 1 t , s 2 t )) (where yt ∈ Y and image domain Y = RH×W×3) contains mixed information with respect to content and style of both inputs x1t and x 2 t . Based on this observation above, we propose to leverage these style-mixed triplets (x1t , x 2 t ,yt) to perform distillation for the I2I and X2I translation models without the need of any real triplet training data. 2

heading: 3.2 DATA-FREE KNOWLEDGE DISTILLATION FOR I2I TRANSLATION
text: We aim to leverage the style-mixed triplets to perform knowledge transfer for the I2I model. Let xt ∈ X and yt ∈ Y indicate the two domains. We expect to map an image xt from domain X into another sample yt which mimics to Y . Note we never get access to any real data. Figure 2 (b) shows the knowledge distillation framework for I2I translation. Our framework is composed of six neural networks: a pretrained teacher GAN (consisting of GT and DT ), and an I2I model with a student generator GS , a student discriminator DS , a student encoder ES that extracts the content information from the input image x1t , and a reference encoder ER that extracts the style information from the reference image x2t . Similarly to StarGANv2, we investigate two cases of I2I translation: the style representation is from noise (i.e., latent-guided synthesis) and from the reference image (i.e., reference-guided synthesis). Here we mainly focus on reference-guided synthesis, since latent-guided synthesis can be adapted easily by replacing the reference encoder with several fully connection layers, which take the noise as input.
Both the teacher discriminator DT and the student encoder ES take the input image x1t as input, extracting the hierarchical representation { DT (x 1 t )l } and { ES(x 1 t )l }
. Here DT (x1t )l represents the output of the lth ResBlock 3. We encourage them to be aligned via the feature-based knowledge distillation loss:
Lekdl = ∑ l γl ∥∥DT (x1t )l − ES(x1t )l∥∥1 (1)
where parameters γl are scalars to balance the terms. We set them to 0.1. When the teacher and student dimensions are not the same, an additional layer can be introduced to map the teacher output to the desired dimensions (similar as the hint distillation used in (Romero et al., 2015)).
Next, we take the image x2t for both the pretrained discriminator DT and the reference encoder ER, extracting the hierarchical representation { DT (x 2 t )l } and { ER(x 2 t )l }
, both of which we encourage to be aligned via the feature-based knowledge distillation loss Lrkdl = ∑ l τl ∥∥DT (x2t )l − ER(x2t )l∥∥1 (2)
where parameters τl are scalars which balance the terms. We set them to 0.1.
Finally, taking the output of the student encoder ES(x1t ) and the output of the reference encoder ER(x2t ), for the student generator GS we obtain the hierarchical representation{ GS(ES(x 1 t ), ER(x 2 t ))l } and the output image ŷs = GS(ES(x1t ), ER(x 2 t )), both of which we encourage to align with the hierarchical representation {G′T (Φ(s1t , s2t ))l}. We also align the output image yt = G′T (Φ(s 1 t , s 2 t )) with yt:
Lgkdl = ∑ l αl ∥∥G′T (Φ(s1t , s2t ))l −GS(ES(x1t ), ER(x2t ))l∥∥1 + β ‖yt − ŷs‖1 . (3)
1We use the the style vector s1t in the first four layers of the generator, and do s2t in the following layers. 2In practice, during experiments we generate style-mixed triplets online in the minibatch. 3After each ResBlock the feature resolution is half of the previous one in both encoder and discriminator,
and two times in generator.
where parameters αl and β are scalars to balance the terms. We set them to 0.1.
We also employ the following GAN loss (Goodfellow et al., 2014) to optimize this problem:
Ladv = Eyt∼Y [logDS (yt)] + Eŷs∼Y [log(1−DS (ŷs)] , (4)
The full objective function of our model is:
min ES ,GS max DS
λadvLadv + λkdl(Lekdl + Lrkdl + L g kdl) (5)
where both λadv and λkdl are hyper-parameters to balance the importance of each terms. We set them to 1.
In conclusion, to improve knowledge transfer for I2I, we have presented two novel contributions. Firstly, we have shown that the pretrained StyleGAN can be used to generate an unlimited number of style-mixed triplets which we can use to perform the training. We are the first to exploit the stylemixing possibilities of StyleGAN to transfer knowledge to I2I systems. Secondly, we have proposed a distillation approach to transfer knowledge from the GAN to the I2I model. We name this datafree distillation because no real data is required and it is based on generated data. our method can transfer knowledge between generators with different architectures.

heading: 3.3 DATA-FREE KNOWLEDGE DISTILLATION FOR X2I TRANSLATION
text: Here we show how the proposed data-free knowledge distillation can be generalized to X2I translation problems. We will perform the distillation based on a noise input to the student generator. For X2I translation, the conditional information c determines the structural information and the semantic information. For instance, in text2image translation on the birds dataset, we take as an input text descriptor (information c), and expect to generate an image which mimics the shape and the appearance of a real bird. However, during the knowledge transfer, the conditional information c is not available. Therefore, we propose to use the teacher GAN discriminator to extract a semantic encoder of the image that can be used to replace the conditional information c. However, when directly applying this for knowledge distillation, we found that this information can still be ignored by the network (see e.g. Figure 13). We therefore introduce two additional techniques. We again use the style-mixed triplets introduced in the previous section to perform the knowledge transfer and we propose a novel semantic diversity loss to diversify the semantic information of the output when varying the condition c.
As shown in Figure 2(c), we additionally introduce a semantic encoder SE to extract a pseudocondition vector c, which assists in distilling the knowledge from the unconditional GAN GT to the conditional GAN GS . The semantic encoder SE is identical to the teacher discriminator after removing the last fully connection layer, and initialized by the teacher discriminator, which is welloptimized on the input images yt and x1t . Utilizing the style-mixed triplets (Sec. 3.1), we extract teacher output x1t = G ′ T (s 1 t ) and yt = G ′ T (Φ(s 1 t , s 2 t )), and the corresponding hierarchical repre-
sentation { G′T (s 1 t )l } and { G′T (Φ(s 1 t , s 2 t ))l } , respectively. Note that G′T (s 1 t )1 and G ′ T (Φ(s 1 t , s 2 t ))1 are identical, since x1t provides the structural information of yt. Semantic encoder SE extracts the semantic attributes from the input images x1t and yt respectively, termed as cx = SE(x 1 t ) and cy = SE(yt) respectively.
For the student generator GS , we take as input the teacher’s latent representation at the first layer, the output the semantic encoder and the noise. Note, the teacher’s latent representation 4 is summed with the corresponding feature of the student generator. When the input of the semantic encoder SE is x1t , we generate the hierarchical representation { GS(zs, G ′ T (s 1 t )1, cx)l } and final student output x̂1s = GS(zs, G ′ T (s 1 t )1, cx). Similarly, when yt is the input of semantic encoder SE , we
have the hierarchical representation { GS(zs, G ′ T (Φ(s 1 t , s 2 t ))1, cy)l } and final student output ŷs = GS(zs, G ′ T (Φ(s 1 t , s 2 t ))1, cy).
By conditioning the student generator with the teacher representation, the student generator can output a similar image (this would otherwise be impossible). The loss is defined as:
4We ignore the latent representation in the second stage (i.e., fintunning on target domain), which means that the student generator has two inputs: the noise and the condition c.
Lkdl = ∑ l αl ∥∥G′T (s1t )l −GS(zs, G′T (s1t )1, cx)l∥∥1 + β ∥∥x1t − x̂1s∥∥1 +∑
l
αl ∥∥G′T (Φ(s1t , s2t ))l −GS(zs, G′T (Φ(s1t , s2t ))1, cy)l∥∥1 + β ‖yt − ŷs‖1 (6)
where parameters αl and β are scalars to balance the terms.
Yang et al. (2019) proposed a loss to promote diversity of the generated images with respect to changes of the input noise z. Inspired by this work, to address the lack of diversity with respect to the conditional vector, we propose the semantic diversity loss is:
Lsrl = −‖x̂1s − ŷs‖1 = −‖GS(zs, G′T (s1t )1, cx)−GS(zs, G′T (Φ(s1t , s2t ))1, cy)‖1 = −‖GS(zs, G′T (s1t )1, cx)−GS(zs, G′T (s1t )1, cy)‖1
(7)
Other than (Yang et al., 2019), our loss promotes diversity with respect to changes of the conditional vectors (i.e., cx and cy) while we retain the input noise vector zs. Since the conditional vector influences the semantic information (e.g., gender, age and hair style), we call our loss the semantic diversity loss.
The full objective function of our model is:
min GS max DS λadvLadv + λkdlLkdl + λsrlLsrl (8)
where λadv , λkdl and λsrl are hyper-parameters to balance the importance of each term, and Ladv is defined in Eq. 4.
The proposed initialization procedure is not applicable to the case when the conditioning is a onehot spatial map (we found the semantic encoder was not able to predict these). Therefore, for this case, we do not initialize the conditional branch and we set c are 1 in the student generator and 0 in the student discriminator during the distillation. We found that even for these cases the knowledge transfer led to large performance gains (Table 2).

heading: 4 EXPERIMENTS
text: In Sec. 4.1, we firstly investigate knowledge transfer for I2I translation. We explore how the proposed method affects image synthesis by using conditional information c (i.e., audio, text and semantic segmentation) in Sec. 4.2. For the experimental evaluation, we apply our proposed knowledge transfer method explained in the previous section (i.e the first stage) and use this distilled model for the downstream task where we finetuned it on the target data (i.e. the second stage). We use the pretrained StyleGAN optimized on HHFQ human face (Karras et al., 2019) except for semantic segmentation-to-image(SS2image), which is on AFHQ dataset (Choi et al., 2020). More detailed information and visualization results can be found in the Appendix A and C. The code is provided in Supp.Mat.

heading: 4.1 DATA-FREE KNOWLEDGE DISTILLATION FOR I2I TRANSLATION
text: We first evaluate our method for I2I translation with a reference image, named reference-guided I2I translation. Then we verify the proposed method without reference image, called latent-guided I2I translation, in which we use random noise to control the style of the output image. We take stateof-the-art StarGANv2 as the student network. Except for the well known metrics: FID (Heusel et al., 2017) and KID (Bińkowski et al., 2018), we also train a real (RC) and a fake classifier (FC) (Shmelkov et al., 2018) to evaluate the ability to generate class-specific images. RC is trained on real data and evaluated on the generated data and vice versa for FC.
Reference-guided I2I translation. We conduct multi-class I2I translation on three datasets: Animal faces (Liu et al., 2019), Birds (Van Horn et al., 2015) and Foods (Kawano & Yanai, 2014). The Animal faces dataset contains 1,490 images and 149 categories in total, Birds has 48,527 images and 555 classes , Foods consists of 31,395 images and 256 classes. We compare to StarGANv2 (Choi et al., 2020). We also explore a wide variety of configurations for our approach, including: no
knowledge distillation (No KD), knowledge distillation only using L1 distance between the teacher generator and the student generator (KD (w-L1, w/o-adv.)), knowledge distillation only using the adversarial loss (KD (w/o-L1, w-adv.)) and knowledge distillation with both losses (KD (w-L1, w-adv.)). We also ablate the style-mixed triplets. We replace both the reference image x2t and the style-mixed output yt with the input image x1t (indicated by KD* (w-L1, w-adv.)).
Knowledge distillation and style-mixed triplets. Figure 4 presents a comparison between several variants of our method in terms of mean FID (mFID) on three datasets. Taking Animal faces as an example, adding either L1 distance or the adversarial loss improves the I2I translation task in general compared to the I2I model trained from scratch (mFID: 134.7). Finally combining both losses obtains the best score (mFID: 96.2), indicating that the proposed method largely improves I2I translation performance. Without the style-mixed triplets, the model performance largely drops (mFID:104.1), even with both the L1 distance and the adversarial loss. This is probably because the generator fails to perform disentanglement, and the reference encoder does not learn the style information. In conclusion, this experiment validates the effectiveness of the proposed style-mixed triplets.
Results. Table 1 (reference-guided I2I translation) reports results for baseline and our method on Animal faces, Birds and Foods datasets with image resolution 256× 256. StarGANv2 training from scratch obtains lower results (e.g., mFID:134.7 on Animal faces dataset), indicating training I2I model from scratch is challenging when given limited data. Our method achieves a significant advantage (mFID: 96.2 on Animal faces dataset), demonstrating that the proposed knowledge transfer facilitates I2I translation. Figure 3 compares the results of mapping the input to the target image domain with a reference image. Given both the input and reference image our method manages to
generate higher quality images than StarGANv2, that retain the pose of the input and keep a similar style as the reference image. It clearly indicates that our method improves the image synthesis model. Note that the existing approaches which address transfer learning for I2I (Wang et al., 2020b; 2021) cannot be applied to this case, since they require the I2I image synthesis architecture to be the same as the pretrained GAN architecture.
Latent-guided I2I translation. To further evaluate the generality of our method for I2I translation, we validate the proposed method without reference image. We compare to StarGANv2 (Choi et al., 2020), SDIT (Wang et al., 2019), DRIT++ (Lee et al., 2020b), DMIT (Yu et al., 2019), DeepI2I (Wang et al., 2020b) and TransferI2I (Wang et al., 2021). We use random noise to control the style information of output images. As reported in Table 1 (latent-guided I2I translation), all baselines training from scratch (from 4rd to 6th row) suffer a significant disadvantage on small dataset. StarGANv2 obtains the best result (131.2 mFID on Animal faces). However, using knowledge transfer TransferI2I performs significantly better (103.5 mFID on Animal faces), and achieves the best score on Foods dataset. Finally, our method obtains the best results on both Animal faces and Birds datasets, and competitive performance on the Foods dataset. This could be due to the fact that the Foods dataset has more training data, and the importance of transfer learning is negligible. We also evaluate our method with higher resolution images (256×256). As shown in the last two rows of Table 1 (latent-guided I2I translation), we still retain a large advantage comparing to StarGANv2, demonstrating that our method still obtains better performance for higher resolution images. Neither DeepI2I nor TransferI2I is applicable, since there is no the released pretrained model for usage (they are based on the BigGAN). The synthesized images on three datasets are in the Appendix C.

heading: 4.2 DATA-FREE KNOWLEDGE DISTILLATION FOR X2I TRANSLATION
text: Our approach can be applied to a wide variety of conditional image synthesis tasks. Given the small target datasets, we perform image synthesis using conditional information c (i.e., text, audio
and segmentation maps). We explore four variants of our method: knowledge distillation only using L1 distance between the teacher generator and the student generator (Ours (w-L1, w/o-adv.)); knowledge distillation only using the adversarial loss (Ours (w/o-L1, w-adv.)); next we set the condition c=1 in the student generator and 0 in the student discriminator (ours †); finally we remove the semantic diversity loss in Eq. 7 from the final objective in Eq. 8 (ours ‡). More details on training and hyperparameters can be found in Appendix A.
Text-to-image. We evaluate the proposed method on the CUB bird dataset (Welinder et al., 2010). It contains 8,855 training images (150 species) and 2,933 test images (50 species), following 10 text descriptions for each bird. Here we use 10 images per class for training, and verify our method on the test dataset. We compare with SSA-GAN (Hu et al., 2021) which is one of state-of-the-art methods. Results are depicted in Figure 1, Figure 5 (a) and Table 2 (left).
Audio-to-image. We evaluate on the commonly-used dataset: Oxford-102 (Nilsback & Zisserman, 2008), where 82 categories and 10 images per category are selected for training, and 20 categories and 1,155 images for test. Note that we split the training and test categories following S2IGAN (Wang et al., 2020a). Results are depicted in Figure 1, Figure 5 (b) and Table 2 (middle).
Semantic segmentation-to-image. Here we conduct experiments by conditioning on segmentation map on the CeleAMask-HQ (Lee et al., 2020a) dataset containing 30,000 image and segmentation mask pairs. We follow the same train/test split as SEAN (Zhu et al., 2020b). We randomly select 500 pairs of data from the train set for training, and 2,000 pairs for testing. To evaluate our method, we compare to three baselines: SPADE (Park et al., 2019), OASIS (Simonyan et al., 2013) and CoCosNetv2 (Zhou et al., 2021), which are trained from scratch. We report the generator performance using the pretrained StyleGAN optimized on AFHQ (Choi et al., 2020)(animal face dataset). Results are depicted in Figure 1, Figure 5 (c) and Table 2 (right, w/o reference). We also compare to SEAN (Zhu et al., 2020b) which additionally uses a reference image to control the style of the output image, the results are shown in Figure 5 (d) and Table 2 (right, w reference).
As reported in Table 2, our approach outperforms all baselines without using knowledge distillation on several metrics. The semantic diversity loss gets the best performance except for the semanticmap-to-image task. The qualitative results in Figure 1 and 5 indicate that our method can generate high-quality images with limited data due to the benefit of the knowledge transferred from the expert teacher network. Figure 6 shows the output images of the X2I generator after knowledge distillation and shows that we synthesize diverse outputs given different condition c. In conclusion, our method is an efficient, general purpose mechanism for conditional image synthesis with limited data.

heading: 5 CONCLUSIONS
text: Conditional image synthesis tasks require vast amounts of training data. This requirement can be alleviated by transfer learning. Therefore, we proposed to transfer knowledge from a unconditional pretrained GAN model to conditional image synthesis tasks. We perform the distillation with style-mixed triplets that can be automatically computed from high-quality GANs. Our experiments confirmed that the proposed transfer learning method obtains state-of-the-art results and allows to train of high-quality X2I systems from fewer labeled samples.

heading: A ARCHITECTURE AND TRAINING DETAILS
text: The proposed method is implemented in Pytorch (Paszke et al., 2017). We transfer knowledge from StyleGAN to X2I translation. We introduce here the architecture and model details for each task. Note we perform the knowledge distillation on one GPU(Quadro RTX6000) with 24GB VRAM.

heading: A.1 KNOWLEDGE DISTILLATION FOR TEXT-TO-IMAGE TRANSLATION
text: For text-to-image translation the training is composed of two stages: data-free distillation from the unconditional prerained StyleGAN to text-to-image translation model, and finetuning text-to-image translation model using the target data and label.
Model details for the first stage. The student model is the same as SSA-GAN (Hu et al., 2021), which is composed of three sub-networks: a text encoder, a student generator GS , a student discriminator DS . The text encoder is the pretrained one provided by (Xu et al., 2017). The generator takes random noise as input, following one fully connection (100 × 8192) and 7 ResBlocks 5. The input feature size of the first ResBlock is B × 512 × 4 × 4 (B is the batch size). We extract the hierarchical representation from StyleGAN (Karras et al., 2019). The first feature from StyleGAN is B × 512 × 8 × 8, and fed into the student generator, which is summed with the student generator feature representation which has the same dimension. The student discriminator contains one convolution layer and 6 ResBlocks.
We optimize the model using Adam (Kingma & Ba, 2014) with batch size of 16. The learning rates of the generator and the discriminator are set as 0.0001 and 0.0004 with exponential decay rates of (β1, β2) = (0.0, 0.9). The model is trained for 300 epochs for knowledge distillation. In the second stage, we iterate 300 epochs, with same batch size, learning rate and exponential decay rate. In Eq. 6 both αl and β are identical. For the specific features of which the dimension is less than 128, we set them 0.1. In other case they are 0.01. In Eq. 8 λadv and λkdl are 1, and λsrl is 0.1.
Model details for the second stage. We finetune the well-initialized student network learned in the first stage. In this stage, we get access to the target data. We keep all the details on training and the hyperparameters of SSA-GAN (Hu et al., 2021). We refer readers for more detailed information to (Hu et al., 2021).

heading: A.2 KNOWLEDGE DISTILLATION FOR AUDIO-TO-IMAGE TRANSLATION
text: For audio-to-image translation the training is composed of two stages: data-free distillation from the unconditional prerained StyleGAN to audio-to-image translation model, and finetuning audioto-image translation model using the target data and label.
Model details for the first stage. We perform knowledge distillation for audio-to-image based on S2IGAN (Wang et al., 2020a). To obtain stable training, S2IGAN proposed a progressive mechanism to generate multi-scale images. S2IGAN introduced a two-stage study, i.e., a speech semantic embedding stage and an image generation stage. The former is to extract the speech embedding with a pretrained speech extractor, the latter takes as an input both random noise and the speech embedding to synthesize photo-realistic images in a multi-stage (coarse-to-fine) way. The generator is composed of one fully connected layer (228× 2048), one batchnorm, one Gaussian Error Linear
5After each ResBlock the feature resolution is half of the previous one in both encoder and discriminator, and two times in generator.
Unit(GLU) (Devlin et al., 2019) and 7 UpsBlocks 6. The first UpsBlock is B × 512 × 4 × 4 (B is the batch size). We extract the hierarchical representation from StyleGAN (Karras et al., 2019). The first feature from StyleGAN isB×512×8×8, and fed into the student generator, which is summed with the student generator feature representation which has the same dimension.
We optimize the model using Adam (Kingma & Ba, 2014) with batch size of 12. The learning rates of both the generator and the discriminator are set as 0.0002 with exponential decay rates of (β1, β2) = (0.5, 0.999). The model is trained for 300 epochs for knowledge distillation. In the second stage, we iterate 300 epochs, with same batch size, learning rate and exponential decay rate. In Eq. 6 both αl and β are identical, and are 10. In Eq. 8 λadv , λkdl and λsrl are 10.
Model details for the second stage. We finetune the well-initialized student network learned in the first stage. In this stage, we get access to the target data. We keep all the details on training and the hyperparameters of S2IGAN (Wang et al., 2020a). We refer readers to check more detailed information to (Wang et al., 2020a).

heading: A.3 KNOWLEDGE DISTILLATION FOR SEGMENTATION MAP-TO-IMAGE TRANSLATION
text: For segmentation map-to-image translation the training is composed of two stages: data-free distillation from the unconditional prerained StyleGAN to segmentation map-to-image translation model, and finetuning segmentation map-to-image translation model using the target data and label.
Model details for the first stage. For segmentation map-to-image task, we use SEAN (Zhu et al., 2020b) as the student model, since it is state-of-the-art. SEAN is composed of one generator, one style encoder and one discriminator. The generator contains several SEAN ResBlks. Each of the ResBlks is followed by a nearest neighbor upsampling layer. We extract the hierarchical representation from StyleGAN (Karras et al., 2019). The first feature from StyleGAN is B × 512× 8× 8. In the knowledge distillation stage, we set the learning rates to 0.0001 and 0.0004 for the generator and discriminator, respectively (Heusel et al., 2017). For the optimizer, we choose Adam (Kingma & Ba, 2014) with β1 = 0, β2 = 0.999, with batch size 4. In second stage (segmentation map-toimage translation), we use same setting except for batch size (i.e., 8). We update 100 epochs for knowledge distillation. In second stage (image-to-image translation) we iterate 100 epochs. In Eq. 6 both αl and β are identical. For the specific features of which the dimension is less than 128, we set them 0.1. In other case they are 0.01. In Eq. 8 λadv and λkdl are 1, and λsrl is 0.1.
Model details for the second stage. We finetune the well-initialized student network learned in the first stage. In this stage, we get access to the target data. We keep all the details on training and the hyperparameters of SEAN (Zhu et al., 2020b). We refer readers to check all detailed information in (Zhu et al., 2020b).

heading: A.4 KNOWLEDGE DISTILLATION FOR IMAGE-TO-IMAGE TRANSLATION
text: Model details. We take StarGANv2 (Choi et al., 2020) as the student model. StarGANv2 contains 5 sub-networks: a context encoder, a mapping network, a style encoder, a generator and a discriminator. For both content encoder and generator, we extract hierarchical representations with dimension size range from 16 to 256. We also extract the corresponding dimension hierarchical representations from the pretrained generator and discriminator respectively. To align the student style encoder with the teacher discriminator, we extract hierarchical representations for both networks with the dimension size ranging from 4 to 256.
We use the Adam (Kingma & Ba, 2014) with β1 = 0, β2 = 0.99, with batch size 2. We update 20,000 iterations for knowledge distillation. In second stage (image-to-image translation), we iterate 100,000.
6Each UpsBlock consists of one interpolate with scale factor 2, one convolutional layer (3× 3 kernal size, stride 1), batchnorm and GLU (Devlin et al., 2019). After each UpsBlock, the resolution is half of the previous one in the discriminator, and two times the previous one in generator.
Model details for the second stage. We finetune the well-initialized student network learned in the first stage. In this stage, we get access to the target data. We keep all the details on training and the hyperparameters of StarGANv2 (Choi et al., 2020). We refer readers to check all detail information in (Choi et al., 2020).
We are able to explicitly control domains/classes as starGANv2 does. For the latent-guided synthesis, starGANv2 introduces class-specific mapping networks (one for each class) to project the class embedding into the shared latent space. Thus starGANv2 uses the class-specific mapping networks to control the domains/classes. In this paper, we follow the same setup as starGANv2. In the first stage of our method, we only learn one mapping network, which is duplicated to initialize all the mapping networks (one for each class) of the second stage. Then we use the target dataset (second stage) to train the well-initialized class-specific mapping networks.
B I2I TRANSLATION

heading: B.1 TWO-CLASS I2I TRANSLATION RESULTS
text: To evaluate the generality of our method, here we validate the proposed algorithm for two-class I2I translation on a two-category dataset: cat2dog-200 (Wang et al., 2021) with an image size of 256 × 256. In cat2dog-200, the training set is composed of 200 images (100 images/per class) and the test set has 200 images (100 images/per class). We investigate two cases of I2I translation: the style representation is from noise (i.e., latent-guided synthesis) and from the reference image (i.e., reference-guided synthesis). As reported Tables 3 and 4, compared to the transfer learning methods (i.e., DeepI2I (Wang et al., 2020b) and TransferI2I (Wang et al., 2021)) we still maintain a large advantage.

heading: B.2 MULTI-CLASS I2I TRANSLATION RESULTS
text: We also further validate the proposed algorithm for Multi-class I2I translation on AFHQ-500 (Choi et al., 2020) with an image size of 256 × 256. In AFHQ-500, the training set is composed of 500 images (100 images/per class) and the test set has 1500 images (500 images/per class). We investigate two cases of I2I translation: the style representation is from noise (i.e., latent-guided synthesis) and from the reference image (i.e., reference-guided synthesis). As reported Tables 5, compared to StarGANv2 we still maintain a large advantage.
B.3 VARIANTS OF THE BASELINES
We adapt both DeepI2I (Wang et al., 2020b) and TransferI2I (Wang et al., 2021) to use a very similar architecture to StarGANv2 (the architectural details are in Table 5, 6, 7 of StarGANv2 (Choi et al., 2020)). Specially, to devise the generator, we preserve the subnets of the discriminator of the pretrained StyleGAN as image encoder which has 16 × 16 × 512output, then additionally add 4 ResBlks like StarGANv2, and finally use the subnets of the generator of the pretrained StyleGAN as the decoder to output the image. We keep the mapping network of the pretrained StyleGAN for the I2I translation generator when performing latent-guided I2I translation. We adapt the discriminator
of the pretrained StyleGAN as the style encoder when performing reference-guided I2I translation. Similar to StarGANv2, we also use a multitask discriminator, which consists of multiple output branches. We use cat2dog-200 dataset to compare the adapted DeepI2I (DeepI2I*), adapted TransferI2I (TransferI2I* ) and ours. As reported in Tables 6 and 7, compared to the transfer learning methods (i.e., Adapting DeepI2I and TransferI2I ) we still maintain a large advantage.

heading: B.4 ABLATION OF THE ENCODER
text: In this paper we use the pre-defined discriminator to initialize the encoder, since the discriminator of the StyleGAN is trained on HHFQ with 70k images, which optimizes it to be an effective feature extractor. The similar technique was also explored in SGD (Shocher et al., 2020) and transferI2I.
To verify that the previously reported results also hold for our method, we also train both the encoder and the reference encoder from scratch in reference-guid translation on Animal faces dataset (10/per class). we achieve 130 of mFID, which is lower than our method (mFID 96.2) . If we train the encoder from scratch, the training suffers from overfitting with limited training images.
C ADDITIONAL RESULTS

heading: C.1 USER STUDY
text: We conduct a user study and ask subjects to select results that are more realistic given the target label, and have the same pose as the input image. We apply pairwise comparisons (forced choice) with 20 users (100 image pairs/user) for I2I translation. Experiments are performed on images from the Animal faces dataset. Fig. 7 shows that our method considerably outperforms the other methods.

heading: C.2 DIVERSE OUTPUTS
text: Figure 8 shows some examples of the synthesized images on three datasets. Taking Animal faces as example, given the target class label our method manages to generate high visual quality images.

heading: C.3 KNOWLEDGE DISTILLATION FOR I2I SYSTEM IN THE FIRST STAGE
text: Figures 9 and 10 show examples generated after transfer learning on I2I system. This shows that our method successfully distills the style-mixing characteristics from the teacher generator to the student generator.

heading: C.4 KNOWLEDGE DISTILLATION FOR X2I SYSTEM IN THE FIRST STAGE
text: Figures 11 and 12 show examples generated after semantic diversity loss. For each triplet, we depict the teacher’s output(left), the student’s output (middle) with condition cx and the one (right) with condition cy . This shows that our semantic diversity loss manages to diversify the output given the input condition c.

heading: C.5 ADDITIONAL RESULTS
text: We additionally depict the generated images in Figures 14- 26.

Review summaries:
The work introduces a flexible method to distill the knowledge of unconditional GANs of images to various image translation tasks, including image-to-image, text-to-image, and audio-to-image translation. While prior work requires using the same architecture of the pre-trained GANs to train the downstream im2im tasks, the proposed method supports distillation on a wide range of X-to-image translation architectures (e.g. starGANv2). The authors have compared their methods with prior work and showed that the proposed method obtained better FIDs.

In this paper, the authors present a unifield learning method to investigate the knowledge transfer for X2I translation. Compared to existing methods, there are two advantages here: (1) this framework can be used for varying kinds of conditional image synthesis tasks; (2) it relieves the limitation for student generator to be the same as the pre-trained GAN. Specifically, the authors leverage the style mixing characteristics of high-quality GANs and introduce the semantic diversity loss to achieve it. Many experiments are conducted with good qualitative and quantitative results.

The paper presents a framework for knowledge distillation in image-to-image translation and arbitrary domain-to-image (e.g., text-to-image) translation. Authors claim following contributions: (1) a unified transfer learning method that leverages only synthetic images, (2) a semantic diversity loss, (3) a style-mixing triplets representing a tuple that includes an input image, a reference image, and a target image. Authors provide results for image-to-image translation tasks and for text-to-image translation tasks. A limited comparison with several modern models is presented. Authors claim that the proposed transfer learning method outperforms modern state-of-the-art.