Distilling GANs with Style-Mixed Triplets for X2I Translation with Limited Data
Authors: Yaxing Wang
Joost van de weijer
Lu Yu
SHANGLING JUI
Venue: ICLR 2022 Poster (2022)
Keywords: Transfer learning
image synthesis
limited data.

Abstract: Conditional image synthesis is an integral part of many X2I translation systems, including image-to-image, text-to-image and audio-to-image translation systems. Training these large systems generally requires huge amounts of training data. 
Therefore, we investigate knowledge distillation to transfer knowledge from a high-quality unconditioned generative model (e.g., StyleGAN) to a conditioned synthetic image generation modules in a variety of systems. To initialize the conditional and reference branch (from a unconditional GAN)  we exploit the style mixing characteristics of high-quality GANs to generate an infinite supply of style-mixed triplets to perform the knowledge distillation. Extensive experimental results in a number of image generation tasks (i.e., image-to-image, semantic segmentation-to-image, text-to-image and audio-to-image) demonstrate qualitatively and quantitatively that our method successfully transfers knowledge to the synthetic image generation modules, resulting in more realistic images than previous methods as confirmed by a significant drop in the FID. 

Published as a conference paper at ICLR 2022
DISTILLING GANS WITH SIXLE-MIXED TRIPLETS FOR X2I TRANSLATION WITH LIMITED DATA
Yaxing , van de Weijer?, Lu Shanglig Jui
I College of Computer Science, Nankai University, China
2 Computer Vision Center, Universitat Autònoma de Barcelona, Spain
School of Computer Science and Engineering, Tianjin University of Technology, China
4 Huawei Kirin Solution, China
[yaxing, joost, Juyu &cve. aecve.ubb.e. es, jui.shangl ingehuawei.com
ABSTRACT
Conditional image synthesis is an integral part of many X2I translation systems, including image-to-image, text-to-image and audio-to-image translation systems. Training these large systems generally requires huge amounts of training data. Therefore, we investigate knowledge distillation transfer knowledge from high-quality unconditioned generative model (e.g., StyleGAN) to a conditioned synthetic image generation modules variety of systems. To initialize the con- ditional and reference branch (from a unconditional GAN) we exploit the style mixing characteristics of high-quality GANs to generate an infinite supply of style-mixed triplets to perform the knowledge distillation. Extensive experimen- tal results in a number of image generation tasks (i.e., image-to-image, semantic sigmentation-to-mmags
text-to-image and audio-to-image) demonstrate qualita- tively and quantitatively that our method successfully transfers knowledge to the synthetic image generation modules, resulting in more realistic images than pre- vious methods as confirmed by a significant drop in the FID. Code is available https: / www.c//////////. www.://www.............///////// hwwps//iwgggggg/Kgaa
INTRODUCTION
Conditional image synthesis, also X2I translation, maps from an input domain (e.g. text, audio, segmentation maps, etc.) to the image domain. Benefiting from GANs (Goodfellow et al., 2014) and its follow-up improved versions (Gulrajani al.] [017]] Kang & Park] 2020 [Salimans et al.) 2016), they obtain remarkable performance on wide variety of image synthesis tasks: image to image (121) (Lee et al. 2018 Zhu et al. 2017). audio to image (Chen et al., 2017: Wang et al, 2020a), text to image (Hu al. 2021 Li al. 2019 2020 Radford et al. 2021 Zhang et al. 2017a) and semantic segmentation map to image Isola et al. 2017; Wang et al. 2018). Despite impressive leaps forward for variety of image synthesis tasks, there are still important challenges. Specifically, to obtain good results, existing works rely on large labelled datasets. Labeling these datasets is both laborious and time-consuming, considerably reducing the practical impact of these methods. It is noteworthy to observe that many of these models (Zhang et al. 2017a) apply transfer learning the text and audio encoders (c.g. using a pretrained LSTM (Reed al.) (016) model for text and pretrained GRU (Merkx et al.) 2019) model for audio), however they train the image synthesis decoder from scratch. This happens because there are no established methods transfer pretrained GANs to conditional image decoders; an omission which we aim to address in this paper.
In this paper, we investigate knowledge transfer for a variety of conditional image synthesis tasks. Traditional knowledge transfer for conditional image synthesis is often not possible, because there might not pretrained network available for the desired translation task (e.g. at the moment no high-quality pretrained network for segmentation map-to-image translation is available). It would therefore be preferable if the wide variety of high-quality GANs available for image generation could be exploited for X2I. Recent works (Wang al.) 2021 2020b) leveraged pretrained GAN to initialize an 12I translation model, managing to transfer knowledge to different image synthesis
Thhe corresponding author.

---

Published as a conference paper at ICLR 2022
Imput: row) audio (2nd row)
Output: image
Input: image: fist row)
Output::
Imput: label 11st row image (2nd row)
Reference image
Output: image
label (2nd row)
image
yellow bird crown and yallow bill
30 looken ducpptes
This flower yellow in colar, with petals that skinny and long
Figure 1: Various examples of method. Using the proposed knowledge transfer method, able generate high-quality images for several image synthesis tasks.
tasks. These methods, however, suffer from three problems: (I) They can only be used for I2I translation and do not generalize to other conditional image synthesis tasks. (II) They are GAN architecture-speciie
approaches, requiring the GAN architecture within the X2I system be exactly the same as that of the pretrained GAN. This limits transfer to current state-of-the-an art X2I systems for which similar GAN architecture exists (like for example StarGANv2 (Choi et al. 2020) for 121). (III) X2I systems are based on a conditional GAN, however existing methods for knowledge transfer for 12I do not initialize the conditional branch during the transfer, and therefore this has to be learned from scratch during the finetuning on the target dataset. Learning this on small target datasets can be problematic.
To address the aforementioned problems, we propose several improvements for knowledge transfer X2I systems: (I) we are the first to investigate knowledge transfer for X2I translation. Therefore, we propose a novel, unified transfer learning method, which can be used for varying kinds of con- ditional image synthesis tasks (Figure which is based on generated images and therefore does not require any real data. (II) The student generator does not need to have the same architecture as the pretrained GAN. Therefore, we can use well-devised specific image synthesis architectures (e.g., SPADE (Par al.) 2019) and StarGANv2 (Choi et al.) (020) by distilling knowledge from the pretrained teacher GAN (e.g.. StyleGAN) to the task-specific student GAN. (III) We use the style mixing characteristic of StyleGAN create style-mixed triplet data, which used transfer the knowledge efficiently to both 121 and X2I translation models. Furthermore, we propose a semantic diversity loss based on the style-mixed triplet, which contributes learn the semantic information of the output image.
We perform experiments wide variety of image synthesis tasks, including text-to-image, audio- to-image, segmentation map-to-image and 121 translations. We demonstrate the efficiency of the proposed knowledge distillation method, providing qualitative and quantitative results. We prove that the single pretrained GAN model can be universally used in varying specific task model. Addi- tionally, leveraging the style mixing character of StyleGAN, further improves 12I translation perfor- mance.
RELATED WORK
GAN-based Conditional Image Synthesis. Benefiting from the advances in GANs and its vari- ants in recent years, conditional image synthesis (also called X2I translation) research has developed rapidly. Two rypical approaches have been investigated for GAN-based image synthesis, namely, un- supervised (Kim al. 2017 Yi et al. 2017) and supervised image generation (Park et al. 2019 Zhang et al. (0176] Zhou et al.) 2020). The latter inputs conditional information (c.g. text, audio, image, segmentation map etc.) to synthesize images which contain the corresponding semantic formation (i.e. the conditional information). Specifically, text-to-image translation (Hu et al.) [021]] Li al., 2020 Zhang et al., 2017a) aims to synthesize high-realistic images which are semantically consistent with the text descriptions. Recent work (Hu et al., (021) introduces semantic-spatial batch normalization to better exploit the text information. Similar to text-to-image translation, both audio-to-iman tomge translation (Chen al., 2017; Wang et al.) 2020a) and segmentation map-to-image translation (Bau et al.) |2200] Park et al. 2019) aim to learn a mapping from the audio/segmentation map the output image. Different the above image synthesis tasks, image-to-image transla-
2

---

Published as a conference paper at ICLR 2022
2017) performs projection from the source to the target image domain. In this paper, we explore transfer learning from GANs to a variation of conditional image synthesis tasks.
tion (Park et al. 2020 Zhu et al.)
Transfer learning. A considerable research effort has investigated transferring knowledge for both
discriminative (Donahue al. 2014 Hinton et al., 2014] Xie et al. 2015; Yu ct al., 2019a Zhou
ct al.
2022 Zhao et al., (020a) and generative tasks (Noguchi & Harada 2019 Zhao et al. (020b).
There also exist several approaches (Goetschalckx al. [2199] Jahanian al. 20200 which focus on
the image manipulation based on the pretrained GAN. Given a target semantic attribute they aim to manipulate the output image of a pretrained GAN. However, these methods do not focus on transfer learning for target data. Some other methods (Abdal et al. 2019, Zhu et al. (020a) reverse the given image into the imput latent space of the pretrained GAN (e.g., StyleGAN), and manage to restructure the target image by optimization of the latent representation. Recent work (Shocher et al. 2020
Wang et al., [202]] 2020b) performed knowledge transfer from a pretrained classification model (e.g., VGG (Simonyan & Zisserman, 2014)) the discriminator (BigGAN (Brock et al. (019)) for 121 translation. However, both Deepl2I (Wang et al., 2020b) and Transfer121 (Wang et al.) (02T)) require that the GAN architecture is identical with the generator used in the 121 architecture. As consequence, these methods cannot be applied to well-designed I2I translation architectures (like starGANv2 (Choi et al. 2020)) since they do not use standard GAN architecture. The proposed method could address these problems.
KNOWLEDGE TRANSFER FOR X2I
Problem setting. Our goal to transfer knowledge from pretrained high- high-quality unconditional GAN to an X2I translation system for the case when training data is limited. The proposed method consist of two stages. In the first stage we perform the data-free knowledge transfer method ex- plained in See. 3.2 and 3.3 This stage does not require any target data. In the second stage, apply a standard finetuning of this distilled model on the target dataset. Distillation can be used to perform knowledge transfer other architectures, however, there is little target data available. For- tunately, the pretrained GAN can generate infinite data for data distillation, meaning that we do not require access to any real data (i.e., data-free). Also, transfer knowledge from unconditional GAN (e.g.. StyleGAN) to a conditional GAN (e.g.. 121 translation); requiring us to propose tech- niques to initialize the conditional branch. Exploiting the style-mixed characteristic (Sec. 3 of StyleGAN we propose two solutions: one for 121 (Sec. 3.2) and another one for X2I (Sec. 3.3) Our method consists of two stages:(1) transfer leaning without any real data (i.e., style-mixed triplets in Sec. 3.1 12I translation in Sec. 3.2 and X21 translation in Sec. 3.3) and (2) finetunning with the real target data.
3.1 STYLE-MIXED TRIPLETS
Benefiting from the style mixing ((Figure 2 (a))) characteristic of StyleGAN, we are able to create an infinite amount of triplet images from any of the two domains. Let z, E R² indicates the input noise of the pretrained generator Gr (teacher). given the input noises z and x? the mapping network Mr of the StyleGAN generator encodes them to style vector s = Mr and s Mz() We define (z) = (Mr(z) where G², takes styles vectors as an input to each of its layers. We further feed these style vectors to the teacher generator G¹ to output x = (s and x² (s,() (where xt, x2 E X' and image domain X' = Hxxxx3 x3) respectively. Here the function (sst, s$) selects to forward st to the layers of Gt important for the content of the generated image (the bottom layers), and se to those important for the style (the top layers ! Interestingly, a new image Yt G (s{ s²)) (where yt y and image domain y = IR. HR contains mixed information with respect to content and style of both inputs x and x²- Based on this observation above, we propose to leverage these miyle triplets (x] x2, y+) to perform distillation for the 12I and X2I translation models without the need of any real triplet training data.
We use the the style vector lt in the first four layers of the generator, and do s in the following layers. In practice, during experiments we generate style-mixed triplets online in the minibatch.
3

---

Published as a conference paper at ICLR 2022
(a)style-mixed triplets
(b) 121
Gr
ER
I
M2
GT
2
Dr
1/+
(c)X21
y/
2?
Suplecs roae
1
KD
KD
26
xt
3/1
KD
Es
Gs
Cy
yx
KD: knowledge distillation
frozen
x
S g: semantic encoder
learnable
Sg
Gs
Figure 2: (a) Synthesis of style-mixed triplets .t). (b) Overview of our data-free knowledge distil- lation for 121 translation. We transfer knowledge from the teacher pretrained discriminator Dr to both the student encoder Es and the reference encoder ER. In addition, we transfer from the pretrained generator Gr the student generator Gs. Here x the input image, x² is the reference image, and yt is the target image. (c) Overview of data-free distillation for X2I translation. The semantic encoder Sg aims extract the at- tribute embedding, and further control the semantic characteristic the student generator output. The semantic encoder is identical to the teacher discriminator and initialized by it.
3.2 DATA-FREE KNOWLEDGE DISTILLATION FOR 121 TRANSLATION
We aim to leverage the style-mixed triplets perform knowledge transfer for the 12I model. Let X; E X' and E y indicate the two domains. We expect to map an image x, from domain X' into another sample Yt which mimics y. Note we never get access to any real data. Figure P (b) shows the knowledge distillation framework for 121 translation. Our framework is composed of six neural networks: a pretrained teacher GAN (consisting of GT and DT), and an 12I model with a student generator Gs. student discriminator Ds. student encoder Es that extracts the content information from the input image x] and reference encoder ER that extracts the style information from the reference image x² Similarly StarGANv2, translation: the style representation is from noise (i.c., latent- guided synthesis) and from the refer- ence image (i.e., reference- guidedd synthesis). Here we mainly focus on enference-guidee synthesis, since latent-guided synthesis can be adapted easily by replacing the reference encoder with several fully connection layers, which take the noise as input.
investigate two cases of I2I
Both the teacher discriminator Dr and the student encoder Es take the input image x as input, extracting the hierarchical representation { (Dr(x))1 } and {Es(xt)t) Here Dr(xt); represents the output of the 4th ResBlock [ We align them via the feature-based knowledge distillation loss:
Liat Σ >u Drr)Dr(()) FEs(x!)
where parameters " scalars balance the terms. We them to 0.1. When the teacher and student dimensions are not the same, an additional layer can be introduced to map the teacher output the desired dimensions (similar as the hint distillation used (Romero et al. 2015)).
Next, we take the image x² for both the pretrained discriminator Dr and the reference encoder En extracting the hierarchical representation { )1} and { {xx(x)) ). both of which we encourage to be aligned via the feature-based knowledge distillation loss
Di Σ Dr(x)) - (x )xll
After each ResBlock the feature resolution is half of the previous one in both encoder and discriminator, and times generator.
4

---

Published as a conference paper at ICLR 2022
where parameters 77 scalars which balance the terms. We set them to 0.1.
Finally, taking the output the student encoder Es(x;) and the output of the reference encoder ER(x,), for the student generator Gş obtain the hierarchical representation (Gs(Es(x)), Ex(x))) and the output image y. = (G(EE(x))) En(x))), both of which we encourage to align with the hierarchical representation ( st))), We also align the output image Yt = (D s()) with y::
kdl Σ on ( +))), Gs(Es(x)), En )xl 8 ly: 8s111-
where parameters and f are scalars to balance the terms. We set them to 0.1.
We also employ the following GAN loss (Goodfellow et al. 2014) to optimize this problem:
Lade y [log Ds (ye)) + Eg.-y [lo((1 Ds (8x)),
(4)
The full objective function of our model is:
min maxxaaeeeude + )kai (haat + Chal + (latt
(5)
Gs Dg
where Aade and Abdl are hyper- parameters to balance their relative importance. We set them to 1.
In conclusion, to improve knowledge transfer for I2I, have presented two novel contributions. Firstly, we have shown that the pretrained StyleGAN can be used to generate an unlimited number of style-mixed triplets which we can use to perform the training. We are the first to exploit the style- mixing possibilities of StyleGAN to transfer knowledge to I2I systems. Secondly, we have proposed a distillation approach transfer knowledge from the GAN to the 12I model. We name this data- free distillation because no real data is required and it is based on generated data. our method can transfer knowledge between generators with different architectures.
3.3 DATA-FREE KNOWLEDGE DISTILLATION FOR X2I TRANSLATION
Here show how the proposed data-free knowledge distillation can be generalized to X2I trans- lation problems. We will perform the distillation based on a noise input to the student generator. For X2I translation, the conditional information с determines the structural information and the mantic information. For instance, in text2image translation on the birds dataset, we take as an input text descriptor (information c), and expect generate an image which mimics the shape and the appearance of real bird. However, during the knowledge transfer, the conditional information not available. Therefore, we propose use the teacher GAN discriminator to extract semantic encoder of the image that can be used to replace the conditional information However, when directly applying this for knowledge distillation, we found that this information can still be ignored by the network (see e.g. Figure [T]. We therefore introduce two additional techniques. We again use the style-mixes triplets introduced in the previous section to perform the knowledge transfer and propose novel semantic diversity loss to diversify the semantic information of the output when varying the condition c.
As shown in Figure [[c), additionally introduce semantic encoder Sg to extract pseudo- condition vector c, which assists in distilling the knowledge from the unconditional GAN Gy to the conditional GAN Gs. The semantic encoder S5 is identical to the teacher discriminator after removing the last fully connection layer, and initialized by the teacher discriminator, which is well- optimized on the input images Yt and x Utilizing the style-mixed triplets (Sec. 3.1). we extract teacher output x! = (s,) and Yt = (4({ and the corresponding hierarchical repre- sentation (G* ) and (G)) )²)); respectively. Note that (st ) and (#(ss) s²)) are identical, since x provides the structural information of yt. Semantic encoder Sg extracts the semantic attributes from the input images x! and yt respectively, termed as Cx Sg(x!) and
Cy = (yt).) respectively.
For the student generator Gs, we take as input the teacher's latent representation at the first layer, the output the semantic encoder and the noise. Note, the teacher's 's latent representation is summed
WWe ignore the latent representation in the second stage (i.c., fintunning target domain), which means that the student generator has two inputs: the noise and the condition c.
5

---

Published as a conference paper at ICLR 2022
with the corresponding feature of the student generator. When the input of the semantic encoder and final student output X = Gs(z,, G (s,)), Cx). Similarly, when Yt is the input of semantic encoder Sg. we have the hierarchical representation [Z, G7 ($(s) )))),, (,y)))) and final student output y. = Gz ( ,,
SE is x. we generate the hierarchical representation (Gs(Zz,
By conditioning the student generator with the teacher representation, the student generator can output a similar image (this would otherwise be impossible). The loss is defined as:
Lkdi = E G Gs(z,, G ex)il1 + +
x x
(6)
Σ a (0(s) Gs ( )), cy): + + Bly+ - 8alli
where parameters and / are scalars to balance the terms.
Yang al. (2019) proposed loss to promote diversity of the generated images with respect to changes of the input noise Inspired by this work, to address the lack of diversity with respect to the conditional vector, we propose the semantic diversity loss is:
Lat = - { 8=111
--Issszs Gyy (ss)), - G7 ) Gs(z.. G7 (,
- GSsss (st),,
Other than Yang al. 2019), our loss promotes diversity with respect to changes of the conditional vectors (i.c., Cx and Cy) while we retain the input noise vector Zz. Since the conditional vector influences the semantic information (e.g., gender, style), we call our loss the semantic diversity loss.
The full objective function of our model is:
minmax AadvLade Akdt + Aest G5 Ds
where Aado, ÀRdl and Aart are trade-off parameters, and Lade is defined in Eq.
The proposed initialization procedure is not applicable to the case when the conditioning is a one- hot spatial map (we found the semantic encoder was not able to predict these). Therefore, for this case, we do not initialize the conditional branch and we set c are 1 in the student generator and 0 in the student discriminator during the distillation. We found that even for these cases the knowledge transfer led to large performance gains (Table
4 EXPERIMENTS
In Sec. 4.1 we investigate knowledge transfer for 12I translation. We explore how the proposed method affects image synthesis by using conditional information с (i.e., audio, text and seman- tie segmentation) in Sec. 4.2] For the experimental evaluation, transfer method explained in the previous section (i.e the first stage) and use this distilled model for the downstream task where we finetuned it the target data (i.e. the second stage). We the pretrained StyleGAN optimized on HHFQ human face (Karras et al.) 2019) except for semantic segmentationnto-inee -o--iatteemS-iimmaaaaa which is on AFHQ dataset (Choi et 2020). More detailed information and visualization results can be found in the Appendix A and
apply
proposed knowledge
4.1 DATA-FREE KNOWLEDGE DISTILLATION FOR 121 TRANSLATION
We first evaluate our method for 121 translation with reference image, named reference-guidee 121 translation. Then we verify the proposed method without reference image, called satent-guided 12I translation, in which
random noise to control the style of the output image. We take state- of-the-art StarGANv2 as the student network. Except for the well known metrics: FID (Heusel et al. 2017) and KID (Biikowski (FC) (Shmelkov et al. 2018) to evaluate the ability to generate class-specific images. RC trained real data and eva evaluated on the generated data and vice versa for FC.
al. 2018), we also train real (RC) and fake classifier

---

Published as a conference paper at ICLR 2022
Input
Reference
Output (StarGANv2)
Output (ours)
.90
146
woadie))
146
Animal races
Bires
Foods
(b)
0.12
0:17
StanANv2
Transfe122
0.83
0.88
Ours
(c)
Figure 3: (a) Qualitative comparison on Animal faces, Birds and Foods datasets.(b) Ablation study variants of our method on Animal faces, Birds and Foods. (c) User study.
Asimaa fares (Hlpprr clannt
Tipper clavs)
Method
m mKIDx 10C.
MRD
RCT
1O0]
mPIDI
RCT
KIDX 100
mFIDI
RCT
Latent-gudeal P3
Translatonn
SINT
314
2856
5.5
4.4
32.7
223.
830
8.71
23.7
236.
113
11.
DMIT
29.5
2801
538
23.5
2504
129
11
2014
8,30
DRITes
270
198.5
SurUANS
1926
Deaph21
1303
Transeri21
9:25
1033
22.3
254
6:25
1183
27.1
25.4
3302
107.8
43.2
24.8
Ours
9.01
94.7
25.6
27.8
6.15
107.4
29.5
36.2
5.52
1152
38.2
21.6
SarGANZZ
103
1194
30.5
35.7
6.08
125.7
19.0
38.7
3.86
1156
45.3
Uuis
937
923
334
39.9
3.88
110.4
32.5
41.3
3.28
1059
48.5
26.1
Referenee guided 121 trunslation
SINGANC
132
1447
29.3
36.6
8.35
128.7
28.6
25.5
6:38
1322
3006
23.6
9,45
96.2
372
395
6.96
103.2
32.6
34.2
392
120.5
39.3
25.4
Table 1: Comparison with baselines. * means the training image resolution is 256
256.
Reference-guided 12I translation. We conduct multi-class 121 translation on three datasets: Ani- mal faces (Liu et al. 2019), Birds (Van Horn et 2015) and Foods (Kawano & Yanai 2014). The Animal faces dataset contains 1, 1,490 images and 149 categories in total, Birds has 48,527 images and 555 classes Foods consists of 31,395 images and 256 classes. We compare StarGANy2 (Choi ct al., 2020). We also explore wide variety of configurations for our approach, including: no knowledge distillation (No KD), knowledge distillation only using L/ distance between the teacher generator and the student generator (KD (w-LI, w/o-adv.), knowledge distillation only using the adversarial loss (KD (w/o-L,, w-adv.) and knowledge distillation with both losses (KD (w-LI, w-adv.).. We also ablate the style-mixed triplets. We replace both the reference image x and the style-mixed output Yt with the input image x (indicated by KD* (w-LI, w-adv.)..
Knowledge distillation and style-mixed triplets. Figure 3b) presents comparison between several variants of our method in terms of mean FID (mFID) on three datasets. Taking Animal faces as an example, adding either LI distance or the adversarial loss improves the 12I translation task in general compared to the 12I model trained from scratch (mFID: 134.7). Finally combining both losses obtains the best score (mFID: 96.2), indicating that the proposed method largely improves 12I translation performance. Without the style-mixed triplets, the model performance largely drops (mFID:104.1), even with both the 7.1 distance and the adversarial loss. This probably because the generator fails to perform disentanglement, and the reference encoder does not learn the style information. In conclusion, this experiment validates the effectiveness of the style-mixed triplets.
Results. Table (reference-guidee 121 translation) reports results for baseline and our method on Animal faces, Birds and Foods datasets with image resolution 256 x 256. StarGANy2 training from scratch obtains lower results (e.g., mFID:134 134.7 on Animal faces dataset), indicating training I2I model from scratch is challenging when given limited data. Our method achieves a significant ad- vantage (mFID: 96.2 Animal faces dataset), demonstrating that the proposed knowledge transfer

---

Published as a conference paper at ICLR 2022
kg Speken desoistion
This flower is white pink color, with petals that have small veins
This small bird has red head and curved beak.
The flower is made of white petals and pistol yallow in color
A small bird with red cheek and pointed beak.
(a) Text2image
(b) Audio2image
(c) Segmentation mapZimage
Input
Reference
Outout
Input
Referance
Output
UE
(d) Segmentation mapžimage with reference image
Figure 4: Qualitative results of the proposed method on various image synthesis tasks,
Outout (taamer)
Output (student,
Rucruc (student
Figure 5: Examples generated after semantic diversity loss. For each triplet, we depict the teacher's out- put(left), the student's output (middle) with condition and the one (right) with condition cy Note the student model SSA-GAN, Results could be found the Appendix C when student model S2IGAN.
facilitates 12I translation. Figure 3 a) compares the results of mapping the input to the target image domain with reference image. Given both the input and reference image our method manages to generate higher quality images than StarGANv2, that retain the pose of the input and keep a simi- lar style as the reference image. clearly indicates that our method improves the image synthesis model. Note that the existing approaches which address transfer learning for I2I (Wang et al. 2020b) 202T) cannot applied to this case, since they require the 12I image synthesis architecture to be the same the pretrained GAN architecture.
Latent-guided 12I translation. We also validate the proposed method without reference image. We compare to StarGANv2, SDIT (Wang et al. 2019), DRIT++ (Lee et al.) 2020b). DMIT YYu) (ct al., 2019b), Deep121 and Transfer121. We use random noise to control the style information of output images. As reported in Table (latent-gun (latent-guided 121 translation), all basclines training from scratch (from 4rd to 6th row) suffer significant disadvantage on small dataset. StarGANv2 ob- tains the best result (131.2 mFID Animal faces). However, using knowledge transfer Transfer121 performs significantly better (103.5 mFID on Animal faces), and achieves the best score on Foods dataset. Finally, our method obtains the best results on both Animal faces and Birds datasets, and competitive performance on the Foods dataset. This could be due to the fact that the Foods dataset has more training data, and the importance of transfer learning is negligible. We also evaluate our method with higher resolution images (256x 256). As shown in the last two rows of Table [ (latent- guided 121 translation), we still retain large advantage comparing to StarGANv2, demonstrating that our method still obtains better performance for higher resolution images. Neither Deep121 nor Transfer121 is applicable, since there is no the released pretrained model for usage (they are based on the BigGAN). We conduct a user study and ask subjects to select the results that is more realistic given the target label, and has the same pose as the input image. We apply pairwise comparisons (forced choice) with 20 users (100 image pairs/user) for I2I translation. Experiments are performed on images from the Animal faces dataset. ((c) shows that our method considerably outperforms the other methods. The synthesized images three datasets are in the Appendix C
8

---

Published as a conference paper at ICLR 2022
Dunas:
SN
Methol
FIDU KID
ES
Mathed
FIIT IITPIEO IST
Method
PID
IST miQUT
MID
SAN
50.45
29
SIIGAN
SPAIE
45.35 SEAN
UE L1,910 12374
1300
USS W LI 101 aBr
B5:4
6.25
250
OASIS
97750
1.74
2.55
458%
Dis (97o LTW 331-8-3rr10 HAXT
1.75
22
XEES FRO0
LT LTWW-3W
10
3390
CocoaNen
26.31
120
0.79
20
Ouzn
88.81
6.52
3.00
Oue
22.96
1.4
2.50 50.4%
Oue 27.47
1.38
3.80
52.1%
Ours
24
0.85
Ous
90.75
7,43
2.09
One
20
Une
2.30
Dhrs
20.19
1.64
4.83
Okin
MIXX
Curr
05.32
2.72 48,33
35.70
2.00
Table 2: The results on a wide range of image synthesis tasks. i means the condition c. is in the student
generator and 0 in the student discriminator. means we remove the semantic diversity loss (Fq.. 7 from final objective (Eq. [). Left: text-to-image. Middle: audio-to-image. Right: semitic segmentation (SS) to image.
4.2 DATA-FREE KNOWLEDGE DISTILLATION FOR X2I TRANSLATION
We apply approach for conditional image synthesis. We consider conditioning information с from text, audio and segmentation maps. We explore four variants of our method: knowledge distillation only using L// distance between the teacher generator and the student generator (Ours (w-LI, w/o-adv.); )); knowledge distillation only using the adversarial loss (Ours (w/o-L1, w-adv.)); next we set the condition c=1 in the student generator and 0 in the student discriminator (ours t); finally we remove the semantic diversity loss in Eq. 7 from the final objective in Eq. (ours f). More details on training and hyperparameters can be found in Appendix A
Text-fo-image. We evaluate the proposed method on the CUB bird dataset (Welinder et al. 2010). It contains 8,855 training images (150 species) and 2.933 test images (50 species), following 10 text descriptions for each bird. Here we use 10 images per class for training, and verify our method on the test dataset. We compare with SSA-GAN (Hu et al. (022) which is one of state-of-the-art methods. Results are depicted in Figure I Figure (a) and Table (left).
tudio-to-image. We evaluate on the commonly-used dataset: Oxford-102 (Nilsback & Zisser- man 2008), where 82 categories and 10 images per category are selected for training, and 20 cat- egories and 1,155 images for test. Note that we split the training and test categories following S2IGAN /Wang et al. 2020a). Results are depicted in Figure I Figure (b) and Table 2 (middle).
Semantic aegmentatioonto-imaae Here we conduct experiments by conditioning on segmentation map the CeleAMask-HHt (Lee al. 2020a) dataset containing 30,000 image and segmentation mask pairs. We follow the same train/test split as SEAN (Zhu et al.) 2020b). We randomly select 500 pairs of data from the train set for training, and 2,000 pairs for testing. To evaluate our method, we compare to three baselines: SPADE (Park et al.) 2019). OASIS (Simonyan et al., (013) and 2oCosNetv2 (Zhou et al. 202T), which are trained from scratch. We report the generator perfor- mance using the pretrained StyleGAN optimized on AFHO Choi et al. (anmmll face dataset). Results are depicted in Figure I Figure 4 (c) and Table 2 (right, w/o reference). We also compare to SEAN (Zhu et al., 2020b) which additionally uses reference image to control the style of the output image, the results are shown in Figure and (riht, w reference).
As reported in T our approach outperforms all baselines without using knowledge distillation on several metrics. The semantic diversity loss gets the best performance except for the semantic- map-to-image task. The qualitative results in Figure ] and 4 indicate that our method can generate high-quality images with limited data due to the benefit of the knowledge transferred from the expert teacher network. [|gure/sshowe the output images of the X21 generator after knowledge distillation and shows that we synthesize diverse outputs given different condition c. In conclusion, our method is an efficient, general purpose mechanism for conditional image synthesis with limited data.
5 CONCLUSIONS
Conditional image synthesis tasks require vast amounts of training data. Therefore, we proposed to transfer knowledge from unconditional pretrained GAN model to conditional image synthesis tasks. We perform the distillation with style-mixed triplets that can be automatically computed from high-quality GANs. Our experiments confirmed that the proposed transfer learning method obtains state-of-thh-aan results and allows to train high-quality X2I systems from fewer labeled samples.
Acknowledgement. We acknowledge the support from Huawei Kirin Solution, and the Spanish Minesterio de Ciencia, Innovacion, y Universidades for funding the project 1002019-104144G0 100, 10. 10....130000 1303//501 100011033.

---

Published as a conference paper at ICLR 2022
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In ICCV, pp. 4432-4441, 2019.
David Bau, Hendrik Strobelt, William Peebles, Bolei Zhou. Jun-Yan Zhu, Antonio Torralba, et al. Semantic photo manipulation with a generative image prior. ACM Transactions on Graphics, 2020.
M Binkkoski,, DJ Sutherland, M Arbel, and Gretton. Demystifying mmd gans. ICLR, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In ICLR, 2019.
Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chenliang Xu. Deep cross-modal audio-visua generation. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pp. 349-357, 2017.
Yunjey Choi, Youngjung Uh. Jaejun Yoo, and Jung- Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In CVPR, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, pp. 647-655, 2014.
Lore Goetschalckx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defini- tions of cognitive image properties. In ICCV, pp. 5744-5753, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, pp. 2672-2680, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training wasserstein gans. NeurIPS, pp. 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In NeurlPS, pp. 6626-6637, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurlPS, 2014.
Kai Hu, Wentong Liao, Michael Ying Yang, and Bodo Rosenhahn. Text image generation with semantic-spatial aware gan, 2021.
Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, pp. 1125-1134, 2017.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerabbilly of generative adversarial networks. In ICLR, 2020.
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image gen- eration. In Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 21357-21369, Curran As- sociates, Inc., 2020, URL htpp:ttttt/////peeiiiin neuriinnnnnneuuuseee cc/ hcc/paper/2220//1eee E490c742cdB 3eee131eeeeeea1112222222aa
Paper. pdff
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, pp. 4401-4410, 2019.
Yoshiyuki Kawano and Keiji Yanai. Automatic expansion a food image dataset leveraging exist- ing categories with domain adaptation. In ECCV, pp. 3-17. Springer, 2014.
10

---

Published as a conference paper at ICLR 2022
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jungkwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. In JCML, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2014.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020a.
Hsin-Ying Ying Lee, Hung-Yo Yu Tseng, Jia-Bin Huang, Maneesh Kumar Singh, and Ming-Hsunn Hsuan Yang. Diverse image-to-image translation via disentangled representations. ECCV, 2018.
Hsin- Hsin-Ying Lee, Hung-V Yu Tseng, Qi Mao, Jia-Bin Huang, Yu-Ding Lu, Maneesh Singh, and Ming- Hsuan Yang. Drit++: Diverse image-to- -mmeag translation via disentangled representations. IJCV, pp. 1-16, 2020b.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr. Controllable text-to-image gen- eration. arXiv preprint Stti::1909.00000 2019.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip H.S. Torr. Manigan: Text-guided image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Ming-Y Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-image translation. In CVPR, pp. 10551-10560, 2019.
Danny Merkx, Stefan L. Frank, and Mirjam Ernestus. Language learning using speech to image retrieval. Interspeech 20/9, Sep 2019. doi: 10.24437/mmeeppech22009-6666 URL http: 2019 3067]
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics Image Processing. pp. 722-729. IEEE, 2008
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. ICCV, 2019.
Taesung Park, Ming-Yo Yu Liu, Ting-Chun Wang, and Jun- Zhu. Semantic image synthesis with opatially-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 2337-2346, 2019.
Taesung Park, Alexci A. Efros, Richard Zhang, and Jun- Yan Zhu. Contrastive learning for condi- tional image synthesis. In ECCV, 2020.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Scott Reed, Zeynep Akata, Bernt Schiele, and Honglak Lee. Learning deep representations of fine- grained visual descriptions, 2016.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, pp. 2234-22444 2242, 2016.
Konstantin Shmelkov, Cordelia Schmid, and Karteck Alahari. How good my gan? In ECCV, pp. 213-229, 2018.
11

---

Published as a conference paper at ICLR 2022
Assaf Shocher, Yossi Gandelsman, Inbar Mosseri, Michal Yarom, Michal Irani, William T Freeman, and Tali Dekel. Semantic pyramid for image generation. In CVPR, pp. 7457-7466, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In CVPR, pp. 595-655- 2015.
Ting- Chuu Wang, Ming-Ye Yu Liu, Jun- Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, pp. 8798- 8807, 2018.
Xinsheng Wang, Tingting Qiao, Jihua Zhu, Alan Hanjalic, and Odette Scharenborg. S2igan: Speech- to-image generation via adversarial learning. arXiv preprint arXiv:2005.06990 2020a.
Yaxing Wang, Abel Gonzalez- Garcia, Joost van de Weijer, and Luis Herranz. SDIT: Scalable and diverse cross-domain image translation. In ACM MM, 2019.
Yaxing Wang, Lu Yu, and Joost van de Weijer. DeepiZi: Enabling deep hierarchical image-to-image translation by transferring from gans. NeurlPS, 2020b.
Yaxing Wang, Hector Laria Mantecon, Joost van de Weijer, Laura Lopez-Fuentes, and Bogdan Raducanu. Transferi2i: Transfer learning for image-to-image translation from small datasets, 2021.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR 2010-001, California Institute of Technology, 2010.
Guo-Sen Xie, Xu-Ya Yao Zhang, Shuicheng Yan, and Cheng-Lin Liu. Hybrid enn and dictionary-based models for scene recognition and domain adaptation. IEEE Transactions Circuits and Systems for Video Technology, (766):2233- 1274, 2015.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works, 2017.
Dingdong Yang. Scunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee, Diversity- sensitive conditional generative adversarial networks. arXiv preprint arXiv: 10X1v:1901.09024 2019.
Zili Yi, Hao Zhang, Ping Tan Gong, al. Dualgan: Unsupervised dual learning for image-to-image translation. ICCV, 2017.
Lu Yu, Vacit Oguz Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, and Arnau Ramisa. Learning metries from teachers: Compact networks for image embedding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2907-2916, 2019a
Xiaoming Yu, Yuanqi Chen, Shan Liu, Thomas Li, and Ge Li. Multi-mapping image-to-image translation via learning disentanglement. In NeurIPS, pp. 2990-2999, 2019b.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE Trans. on PAMI, 2017a.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dim- itris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adver- sarial networks. In ICCV, pp. 5907-5915, 2017b.
Fang Zhao, Shengcai Liao, Guo-Sen Xie, Jian Zhao, Kaihao Zhang, and Ling Shao. Unsupervised domain adaptation with noise resistible mutual-training for person re-identification. In European Conference on Computer Vision, pp. 526-544. Springer, 2020a.
12

---

Published as a conference paper at ICLR 2022
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for limited-dar generation. ICML, 2020b.
Tao Zhou, Huazhu Fu, Geng Chen, Jianbing Shen, and Ling Shao. Hi-net: hybrid-fusion network for multi-modal mr image synthesis. IEEE transactions on medical imaging, 399))2772-2781,, 2020.
Tao Zhou, Huazhu Fu, Chen Gong, Ling Shao, Fatih Porikli, Haibin Ling, and Jianbing Shen. Consistency and diversity induced human motion segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. Cocosnet v2: Full-resolution correspondence learning for image translation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11465- 11475, 2021.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. ECCV, 2020a.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imas image translation using cycle- cyclssconsistent adversarial networks. In ICCV, pp. 2223-222222222 2232, 2017.
Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesis with semantic region-adaptiv normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5104-5113, 2020b.
A ARCHITECTURE AND TRAINING DETAILS
The proposed method is implemented in Pytorch (Paszke et al. 2017). We transfer knowledge from StyleGAN to X2I translation. We introduce here the architecture and model details for each task. Note we perform the knowledge distillation on one GPU(Quadro RTX6000) with 24GB VRAM.
A.1 KNOWLEDGE DISTILLATION FOR TEXT-TO-IMACE TRANSLATION
For text-to-image translation the training is composed of two stages: data-free distillation from the unconditional prerained StyleGAN to text-to-image translation model, and finetuning text-to-image translation model using the target data and label.
Model details for the first stage. The student model is the same as SSA-GAN (Hu et al. 202 which composed of three sub-networks: text encoder, a student generator Gs, student dis- criminator Ds. The text encoder is the pretrained one provided by (Xu et al. 2017). The generator takes random noise as input, following one fully connection (100 x 8192) and ResBllock F The input feature size of the first ResBlock is B x 512 x 4 x 4 (B is the batch size). We extract the hierarchical representation from StyleGAN (Karras et al.) 2019). The first feature from StyleGAN is B x 512 x 8 x 8, and fed into the student generator, which is summed with the student gener- ator feature representation which has the same dimension. The student discriminator contains one convolution layer and 6 ResBlocks.
We optimize the model using Adam (Kingma & Ba 2014) with batch size of 16. The learning rates of the generator and the discriminator are set as 0.0001 and 0.0004 with exponential decay rates of (81, (2) = (0.0, 0.9). The model is trained for 300 epochs for knowledge distillation. In the second stage, iterate 300 epochs, with same batch size, learning rate and exponential decay rate. In Eq, both and /3 are identical. For the specific features of which the dimension is less than 128, we set them 0. In other case they are 0.01. Eq. Xadv and Akal are 1, and Aari is 0.1.
Model details for the second stage. We finetune the well-initialized student network learned in the first stage. In this stage, we get access the target data. We keep all the details on training and the hyperparameters of SSA-GAN (Hu et al.) 2021). We refer readers for more detailed information to (Hu et al. 2021
After each ResBlock the feature resolution is half of the previous one in both encoder and discriminator, and times generator.
13

---

Published as a conference paper at ICLR 2022
A.2 KNOWLEDGE DISTILLATION FOR AUDIO-TO-IMAGE TRANSLATION
For audio-to-image translation the training composed of two stages: data-free distillation from the unconditional prerained StyleGAN to audio-to- audio-to-image translation model, and finetuning audio- to-image translation model using the target data and label.
Model details for the first stage. We perform knowledge distillation for audio-to-image based on S2IGAN (Wang et al., 2020a). To obtain stable training. S2IGAN proposed progressive mecha- nism to generate multi-scale images. S2IGAN introduced two-stage study, i.e., a speech semantic embedding stage and an image generation stage. The former is to extract the speech embedding with a pretrained speech extractor, the latter takes as an input both random noise and the speech embedding to synthesize photo-realistic images in a multi-stage (coarse-to- to- to-fine)) way. The generator is composed of one fully connected layer (228 x 2048), one batchnorm, one Gaussian Error Linear Unit(GLUU) (Devlin et al., 2019) and 7 UpsBlocks 6 The first UpsBlock is B x 512 x 4 x 4 (B is the batch size). We extract the hierarchical representation from StyleGAN (Karras al. 2019). The first feature from StyleGAN is B x 512 x 8 x 8, and fed into the student generator, which is summed with the student generator feature representation which has the same dimension.
We optimize the model using Adam (Kingma & Ba 2014) with batch size of 12. The learning rates of both the generator and the discriminator are (31. (2) (0.5, 0.999). The model is trained for 300 epochs for knowledge distillation. In the second stage, we iterate 300 epochs, with same batch size, learning rate and exponential decay rate. In Eq. 6 both and / are identical, and are 10. In Eq. 8 Xado, ARat and Asr are 10.
0.0002 with exponential decay rates of
Model details for the second stage. We finetune the well-initialized student network learned in the first stage. In this stage, we get access to the target data. We keep all the details on training and the hyperparameters of S2IGAN (Wang et al.. 2020a). We refer readers to check more detailed information to (Wang et al. 2020a).
A.3 KNOWLEDGE DISTILLATION FOR SEGMENTATION MAP-TO-IMAGE TRANSLATION
For segmentation map-to to-iimge translation the training is composed of two stages: data-free distil- lation from the unconditional prerained StyleGAN to segmentation map-to-image translation model, and finetuning segmentation map-to-image translation model using the target data and label.
Model details for the first stage. For segmentation sap-to-image task, we use SEAN ZZu et al. 2020b) the student model, since it is ofate-of-the-art. SEAN is composed of one generator, one style encoder and one discriminator. The generator contains several SEAN ResBlks. Each of the ResBlks is followed by nearest neighbor upsampling layer. We extract the hierarchical representation from StyleGAN (Karras et al. 2019). The first feature from StyleGAN is B x 512 x 8 3x 8.
We use the pretrained StyleGAN optimized on HHFQ human face (Karras et al. 2019) except for semantic gegmmentatio--to-mmae(SSSS we are able to use the pretrained one from HHFQ, which further improve the performance.
which is on AFHQ dataset (Choi et al. 2020). In fact
In the knowledge distillation stage, we set the learning rates to 0.0001 and 0.0004 for the generator and discriminator, respectively (Heusel et al. 2017). For the optimizer, we choose Adam (Kingma Ba 2014) with 31 0, f2 0.999, with batch size 4. In second stage (segmentation map-to- image trionn we use same setting except for batch size (i.e., 8). We update 100 epochs for knowledge distillation. In second stage (image-to-image translation) we iterate 100 epochs. In Eq, both at and 3 are identical. For the specific features of which the dimension is less than 128, we set them O. In other case they are 0.01. In Eq. Aado and Akal are I, and Aar is O.
Model details for the second stage. We finetune the well-initialized student network learned in the first stage. In this stage, we get access to the target data. We keep all the details on training and
'Each UpsBlock consists of one interpolate with scale factor 2, one convolutional layer (3 x 3 kernal size, stride 1). batchnorm and GLU (Devlin al. 2019). After each UpsBlock, the resolution is half of the previous in the discriminator, and times the previous one in generator.
14

---

Published as a conference paper at ICLR 2022
the hyperparameters of SEAN (Zhu et al. 2020b). We refer readers to check all detailed information in (Zhu et al. 2020b).
A.4 KNOWLEDGE DISTILLATION FOR IMAGE-TO-IMAN TRANSLATION
Model details. We take StarGANv2 (Choi al. 2020 the student model. StarGANv2 con- tains sub-networks: a context encoder, a mapping network, a style encoder, a generator and a discriminator. For both content encoder and generator, we extract hierarchical representations with dimension size range from 16 to 256. We also extract the corresponding dimension hierarchical rep- resentations from the pretrained generator and discriminator respectively. To align the student style encoder with the teacher discriminator, we extract hierarchical representations for both networks with the dimension size ranging from to 256.
We use the Adam (Kingma & Ba 2014) with 31 = 0, 82 = 0.99, with batch size 2. We update 20,000 iterations for knowledge distillation. second stage (image-to-image translation), we iterate 100,000.
Model details for the second stage. We finetune the well-initialized student network learned the first stage. In this stage, we get access to the target data. We keep all the details on training and the hyperparameters of StarGANv2 (Choi et al. 2020). We refer readers to check all detail information in (Choi et al. 2020).
We are able to explicitly control domains/classes as starGANv2 does. For the latent- guint- synthe- sis, StarGANv2 introduces class-specific mapping networks (one for each class) to project the class embedding into the shared latent space. Thus starGANv2 uses the class-specific mapping network to control the domains/classes. In this paper, follow the same setup starGANv2. In the first stage of our method, we only learn one mapping network, which is duplicated to initialize all the mapping networks (one for each class) of the second stage. Then stage) to train the well-initialized class-specific mapping networks.
the target dataset (second
B 121 TRANSLATION
B.1 TWO-CLASS 121 TRANSLATION RESULTS
To evaluate the generalization of our method, here we validate the proposed algorithm for two-class 12I translation on a two-category dataset: cat2dog-200 (Wang et al., 2021 with an image size of 256 256. In cat2dog-200, the training set composed of 200 images (100 images/per class) and the test set has 200 images (100 images/per class). We investigate two cases of 121 translation: the style representation from noise (i.e., latent-guided synthesis) and from the reference image (i.e., reference- frference-guided guided synthesis). As reported Tables 3 and compared to the transfer learning methods (i.e., Deep12I (Wang et 2020b) and Transfer121 (Wang al., (021)) we still maintain a large advantage.
B.2 MULTI-CLASS 121 TRANSLATION RESULTS
We also further validate the proposed algorithm for multi-class 12I translation on AFHQ-500 (Choi al. 2020) with image size of 256 256. In AFHQ-500, the training set is composed of 500 images (100 images/per class) and the test set has 1500 images (500 images/per class). We investigate two cases of 121 translation: the style representation is from noise (i.e., latent-guided synthesis) and from the reference image (i.e., diference-guides synthesis). As reported Tables compared to StarGANv2 we still maintain large advantage.
B.3 VARIANTS OF THE BASELINES
We adapt both Deepl21 WWang et al. 2020b) and Transfer121 (Wang et al. 2021 use very sim- ilar architecture to StarGANv2 (the architectural details are in Table 5, 6, 7 of StarGANy2 (Choi et al. 20200). Specially, devise the generator, we preserve the subnets of the discriminator of the pretrained StyleGAN as image encoder which has 16 x 16 x 512output, then additionally add ResBlks like StarGANy2, and finally use the subnets of the generator of the pretrained StyleGAN as
15

---

Published as a conference paper at ICLR 2022
dog2cat
cat2dog
FID
KID*IOO
FID
KID*I00
Deep121
154.6
6.97
194.6
24.53
Transfer121
137.2
6,48
182.1
24.14
Ours
60.17
4.83
86.4
5.46
Table 3:
The metric results of reference-guidee synthesis car2dog- 200 dataset. Note multiply 100 for
KID.
dog2cat
caf2dog
FID
KID*IOO
FID
KID*I00
Deep121 Transfer121
83.71
4.26
112.4
5.67
55.2
3.97
83.6
4,56
Ours
42.7
3,46
74.2
4.03
Table 4: The metric results of Latent-guided synthesis on cat2dog- 200 dataset. Note we multiply 100 for KID.
the decoder to output the image. We keep the mapping network of the pretrained StyleGAN for the 12I translation generator when performing latent-guided I2I translation. We adapt the discriminator of the pretrained StyleGAN the style encoder when performing reference-guides I2I translation. Similar to StarGANv2, we also use a multitask discriminator, which consists of multiple output branches. We use cat2dog-200 200 dataset to compare the adapted DeepI21 (Deep12I*), adapted Trans- fer121 (Transfer121s and ours. As reported in Tables ( and 7 compared to the transfer learning methods (i.e., Adapting Deep121 and Transfer121) still maintain large advantage.
B.4 ABLATION OF THE ENCODER
In this paper we use the pre-defined discriminator to initialize the encoder, since the discriminator of the StyleGAN is trained on HHFQ with 70k images, which optimizes it to an effective feature extractor. A similar technique was also explored in SGD (Shocher et al., 2020) and transfer121.
To verify that the previously reported results also hold for our method, also train both the en- coder and the reference encoder from scratch in reference-guide (10/per class). we achieve 130 of mFID, which lower than our method (mFII) 96.2). If we train the encoder from scratch, the training suffers from overfitting with limited training images.
translation on Animal faces dataset
C ADDITIONAL RESULTS
C.I DIVERSE OUTPUTS
Figure 6 shows some examples of the synthesized images three datasets. Taking Animal faces as example, given the target class label our method manages to generate high visual quality images.
C.2 KNOWLEDGE DISTILLATION FOR 12I SYSTEM IN THE FIRST STAGE
Figures 7 and show examples generated after transfer learning on 12I system. This shows that method successfully distills the style-mixing characteristics from the teacher generator to the student generator.
C.3 KNOWLEDGE DISTILLATION FOR X2I SYSTEM IN THE FIRST STAGE
Figures and show examples generated after semantic diversity loss. For each triplet, we depict the teacher's output(left), the student's output (middle) with condition c, and the one (right) with
16

---

Published as a conference paper at ICLR 2022
AFH1Q:500Lalent-guuuee synthesis)
syathesis)
FID
LPIPS
FID
LPIPS
SturGANv2
40.21
41.62
Ours
35.36
0.48
34.75
0.45
Table The metric results on AFHQ-500 dataset.
dog2cal
cal2dog
FID
KID>
K11>
Deepi21
189.3
15.4
205.6
29.2
Transfer121*
182.6
16.7
198.4
26.7
Ours
60.17
4.83
86.4
5.46
Table 6: The metric results of inference-guided synthesis cat2dog-200 dataset. Note we multiply 100 for KID.
condition cy. This shows that our semantic diversity loss manages to diversify the output given the input condition c.
C.4 ADDITIONAL RESULTS
We additionally depict the generated images in Figures 12 24
17

---

Published as a conference paper at ICLR 2022
dog2cat
car2dog
FID
KID#I00
FID
KID
Deep121
210.7
20.7
21.8
Transter|21
197.8
19.4
182.4
20.7
Ours
42.7
2.46
74.2
4.03
Table The metric results of Latent Latent-guided symthesis on cat2dog- 200 dataset. Note we multiply 100 for KID.
Input
Maltese dog
Blenheim spaniel
Boston bull
Sussex spaniel
Siberian husky
Tabby
Lion
American black bear
Meerkat
Grilled eggplant
Roll bread
Sauteed spinach
Rice gratin
Input
California Gull (Adult)
Cro.
Figure 6: Qualitative results the proposed method on multi-class translation task. Ame. Amerrcaa Goldfinch (Breeding Malc), Red Cro.:Red Crossbill (Adult Malc), Ind. Bun.: Indigo Bunting (Adult Malc).
18

---

Published as a conference paper at ICLR 2022
Input
Referance
Input
Reference
Curput
Figure 7: Examples generated after transfer leaning for 121 system. The student model is StarGANv2, Note it under reference-guide mode.
19

---

Published as a conference paper at ICLR 2022
Input
Figure 8: Examples generated after transfer leaning for 121 system. For one input, we sample two noises to generate images. The student model is StarGANy2. Note is under latent-guided mode.
20

---

Published as a conference paper at ICLR 2022
Output (teacher)
Output (student,
(stuciet,9a
Output (teacher)
Output (student,r)
Ourput (student,S)
Figure 9: Examples generated after semantic diversity loss. For each triplet, we depict the teacher's out- plt(left), the student's output (middle) with condition e. and the one (right) with condition cy. Note the student model is SSA-GAN.
21

---

Published as a conference paper at ICLR 2022
(teacher)
Output (student, ()
Output (teacher)
Output (student,r)
Ourput (student,5)
Examples generated after semantic diversity loss. For each triplet, we depict the teacher's out- plt(left), the student's output (middle) with condition e, and the one (right) with condition cy. Note the student model is S2IGAN.
Figure 10:
22

---

Published as a conference paper at ICLR 2022
Output (teacher)
Output (student,
Output (student,
Output (teacher)
Output (student,cg)
Output (student,Ag P
Figure 11: Examples generated without semantic diversity loss for tasks: audio2image (top) and text2image (bottom). For each triplet, we depict the teacher's output, the student's output with condition and the one with condition Cy.
23

---

Published as a conference paper at ICLR 2022
Input
SSA-GAN
Ours
GT
This bird is red and black in color with a black beak and red eye rings
This bird has a red breast, with red and white belly
A black bird with shiny green breast.
The bird has a long curved black bill and tan belly
This bird has wings that grey and has a white belly
A small bird with short, flat beak and vibrant blue body
This yellow bird's wings appear black and white in color
This bird has wings that are black and has a yellow belly
Figure 12: Qualitative results text-to-image synthesis task.
24

---

Published as a conference paper at ICLR 2022
Input
S2IGAN
Ours
GT
Spoken description
The petals on this flower are purple with yellow
stamen
flower with bright petals stripes of black
This flower has petals that are yellow and has black lines
This flower has pointy long yellow petals and one sepal
This flower has petals that are white with yellow patch
The flower shown has thin yellow petals as its main feature
The petals are pink and rounded and cover the pistil
This flower has light pink petals and bell shape
Figure 13: Qualitative results on audio-to-image synthesis task.
25

---

Published as a conference paper at ICLR 2022
Input
SPADE
Ours
GT
TH
FOR
ET
si
Figure 14: Qualitative results on segmentation map-to-image synthesis task.
26

---

Published as a conference paper at ICLR 2022
Input
Reference
SEAN
Ours
S
sa
LE
Figure 15: Qualitative results with reference images segmentation map-to-image synthesis task.
27

---

Published as a conference paper at ICLR 2022
Figure 16: Qualitative results on image-to-ing to-iage synthesis task. We translate the input image (top left) into all categories. Please zoom-in for details.
28

---

Published as a conference paper at ICLR 2022
Input
Refe.
Output
Reco.
Input
Refe.
Output
Reco.
Input
Refe.
Output
Reco.
Figure 17: Qualitative results with reference image for image-to-image synthesis task on Animal datasct
Refe:: reference image, Reco.: reconstructed image.
29

---

Published as a conference paper at ICLR 2022
Input
Refe.
Output StarGANv2
Output Ours
Input
Refe.
StarGANv2
Output
Output Ours
Input
Refe.
Output StarGANw2
Output Ours
Figure 18: Qualitative comparison Animal faces datascts.
30

---

Published as a conference paper at ICLR 2022
Figure 19: Qualitative results on image-to- to-image synthesis task. We translate the input image (top left) into
all 555 categories. Please zoom-in for details.
31

---

Published as a conference paper at ICLR 2022
Input
Refe.
Output
Reco.
Input
Refe.
Output
Reco.
Input
Refe.
Reco. Output
Figure 20: Qualitative results with reference image for smage-lo-image synthesis task on Birds dataset. Refe.:
reference image, Reco.: reconstructed image
32

---

Published as a conference paper at ICLR 2022
Input
Refe.
Output StarGANv2
Output Ours
Input
Refe.
StarGANv2
Out
Output Ours
Input
Refe.
Output StarGANv2
Output Ours
Figure 21: Qualitative comparison on Birds datasets.
33

---

Published as a conference paper at ICLR 2022
Figure 22: Qualitative results image-lo-imagee synthesis task. We translate the input image (top left) into
all 256 categories. Please zoom-in for details.
34

---

Published as a conference paper at ICLR 2022
Input
Refe
Output
Reco
Input
Refe.
Output
Reco.
Input
Refe.
Output
Reco.
Figure 23: Qualitative results with reference image image-to-image synthesis task Foods dataset. Refe.: reference image, Reco.: reconstructed image
35

---

Published as a conference paper at ICLR 2022
Input
Refe.
Output StarGANv2
Output Ours
Input
Refe.
StarGANv2
Output
Output Ours
Input
Refe.
Output StarGANv2
Output Ours
Figure 24: Qualitative comparison Foods datasets.
36


Reviews:
[{"review_id": "SxmYH8Smt0f", "reviewer": "Reviewer_BBix", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-------------------\nStrengths\n-------------------\n- Authors evaluated their approach on multiple datasets and multiple tasks. Demonstrated results prove that qualitatively the model is comparable to modern state-of-the-art such as TransferI2I.\n-------------------\nWeaknesses\n-------------------\n- The proposed approach for transfer learning is obviously inspired by TransferI2I. A proper definition of main contributions to the TransferI2I model for the image-to-image translation problem will make the paper more sound.\n- The provided comparison with the baselines is limited and doesn't present modern image-to-image translation models, e.g. [a,b]. \n[a] Schönfeld, Edgar, et al. \"You only need adversarial supervision for semantic image synthesis.\" International Conference on Learning Representations. 2020.\n[b] Zhou, Xingran, et al. \"CoCosNet v2: Full-Resolution Correspondence Learning for Image Translation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n- The paper is hard to follow, and contains many vague sentences, e,g. Section 3.3, page 5:\nHowever, when naively applying this for knowledge distillation the conditional information can still be ignored\nWhy conditional information can be ignored?\n- The semantic diversity loss is declared to be novel in the abstract. Yet, equation (7) demonstrates that it is a small modification of a loss from paper  by (Yang et al., 2019). The modification is described as 'solely change the conditional terms'. A more detailed explanation why conditional terms must be changed, will significantly improve the description of the proposed semantic diversity loss.\n\n", "summary_of_the_paper": "The paper presents a framework for knowledge distillation in image-to-image translation and arbitrary domain-to-image (e.g., text-to-image) translation. Authors claim following contributions: (1) a unified transfer learning method that leverages only synthetic images, (2) a semantic diversity loss, (3) a style-mixing triplets representing a tuple that includes an input image, a reference image, and a target image.\nAuthors provide results for image-to-image translation tasks and for text-to-image translation tasks. A limited comparison with several modern models is presented. Authors claim that the proposed transfer learning method outperforms modern state-of-the-art.", "summary_of_the_review": "The paper would benefit from the following improvements:\n- Clear definition of main technical contribution in the Introduction\n- Evaluation of the semantic-to-image task in terms of mIoU (i.e. comparison of the input segmentation with the results of a segmentation model on a synthesised image)\n- Extended explanation of the semantic diversity loss\n- Extended evaluation of the semantic-to-image task with modern models such as [a,b]", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "STzz78HamXy", "reviewer": "Reviewer_76so", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this submission is interesting. It is well written and clearly explains the ideas. I really like the figures which are clean and helpful in understanding the ideas of the paper and performance of the method. I am impressed by the variety of experiments, which answered the questions which arose to me when understanding the method. In the following part, I will detail my comments point by point.\n\nPROS\n1. The problem that this submission aims to solve, how to extend the application range of transfer learning for images, is important. And the authors make a step forward.\n2. The proposed method is simple and easy to implement, which is also painless to understand.\n3. The writting is quite good and I can follow it easily.\n\nCONS\n\n1. While the authors present a number of quantitative comparisons with existing methods, I find the corresponding qualitative results are limited. I can only see Fig. 3 in the submission file. Please attach more visual comparison results to highlight your advantages.\n2. How to evaluate the results of each method is quite subjective. And there is not a canonical metric for quantitative evaluations. Thus I suggest the authors do a user study here to reduce the unwanted subjective influences.\n3. Currently, this project performs the experiments on the Animal faces, Birds and Foods datasets, which are not the commonly used datasets in many I2I methods. However, with the default settings, they have the best performances on their chosen dataset. So the authors might need to do some experiments on those datasets with corresponding metrics for a fairer comparison.\n4. There are also not qualitative comparison figures for tasks of audio2image and text2image.\n5. In Fig. 3, I cannot see too much visual improvement especially for the bird figures. For example, StarGAN seems to generate better details. On the $4$-th row,  the feather colors of the proposed method are brighter than StarGAN. \n6. Also, another issues is that when the target dataset becomes bigger and more complex, the expected advantages become smaller and the developed method even behaves worse than previous methods (e.g. in Tab. 1 for Foods). This makes me hesitate the practical values of this method when working in the real world, which is very complicated.\n7. Are there some examples to support the claim about the style mixing characteristics? Or can the authors present some citations to support it?\n8. In Sec. 3.2 & Sec. 3.3, why should use the pre-defined discriminator to initialize the encoder and optimize it? Are there some specific motivations to do so? I assume more discussions are needed here to make it more self-contained.\n9. I am looking forward to more special designs when trying to remove the constraint that requires the GAN architecture within the X2I system to be exactly the same as that of the pretrained GAN, instead of justing using several layers for dimension mapping.", "summary_of_the_paper": "In this paper, the authors present a unifield learning method to investigate the knowledge transfer for X2I translation. Compared to existing methods, there are two advantages here: (1) this framework can be used for varying kinds of conditional image synthesis tasks; (2) it relieves the limitation for student generator to be the same as the pre-trained GAN. Specifically, the authors leverage the style mixing characteristics of high-quality GANs and introduce the semantic diversity loss to achieve it. Many experiments are conducted with good qualitative and quantitative results.", "summary_of_the_review": "I think the paper could be strengthened by further analysis, experiments, and presentations fixing. However, I like the general idea and I am favorable to accept it considering a large number of work they have done and signifance of this problem.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "B2Xel8fxPWK", "reviewer": "Reviewer_EUGu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(Update: after reading the reviews and the authors' reply, I would recommend the paper to be accepted to the conference. I'm raising the score from 6 to 8.)\n\nStrengths:\n1. The proposed method obtains better image quality against prior work in various settings, as suggested by lower FIDs.\n2. Strong results on X2I translation tasks are shown, indicating that the distillation method is effective for X2I tasks.\n3. The method improves image translation performance when training on fewer labeled samples.\n\nWeaknesses:\n1. For clarification purposes, after the second stage of training of I2I models, can the model explicitly control domains/classes in the latent-guided synthesis, just as how starGANv2 is set up?\n2. The clarity of writing can be improved. For example, it takes me a while to understand the setup for the entire training procedure. It might be better to mention more clearly that the distilled model is used as a pre-trained model for the downstream task.\n3. Would it be possible to extend the ablation study (Figure 4) to more datasets and settings? It would be more convincing to the reader if the trend is similar in various settings.\n\nAdditional questions:\n1. During training, I am wondering if the style-mixed triplets are pre-computed and stored, or randomly sampled on the fly. If it is the former case, maybe one can check if increasing the number of stored style-mixed triplets can eliminate the need for knowledge distillation.", "summary_of_the_paper": "The work introduces a flexible method to distill the knowledge of unconditional GANs of images to various image translation tasks, including image-to-image, text-to-image, and audio-to-image translation. While prior work requires using the same architecture of the pre-trained GANs to train the downstream im2im tasks, the proposed method supports distillation on a wide range of X-to-image translation architectures (e.g. starGANv2). The authors have compared their methods with prior work and showed that the proposed method obtained better FIDs.", "summary_of_the_review": "This work introduces a nice idea that improves image translation methods by distilling the knowledge of unconditional GANs. The setup is more flexible compared to prior work, where different architectures for image translation can be used. While the clarity of the writing can be improved, I think it is good to accept this paper to the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of distilling the knowledge present in different GAN-based image generation tasks. The paper received mixed reviews. The reviewers had difficulty understanding some details regarding the approach, and requests for ablations and clarifications on existing empirical evaluation. The authors provided a strong thoughtful rebuttal that addressed many of those concerns. The paper was discussed and two reviewers updated their reviews in the post-rebuttal phase. Reviewers generally agree that the paper should be accepted but still have concerns regarding contribution and writing. AC agrees with the reviewers and suggests acceptance. However, the authors are urged to look at reviewers' feedback and incorporate their comments in the camera-ready.
Decision: Accept (Poster)