Distilling GANs with Style-Mixed Triplets for X2I Translation with Limited Data

1 INTRODUCTION . Conditional image synthesis , also X2I translation , maps from an input domain ( e.g . text , audio , segmentation maps , etc . ) to the image domain . Benefiting from GANs ( Goodfellow et al. , 2014 ) and its follow-up improved versions ( Gulrajani et al. , 2017 ; Kang & Park , 2020 ; Salimans et al. , 2016 ) , they obtain remarkable performance on a wide variety of image synthesis tasks : image to image ( I2I ) ( Lee et al. , 2018 ; Zhu et al. , 2017 ) , audio to image ( Chen et al. , 2017 ; Wang et al. , 2020a ) , text to image ( Hu et al. , 2021 ; Li et al. , 2019 ; 2020 ; Radford et al. , 2021 ; Zhang et al. , 2017a ) and semantic segmentation map to image ( Isola et al. , 2017 ; Wang et al. , 2018 ) . Despite impressive leaps forward for a variety of image synthesis tasks , there are still important challenges . Specifically , to obtain good results , existing works rely on large labelled datasets . Labeling these datasets is both laborious and time-consuming , considerably reducing the practical impact of these methods . It is noteworthy to observe that many of these models ( Zhang et al. , 2017a ) apply transfer learning to the text and audio encoders ( e.g . using a pretrained LSTM ( Reed et al. , 2016 ) model for text and pretrained GRU ( Merkx et al. , 2019 ) model for audio ) , however they train the image synthesis decoder from scratch . This happens because there are no established methods to transfer pretrained GANs to conditional image decoders ; an omission which we aim to address in this paper . In this paper , we investigate knowledge transfer for a variety of conditional image synthesis tasks . Traditional knowledge transfer for conditional image synthesis is often not possible , because there might not be a pretrained network available for the desired translation task ( e.g . at the moment no high-quality pretrained network for segmentation map-to-image translation is available ) . It would therefore be preferable if the wide variety of high-quality GANs available for image generation could be exploited for X2I . Recent works ( Wang et al. , 2021 ; 2020b ) leveraged a pretrained GAN to initialize an I2I translation model , managing to transfer knowledge to different image synthesis tasks . These methods , however , suffer from three problems : ( I ) They can only be used for I2I translation and do not generalize to other conditional image synthesis tasks . ( II ) They are GAN architecture-specific approaches , requiring the GAN architecture within the X2I system to be exactly the same as that of the pretrained GAN . This limits transfer to current state-of-the-art X2I systems for which no similar GAN architecture exists ( like for example StarGANv2 ( Choi et al. , 2020 ) for I2I ) . ( III ) X2I systems are based on a conditional GAN , however existing methods for knowledge transfer for I2I do not initialize the conditional branch during the transfer , and therefore this has to be learned from scratch during the finetuning on the target dataset . Learning this on small target datasets can be problematic . To address the aforementioned problems , we propose several improvements for knowledge transfer to X2I systems : ( I ) we are the first to investigate knowledge transfer for X2I translation . Therefore , we propose a novel , unified transfer learning method , which can be used for varying kinds of conditional image synthesis tasks ( Figure 1 ) which is based on generated images and therefore does not require any real data . ( II ) The student generator does not need to have the same architecture as the pretrained GAN . Therefore , we can use well-devised specific image synthesis architectures ( e.g. , SPADE ( Park et al. , 2019 ) and StarGANv2 ( Choi et al. , 2020 ) ) by distilling knowledge from the pretrained teacher GAN ( e.g. , StyleGAN ) to the task-specific student GAN . ( III ) We use the style mixing characteristic of StyleGAN to create style-mixed triplet data , which are used to transfer the knowledge efficiently to both I2I and X2I translation models . Furthermore , we propose a semantic diversity loss based on the style-mixed triplet , which contributes to learn the semantic information of the output image . We perform experiments on a wide variety of image synthesis tasks , including text-to-image , audioto-image , segmentation map-to-image and I2I translations . We demonstrate the efficiency of the proposed knowledge distillation method , providing qualitative and quantitative results . We prove that the single pretrained GAN model can be universally used in varying specific task model . Additionally , leveraging the style mixing character of StyleGAN , further improves I2I translation performance . 2 RELATED WORK . GAN-based Conditional Image Synthesis . Benefiting from the advances in GANs and its variants in recent years , conditional image synthesis ( also called X2I translation ) research has developed rapidly . Two typical approaches have been investigated for GAN-based image synthesis , namely , unsupervised ( Kim et al. , 2017 ; Yi et al. , 2017 ) and supervised image generation ( Park et al. , 2019 ; Zhang et al. , 2017b ) . The latter inputs conditional information ( e.g . text , audio , image , segmentation map etc . ) to synthesize images which contain the corresponding semantic information ( i.e . the conditional information ) . Specifically , text-to-image translation ( Hu et al. , 2021 ; Li et al. , 2020 ; Zhang et al. , 2017a ) aims to synthesize high-realistic images which are semantically consistent with the text descriptions . Recent work ( Hu et al. , 2021 ) introduces semantic-spatial batch normalization to better exploit the text information . Similar to text-to-image translation , both audio-to-image translation ( Chen et al. , 2017 ; Wang et al. , 2020a ) and segmentation map-to-image translation ( Bau et al. , 2020 ; Park et al. , 2019 ) aim to learn a mapping from the audio/segmentation map to the output image . Different to the above image synthesis tasks , image-to-image translation ( Park et al. , 2020 ; Zhu et al. , 2017 ) performs projection from the source to the target image domain . In this paper , we explore transfer learning from GANs to a variation of conditional image synthesis tasks . Transfer learning . A considerable research effort has investigated transferring knowledge for both discriminative ( Donahue et al. , 2014 ; Hinton et al. , 2014 ) and generative tasks ( Noguchi & Harada , 2019 ; Zhao et al. , 2020 ) . There also exist several approaches ( Goetschalckx et al. , 2019 ; Jahanian et al. , 2020 ) which focus on the image manipulation based on the pretrained GAN . Given a target semantic attribute they aim to manipulate the output image of a pretrained GAN . However , these methods do not focus on transfer learning for target data . Some other methods ( Abdal et al. , 2019 ; Zhu et al. , 2020a ) reverse the given image into the input latent space of the pretrained GAN ( e.g. , StyleGAN ) , and manage to restructure the target image by optimization of the latent representation . Recent work ( Shocher et al. , 2020 ; Wang et al. , 2021 ; 2020b ) performed knowledge transfer from a pretrained classification model ( e.g. , VGG ( Simonyan & Zisserman , 2014 ) ) or the discriminator ( BigGAN ( Brock et al. , 2019 ) ) for I2I translation . However , both DeepI2I ( Wang et al. , 2020b ) and TransferI2I ( Wang et al. , 2021 ) require that the GAN architecture is identical with the generator used in the I2I architecture . As a consequence , these methods can not be applied to well-designed I2I translation architectures ( like starGANv2 ( Choi et al. , 2020 ) ) since they do not use a standard GAN architecture . The proposed method could address these problems . 3 KNOWLEDGE TRANSFER FOR X2I . Problem setting . Our goal is to transfer knowledge from a pretrained high-quality unconditional GAN to an X2I translation system for the case when training data is limited . The proposed method consist of two stages . In the first stage we perform the data-free knowledge transfer method explained in Sec . 3.2 and 3.3 . This stage does not require any target data . In the second stage , we apply a standard finetuning of this distilled model on the target dataset . Distillation can be used to perform knowledge transfer to other architectures , however , there is little target data available . Fortunately , the pretrained GAN can generate infinite data for data distillation , meaning that we do not require access to any real data ( i.e. , data-free ) . Also , we transfer knowledge from an unconditional GAN ( e.g. , StyleGAN ) to a conditional GAN ( e.g. , I2I translation ) ; requiring us to propose techniques to initialize the conditional branch . Exploiting the style-mixed characteristic ( Sec . 3.1 ) of StyleGAN we propose two solutions : one for I2I ( Sec . 3.2 ) and another one for X2I ( Sec . 3.3 ) . Our method consists of two stages : ( 1 ) transfer leaning without any real data ( i.e. , style-mixed triplets in Sec . 3.1 , I2I translation in Sec . 3.2 and X2I translation in Sec . 3.3 ) and ( 2 ) finetunning with the real target data . 3.1 STYLE-MIXED TRIPLETS Benefiting from the style mixing ( ( Figure 2 ( a ) ) ) characteristic of StyleGAN , we are able to create an infinite amount of triplet images from any of the two domains . Let zt ∈ RZ indicates the input noise of the pretrained generator GT ( teacher ) . given the input noises z1t and z 2 t , the mapping network MT of the StyleGAN generator encodes them to a style vector s1t = MT ( z 1 t ) and s 2 t = MT ( z 2 t ) . We define GT ( z ) = G′T ( MT ( z ) ) where G ′ T takes styles vectors as an input to each of its layers . We further feed these style vectors to the teacher generator G′T to output x 1 t = G ′ T ( s 1 t ) and x 2 t = G′T ( s 2 t ) ( where x 1 t , x 2 t ∈ X and image domain X = RH×W×3 ) respectively . Here the function Φ ( s1t , s 2 t ) selects to forward s 1 t to the layers of G ′ T important for the content of the generated image ( the bottom layers ) , and s2t to those important for the style ( the top layers ) 1 . Interestingly , a new image yt = G′T ( Φ ( s 1 t , s 2 t ) ) ( where yt ∈ Y and image domain Y = RH×W×3 ) contains mixed information with respect to content and style of both inputs x1t and x 2 t . Based on this observation above , we propose to leverage these style-mixed triplets ( x1t , x 2 t , yt ) to perform distillation for the I2I and X2I translation models without the need of any real triplet training data . 2

Summary: In this paper, the authors present a unifield learning method to investigate the knowledge transfer for X2I translation. Compared to existing methods, there are two advantages here: (1) this framework can be used for varying kinds of conditional image synthesis tasks; (2) it relieves the limitation for student generator to be the same as the pre-trained GAN. Specifically, the authors leverage the style mixing characteristics of high-quality GANs and introduce the semantic diversity loss to achieve it. Many experiments are conducted with good qualitative and quantitative results.