Data Instance Prior for Transfer Learning in GANs

1 INTRODUCTION . Generative Adversarial Networks ( GANs ) are at the forefront of modern high-quality image synthesis in recent years ( Brock et al. , 2018 ; Karras et al. , 2020b ; 2019 ) . GANs have also demonstrated excellent performance on many related computer vision tasks such as image manipulation ( Zhu et al. , 2017 ; Isola et al. , 2017 ) , image editing ( Plumerault et al. , 2020 ; Shen et al. , 2020 ; Jahanian et al. , 2020 ) and compression ( Tschannen et al. , 2018 ) . Despite the success in large-scale image synthesis , GAN training suffers from a number of drawbacks that arise in practice , such as training instability and mode collapse ( Goodfellow et al. , 2016 ; Arora et al. , 2017 ) . It has been observed that the issue of unstable training can be mitigated to an extent by using conditional GANs . However , this is expected as learning the conditional model for each class is easier than learning the joint distribution . The disadvantages of GAN training have prompted research in several non-adversarial generative models ( Hoshen et al. , 2019 ; Bojanowski et al. , 2018 ; Li & Malik , 2018 ; Kingma & Welling , 2014 ) . These techniques are implicitly designed to overcome the mode collapse problem , however , the quality of generated samples are still not on par with GANs . Current state-of-the-art deep generative models require a large volume of data and computation resources . The collection of large datasets of images suitable for training - especially labeled data in case of conditional GANs - can easily become a daunting task due to issues such as copyright , image quality and also the training time required to get state-of-the-art image generation performance . To curb these limitations , researchers have recently proposed techniques inspired by transfer learning ( Noguchi & Harada , 2019 ; Wang et al. , 2018 ; Mo et al. , 2020 ) and data augmentation methods ( Karras et al. , 2020a ; Zhao et al. , 2020b ; Zhang et al. , 2019 ) . Advancements in data and computation efficiency for image synthesis can enable its applications in data-deficient fields such as medicine ( Yi et al. , 2019 ) where labeled data generation can be difficult to obtain . Transfer learning is a promising area of research ( Oquab et al. , 2014 ; Pan & Yang , 2009 ) that leverages prior information acquired from large datasets to help in training models on a target dataset under limited data and resource constraints . There has been extensive exploration of transfer learning in classification problems that have shown excellent performance on various downstream data-deficient domains . Similar extensions of reusing pre-trained networks for transfer learning ( i.e . fine-tuning a subset of pre-trained network weights from a data-rich domain ) have also been recently employed for image synthesis in GANs ( Wang et al. , 2018 ; Noguchi & Harada , 2019 ; Mo et al. , 2020 ; Wang et al. , 2020 ; Zhao et al. , 2020a ) in the limited data regime . However , these approaches are still prone to overfitting on the sparse target data , and hence suffer from degraded image quality and diversity . In this work , we propose a simple yet effective way of transferring prior knowledge in unsupervised image generation given a small sample size ( ∼ 100-2000 ) of the target data distribution . Our approach is motivated by the formulation of the IMLE technique ( Li & Malik , 2018 ) that seeks to obtain mode coverage of target data distribution by learning a mapping between latent and target distributions using a maximum likelihood criterion . We instead propose the use of data priors in GANs to match the representation of the generated samples to real modes of data . In contrast to ( Li & Malik , 2018 ) , we use the images generated using data priors to find the nearest neighbor match to real modes in the generator ’ s learned distribution . In particular , we show that using an informative data instance prior in limited and large-scale unsupervised image generation substantially improves the performance in image synthesis . We show that these data priors can be derived from commonly used computer vision pre-trained networks ( Simonyan & Zisserman , 2014 ; Zhang et al. , 2018 ; Noguchi & Harada , 2019 ; Hoshen et al. , 2019 ) or self-supervised data representations ( Chen et al. , 2020 ) ( without any violation of the target setting ’ s requirements , i.e . ensuring that the pre-trained network has not been trained on few-shot classes in the few-shot learning setting , for instance ) . In case of sparse training data , our approach of using data instance priors leverages a model pre-trained on a rich source domain to the learn the target distribution . Different from previous works ( Noguchi & Harada , 2019 ; Wang et al. , 2020 ; 2018 ) which rely on fine-tuning models trained on a data-rich domain , we propose to leverage the feature representations of our source model as data instance priors , to distill knowledge ( Romero et al. , 2015 ; Hinton et al. , 2015 ) into our target generative model . We note that our technique of using data instance priors for transfer learning becomes fully unsupervised in case the data priors are extracted from self-supervised pre-trained networks . Furthermore , in addition to image generation in low data domain , we also achieve state-of-the-art Fréchet inception distance ( FID ) score ( Heusel et al. , 2017 ) on large-scale unsupervised image generation and also show how this framework of transfer learning supports several image editing tasks . We summarize our main contributions as follows : • We propose Data Instance Prior ( DIP ) , a novel transfer learning technique for GAN image synthesis in low-data regime . We show that employing DIP in conjunction with existing few-shot image generation methods outperforms state-of-the-art results . We show with as little as 100 images our approach DIP results in generation of diverse and high quality images ( see Figure 3 ) . • We demonstrate the utility of our approach in large-scale unsupervised GANs ( Miyato et al. , 2018 ; Brock et al. , 2018 ) achieving the new state-of-the-art in terms of image quality ( Heusel et al. , 2017 ) and diversity ( Sajjadi et al. , 2018 ; Metz et al. , 2017 ) . • We show how our framework of DIP by construction enables inversion of images and common image editing tasks ( such as cutmix , in-painting , image translation ) in GANs . We call our method a data instance prior ( and not just data prior ) , since it uses representations of instances as a prior , and not a data distribution itself . 2 RELATED WORK . Deep Generative Models In recent years , there has been a surge in the research of deep generative models . Some of the popular approaches include variational auto-encoders ( VAEs ) ( Rezende et al. , 2014 ; Kingma & Welling , 2014 ) , auto-regressive ( AR ) models ( Van Oord et al. , 2016 ; Van den Oord et al. , 2016 ) and GANs ( Goodfellow et al. , 2014 ) . VAE models learn by maximizing the variational lower bound of likelihood of generating data from a given distribution . Auto-regressive approaches model the data distribution as a product of the conditional probabilities to sequentially generate data . GANs comprise of two networks , a generator and a discriminator that train in a min-max optimization . Specifically , the generator aims to generate samples to fool the discriminator , while the discriminator learns distinguish these generated samples from the real samples . Several research efforts in GANs have focused around improving the performance ( Karras et al. , 2018 ; Denton et al. , 2015 ; Radford et al. , 2016 ; Karras et al. , 2020b ; 2019 ; Brock et al. , 2018 ; Zhang et al. , 2019 ) and training stability ( Salimans et al. , 2016b ; Gulrajani et al. , 2017 ; Arjovsky et al. , 2017 ; Miyato et al. , 2018 ; Mao et al. , 2017 ; Chen et al. , 2019 ) . Recently , the areas of latent space manipulation for semantic editing ( Shen et al. , 2020 ; Jahanian et al. , 2020 ; Zhu et al. , 2020 ; Plumerault et al. , 2020 ) and few-shot image generation ( Wang et al. , 2020 ; Mo et al. , 2020 ; Noguchi & Harada , 2019 ) have gained traction in an effort to mitigate the practical challenges while deploying GANs . Several other non-adversarial training approaches such as ( Hoshen et al. , 2019 ; Bojanowski et al. , 2018 ; Li & Malik , 2018 ) have also been explored for generative modeling , which leverage supervised learning along with perceptual loss ( Zhang et al. , 2018 ) for training such models . Transfer Learning in GANs While there has been extensive research in the area of transfer learning for classification models ( Yosinski et al. , 2014 ; Oquab et al. , 2014 ; Tzeng et al. , 2015 ; Pan & Yang , 2009 ; Donahue et al. , 2014 ) , relatively fewer efforts have explored this on the task of data generation ( Wang et al. , 2018 ; 2020 ; Noguchi & Harada , 2019 ; Zhao et al. , 2020a ; Mo et al. , 2020 ) . ( Wang et al. , 2018 ) proposed to fine-tune a pre-trained GAN model ( often having millions of parameters ) from a data-rich source to adapt to the target domain with limited samples . This approach , however , often suffers from overfitting as the final model parameters are updated using only few samples of the target domain . To counter overfitting , the work of ( Noguchi & Harada , 2019 ) proposes to update only the batch normalization parameters of the pre-trained GAN model . In this approach , however , the generator is not adversarially trained and uses supervised L1 pixel distance loss and perceptual loss ( Johnson et al. , 2016 ; Zhang et al. , 2018 ) which often leads to generation of blurry images in the target domain . Based on the assumption that source and target domain support sets are similar , ( Wang et al. , 2020 ) recently proposed to learn an additional mapping network that transforms the latent code suitable for generating images of target domain while keeping the other parameters frozen . We compare against all leading baselines including ( Noguchi & Harada , 2019 ; Wang et al. , 2020 ) on their respective tasks , and show that our method DIP outperforms them substantially , while being simple to implement . A related line of recent research aims to improve large-scale unsupervised image generation in GANs by employing self-supervision - in particular , an auxiliary task of rotation prediction ( Chen et al. , 2019 ) or using one-hot labels obtained by clustering in the discriminator ’ s ( Liu et al. , 2020 ) or ImageNet classifier feature space ( Sage et al. , 2018 ) . In contrast , our method utilizes data instance priors derived from the feature activations of self-supervised/supervised pre-trained networks to improve unsupervised few-shot and large-scale image generation , leading to simpler formulation and higher performance as shown in our experiments in Section 5.3 and Appendix A . Recently , some methods ( Karras et al. , 2020a ; Zhao et al. , 2020b ; Zhang et al. , 2019 ; Zhao et al. , 2020c ) have leveraged data augmentation to effectively increase the number of samples and prevent overfitting in GAN training . However , data augmentation techniques often times alter the true data distribution and there is a leakage of these augmentations to the generated image , as shown in ( Zhao et al. , 2020c ; b ) . To overcome this , ( Zhao et al. , 2020b ) recently proposed to use differential augmentation and ( Karras et al. , 2020a ) leveraged an adaptive discriminator augmentation mechanism . We instead focus on leveraging informative data instance priors , and in fact , show how our DIP method can be used in conjunction with ( Zhao et al. , 2020b ) to improve performance .

Summary: This submission deals with transfer learning for training GANs with limited label data. The challenge is that training with limited data can result in mode collapse. This submission proposes to use data priors for each instance of the target distribution, transformed through knowledge from a source domain, as conditional information in GAN to ensure mode coverage of the target data distribution. A pre-trained feature extractor is used to provide the information to condition the GAN. A range of experiments is performed with the features extracted from VGG16, SIMCLR. They show consistent improvements in the image quality and diversity, measured via FID and precision-recall,  for few-shot, limited data, and even large scale data settings.