Data Instance Prior for Transfer Learning in GANs
Authors: Puneet Mangla
Nupur Kumari
Mayank Singh
Vineeth N. Balasubramanian
Balaji Krishnamurthy
Venue:  (2021)
Keywords: GAN
transfer learning
fewshot learning
image generation

Abstract: Recent advances in generative adversarial networks (GANs) have shown remarkable progress in generating high-quality images. However, this gain in performance depends on the availability of a large amount of training data. In limited data regimes, training typically diverges, and therefore the generated samples are of low quality and lack diversity. Previous works have addressed training in low data setting by leveraging transfer learning and data augmentation techniques. We propose a novel transfer learning method for GANs in the limited data domain by leveraging informative data prior derived from self-supervised/supervised pre-trained networks trained on a diverse source domain. We perform experiments on several standard vision datasets using various GAN architectures (BigGAN, SNGAN, StyleGAN2) to demonstrate that the proposed method effectively transfers knowledge to domains with few target images, outperforming existing state-of-the-art techniques in terms of image quality and diversity. We also show the utility of data instance prior in large-scale unconditional image generation and image editing tasks.

Under review as a conference paper at ICLR 2021
DATA INSTANCE PRIOR FOR TRANSFER LEARNING IN GANS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in generative adversarial networks (GANs) have shown remark- able progress in generating high-quality images. However, this gain in perfor- mance depends on the availability of a large amount of training data. In limited data regimes, training typically diverges, and therefore the generated samples are of low quality and lack diversity. Previous works have addressed training in low data setting by leveraging transfer learning and data augmentation techniques. We propose novel transfer learning method for GANs in the limited data domain by leveraging informative data prior derived from seffssupervieed/superiiee pre- diverse source domain. We perform experiments on several standard vision datasets using various GAN architectures (BigGAN, SNGAN, StyleGAN2) to demonstrate that the proposed method effectively trans- fers knowledge to domains with few target images, outperforming existing state- of-the-art techniques terms of image quality and diversity. We also show the utility of data instance prior in large-scale unconditional image generation and image editing tasks.
trained networks trained
INTRODUCTION
Generative Adversarial Networks (GANs) are at the forefront of modern high-quality image synthe- sis in recent years (Brock et al.) 2018 Karras et al.) 2020b; 2019). GANs have also demonstrated excellent performance on many related computer vision tasks such as image manipulation (Zhu 2017: Isola et al. 2017), image editing (Plumerault et al. 2020; [hhen et al., 20200 Jahanian et al. al. 2020) and compression (Tschannen et al. 2018). Despite the success in large-scale image synthesis, GAN training suffers from a number of drawbacks that arise in practice, such as training instability and mode collapse (Goodfellow et al., 2016 Arora et al. 2017). It has been observed that the issue of unstable training can be miligated to extent by using conditional GANs. However, this is expected as learning the conditional model for each class is easier than learning the joint distribution. The disadvantages of GAN training have prompted research in several -ovaasreral generative models (Hoshen et al.) 2019 [Bojanowsii et al., 2018 Li & Malik] 2018] Kingma & Welling 2014). These techniques are implicitly designed to overcome the mode collapse problem, however, the quality of generated samples are still not on par with GANs.
Current state-of-the-art deep generative models require large volume of data and computation resources. The collection of large datasets of images suitable for training especially labeled data in case of conditional GANs can easily become daunting task due issues such copyright, image quality and also the training time required to get saute-of-the-ald image generation performance. To curb these limitations. researchers have recently proposed techniques inspired by transfer learning (Nouchhi & Harada 2019 Wang et al. 2018 Mo et al.) 2020) and data augmentation methods (Karras et al. (2220a] Zhao al. (020b] Zhang et al. 2019). Advancements in data and computation efficiency for image synthesis can enable its applications in data-deficient fields such medicine Yi ct al. 2019) where labeled data generation can be difficult to obtain.
Transfer learning is promising area of research (Oquab et al. 2014 Pan & Yang 2009) that lever- ages prior information acquired from large datasets to help in training models on a target dataset un- der limited data and resource constraints. There has been extensive exploration of transfer learning in classification problems that have shown excellent performance on various downstream data-deficient domains. Similar extensions of reusing pre-trained networks for transfer learning (i.e. fine-tuning

---

Under review as a conference paper at ICLR 2021
subset of pre-trained network weights from data-rich domain) have also been recently employed for image synthesis in GANs (Wang et al. 2018: Noguchi & Harada 2019; Mo et al. 20200 Wang et al. 2020 Zhao et al. 2020a) in the limited data regime. However, these approaches are still prone to overfitting on the sparse target data, and hence suffer from degraded image quality and diversity.
In this work, propose simple yet effective way of transferring prior knowledge in unsuper- vised image generation given a small sample size (~ 100-2000) of the target data distribution. Our approach is motivated by the formulation of the IMLE technique (Li & Malik 2018) that seeks to obtain mode coverage of target data distribution by learning a mapping between latent and target dis- tributions using a maximum likelihood criterion. We instead propose the use of data priors in GANs to match the representation of the generated samples to real modes of data. In contrast to (Li & Ma- lik, 2018), we use the images generated using data priors to find the nearest neighbor match to real modes in the generator's s learned distribution. In particular, we show that using an informative data instance prior in limited and large-scale unsupervised image generation substantially improves the performance in image synthesis. We show that these data priors can be derived from commonly used computer vision pre-trained networks (Simonyan & Zisserman 2014 Zhang et al., 2018 Noguchi & Harada 2019 Hoshen et al., 2019) or self-supervised data representations (Chen al. 2020) (without any violation of the target setting's 's requirements, i.e. ensuring that the pre-trained network has not been trained few-shot classes in the few-shot learning setting, for instance). In case of sparse training data, our approach of using data instance priors leverages a model pre-trained on a rich source domain the learn the target distribution. Different from previous works (Noguchi & Haradal 2019: Wang et al. 22020; 2018) which rely on fine-tuning models trained on a data-rich do- main, we propose to leverage the feature representations of our source model as data instance priors, to distill knowledge (Romero et al. 2015 Hinton et al. 2015) into our target generative model.
We note that our technique of using data instance priors for transfer learning becomes fully unsuper- vised in case the data priors are extracted from supe-vupe pre- pre-trained networks. Furthermore, in addition to image generation in low data domain, we also achieve state-of-the-art Fréchet incep- tion distance (FID) score (Heusel et al. (017) on large-scale unsupervised image generation and also show how this framework of transfer learning supports several image editing tasks.
We summarize our main contributions as follows:
We propose Data Instance Prior (DIP), a novel transfer learning technique for GAN image synthesis in low-data regime. We show that employing DIP in conjunction with existing few-shot image generation methods outperforms sfate-of-the-art results. We show with as little 100 images approach DIP results in generation of diverse and high quality images (see FigureB)
We demonstrate the utility of our approach in large-scale unsupervised GANs MMyyato et al. 2018 Brock et al. 2018) achieving the new tfate-of-the-art in terms of image quality (Heusel et al. 2017) and diversity (Sajjadi et al. 20188) Metz et al. (017).
We show how our framework of DIP by construction enables inversion of images and com-
mon image editing tasks (such as cutmix, in in-painting, image translation) in GANs.
We call our method a data instance prior (and not just data prior), since it uses representations of instances as a prior, and not data distribution itself.
2 RELATED WORK
Deep Generative Models In recent years, there has been a surge in the research of deep generative models. Some of the popular approaches include variational auto-encoders (VAEs) (Rezende al., 2014 Kingma& & Welling 2014), rego-sege (AR) models (Van Oord et al. 2016: Van den Oord let al. 2016) and GANs (Goodfellow et al., 2014). VAE models learn by maximizing the variational lower bound of likelihood of generating data from given distribution. Auto-regressive approaches model the data distribution as a product of the conditional probabilities to sequentially generate data. GANs comprise of two networks, generator and a discriminator that train in a min-max optimization. Specifically, the generator aims to generate samples to fool the discriminator, while the discriminator learns distinguish these generated samples from the real samples. Several research efforts in GANs have focused around improving the performance (Karras et al. 2018 D et al.)
2015 Radford et al. 2016 Karras al. 2020b 2019 Brock al. 2018 Zhang al. 2019) and
2

---

Under review as a conference paper at ICLR 2021
training stability (Salimans et al., 2016b Gulrajani et al. 2017 Arjovsky al. 2017: Miyato al. 2018 Mao et al. 2017; Chen et al. 2019). Recently, the areas of latent space manipulation for semantic editing (Shen et al.] 2020 Jahanian al. 2020; Zhu et al. 2020; Plumerault et al. 2020) and few-shot image generation (Wang et al. 2020 Mo et al. 2020 Noguchi & Harada 2019) have gained traction in effort to mitigate the practical challenges while deploying GANs. Several
other non-adversarial training approaches such as (Hoshen et al.] 2019; Bojanowski et al., 2018
Li & Malik 2018) have also been explored for generative modeling, which leverage supervised learning along with perceptual loss (Zhang et al. 2018) for training such models.
Transfer Learning in GANs While there has been extensive research in the area of transfer learn- ing for classification models YYsiinkii et al.) 2014 Oquab et al., 2014] Tzeng et al. 2015 Pan & Yang, 2009 Donahue et al. 2014). relatively fewer efforts have explored this on the task of data
generation (Wan et al. 2018 2020 Noguchi & Harada, 2019; Zhao et al. 2020a Mo et al. 2020). (Wang et al. 2018) proposed to fine-tune pre-trained GAN model (often having millions of pa- rameters) from a data-rich source to adapt to the target domain with limited samples. This approach, however, often suffers from overfitting as the final model parameters are updated using only few samples of the target domain. To counter overfitting, the work of (Noguchi & Harada 2019) pro- poses to update only the batch normalization parameters of the pre-trained GAN model. In this approach, however, the generator is not adversarially trained and uses supervised T, pixel distance loss and perceptual loss (Johnson et al. 2016 ZZang et al., 2018) which often leads to generation of blurry images in the target domain. Based on the assumption that source and target domain sup- port sets are similar, (Wang et al.) 2020) recently proposed learn an additional mapping network that transforms the latent code suitable for generating images of target domain while keeping the other parameters frozen. We compare against all leading baselines including (Noguchi & Harada 2019 Wang et al., (2020) on their respective tasks, and show that our method DIP outperforms them substantially, while being simple to implement.
A related line of recent research aims to improve large-scale unsupervised image generation in GANs by employing self-supervision in particular, an auxiliary task of rotation prediction (Chen)
jet a (2119) or using one-hot labels obtained by clustering in the discriminator's 's (Liu et al. 2020 or ImageNet classifier feature space (Sage et al., 2018). contrast, our method utilizes data instance priors derived from the feature activations of sefF-supeeriisedspprrvvssd pre-trained networks to improve unsupervised few-shot and large-scale image generation, leading to simpler formulation and higher performance some methods (Karras et al. 2020a Zhao et al. 2020b Zhang et al. [209 Zhao et al. (202cc) have leveraged data augmentation to effectively increase the number of samples and prevent overfitting in GAN training. However, data augmentation techniques often times alter the true data distribution and there is a leakage of these augmentations to the generated image, as shown in (Zhao et al., b). To overcome this, (Zhao et al. 2020b) recently proposed to use differential augmentation and (Karras et al.] 2020a) leveraged an adaptive discriminator augmentation mechanism. We instead focus on leveraging informative data instance priors, and in fact, show how our DIP method can be used conjunction with (Zhao et al. 2020b) to improve performance.
shown in experiments in Section 5.3 and Appendix A Recently,
PRELIMINARIES
We now present the related background on generative models that are essential for our methodology.
It consists of a generator G R XY + R2 and a discriminator D R. XY > [0, 1] which are trained ad adversarially to Iearn a target data distribution (xxly,, where x e R", E Y, the space of class labels. The goal of G to generate samples from noise z ~ p(z). z E R given a conditional label
Conditional GAN:
q(v) that matches the target distribution and the aim of D is to distinguish between real samples and those generated from G. Conditional GANs use class-level information y of data in the generator and discriminator. The standard hinge loss (Tran et al. 2017) for training
GANs is given by:
Lp Ey-a(o) [x x v) mmax (0, 1 D(x, v))]] + Byy-ats) [Ea-pin) [max((, 1 ((G v))
LG = - Eyrg(() [e
3

---

Under review as a conference paper at ICLR 2021
where the discriminator score (xx.)) 3) depends input image (either real or fake) and its class MMyytoo& & Koyama| 2018: Odena et al. 2017). Generally, the class information is passed into G through onc-hot vector concatenated with through conditional batch norm layers (De Vries et al. 2017 Dumoulin et al. 2016).
Implicit Maximum Likelihood Estimation IMLE (Li & Malik 2018 is non-adversarial tech- nique that uses a maximum likelihood criterion to train the generative model. During training, each real sample of the data distribution is matched a generated image through nearest neighbour search and the distance between the two is minimized. The optimization objective to update the parameters of network G' in each training step given by:
min 2m....1 x [II((z (x)) - x x1
where z min [[G(z) x12
Here, the inner optimization of finding latent code z (x) from the batch {zı..Zzm such that G(z" (x)) is nearest to the real image from the data distribution is implemented using the nearest neighbor search technique (Li & Malik (0177). The above objective promotes that each real example close to some generated sample from the trained generator.
4 DIP: PROPOSED METHODOLOGY
We propose a transfer learning framework DIP for training GANs that exploits knowledge extracted from seffssuuperrieedd/peeriiesd networks, pre-trained on a rich and diverse source domain in the form of data instance priors. It has been shown that providing class label information in GANs sig- nificantly improves training stability and quality of generated images as compared to unconditional setting MMyato & Koyama 2018 Chen et al. 2019). However, in practice, GANs observed to be prone to mode-collapse that is further exacerbated in case of sparse training data. We take mo- tivation from the reconstructive framework of IMLE (Li & Malik 2018) and propose to condition GANs on image instance priors that act as regularizer to prevent mode collapse and discriminator overfitting.
Knowledge Transfer in GAN GANs are a class of implicit generative models that minimize a divergence measure between the data distribution q(x) and the generator output distribution Giz) where 2 ~ p(z) denotes the latent distribution. Intuitively, this minimization of a divergence objec- tive ensures that each generated sample (iz) close to some data example x ~> q(x). However, this does not ensure the converse, i.e. that each data instance is close to generated sample, which can result mode dropping. To counter this, especially in limited data regime, we propose to update the parameters of the model so that each real data example is close some generated sample similar to (Li & Malik] (0188) by using data instance priors as conditional label in GANs. Moreover to enable transfer of knowledge, image features extracted from networks pre-trained on a large source domain are used as the instance level prior.
Given a pre-trained feature extractor C: R - Rd. x RP, which is trained source domain using supervisory signals or self-supervision, we use its output C(x) as condition in the generator. G is conditioned on C(x) using conditional batch-norm (Dumoulin et al. (016) whose input is Gems (C))), where Gemb is a learnable matrix. During training we enforce that G(z[C(xx) is discriminator feature space (similar Giz" (x)) being close x in Eq 2). Let the discriminator be D = D, D, (o denotes composition) where D is discriminator's 's last Teature layer and Dj is the final linear layer. To enforce the above objective we map C(x) to the dimension equal to discriminator's 's feature layer D, using trainable matrix Demb and minimize distance between Demo (xx)) and Dy of both real image x and generated image G(z) C(x)) in an adversarial manner. Hence, our final GAN training loss for the discriminator and generator is given by:
close to the real image
D(x, C(x))) + axpg 1 Ln = Ex=q(e)[max((0 -
C(x))
LG =
(3)
where
D(x,y)) Demi(y)) Dx + Dj Dy(x)
(4)
4

---

Under review as a conference paper at ICLR 2021
((x)
Pemb
Projection loss
Gamb
Training
G(((C())
D
Advennarid ial
Df
oo
108 Sumple framo BMM
Inference
21o1 Clustering
Zitj Somple from: the cluster centers with some cote
((x) P
Prior sample
Figure I: Overview of our proposed technique, Data Instance Priors (DIP) transfer learning GANs. Top: DIP training with feature C(x) of real sample
conditional prior in the conditional GAN framework of (Miyato Koyama 2018). C is a pre-trained network rich source domain from which we wish transfer knowledge. Bottom: Inference over trained GAN involves learning distribution over the set training data
prior {C) } to enable sampling of conditional priors.
Algorithm 1: Data Instance Prior Training (DIP)
Input:G, network with parameters Og and 0p, pre-trained model C for extracting prior condition, samples from real data distribution s(x) and latent distribution p(=), batch size b, number of training iterations, discriminator update steps datep for each generator update, Adam hyperparameters o, 81, $2. for number of training iterations do
for t: d...dapp do Sample batch i ~ q(x), ~ p(zz)
Jfahe Gx))C(x))
Calculate D(x, ((=)) Df(x) Dem)(((==) + D2
(emne)))DDe(C(a Calculate D (Zjane, C(+)) = D2 (2jaso) Demb (C(z)) + Dro (2(apt))
LD max(0, 1 (xx, C(x))) max(0,1 1 + DiTaaee C(x)))
Update On Adam(Lox a, 32)
10
end
Sample p(=)
12
Generate images Ijaae ((z)
13
Calculate (ITaae C(x)) = Dy (Tjate) Demo(C(())) + D2 Dj (Z
LG -Daaaaee C(x))
15
Update tG + Adam( a, 31,1,82)
16
end
17
return 0o.0p. AD.
In the above formulation, the first term in Eq. is the projection loss as in (Miyato & Koyama 2018)
between input image and conditional embedding of discriminator. Since conditional embedding is extracted from a pre-trained network, above training objective leads to feature level knowledge distillation from C. It also acts regularizer on the discriminator reducing its overfitting in the limited data setting as shown in Figure 2 The gap between discriminator score (D; Dy) of training and validation images keeps on increasing and FID quickly degrades for baseline model as compared to DIP when trained only 10% data of CIFAR- 100. Moreover, enforcing feature Dy (Zz) (x))) to similar to Demo (C(x) promotes that for each real sample, there exists a generated sample close to it and hence promotes mode coverage of target data distribution. We demonstrate that the above proposed use of data instance priors from pre-trained feature extractor, while designed for limited data setting, also benefits in large-scale image generation. Our overall methodology is illustrated Figure and pseudo code Algorithm
Random image generation at inference Given the training set Dimage {x;}リ=1 of sample size and its corresponding training data prior set Bprior {C(x,))} the generator requires access to D Pprsor for sample generation. In case of few-shot image generation where size of DD priior limited (~ 100), create more variations generate images conditioned interpolation of two randomly sampled prior i.e.
G(z|A P1 (1 - A))2)) where P1, P2 Dprior; X Uniform (0, 1]
(5)
5

---

Under review as a conference paper at ICLR 2021
DIP
Baseline
110
120
6-
Real/ ake soore
100-
Real
Readine
Real/t aike
Ganarated Valistatiin
64-
Real
Genernted
Validation
0-
6k
102
201
N00
10k
S0s
6lk
70k
06.
10%
26
302
40k
501
60k
70
0k
10L
20k
Trnining iterations
Training iterations
Training iterations
Figure 2: Comparison between DIP and Baseline when trained 10% data of CIFAR- 100. left: FID (in Pytorch) of baseline training starts increasing very early in training (around 15k) compared to FID of DIP training. middle: Discriminator score on training and validation images remain similar each other and consistently higher than score of generated images for DIP model. right: Discriminator score on training and validation images start diverging and training collapses for the baseline model.
In case of large-scale image generation, to avoid storing Dprior corresponding complete training set (possibly in order of millions), we propose to cluster (Sculley) 20100) or build a Gaussian Mixture Model (GMM) (Xu & Jordan, 1996) on Pprior and store only the cluster centroids and thus enable memory efficient sampling of conditional prior from the distribution fit during inference as follows:
G(zlu+ + ((0, Ie)) where μ ~ K-MeanSCentrrids((eeem (Drrroo) or ((z)N(p, Σ)) where μ., E ~ GMM (D (ppioor)
(6)
Controlled image generation through semantic diffusion We observed that high-level seman- tics (e.g, smile, hair, gender, glasses, etc in case of faces) of generated image, G(z[C(x)), relied on the conditional prior, C'x)). Complementarily, variations in the latent code z ~ (0,, induced fine-grained changes such skin texture, face shape, etc. This suggests that conditional prior, C(x)), to get some control over the image generation's 's high-level semantics. We show that by altering as our new input prior helps in generating samples with the desired attributes, as shown in Fig 5 In similar manner, DIP also allows generation of images with certain cues (like sketch image generation, as shown in Fig and Appendix). We note that the generation of samples at inference, this case, can simply be done by using C(x) as condition in G.
can exploit our
image x (through CutMix, CutOut, etc) and using C(x) of the altered image
EXPERIMENTS
We perform extensive experiments to highlight the efficacy of our data instance prior module DIP in unsupervised training based on SNGAN (Miyato al. 2018 Big-GAN (Brock et al., 20188) and StyleGAN2 (Karras et al. (020b) architectures. For extracting image prior information, we use the last layer activations of: (1) Vggló (Simonyan & Zisserman, 2014) classification network trained on ImageNet; and (2) SimCLR (Chen et al. 2020) network trained using self-supervision on ImageNet. We conduct experiments on few-shot (~ 25-100 images), limited (~ 2k-5k images) and large-scale (~ 50k-1M images) data settings. For evaluation, we use FID (Heusel et al. (017), precision and
recall scores (K Kyykäänniemi et al. 2019 to assess the quality and vonaaacovrrgee//iveesstty of the generated images.
5.1 FEW-SHOT IMAGE GENERATION
Baselines and Datasets For comprehensive evaluation, compare and augment our methodol- ogy DIP with training SN-GAN from scratch and the following leading baselines: Batch Statistics Adaptation (BSA) (Noguchi & Harada 2019), TransferGAN (Wang al., 2018), FreezeD (Mo) et al. 2020) and DiffAugment (Zhao et al.) 20206). In case of BSA, a non-adversarial variant, GLANN (Hoshen al. 2019) is used which optimizes for image embeddings and generative model

---

Under review as a conference paper at ICLR 2021
SN-GAN (128 128)
Method
Pre-training
Anime P1
Faees
Flower
FID
FID
P1
R1
R
From scratch
120.38
0.61
0.00
140.66
0.31
0.00
124.02
0,30
0.09
+ 1gg16
66.85
0.03
68.49
TransforGAN
102.75
0.70
0.00
101.15
0.85
0.00
113.35
0.71
0.09
+ Vigl
86.96
0.02
75.21
0.10
110.24
0.55
Freezell
109.40
0.67
0.00
107.83 83
0.83
0..00
91.80
0.69
0.14
+. Vggl6
93.36
0.56
0.03
77.09
0.68
0.14
120.43
0.53
DIP DIP-SimCL.R
89,39
0.46
0.025
70.40
0.74
0.22
120.13
0.63
0.33
DiffAugment
85.16
0.95
0.00
109.25
0.84
0.00
83.45
0.75
0.23
+ DIP 1
48.67
0.82
0.03
62.44
0.80
0.19
79.86
0.79
0.57
+ DIP SimCLR
52.41
0.77
0.04
64.53
0.78
0.22
87.92
0.75
0.54
BSA*
92.0
123.2
129.8
GLANN-DIP-Vg316
67.07
0.87
0.01
60.11
0.95
0.08
97.73
0.95
0.51
Table 1: Few-shot image generation results using 100 training images (J: lower is better; t: higher better). Precision and Recall scores are based on (Kynkäänniemi et al. 2019). FID is computed between 10k, 7k, 5k generated and 10k, 7k, 251 real samples for Anime, Faces and Flower respectively. denotes directly reported from the paper.
Big-GAN x 125)
Method
Places2.5k FID
FFHQZk
CUB6k
FID.
RID
MineGAN
75,50
75.91
69.64
TranderGAN
162.9
126.23
138.87
+DIP-Vge16
57.35
44.43
23.37
Freezel)
161.87
142.47
+ +DII-Vae16
50.58
43,90
26.90
DiffAugment 30.76
+ -DIP-Vag16
23.19
15.81
DIP-SimCLR 26.65
21.06
12.36
Table 2: Limited data image generation using different ap- proaches. FID (lower hetter) computed hetween 10k, 6k generated and real samples (disjoint from training set) for Places2.5%, FFHQZK, CUB datasets respectively. BigGAN pre- trained ImageNet fine tuned in all approaches.
Figure 100-shot image interpolation between instance-level prior for DiffAugment (Zhao et al. 2020b] (Rows 1.3,5) and Dif'Augment Vgglo (Rows 2.4.6 Anime, Faces and Flower datasets re- spectively.
through perceptual loss] We use our data priors distill knowledge over these image embeddings. For more training and hyperparameter details, please refer to Appendix A
We perform experiments randomly chosen 100 images at 1 188 x 128 resolution from: (1) Anime
(2) FFHQ Karraa et al., 2019) and (3) Oxford 102 flowers (Nilsback & Zisserman 2008) (we restrict to the Passion flower class following (Noguchi & Harada 2019), to avoid overlap with ImageNet classes) datasets. The above datasets are chosen to ensure that there is no class label intersection with finetune SNGAN pre-trained
ImageNet as done in (Noguchi & Harada 2019). We also show additional results at 256 x 256 resolution as well additional datasets (Pandas, Grumpy Cat, Obama) with StyleGAN-2 (Karras et al. 2020b)
ImageNet classes. For methods with pre-training,
in Appendix A
Results Using DIP shows consistent improvement in FID and Recall over all baseline methods as shown in Table 1 Fig 3 shows samples generated via interpolation between conditional embedding of models trained via DIP-Vgg on DiffAugment and vanilla DiffAugment. These results show qualitatively the significant improvement obtained using our DIP-based transfer learning approach. Comparatively, the baseline, vanilla DigfAugment, fails to generate realistic interpolation and for
'The code provided with BSA was not reproducible, and hence this choice www..gwer.neet. net/ Dat.emmmnee/aaaoooo2225
7

---

Under review as a conference paper at ICLR 2021
Method
Inference
CIFAR-IU
CIFAR-IOU
FEHQ P
LSUN-Bedroom
Imaaee NelSAa32
FID
Pt
R
FID
P
R.
FID
FID
P
R1
FID
P
R
Baseline
19.73
0.64
0.70
24.66
0.61
0.67
21.67
0.77
0.47
9.89
0.58
0.42
16.19
0.60
0.67
SSGAN
15.65
0.67
0.68
21,02
0.61
0.65
7.68
0.59
0.50
17.18
0.61
0.65
Self-Cond GAN
16.72
0.71
0.64
21.8
0.64
0.60
15.56
0.66
0.63
Durter
10.57
0.75
0.65
14.11
0.70
0.66
13.88
0.85
0.55
3.77
0.67
0.57
9.93
0.66
0.63
DIP 12g16
GMM
11.24
0,74
0.64
15.71
0.70
0.62
15.83
0.76
0.55
4.99
0.66
0.54
12.11
0.64
0.62
K-means
15.75
4.69
Epeiss GMM
12,60
0.69
0.70
16.36
0.67
0.70
13.50
0.83
0.59
3.85
0.67
0.56
11.43
0.62
0.68
DIP SinCLR
14.42
0.68
0.65
20.08
0.62
16.62
0.27
0.53
4.92
0.53
14.99
0.60
0.63
K-meams
14.27
0.67
0.70
17.96
0.66
0.699
14.10
0.82
0.52
5,48
0.74
0.45
15.58
0.62
0.64
Table 3: Comparison of FID, Precision and Recall metrics of DIP with Baseline and SSGAN for large-scale image generation. Best values obtained by using complete training set Dprior are underlined and the best value among all other approaches are bold.
the most part, presents memorized training set images. DIP performs better when training is done from scratch compared to FreezeD and TransferGAN but is worse than DiffAugment+DDP. We present additional ablation studies in Appendix A including more qualitative comparisons, to study the benefit of using DIP in few-shot image generation.
5.2 LIMITED DATA IMAGE GENERATION
In many practical scenarios, we have access to moderate number of images (1k-5k) instead of just few examples, however the limited data may still not be enough to achieve stable GAN training. We show the benefit of our approach in this setting and compare results with: MineGAN(Wang. et al.) 2020). TransferGAN, FreezeD, and Dugmeut We perform experiments on three 128 x 128 res- olution datasets: FFHQ2k, Places2.5k and CUB6k following (Wang al. 2020). FFHQ2K contains 2K training samples from FFHQ (Karras et al. (019)) dataset. Places2.5k is a subset of Places365 dataset (Zhou et al. 2014) with 500 examples each sampled from 5 classes (alley, arch, art gallery, auditorium, ball-room). CUB6k is the complete training split of CUB-200 dataset (Wah et al., 201T). We use the widely used BigGAN (Brock et al. 2018) architecture, pre-trained on ImageNet for fine- tuning. Table [ shows our results; using DIP consistently improves FID on existing baselines by a significant margin. More implementation details are given in Appendix B and sample generated images via our approach are shown in Fig We also compare our approach with DiffAugment on CIFAR-10 10 and CIFAR- 100 dataset while varying the number of sample used for training in Table Appendix B
5.3 LARGE-SCALE IMAGE GENERATION
In order to show the usefulness of our method on large-scale image generation, we carry out experi- ments on CIFAR-10, CIFAR-100 (Krihhvsky et al. 2010) and ImageNet-32 x 32 datasets with 50k, 50k and ~ 1.2M training images respectively at 32 x 32 resolution. For higher 128 x 128 reso- lution, we perform experiments on FFHQ and LSUN- bedroom (Yu et al. 2015) datasets with 63k and 3M training samples. We use ResNet-based architecture for both discriminator and generator similar to BigGAN (Brock et al., 2018) for all our experiments. We also compare DIP with SS- GAN (Chen et al. (019) and Celf-Conditional GAN (Liu et al. 2020). Implementation and training hyperparameter details are provided in Appendix C
Table Treootss the FID, precision and recall score the generated samples and the test set for base- lines and our approach (DIP). For K-means clustering on Gemas (ID (DDrrio)), we set number of clusters 10K and for GMM, the number of components are fixed 1K for all datasets. DIP achieves better FID, precision and recall scores compared to leading baselines. Sample qualitative results are shown in the Appendix (Figure T1.. To further analyze the role of prior in our methodology, we train CIFAR- 100 dataset with DIP using priors from different pre-trained models. As shown in the results in 5 the FID metric remains relatively similar for different priors when compared to the baseline. We also evaluate the quality of inverted images for 128 x 128 resolution LSUN datasets using Inference via Optimization Measure (IvOM) (Metz et al. 2017) to emphasize the high instance-level data coverage in the prior space of GANs trained through tails on IvOM calculation are provided in Appendix C). Table - shows the IvOM and FID metric between inverted and real query images. Figure shows sample inverted images. We observe both
FTHQ and
approach (de-
8

---

Under review as a conference paper at ICLR 2021
Method
FFHQ
LSUN-
BUUN-Bedrom
IvOM
FID
IvOM
FID
Baseline
0.0386
85.06
0.0517
115.02
Die-
0.0142
73.85
0.0191
129.4
SinCRR
0.0161
Table 4: IvOM and FID on 500 random
images of FPHQ and LSUN-Bedroom datasets
Method
CIFAR-1O0
Baseline
24.66
FFEQ
LSUN-Bedroom
DIP-SinCLR (ImageNet)
16.26
DIP-SimCLR (CIFAR
14.62
Figure 4: Images generated through IvOM for randomly sampled test set images FFHQ and LSUN-Bedroom, (Top Bottom:) Original im- ages, Baseline, Baseline + DIP- Vgg 16, Baseline + DIP-SimCLR.
DIP-ResNet50 (Places-365)
14.68
Table 5: Comparison of FID when using prior from
various pre-trained models CIFAR-100
Figure 5:
Semantic diffusion for image manipulation using DIP-Vgg16 model on FFHQ dataset. (Left to Right:) Custom Editing, Inpainting, Skelch-to-Image Translation and Colorization.
from qualitative and quantitative perspective, models trained via DIP invert a given query image better than the corresponding baselines.
Semantic Diffusion and Variations As described in Section our conditional DIP module pro- vides with framework to alter input images and thus generate images with specific semantics. demonstseessssstttes how controlled semantic diffusion can be leveraged several image manip- ulation tasks. We perform: (1) Custom Editing by using CutMix (i.e pasting desired portion of one image upon another); (2) Inpainting by using CutOut (i.e removing random portions example image); (3) Sketch-to-Image by providing the feature of sketch conditional prior; and (4) Col- orization by using the feature of a given grayscale image for conditioning. As evident from Figure 5 our trained generator is able to generalize well through its ability diffuse the semantic infor- mation from altered (cutmix and cutout) as well as out-of-domain (sketches and gray-scale) images. For more qualitative results, please Fig T0 in Appendix. We can also use interpolation, noise injection and Mixup in the conditional space to generate meaningful variations of given image as shown in Fighin in the Appendix.
CONCLUSION
In this work, we present novel data instance prior based transfer learning approach improve the quality and diversity of images generated using GANs when few training data samples are available. By leveraging features as priors from rich source domain limited unsupervised image synthesis, we show the utility of our simple yet effective approach on various standard vision datasets and GAN architectures. We demonstrate the efficacy of our approach in image generation with limited data, where it achieves the new state-of-the performance, well Furthermore, using our framework of training via instance level priors, we show how easily we can perform common image editing tasks by manipulating these priors. As future work, it would be interesting to explore the application of prior information in other potential image editing tasks.
large-scale settings.
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223, 2017.

---

Under review as a conference paper at ICLR 2021
Sanjeev Arora, Rong Ge, Yingyu Liang, Tengyu Ma, and Yi Zhang. Generalization and equilibrium in generative adversarial nets (gans). In JCML, 2017.
Piotr Bojanowski, Armand Joulin, David Lopez-Pas, and Arthur Szlam. Optimizing the latent space of generative networks. In International Conference Machine Learning, pp. 100-609, 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations, 2018.
Ting Chen, Mario Lucie, Neil Houlsby, and Sylvain Gelly, On self modulation for generative adver- sarial networks. In International Conference Learning Representations, 2018.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12154-122 12161,21,,,, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, pp. 6594-6604, 2017.
Emily Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. In Advances in neural information processing systems, pp. 1486-1494, 1494, 2015.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darelll Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference on machine learning, pp. 647-6677 2014.
Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. arXiv preprint arXiv:1610.07629, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural infor- mation processing systems, pp. 2680, 2014.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. In Advances in neural information processing systems, pp- 5767-5777, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In NeurlPS, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop. 2015. URL htttt://p//xxiv org/ abs//bb//oe 1503.03. 02531
Yedid Hoshen, Ke Li, and Jitendra Malik. Non-adversarial image synthesis with generative la- tent nearest neighbors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 181119,, 2019.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In CVPR, 2017.
Ali Jahanian, Lucy Chai, and Phillip Isola. On the "steerability" of generative adversarial networks. In ICLR, 2020.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
10

---

Under review as a conference paper at ICLR 2021
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In ICLR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. arXiv preprint arXiv:2006. 066666,00606666, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 8110-8119, 2020b.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10. URL http: www. cs woronto......oo.ooo edu/kriz/cifan html 2010.
Tuomas Kynkääänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems, pp. 3927-3936, 2019.
Ke Li and Jitendra Malik. Fast k-nearest neighbour search via prioritized dei. In Proceedings of the 34th International Conference on Machine Learning-Volle Volume 70, pp. 2081-2090, 2017.
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation. arXiv:1809.0900 2018.
arXiv preprint
Steven Liu, Tongzhou Wang, David Bau, Jun- Zhu, and Antonio Torralba. Diverse image gener- ation via self-conditioned gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Henry B Mann and Donald R Whitney. On test of whether of two random variables is stochas- tically larger than the other. The annals of mathematical statistics, pp. 50-60, 1947.
Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE international confer- ence on computer vision, pp. 2794-2802, 2017.
Casey Meehan, Kamalika Chaudhuri, and Sanjoy Dasgupta. A non-parametric test to detect data- copying in generative models. arXiv preprint 2rXiv:2004.05675. 2020.
Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. ICLR, 2017.
Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In International Con- ference on Learning Representations, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the discriminator: a simple baseline for fine- tuning gans. In CVPR for Content Creation Workshop, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Atsuhiro Noguchi and Tatsuya Harada. Image generation from small datasets via batch statistics adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2750- 2758, 2019.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In International conference on machine learning, pp. 2642-2651, 2017.

---

Under review as a conference paper at ICLR 2021
Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring mid- im- age representations using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Viston and Pattern Recognition (CVPR), June 2014.
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Transactions on knowledge and data engineering, 22410):335-,335, 2009.
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. In ICLR, 2020. URL.https://opeeeeeieee net/ forum?
Hd-H1laeJrKDB
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In International Conference Learning Repre- sentations, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, 2014.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Alexander Sage, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. Logo synthesis and manipu- lation with clustered generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5879-5888, 2018.
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lučié, Olivier Bousquet, and Sylvain Gelly. Assessing Generative Models via Precision and Recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NewrlPS, 2016a.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 222222232222 2016b.
Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. u-net based discriminator for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8207-8216, 2020.
D. Sculley. Web-scale k-means clustering. Proceedings of the 19th International Conference World Wide Web, WWW '10, pp. 1177-1178, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 978 9781605587998. doi: 711144//77226901177282.. URL https:
//doi. 10 1145 01155/7776999 1772867
Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In CVPR, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Dustin Tran, Rajesh Ranganath, and David M Blei. Deep and hierarchical implicit models. arXiv 7:3, 2017.
preprint ArXiv:1702.000
Michael Tschannen, Eirikur Agustsson, and Mario Lucic. Deep generative models for distribution- preserving lossy compression. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE International Conference on Computer Vision, pp.
2015.
Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Con- ditional image generation with pixelcnn decoders. In Advances in neural information processing systems, pp. 4790-4798, 2016.
12

---

Under review as a conference paper at ICLR 2021
Aaron Van Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp. 1747-1756, 2016.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-00 California Institute of Technology, 2011.
Yaxing Wang. Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bog- dan Raducanu. Transferring gans: generating images from limited data. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 218-234,4 2018.
Yaxing Wang, Abel Gonzalez-Gare Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: Effective knowledge transfer from gans to target domains with few images. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Lei Xu and Michael Jordan. On convergence properties of the em algorithm for gaussian mixtures. Neural Computation, 8(1):129-551, January 1996. ISSN 0899-7667. doi: 10. 10.1162// 1.129. httphttss://oii wtt.oo//......ee........ 1996. 8.1.129
Xin Yi, Ekta Walia, and Paul Babyn. Generative adversarial network in medical imaging: review. Medical image analysis, 58:101552, 2019.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 27, pp. 3320-3328, 2014.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 03365, 2015.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In International Conference on Learning Representations, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for generation with limited data. In ICML., 2020a.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. arXiv preprint arXiv:2006.1015 2020b.
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Im- proved consistency regularization for gans. arXiv preprint arXiv:2002.04724 2020c.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene enns. arXiv preprint 142.6.141..6856. 2014.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In Proceedings of European Conference Computer Vision (ECCV), 2020.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networkss. In Computer Vision (ICCV), 2017 IEEE Interna- fional Conference on, 2017.
13

---

Under review as a conference paper at ICLR 2021
Appendix
A FEW-SHOT IMAGE GENERATION
Performance varying number of training images We vary the number of training examples in Anime dataset from 25-500 for baseline few-shot algorithms and their respective augmentations with DIP-Vggl16 The FII) metric comparison in Figl6ajshows the benefit of our approach when used with existing training algorithms. The FID metric for all approaches improves (decreases) with the increase the number of training images with DIP out-performing corresponding baselines. Sample images generated by our approach are shown in Fig 6b
Memorization in GANs To evaluate whether trained GANs are actually generating novel images instead of only memorizing the training set, we calculate FID between images randomly sampled from training set with repetition and the separate test set for Anime and FFHQ dataset. For Anime dataset, we get an FID of 81.23 and for FFHQ. 100.07. On comparing these numbers to Table [ we observe that only on using DIP with existing algorithms,, we achieve a better FID score suggesting the usefulness of the proposed approach.
Anime
225
BSA
FreezeD Drp
DiEAugment
GLANN Vug
DIP-/gg16
DIP
200
FreezeD
175
FreezeD DIP-Vgg16
25
DiffAugment
DiffAugment DIP-Vgg16
50
125
100
100
75
50
500
100
500
(b)
#images in training dalasel
(a)
Figure 6: (a) FID (lower is better) performance graph of few-shot image generation on 25-500 images of Anime dataset using various approaches on SNGAN model; (b) Samples of few-shot image generation 25-500 images Anime dataset using DIP on various approaches on SNGAN.
Few-shot image generation with StyleGAN-2 For 256 256 resolution dataset with StyleGAN- 2 architecture, we follow (Zhao et al. 2020b) and perform experiments on 100-shot Obama, Panda and Grumpy Cat dataset with pre-trained models FFHQ (Karras al., 2019) dataset. Table shows consistent improvement in FID scores when trained with DIP irrespective of baseline training algorithm except on Grumpy Cat dataset. We hypothesize that this because the prior features of this dataset has low diversity and are not informative enough to lead to improved performance with data instance prior training.
Impact of loss function analyze how DIP performs when loss function is changed. We com- pare between the following three loss functions: hinge loss used originally in our experiments, non- saturating loss (Goodfellow al.) 20144) and the wasserstein loss (Arjovsky al.) 2017). Table shows the corresponding results when DIP is used with FreezeD and DiffAugment. We observe that in case of FreezeD+DIP wasserstein loss significantly outperforms non-saturating loss and hinge loss. In case of DiffAugment hinge loss performs best followed by non-saturating loss and wasser- stein loss.
Implementation Details In SN-GAN architecture, while training with data instance prior, Gemb and Demb are matrices which linearly transform the pre- traitre features into generator conditional space of dimension 128 and discriminator feature space of dimension 1024. For baseline training.
14

---

Under review as a conference paper at ICLR 2021
(a) Anime Faces
(b) Human Faces
(c) Flower
Figure 7: Few-shot interpolation samples between instance-level priors: Scratch (Row 1), Scratch + DIP-
Vgg16 (Row 2). FreezeD (Row 3), FreezeD DIP-Vgg16 (Row 4), DiffAugment (Row 5), DiffAugment DIP-Veg16 (Row 6)
Style-GAN 2 (256 x 256)
Method
Anime (SNGAN) FID
Method
Panda FID
Grumpy Cal
Obama
FID
FID
FreezeD
16.69
29.67
62.26
Freezel DIP
93.36
Vggl6
54.87
FreezeD Logo-GAN (K=5)
226. 60
FreezeD Logo-GAN (K=10)
183.38
DillAugment
46.87
+ DIP-Vgg16
11.14
28.45
43.79
DiffAugment DIP
48.67
DiffAugment Logo-GAN (K=5)
130.54
BSA*
21.38
34.20
50.72
GLANN Vigl
38.57
DiffAugment Logo-GAN (K=10)
190.59
Table 6: 100-shot image generation results using StyleGAN-2 (Karras et al. 2020b) architecture on Panda, Grumpy-cat and Obama datasets. FID is computed between generated training dataset. denotes directly reported from
Table 7: 100-shot image generation com-
parison of DIP with Logo-GAN (Sage et al. 2018) on Anime dataset using Vggló net- work trained on ImageNet. FID is com- puted between 10k generated and real sam- ples (disjoint from training set).
the complete the paper (Zhao et al. 2020b).
embedding for each of the 100 training images to ensure minimal difference between baseline and our approach without increasing number of parameters. We also experimented with medf-modulated (Chen al., (018)) and unconditional training which resulted in either training col- lapse or worse results in all approaches. In DiffAugment, we use three augmentations: translation, cutout, and color with consistency regularization hyperparameter as 10 and training is done from scratch following the implementation in their paper (Zhao et al.) 2020b). In FreezeD, we freeze the first five blocks of the discriminator and finetune the rest. We use spectral normalization for both generator and discriminator during training with batch size of 25, number of discriminator steps as 4. G and D learning rate as 2e 4, z dimension as 120 and maximum number of training steps as 30K. During evaluation, moving average weights (Salimans et al. (016a) of the generator is used in all experiments unless stated otherwise. For FID calculation, we select the snapshot with best FID similar to (Chen et al. [0119] Zhao et al.) (220bb.. For calculating precision and recall based on the k-nearest neighbor graph of inception features, as in (Kynkäänniemi et al. 2019). we use k as 10 for Precision and 40 for Recall.
For StyleGAN-2, Gemb is a 2-layer MLP with ReLU non-linearity which maps C(x) to a 512- dimensional generator conditional space. It is then concatenated with random noise z of dimension 512 which is used as input in the mapping network. Demb is a linear transformation matrix and discriminator loss is projection loss combined with real/fake loss. Training is done with batch-size of 16 for DiffAugmen and 8 for Freeze till 20k steps.
In case of BSA, we show that DIP can be used improve the results on similar non-adversarial generative models. Specifically, we perform experiments with GLANN which is a two step training
"https///gtthub... hhtpps//gituh..oom/anggggg/ http://wwww..cmc////goo////////a butps//ggthuh..mm//cclhingggi
15

---

Under review as a conference paper at ICLR 2021
SN-GAN (128 128)
Method
Pre-truining
Anime NS
Faces
W
NS
FreezeD
40
102.43
148,99
107.85 83
105.34
209.23
DIP 1ggl6
93.36
82.49
74.91
77.09
77.38
71.05
DiffAugment lgg-
85.16
106.96
252.11
109.25
107.18
325.85
48.61
56.43
62.44
68.66
81.03
Table 8: Comparison of loss function in few-shot image generation using 100 training images (FID: lower is better). H is hinge loss, NS is non saturating loss and W is wasserstein loss.
procedure, follows: (1) Optimize for image embeddings {er} of all training images {x, jointly with a generator network G using perceptual loss; and (2) Learn a sampling function z - through IMLE for generating random images during inference. For adding data instance prior in the training procedure of GLANN, instead of directly optimizing for (ei}, we optimize for the following modified objective:
arg min
Σ Inperceptual (G Gemb o x()
G.G.mb 1
where {e,} = {Gemb C(x:)}
We finetune the pre-trained generator on batch-size of 50 with a learning rate of 0.01 for 4000 epochs. During second step of IMLE optimization, we use a 3-layer MLP with dimension as 64 and train for 500 epochs with a learning rate of 0.05.
Comparison with Logo-GAN (Sage et al. 2018 Logo-GAN has shown advantage of using fea- tures from pre-trained ImageNet network in unconditional training by assigning class label to each instance based clustering in the feature space. We compare approach with this method in the few-shot data setting. For implementing logo-GAN, we perform class-conditional training (Miyato al., 2018) using labels obtained by K-means clustering on Vgg 16 features of 100-shot Anime dataset. The results reported in Table /show the benefit of directly using features as data instance prior instead of only assigning labels based on feature clustering.
B LIMITED DATA IMAGE GENERATION
Experiments on CIFAR-10 and CIFAR-100 We experiment with unconditional BigGAN and StyleGAN2 model on CIFAR-10 and 100 while varying the amount of data as done in Zhao et al. 20205). We compare DIP with DiffAugment on all settings and the results are shown in Table 9 In the limited data setting (5% and 10%) augmenting DiffAugment with DIP gives the best results in terms FID for both BigGAN and StyleGAN2 architectures. When trained on complete training dataset DIP slightly outperforms DiffAugment on BigGAN architecture. BigGAN model used for training CIFAR-10 and CIFAR-100 is same as the one used for large scale experiments in Section 5.3 In DiffAugment with BigGAN architecture, we use all three augmentations: translation, cutout, and color along with consistency regularization hyperparameter as 10. In DiffAugment DIP consistency regularization hyperparameter is changed to 1. For experiments StyleGAN2 architecture we use the base of DiffAugment
Implementation details of experiment on 128 Resolution datasets in Section 5.2 We use our approach in conjunction with existing methodologies in similar way as the few-shot setting with Gemb and Demb as linear transformation matrices which transform the data priors into the genera- tor's conditional input space of dimension 128 and discriminator feature space of dimension 1 1536. During baseline training. we use self- modf-modulation (Chen et al. (018)) in the batch-norm layers similar to (Chen et al. 2019; Schonfeld et al. 2020). In DiffAugment, we use three augmentations: trans- lation, cutout, and color with consistency regularization hyperparameter training, we freeze the first 4 layers of discriminator. For TransferGAN, FreezeD, MineGAN and its augmentation with DIP, we use the following hyperparameter setting: batch size 256, G and D
10. During FreezeD
16

---

Under review as a conference paper at ICLR 2021
Method
CTFAR-I0
CIFAR- 100
100% data
20% data
10% data
100% data
20% data
10% data
BigGAN
17.22
31.25
42.59
20.37
33.25
42.43
+DIP
9,70
16.24
27.86
12.89
21.70
31.48
+DiffAugment
10.39
15.12
18.56
13.33
19.78
23.80
+DiffAugment & DIP
9.52
14.24
18.50
12.70
16.91
20.47
StyleGAN2
11.07
23.08
36.02
16.54
32.30
45.87
+DiffAugment
9.89
12.15
14.5
15.22
16.65
20.75
+DiffAugment & DIP
9.50
10.92
12.03
14.45
15.52
17.33
Table 9: Comparison of FID on CIFAR-I0 and CIFAR-100 while varying the amount data used during training. Above all approaches are trained with amdom-horizontal flip augmentation of real images. BigGAN- DisfAuumeett includes consistency regularization (Zhang el al. 2019) following the implementation provided by authors (Zaa et al. 2020b). Best FID values reported for each model. denotes directly reported from paper.
(a) Places (2.5k)
(b) FFHQ (2K)
(c) CUB (6K)
Figure 8: Sample generated images on limited data training: Freezel) (Row 1), FreezeD + DIP- Vgg16 1 (Row 2), Diff Diffmnttent (Row 3) and DiffAugment DIP-Vgg16 (Row 4)
2e and dimension 120. For DiffAugment, batch size 32, D-steps is 4 and rest of the hy- perparameters same. Training is done till 30k steps for DiffAugment, FreezeD, and 5k steps for rest. The moving average weights of the generator are used for evaluation. We use pre-trained network from Brock et al. 2018 for finetuning.
C LARGE-SCALE IMAGE GENERATION
Test for memorization in trained model For analyzing memorization in GANs, evaluate it on the recently proposed test to detect data copying (Meehan et al.) 2020). The test calculates whether generated samples closer to the training set compared to separate test set in the inception feature space using three sample Mann-Whitnee U test (Mann & Whitney| 1947). The test statistic O represents underfitting. We average the test statistic over 5 trials and report the results in Table I0 We can see that using data instance priors during GAN training does not lead to data-copying according the test statistic except in case of FFHQ dataset where both DIP and baseline Cr values are also negative.
Cr <<<<< 0 represents overfitting and data-copying, whereas Cr
Image inversion To invert query image, x,, using our trained model, we optimize the prior (after passing it to Gemb) that is used to condition each resolution block, independently. Mathematically. we optimize the following objective:
z z', CiCi = arg min [G(z)C1....C xy12. 2,C....C2
inv x
G(zz (
Here, C₁ (after passing it through Gemb) is the prior that is used to condition the jth E k1...) resolution block. To get faster and better convergence, initialize all C, as (Cx,)) The optimization is achieved via back using Adam optimizer with learning rate of 0.1.
17

---

Under review as a conference paper at ICLR 2021
Figure 9: Semantic variations using pre-trained Vggló conditional DIP module on FFHQ dataset. (Left:) Random samples generated with prior feature of the first image in each row; (Right:) first and second in both images shows generated samples by interpolation and Bernoulli mixup between two image priors respectively.
19
Figure 10: Examples of semantic diffusion used in image manipulation on FFHQ dataset using DIP-VggIO 16 approach. Top-Left: Custom Editing; Top-Right: Sketch-to-Image: Bottom-Left: Inpainting; Bottom-Right Colorization
Semantic diffusion for image manipulation Figure 10 shows more examples of semantic diffu- sion being used standard image manipulations like colorization, editing, sketch-to-ing to-ic- transla- tion and inpainting.
18

---

Under review as a conference paper at ICLR 2021
(a) CIFAR-10
(h) CIFAR-100
(c) ImageNet Net-32x32
FFHQ
(e) LSUN- -Bedroom
Figure 11: Samples generated by our DIP-Vggll approach on large-scale image generation
Methods
CIFAR-10 C
CIFAR-100
FFHQ CT
LSUN
ImageNet32232
Cr
Cr
Cr
Baseline
3.02
4.26
-0.15
2.59
10.5
DIP-Vgg16
1.24
2.50
-0.63
3.31
7.48
DIP-Vgg 16 (GMM)
1.58
3.05
-0.81
1.06
DIP-Vgg (K-means)
1.96
3.70
-0.46
8.47
DIP-SimCLR
2.23
3.30
-1.14
2.49
9.70
DIP-SimCLR (GMM)
2.86
3.48
-1.49
0.13
9.91
DIP-SinCLR (K-means)
2.45
3.91
-1.84
-0.12
10.11
Table 10: Test evaluating data-copy memorization GANs (Meehan al.
2020
different ap-
proaches and datasets. Test statistic Cr << denotes overfitting and data-copying, and Cr >> 0 represents unternfittin.
Implementation Details We single linear layer to transform the pre-trained image features to the generator's conditional input space of 128 dimensions, and discriminator feature space of 1024 dimensions respectively. hierarchical latent structure similar (Brock al. 2018) used during DIP training. During evaluation with K-means and GMM on ImageNet and LSUN-Bedroom we first randomly sample 200K training images and then fit the distribution since clustering on com- plete training set which is in the order of millions is infeasible. In the training of the unconditional baseline, we use self-modulation (Chen et al., 2018). In SSGAN, for rotation loss we use the de- fault parameter of 0.2 for generator and 1.0 for discriminator mentioned in (Chen et al.) 2019). For training Self-Cor Conditional GAN (Liu et al. 2020), we set the number of clusters to 100 for all datasets. For 10 and CIFAR-ION for ImageNet, at every 75k iterations with 50k samples following default implementation as in (Liu al. 2020). Following standard practice (Zhang et al. 2019), call between test split and an equal number of generated images for- 10, CIFAR- 100, and ImageNet 32 x 32, i.e., 10k, 10k, and 50k, respectively. For FFHQ and LSSN--eerrom datasets, use 7k and
re-cluster at every 25k iterations with 25k samples, and
calculate FID, Precision and Re-
19

---

Under review as a conference paper at ICLR 2021
30k generated and real (disjoint from training) samples, respectively. For all datasets and methods, training is done with batch size of 64, G and learning rate is set to 0.0002, z dimension equals 120 and spectral normalization is used in both generator and discriminator networks. Training is done till 100k steps for all datasets except ImageNet which trained for 200k steps and moving average weights of generator used during evaluation.
20


Reviews:
[{"review_id": "hSCtanfL2fP", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThe paper focuses on  improving the performance of training generative adversarial networks (GANs) with limited target data. With the low diversity and quality when traing GANs with few data,  the paper proposes to use data instance prior to reduce the overfitting. Specially, taking the target sample as input, the data prior is extracted by a pre-trained network /  self-supervised model  , and then mapped into the embedding by  both G_emb and D_emb. The former acts as the class embedding, and the latter is the image embedding combined with the discriminator.     Authors also extend the proposed method into the large dataset, and provide the cluster method or a Gaussian Mixture Model.  The quantitative and qualitative result support the proposed method\n\n##########################################################################\n\nPros:\n+ The idea of utilizing the informative data prior to help train GANs is interesting.  Authors leverage the pre-trained model to extract data prior, and combine it with conditional GANs. Basically, SNGAN is considered in this peper, which uses the projection loss instead of cross-entropy loss to perform conditional image generation. Based on form of the projection loss, authors are able to avoid the problem the label of target data, and directly use the extracted embedding to conduct image generation.\n\n+ For  large dataset, authors also provide a simple and effective method to get the semantic embedding.\n\n+ For experiment, the paper uses current SOTA methods (BigGAN, SNGAN and StyleGAN2) and a series of datasets to evaluate the proposed method. Besides, all latest methods (to my best knowledge)  is compared to the proposed method, even the unpublished papers (DiffAugment),  which indicates that the proposed method is effective and convincing .\n\n##########################################################################\n\n Cons:\n- For me, the paper is so clear to understand, and miss a few information. \n  (1) Does the final objective contains E.q 2 ?  From the description above E.q 3 and the architecture , it seems to contain two losses.\n\n  (2) Is E.q 2 used at inference time which is combined to the Inference section? I fail to connect the inference section to the whole paper.\n\n  (3) If E.q 2 is utilized in this paper, what is different to BSA?  From my point, BSA assigns the noise and the real sample pair, and optimize  the input noise as well as the parameter of the batchnorm,  but fails to consider the adversarial loss,  which results in generating blur image. In this paper, authors  additionally consider the adversarial loss, and improve the reality of the synthesized image. \n\n\n- With conditional GANs is selected in this paper, I am wondering how to combine to the StyleGAN2, although the description is provided in Appendix. To be honest, I am not sure it works well when combining StyleGAN2 with project loss.\n\n- What is the goal of both G_emb and D_emb? Do them just map the extracted embedding  to the same dimension required by conditional GANs?  authors mention that  it is   on-linearity  or   linear transformation matrices for different GANs frameworks, and  varying for  varying GANs architectures.  What is  linear transformation matrices?  what is the role to design    both G_emb and D_emb?\n\n- In table 1, the result when combining FreezeD on Flower is low. Could authors explain it?\n\nMinor comments: \n(1) Is  TransferGAN cited correctly in section of Baselines and Datasets  in page 6 ?  I think it is the one [1], which is first paper to perform transfer learning for GANs with limited data.\n\n(2) What is the x and hat of x in E.q 3. \n\n(3) In abstract,  authors mention ' Previous works have addressed training inlow data setting by leveraging transfer learning and data augmentation techniques with limited success'. Is it  'with limited success' or 'with limited data'? \n\n[1] Transferring gans:  generating images from limited data.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TFaPtWq-J7x", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposed a transfer learning method for GANs in the limited data domain. It borrow ideas from IMLE (to overcome mode-collapse) and conditional GAN (to improve training stability and generation quality), by introducing data instance prior (plays a similar role to that of  label information in conditional GAN) and knowledge distillation techniques, the model is claimed to be effective in preventing mode collapse and discriminator overfitting.\n\nThough the main idea makes sense to some extent, the writing is a little weak, especially the equations and some statements that are not correctly verified in the experiments, making it difficult to go through the paper. \n\nDetailed concerns are listed below.\n\n1. Eq 3 is not correct, the sample process should be placed under “Expectation” rather than “minimize”; also, there is no information about which parameter is going to be optimized here. What’s the relation ship between x and x~, are they independently sampled from the target dataset? \n\n2. Figure 1 is confusing. It shows that the adversarial loss is depend on the projection loss, but there is no equation to show the relationship between these two losses.\n\n3. There are many network components in the proposed method, it is not easy to guess the objective related to the real/fake score in an adversarial manner, a clear equation for the adversarial training is necessary. Also, a detailed training process is necessary, e.g. how to sample, when to optimize the generator, discriminator, and the embedding networks?\n\n4. How to make sure that “enforcing feature Df(G(z|C(x))) to be similar to Demb(C(x)) ensures that for each real sample, there exists”? Apparently, the constrain is made on some latent space and there is no statement to show that close in latent space is equivalent to close in image space.\n\n5. When the data is large scale, doing clustering is inhibitive.\n\n6. On Table 3, the results are not state of the art on CIFAR10 and fall behand the recent works on generation with limited data, e.g. discriminator augmentations (DA) [1],  DiffAugment [2], then what’s the advantage of the proposed method when compare to these methods?\n\n7. An ablation study is suggested. It’s not easy to see the contribution of each part (e.g. knowledge distillation, covering real data modes) to the final performance.\n\n8. It claims that the proposed method can prevent discriminator overfitting, but in the experiments, it is not shown.  \n\n9. What’s the motivation to do interpolation on the data instance prior? How to make sure that the interpolation on the prior (Eq.4) is smooth? \n\n10. On semantic diffusion for image manipulation. It is not clear how to obtain the results shown in Figure 4/9 on custom editing, e.g. there is only one input image, how to compute and exchange the C(x)? Also, the effects of manipulating high-level semantics and fine-grained details are not observed/discussed in the experiments.\n\n\n[1] Karras T, Aittala M, Hellsten J, et al. Training generative adversarial networks with limited data[J]. arXiv preprint arXiv:2006.06676, 2020.\n[2] Zhao S, Liu Z, Lin J, et al. Differentiable augmentation for data-efficient gan training[J]. arXiv preprint arXiv:2006.10738, 2020.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ae1pqab0r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission deals with transfer learning for training GANs with limited label data. The challenge is that training with limited data can result in mode collapse. This submission proposes to use data priors for each instance of the target distribution, transformed through knowledge from a source domain, as conditional information in GAN to ensure mode coverage of the target data distribution. A pre-trained feature extractor is used to provide the information to condition the GAN. A range of experiments is performed with the features extracted from VGG16, SIMCLR. They show consistent improvements in the image quality and diversity, measured via FID and precision-recall,  for few-shot, limited data, and even large scale data settings. \n\nStrong points\n- The idea of creating useful guides based on pre-trained features for conditional GANs makes sense.\nThe experiments are extensive and they show improved quality and diversity for a wide range of settings.\n\nCorrectness\n- The idea and the reported experiments make sense. \n\nReproducibility\n- The code has been provided with the code for pre-training and other details included. \n\nMore suggestions and comments\n- IMLE description in (2) is vague. It seems that a gaussian sample z is taken, but on the other hand, for a given network G, z is optimized to match x. The restrictions on z is unclear.\nIt seems that hinge-loss is used for GAN based on (1)? It would be important if the authors could comment on the choice of the divergence/loss in this setting. One may wonder that the limited data and mode-collapse could be better handled with Wasserstein distance. An ablation study would be very useful to clarify the role of the distance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Ec335jbOZ-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper illustrates how they train GANs with small sample sizes with the help of Transfer Learning. The paper tackled a very specific problem: what should we do with a small sample training size if we want to train a GAN. The authors have supported their arguments by a proof in Data In Prior and experiment results. They illustrated well in both aspects.  \n\nHere are my point of views:\nTransfer learning is a good way to help GANs when sample size is limited. but I have two concerns over this paper:\n1. The datasets are very popular in the filed of GAN, however, for the Anime one, I am just curious how the VGG pretrained network can also help.  \n\n2.  As for the experiment, it lacks of comparison with results that transfer learning is not applied. \n\nGenerally, the paper is good and it can help data augmentation in other applications.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to use a feature extractor (encoder) $C(x)$, pre-trained with label supervision or contrastive learning on a large image dataset, to both regularize the discriminator's last feature layer $D_f(x)$ and encode the data $x$ itself as the conditional input of the generator $G(z|G_{emb}(C(x)))$. The main purpose is to help the training of GANs when there is a limited number of images in the target domain. A clear concern of this approach is that to generate a fake image, one will need to first sample a true image, making the model unattractive if the training dataset size is large (need to store the whole training dataset even after training). To mitigate this issue, the authors propose to fit up to 200k randomly sampled $G_{emb}(C(x))$ with a GMM with 1k components. To validate the practice of requiring a GMM (a shallow generative model) to help a GAN (a deep generative model) to generate, the authors have done a rich set of experiments under state-of-the-art GAN architectures or training methods (SNGAN, BigGAN, StyleGAN2, DiffAugment) to illustrate the efficacy of the proposed data instance prior and its compatibility with the state-of-the-art methods in a variety of settings. In the AC's opinion, the paper is missing references to 1) related work that combines VAE (or some other type of auto-encoder) and GAN, which often helps stabilize the GAN training [1,2,3], 2) VAE with a VampPrior [4], and 3) more broadly speaking, empirical Bayes related methods where the prior model is learned from the observed data (see [5] and the references therein). The potential advantages of using a VAE rather than a GMM to help a GAN to generate include: 1) there is no need to store 1k GMM components, which may require a large amount of memory; 2) there is no need to subsample the training set; and 3) the VAE and GAN can be jointly trained. The AC recommend the authors to discuss the connections to these related work in their future submission.

[1] Larsen, Anders Boesen Lindbo, et al. "Autoencoding beyond pixels using a learned similarity metric." International conference on machine learning. PMLR, 2016.

[2] Zhang, Hao, et al. "Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling." International Conference on Learning Representations. 2019.

[3] Tran, Ngoc-Trung, Tuan-Anh Bui, and Ngai-Man Cheung. "Dist-gan: An improved gan using distance constraints." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

[4] Tomczak, Jakub, and Max Welling. "VAE with a VampPrior." International Conference on Artificial Intelligence and Statistics. PMLR, 2018.

[5] Pang, Bo, Tian Han, Erik Nijkamp, Song-Chun Zhu, and Ying Nian Wu. "Learning Latent Space Energy-Based Prior Model." Advances in Neural Information Processing Systems 33 (2020).

Decision: Reject