Few-shot Cross-domain Image Generation via Inference-time Latent-code Learning
Authors: Arnab Kumar Mondal
Piyush Tiwary
Parag Singla
Prathosh AP
Venue: ICLR 2023 notable top 25% (2023)
Keywords: generative domain adaptation
generative adversarial network

Abstract: In this work, our objective is to adapt a Deep generative model trained on a large-scale source dataset to multiple target domains with scarce data. Specifically, we focus on adapting a pre-trained Generative Adversarial Network (GAN) to a target domain without re-training the generator. Our method draws the motivation from the fact that out-of-distribution samples can be `embedded' onto the latent space of a pre-trained source-GAN. We propose to train a small latent-generation network during the inference stage, each time a  batch of target samples is to be generated. These target latent codes are fed to the source-generator to obtain  novel target samples. Despite using the same small set of target samples and the source generator, multiple independent training episodes of the latent-generation network results in the diversity of the generated target samples. Our method, albeit simple, can be used to generate data from multiple target distributions using a generator trained on a single source distribution. We demonstrate the efficacy of our surprisingly simple method in generating multiple target datasets with only a single source generator and a few target samples.

Published as a conference paper at ICLR 2023
FEW-SHOT CROSS-DOMAIN IMAGE GENERATION VIA INFERENCE-TIME LATENT-CODE LEARNING
Arnab Kumar Mondal", Piyush Tiwary', Parag Singla' & Prathosh AP" IIT Delhi, IISc Benglauru
ABSTRACT
In this work, our objective is to adapt Deep generative model trained a large- scale source dataset to multiple target domains with scarce data. Specifically, we focus on adapting a pre-trained Generative Adversarial Network (GAN) to a tar- get domain without re-training the generator. Our method draws the motivation samples can be "embedded" onto the latent
from the fact that space of pre-trained source-GAN, We propose to train small latent- -eeet-generttion network during the inference stage, each time a batch of target samples is be generated. These target latent codes are fed to the source-generator to obtain novel target samples. Despite using the same small set of target samples and the source generator, multiple independent training episodes of the aatent-generatte results in the diversity of the generated target samples. Our method, albeit simple, be used generate data from multiple target distributions using a generator trained on a single source distribution. We demonstrate the efficacy of our sur- prisingly simple method in generating multiple target datasets with only single source generator and a few target samples. The code of the proposed method is available at: Intps:: /ttt:/u//ttubbb. com/ arnabbronnd
network
INTRODUCTION
1.1
FEW SHOT IMAGE GENERATION
Deep generative models learn generate novel data points from unknown underlying distribu- tion. The family of auto-encoder based generative models (Kingma & Welling (2014) use variational inference maximize evidence lower bound (ELBO) on the data likelihood; adversarial generators such as GANs (Goodfellow et al. 2014) learn to sample by solving a min-max optimization game and the normalizing flow-based methods utilize tractable transformations between the latent and data distributions (Kobyzev et al.
2020). All such models are shown to be successful in generating
high-quality realistic data such as images (Karras et al. 2018 2019 2020b).
However, one of the caveats in deep generative models is that they require thousands of images for proper training. limiting the scope of what can be explored (Sushko et al. 202I]. This problem poses the applications of deep generative models, the number of training data often limited to the order of hundreds or even tens at times; making it crucial to adapt generative models for few-shot settings. One natural way accomplish the above objective is to the 'prior- knowledge" that is already there in generative model built on a larger, but 'close" source dataset (Wang et al. 2018 2020b). Several ideas ranging from learning latent transformations (Wang al.) 2020b) to re-training generators on target data with regularizers such as Ellasti--eightt consolida- tion (i et al. 2020) and cross-domain correspondence (Ojha et al. 2021) have been proposed (See section 2 for a detailed description). The basic principle in all these is to adapt the generator of Gen- erative Adversarial Network (GAN) (Goodfellow et al.) 2014),, trained on large source dataset, on to the target dataset such that the re-trained generator imbibes the 'style" of the target while retain- ing the 'variability" the source domain. In other words, the re-training is geared towards reducing the infamous problem of the catastrophic forgetting that bogs the realm of transfer learning (Mc- Closkey & Cohen 1989). While the aforementioned methods show good progress towards adapting pretrained GAN, there still shortcomings such as lack of diversity due to wert-fiting. Further, these methods require novo re-training on every new target, which possibly leads to catastrophic forgetting. In this paper, intend to tackle some of these issues by addressing the following ques-
practical restrictions

---

Published as a conference paper at ICLR 2023
(d)
Figure 1: (a) Emoji (Hua et al. 2017). (b) Amedeo Modigliani's 's Art Yaniv et al. 2019), (c) Fernand
Leger's Art (aaniy et al. 2019). (d) Moise Kisling's Art (Yaniv et al. 2019), (e) Sketches (Wang
& Tang 2009). The left image in each pair is the original image, the right image is the image
reconstructed using its embedding in the extended intermediate latent space of StyleGAN2 (Karras
al. 2020b) trained on FFHQ dataset. The latent space accommodates a wide array of data.
tion- Can a GAN trained on single large-scale source dataset be adapted multiple target domains containing very few examples without re-training the pretrained source generator?
1.2 MOTIVATION AND CONTRIBUTIONS
Recently, it has been observed that the latent space of high-fidelity GANs (e.g. StyleGAN2 (Karaas et al. 2020b)) is versatile as it can 'accommodate" a large variety of out-of aut-of-distribution data Abdal et al, 2019 202T Richardson et al. 2021; Tov et al. (022T). In other words, given pretrained GAN on source dataset (source-GAN) and samples from certain target distribution, the corresponding representations of them can be found out in the latent space of the source-GAN (Abdal et al. 2019 2021 Richardson al. 2021 Tov al., (22II) For instance, Fig. Presents images from several target distributions and corresponding reconstructed images using embeddings from the latent space of a StyleGAN2 trained a large-scale source dataset (FFHQ). It is seen that wide range of out- of-distribution samples can be embedded in the latent space of source generators. This motivates us to hypothesize the existence of target-data manifold in the latent space of the source- GAN.
To achieve the aforementioned objective, one straightforward way is to re-train the source-GAN with custom regularization in (Li al. 2020). However, these methods are prone over-fitting when the target data has very few samples (of the order of tens). To alleviate these issues, we propose to find the latent vectors that generate the target data on the fly during the inference without the need to re-train the source-genee nource-generator. This is accomplished by solving inference-time optimization problem on the latent space pretrained GAN. Recent works (Zhang et al., 2020; Pandey et al.) 2021 Wu et al. 2019) have shown the advantage of inference-tine latent optimization for several tasks which motivates us to explore the use inference-ting optimization for few-shot generation. We list the contributions of this work below:
We propose a simple procedure to utilize a GAN trained on large-scale source-data to generate samples from target domain with very few (1-10) examples.
2. Our procedure shown be capable of generating data from multiple target domains using single source-GAN without the need for re-training or fine-tuning it.
3. Extensive experimentation shows that method generates diverse and high-quality target samples with very few examples surpassing the performance of the bascline methods.
2 RELATED WORK
Few shot generative domain adaptation: In 'generative domain adaptation', base model pre- trained on source domain is adapted to a related target domain by using few examples. Generally, this done by re-training the model on the target data via appropriate losses. For example, the authors of Transfer-Gi GAN (Wang et al. (018) demonstrated that fine-tuning from single pretrained GAN (Goodfellow et al. 2014) is beneficial for domains with scarce data. Later, the authors in Noguchi & Harada 2019) observed that this technique leads to mode collapse, and hence they only fine-tune the scale and shift parameters of the generator. However, this may limit the flexibility of the network. To address this concern, the authors in MineGAN (Wang al. 2020b) prepend miner network to the generator to transform the input latent space modeled by multivariate normal distri- bution so that the generated images resemble the target domain. They propose two step-training
2

---

Published as a conference paper at ICLR 2023
procedure that trains the miner network first, and then the entire pipeline is re- re-trained using the tar- get data. (Li et al., 2020) adopts Elastic Weight Consolidation (EWC) (Kirkpatrick et al. 2017) 7 to penalize large deviation of the important weights (estimated using Fisher information) while fine- tuning a GAN pretrained on a source domain. (Mo et al. 2020) demonstrates that simple fine-tuning of GANs by freezing the lower layers of the diseriminator generates diverse good images in the low data regime. (Zhao et al., 2020aa demonstrates that the filters in the layers close to the obser- vations of both the generator and discriminator of pretrained GANs can be transferred to facilitate generation in a perceptually distinct target domain with limited training data. Further, they propose adaptive filter modulation (AdaFM) adapt the transferred filters to the target domain. (Ojha al.) 2021 leverage cross-domain correspondence while fine-tuning the source-model. (Lee et al. 2021 uses pairs of positive and negative images from two different domains to learn the cross-domain correspondence. (Xiao et al., 2022) aligns the spatial structural information between the generated image source and target domain using cross-domain spatial structural consistency loss.
Few shot augmented generation without pre-training: In (Karras et al.) (020a),, the authors achieve state-of-the-an generation quality with just a few thousand training examples through an adaptive discriminator augmentation mechanism. (Zhao al., (2020)) generates high-fidelity im- ages using just 100 images utilizing differentiable augmentation (DiffAugment) technique. (Liu al.) 2 designs skip-layer channel-wise excitation module and a self-supervised discrimina- for trained as reature-encoder to achieve good generation quality using just 100 images. However, the effectiveness of these techniques reduces in extreme few shot e.g.. 10-shot settings. In the ex- treme case, the generative model is trained on single image (Rott Shaham et al., 2019 Sushko al. 202T). However, the learned model only manipulates the repeated patterns in that image, fails to generate something that does not exist within the training data.
| de- velops an interesting approach of CLIP-guided (Radford et al. 2021 training of the generator for zero-shot domain adaptation. (Zhu et al., 2022) builds upon StyleGAN -ADA to develop a one-shot adaptation model and introduces several regularizations to improve generation quality.
Text-guided Domain Adaptation for Image Generators: StyleGAN-NADA (Gal et al. 2021
Image-to-Image translation and Style Transfer: Several methods (Kwong et al., 2021 Pinkney & Adler 2020; Song et al.] (021) leverage the correspondence between closely related domains and the rich embedding space of StyleGAN2 (Karras et al., (020b)) to perform image-to-image transla- tion. These methods first embed a source domain image into the latent space of StyleGAN2. Next, the latent representation is passed through a model fine-tuned on target data (Song et al. 202T or a hybrid model obtained by swapping the layers between the source and fine-tuned model (Kwong
et al. | Pinkney & Adler] (220).
Example-based neural style transfer methods (Gatys et al. 2016; Huang Belongie, 2017 Li et al.) 2017) may be employed to transfer the style of a few target domain examples to the plentiful source domain data. However, one single example fails to represent the consistent style across the target domain fully. For instance, the higher-level geometric shape differ from domain domain. However, these methods are mostly effective in transferring the color and texture which limits the scope of these methods in generative domain adaptation.
Uniqueness of our Approach: Our method is close in spirit to MineGAN (Wang et al. 20205) that aims find a latent transformation yielding the target data. However, there are many differences-
1. Proposed method finds point estimates via inference- siterencee-imeeeee optimization compared to (Wang) al. (020b) where a distribution- transformation sought, leading to overfitting with few samples.
2. We train a latent learner every time we desire to generate a batch of images. This contributes to the diversity.
3. This method does not retrain the source generator with target data which preserves the diversity imbibed via source-data and enables generation from multiple targets at once.
3 PROPOSED METHOD
3.1 OVERVIEW
Given GAN trained on source dataset and few samples from the target distribution, our method (Overview in Fig. 2) optimizes a new multi-layer perceptron to output 'novll' target latent vectors. These are then fed to the source-generator to produce novel images from the target domain. Note
3

---

Published as a conference paper at ICLR 2023
that the generator is not trainable module in method, it is fixed throughout the training. The only trainable module in our method of the latent learning network.
3.2
BACKGROUND
Inference Stage
3.2.1 GAN STYLEGAN
F(-1
Ratghe + Loutt
As in previous works Ojha et 2021); Xiao et al. (2022), we utilize StyleGAN2 Karras et al. (2020b) for adaptation task. StyleGAN2 con- sists of two latent spaces: (i) initial latent space, C R512 and (ii) intermediate latent space, W S 1 A 8-layer feed forward network maps a E Z to w E W. Manipulating an im- age needs finding its latent representation. Previ- ous research (Abdal et al.) 2019) suggests that a shared latent representation in the Z or W space may fail to faithfully embed given image. Re- sults improve if separate code is selected for each layer of the StyleGAN. We call this ex- tended intermediate latent space and denote as W+ We operate on W+
Gos
((
Dow
Raae
Go. Pre-trained Generator
Trainable Medule
Medule
Dg. trained Discrtminator
Buckward
11 Latent Learner
Gratlien
Feature
Figure 2: Proposed Model: Given a trained StyleGAN et al. 2019 2020b) on source dataset and a few examples from the target domain, our method trains latent-learner Lo,, (a MLP) to generate new latent code starting from random Gaussian noise vector. The newly gen- erated latent code is then fed to the source Gener- ator to generate a 'novel' target domain sample. Solid border indicates frozen modules and train- able modules are shown with dotted border. For- ward path and backpropagation are shown with solid and dotted lines respectively.
3.2.2 PROBLEM FORMULATION
Consider a StyleGAN2 model with generator Go having parameters AG and discriminator Dap with parameters 0p. Let (s) be the un- derlying distribution of the source data S which it was trained. Let the extended intermediate latent space of Gea be represented by WT We are given few samples from target data T ~ pr (t). Our objective is to use the trained networks Ge Dep and target images T generate samples I ~ (t). Note, Ge (w)) E RR represents the images generated by the generator Ge and w E W+ E R. as the input from the extended latent space. Note that here m < d.
3.3 LATENT LEARNING NETWORK
objective is to find latent vector wr. E W that lies on the target manifold. To achieve this, we use a feed forward network with ReLU activations which we call the latent learner Ls,, with parameters 0z. The imput this network is random vector X where ~ (O, I) is a sampled from the normal distribution with arbitrary dimensions. Therefore, the objective is obtain wt To, (X) such that Gog (WL) pr()).
During inference/generation
To train the latent learner Lo,, we enforce properties of target domain on Gag(w)) using appro- priate losses involving two components. The first component is the standard style loss (Gatys al., 2016) between and Go. (w,) to capture the target domain features from the image. The second component of the loss function an adversarial loss: the discriminator De, of StyleGAN2 acts as a critic which aims to discriminate between generated image Go (WL) and original target image E T, whereas La,, aims fool the discriminator, thus making the generated image more like that of the target domain. Note, here we are fine-tuning the discriminator Dap along with the training of Leg while the source generator Goa is untouched (Fig. 2 Our training objective is as follows:
02,0% arg min max Or. 0p
(ooyyte + Lodu
(1)
Latyle = E x,t
Σ
Bt
AlbGGg (x)) -
(2)
C,
Lado = log DOD (1) + + [log(1 - ( (Log (X)))
(3)
Here, Latyte is the style loss, R₁ and C₁ are dimensions of feature map 1th layer. A' denotes gram matrix of corresponding feature maps shown in (Gatys et al. 2016). We have used VGG-19 model
4

---

Published as a conference paper at ICLR 2023
(F()) in Fig. 1) to get the feature maps. B1 is the hyperparameter corresponding to -layer which is set to 1 for all I. Lade is the adversarial loss explained above. While training, generator parameters 8g are kept constant, but the gradient is propagated to the latent learner. To update the latent learner, Lo,, we use non-saturating adversarial loss as outlined in (Goodfellow et al. 2014). Also note that fisyle in Equation T doos not contribute to the maximization step but only the minimization step of the min-max game, Once the latent learner converges, its output wi = Lø; (X) is used as the input to the generator to obtain the final generated target images t* = Gaa (w),).
3.3.1 INFERENCE-STAGE LATENT LEARNER RETRAINING
We train the latent learner afresh during inference- stage, whenever we desire to generate batch of im- ages from the target domain. Note that the same set of target samples are used for training Lg, ev- ery time, albeit it is trained afresh with different ini- tialization. This would ensure that Lg, would not overfit on the few target data points (unlike in (Wang) ctt al., (020b)), thus inducing diversity in the gener- ated samples. This procedure nevertheless, comes
Algorithm 1: Proposed Method's Pseudo Code Input: Few target domain images T, pretrained generator Ga and Discriminator Dep
Output: A batch of latent vectors wi, and corresponding generated images t' (wt(
Sample X~ IU,
while not converged do
L Estule Lodu 8b + 0b VV L
at the cost of an increased inference time which is manageable in many practical settings since the la- tent learner is simple 3-layer MLP.
8 - 62 Nor
Generate wi, Log (X) Generate batch of target images t" Goa (1r%)
3.4 THEORETICAL DISCUSSIONS
Figure 3: Pseudo code for training.
In this section, we discuss possible theoretical justi- fication for the proposed inference time optimization framework. Our hypothesis is mostly based on theory developed in Arjovsky & Bottou (017)) The summary of our discussion is that there exist infinitely many optimal discriminators for the min-max problem in Eq. [ Hence, due to different sources of stochasticity in the proposed method, the algorithm could latch on to any of the optimal discriminators leading to different latent learner networks, during each instance of inference. To make the argument more formal, let wr denote the random variable for embedding of the target domain samples T. Also, let wt = Lo; (X) denote the random variable for the latent vectors gen- erated by the latent learner Løz upon convergence. Further, let Mr and Mr denote the manifolds spanned by Gag (WT) and GGgcwr, respectively.
Now, as m < d (refer to Section 3.2.2 we have dim(wr) < diim(Gsa (wr)) and dim(wL) < dim (Gea (w))), hence Mr and My will lie in a subspace of dimension less than d. In other words, Goo (WT) and Ge (wy) do not have full dimension. Also, MMr and Mr will not align perfectly almost surely. From this we can conclude that -= MM n ML has
measure under both Mr and Mr (Lemma 1, Lemma 2 and, Lemma 3 AArovssy & Bottou (2017). From Theorem 2.2 of Arjovsky & Bottou (2017), there exists discriminator, Dar, R2 + 1J that would optimally discriminate between the samples of the generated and real data distributions. Now we hypothesise that due to the stochasticity originating from different sources (such as initialization, SGD, etc.), the proposed method will discover different manifold Mr at each instance in inference, leading to a different optimal discriminator Døy. This in turn leads to a different Lø; during each inference- time optimization. The aforementioned effect is pronounced especially in the case of extremely low target data regime. This is because every novel inference time optimization routine facilitates the Laj network to quickly latch on to a new manifold, MM, We believe that this serves as plausible explanation for the diversity in the generated samples that is observed each inference time of our approach. Developing these ideas further, including formally incorporating the effect of number of target samples on the quality of generation, is a direction for future work.
3.5 IMPLEMENTATION DETAILS
The proposed approach is but inference time optimization. During inference, we solve an optimiza- tion problem in the parameter space of the latent learner using combination of style loss (Equation 2) and adversarial loss (Equation We use 199-19 model to calculate style loss; the layers used to extract feature maps are (convl 2, conv2 2, conv3 4, conv4 4, conv5 4). The latent leamer
5

---

Published as a conference paper at ICLR 2023
is 3-layer MLP with 512 neurons in each layer throughout all the experiments unless otherwise specified. The hidden layers employ 'ReLU" activation, and the final layer has no activation. All the few-shot target samples used in a single batch (of size 8) for computing the style and adversarial losses. The pseudo-code for the entire algorithm is shown in Algorithm (Fig.
4 EXPERIMENTS AND RESULTS
4.1 DATASETS
Following previous work (Li et al. 2020 Ojha et al. 2021 we consider Flickr Faces HQ (FFHQ)
(Karras et al. 2019) as one of the source domain datasets and adapt the following target do-
mains (i) FFHQ-Babies (Ojha et al. 2021), (ii) FFHO-Sunglasse (Ojha et al. 2021). (iii) face
sketches (Wang& & Tang 2009). (iv) emoji faces from bitmoji com API (Taigman et al. 2016
Hua et al. 2017). and (v) portrait paintings from the artistic faces dataset (Yaniv et al. 2019). Next,
we consider LSUN Church (Yu et al. 2015) as source domain and adapt to (i) haunted houses (Ojha
eet al.,
2021 and (ii) Van Gogh's s house paintings (Ojha et al. 2021). As in previous works (Li
et al. 2020 Ojha et al., 2021). we also consider combinations of unrelated source and target do-
mains (FFHQ + haunted house and Church - face sketches) to probe if the proposed method can
adapt successfully when the source and target domains are not 'close' (cf. supp.). We work with 256 256 images for both source and target domain. All our experiments consider 10 randomly sampled target samples as in previous work unless otherwise specified. For source models, we reuse a pretrained model (Seonghycon| 2019) for FFHQ 256 x 256 resolution and the official pretrained (icippoon((Karaas & Hellsten, 2019) for LSUN Church.
4.2 METHODOLOOGY, METRICS AND BASELINES
Methodology: Even though very few (1, or 10) target examples are used in the method for adaptation, the evaluation is conducted on larger target set. For example, there are approximately
300, 2500, 2700 and unlimited examples in the sketches (Wang & Tang, 2009). PFHQ-Babic (Ojhal et al. 2021), FFHQ-Sunglasses (Ojha et al., 2021), and emoji (Taigman et al. 2016 Hua et al. 2017) datasets, respectively. In such settings, we use the entire target dataset (10000 samples for emoji) for evaluation purposes by generating so many targets using our method. All the mentioned datasets have a dimension of 256 x 256, results on higher resolution images can be found in appendix.
Metrics: We compute and report the widely used Fréchet Inception Distance (FID) (Heusel et al.) 2017) for measuring the similarity of the generated images to the real ones. A lower FID score indicates high similarity. However, being uni-dimensional score, FID cannot disentangle the two aspects of sample quality and diversity. To alleviate this issue, (Nacem et al. (2200) proposed den- sity and coverage metrics to quantify quality and diversity, respectively. Density is unbounded, and a higher density score indicates better quality. Coverage is bounded by 1, and a higher coverage score is preferred. Another metric that we utilize is the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) metric, which gives an idea about overfitting on the small amount of target data as shown (Ojha et al. (202T).
Baselines: We compare our method against the following baselines (discussed in Sec. 2) 2 Trans- ferring GAN (TGAN) (Wang et al., 2018), Batch Statistics Adaptation (BSA) (Noguchi & Harada 2019) MineGAN (Wang et al.) (0200b. Freeze-D (Mo et al. 2020), Non-leaking Adaptive Data Augmentation (Karras et al. 2020a|| Zhao et al. 2020b) (TGAN ADA), Elastic Weight Consol- idation (EWC) (Li et al. 2020), Few-shot Image Generation via Cross-domain Correspondence
(CDC) Ojha et al. 2021 ), C.: Contrastive Learning for Cross-domain Correspondence (Lee et al.,
2021
and Relaxed Spatial Structural Alignment (Xiao et al. 2022).
4.3
RESULTS
As can be seen from Table [ our method outperforms the current state-of-the-art methods on all of the four datasets as measured by FID, From Table 2 it can be seen that the proposed method achieves the best density and coverage scores, indicating the generated samples are diverse and of high quality.
We have taken the numbers from the paper itself, as the code is not yet published by the authors. For the same reason, we are unable to present qualitative comparison with this method.

---

Published as a conference paper at ICLR 2023
Table 1: FID scores (4) for different target datasets.
Method
Babies
Sunglasses
Sketches
Bitmoji
TGAN (Wang al. 2018
104.79 = 0.03
55.61 = 0.04
53.41 = 0.02
66.69 = 0.05
TGAN ADA (Karras el
2020a
102.58 1 0.12
53.64 1: 0.08
66.99 + 0.01
68.71 1: 0.07
BSA (Noguchi & Harada
2019
140.34 = 0.01
76.12
69.32 0.02
105.56 = 0.03
FreezeD (Mo et 2020
110.22
51.29
4.05
71.16 0.01
MineGAN (Wan cl
2020b
68.91 0.03
64.34
86.40 0.04
EWC (Li ct al. 2020
87.41
59.73
0.02
73.82 0.04
CDC (O]ha ot al. 2021
74.39 0.03
42.13
45.67
0.02
69.54 0.05
C eL al. 2021
67.55 2.23
36.69
41.50 1.64
RSSA [Xtao et al. 2022
75.67 0.39
44.35 0.06
54.58 0.51
67.14 0.63
Proposed Method
0.05
35.64 0.15
35.59
64.50 0.12
Table 2: Comparison of Density (1) and Coverage (1) scores for FFHQ babies and sketches datasets.
Table 3: Intra-cluster pairwise LPIPS distance (1)
Method
Amedeeo's Paintings
Sketches
TGAN (Wang 2018
0,41:0055
0.39300:5
Mathod
Deasily
Dessity
TGAN FADD (Kamas 2020a
RSA NNguehii & Halne 2019
0.51:0004
0.39=00.4
0.41:0005
0.35+00.11
T6AN COOEE
0,219
[1:20
Freez (Mpore 2000
0.40=:0.03
0.39::0.33
TCIN
Frused)
Mine(i1
Nt Wiree 20000
Maxt prant
0415
0.454
0,M12
0.211
(1.865
0.125
1L396
D415
0.195
0:174
0.263
0.456
MineGAN
40206
0.5210.03
0.4220.03
0,1210.03
0.40:10.55
0.301
0.325
CDC Ore urre sual
0.60::001
0.45:00.22
CDC
0.690
0.667
0.149
04492
0.4510.03
RSS4
0.961
0.800
0.070
0.91
RSS
6022
0.53-0007
0.43-40.05
0.611
0.418
Proposed Aeturco
0.61=0.02
As in (Ojha et al. 2021), assign 1000 gener- ated images to one of the lt possible clusters (for
Table 4: Ablation studies to understand the impact of different loss terms.
k-shot generative domain adaptation, k 10 in [able based on the lowest LPIPS distance (Zhang et al. 2018). Next, we compute the av- erage pair-wise LPIPS metric among the mem- bers of the same cluster. Finally, we take the average over the & clusters. A method will have a zero score if it reproduces the original images.
Loss Components
FID Score (c)
Adv. Loss
Sty. Loss
Babies
Sketches
X
69.78
68.99
X
/
212.87
139.12 12
1
62.14
35.59
A lower value of this metric implies the gen- erated images are similar or, in other words, less diverse. As can be seen from Table 3 our proposed method achieves the best intra- cluster pair-wise LPIPS distance for Amedeo Modigliani's 's paintings and sketches datasets. Figure andd a few examples of gener- ated targets from our method and the previous SoTA (Ojha al. 2021: Xiao al.] (022). It can be seen that our method can generate di- verse examples on multiple target domains. (See appendix for more qualitative results on different target domains. Appendix also contains examples of editing using the learnt representations.)
Table Impact of the capacity of the L8, on gen- eration quality.
FID Score (4) Sketches
Latent learner Capacity
Babies
Small Network
Trainable parameters: 787,456
66.28
35.22
Medium Network
Trainable parameters: 1,050,112
62.14
35.59
Large Network
Trainable parameters: 2,624,000
64.43
38.61
4.4 ABLATION STUDIES
Loss Components: Table
Table 6: Effect of & in k-shot adaptation on generation quality as mea- sured by FID score (J).
4
presents the impact of the two loss components using two datasets FFHQ-Babies and Sketches. The best FII score is achieved when both the loss components are present. Further, the style loss acts more regularizer, whereas adversarial loss improves quality.
I-shot
3-shol
10-shot
Method
Bables
Sketches
Bables
Sketches
Bables
Sketches
CDC Olba et al 2021
105.5 58
81.95
73.63
51.01
74.39
45.67
RSSA Xino et al 2002
157.84
119.66
96.42
63.34
75.67
54.58
Proposed Method
65.47
41.88
35.59
Lo, Capacity: Next, to understand the impact of the complexity of the latent learner (overfitting vs underfitting) generation quality. in Table 5 we consider three capacities for the latent learner. The smaller capacity Lo, is a 2-yeyer MLP with 512 neurons in each layer, the larger capacity Le,, is again layer MI.P with 1024 neurons in the hidden layers and 512 neurons in the output layer, and
7

---

erated Fernand Leger's 's art. Published as a conference paper at ICLR 2023
(a) 10-shot training exam- (b) Babies generated using (c) Babies generated using (d) Babies generated using
ples of Babies.
CDC.
RSSA.
our method.
(e) 10-shot training exam- (I) Images generated using (g) Images generated using (h) Images generated using
ples of Sunglasses.
CDC.
RSSA.
method.
(i) 10-shot sketch exam- ples.
CDC
generated (k) RSSA
generated
Sketches generated
sketches.
sketches.
ing our method.
(m) 10-shot examples (n) Emojis generated using (o) Emojis generated using (p) Emojis generated using CDC.
bitmo ji com emojis.
RSSA.
our method.
(q) 10-shot training exam- (r)
CDC
generated (s) RSSA generated paint-
(1) Amedeo's painting gen- erated using our method.
ples of Amedeo's 's painting. Amedeo's 's painting.
ing.
(u) 10-shot training exam- (v) CDC generated Moise (w)
RSSA Moise Kisling's art.
generated (x) Our method generated
ples of Moise Kisling's art. Kisling's art.
Moise Kisling's art.
25
(y) 10-shot training
CDC generated Fer- (aa) RSSA generated Fer- (ab) Proposed method gen- nand Leger's art.
ples Fernand Leger's art. nand Leger's art.
Figure 4: Source: FFHQ. Each row represents one target domain among Babies, Sunglasses, Sketches, Emoji, Amedeo Modigliani's paintings, Moise Kisling's art, and Fernand Leger's art. The 1st column presents 10-shot target examples. The 2nd 3rd and 4th columns present 10 images generated by CDC (Ojha et al. 2021). RSSA (Xiao et al. 2022), and proposed method respectively.
medium capacity is 3-layer MLP with 512 neurons in each layer. It is seen, the medium-capacit network used across all the experiments achieves the optimum performance.
Number of Target Examples: Till now, the results presented utilize 10 target domain images for generative domain adaptation. Table 6 presents the impact lesser training data on generation quality. We compare the proposed method with the current SOTA methods (Ojha et al., 2021: Xiao et al.) 2022). It is seen, the generation quality improves with k, and the proposed method performs considerably better in almost all cases considered. Refer to the appendix for qualitative analysis.
4.5 APPLICATION: IMAGE TRANSLATION
One potential application of our method is to transfer the style of the image of one domain to another domain. Figure presents some examples where source domain (FFHQ) images have been trans-
8

---

Published as a conference paper at ICLR 2023
(a) Random images generated using source StyleGAN2 trained on LSUNN Church (uu et al. 2015).
(b) 10-shot training exam- (c) Haunted house gener- (d) Haunted house gener- (c) Proposed method gen-
ples of haunted house. ated using CDC.
ated using RSSA.
erated haunted houses.
(I) 10-shot examples of (g) CDC generated Van (h) RSSA generated Van Proposed method gener- Van Gogh's paintings.
Gogh style paintings.
Gogh style paintings. ated Van Gogh paintings.
Figure 5: Source: LSUN Church (Yu et al. 2015). Target: Haunted house (Ojha et al. 2021 and Van Gogh's 's house paintings (Ojha al. (022). The 1st column of 2nd and 3 row presents 10- shot examples from the target domains. The 2nd 3rd and 4th columns present 10 images generated using CDC (Ojha et al. 2021), RSSA (Xiao et al. 2022), and proposed method respectively.
(a)
(b)
(c)
(d)
Figure 6: Transferring source domain (FFHQ) image (left most image in each sub-figure) to target domains Babies (center image) and Sketches (right most image).
lated to target domains babies and sketches. To do this, we use the embedding of the source image and target images in place of random input noise to the latent learner. Further, we incorporate two additional loss terms: (i) Multi-scale SSIM loss between the source image and generated image, (ii) Structural loss between source, target, and generated images. Please note, as before, we keep the source generator fixed and update only the latent learner parameters, See appendix for more details.
LIMITATIONS, RISK AND CONCLUSION
We conclude this article with discussion on limitation of this work and possible future directions. I Limitation: In our method, inference time optimization mandates that we solve optimization problem every time we want to generate new images, which may increase the inference time for certain applications. One way to address this issue to store the outputs of the latent learner solve inference-time optimization and train another generative model to sample from the manifold of interest in the latent space. To validate the proof of this concept, we trained WGAN (Arjovsky jet al. 2017 Gulrajani et al. 2017) to generate latent codes for the babies and sunglasses datasets and achieved comparable FID scores. We intend to probe this direction further in our future work. Risk: One of the possible negative social impacts of deep generative frameworks is the creation of deepfakes (Vaccari & Chadwick 2020). However, significant amount of work (Wang et al., 22200a] Dolhansky et al. 2020, Mirsky & Lee, (22T]) have been done to tackle the problem of identifying real vs. fake images to counter deepfakes.
Conclusion: In this work, we present methodology to build data-efficient generative models, demonstrating that existing source models can be used as is (without retraining or fine-tuning) to model new distributions with less data. We believe that our work is an important step towards few- shot "generative domain adaptation' where we have demonstrated that the same source generator can be utilized effectively to generate source domain samples and multiple target domain samples. Additionally, our proposed method can viewed as a step towards continual learning for generative task where the same generator generate data from different domains.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
To ensure that the proposed work is reproducible, we have included an Algorithm (Refer to Fig. 3.. We have clearly mentioned different loss terms (Refer to Eq. 2 3). Further, we have an explicit section (Refer to Sec. 3.5p on implementation details. The code of the proposed method available at: https: : githab. com woom//rnaabmondaa//aaoom /GenDA This an anonymous link which doesn't I reveal author identity. For convinience, we have also included a ReadMe file using which our results can be reproduced.
ACKNOWLEDGMENTS
We thank IT Delhi HPC facility for computational resources. Parag Singla is supported by IBM AI Horizon Networks (AIHN) grant and IBM SUR awards. Any opinions, findings, conclusions or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views or official policies, either expressed implied, of the funding agencies.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proc. ICCV, pp. 4432 4441, 2019.
Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka. Styleflow: Attribute-conditioned exploration of stylegan-generaae generated images using conditional continuous normalizing flows. ACM doi: 10.1145/344764.
Trans. Graph., 40(3), 2021. ISSN 0730-0301.
Martin Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In Proc. of ICLR, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proc. of ICML., pp. 214-223, 2017.
Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. The deepfake detection challenge (dfdc) dataset. arXiv preprint arXiv:2006.07399 2020.
Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. StyleGAN-NADA: Clip-guided domain adaptation of image generators. arXiv preprint arXiv:2108, 00946, 2021.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. Image Style Transfer Using Convolutional Neural Networks. In Proc. of CVPR, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Nets. In Proc. of NeuRIPS, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training wasserstein gans. In Proc. of NeuRIPS, 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Proc. of NeurlPS, 2017.
Xinru Hua, Davis Rempe, and Haotian Zhang. Unsupervised Cross-Domain Image Generation.
Stanford Course, CS229 Project, 2017.
Xun Huang and Serge Belongie. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. Proc. ICCV 2017.
Tero Karras and Janne Hellsten. stylegan2. http:://ggttubooocmm//aaaaattttt 2019.
http://ssperrcoppttinng.
10

---

Published as a conference paper at ICLR 2023
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive Growing of GANs for Improved Quality, Stability, and Variation. In Proc. of ICLR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A Style-Based Generator Architecture for Generative Adversarial Networks. In Proc. of CVPR, 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In Proc. of NeurlPS, 2020a.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and Improving the Image Quality of StyleGAN. In Proc. of CVPR, 2020b.
Diederik Kingma and Max Welling. Auto-encoding Variational Bayes. In Proc. ICLR, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Demis Hass-
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521-3520 2017. ISSN 0027-8424.
Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Sam Kwong, Jialu Huang, and Jing Liao. Unsupervised Image-to-Image Translation via Pre-trained StyleGAN2 Network. IEFE Transactions on Multimedia, 2021.
Hyuk-Gi Lee, Gi-Cheon Kang, Chang-Hoon Jeong, Han-Wool Sul, and Byoung-Tak Zhang. C²: Contrastive learning for cross-domain correspondence in few-shot image generation. In Proceed- ings of Workshop on Controllable Generative Modeling in Language and Vision (CtrlGen) at NeurlPS 2021, 2021.
Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Universal style transfer via feature transforms. In Proc. of NeurlPS, 2017.
Yijun Li, Richard Zhang, Jingwan (Cynthia) Lu, and Eli Shechtman. Few-shot Image Generation with Elastic Weight Consolidation. In Proc. of NeurlPS, 2020.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis. In Proc. of ICLR, 2021.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. 1989.
Yisroel Mirsky and Wenke Lee. The creation and detection of deepfakes: A survey, ACM Computing Surveys (CSUR), 54(1):1-41, 2021.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze the Discriminator: Simple Baseline for Fine- Tuning GANs. In CVPR Al for Content Creation Workshop. 2020.
Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models. Proc. of ICMI, 2020.
A. Noguchi and T. Harada. Image Generation From Small Datasets via Batch Statisties Adaptation. In Proc. of ICCV, 2019.
Utkarsh Ojha, Yijun Li, Cynthia Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. In Proc. of CVPR, 2021.
Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh AP. Generalization on unseen domains via inference-time time label-preserving target projections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12924-12933, 2021.
Justin NM Pinkney and Doron Adler. Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains. arXiv preprint arXiv:2010.05334 2020.

---

Published as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. of ICML. pp.
2021.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-image translation. In Proc. of CVPR, 2021.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. SinGAN: Learning a Generative Model from Single Natural Image. In Proc. of ICCV, 2019.
Kim Seonghyeon
Eacggan--pytorch
https://ggttubb.comrrrssiiiiliit
Stylegan2-pytoree 2019.
Guoxian Song, Linjie Luo, Jing Liu, Wan-Chun Ma, Chunpong Lai, Chuanxia Zheng, and Tat-Jen
Cham. AgileGAN: Stylizing Portraits by Inversion-Consisten Transfer Learning. ACM Transac- fions on Graphics (TOG), 40(4), 2021. ISSN 0730-0301. doi: 10.1 10.1145/34502666455977 URL
https://doi.org/ 10. 1141/145553131 3459771
Vadim Sushko, Jürgen Gall, and Anna Khoreva. One-Sho GAN: Learning Generate Samples from Single Images and Videos. In Proc. of CVPRW, 2021.
Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised Cross-Domain Image Generation. arXiv preprint arXiv: 1611 1.02200, 2016.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Trans. Graph., 40(4), 2021. ISSN 0730-0301. doi: 10. 00.1145/345502664555333
Cristian Vaccari and Andrew Chadwick. Deeplakes and disinformation: Exploring the impact of synthetic political video deception, uncertainty, and trust in news. Social Media+ Society, (1):20563051 129003008 2020.
Sheng- Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Con-generated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8695-8704, 2020a.
Xiaogang Wang and Xiaoou Tang. Face photo-sketch synthesis and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(11):1955-166 2009. doi: 10. 109/TPPMM. 2008.222.
Yaxing Wang, Chenshen Wu, Luis Herranz, Joost van de Weijer, Abel Gonzalez-Garcia, and Bogdan Raducanu. Transferring GANs: generating images from limited data. In Proc. of ECCV, 2018.
Yaxing Wang, Abel Gonzalez-Garcia, David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. MineGAN: effective knowledge transfer from GANs to target domains with few images. In Proc. of CVPR, 2020b.
Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap. Logan: Latent optimisation for generative adversarial networks. arXiv preprint arXiv:1912.0095: 2019.
Jiayu Xiao, Liang Li, Chaofei Wang, Zheng-Jun Zha, and Qingming Huang. Few shot generative model adaption via relaxed spatial structural alignment. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 11204-12213, 2022.
Jordan Yaniv, Yacl Newman, and Ariel Shamir. The face of art: Landmark detection and geometric style in portraits. ACM Trans. Graph., 38(4), 2019. doi: 10. 00.114553306344332229844
Fisher Yu, Yinda Zhang, Shuran Song. Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 1506v:1506.03365. 2015.
Jianfeng Zhang, Xuecheng Nie, and Jiashi Feng. Inference stage optimization for cross-scenario 3d human pose estimation. arXiv preprint arXiv:2007.02054. 2020.
12

---

Published as a conference paper at ICLR 2023
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness deep features as perceptual metric. In CVPR, 2018.
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On Leveraging Pretrained GANs for Limited-Data Generation. In ICML, 2020a.
Shengyu Zhao, Zhijian Liu, Lin, Lun-Yan Zhu, and Song Han. Differentiable Augmentation for Data-Efficient GAN Training. In Proc. of NeurlPS, 2020b.
Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka. Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks. In Proc. of ICLR, 2022.
13


Reviews:
[{"review_id": "tTpgkKcZI8", "reviewer": "Reviewer_ubZS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method to generate samples from a target domain containing very few shots using a fixed pretrained GAN. ", "summary_of_the_review": "The proposed method is simple and intuitive, with strong quantitative/qualitative results. However, the focus on exclusively extremely few shot setting is somewhat limiting, so I advocate for a weak accept.\n\nPost rebuttal: My concerns have been addressed, therefore I increase my score to accept. ", "strength_and_weaknesses": "Strengths:\n\n- concise and straightforward method for performing the task at hand\n- generally well-written and easy to follow\n- compelling results for this very-few-shot domain adaptation generation\n\nWeaknesses:\n\n- The paper feels like it could be a bit shortened. The related work and background are longer than I'd expect, and there are fewer qualitative examples than I'd like in the main text. \n- I am concerned about this method's generalization to higher number of shots. I know other few-shot works use 30 or 100 examples. (I don't mind needing to retrain the networks for each new generation batch). How does this method perform at different shot capacity?\n- I am curious about the effects of latent learner's capacity. What happens with an extremely small model (1 fully connected layer)? When the model is the identity function, do the learned point simply become training set? \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZXuxGTWcm9", "reviewer": "Reviewer_jAjp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper shows a few shot GAN by embed latent factor into an existing GAN to transfer to a new style. This paper tries to mix pre-trained\nGAN with few shot learning. \n\nTheir contribution can be concluded as: \n-propose a simple procedure to utilize a GAN trained on large-scale source-data to\ngenerate samples from a target domain with very few (1-10) examples.\n- Their procedure is shown to be capable of generating data from multiple target domains\nusing a single source-GAN without the need for re-training or fine-tuning it.\n- Extensive experimentation shows that our method generates diverse and high-quality target\nsamples with very few examples surpassing the performance of the baseline methods.", "summary_of_the_review": "In this work, authors present a methodology to build data-efficient generative models, demonstrating that existing source models can be used as it is (without retraining or fine-tuning) to model new distributions with less data. The method is not very complicate but it shows a \ngood direction and very reproducible work as a direction.", "strength_and_weaknesses": "Strength:\n- Highly achievable and reasonable.\n  Generally the pain part of GAN is that latent space is always not fully exploitable through training. Existing methods focus too much on \nmodel itself but not build on transfer learning. \n\nWeakness:\n- This method still needs some high resolution and complex dataset to demonstrate. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KbuBRyR5_G", "reviewer": "Reviewer_8gcS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a simple approach to learn a particular style of generative model using a base model as StyleGAN. It learns a small latent encoder embedding network which encodes the normal distribution into the target latent space. It learns using the style loss proposed by Gatys et al and the StyleGAN Discriminator's adversarial loss. The results are comprehensive and the technique is beneficial for face editing applications.", "summary_of_the_review": "The simplicity of the approach by learning an encoder to transform normal distribution to the appropriate target latent space of the StyleGAN network provides an interesting approach to face stylization. The work provides the right references as well as compares against well known techniques such as StyleGAN NADA on well backed metrics. \n\nThe paper deserves to be accepted to ICLR as it adheres to its high standards. ", "strength_and_weaknesses": "Strengths:\n\n* Simple technique for learning a particular style using a set of few images (~10)\n* Easy to train and the training converges in a reasonable time\n* Provides an alternative to StyleGAN NADA where the style cannot be explicitly described in a few words \n\nWeaknesses:\n\n* The style loss using Gram Matrix only allows particular kinds of texture based styles to be enabled by this technique\n* Geometric transformations such as adding a smile or wink would be harder with this kind of technique.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JK14SOAh3x", "reviewer": "Reviewer_G9qY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel method to generate realistic images on the target sparse domain by transferring a large-scale generator from the source domain without retraining the generator. Also, the proposed method can generate diverse target samples. The key idea is to solve an optimization problem on the latent domain of the generator at inference time. The proposed methodology is clearly stated and extensive comparisons with state-of-the-art methods have been studied. The proposed method has shown very competitive numbers in the experiments.", "summary_of_the_review": "The proposed method shows great potential in mitigating the overfitting and diversity issues in solving domain transfer problems. The experiments have shown promising results with the proposed method. The paper is overall well presented. I would recommend acceptance of the paper.", "strength_and_weaknesses": "*Strengths*\n- The presentation quality of the paper is of good quality: the proposed method is well stated.\n- Comparison with previous literature is well studied.\n- The proposed method is novel in that it provides an intuitive yet efficient way to reduce overfitting and increase diversity.\n- Numerical studies and comparisons with previous methods have been well conducted. The results show that the proposed method is at least comparable to or better than state-of-the-art methods.\n*Weaknesses*\n- Although this paper is experiment-biased, it could be more plausible to include theoretical discussions on why the introduction of the latent space learner could help reduce overfitting and increase diversity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%