Adaptive IMLE for Few-shot Image Synthesis
Authors: Mehran Aghabozorgi
Shichong Peng
Ke Li
Venue: Submitted to ICLR 2023 (2023)

Abstract: Despite their success on large datasets, GANs have been difficult to apply in the few-shot setting, where only a limited number of training examples are provided. Due to mode collapse, GANs tend to ignore some training examples, causing overfitting to a subset of the training dataset, which is small to begin with. A recent method called Implicit Maximum Likelihood Estimation (IMLE) is an alternative to GAN that tries to address this issue. It uses the same kind of generators as GANs but trains it with a different objective that encourages mode coverage. However, the theoretical guarantees of IMLE hold under restrictive conditions, such as the requirement for the optimal likelihood at all data points to be the same. In this paper, we present a more generalized formulation of IMLE which includes the original formulation as a special case, and we prove that the theoretical guarantees hold under weaker conditions. Using this generalized formulation, we further derive a new algorithm, which we dub Adaptive IMLE, which can adapt to the varying difficulty of different training examples. We demonstrate on multiple few-shot image synthesis datasets that our method significantly outperforms existing methods. 

Under review as a conference paper at ICLR 2023
ADAPTIVE IMLE FOR FEW-SHOT IMAGE SYNTHESIS
Anonymous authors
Paper under double-blind review
ABSTRACT
Despite their success on large datasets, GANs have been difficult apply in the few-shot setting, where only limited number of training examples are provided. Due to mode collapse. GANs tend to ignore some training examples, causing overfitting to a subset of the training dataset, which is small to begin with. A recent method called Implicit Maximum Likelihood Estimation (IMLE) an alternative to GAN that tries to address this issue. It uses the same kind of generators as GANs but trains it with a different objective that encourages mode coverage. However, the theoretical guarantees of IMLE hold under restrictive conditions, such as the requirement for the optimal likelihood at all data points to be the same. In this paper, we present more generalized formulation of IMLE which includes the original formulation as a special case, and we prove that the theoretical guarantees hold under weaker conditions. Using this generalized formulation, we further derive a new algorithm, which we dub Adaptive IMLE, which can adapt to the varying difficulty of different training examples. We demonstrate multiple few- shot image synthesis datasets that our method significantly outperforms existing methods.
INTRODUCTION
Image synthesis has achieved significant progress over the past decade with the emergence of deep
learning. Deep generative models such as GANs (Goodfellow et al. 2014 Brock al.
2019 Karras
et al. 2019: 2020; 2021). VAEs (Kingma & Welling 2013: Vahdat & Kautz
2020
Child 1021
Razavi et al. 2019). diffusion models (Dhariwal & Nichol 2021 Ho et al.
2020). score based
models (Song et al. 2021 Song & | 2019), normalizing flows (Dinh et al. |017]] Kobyzev
al.
2021 Kingma & Dhariwal 2018) and autoregressive models (Salimans et al., 201 van den
Oord et al. 2016b have made incredible improvements in generated image quality, which makes it
possible to generate photorealistic images using these models.
Many of these deep generative models require training quality images. However, there are many real-life scenarios in that only a limited number training examples available, such as orphan diseases in the medical domain and rare events for training autonomous driving agents. One way to address this issue is by fine-tuning a model pre-trained on large auxiliary dataset from similar domains (Wang et al.) 2020 Zhao al.. 2020a Mo et al.; (020). Nonetheless, a large auxiliary dataset with a sufficient degree of similarity to the task at hand may not be available in all domains. If insufficient similar auxiliary dataset were used regardless, image quality may be adversely impacted, as shown in (Zhao et al. 2020b). In this paper, we focus on the challenging setting of few-shot unconditional image synthesis without auxiliary pre-training.
large-scale datasets to produce high-
The scarcity of training data in this setting makes it especially important for generative models to make full use of all training examples. This requirement sets it apart from the many-shot setting with abundant training data, where ignoring some training examples does not cause as big an issue. As result, despite achieving impressive performance in the many-shot setting, GANs are challenging to apply to the few-shot setting due to the well-known problem of mode collapse, where the generator only learns from a subset of the training images and ignores the rest. A recent work (Li & Malik 2018) proposed alternative technique called Implicit Maximum Likelihood Estimation (IMLE) for unconditional image synthesis. Similar to GAN, IMLE uses a generator, but rather than adopting an adversarial objective which encourages each generated image to be similar to some training images, IMLE encourages each training image to have some similar generated images. Therefore, the generated images could cover all training examples without collapsing to subset of the modes.

---

Under review as a conference paper at ICLR 2023
IMLE
Generated Samples
Real Data
Generated Sumples
Real Data
Adaptive IMLE
Generated Samples
Real Data
Generated Samples
Real Dats
Figure I: Schematic illustration that compares vanilla IMLE (Li & Malik 2018) (top row) with the proposed algorithm, Adaptive IMLE (bottom row). While IMLE treats all training examples (denoted by the squares on the left) equally and pulls the generated samples (denoted by the circles on the left) towards them uniform pace, Adaptive IMLE adapts to the varying difficulty of each training example and pulls the generated samples towards them at an individualized pace that depends on the training example. The dashed line on the left figure illustrates the progression towards three data points at four comparable epochs with the starting positions highlighted. The corresponding generated samples are shown on the right, As shown, Adaptive IMLE can converge to the various data points faster and closer than IML.E
However, the theoretical guarantees of IMLE hold under restrictive conditions, of which is that all data points should have an identical optimal likelihood. The IMLE algorithm, therefore, treats all training examples equally when optimizing the model parameters and ignores the varying difficulty in learning from different training examples. As shown in the top row of Fig. [ the generated samples make uneven progress toward different training examples using IMLE, leading overfitting some examples and underfitting to others. While this may not cause a major issue in the many-shot setting because many data points are expected to have similar optimal likelihoods, it can be quite problematic in the few-shot setting, since incorrectly weighting even a few training examples can impact the model quality substantially due the small total number of training examples that the model is trained on.
In this paper, we introduce a generalized formulation of IMLE, which in turn enables the derivation of a new algorithm that requires fewer conditions and gets around the aforementioned issue. In particular, we mathematically prove that the theoretical guarantees of the generalized formulation hold under weaker conditions and subsumes the IMLE formulation as special case. Furthermore, we derive an algorithm called Adaptive IMLE using this generalized formulation, which could adapt to points with different difficulties, illustrated in the bottom row of Fig. []] We compare our method to existing few-shot image synthesis baselines over six datasets and show significant improvements over the baselines in terms of mode modelling accuracy and coverage.
2 RELATED WORK
There are two broad families of work on few-shot learning, one that focuses discriminative tasks such as classification (O' Mahony et al. 2019 Finn et al.) 2017]] Snell et al. 2017) and another that focuses generative tasks. In this paper, we focus the latter. Similar to many generation tasks, few-shot generation tasks take a limited number of training examples as input and aim to generate samples that are similar to those training examples. What is different from the many-shot
2

---

Under review as a conference paper at ICLR 2023
setting is that it is crucial for the generative model to utilize all the training examples in the few-shot setting. Due to the scarcity of available data for training, ignoring even just a few data points would cause more serious issue in the few-shot setting than in the many setting. One line of work focuses on pre-training on large-scale auxiliary datasets from similar domains and adapting the pre-trained models for the few-shot task. This has been applied to unconditional image generation (Li)
et 2020; Zhao et al. [2020a; Mo et al. [2020] Ojha et al. 2021; Wang et al., (020), conditional
image generation (Sinha et al. 2021| Liu al. 2019) and video generation ( (Wang et al., 2019).
However, there are no guarantees on the existence of such large-scale auxiliary datasets for all
domains, and recent studies (Zhao et al., 2020b; Kong et al.] (0022) also showed that fine-tuning from
a dissimilar domain could even lead to the degradation of generated image quality.
In this paper, we focus on the setting without fine-tuning pre-trained models from auxiliary datasets. Most prior work considered applying GANs to this setting and developed methods for alleviating the well-known mode collapse problem of GANs. FastGAN (Liu et al.) 202T) introduced a skip-layer excitation module for faster training and used self-supervision for the discriminator to learn more descriptive features, which aids better mode coverage of the generator. MixDL (Kong et al. 2022) introduced two-sided distance regularization to facilitate learning smooth and mnoee-preeerving latent space. Despite these improvements, some degree of mode collapse still remains. A recent method called Implicit Maximum Likelihood Estimation (IMLE) (Li & Malik] 2018) adopted different objective function and showed promising results towards alleviating mode collapse on unconditional image synthesis tasks. Prior IMLE-based methods mainly focused on conditional image synthesis (Li* et al. (2000) Peng et al., 2022). In this work, we build on (Li & Malik, (0188) and introduce a novel and more generalized formulation of IMLE make more suitable for the unconditional few-shot setting.
3 BACKGROUND: IMPLICIT MAXIMUM LIKELIHOOD ESTIMATION (IMLE)
In unconditional image synthesis. the goal is to learn the unconditional probability distribution of images p(x), from which samples can be drawn to yield new synthesized images. It is common to use an implicit generative model one example of such model is the generator in GANs, which takes the form of a function To parameterized as a neural network with parameters 6 which maps latent codes drawn from standard Gaussian N(0, I) to images x. One way to learn this model is with the GAN objective, which introduces a discriminator that aims to distinguish between generated images Tg (z) and real images The generator is trained to produce more realistic images that would fool the discriminator. However, the output To (z) tends to recover only subset of the training examples even when varying all values of z. This issue is known mode collapse, and the intuitive reason behind it is that the adversarial objective of GAN only encourages each generated sample to be similar to some training examples, but there is no guarantee that all training examples will have some similar generated samples. In the few-shot image synthesis setting, the issue of mode collapse is even more significant given the limited number of training examples that are available in the first place.
A more recent method known as Implicit Maximum Likelihood Estimation (IMLE) (Li & Malik 2018) proposed an alternative objective to address this issue. Instead of making each generated sample similar to some training examples, IMLE tries to ensure that samples can be generated around each training example x,. The generator Te is encouraged to pull some samples Tz,y) towards each x,, thereby rewarding coverage of the modes associated with all training examples.
More precisely, the IMLE objective takes the following form:
min E N ((.J) 0
Σ min d (x,, To(z,)) JE(m)
where d(.,-) is distance metric, is hyperparameter, and x, is the 7th training example. The training procedure involves finding the nearest generated sample index (i)) to each training example X1, and optimizing the model parameter 0 by minimizing the distance from the selected sample To (2)(1) to the target data x,. Detailed pseudocode of the algorithm can be found in the appendix.
Despite the algorithm's simplicity, restrictive conditions need to be satisfied for the theoretical guarantees of IMLE to hold, such as requiring a uniform optimal likelihood for all data points. As an example, consider dataset with two clusters with the same number of points where one cluster
3

---

Under review as a conference paper at ICLR 2023
has large variance and the other has small variance. In this case, the training examples from the high-variance cluster are more difficult to learn than the training examples from the low-variance cluster, because of sparser coverage of the space in the former cluster. If we consider what the ground truth data distribution looks like, is a bimodal distribution, with the mode corresponding to the low-variance cluster having higher likelihood than the other. So requiring uniform optimal likelihood for all data points, as IMLE does, will result in overfitting to the low-variance cluster and underfitting the high-variance cluster, which is not optimal. We refer readers the IMLE paper 2018) for more details.
(Li & Malik
4 METHOD
In this paper, we devise generalized formulation of IMLE, whose theoretical guarantees hold under more general conditions than vanilla IMLE (See/4.T). This formulation subsumes vanilla IMLE as special case and also gives rise to new algorithm which we call Adaptive IMLE. It turns out that Adaptive IMLE offers theoretical and practical advantages over IMLE, which we will demonstrate (Sec 4.2)
4.1 GENERALIZED FORMULATION
Since To is implicit generative model, the likelihood induced by the model cannot in general be expressed in closed form, and so evaluating it numerically is typically computationally intractable. In order to train the generative model, we would like to maximize the likelihood of the training examples without actually needing to evaluate the likelihood. Below we will consider the generalized objective we propose and show that optimizing the objective is equivalent maximizing the sum of likelihoods at the training examples, without requiring the evaluation of likelihood.
Consider the following optimization problem:
1 m
max C (T:), max E E ((.00)
T:
(d(x,
(d(x,,)((z))) To(zz)))
(2)
12
wi
where Te, d(.,-) and as defined in Equ insight revealed by lemmas below.
I
We will choose and (-) later based on the
We will present the high-level sketches of our key lemmas (omitting technicalities) and delineate their interpretations and significance. The precise statements of the lemmas and their proofs are left the appendix.
We will first present a lemma that relates an expectation of a random variable to the weighted integral of one minus its cumulative density function (CDF) evaluated at different points, which we will refer to as cumulative densities.
Lemma 1. Let X be a non-negative random variable and be a continuous function on [0, oo). If क' is integrable all closed intervals in [0, oo).
/ ('())Pr(X 2 t)dt
E =(())
This lemma is useful because the left-hand side (LHS) is easy to approximate with Monte Carlo estimates of expectations, and the right-hand side (RHS) is weighted integral of one minus cumula- tive densities, which are intractable to compute general. It enables us to control the weighting of different cumulative densities by choosing the function Φ.
Recall that our goal is to maximize the likelihood at each training example without actually computing the likelihood. We can leverage Lemma for this purpose, by choosing the non-negative random variable X appropriately. We choose X to be the distance between training example and a generated sample To With this choice. Lemma gives us way to relate weighted integral of the average likelihoods within differently sized neighbourhoods around the training example x; (RHS) to the expectation of a function of the distance d(x;, (z(2))) (LHS).
4

---

Under review as a conference paper at ICLR 2023
Moreover, we'd like restrict the average likelihoods integrate neighbourhoods of certain sizes rather than from O to cc. Specifically, we'd like to integrate from or to т, where 0 is the radius of the largest neighbourhood and 0 S 8 < 1 is a tightening threshold. To this end, we can choose the weighting function ( to be when 87 S & r and 0 otherwise. One choice of such (-)-) that satisfies this condition and its associated $ (1) are:
to only those within
ÔT 1 << dt dT ST
<<
(1) =
(1)
St
1>T
0 127
Using this choice of (-).). we obtain the following lemma for particular training example X,.
Lemma 2. Under the choice of & (-) above and its associated (-)(),
Eg E Dr (dxxx, TT(z, = f (r(d(x,,Tozz))) < t)dt.
This lemma shows that, for one training example X,, the expectation on the LHS reduces to minus the integral of the average likelihoods within balls whose radii lie between år, and T5. Applying Lemma 2 to all training examples ,I. obtain the following lemma that reveals what the overall objective in Eqn optimizes.
Lemma 3. Under the choice of +.) above and its associated (f(()),
[rr.. (0)
(T(((x;;T(z,,) < l)dl.
mar
j=1
BT
Lemma 3 shows that [rr,), (0) implicitly computes the average likelihood that the generative model assigns the neighbourhood of each data point. The choice of r; controls the radius. Since we would like to maximize probability in the immediate neighbourhood of each data point, we would like т; to be small.
So should we choose an arbitrarily small value for T?? Recall that by definition of 4 2,,()). (.), if d(x,, T' (2,)) Ts. 4,, (d(x,, (z))))) T.. So. for a very small T, may well be the case that d(x,, To (zz))
Ti Vj, which would make the Monte Carlo estimate of L(r., (0), i.c.,
DE T. Σド D., (o(z))) zero. Since this is constant, the gradient w.r.t. the parameters zero, which makes gradient- based learning impossible. This would happen whenever minze[m) d(x;, To (z(2))). )), and the smallest that can be chosen minje(m] d(x;; To(2))) (which is treated as a constant rather than a function of 0).
With this choice of Tr, assuming that there is unique jr such that d(x,, Tz(z))) minje[m] d(x;, To(z;)) (which happens almost surely), the objective can be simplified to:
(0) E
(U,I)
max( min d(x,, To(;)) jE(m)
(3)
Yum
If we minimize the objective in Eqn. 3 objective. The solution to the Adaptive IMLE objective can be expressed as:
we get novel objective known as the Adaptive IMLE
arg max (0) = arg min E......... 2 (0,T)
Σ
max( min d(x, To (z 8T:) jE(m
(4)
8
5

---

Under review as a conference paper at ICLR 2023
It turns out that the vanilla IMLE objective can be recovered special case, by choosing ò 0 and W1 = W2
Wp-
n
arg max (..): arg min Ee Pm ((0.1)
max( min d (x,, To(z,)) (o(z;)),00 jelm]
Wi
1
arg min P
((0..
min (x, To(2)))
Wi jE(m
arg min N(0.))
min d (xi, To(z)))
jE[m]
4.1.1 CURRICULUM LEARNING
Recall that our goal is to maximize the likelihood of the immediate neighbourhood around each data point, and the size of this neighbourhood is controlled by т; Therefore, want to make т₃ small. In order to make т; small without impeding learning, we need to make (000..... [Ti] =
Eg 2.....mmm~1110 (0), [minjemm] d(x;; To (2,)) small. To this end, we can either increase the number of samples, or train To so that the samples it produces are close to the data point x₃. The former computationally expensive, and
will devise method to achieve the latter.
We propose a curriculum learning strategy, which solves a sequence optimization problems with different 74's 's, such that T, s get smaller for optimization problems later in the sequence. The earlier optimization problems After each optimization problem solved to convergence, start solving the next optimization problem with 0 initialized to the solution found previously.
the sequence help train T3 to produce samples close to the data points.
This will make 's smaller and smaller. If they eventually converge to zero, then it turns out that would have equivalently maximized the sum of likelihoods po(x;) of the training examples under the probability distribution induced the generative model, shown in the lemma below.
Lemma 4. Suppose Po is continuous at all data points X1,...y Xus under the choice of wi = Joh (x.) == 18r, J..O.) dxdt, where Br (x) {yld(y, x) < r is an open ball of radius r centred x,
lim (0)
Σ (x(x:)
(---0+);
This lemma shows the theoretical guarantees of Adaptive IMLE hold under more general conditions that those required by vanilla IMLE.
4.2 ADAPTIVE IMLE
The key difference from the objective in Eqn. 4 to the original IMLE formulation in Eqn. is the individualized neighbourhood radius t; around each data point x, This change the objective crucial, as it allows the model to adapt to the varying difficulty in learning different training examples, hence the algorithm name, Adaptive IMLE.
As mentioned in Sect. 4.1.1 need to gradually decrease in order to make the learning feasible. This could achieved by decreasing the tightening threshold 8r,. To this end, algorithm optimizes the model's 's parameter until the distance between the generated sample and the target data T(x;,T6 T6 (2,)) decreases to 8r, Once the threshold is reached, the algorithm decreases the threshold by multiplying it by 8 as 0 < 8 < 1. This updated threshold then serves as the target for learning x,. Intuitively, the tightening coefficient determines the amount of progress required for the selected sample towards each training example.
Now let's turn our attention to the optimization problem we solve in each stage of the curriculum. Consider the following unweighted variant the objective in Eqn. without the in. factor:
arg min Eg Fm ((..) max min d(x, To(zz)), 6T:) jE(m]
(5)

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Adaptive IMLE Procedure
Require:
The set of inputs {xi)._- tightening coefficient å E [0, I: Initialize the parameters 0 of the generator T.
Draw latent codes Z Z1 ... Ze from N( I)
3: o(i) f- arg [in d(x,, (2)))) vi E [n]
4: T, < d (x1, To (2=(a)) vi [n]
Initialize the threshold each data point
5: for == to K do
6: Pick a random batch S S [n]] Vo (E ies d (xi, To (zo(a)) ))) /|S|
7: 0 4 0-700
Draw latent codes f- 21, Zm from (0,I) for E 5 do
9:
10:
if d (xx, To (Zo(a)) ) 2 6r; then
D Only update af1) when getting into the threshold Tightening the threshold
11:
4 T.
12:
(i1) + arg [m]] d(x,, (2)))
13:
end
14:
end for
15:
end for
16:
return 0
If we were run stochastic gradient descent (SGD) without replacement on the weighted objective (Eqn. F and the unweighted objective (Eqn. 5 and consider the updates made by each, we will find that updates induced by the weighted objective are just scalar multiples of those induced by the unweighted objective, since and therefore w;; is fixed during each stage of the curriculum. So, optimizing the weighted objective is equivalent to optimizing the unweighted objective, with a different step size chosen for each update. We consider optimizing the unweighted objective with constant step size that is smaller than these per-iteration step sizes. Then, if SGD on the weighted objective converges to solution where minje[m] d(x;; To (zy)) falls below 87, (which is global minimum), SGD on the unweighted objective with such a step size will do the same, because the latter is just choosing more conservative step sizes than implied by the weighted objective.
Now, if we putting everything together, we obtain the Adaptive IMLE algorithm. The details are shown in Algorithm
EXPERIMENTS
Baselines We compare our method to recent few-shot unconditional image synthesis methods that operate in the same setting we consider, namely without needing to pre-train on auxiliary datasets.
Two of such recent methods are FasIGAN (Liu et al. (022) and MixDL (Kong et al. 2022).
Training Details Our network architecture is modified from Child (202)). where keep the
decoder architecture and replace the encoder with a fully-connected mapping network inspired by
Karras al. (2019). We choose an input latent dimension of 1024 and tightening coefficient 0.9.
We train our model for 500k iterations with mini-batch size of 4 using the Adam optimizer (Kingma
& Ba 2015 with learning rate of 2 x 10-6 on a single NVIDIA V100 GPU.
Datasets We evaluate method and the baselines wide range of natural image datasets 256 x 256 resolution, which includes Animal-Face Dog and Cat (Si & Zhu 2012), Obama, Panda, and Grumpy-cat (Zhao al. 2020b) and Flick--FaceHQ (FFHQ) subset (Karras et al. 2019). All datasets contain 100 images except for Dog and Cat which contain 389 and 160 images respectively. The FFHQ subset consists of 100 FFHQ images with similar backgrounds, in order highlight diversity in the generation of foregrounds.
We use the Fréchet Inception Distance (FID) HHeusel et al. 2017) to measure the perceptual quality of the generated images, where randomly generate 5000 images and compute FID between the generated samples and real images in each dataset. To evaluate the mode modelling accuracy (precision) and coverage (recall), we the precision metric of Kfnkkääännnnn et al. (2019) to measure the former, and use the recall metric of Kynkäänniemi et al. (2019) and
Evaluation Metrics
'The weighted objective will always converge such solution since we choose 8 he close
7

---

Under review as a conference paper at ICLR 2023
MixDL
FastGAN
Adaptive IMLE (Ours)
Cat
Dog
FFHQ subset
Panda
Grumpy Cat
Obama
Figure 2: Qualitative comparison of images generated by our method and those generated by the
baselines, FastGAN (Liu et al. 202T) and MixDL (Kong et al.) 2022).. shown, our samples are of higher quality and have greater diversity. On the other hand, the samples generated by the baselines show more limited diversity, which is validated by the recall results in Table. which suggest that the baselines exhibit mode collapse.
Grumpy Cat
Ohama
Panda
Cat
Dog
FFHQ subset
FID,,
LPIPS)
FIDJ
LPIPS),
FID4
LPIPS),
FID,
LPIPS)
FID,,
LPIPS))
FIDJ
IPPPSS
FastGAN [Liu et al. |021
26.6
0.357
41.1
0.370
10.0
0.339
35.1
0.467
50.7
0.430
54.2
0.357
MixDL Kong al. 2022
24.5
0.296
43.4
0.276
10.6
0.264
26.5
0.305
81.2
0.274
62.3
0.221
Adaptive IMLE (Owrs)
23.4
0.058
30.7
0.036
8.5
0.039
29.9
0.074
50.6
0.072
43.9
0.014
Table 1: We compute FID (Hessel et al.
between the real data and 5000 randomly generated samples in all cases. LPIPS above represents LPIPS backtracking score (Liu et al., (021). For this metric, each model is trained on 90% of the dataset. The resulting model is used to backtrack in the latent space and reconstruct the remaining 10%. Lower LPIPS backtracking score shows better mode coverage of the training data.
2017)
LPIPS backtracking score (Liu et al. 2021) to measure the latter. For LPIPS backtracking, use 90% the full dataset for training and evaluate the metric using the remaining 10% of the dataset.
5.1 QUANTITATIVE RESULTS
We compare the FID and LPIPS backtracking scores across all methods in Tab. As shown, our method outperforms the baselines in terms of both metrics on all datasets except for Cat, where our FID is the second best and LPIPS backtracking score is the best. We compare the mode accuracy and coverage in Tab. 2 As shown, method achieves better precision than the baselines and significantly outperforms the baselines in terms of recall. These results show that our method could produce high-quality images while obtaining better mode coverage compared to the baselines.
8

---

Under review as a conference paper at ICLR 2023
Grumpy Cat
Obama
Panda
Cat
Dog
FFHQ subset
Prec.
Rec.
Prec.f
Rec.1
Prec.
Rec.t
Prec.
Rec.
Prec.1 f
Rec.
Prec.
Rec.
FastGAN (Lu
2027
0.909
0.13
0.918
0.09
0.957
0.16
0.971
0.08
0.961
0.19
0.913
0.13
MixDL Kong et at. 2022
0.931
0.35
0.910
0.47
0.933
0.30
0.910
0.50
0.857
0.15
0.770
0.30
Adaptive IMLE (Ours)
0.987
0.73
0.967
0.85
0.990
0.87
0.972
0.71
0.972
0.48
0.998
0.76
Table 2: Precision and recall (Kynkäänniemi et al. 2019) is computed across 1000 randomly generated samples and the target dataset. Our method performs better for both precision and recall in all cases. Higher precision shows better fitting to the target dataset and higher recall corresponds to better mode coverage.
5.2 QUALITATIVE RESULTS
We show the qualitative comparison of our method to the baselines in Fig. 2 As shown, method generates higher quality samples which better preserve the semantic structures compared to the baselines, such the eyes in Cat, the facial structure in Dog and the mouth and hair in the FFHQ subset. addition, our method generates more diverse results while the baselines suffer from mode collapse and generate similar samples, such as in Panda, Grumpy Cat and Obama. Additional samples from our model can be found in the appendix.
We show the final reconstruction of the target image found using LPIPS backtracking (Liu al. 202I) on the models trained with different methods in Fig. 3 As shown, our method is the only one where the reconstruction is structurally similar to the target image, demonstrating that our model successfully covers the mode that the target image belongs to.
We also compare our method the baselines on the quality of interpolations between two samples in the latent space. As shown in Fig. E our method interpolates more smoothly and naturally than the baselines, thereby indicating that our model less overfitted the training examples.
Target image
MixDL
FastGAN
Ours
Target image
MixDL
FastGAN
Ours
Figure 3: Visualizations of the reconstructions of an unseen target image from LPIPS backtracking. While the reconstructions of MixDL and FastGAN are structurally dissimilar from the target images, the reconstructions of our method are structurally similar to the target images.
Obama
FFHQ subset
MixDL
FastGAN
Ours
Figure 4: Latent space interpolation results. Our results show smooth and meaningful transitions and higher quality images generated from intermediate points along the interpolation line in the latent space. Start and end of interpolations are nearest neighbours of the same data examples among 200 samples generated by each method.
5.3 ABLATION STUDY
We compare the FID, precision, and recall between the proposed method, Adaptive IMLE, and vanilla IMLE on the more challenging datasets, Obama and FFHQ subset. As shown in Tab. 3 Adaptive IMLE significantly improves upon vanilla IMLE in terms of FID and recall while achieving similar precision, validating the effectiveness of the proposed method under the few-shot setting.

---

Under review as a conference paper at ICLR 2023
Obama
FFHQ subset
FIDJ Pree. Rec.t
FID
Prec.1
Rec.1
Vanilla IMLE Li & Malik 2018)37,4 37,4 0.973
0.61
54.1 0.999
0.51
Adaptive IMLE (Our)
30.7 0.967 0.85
43.9 0.998
0.76
Table 3: Adaptive IMLE significantly improves perceptual quality (FID) and recall compared to vanilla IMLE, while maintaining similarly high levels of precision.
CONCLUSION
We developed method for the challenging few-shot image synthesis setting that does not depend on pre-training on auxiliary datasets. We presented more generalized formulation of IMLE and proved that the theoretical guarantees of this generalized formulation hold under weaker conditions. We further derived novel algorithm based on this formulation which can adapt to different training examples of varying difficulty. We showed that our method significantly outperforms existing baselines in terms of mode modelling accuracy and coverage on six few-shot benchmark datasets.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. ArXiv, abs/1809. 1096, 2019.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. ArXiv, abs/2011.10650 2021.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. ArXiv, abs/2105.05233 2021.
Laurent Dinh, Jascha Narain 100500803,.
and Samy Bengio. Density estimation using real nvp.
ArXiv, abs/1605, 08803, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 06-11 Aug 2017. URL https:://proceeeiings ml www..preess/vv
Finn17a.htr
Ian Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farkee Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In NIPS, 2017.
Jonathan Ho, Ajay Jain, and Abbeel. Denoising diffusion probabilistic models. ArXiv, abs/2006. 1139, 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1401-4410 2019.
Tero Karras, Samuli I Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. 2020 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8107-8116, 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. In NeurIPS, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980,, 2015.
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. ArXiv, 1070000039, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1r1iv:1312.611 14,611114000 2013.
arXiv preprint
Ivan Kobyzev, Simon Prince, and Marcus A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43: 3964-3979, 2021.
Chaerin Kong, Jeesoo Kim, Donghoon Han, and Nojun Kwak. Smoothing the generative latent space with mixup-based distance learning. In European Conference on Computer Vision, 2022.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, 32, 2019.
Ke Li and Jitendra Malik. Implicit maximum likelihood estimation, arXiv preprint arXiv:1809.09053 2018.
11

---

Under review as a conference paper at ICLR 2023
Ke Li Shichong Peng*. Tianhao Zhang*, and Jitendra Malik. Multimodal image synthesis with conditional implicit maximum likelihood estimation. International Journal of Computer Vision. doi: 10.1007/s1 10. 10.100//s111 263 020 01325-y
May 2020. ISSN 1573-1405.
URL https::/
Yijun Li, Richard Zhang, Jingwan Lu, and Eli Shechtman. Few-shot image generation with clastic weight consolidation. ArXiv, abs/2012.02780, 2020.
Bingchen Liu, Yizhe Zhu, Kunpeng Song, and Ahmed Elgammal. Towards faster and stabilized GAN training for Sigh-fidelity shot image synthesis. CoRR, abs/2101.04775, 2021. URL
fttts://ariie
org attpaa/s//1.1111 04775
Ming- Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz. Few-shot unsupervised image-to-iman translation. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 10550-10559, 2019.
Sangwoo Mo, Minsu Cho, and Jinwoo Shin. Freeze discriminator: A simple baseline for fine-tuning gans. ArXiv, abs/2002. 10964, 2020.
Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. Few-shot image generation via cross-domain correspondence. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10738-10747 2021.
N. O' Mahony, Sean Campbell, Anderson Carvalho, Krpalkova, Gustavo Velasco Hernan- dez, Suman Harapanahalli, D. Riordan, and J. Walsh. One-shot learning for custom identifi- cation tasks; a review. Procedia Manufacturing. 38: 186-193, 2019. ISSN 2351-9789. doi: http:///rrr........oo//omoooom/p 20200.100.............. 0255. URL ht ptttps:/// www.. c enc.eeeneddrrre com/ 63 29th International Conference on Flexi- ble Automation and Intelligent Manufacturing [FAM 2019), June 24-28, 2019, Limerick, Ireland, Beyond Industry 4.0: Industrial Advances, Engineering Education and Intelligent Manufacturing.
Shichong Peng, Seyed Alireza Moazenipourasil, and Ke Li. Chimle: Conditional hierarchical imle. In NewrlPS, 2022.
Ali Razavi, Aäron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. ArXiv, abs/1906.00kl 000446, 2019.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelenn++: Improving the pixel- cnn with discretized logistic mixture likelihood and other modifications. ArXiv, 1701.05517, 2017.
Zhangzhang Si and Song-Chin Chun Zhu. Learning hybrid image templates (hit) by information projection. IEEE Transactions Pattern Analysis and Machine Intelligence, 34:1 134--33667,, 2012.
Abhishek Sinha, Jiaming Song, Chenlin Meng, and Stefano Ermon. D2c: Diffusion-denoising models for few-shot conditional generation. ArXiv, abs/2106.068 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL htttss:// ngs. neurips wwwrips.cc/ppeeeeepeeeeeee htt.r/isssssccccpppe////0111111 httpccc/pppee//21///11. cb8ca677 7461 F20FF2122222
P pdd
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. ArXiv, /e///907.05660 2019.
Yang Song, Jascha Narain Sohl-Dickstein Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ArXiv, abs/2011. 13456, 2021.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. ArXiv. hbs/2007.03893 2020.
12

---

Under review as a conference paper at ICLR 2023
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, and Alex Graves. Conditional image generation with pixelenn decoders. ArXiv, abs/1606.05328, 2016a.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks.
ArXiv, 0ess/6601.06753 2016b.
Ting-Chun Wang, Ming- Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video synthesis. ArXiv, 1tt/1910.11211. 2019.
Yaxing Wang, Abel Gonzalez-Garcia David Berga, Luis Herranz, Fahad Shahbaz Khan, and Joost van de Weijer. Minegan: Effective knowledge transfer from gans to target domains with few images. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 2020.
Miaoyun Zhao, Yulai Cong, and Lawrence Carin. On leveraging pretrained gans for generation with limited data. In ICML, 2020a.
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. ArXiv, abs/2006. 10738, 2020b.
13

---

Under review as a conference paper at ICLR 2023
A PROOFS
Lemma 1. Let X be non-negative random variable and & be continuous function on (0) If e' integrable on all closed intervals in [0, oo),
[[ = +(0) /0 ('(1)Pr(X 2 t)dt
Proof.
2(0) / ('())Pr(X 2 =)d==((0)) /// J.a d'()p(a)dadi
=00)=
x ('(t)p(z)
(x21/20)
(00++
{(()p(z)
(122,120}
L' d'(pl(z)dtdr
(0))
(J, ((())dt) p(a)dr
(0) +
(0)) + (0(x) +(0)) p(x)do (2nd FTC)
4(0) + ((v)p(o)dc L0/ ((0)(())dr
((z)p(r)dz 20) Lo/ p(z)dz
200
+00)) + E [+ -
= [x()]
Lemma 2. Under the choice of x.()) above and its associated Φ (.),
N110.) o (d(x, To n. f (o(d(), t)dt.
Proof. By definition, (0) = -8T.
Eg E Tm A(o.1) [4,, (d(x;, Ta(z)))] = (0) + / (')(()))((( 2 t)dt (Lemma 1)
=8T;
(T(x)x; 2 1)dt
8T, +
(1 (1- Pr(((x(xx,oo t)) dt
J
1 Pr(d(xx, To(z)) < l)t
=ōT; + (Ti - (T.))
T
Pr(d(x,, To(z)) < t)dt
br.
14

---

Under review as a conference paper at ICLR 2023
Lemma 3. Under the choice of (-) above and its associated (-),,).
(0))
< t)dt.
mww ==
Proof.
(0) = E S (((01))
(d To(z,)))
W/
1=1
j=1
T:
8m N(0.1) (0, (d(x;, To(2))))
17
==1
T
Ts
To(zz) < t)dt
(Lemma 2)
m
Pr(d(x;, Ta(z;)) s)dl
11
Equation 3.
Proof.
((T))(0) = E. Em
12
m
Tq
1
max min d(x;, To (2 (2;)), ÖT.) Je(m.
W%
m
m
i=1
E (0011)
73
max min d(x, To (z 85:)
W2
m
m
/Elm
i=1
- N(01))
T3
max( min d(x,, To To(z))), jE/m]
7um
W
Equation 4.
Proof.
1
n
1
arg max ((T))
arg max E. B No(11)
max( min d(x;, (z (z;)) ÖT.) jE(m
7177
Wi
arg max No(.1)
Ti
max( min d(x,, To To(z))), 5E(m]
Wl
n
arg max E. (..00.
max min d(x; To(z;)),ō7á JE(m]
W/s
W1
n
arg max E
(0.1))
max min d(x;, To (z,))) ÖTC)
Wi;
JE(m)
arg min E.
max min d(x,, To (z, je/m
15

---

Under review as a conference paper at ICLR 2023
Lemma 4. Suppose po is continuous all data points X1, x....., Xw. under the choice of Jor, (x)( == f8r, (x())) dxdt, where (x = {yld(y, x) < r is an open ball of radius
r centred at x,
lim (T)) (7=))+)
(0) = Σ (x(
Proof.
(T.).
Prid(x;,Tola < t)dt (Lemma
71
mu
11
m
po dx)xx
71
mt;;
ÔT: (x()).)
po (x dxdt
Tu
Wi
GT; Bx(x.)
Jon (oo))
(oc.) dxxxt Sat, S..os dxdt
mm
m
Jor, B.....) Po Po(x))xxxt J J dxet Jonn Spice Po ddxdt Str, Sp(())) dxdt
lim
(0)
lim T-
Σ
(ne+0t),
nm
n
m
lim
nm
To +
JB,, (xx.) po(x) 6 SBar, (xx) (e(xx)ss
lim
(L' (L'Hôpital and 2nd FTC)
Sp,, (xx) dx ô "Sro. (xx)) dx
To 7--00
SB., (x.) Po (x(( -51 Ba-, (x.) (x))dx
lim
TMT
7--
[B,((x)) 1 51 Bar, (xx.) (x xxx
m
50 81 (1(())) (r)) [[x]a(xexx) Po (x(x)xxdr
dxdr
lim
1o - 01 (rcon) (r)) [ (x()(xxx))==) dxdr
nm
--0
[x pe xx
lim
(L'Hôpital and 2nd FTC)
TiTTI
[xlddxxx.)=x.) dx
Po (xx))
(Continuity of pa)
f=1 1=1
pe (x,)
Note that under common metrics like lp distances, wo can be found in closed form, i.e.,
vol(B, (2r) ((1+1//) and so w, = f&r vol(B((x,))tt = foy (2t) 1(1+1/p)" T(1+d/p)
T(1+d/p)
r(111/p) (2(1-8)7)441
2(d+1) T(1+d/p) where I'(-) denotes the gamma function.
16

---

Under review as a conference paper at ICLR 2023
B PSEUDO CODE FOR IMLE
Algorithm 2 Implicit maximum likelihood estimation (IMLE) procedure
Require: The set of inputs fx; )
I: Initialize the parameters of the generator To
2: for === 11 K do
3: Pick a random batch 5 S [n]]
4: Draw latent codes Z +- , Zm from (0,
5: (()) <- arg minj d(x,, (zj)) Vi
6:
for / = 1 to L do
7:
Pick random mini batch $ C S
8:
0 0 V (Eies d (xi, To (zo()))) ))) /
9:
end for end for return
11:
17


Reviews:
[{"review_id": "rrgpHBLPeF4", "reviewer": "Reviewer_owTn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new objective for the problem of few-shot image synthesis based on extending implicit maximum likelihood estimation (IMLE). Specifically, they derive a data-adaptive IMLE objective for training the implicit generative model. Experiment results on six few-shot image synthesis datasets, namely Grumpy Cat, Obama, Cat, Panda, Dog, and the subset of FFHQ, demonstrate the proposed method outperforms the previous few-shot image synthesis methods.", "summary_of_the_review": "This paper tackles the few-shot image synthesis problem by extending the IMLE objective to be “image-adaptive”. However, I feel the overall clarity could be improved. For the evaluation side, I am not fully convinced due to the missing baseline and the concerns about ablation studies. In these respects, I am currently on the negative side. ", "strength_and_weaknesses": "### Strengths\n* The problem setup of few-shot image synthesis is practical and interesting.\n* The paper performs extensive experiments across diverse datasets.\n* The paper shows solid experimental results, both quantitatively and qualitatively. \n\n### Weaknesses\n* While the problem definition (few-shot image synthesis) is interesting in general, I am not convinced about the detailed problem setup in not using auxiliary datasets. An explanation highlighting the importance of such a setup or comparing the results with fine-tuning-based methods would be welcome.\n* The paper is hard to follow. Considering that the paper proposes the generalized version of IMLE, the clarity of the paper would be much better if it explicitly mentioned the limitations of IMLE (at least in brief) in the theoretical aspect. However, the paper only mentions that “restrictive conditions need to be satisfied for the theoretical guarantees of IMLE to hold, such as requiring a uniform optimal likelihood for all data points”. Why, what, and how are such conditions restrictive, while the proposed method is not? \n*  I think the ablation study (Table 3) should be plotted (instead of Table) to compare IMLE and adaptive IMLE. Few-shot synthesis problems are generally vulnerable to the overfitting of the generator. As the proposed adaptive IMLE objective includes additional regularizations (e.g., $\\min$ with $\\delta \\tau_i >0$)than IMLE, hence more difficult to optimize, I wonder whether the performance gain is simply from alleviating the overfitting or from indeed “better” objective design. At least, I strongly recommend authors put more details on how the ablation study is performed (e.g., “best” performance during the training is reported); but still, showing plots with respect to number of iterations would be much more welcome.\n* While the proposed method shows superior performance among the reported baselines, the current manuscript misses some recent baselines (even outperforming adaptive IMLE), e.g., [1]. At least, a discussion with this baseline would strengthen the manuscript more.\n\n[1] FakeCLR: Exploring Contrastive Learning for Solving Latent Discontinuity in Data-Efficient GANs, ECCV 2022. (arXiv: Jul 2022)\n\n### Minor comment\n* The equation number is not stated in the sentence between Eqn. 3 and Eqn. 4.\n\n### Questions\n* Why does the precision of IMLE be better than the proposed Adaptive IMLE?\n* The proposed method can be applied to any few-shot synthesis problem; do the authors try to apply the method in different domains?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bOkVkqyWCv", "reviewer": "Reviewer_xM5B", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper enables training GANs with a few amount of data from scratch. It is built on top of an existing method IMLE which is improved to adaptive IMLE to avoid model collapse or model overfitting. The more generalized form of IMLE is presented which covers the original formulation as a special case. Extensive experiments on different datasets are conducted to show that the proposed method could generate higher quality and more diverse results.", "summary_of_the_review": "A new perspective to tackle the few shot generation problem. Good work but I worried a bit about its performance over other non-facial dataset and eventually how useful it will be compared to transfer learning setting. ", "strength_and_weaknesses": "+ Strict prove on theoretical guarantees of the generalized formulation hold under weaker condition\n+ A systematic level up of vanilla IMLE algorithm\n+ Obvious improvement of results under few-shot setting\n\nA good work about making GAN training less data-hungry and potentially using less compute resources, which is applicable to personalized generation with a small library of data. Please find my minor concerns below:\n\nTo understand how the number of training data is affecting the performance, it is better to conduct an experiment by ablating this hyper-parameter and show a curve about how FID will increase with less training examples.\n\nJust curious what is the smallest number of images that the proposed method could work? 100? 10? 1? The previous transferring learning based methods, though requiring pretraining on an external data, are more aligned with human learning process of learning new knowledge quickly after learning old knowledge and show even 1-shot setting is possible. The proposed method belongs to learning from scratch with a few amount of data. Is it limited in for example just learning 100 faces of one person (obama) instead of learning 100 faces of different persons? Essentially, GAN should be a data engine to benefit more downstream tasks. If the generated is still less diverse, it may not be quite useful as compared to transfer learning approaches.\n\nIt is suggested to include some results on non-facial data as face has been heavily explored, in order to see how robust the algorithm is cross different type of datasets.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W_Gsojqgvj", "reviewer": "Reviewer_jG1N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a generalized formulation of IMLE, Adaptive IMLE, for few-shot image synthesis. As the generator trained in GAN suffers from the mode collapse issue, i.e., GAN may overfit to a subset of training samples. Then, IMLE is proposed to avoid this issue by optimizing training images to have some similar generated images. However, as strict restrictions are applied on IMLE and may impede training efficiency in the few-shot setting, this paper proposed Adaptive IMLE by adjusting the radius w.r.t. each training data. Detailed theoretical analysis is provided, and the experimental study demonstrates its effectiveness. However, the discussion is very limited.", "summary_of_the_review": "The discussion is very limited. Please see my comments above.", "strength_and_weaknesses": "Strength:\n1.\tFormulating a generalized form of techniques is essential for further development, and the few-shot learning setting is specifically studied.\n2.\tClear paper writing about background introduction and method derivation & development.\n3.\tConsistent performance gain and promising results. Visualization are also provided.\n\nWeakness:\n1.\tMy biggest concern is regarding ignoring w_i in the Eq.5. The explanation seems incomplete (the unweighted objective, since tau_i and there fore w_i is fixed). Please double-check and clearly explain why w_i can be ignored.\n2.\tWhy Prec drops slightly while FID and Rec. are improved largely? It is necessary to explain the reason. Acturally, Prec are very high and I am not sure whether it Is a good metric.\n3.\tTraining details should be provided. Do you need additional large-scale data for pre-training? Also, do you need to find the nearest generated samples for all training data within each batch? If yes, the memory usage is pretty large and I wonder whether your approach can address this issue.\n4.\tThe comparison between IMLE and adaptative IMLE is only on Obama and FFHQ while more experimental comparisons are performed in Table 1-2. Can you explain the reason? From my perspective, the comparison with vanilla IMLE is much more critical.\n5.\tFor the optimization of IMLE, one weakness is that the neighbor search is centered w.r.t. each training data, then, it is possible that a very limited number of generated samples are selected in one batch. Will that be risky, and do your methods can mitigate this issue also?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Aysp3MOZEX", "reviewer": "Reviewer_uhYC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work designs a more general IMLE objective by introducing an adaptive neighbourhood radii. The theoretical derivation and the proposed curriculum learning strategy are beautiful and impressive. Some experiments are also conducted to verify the effectiveness of the proposed method.", "summary_of_the_review": "This work devises a theoretically sound objective to generalize the IMLE, and achieves stable and better results. Even though the writing of this paper should be improved, it is still a beautiful work.", "strength_and_weaknesses": "Strength:\n1. The idea of introducing an adaptive radii for IMLE is interesting, which increase the degree of the vanilla IMLE.\n2. Leveraging lemma 1 to construct the adaptive is very clever. It provides a very useful tool to depict the distribution of the distance between the training example and a generated sample.\n3. The visual results tend to prove that this adaptive objective could produce more diverse example.\n\nWeaknesses:\n1. The written of this paper is not very clear. Beginning from lemma 3, the notation $w_i$ is frequently appeared. But there isn't any explanation about this notation, and why does introduce $w_i$?\n2. I don't understand the definition of $L_{\\{\\tau_i\\}_i}$ in the first line of the proof for lemma 3 in appendix. If possible, please provide more details.\n3. The number of sample $m$ should play an important role in this work. Even this work design the curriculum strategy to avoid increasing $m$. It would be better to offer some results under different values of $m$.\n4. Comparing with the vanilla IMLE, why does the proposed objective achieve better results? This paper does not clearly claim this point.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "9EIQA3wc3I", "reviewer": "Reviewer_Me9N", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the problem of few-shot image synthesis, i.e., learning to generate samples from a distribution given only few examples. A previous approach, dubbed Implicit Maximum Likelihood Estimation (IMLE), showed that one perform few-shot synthesis by minimizing a loss that penalizes the distance between generated samples and true samples. This loss was shown to exhibit nice properties, such as convergence to the true likelihood, but operated under strong assumptions (e.g., all training images have the same likelihood). This work builds upon IMLE by proposing an adaptive version that is able to weigh examples from the training set based on their difficulty and theoretically show that their loss is a generalization of IMLE operating under weaker assumptions. Results on few-shot image synthesis benchmarks show that learning with the Adaptive IMLE can improve generation.", "summary_of_the_review": "Overall, I think that the paper presents a nice extension of previous work in order to improve few-shot image synthesis. The experimental results seem promising, but there is some lack of polish in the presentation of the material. I am leaning more on the positive-side based on the results.\n\n", "strength_and_weaknesses": "\n**Strengths:**\n- The new adaptive loss function is principled in that it can be shown in a limiting sense to converge to a maximum likelihood-based objective.\n- The empirical results show quantitative and qualitative improvements in terms of generation quality and diversity of the images. To the reviewer's knowledge, comparisons against strong few-shot image synthesis baselines are considered for comparison.\n\n**Weaknesses:**\n- Some notions of the theory would be impractical to implement, such as choosing the correct weights $w_i$ in the optimization for certain distance metrics, which requires computing the volume of a neighborhood around a given datapoint.\n- There are some issues in the clarity/derivations that would be good to improve upon.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject