Instance-Conditioned GAN
Authors: Arantxa Casanova
Marlene Careil
Jakob Verbeek
Michal Drozdzal
Adriana Romero
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: generative adversarial networks
GAN
conditional GAN
image generation
scene generation
kernel density estimation
class conditional GAN
unbalanced classes
long-tail distribution

Abstract: Generative Adversarial Networks (GANs) can generate near photo realistic images in narrow domains such as human faces. Yet, modeling complex distributions of datasets such as ImageNet and COCO-Stuff remains challenging in unconditional settings. In this paper, we take inspiration from kernel density estimation techniques and introduce a non-parametric approach to modeling distributions of complex datasets. We partition the data manifold into a mixture of overlapping neighborhoods described by a datapoint and its nearest neighbors, and introduce a model, called instance-conditioned GAN (IC-GAN), which learns the distribution around each datapoint. Experimental results on ImageNet and COCO-Stuff show that IC-GAN significantly improves over unconditional models and unsupervised data partitioning baselines. Moreover, we show that IC-GAN can effortlessly transfer to datasets not seen during training by simply changing the conditioning instances, and still generate realistic images. Finally, we extend IC-GAN to the class-conditional case and show semantically controllable generation and competitive quantitative results on ImageNet; while improving over BigGAN on ImageNet-LT. Code and trained models to reproduce the reported results are available at https://github.com/facebookresearch/ic_gan.


Instance-Conditioned GAN
Arantxa Casanova Facebook Al Research École Polytechnique de Montréal Mila, Quebec Al Institute
Marlène Careil Facebook Research
Jakob Verbeek Facebook Al Research
Télécom Paris
Michal Drożdžal* Facebook Al Research
Adriana Romero-Soriano"
Facebook AI Research
McGill University
Abstract
Generative Adversarial Networks (GANs) can generate near photo realistic images in narrow domains such as human faces. Yet, modeling complex distributions datasets such as ImageNet and COCO-Stuff remains challenging in unconditional settings. this paper, we take inspiration from kernel density estimation techniques and introduce non-parametric approach to modeling distributions complex datasets. We partition the data manifold into a mixture of overlapping neighborhoods described by datapoint and its nearest neighbors, and introduce model, called instance-coniee conditioned GAN (IC-GAN), which learns the distribution around each datapoint. Experimental results ImageNet and COCO-Stut show that IC-GAN significantly improves over unconditional models and unsupervised data partitioning baselines. Moreover, we show that IC-GAN effortlessly transfer to datasets not seen during training by simply changing the conditioning instances, and still generate realistic images. Finally, extend IC-GAN the class-conditional case and show semantically controllable generation and competitive quantitative results on ImageNet: while improving over BigGAN on ImageNet-LT. Code and trained models to reproduce the reported results are available https: wwt.bb/..c./acccbbooeeeeeer/e//e
Introduction
Generative Adversarial Networks (GANs) 18] have shown impressive results in unconditional Despite their success, GANs present optimization difficulties and can suffer from mode collapse, resulting in the generator not being able to obtain a good distribution coverage, and often producing poor quality and/or low diversity generated samples. Although many approaches attempt to mitigate this problem e.g. 20 such as the one in ImageNet
image generation 27 29
35
KS
complex data distributions
45
remain a challenge for unconditional GANs
3 Bo
Class-
ease the task of learning the data distribution by conditioning on class labels, effectively partitioning the data. Although they provide higher quality samples than their unconditional counterparts, they require labelled data, which may be unavailable or costly to obtain.
conditional GANs (5] 39 40 56
Several recent approaches explore the of unsupervised data partitioning to improve GANs [2] 23
14] 17
While these methods are promising and yield visually appealing samples, their quality still far from those obtained with class-conditional GANs. These methods make of relatively coarse and non-overlapping data partitions, which oftentimes contain data points from different types of objects or scenes. This diversity of data points may result in manifold with low
Equual contribution.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Ausly
camel
(a) IC-GAN samples
(b) Class-conditional IC-GAN samples
husky
husky
(c) IC-GAN transfer samples
(d) Class-conditional IC-GAN transfer samples
Figure 1: Samples from unlabeled (a) and class-conditional (b) IC-GAN trained the 256 x 256 ImageNet dataset. For each subfigure, the first column represents instances used to condition the model and the next three columns depict model samples. For class-conditional generation in (b) include samples conditioned on the same image but different labels. We highlight the generalization capacities of IC-GAN by applying the ImageNet-trained model to instances from other datasets in unlabeled (c) and class-conditional (d) scenarios. Panels (c) and (d) display samples conditioned on instances from the COCO-Stufi Cityscapes, MetFaces, and PACS datasets (from top bottom).
density regions, which degrades the quality of the generated samples Α Using finer partitions, however, tends to deteriorate results 33] 36 42 because the clusters may contain too few data points for the generator and discriminator to properly model their data distribution.
In this work, we introduce new approach, called instance-conditionned GAN (IC-GAN), which extends the GAN framework to model mixture of local data densities. More precisely, IC-GAN learns model the distribution the neighborhood of data point, also referred to as instance, by providing representation of the instance as additional input to both the generator and diseriminator, and by using the neighbors of the instance as real samples for the discriminator. By choosing a sufficiently large neighborhood around the conditioning instance, avoid the pitfall

---

of excessively partitioning the data into small clusters. Given the overlapping nature of these clusters, increasing the number of partitions does not come at the expense of having less samples in each of them. Moreover, unlike when conditioning on discrete cluster indices, conditioning on instance representations naturally leads the generator to produce similar samples for similar instances. Interestingly, once trained, our IC-GAN can be used to effortlessly transfer to other datasets not seen during training by simply swapping-out the conditioning instances at inference time.
IC-GAN bears similarities with kernel density estimation (KDE), a non-parametric density estimator in the form of a mixture of parametrized kernels modeling the density around each training data point - see e.g. [4] Similar to KDE, IC-GAN can be seen as mixture density estimator, where each component is obtained by conditioning on training instance. Unlike KDE, however, we do not model the data likelihood explicitly, but take an adversarial approach in which we model the local density implicitly with a neural network that takes as input the conditioning instance as well as noise vector. Therefore, the kernel in IC-GAN is no longer independent on the data point on which we condition, and instead of a kernel bandwidth parameter, we control the smoothness by choosing the neighborhood size of an instance from which we sample the real samples to be fed to the discriminator.
We validate our approach on two image generation tasks: (I) unlabeled image generation where there is no class information available, and (2) class-conditional image generation. For the unlabeled scenario, we report results on the ImageNet and COCO- -tuff datasets. We show that IC-GAN outperforms previous approaches in unlabeled image generation on both datasets. Additionally, we perform a series of transfer experiments and demonstrate that an IC-GAN trained on ImageNet achieves better generation quality and diversity when testing on COCO-Stuff than the same model trained on COCO-Stuff. with controllable semantics by adapting both class and instance-, while achieving competitive sample quality and diversity on the ImageNet dataset. Finally, we test IC-GAN in ImageNet- LT, long-tail class distribution ablated version of ImageNet, highlighting the benefits of non- parametric density estimation in datasets with unbalanced classes. Figure | shows IC-GAN unlabeled ImageNet generations (a), IC-GAN class-conditional ImageNet generations (b), and IC-GAN transfer generations both in the unlabeled (c) and controllable class conditoonii (d) setting.
the class-conditional setting, we show that IC-GAN can generate images
2 Instance-conditionee GAN
The key idea of IC-GAN is to model the distribution of a complex dataset by leveraging fine-grained overlapping clusters in the data manifold, where each cluster is described by datapoint x; referred to as instance and its nearest neighbors set A; in a feature space. Our objective is to model the underlying data distribution p(x) mixture of conditional distributions p(x around each of M instance feature vectors h, in the dataset, such that p(x) R 37 p((hh))
More precisely, given an unlabeled dataset D = ) with M data samples x; and an embedding function parametrized by o, we start by extracting instance features h, fo (x:) Vx, E D, where (-) is learned in an unsupervised or self-supervised manner. We then define the set A; of nearest neighbors for each data sample using the cosine similarity is common in nearest neighbor over the features h,. Figure 2a depicts a sample x; and its nearest neighbors.
classifiers, egg. 53
We are interested in implicitly modelling the conditional distributions p(x h:) with generator Geo (z, h.), implemented by a deep neural network with parameters OG. The generator transforms samples from a unit Gaussian prior -N(0, I) into samples from the conditional distribution p(x(b,), where h; is the feature vector of an instance x; sampled from the training data. In IC-GAN, adopt an adversarial approach to train the generator Geg. Therefore, our generator is jointly trained with a discriminator Dop (x, h;) that discerns between real neighbors and generated neighbors of h, as shown in Figure 2b Note that for each h; real neighbors are sampled uniformly from A,,
Both G and D engage in two player min-max game where they try to find the Nash equilibrium for the following equation:
min max (e( D(x,, fo(x:)) + G
D
(1)
Ex Ex.roptx) - (x)))
3

---

fa(o,)
h
X
Z
Giz, h,)
X
D(x h)
fake
Shared weightx
p p(x|h)
X
D h,)
real
Real samples (neighbors)
(a) Neighborhood A; of instance h;
(b) Schematic illustration of the IC-GAN workflow
Figure 2: Overview of IC-GAN. (a) The goal of the generator is to generate realistic images similar to the neighbors of h, defined in the embedding space using cosine similarity. Five out of seven neighhors shown in the figure. Note that images in the same neighborhood may belong different classes (depicted as different shapes). (b) Conditioned on instance features h, and noise 2, the generator produces synthetic sample Xg- Generated samples and real samples (neighbors of h,) fed to the discriminator, which is conditioned on the same h;.
Note that when training IC-GAN we use all available training datapoints to condition the model. At inference time, as non-parametric density estimation methods such as KDE, the generator IC- GAN also requires instance features, which may come from the training distribution or a different one.
Extension to class-conditional generation. We extend IC-GAN for class-conditional generation by additionally conditioning the generator and discriminator on a class label y. More precisely, given a labeled dataset D, y:)), with M data sample pairs (x,, ya) and embedding function fo. we extract instance features h; = fo (x:) Vx; E D, where fo(.) is learned in an unsupervised, self-supervised, supervised manner. We then define the set A; of k: nearest neighbors for each data sample using the cosine similarity over the features h, where neighbors may be from different classes. This results in neightorhoods, where the number of neighbors belonging to the same class as the instance h; is often smaller than k. During training, real neighbors x, and their respective labels y, are sampled uniformly from A, for each h, In the dlass-coniitional case, model p(x[h,, y;) with generator Goa (z, Y,) trained jointly with a discriminator Dop (x, h;)
3 Experimental evaluation
We describe experimental setup in Section 3.1 followed by results presented in the unlabeled
setting in Section 3.2 dataset transfer in Section 33 and class-conditional generation in Section 3.4
We analyze the impact of the number of stored instances and neighborhood size in Section 3.5
3.1 Experimental setup
Datasets. We evaluate our model in the unlabeled scenario on ImageNet 45 and COCO-Stuff 6 The ImageNet dataset contains 1.2M and 50k images for training and evaluation, respectively. COCO- Stuff is very diverse and complex dataset which contains multi-object images and has been widely used for complex scene generation. We use the train and evaluation splits of [S] and the (un)seen subsets of the evaluation images with only class combinations that have (not) been seen during training. These splits contain 76k, 2k, 675 and 1.3k images, respectively. For the class-conditional image generation, we use ImageNet well ImageNet-LT [3]] The latter is long-tail variant of ImageNet that contains a subset of 15kk samples, where the 1,000 classes have between 5 and 1 1,280 samples each. Moreover, abilities of IC-GAN: Cityscapes [1O.. MetFaces
some samples of four additional datasets to highlight the transfer
2 PACS [1I and Sketches [15].
Evaluation protocol. We report Fréchet Inception Distance (FID) 22 Inception Score (IS) 47 and LPIPS 57 LPIPS computes the distance between the Alex activations of two images generated with two different latent vectors and same conditioning. On ImageNet, we follow [.] and

---

compute FID over 50k generated images and the 50k real validation samples are used as reference. On COCO-Stuff and ImageNet-LT, we compute the FID for each of the splits using all images in the split as reference, and sample the same number images. Additionally, in ImageNet-ITT we stratify the FID by grouping classes based on the number of train samples: more than 100 (many-shot FID), between 20 and 100 (med-shot FID), and less than 20 (few-shot FID). For the reference set, we split the validation images along these three groups of classes, and generate a matching number of samples per group. In order to compute all above-mentionee metries, IC-GAN requires instance features for sampling. Unless stated otherwise, we store 1,000 training set instances by applying k-means clustering to the training set and selecting the features of the data point that is the closest to each one of the centroids. All quantitative metrics for IC-GAN are reported over five random seeds for the input noise when sampling from the model.
Network architectures and hyperparameters. As feature extractor fa. we use ResNet50 21 trained in self-supervised way with SwAV (7 for the unlabeled scenario; for the class-conditional IC-GAN, we use ResNet50 trained for the classification task on either ImageNet or ImageNet- For ImageNet experiments, we use BigGAN [ as baseline architecture, given its superior image quality and ubiquitous use in conditional image generation. For IC-GAN, we replace the class embedding layers in the generator by fully connected layer that takes the instance features as input and reduces its dimensionality from 2,048 to 512; the same approach is followed to adapt the discriminator. For COCO- StyleGAN2 architecture
LT 26
we additionally include the state-of- fftte-of-the-eee the-art art unconditional 24 as it has shown good generation quality and diversity in the lower data We follow its conssconniiional version 2 to extend it to IC-GAN by replacing the input class embedding by the instance features. Unless stated otherwise, we set the size of the neighborhoods to k == 50 for ImageNet and k = for both COCO-Stuff and ImageNet-LTT. See the supplementary material for details on the architecture and optimization hyperparameters.
regime
28 2y
3.2 Unlabeled setting
ImageNet. We start by comparing IC-GAN against previous work in Table M Note that unconditional BigGAN baseline is trained by set- ting all labels in the training set to zero, following 36 42]. IC-GAN surpasses all previous approaches at both 64 x 64 and 128 x 128 res- olutions in both FID and IS scores, At 256 x 256 resolution, IC-GAN outperforms the concurrent uncon- ditional diffusion- based model of [2]: the only other result we are aware of in this setting. Additional results in terms of precision and re- call can be found in Table g in the supplementary material.
Table 1: Results for ImageNet in unlabeled setting. For fair comparison with 442 64 x 64 resolution, trained an unconditional BigGAN model and report the non-official FID and IS scores computed with Pytorch rather than TensorFlow indicated with * increased parameters to match IC-GAN capacity. DA: 50% horizontal flips in (d) real and fake samples, and (i) conditioning instances. chx: Channel multiplier that affects network width as in BigGAN.
Method
Res.
FFID
IIS
Self-sup. GAN
12
64
19.2*
16.5*
Uncond. BigGAN IC-GAN IC-GAN + DA (d,i)
64
16.9* 1 0.0
14.6 + 0.1
64
10.4 1
21.9* + 0.1
64
9.2 =
23.5 + 0.1
MGAN 2X
128
58.9
13.2
PacGAN2 32
128
$7.5
13.5
Logo-GAN-AE
46
128
50.9
14.4
As shown in Figure Ia IC-GAN generates high quality images pre- serving most of the appearance of the conditioning instance. Note that generated images are not mere train- ing memorizations; as shown the supplementary material, generated images differ substantially from the nearest training samples.
Self-cond. GAN
3
128
41.7
14.9
Uncond. BigGAN
36
128
25.3
20.4
SS-cluster GAN
36
128
22.0
23.5
PGMGAN 2 IC-GAN
128
21.7
23.3
128
13.2 + 0.0
45.5 + 0,2
IC-GAN + DA (d.i)
128
11.7 + 0.0
48.7 + 0.1
ADM 12
256
32,5
37.6
IC-GAN (ch x 64)
256
17.0 +: 0.2
53.0 + 0.4
IC-GAN (ch x 64) DA (d.i)
256
17.4 + 0.1
53.5 + 0.5
IC-GAN (ch x (69 + DA (d)
256
15.6 0.1
59.0 + 0.4
COCO-Stuff. We proceed with the evaluation of IC-GAN on COCO-
Stuff in Table g We also compare to state-of-the- art complex scene generation pipelines which rely on labeled bounding box annotations conditioning LostGANy2 [9] and OC-GAN (50). Both of
5

---

(a) (b) IC-GAN (StyleGAN2)
(c) IC-GAN (BigGAN)
(d) IC-GAN (BigGAN, transf.)
Figure 3: Qualitative comparison for scene generation on 256 x 256 COCO-Stuff.
these approaches use tailored architectures for complex scene generation, which have at least twice the number of parameters of IC-GAN. Our IC-GAN matches improves upon the unconditional version of the same backbone architecture in terms of FID all cases, except for training FID with the StyleGAN2 backbone at 256 x 256 resolution. Overall, the StyleGAN2 backbone is superior to BigGAN on this dataset, and StyleGAN2-based IC-GAN achieves the state-of-the-art FID scores, when compared the bounding- conditioned LostGANv2 and OC-GAN. IC-GAN exhibits notably higher LPIPS than LostGANv2 and OC-GAN, which could be explained by the fact that the latter only leverage one real sample per input conditioning during training; whereas IC-GAN uses multiple real neighboring samples per each instance, naturally favouring diversity in the generated images, As shown in figures 3b and 3d. IC-GAN generates high quality diverse images given the input instance. A qualitative comparison between LostGANv2, OC-GAN and IC-GAN can be found in Section E of the supplementary material.
Table 2: Quantitative results on COCO-Stuff. IC-GAN trained on ImageNet indicated as "transf".
Some non-zere standard deviations are reported as 0.0 because rounding.
LFID
LPIPS
128 128
# prms.
train
eval
eval seen
eval unseen
eval
LostGANv2 49
41 M
12.8 + 0.1
40.7 + 0.3
80.0 0.4
55.2 - 0.5
0.45 0.1
OC-GAN 500
170 M
45.1
+ 0.3
85.8 + 0.5
60.1 = 0.2
0.13 0.1
Unconditional (BigGAN)
18 M
46.9 0.5
103.8 + 0.8
60.9 0.7
0.68 0.1
IC-GAN (BigGAN)
22 M
16.8 0.1
44.9 0.5
81.5 1.3
60.5 0.5
0.67 0.1
IC-GAN (BigGAN, transf.)
77 M
8.5
3.26
77.0
48.9 0.2
0.69
Unconditional (StyleGAN2)
23
8.8
37.8
92.1 1.0
53.2 0.5
0.68 0.1.
IC-GAN (StyleGAN2)
24
8.9 0.0
36.2 0.2
74.3 0.8
50.8 0.3
0.67 0.1
256
LostGANy2 49
46 M
18.0
47.6 0.4
88.5 + 0.4
62.0 - 0.6
0.56 +
OC-GAN 50
190 M
$7.0
98.7 - 1.2
71.4 0.5
0.21 0.1
Unconditional (BigGAN)
21 M
0.1
81.6
135.1 1.6
95.8 1.1
0.77 0.1
IC-GAN (BigGAN)
26 M
24.6
53.1
88.5
69.1
0.73 0.1
IC-GAN (BigGAN, transf.)
90
1.1
4.3
1.2
$5.6 0.6
0.76
Unconditional (StyleGAN2)
23
7.1 0.0
0.4
98.1
59.9
0.76
IC-GAN (StyleGAN2)
25
9.6 + 0.0
41.4
76.7 0.6
57.5 0.5
0.74
3.3 Off-the-shelf transfer to other datasets
In our first transfer experiment, we train IC-GAN with a BigGAN architecture on ImageNet, and use generate images from COCO-Stuf" instances test time. Quantitative results are reported as "IC-GAN (transf.)" in Table In this setup. no COCO-Stu Stuff images are used to train the model, thus, all splits contain unseen objects combinations. Perhaps surprisingly. IC-GAN trained on ImageNet outperforms the same model trained on COCO- Stuff for all splits: 8.5 vs. 16.8 train FID at 128 resolution. This raises the question of how close ImageNet and COCO-Stuff data distributions are. We compute the FID between real data train split of the two datasets at 128 x 128 resolution and obtain score of 37.2. Hence, the remarkable transfer capabilities of IC-GAN are not explained by dataset similarity and may be attributed to the effectiveness of the ImageNet pre- me-ttraine feature

---

extractor and generator. When we replace the conditioning instances from COCO- StoffStuff with those of ImageNet, we obtain train FID score of 43.5, underlining the important distribution shift that can be implemented by changing the conditioning instances.
Interestingly, the transferred IC-GAN also outperforms LostGANv2 and OC-GAN which condition on labeled bounding box annotations. Transferring the model from ImageNet boosts diversity w.r.t. the model trained on COCO-Stuff (see LPIPS in Table 2b. which may be in part due to the larger 50 used for ImageNet training, compared to k= 5 when training on COCO-Stuff. Qualitative results of COCO-Stuff generations from the ImageNet pre- trained IC-GAN can be found in Figure Ic (top row) and Figure bd These generations suggest that IC-GAN is able to effectively leverage the large scale training on ImageNet improve the quality and diversity of the COCO-Stuff scene generation, which contains significantly less data to train.
We further explore how the ImageNet trained IC-GAN transfers conditioning on other datasets using Cityscapes, MetFaces, and PACS in Figure Tc Generated images still preserve the semantics and style of the images for all datasets, although degrading their quality when compared to samples in Figure La as the instances in these datasets -in particular MetFaces and PACS- are very different from the ImageNet ones. See Section F in the supplementary material for more discussion, additional evaluations, and more qualitative examples of dataset transfer.
3.4 Class-conditional setting
ImageNet. In Table B we
Table 3: Class-conditional results on ImageNet. *: Trained using open source code. DA: 50% horizontal flips in (d) real and fake samples, and (i) conditioning instances. chx: Channel multiplier that affects network width. i; numbers from the original paper, as training diverged with the BigGAN opensourced code.
show that the dlass-cnniiiioned IC-GAN outperforms BigGAN terms of both FID and IS across all resolutions except the FID at 128 x 128 resolution, It is worth mentioning that, unlike BigGAN, IC-GAN can control the seman- ties of the generated images by either fixing the instance features and swapping the class condition- ing, by fixing the class condi- tioning and swapping the instance features; see Figure lb As shown in the figure, generated images preserve semantics of both the class label and the instance, gen- crating different dog breeds on similar backgrounds, or generat- ing camels in the snow, an unseen scenario in ImageNet to the best of our knowledge. Moreover, in Figure ld we show the transfer capabilities of our class- conds-oondiiional IC-GAN trained on ImageNet and conditioned on instances from other datasets, generating camels in the grass, zebras in the city, and husky dogs with the style of MetFaces and PACS instances. These controllable conditionings enable the generation of images that are not present or very rare the ImageNet dataset, egg. camels surrounded by snow or zebras in the city. Additional qualitative transfer results which either fix the class label and swap the instance features, or vice-versa, can be found in Section F of the supplementary material.
Res.
.FID
1S
BigGAN
64
12.3 1 0.0
27.0 = 0.2
BigGAN 5
+ +DA (d)
64
10.2 0.1
30.1 = 0.1
IC-GAN
64
8.5 0.0
39.7 0.2
IC-GAN DA(d, i)
6.7 0.0
45.9 0.3
BigGAN S
128
9.4
98.7 = 1.1
BigGAN
DA(d)
128
8.0 0.0
107.2 = 0.9
IC-GAN
128
10.6
100.1 100.1000 0.5
IC-GAN + DA(d, i)
128
9.5 0.1
108.6 = 0.7
BigGAN 5 (ch x 64)
256
8.0 0.1
139.1 = 0.3
BigGAN S (ch x 64) + DA(d)
256
8.3
125.0 1.1
IC-GAN (ch x 64)
256
8.1
143.7 = 1.1
IC-GAN (ch x 64) + DA(d,
256
7.0
152.6 =
BigGANt [5] (ch x 96)
256
8.1
144.2
IC-GAN (ch 96) + DA(d)
256
8.2
173.8
0.9
ImageNet-LT, Due the class imbalance in ImageNet-LT, selecting a subset instances with either k-means or uniform sampling can easily result in ignoring rare classes, and penalizing their generation. Therefore, for this dataset we use all available 1 115k training instances sample from the model and compute the metrics. In Table E we compare to BigGAN, showing that IC-GAN is better in terms of FID and IS for modeling this long-tailed distribution. Note that the improvement is noticeable for each of the three groups of classes with different number of samples, see many/med/few column. In Section G of the supplementary material we present experiments when using class balascialnninng to train BigGAN, showing that it does not improve quality nor diversity of generated samples. We
7

---

Table Class-conditie results on ImageNet-LT. *: Trained using open source code.
Res.
[train FID
1rain IS
Lval FID
many/med/few .val FID
1val IS
BigGAN
S
64
27.6 = 0.1
18.1 = 0.2
28.1 = 0.1
28.8/32.8 / 48.4 = 0,2
16.0 = 0,1
IC-GAN
64
23.2 = 0.1
19.5 = 0.1
23.4 0.1
23.8/ /4.77 = 0.1
42.7
17.6 = 0.1
BigGAN IC-GAN
5
128
31.4 + 0.1
30.6 + 0.1
35.4 + 0.1
34.0/43.5 /64.4 + 0.2
24.9 + 0.2
128
23.4 - 0.1
39.6 = 0.2
24.9 = 0.1
24.3/314/
0.3
32.5 = 0.1
BigGAN *
256
27.8 + 0.0
58.2 = 0.2
3.44
28.1/40.9 66.66 0.3
44.7 = 0.2
IC-GAN
256
21.7 + 0.1
66.5 + 0.3
2.14
20.6 60.0 = 0.2
51.7 + 0.1
hypothesize that oversampling some classes may result in overfitting for the discriminator, leading to low quality image generations.
3.5 Selection of stored instances and neighborhood size
In this section, we empirically justify the k-means procedure to select the instances to sample from the model, consider the effect of the number of instances used to sample from the model, as well as the effect of the size & of the neighborhoods A, used during training. The impact of different choices for the instance embedding function (x) is evaluated in the supplementary material.
Selecting instances to sample from the model. In Figure 3 (left), compare two instance selection methods in terms of FID: uniform sampling (Random) and k-means (Clustered), where we select the closest instance to each cluster centroid, using k 50 neighbors during training (solid and dotted green lines). Random selection consistently outperformed by k-means; selecting only 1,000 instances with k- m-means results in better FID than randomly selecting 5,000 instances. Moreover, storing more than 1,000 instances selected with k-means does not result in noticeable improvements in FID. Additionally, to the k-means cluster centers, obtaining 41.8 + 0.2 FID, which is considerably higher than the 10.4 + 0.1 FID we obtain with IC-GAN (k 50) when using the same 1,000 cluster centers. This supports the idea that IC-GAN is generating data points that go beyond the stored instances, better recovering the data distribution.
computed FID metrics for the 1,000 ground truth images that closest
We consider precision (P) and recall (R) 30 (using an Inception 51 as feature extractor and sampling 10,000 generated and real images) disentangle the factors driving the improvement in FID, namely image quality and diversity (coverage) - see Figure (right). We see that augmenting the number of stored instances results in slightly worse precision (image quality) but notably better recall (coverage). Intuitively, this suggests that by increasing the number of stored instances, we better recover the data density at the expense of slightly degraded image quality in lower density regions of the manifold - see e.g.
Required storage 8MH
Required storage
0.4MB
0.8MB
4MB
16MB
40MB
0.4MB
0.8MB
AMB
ВMB
16MB
10MB
30-
k- 5 (Clustered)
0.6
k= 20 (Clusterec)
25
k= 50 (Clusterec)
50 (Random)
(Clustered)
20
100 (Clustered)
20 (Clustered)
500 (Clustered)
0.4
k= 50 (Clustered)
15
km 190 (Clusterec))
5=0 (Clusterec)
10-
02-
50
100
500
1000
2000
5000
100
500
1000
2000
5000
Number of stored instances
Number of stored instances
Figure 4: Impact on the number of stored instances used to evaluate IC-GAN and the size of the neighborhood k. Experiments performed on the 64 x 64 unlabeled ImageNet dataset.

---

Neighborhood size. In Figure 4 (both panels) we analyze the interplay between the neighborhood size and the number of instances used recover the data distribution. For small numbers of stored instances, we observe that larger the neighborhoods lead to better (lower) FID scores (left-hand side of left panel). For recall, we also observe improvements for large neighborhoods when storing few instances (left-hand side of right panel), suggesting that larger neighborhoods are more effective in recovering the data distribution from few instances. This trend is reverted for large numbers of stored instances, where smaller values of k are more effective. This supports the idea that the neighborhood size acts as a bandwidth parameter - similar to KDE -, that controls the smoothness of the implicitly learnt conditional distributions around instances. For example, k 500 leads to smoother conditional distributions than & = 5, and as a result requires fewer stored instances to recover the data distribution. Moreover, expected, we notice that the value of k does not significantly affect precision (right panel). Overall, k = 50 offers a good compromise, exhibiting top performance across all metrics when using least 500 stored instances. We visualize the smoothness effect by means of qualitative comparison across samples from different neighhorhood sizes in Section K of the supplementary material. Using (very) small neighborhoods (e.g. of k = 5), results in lower diversity in the generated images.
4 Related work
Data partitioning for GANs. Previous works have attempted to improve the image gen- eration quality and diversity of GANs by partitioning the data manifold through clustering techniques 22 19
33
36
46). or by leveraging mixture models in their design [4 17 23
In particular, 36
apply k-means on representations from pre-trained feature extractor to cluster the data, and then use cluster indices to condition the generator network. Then, 19 3
introduce alternating two-stage approach where the first stage applies k-means to the discriminator feature space and the second stage trains a GAN conditioned on the cluster indices. Similarly, [2] proposes to train a clustering network, which outputs pseudolabels, in cooperation with the generator. Further, [2] trains a feature extractor with self-supervised pre-training tasks, and creates a k-nearest neighbor graph in the learned representation space to cluster connected points into the same sub-manifold. In this case, a different generator is then trained for each identified sub-manifold. By contrast, IC-GAN uses fine-grained overlapping data neighborhoods in tandem with conditioning on rich feature embeddings (instances) to learn a localized distribution around each data point.
Mitigating mode collapse in GANs. Works which attempt mitigate mode collapse may also bear some similarities to ours. In 32]. the discriminator takes into consideration multiple random samples from the same class to output decision. In [35] mixed batch of generated and real samples is fed to the discriminator with the goal of predicting the ratio of real samples in the batch. Other works use
mixture of generators 17 2X and encourage each generator focus on generating samples from different mode. Similarly, in [4), the discriminator is pushed to form clusters in its representation space, where each cluster is represented by Gaussian kernel. In turn, the generator tends to learn to generate samples covering all clusters, hence mitigating mode collapse. By contrast, we focus on discriminating between real and generated neighbors of an instance conditioning, by using a single generator network trained following the GAN formulation.
Conditioning on feature vectors. Very recent work 37 uses image self-superviser feature repre- sentations to condition a generative model whose objective is to produce a good input reconstruction; this requires storing the features of all training samples. In contrast, our objective is to learn localized distribution (as captured by nearest neighboring images) around each conditioning instance, and we only need to save very small subset of the dataset features to approximately recover the training distribution.
Kernel density estimation and adversarial training. Connections between adversarial training and nonparametric density estimation have been made in prior work M. However, to the best of our knowledge, no prior work models the dataset density in nonparametric fashion with a localized distribution around each data point with a single conditional generation network.
Complex scene generation. Existing methods for complex scene generation, where natural looking scenes contain multiple objects, most often aim at controllability and rely on detailed condition- ings such as a scene graphs K 25 bounding box layouts [48 58]. semantic segmentation masks
1 4
551 or more recently, freehand sketches [6.. All these methods leverage
intricate pipelines to generate complex scenes and require labeled datasets. By contrast, our approach

---

relies on instance conditionings which control the global semantics of the generation process, and does not require any dataset labels. It is worth noting that complex scene generation is often characterized by unbalanced, strongly long tailed datasets. Long-tail class distributions negatively affect class- conditional GANs, as they struggle to generate visually appealing samples for classes in the tail [8. However, to the best of our knowledge, no other previous work tackles this problem for GANs.
5 Discussion
Contributions. We presented instance-conditionee GAN (IC-GAN), which models dataset dis- tributions in a pon-maaaettiic way by conditioning both generator and discriminator on instance features. We validated our approach the unlabeled setting, showing consistent improvements over baselines on ImageNet and COCO-Stuff. Moreover, we showed through transfer experiments, where condition the ImageNNettrained model on instances of other datasets, the ability of IC-GAN to produce compelling samples from different data distributions. Finally, we validated IC-GAN in the clndiinoniiio setting, obtaining competitive results on ImageNet and surpassing the Big- GAN baseline on the challenging ImageNettLT: and showed compelling controllable generations by swapping the class-conditioning given fixed instance or the instance given fixed conditioning.
Limitations. IC-GAN showed excellent image quality for labeled (class-conditional) and unlabeled image generation. However, any machine learning tool, it has some limitations. First, as kernel density estimator approaches, IC-GAN requires storing training instances to use the model. Experi- mentally, we noticed that for complex datasets, such as ImageNet, using 1,000 instances is enough to approximately cover the dataset distribution. Second, the instance feature vectors used to condition the model are obtained with a pre-trained feature extractor (self-supervised in the unlabeled case) and depend on it. We speculate that this limitation might be mitigated if the feature extractor and the generator are trained jointly, and leave it as future work. Third, although, we highlighted excellent transfer potential of our approach to unseen datasets, we observed that, in the case of transfer to datasets that are very different from ImageNet, the quality of generated images degrades.
Broader impacts. IC-GAN brings with it several benefits such as excellent image quality in labeled conditiondii and unlabeled image generation tasks, and the transfer potential unseen datasets, enabling the use of our model on a variety of datasets without the need of fine-tume see-tunnng or re-training. Moreover, in the case of class-conditional image generation, IC-GAN enables controllable generation of content by adapting either the style- - by changing the instance - or the semantics - by altering the class- Thus, we expect that our model can positively affect the workflow for creative content generators. That being said, with improving image quality in generative modeling, there is some potential for misuse. A common example are deepfakes, where a generative model is used to manipulate images or videos well enough that humans cannot distinguish real from fake, with the intent misinform. We believe, however, that open research on generative image models also contributes to better understand such synthetic content, and to detect it where it is undesirable. Recently, the community has also started to undertake explicit efforts towards detecting manipulated content by organizing challenges such as the Deepfake Detection Challenge [13
References
[1] M Ehsan Abbasnejad, Qinfeng Shi, Anton van den Hengel, and Lingqiao Liu. A generative adversarial density estimator. In CVPR, 2019.
[2] Mohammadreza Armandpour, Ali Sadeghian, Chunyuan Li, and Mingyuan Zhou, Partition-guided GANs. arXiv preprint, (arXiv:2104.008 2021.
[3] Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene generation. In CVPR, 2019.
[4] C. Bishop. Pattern recognition and machine learning. Spinger- 2006.
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In ICLR. URL httpltttt//pperreiiiicoo hwtp://eeeeeeeeeeeeooeeee mott/forumiid-i hotf//orur/ddd-11e00099)
[6] Holger Caesar. Jasper Uijlings, and Vittorio Ferrari. COCO-Stun Stuff: Thing and stuff classes in context. In CVPR, 2018
10

---

[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
[8] Arantxa Casanova, Michal Drozdzal, and Adriana -omero-Soriano. Generating unseen complex scenes: are we there yet? arXiv preprint, (arX1w:2012.04022), 2020.
[9] Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks. In CVPR, 2017.
[10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson. Uwe Franke, Stefan Roth, and Bertt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, 2016.
[11] Terrance DeVries, Michal Drozdzal, and Graham W. Taylor. Instance selection for GANs, In NeurlPS, 2020.
[12] Prafulla Dhariwal and Alex Nichol. Diffusion models beat GANs on image synthesis. arXiv preprint, (arXiv:2105.022)) 2021.
[13] Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton-Ferrer, The deepfake detection challenge (DFDC) preview dataset. CoRR, abs/1910.08854, 2019. URL Jttpp//////rxxi org/ 1bs/ 08854
[14] Hamid Eghbal-zadeh, Werner Zellinger, and Gerhard Widmer. Mixture density generative adversarial networks. In CVPR, 2019.
[15] Mathias Fitz, James Hays, and Mare Alexa, How do humans sketch objects? ACM Trans. Graph. (Proc. SIGGRAPH). :1(4):44:1-44:: 2012.
[16] Chengying Gao, Qi Liu, Qi Xu, Limin Wang, Jianzhuang Liu, and Changqing Zou. SketchyCOCO: image generation from freehand scene sketches. In CVPR, 2020.
[17] Arnab Ghosh, Viveka Kulharia, Vinay P. Namboodiri, Philip H.S. Tom,, and Puneet K. Dokania. Multi-agent diverse generative adversarial networks. CVPR, 2018.
[18] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In NewrIPS, 2014.
[19] Guillermo L Grinblat, Lucas C Uzal, and Pablo M Granitto. Class-splitting generative adversarial networks. arXiv preprint, (arXiv:1 (tt.11.99997))) 2017.
[20] Ishaan Gulrajani. Faruk Ahmed, Martin Arjovsky. Vincent Dumoulin, and Aaron Courville. Improved training of Wasserstein GANs. In NeurlPS, 2017.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[22] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In NewrlPS, 2017.
[23] Quan Hoang. Tu Dinh Nguyen, Trung Lc, and Dinh Phung. MGAN: Training generative adversarial nets with multiple generators. In ICLR, 2018.
[24] Jeff' Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.087:33 087344 2017.
[25] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In CVPR, 2018.
[26] Bingyi Kang. Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng. and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In /CLR, 2020.
[27] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
[28] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In NewrIPS, 2020.
[29] Tero Karras, Samuli Laine, Miia Aittala, Janne Hellsten. Jaakko Lehtinen, and Timo Aila, Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.

---

[30] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models, arXiv preprint arXiv:1904.0699,, 2019.
[31] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In CVPR, 2017.
[32] Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. PacGAN: The power of two samples in generative adversarial networks, In NeurlPS, 2018.
[33] Steven Liu, Tongzhou Wang, David Bau, Jun- Yan Zhu, and Antonio Toralba. Diverse image generation via self-conditioned GANs. In CVPR, 2020.
[34] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. Large-scale long-tailed recognition in open world. In CVPR, 2019.
[35] T. Lucas, C. Tallee, Y. Ollivier, and J. Verbeek. Mixed batches and symmetric discriminators for GAN training. In ICML. 2018.
[36] Mario Lučié, Michael Tschannen, Marvin Ritter, Xiaohua Zhai, Olivier Bachem, and Sylvain Gelly. High-fidelity image generation with fewer labels. In ICML,, 2019.
[37] Puncet Mangla, Nupur Kumari, Mayank Singh, Balaji Krishnamurthy, and Vineeth N. Balasubramanian. Data instance prior (DISP) in generative adversarial networks. WACV, 2022.
[38] Luke Metz, Ben Poole, David Pfau, and Jascha Sohl-Dickstein. Unrolled generative adversarial networks. arXiv preprint, (arXiv:1611.021e) 2016.
[39] Mehdi Mirza and Simon Osindero.
Conditional generative adversarial nets. arXiv preprint,
(44 2014.
[40] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint, (arXiv:1802.05957) 2018.
[41] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In ICML, 2020.
[42] Mehdi Noroozi. Self-labeled conditional GANs. arXiv preprint, (arXiv:2012.021122..2 2020.
[43] Taesung Park, Ming- Yu Liu, Ting-Ch Chun Wang, and Jun Yan Zhu. Semantic image synthesis with spatially- adaptive normalization. In CVPR, 2019.
[44] Xiaojuan Qi, Qifeng Chen, Jiaya Jia, and Vladlen Koltun. Semi- parametric image synthesis. In CVPR, 2018.
[45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bersstcin, Alexander C. Berg, and Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3)21 2015. doi:
10.1007/s 1263-015-0816-y.
[46] Alexander Sage, Eirikur Agustsson, Radu Timofte, and Lue Van Gool. Logo synthesis synthesss and manipulation with clustered generative adversarial networks. In CVPR, 2018
[47] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training GANs. In NeurlPS, 2016.
[48] Wei Sun and Tianfu Wu. Image synthesis from reconfigurable layout and style. In CVPR, 2019.
[49] Wei Sun and Tianfu Wu. Learning layout and style reconfigurable GANs for controllable image synthesis. arXiv preprint, (arXiv:2003. 15511), 2020.
[50] Tristan Sylvain, Pengchuan Zhang. Yoshua Bengio, R Devon Hjelm, and Shikhar Sharma. Object-centric image generation from layouts. arXiv preprint, SatXi:20003.04449 2020.
[51] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
[52] Hao Tang, Dan Xu, Yan Yan, Philip HS Torr, and Nicu Sebe. Local class-specific and global image-level generative adversarial networks for semantic-guu -uiieed scene generation. In CVPR, 2020.
12

---

[53] Hugo Touvron, Alexandre Sablayrolles. Mathhij Douze. Matthieu Cord, and Hervé Jégou, Grafit: Learning fine-grained image representations with coarse labels. arXiv preprint, (arXiv:201 1. 12982). 2020.
[54] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for shot learning. In NeurlPS, 2016. URL https://axxi org/ ab1///6b.166...... .40800
[55] Ting-Chun Wang, Ming-Yo Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional GANs. In CVPR, 2018.
[56] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, 2019.
[57] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
[58] Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In CVPR, 2019.
[59] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Diffcrentiable augmentation for data- efficient GAN training. In NeurlPS, 2020.
13


Reviews:
[{"review_id": "CuxUUcov0Up", "reviewer": "Reviewer_8u3S", "summary": "The authors propose a new way of training GANs, which they call instance-conditioned GANs. This method is similar to conditional GANs, but instead of using a label, they use a feature vector extracted from some feature function (by using ResNet50 in the experiments).This bypasses the need for labels for conditional GANs. However, there is another difference. The features are assumed to be locally similar so when evaluating the loss over batches, real images are chosen to be neighbors of the given instance. This creates a kernel-density estimation type of model. Experimentally, these Instance-condition GANs perform very well, beating self-supervised, and unconditional GANs. Moreover, the conditional version of instance-conditioned GANs outperform conditional GANs as well in some smaller-resolution settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThis paper is very original and interesting.\n\nQuality:\n\nThe paper is a bit confusing (see clarity below) but the research does seem to be well developed. The experiments are very impressive as well!\n\nClarity:\n\nThe overall clarity of this paper is only okay. Some of the terms are a bit confusing. For example, on line 71, the authors say “overlapping partitions”. What are overlapping partitions of a manifold? Typically, given a set A, we partition it. Meaning that there is a singular partition of disjoint sets whose union is the whole set A. I can only conclude that overlapping partitions means a collection of sets whose union is A but they are not disjoint. \n\nIt is also confusing why they call a datapoint an instance. In fact this is defined on line 72 but used without definition on line 38, causing more confusion.\n\nLine 79 says that Figure 2a shows 7 neighbors which is confusing because there are only 4 images, although there are 7 darker shapes. And the different shapes in the figure make it also confusing because one would think that neighboring shapes be similar. \n\nSignificance:\n\nThis is a very interesting idea and significant because it can be used on completely unsupervised data.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations:\n\nThe embedding depends on a pretrained network.\n\nPotential negative societal impacts:\n\nNone that I can think of", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "6VZvsy9O5zz", "reviewer": "Reviewer_MiBY", "summary": "To model complex distributions of datasets, this paper proposed to partition datasets into a mixture of overlapping neighborhoods described by a datapoint and its nearest neighbors. An instance-conditioned generator is trained to learn the distribution around each datapoint, namely the generator learns to generate images similar to the conditional images. The proposed approach is evaluated on ImageNet and Coco-Stuff.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Pros:\n    - The paper is well written and easy to read.\n    - Comprehensive experiments are conducted, and multiple metrics are adopted to evaluate the proposed method.\n\n- Cons:\n    - The contribution is marginal. The authors claim the proposed method can learn complex distributions, however the generator can only generate images similar to the given conditional image, and the the number of modes to display during test phase largely depends on the conditional images provided.\n    - Can author provide several real-life applications when this model could be helpful?\n    - Is the proposed approach indeed under unconditional setting? It is more like image-to-image translations, instead of unconditional generations.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to Cons in the main review sections.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "0gRrXMyyeLg", "reviewer": "Reviewer_pxsk", "summary": "This paper introduces Instance Conditioned GAN (IC-GAN) which aims to model complex multi modal distributions in an unconditional manner. The model partitions the target distribution into sub distributions learned by conditioning on a single training point and its nearest neighbors. IC-GAN improves unconditional image performance baselines on ImageNet and COCO-stuff with a variety of architectures. The authors extend the model to perform class conditional generation and transfer learning.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Unconditional generation for datasets like ImageNet is a very hard problem and this paper proposes a novel way of partitioning the dataset. The authors draw parallels to Kernel density estimation (KDE) to motivate the approach which to my knowledge has not been done. The ease of performing transfer learning adds to the novelty of the approach.\n\nThe paper is very well written and easy to understand. The authors have conducted a thorough set of experiments and ablation studies to back up their claims.\n\nSome observations:\n1. The authors claim their method produces a diverse set of images but they do not share recall values anywhere apart from Fig 4. I think adding some metric of diversity in addition to FID/IS to at least some experiments would help.\n2. In the transfer learning section, I would have liked a little bit more discussion about what the authors think is being transferred. For example, if we condition on an image from a different dataset, what property of that dataset its discarded and what what property is preserved/ transferred?\n\nIn all, I think the authors have performed a comprehensive set of experiments which help push forward our understanding in unconditional generation and, therefore, I would recommend acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think the authors have adequately addressed limitations and societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two of the reviewers strongly support the paper and find the idea original. Another reviewer thought that the ability to perform good transfer learning is also of interest to the community. I agree with their observations and recommend acceptance.
Decision: Accept (Spotlight)