Instance-Conditioned GAN

1 Introduction . Generative Adversarial Networks ( GANs ) [ 18 ] have shown impressive results in unconditional image generation [ 27 , 29 ] . Despite their success , GANs present optimization difficulties and can suffer from mode collapse , resulting in the generator not being able to obtain a good distribution coverage , and often producing poor quality and/or low diversity generated samples . Although many approaches attempt to mitigate this problem – e.g . [ 20 , 32 , 35 , 38 ] – , complex data distributions such as the one in ImageNet [ 45 ] remain a challenge for unconditional GANs [ 33 , 36 ] . Classconditional GANs [ 5 , 39 , 40 , 56 ] ease the task of learning the data distribution by conditioning on class labels , effectively partitioning the data . Although they provide higher quality samples than their unconditional counterparts , they require labelled data , which may be unavailable or costly to obtain . Several recent approaches explore the use of unsupervised data partitioning to improve GANs [ 2 , 14 , 17 , 23 , 33 , 42 ] . While these methods are promising and yield visually appealing samples , their quality is still far from those obtained with class-conditional GANs . These methods make use of relatively coarse and non-overlapping data partitions , which oftentimes contain data points from different types of objects or scenes . This diversity of data points may result in a manifold with low ⇤Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . density regions , which degrades the quality of the generated samples [ 11 ] . Using finer partitions , however , tends to deteriorate results [ 33 , 36 , 42 ] because the clusters may contain too few data points for the generator and discriminator to properly model their data distribution . In this work , we introduce a new approach , called instance-conditioned GAN ( IC-GAN ) , which extends the GAN framework to model a mixture of local data densities . More precisely , IC-GAN learns to model the distribution of the neighborhood of a data point , also referred to as instance , by providing a representation of the instance as an additional input to both the generator and discriminator , and by using the neighbors of the instance as real samples for the discriminator . By choosing a sufficiently large neighborhood around the conditioning instance , we avoid the pitfall of excessively partitioning the data into small clusters . Given the overlapping nature of these clusters , increasing the number of partitions does not come at the expense of having less samples in each of them . Moreover , unlike when conditioning on discrete cluster indices , conditioning on instance representations naturally leads the generator to produce similar samples for similar instances . Interestingly , once trained , our IC-GAN can be used to effortlessly transfer to other datasets not seen during training by simply swapping-out the conditioning instances at inference time . IC-GAN bears similarities with kernel density estimation ( KDE ) , a non-parametric density estimator in the form of a mixture of parametrized kernels modeling the density around each training data point – see e.g . [ 4 ] . Similar to KDE , IC-GAN can be seen as a mixture density estimator , where each component is obtained by conditioning on a training instance . Unlike KDE , however , we do not model the data likelihood explicitly , but take an adversarial approach in which we model the local density implicitly with a neural network that takes as input the conditioning instance as well as a noise vector . Therefore , the kernel in IC-GAN is no longer independent on the data point on which we condition , and instead of a kernel bandwidth parameter , we control the smoothness by choosing the neighborhood size of an instance from which we sample the real samples to be fed to the discriminator . We validate our approach on two image generation tasks : ( 1 ) unlabeled image generation where there is no class information available , and ( 2 ) class-conditional image generation . For the unlabeled scenario , we report results on the ImageNet and COCO-Stuff datasets . We show that IC-GAN outperforms previous approaches in unlabeled image generation on both datasets . Additionally , we perform a series of transfer experiments and demonstrate that an IC-GAN trained on ImageNet achieves better generation quality and diversity when testing on COCO-Stuff than the same model trained on COCO-Stuff . In the class-conditional setting , we show that IC-GAN can generate images with controllable semantics – by adapting both class and instance– , while achieving competitive sample quality and diversity on the ImageNet dataset . Finally , we test IC-GAN in ImageNetLT , a long-tail class distribution ablated version of ImageNet , highlighting the benefits of nonparametric density estimation in datasets with unbalanced classes . Figure 1 shows IC-GAN unlabeled ImageNet generations ( a ) , IC-GAN class-conditional ImageNet generations ( b ) , and IC-GAN transfer generations both in the unlabeled ( c ) and controllable class-conditional ( d ) setting . 2 Instance-conditioned GAN . The key idea of IC-GAN is to model the distribution of a complex dataset by leveraging fine-grained overlapping clusters in the data manifold , where each cluster is described by a datapoint xi – referred to as instance – and its nearest neighbors set Ai in a feature space . Our objective is to model the underlying data distribution p ( x ) as a mixture of conditional distributions p ( x|hi ) around each of M instance feature vectors hi in the dataset , such that p ( x ) ⇡ 1M P i p ( x|hi ) . More precisely , given an unlabeled dataset D = { xi } Mi=1 with M data samples xi and an embedding function f parametrized by , we start by extracting instance features hi = f ( xi ) 8xi 2 D , where f ( · ) is learned in an unsupervised or self-supervised manner . We then define the set Ai of k nearest neighbors for each data sample using the cosine similarity – as is common in nearest neighbor classifiers , e.g . [ 53 , 54 ] – over the features hi . Figure 2a depicts a sample xi and its nearest neighbors . We are interested in implicitly modelling the conditional distributions p ( x|hi ) with a generator G✓G ( z , hi ) , implemented by a deep neural network with parameters ✓G . The generator transforms samples from a unit Gaussian prior z ⇠ N ( 0 , I ) into samples x from the conditional distribution p ( x|hi ) , where hi is the feature vector of an instance xi sampled from the training data . In IC-GAN , we adopt an adversarial approach to train the generator G✓G . Therefore , our generator is jointly trained with a discriminator D✓D ( x , hi ) that discerns between real neighbors and generated neighbors of hi , as shown in Figure 2b . Note that for each hi , real neighbors are sampled uniformly from Ai . Both G and D engage in a two player min-max game where they try to find the Nash equilibrium for the following equation : min G max D Exi⇠p ( x ) , xn⇠U ( Ai ) [ logD ( xn , f ( xi ) ) ] + Exi⇠p ( x ) , z⇠p ( z ) [ log ( 1 D ( G ( z , f ( xi ) ) , f ( xi ) ) ) ] . ( 1 ) Note that when training IC-GAN we use all available training datapoints to condition the model . At inference time , as in non-parametric density estimation methods such as KDE , the generator of ICGAN also requires instance features , which may come from the training distribution or a different one . Extension to class-conditional generation . We extend IC-GAN for class-conditional generation by additionally conditioning the generator and discriminator on a class label y . More precisely , given a labeled dataset Dl = { ( xi , yi ) } Mi=1 with M data sample pairs ( xi , yi ) and an embedding function f , we extract instance features hi = f ( xi ) 8xi 2 Dl , where f ( · ) is learned in an unsupervised , self-supervised , or supervised manner . We then define the set Ai of k nearest neighbors for each data sample using the cosine similarity over the features hi , where neighbors may be from different classes . This results in neighborhoods , where the number of neighbors belonging to the same class as the instance hi is often smaller than k. During training , real neighbors xj and their respective labels yj are sampled uniformly from Ai for each hi . In the class-conditional case , we model p ( x|hi , yj ) with a generator G✓G ( z , hi , yj ) trained jointly with a discriminator D✓D ( x , hi , yj ) . 3 Experimental evaluation . We describe our experimental setup in Section 3.1 , followed by results presented in the unlabeled setting in Section 3.2 , dataset transfer in Section 3.3 and class-conditional generation in Section 3.4 . We analyze the impact of the number of stored instances and neighborhood size in Section 3.5 . 3.1 Experimental setup . Datasets . We evaluate our model in the unlabeled scenario on ImageNet [ 45 ] and COCO-Stuff [ 6 ] . The ImageNet dataset contains 1.2M and 50k images for training and evaluation , respectively . COCOStuff is a very diverse and complex dataset which contains multi-object images and has been widely used for complex scene generation . We use the train and evaluation splits of [ 8 ] , and the ( un ) seen subsets of the evaluation images with only class combinations that have ( not ) been seen during training . These splits contain 76k , 2k , 675 and 1.3k images , respectively . For the class-conditional image generation , we use ImageNet as well as ImageNet-LT [ 34 ] . The latter is a long-tail variant of ImageNet that contains a subset of 115k samples , where the 1,000 classes have between 5 and 1,280 samples each . Moreover , we use some samples of four additional datasets to highlight the transfer abilities of IC-GAN : Cityscapes [ 10 ] , MetFaces [ 28 ] , PACS [ 31 ] and Sketches [ 15 ] . Evaluation protocol . We report Fréchet Inception Distance ( FID ) [ 22 ] , Inception Score ( IS ) [ 47 ] , and LPIPS [ 57 ] . LPIPS computes the distance between the AlexNet activations of two images generated with two different latent vectors and same conditioning . On ImageNet , we follow [ 5 ] , and compute FID over 50k generated images and the 50k real validation samples are used as reference . On COCO-Stuff and ImageNet-LT , we compute the FID for each of the splits using all images in the split as reference , and sample the same number images . Additionally , in ImageNet-LT we stratify the FID by grouping classes based on the number of train samples : more than 100 ( many-shot FID ) , between 20 and 100 ( med-shot FID ) , and less than 20 ( few-shot FID ) . For the reference set , we split the validation images along these three groups of classes , and generate a matching number of samples per group . In order to compute all above-mentioned metrics , IC-GAN requires instance features for sampling . Unless stated otherwise , we store 1,000 training set instances by applying k-means clustering to the training set and selecting the features of the data point that is the closest to each one of the centroids . All quantitative metrics for IC-GAN are reported over five random seeds for the input noise when sampling from the model . Network architectures and hyperparameters . As feature extractor f , we use a ResNet50 [ 21 ] trained in a self-supervised way with SwAV [ 7 ] for the unlabeled scenario ; for the class-conditional IC-GAN , we use a ResNet50 trained for the classification task on either ImageNet or ImageNetLT [ 26 ] . For ImageNet experiments , we use BigGAN [ 5 ] as a baseline architecture , given its superior image quality and ubiquitous use in conditional image generation . For IC-GAN , we replace the class embedding layers in the generator by a fully connected layer that takes the instance features as input and reduces its dimensionality from 2,048 to 512 ; the same approach is followed to adapt the discriminator . For COCO-Stuff , we additionally include the state-of-the-art unconditional StyleGAN2 architecture [ 29 ] , as it has shown good generation quality and diversity in the lower data regime [ 28 , 29 ] . We follow its class-conditional version [ 28 ] to extend it to IC-GAN by replacing the input class embedding by the instance features . Unless stated otherwise , we set the size of the neighborhoods to k=50 for ImageNet and k=5 for both COCO-Stuff and ImageNet-LT. See the supplementary material for details on the architecture and optimization hyperparameters . 3.2 Unlabeled setting ImageNet . We start by comparing IC-GAN against previous work in Table 1 . Note that unconditional BigGAN baseline is trained by setting all labels in the training set to zero , following [ 36 , 42 ] . IC-GAN surpasses all previous approaches at both 64⇥64 and 128⇥128 resolutions in both FID and IS scores . At 256⇥256 resolution , IC-GAN outperforms the concurrent unconditional diffusion-based model of [ 12 ] ; the only other result we are aware of in this setting . Additional results in terms of precision and recall can be found in Table 8 in the supplementary material . As shown in Figure 1a , IC-GAN generates high quality images preserving most of the appearance of the conditioning instance . Note that generated images are not mere training memorizations ; as shown in the supplementary material , generated images differ substantially from the nearest training samples . COCO-Stuff . We proceed with the evaluation of IC-GAN on COCOStuff in Table 2 . We also compare to state-of-the-art complex scene generation pipelines which rely on labeled bounding box annotations as conditioning – LostGANv2 [ 49 ] and OC-GAN [ 50 ] . Both of these approaches use tailored architectures for complex scene generation , which have at least twice the number of parameters of IC-GAN . Our IC-GAN matches or improves upon the unconditional version of the same backbone architecture in terms of FID in all cases , except for training FID with the StyleGAN2 backbone at 256⇥256 resolution . Overall , the StyleGAN2 backbone is superior to BigGAN on this dataset , and StyleGAN2-based IC-GAN achieves the state-of-the-art FID scores , even when compared to the bounding-box conditioned LostGANv2 and OC-GAN . IC-GAN exhibits notably higher LPIPS than LostGANv2 and OC-GAN , which could be explained by the fact that the latter only leverage one real sample per input conditioning during training ; whereas IC-GAN uses multiple real neighboring samples per each instance , naturally favouring diversity in the generated images . As shown in figures 3b and 3c , IC-GAN generates high quality diverse images given the input instance . A qualitative comparison between LostGANv2 , OC-GAN and IC-GAN can be found in Section E of the supplementary material .

Summary: To model complex distributions of datasets, this paper proposed to partition datasets into a mixture of overlapping neighborhoods described by a datapoint and its nearest neighbors. An instance-conditioned generator is trained to learn the distribution around each datapoint, namely the generator learns to generate images similar to the conditional images. The proposed approach is evaluated on ImageNet and Coco-Stuff.