Instance-Conditioned GAN

1 Introduction . Generative Adversarial Networks ( GANs ) [ 18 ] have shown impressive results in unconditional image generation [ 27 , 29 ] . Despite their success , GANs present optimization difficulties and can suffer from mode collapse , resulting in the generator not being able to obtain a good distribution coverage , and often producing poor quality and/or low diversity generated samples . Although many approaches attempt to mitigate this problem – e.g . [ 20 , 32 , 35 , 38 ] – , complex data distributions such as the one in ImageNet [ 45 ] remain a challenge for unconditional GANs [ 33 , 36 ] . Classconditional GANs [ 5 , 39 , 40 , 56 ] ease the task of learning the data distribution by conditioning on class labels , effectively partitioning the data . Although they provide higher quality samples than their unconditional counterparts , they require labelled data , which may be unavailable or costly to obtain . Several recent approaches explore the use of unsupervised data partitioning to improve GANs [ 2 , 14 , 17 , 23 , 33 , 42 ] . While these methods are promising and yield visually appealing samples , their quality is still far from those obtained with class-conditional GANs . These methods make use of relatively coarse and non-overlapping data partitions , which oftentimes contain data points from different types of objects or scenes . This diversity of data points may result in a manifold with low ⇤Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . density regions , which degrades the quality of the generated samples [ 11 ] . Using finer partitions , however , tends to deteriorate results [ 33 , 36 , 42 ] because the clusters may contain too few data points for the generator and discriminator to properly model their data distribution . In this work , we introduce a new approach , called instance-conditioned GAN ( IC-GAN ) , which extends the GAN framework to model a mixture of local data densities . More precisely , IC-GAN learns to model the distribution of the neighborhood of a data point , also referred to as instance , by providing a representation of the instance as an additional input to both the generator and discriminator , and by using the neighbors of the instance as real samples for the discriminator . By choosing a sufficiently large neighborhood around the conditioning instance , we avoid the pitfall of excessively partitioning the data into small clusters . Given the overlapping nature of these clusters , increasing the number of partitions does not come at the expense of having less samples in each of them . Moreover , unlike when conditioning on discrete cluster indices , conditioning on instance representations naturally leads the generator to produce similar samples for similar instances . Interestingly , once trained , our IC-GAN can be used to effortlessly transfer to other datasets not seen during training by simply swapping-out the conditioning instances at inference time . IC-GAN bears similarities with kernel density estimation ( KDE ) , a non-parametric density estimator in the form of a mixture of parametrized kernels modeling the density around each training data point – see e.g . [ 4 ] . Similar to KDE , IC-GAN can be seen as a mixture density estimator , where each component is obtained by conditioning on a training instance . Unlike KDE , however , we do not model the data likelihood explicitly , but take an adversarial approach in which we model the local density implicitly with a neural network that takes as input the conditioning instance as well as a noise vector . Therefore , the kernel in IC-GAN is no longer independent on the data point on which we condition , and instead of a kernel bandwidth parameter , we control the smoothness by choosing the neighborhood size of an instance from which we sample the real samples to be fed to the discriminator . We validate our approach on two image generation tasks : ( 1 ) unlabeled image generation where there is no class information available , and ( 2 ) class-conditional image generation . For the unlabeled scenario , we report results on the ImageNet and COCO-Stuff datasets . We show that IC-GAN outperforms previous approaches in unlabeled image generation on both datasets . Additionally , we perform a series of transfer experiments and demonstrate that an IC-GAN trained on ImageNet achieves better generation quality and diversity when testing on COCO-Stuff than the same model trained on COCO-Stuff . In the class-conditional setting , we show that IC-GAN can generate images with controllable semantics – by adapting both class and instance– , while achieving competitive sample quality and diversity on the ImageNet dataset . Finally , we test IC-GAN in ImageNetLT , a long-tail class distribution ablated version of ImageNet , highlighting the benefits of nonparametric density estimation in datasets with unbalanced classes . Figure 1 shows IC-GAN unlabeled ImageNet generations ( a ) , IC-GAN class-conditional ImageNet generations ( b ) , and IC-GAN transfer generations both in the unlabeled ( c ) and controllable class-conditional ( d ) setting . 2 Instance-conditioned GAN . The key idea of IC-GAN is to model the distribution of a complex dataset by leveraging fine-grained overlapping clusters in the data manifold , where each cluster is described by a datapoint xi – referred to as instance – and its nearest neighbors set Ai in a feature space . Our objective is to model the underlying data distribution p ( x ) as a mixture of conditional distributions p ( x|hi ) around each of M instance feature vectors hi in the dataset , such that p ( x ) ⇡ 1M P i p ( x|hi ) . More precisely , given an unlabeled dataset D = { xi } Mi=1 with M data samples xi and an embedding function f parametrized by , we start by extracting instance features hi = f ( xi ) 8xi 2 D , where f ( · ) is learned in an unsupervised or self-supervised manner . We then define the set Ai of k nearest neighbors for each data sample using the cosine similarity – as is common in nearest neighbor classifiers , e.g . [ 53 , 54 ] – over the features hi . Figure 2a depicts a sample xi and its nearest neighbors . We are interested in implicitly modelling the conditional distributions p ( x|hi ) with a generator G✓G ( z , hi ) , implemented by a deep neural network with parameters ✓G . The generator transforms samples from a unit Gaussian prior z ⇠ N ( 0 , I ) into samples x from the conditional distribution p ( x|hi ) , where hi is the feature vector of an instance xi sampled from the training data . In IC-GAN , we adopt an adversarial approach to train the generator G✓G . Therefore , our generator is jointly trained with a discriminator D✓D ( x , hi ) that discerns between real neighbors and generated neighbors of hi , as shown in Figure 2b . Note that for each hi , real neighbors are sampled uniformly from Ai . Both G and D engage in a two player min-max game where they try to find the Nash equilibrium for the following equation : min G max D Exi⇠p ( x ) , xn⇠U ( Ai ) [ logD ( xn , f ( xi ) ) ] + Exi⇠p ( x ) , z⇠p ( z ) [ log ( 1 D ( G ( z , f ( xi ) ) , f ( xi ) ) ) ] . ( 1 ) Note that when training IC-GAN we use all available training datapoints to condition the model . At inference time , as in non-parametric density estimation methods such as KDE , the generator of ICGAN also requires instance features , which may come from the training distribution or a different one . Extension to class-conditional generation . We extend IC-GAN for class-conditional generation by additionally conditioning the generator and discriminator on a class label y . More precisely , given a labeled dataset Dl = { ( xi , yi ) } Mi=1 with M data sample pairs ( xi , yi ) and an embedding function f , we extract instance features hi = f ( xi ) 8xi 2 Dl , where f ( · ) is learned in an unsupervised , self-supervised , or supervised manner . We then define the set Ai of k nearest neighbors for each data sample using the cosine similarity over the features hi , where neighbors may be from different classes . This results in neighborhoods , where the number of neighbors belonging to the same class as the instance hi is often smaller than k. During training , real neighbors xj and their respective labels yj are sampled uniformly from Ai for each hi . In the class-conditional case , we model p ( x|hi , yj ) with a generator G✓G ( z , hi , yj ) trained jointly with a discriminator D✓D ( x , hi , yj ) . 3 Experimental evaluation . We describe our experimental setup in Section 3.1 , followed by results presented in the unlabeled setting in Section 3.2 , dataset transfer in Section 3.3 and class-conditional generation in Section 3.4 . We analyze the impact of the number of stored instances and neighborhood size in Section 3.5 . 3.1 Experimental setup . Datasets . We evaluate our model in the unlabeled scenario on ImageNet [ 45 ] and COCO-Stuff [ 6 ] . The ImageNet dataset contains 1.2M and 50k images for training and evaluation , respectively . COCOStuff is a very diverse and complex dataset which contains multi-object images and has been widely used for complex scene generation . We use the train and evaluation splits of [ 8 ] , and the ( un ) seen subsets of the evaluation images with only class combinations that have ( not ) been seen during training . These splits contain 76k , 2k , 675 and 1.3k images , respectively . For the class-conditional image generation , we use ImageNet as well as ImageNet-LT [ 34 ] . The latter is a long-tail variant of ImageNet that contains a subset of 115k samples , where the 1,000 classes have between 5 and 1,280 samples each . Moreover , we use some samples of four additional datasets to highlight the transfer abilities of IC-GAN : Cityscapes [ 10 ] , MetFaces [ 28 ] , PACS [ 31 ] and Sketches [ 15 ] . Evaluation protocol . We report Fréchet Inception Distance ( FID ) [ 22 ] , Inception Score ( IS ) [ 47 ] , and LPIPS [ 57 ] . LPIPS computes the distance between the AlexNet activations of two images generated with two different latent vectors and same conditioning . On ImageNet , we follow [ 5 ] , and compute FID over 50k generated images and the 50k real validation samples are used as reference . On COCO-Stuff and ImageNet-LT , we compute the FID for each of the splits using all images in the split as reference , and sample the same number images . Additionally , in ImageNet-LT we stratify the FID by grouping classes based on the number of train samples : more than 100 ( many-shot FID ) , between 20 and 100 ( med-shot FID ) , and less than 20 ( few-shot FID ) . For the reference set , we split the validation images along these three groups of classes , and generate a matching number of samples per group . In order to compute all above-mentioned metrics , IC-GAN requires instance features for sampling . Unless stated otherwise , we store 1,000 training set instances by applying k-means clustering to the training set and selecting the features of the data point that is the closest to each one of the centroids . All quantitative metrics for IC-GAN are reported over five random seeds for the input noise when sampling from the model . Network architectures and hyperparameters . As feature extractor f , we use a ResNet50 [ 21 ] trained in a self-supervised way with SwAV [ 7 ] for the unlabeled scenario ; for the class-conditional IC-GAN , we use a ResNet50 trained for the classification task on either ImageNet or ImageNetLT [ 26 ] . For ImageNet experiments , we use BigGAN [ 5 ] as a baseline architecture , given its superior image quality and ubiquitous use in conditional image generation . For IC-GAN , we replace the class embedding layers in the generator by a fully connected layer that takes the instance features as input and reduces its dimensionality from 2,048 to 512 ; the same approach is followed to adapt the discriminator . For COCO-Stuff , we additionally include the state-of-the-art unconditional StyleGAN2 architecture [ 29 ] , as it has shown good generation quality and diversity in the lower data regime [ 28 , 29 ] . We follow its class-conditional version [ 28 ] to extend it to IC-GAN by replacing the input class embedding by the instance features . Unless stated otherwise , we set the size of the neighborhoods to k=50 for ImageNet and k=5 for both COCO-Stuff and ImageNet-LT. See the supplementary material for details on the architecture and optimization hyperparameters . 3.2 Unlabeled setting ImageNet . We start by comparing IC-GAN against previous work in Table 1 . Note that unconditional BigGAN baseline is trained by setting all labels in the training set to zero , following [ 36 , 42 ] . IC-GAN surpasses all previous approaches at both 64⇥64 and 128⇥128 resolutions in both FID and IS scores . At 256⇥256 resolution , IC-GAN outperforms the concurrent unconditional diffusion-based model of [ 12 ] ; the only other result we are aware of in this setting . Additional results in terms of precision and recall can be found in Table 8 in the supplementary material . As shown in Figure 1a , IC-GAN generates high quality images preserving most of the appearance of the conditioning instance . Note that generated images are not mere training memorizations ; as shown in the supplementary material , generated images differ substantially from the nearest training samples . COCO-Stuff . We proceed with the evaluation of IC-GAN on COCOStuff in Table 2 . We also compare to state-of-the-art complex scene generation pipelines which rely on labeled bounding box annotations as conditioning – LostGANv2 [ 49 ] and OC-GAN [ 50 ] . Both of these approaches use tailored architectures for complex scene generation , which have at least twice the number of parameters of IC-GAN . Our IC-GAN matches or improves upon the unconditional version of the same backbone architecture in terms of FID in all cases , except for training FID with the StyleGAN2 backbone at 256⇥256 resolution . Overall , the StyleGAN2 backbone is superior to BigGAN on this dataset , and StyleGAN2-based IC-GAN achieves the state-of-the-art FID scores , even when compared to the bounding-box conditioned LostGANv2 and OC-GAN . IC-GAN exhibits notably higher LPIPS than LostGANv2 and OC-GAN , which could be explained by the fact that the latter only leverage one real sample per input conditioning during training ; whereas IC-GAN uses multiple real neighboring samples per each instance , naturally favouring diversity in the generated images . As shown in figures 3b and 3c , IC-GAN generates high quality diverse images given the input instance . A qualitative comparison between LostGANv2 , OC-GAN and IC-GAN can be found in Section E of the supplementary material .

Summary: This paper introduces Instance Conditioned GAN (IC-GAN) which aims to model complex multi modal distributions in an unconditional manner. The model partitions the target distribution into sub distributions learned by conditioning on a single training point and its nearest neighbors. IC-GAN improves unconditional image performance baselines on ImageNet and COCO-stuff with a variety of architectures. The authors extend the model to perform class conditional generation and transfer learning.