Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces Instance Conditioned GAN (IC-GAN) which aims to model complex multi modal distributions in an unconditional manner. The model partitions the target distribution into sub distributions learned by conditioning on a single training point and its nearest neighbors. IC-GAN improves unconditional image performance baselines on ImageNet and COCO-stuff with a variety of architectures. The authors extend the model to perform class conditional generation and transfer learning.
main_review: Unconditional generation for datasets like ImageNet is a very hard problem and this paper proposes a novel way of partitioning the dataset. The authors draw parallels to Kernel density estimation (KDE) to motivate the approach which to my knowledge has not been done. The ease of performing transfer learning adds to the novelty of the approach.

The paper is very well written and easy to understand. The authors have conducted a thorough set of experiments and ablation studies to back up their claims.

Some observations:
1. The authors claim their method produces a diverse set of images but they do not share recall values anywhere apart from Fig 4. I think adding some metric of diversity in addition to FID/IS to at least some experiments would help.
2. In the transfer learning section, I would have liked a little bit more discussion about what the authors think is being transferred. For example, if we condition on an image from a different dataset, what property of that dataset its discarded and what what property is preserved/ transferred?

In all, I think the authors have performed a comprehensive set of experiments which help push forward our understanding in unconditional generation and, therefore, I would recommend acceptance.
limitations_and_societal_impact: I think the authors have adequately addressed limitations and societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a new way of training GANs, which they call instance-conditioned GANs. This method is similar to conditional GANs, but instead of using a label, they use a feature vector extracted from some feature function (by using ResNet50 in the experiments).This bypasses the need for labels for conditional GANs. However, there is another difference. The features are assumed to be locally similar so when evaluating the loss over batches, real images are chosen to be neighbors of the given instance. This creates a kernel-density estimation type of model. Experimentally, these Instance-condition GANs perform very well, beating self-supervised, and unconditional GANs. Moreover, the conditional version of instance-conditioned GANs outperform conditional GANs as well in some smaller-resolution settings.
main_review: Originality:

This paper is very original and interesting.

Quality:

The paper is a bit confusing (see clarity below) but the research does seem to be well developed. The experiments are very impressive as well!

Clarity:

The overall clarity of this paper is only okay. Some of the terms are a bit confusing. For example, on line 71, the authors say “overlapping partitions”. What are overlapping partitions of a manifold? Typically, given a set A, we partition it. Meaning that there is a singular partition of disjoint sets whose union is the whole set A. I can only conclude that overlapping partitions means a collection of sets whose union is A but they are not disjoint. 

It is also confusing why they call a datapoint an instance. In fact this is defined on line 72 but used without definition on line 38, causing more confusion.

Line 79 says that Figure 2a shows 7 neighbors which is confusing because there are only 4 images, although there are 7 darker shapes. And the different shapes in the figure make it also confusing because one would think that neighboring shapes be similar. 

Significance:

This is a very interesting idea and significant because it can be used on completely unsupervised data.
limitations_and_societal_impact: Limitations:

The embedding depends on a pretrained network.

Potential negative societal impacts:

None that I can think of
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: To model complex distributions of datasets, this paper proposed to partition datasets into a mixture of overlapping neighborhoods described by a datapoint and its nearest neighbors. An instance-conditioned generator is trained to learn the distribution around each datapoint, namely the generator learns to generate images similar to the conditional images. The proposed approach is evaluated on ImageNet and Coco-Stuff.
main_review: - Pros:
    - The paper is well written and easy to read.
    - Comprehensive experiments are conducted, and multiple metrics are adopted to evaluate the proposed method.

- Cons:
    - The contribution is marginal. The authors claim the proposed method can learn complex distributions, however the generator can only generate images similar to the given conditional image, and the the number of modes to display during test phase largely depends on the conditional images provided.
    - Can author provide several real-life applications when this model could be helpful?
    - Is the proposed approach indeed under unconditional setting? It is more like image-to-image translations, instead of unconditional generations.
limitations_and_societal_impact: Please refer to Cons in the main review sections.
review_title: null

Meta-review: Two of the reviewers strongly support the paper and find the idea original. Another reviewer thought that the ability to perform good transfer learning is also of interest to the community. I agree with their observations and recommend acceptance.
Decision: Accept (Spotlight)