Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper propose to add instance discrimination (aka contrastive learning) objective for training generative adversarial networks. Specifically, (a) instance discrimination among real samples & (b) instance discrimination among fake samples are used as two axillary objectives. In addition, noise perturbation whereby the method uses z & z+epsilon to create two views needed for contrastive learning. (conventional contrastive learning uses 2 different augmentations of an image to create 2 views). Experimental results show that the proposed method InsGen outperforms prior methods that use data augmentations for training image GANs.
main_review: [Strengths]
- Overall, the paper is well-written and easy to follow.
- Using noise perturbation to create two views is pretty novel.
- The method can have high impact to the field, as it is simple but highly effective. I think it is much easier to use compared to ContraD --- which is another method that used contrastive learning for GAN training. Personally, I have tried using ContraD for my project, but it was not easy to implement. This method can be a good alternative to ContraD.

[Weaknesses]
- Novelty and Comparison against prior art: There is prior works in GAN literature that used instance discrimination objectives: ContraD [20] and DC-VAE [a]. Conceptually, InsGen is pretty similar to these works.
I believe comparison against these works should be written explicit in the paper.
This submission (InsGen) has subtle differences against ContraD:
(1) ContraD does not use instance discrimination for training the generator. InsGen does.
(2) A novel way of creating two views (noise perturbation).
(3) ContraD uses SimCLR (for real samples) & SupCon (for fake samples), whereas InsGen uses MoCo.

I think (1) & (2) are addressed in ablation, but (3) is not well-studied. The effect of different contrastive learning methods should be investigated. It is possible that the performance difference between ContraD and InsGen is due to the difference between SimCLR and MoCo.

Also, Table 1 (FFHQ) does not include ContraD. ContraD should be compared.

- Missing details about augmentation. What kinds and strengths of data augmentation operations are used for MoCo? Does it follow the original setting of MoCoV2?

- Missing results on CIFAR10. Since CIFAR10 is the most widely studied benchmark for Image GANs, it is better to include it.

[Missing References]
[a] Gaurav Parmar, Dacheng Li, Kwonjoon Lee, and Zhuowen Tu. Dual contradistinctive generative autoencoder. In CVPR, 2021.

======= Post-Rebuttal ======
After reading the latest results, I increased my rating from 5 to 6 for following reasons:
(a) the improvement is due to the difference in formulation (not the underlying contrastive learning method);
(b) this method is cleaner and easier to use than ContraD.
The formulation of ContraD is not natural and kind of hard to implement (from my experience). Although ContraD is a similar work on introducing contrastive learning to the GAN training, I believe the community will benefit from easier-to-use and better performing variant. Hence, I vote for accepting this paper.
limitations_and_societal_impact: Limitations or potential negative societal impact are not included in this paper. Including limitations as well as the position of this paper against prior literature will greatly enhance the quality of this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a novel GAN training method, named InsGen. This work effectively overcomes the performance degradation problem of GAN training with a small amount of the train dataset. The main idea is to train a discriminator by assigning an instance discrimination task as well as a bi-classification task (e.g. real or fake). The experimental results show that, with the smaller amount of data used, the synthesized image quality of InsGen outperforms the images from SOTA methods.

main_review: Strength
* This paper is well written and easy to follow up.
* The idea of noise perturbation is novel (and clever!). 
* The idea of enhancing discriminator by the instance discrimination does help enhance overall generation quality of GANs. 
* Experimental results show a significant improvement when training GANs using a small amount of data. 

Weakness
* Adding the experiments using datasets with various classes would be more convincing. At least, the experiment on cifar-10 should be appended. 
* This method does not seem to require a specific model structure, isn't it? It seems that the training method of InsGen can be applied not only to StyleGAN v2 but also to other structures such as BigGAN. However, in the current information, it is not clear. (If the method does not require specific model architectures, the authors should advertise themselves.)
* According to Tab.3, the 70K FID of the 3rd-row increases (which means quality degradation) compared to the 2nd-row. Any explanation?  
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a kind of contrastive loss that aims to discriminate both real and generated instances for GAN training. Experiments on FFHQ and AFHQ demonstrate that the proposed method implemented upon StyleGAN2-ADA can improve FID especially with limited data.
main_review: Strengths:
- The proposed method sounds novel and well motivated.
- The proposed method can improve StyleGAN2-ADA's FID especially with limited data. It is good to see that the proposed method can also improve its FID on the full FFHQ training set.

Weaknesses:
- My main concern is that the authors did not provide any empirical evidence on the generator diversity as the authors claimed. With limited data, the generator can memorize the full training set very easily, i.e., the generator may simply produce identical training images and thus achieve a low FID. Thus it is necessary to conduct nearest neighbor tests, style space interpolation, or measure the diversity quantitatively using e.g. precision/recall.
- I think the proposed loss combination is a bit complex but not fully justified. For example, why can't we simply apply data augmentation to fake images as done in real images instead of noise perturbation?
- The experiments are limited to the StyleGAN2-ADA backbone, so it is not clear whether the proposed method can generalize over other GAN architectures/algorithms. The datasets are also a bit limited.
- In the supplementary material, only the fact that the discriminator achieves better accuracy on the training set is not meaningful.

====== Post Rebuttal ======

Changed to 6 provided that the authors have promised to include diversity experiements in the revision and addressed my remaining concerns.
limitations_and_societal_impact: Good
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work adds an auxiliary contrastive learning task to train GANs in the regime of limited data. 
In particular, they add an instance discrimination task to the discriminator and update both, generator and discriminator weights based on the modified objective. 
main_review: Originality: 
The paper integrates well-known ideas from contrastive learning into GAN training to improve performance, particularly in the low-data regime. While the idea is simple, combining both fields is a very interesting approach. The related work is well-written and gives the reviewer a good overview of the current state of the field and this work's contribution.

Quality: 
Overall, the claims made in this paper are well supported, e.g. by the ablation in Section 4.2. However, the experimental evaluation could be more thorough: 
   - To support the generalizability of this approach it would be good to extend the evaluation to a second architecture, e.g. https://arxiv.org/abs/2101.04775.
   - Please discuss the following limitations: 
     i ) how does the contrastive objective impact the training time? Since the loss is computed over various samples does this slow down training or increase memory significantly? 
     ii) could this additional objective prevent the GAN from converging to the optimal solution? How important is the weighting of the losses and how are the hyperparameters from the appendix selected?
    - Which choices of augmentations are most effective? Are the same augmentations needed as for ADA or are some less effective for the task of instance discrimination?
   - Additional ablations that should be discussed: 
      i) If more synthetic samples keep improving FID (Figure 3), why stop at 1k? If feasible please add more results or discuss why this is not feasible in practice.
     ii) What is the value of epsilon (missing in hyper-parameters in the appendix) and how sensitive is the approach to this choice? Can you include some samples within the range of epsilon to give the reader an intuition on how similar these samples are?
      iii) Did you try using the same head for real/fake images (phi^r, phi^f)? While the given intuition (L.163) is comprehensible, an ablation study in the appendix would be good.
      iv) Why is the noise perturbation not applied to the generator? You could either give an intuition or add results of this ablation to the Table 3.
     v) Why do you use a momentum encoder? How much does this alone affect the results? Is this the standard (e.g. does ADA also use it?) or sth that you propose to do in this work? 

Clarity:
The paper is well-written and well-structured. In particular, I like that it is self-contained and that the related work gives a good overview of the field.
Minor: L.207, L_D^r and L_D^f are not defined previously

Significance: 
The results from this work might be useful to the community because it proposes a relatively simple method to improve performance in the low-data regime. To judge the practical value of the proposed method, it would be good to include an analysis on the overhead wrt training time / memory consumption regarding the added auxiliary task.
limitations_and_societal_impact: As mentioned above, please discuss the limitations of your work, in particular, wrt training time and memory consumption. Also please discuss the societal impact.
review_title: null

Meta-review: All reviewers agree that the paper should be accepted, given it's effectiveness. I agree with the reviewers. The contents of the rebuttal are important to distinguish the paper from previous work (especially ContraD) and therefore should be incorporated into the final version.
Decision: Accept (Poster)