Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ==== Summary ====

This paper improves upon state-of-the-art GANs by incorporating recent advances of contrastive representation learning into the training of discriminator. In particular, the discriminator loss function consists of three terms: (1) the original SimCLR loss on the multi-view real data pairs; (2) the supervised contrastive loss (Khosla et al, 2020) that assigns high scores among the fake sample pairs and giving lower score among the real data pairs; (3) the usual discriminator loss in GAN training. While each of these terms alone is not entirely new, the author proposes several tricks to make the training of GANs together with the contrastive loss works. Empirically, the proposed method outperforms other GAN methods trained with auxiliary data augmentation techniques, and demonstrates good representations under the linear classifier probing setup. 

Pros:

(1) Writing is clear and easy to follow

(2) Strong empirical performance in both image generation and classifier probing 
Solid experiment designs with thorough ablation studies

Cons:

(1) Rather limited novelty in terms of technical contributions

==== Technical Questions ====

Q1: Regarding the supervised contrastive loss for fake samples (i.e., L_{con}^{-} in Figure 1).
I am wondering if it is beneficial to also consider data augmentation on the fake sample produced by generators. In that case, the loss function matrix in Figure 1 will look closer to the L_{con}^{+} part, which encourages different views of the same fake sample to have similar representations.

Q2: I feel like there’s one ablation setting missing in Table 1: using “HFlip, Trans” data augmentation for the ContraD. This way, we can see the true benefit of combining those three losses.

Q3: For the linear evaluation results in Table 3, SimCLR results seem not consistent with their original paper? What causes the difference?
review_title: a well-executed empirical paper with strong performance

final_score: rating: 7: Good paper, accept
review_content: **Summary**
The manuscript proposes ContraD - a method that incorporates the recent SimCLR self-supervised learning method for images into the GAN training framework. Experimental results show that the proposed method consistently improves on strong baselines in terms of FID scores.

**Score justification**
The paper is well-written, the introduction carefully synthesizes a lot of recent work on self-supervised representation learning and GANs; results are convincing (albeit mostly obtained on smaller-scale datasets) and ablations support paper claims.


**Major comments**
* Ablation of $L_{con}^{+}$ in Table 6 tells a mixed story - it breaks down for the SNResNet discriminator, but seems to work well for the SNDCGAN discriminator. It's hard to conclude something definitive from the failed SNResNet experiment (could have worked with more hyper-parameter tuning?), but the successful SNResNet experiment suggests that the $L_{con}^{+}$ may not be necessary at all, which is at odds with the authors' narrative. Would be great to see more ablations showing that this is not the case, including a combined ablation of $L_{con}^{+}$ , $L_{con}^{-}$ and stop gradients.

* It would be interesting to see linear separation and transfer to other datasets (as in the SimCLR paper) for the ImageNet models. Strong results there would support the authors' narrative of the synergy GANs and SimCLR training.
* I am not fully convinced by the strong claims of strong coherence between GANs and contrastive representation learning (e.g. "[...] these two representations greatly complement each other [...]", "[...] a good contrastive representation is good for GANs and vise versa"). In particular it seems to be that GAN training benefits from contrastive learning (it seems to be a good auxiliary loss in presence of strong augmentation), but the claims of benefit to SimCLR don't seem fully substantiated.

**Minor comments**
* Multiple citations do not have correct capitalization (e.g. "gan")
* Results presented in Table 3 (linear evaluation on CIFAR-10) can be a bit misleading - at first glance they seem to suggest that * ContraD improves on SimCLR results from the original paper (model pre-trained on ImageNet and transferred to CIFAR-10). Making it clear that this is not the case (e.g. by including the original SimCLR results and/or supervised performance) would be helpful to the readers.
* Similarly, Table 4 would be easier to take in in the context of FID scores that are achieved by class-conditional GANs.

review_title: Good work on synthesizing self-supervised training and GANs; but some claims are perhaps too strong

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors suggest using the contrastive loss to improve the training of the discriminator and further stabilize the GAN training process. More specifically, the proposed method incorporates the self-supervised simCLR contrastive loss on a pair of transformed real images and supervised contrastive loss on the fake ones. The proposed method is evaluated on the image synthesis task on CIFAR10/100 and CelebA-HQ-128 images and over several different GAN models.

Strengths:
* The idea of using self-supervised learning for improving the training dynamics of the discriminator makes sense and is an interesting exploration area.
* Empirical evaluations show a consistent and significant advantage for the proposed methods and ablation studies verify the contributions of the different proposed components.

weaknesses:
* The proposed method is rather a careful ensembling of existing components, e.g. simCLR self-supervised or supervised contrastive loss in the right context, rather than a radically novel methodological contribution.
* The proposed method is only tested on relatively low-resolution datasets, namely CIFAR10/100 and CelebA-HQ-128. It would have been interesting to also demonstrate the contributions on the more challenging higher resolution datasets.

Detailed comments:
* Equation 7: If I am not mistaken, it doesn't seem quite the same as in the referred supervised contrastive loss from Khosla et al.; more specifically, I think the order between log and \sum_{v_{i+}^{(2)} \in V_{i+}^{(2)}} need to be reversed. Please clarify.
* An ablation study I was missing was having the proposed method without the extensive simCLR augmentations and see how it compares to the other methods.
* It would have been interesting to also compare with some other recent and relevant work, e.g. ADA (Karras et al.).
* "Remark that we use an independent projection header h_f instead of h_r" => Is this making a significant difference? Are there ablation studies showing this?
* Typo: "approaches that handles"

review_title:  

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose to improve GAN training by incorporating augmentations from contrastive learning. Specifically, a new contrastive discriminator, named ContraD, is proposed for GANs; with ContraD, the encoder part of the discriminator is trained with (two) contrastive learning losses, while the left discriminator head and the GAN generator are trained as usual. The authors argue that this specific fusion of GAN and contrastive learning signiﬁcantly stabilizes GAN training and, moreover, the fused two research fields could benefit each other.

The paper is well written overall. The idea is interesting, and the technical details may be sound. However, I am not fully convinced because of the concerns listed below.
 
Although the approach is proposed to stabilize GAN training, this aspect was not highlighted in the experiments.  
 
In Section 3.1, the discriminator encoder is trained with two specifically-chosen self-supervised-learning losses, and the experiments show that such a specific combination is essential for a good performance; otherwise, “the GAN loss could completely negate the effectiveness of ContraD. ” So why would that happen? How to choose those losses in practice? Detailed discussions are necessary here.
 
The first sentence of Section 3.2 is contradictory with Eq (12) and Algorithm 1.
 
In Eq (14), how to choose the v for a specific class? Was the label information used to get the v?
 
In Table 1, it seems the performance of StyleGAN2 is much lower than that reported in the original paper. Please elaborate on that.
 
In the experiments, larger batch sizes are used for the proposed method. Will that affect the fairness? Discussions are necessary.
review_title: A combination of GAN and contrastive learning, but with unaddressed concerns

Meta-review: This paper aims to improve the training of generative adversarial networks (GANs) by incorporating the principle of contrastive learning into the training of discriminators in GANs. Unlike in an ordinary GAN which seeks to minimize the GAN loss directly, the proposed GAN variant with a contrastive discriminator (ContraD) uses the discriminator network to first learn a contrastive representation from a given set of data augmentations and real/generated examples and then train a discriminator based on the learned contrastive representation. It is noticed that a side effect of such blending is the improvement in contrastive learning as a result of GAN training. The resulting GAN model with a contrastive discriminator is shown to outperform other techniques using data augmentation.

**Strengths:**
  * It proposes a new way of training the discriminators of GANs based on the principle of contrastive learning.
  * The paper is generally well written to articulate the main points that the authors want to convey.
  * The experimental evaluation is well designed and comprehensive.

**Weaknesses:**
  * Even though the proposed learning scheme is novel, the building blocks are based on existing techniques in GAN and contrastive learning.
  * The claim that GAN helps contrastive learning is not fully substantiated.
  * It is claimed in the paper that the proposed contrastive discriminator can lead to much stronger augmentations *without catastrophic forgetting*. However, this “catastrophic forgetting” aspect is not really empirically validated in the experiments.
  * The writing has room for improvement.

Despite its weaknesses, this paper explores a novel direction of training GANs that would be of interest to the research community.

Decision: Accept (Poster)