Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Summary:
The paper presents a new regularization technique termed consistency regularization for training GANs. The idea is the following: the authors propose to penalize the sensitivity of the last layer of the discriminator to augmented images. This idea is simple yet efficient: it is easy to implement, a regularization term is gradient-free, and its computation is up to 1.8 times faster than standard gradient-based regularization techniques. The authors tested different augmentation techniques and concluded that simple ones behave better (e.g., shifting and flipping). The experimental results show an impressive gain in FID measure, renewing the current state-of-the-art score for class conditional image generation on CIFAR-10 dataset. 

Pros: 
The proposed technique is very simple and intuitive; it easy to implement, and it is computationally cheap. The experiments were held for three runs with different random seeds, supporting its consistency.  The paper is overall clearly written and easy to understand. 

Cons:
The reported experimental results are held only for BigGAN architecture while not considering different networks to ensure the stability of the proposed regularization. Also, the paper would benefit from a clear experiment description on CelebA dataset (e.g., adding the results to Table 1). 

Questions:
-Have you tried other transforms, which potentially keep images on the manifold, including zoom, resize, rotation, brightness adjustment, etc.?
-How the number of layers in $L_{cs}$ (formulas 2-3) affects FID? 
-Have you considered an unconditional setting? 

Minor comments: 
-It would be more convenient if the authors explicitly numerate subplots; e.g., in Figure 2, it is confusing to refer the subplots labeled by (a)-(f) as written in caption. 
-Additionally, it would be nice to include say best FID scores over different loss functions (from Figure 2) to Table 1.
- In section 4.3, you wrote that you tried different $\lambda$ values:  {0,1,10, 100}, but Figure 4 does not cover all of them. 
- It would be nice to add implementation details (e.g., optimizer, learning rate parameters, steps per discriminator, etc.) for better reproducibility.
-The paper would benefit from illustrations of generated samples.
-Please check the spelling of the penultimate article name in references (Zhai et al., 2019). 

review_title: Official Blind Review #6

final_score: rating: 6: Weak Accept
review_content: This paper proposes to use Consistency Regularization for training GANs, a technique known to work well in unsupervised learning. The technique consists in applying a transformation to real images and enforcing that the features of the discriminator between the transformed inputs and the original inputs are similar. The author show that using this technique enables them to improve the performance of a standard GAN significantly on CIFAR10. They also carry an ablation study studying the influence of the different part of the proposed technique.

Overall I'm in favor of accepting this paper. The paper is well written, with convincing experiments and an interesting ablation study. However I have several minor issues that I think could greatly improve the paper if addressed.

Minor comments:
- I think an idea which is somewhat related but hasn't been mentioned in the paper, is the idea of adding noise to the input when training GANs [1]. I think this is worth mentioning in the related work.

- Related to the previous point, why penalizing features and not directly output ? What about also trying to classify the transformed images as real ? Also you say that penalizing the last layer, I think including the influence of m (eq 2) in the ablation study would be interesting.

- The authors provide some measure of standard deviation on some experiments but not on all of them. It would be nice to systematically report the standard deviation for every experiments.

- In figure 1 the author make the hypothesis that the discriminator will output very different score to images semantically close together. Did the author verify this hypothesis experimentally ?

- Also why penalizing only the samples from the real distribution and not from the generator ? have you tried both ?

- When the test accuracy of the discriminator is low, it could also be that the discriminator is under-fitting, it would be nice to also report the train accuracy for the discriminator.

- I think the conclusion about the effect of consistency regularization vs data augmentation is a bit vague since consistency regularization has no sense without data-augmentation. 

- It's quite interesting but also disappointing that combining transformations doesn't give that much of an improvement. Do the author have any intuition why this is the case ? and why learning them one after the other would work ?

References:
[1] Arjovsky and Bottou. "Towards Principled Methods for Training Generative Adversarial Networks." (ICLR 2017)
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The topic of this paper is out of the reviewer's domain (Bayesian optimization, RL, and neuroscience). The reviewer has been reviewing ICLR for several years. Such mismatches had not happened in the past.

The reviewer doesn't think this paper reached the bar of a good ICLR paper but hesitates to reject.


This work proposed a training stabilizer for GANs based on the notion of Consistency Regularization. Experimentally, the authors had augmented data passed into the GAN discriminator and penalize the sensitivity of the ultimate layer of the discriminator to these augmentations.

The authors claimed "We conduct a series of ablation studies to demonstrate that the
consistency regularization is compatible with various GAN architectures and loss
functions. Moreover, the proposed simple regularization can consistently improve
these different GANs variants significantly. "











review_title: Official Blind Review #1

Meta-review: The paper proposes a simple and effective way to stabilize training by adding consistency term to discriminator. Given the stochastic augmentation procedure $T(x)$ the loss is just a penalty on $D$. The main unsolved question why it help to make discriminator "smoother" in the consistency case for a standard GAN (since typically, no constraints are enforced). Nevertheless, at the moment this a working heuristics that gives new SOTA, and that is the main strength. The reviewer all agree to accept, and so do I.
Decision: Accept (Poster)