Consistency Regularization for Generative Adversarial Networks
Authors: Han Zhang
Zizhao Zhang
Augustus Odena
Honglak Lee
Venue:  (2020)
Keywords: Generative Adversarial Networks
Consistency Regularization
GAN

Abstract: Generative Adversarial Networks (GANs) are known to be difficult to train, despite considerable research effort. Several regularization techniques for stabilizing training have been proposed, but they introduce non-trivial computational overheads and interact poorly with existing techniques like spectral normalization. In this work, we propose a simple, effective training stabilizer based on the notion of consistency regularization—a popular technique in the semi-supervised learning literature. In particular, we augment data passing into the GAN discriminator and penalize the sensitivity of the discriminator to these augmentations. We conduct a series of experiments to demonstrate that consistency regularization works effectively with spectral normalization and various GAN architectures, loss functions and optimizer settings. Our method achieves the best FID scores for unconditional image generation compared to other regularization methods on CIFAR-10 and CelebA. Moreover, Our consistency regularized GAN (CR-GAN) improves state of-the-art FID scores for conditional generation from 14.73 to 11.48 on CIFAR-10 and from 8.73 to 6.66 on ImageNet-2012.

Published as a conference paper at ICLR 2020
CONSISTENCY REGULARIZATION FOR GENERATIVE ADVERSARIAL NETWORKS
Han Zhang, Zizhao Zhang, Augustus Odena, Honglak Lee Google Research
(zhanghan, zi zhaoz, augustusodena, honglak aak @google.com
ABSTRACT
Generative Adversarial Networks (GANs) are known to be difficult to train, de- spite considerable research effort. Several regularization techniques for stabilizing training have been proposed, but they introduce non-trivial computational over- heads and interact poorly with existing techniques like spectral normalization. this work, propose simple, effective training stabilizer based on the notion of consistency regularization-a popular technique in the semi-supervised learning literature. In particular, we augment data passing into the GAN discriminator and penalize the sensitivity the discriminator to these augmentations. We conduct a series of experiments to demonstrate that consistency regularization works effee- tively with spectral normalization and various GAN architectures, loss functions and optimizer settings. Our method achieves the best FID scores for unconditional image generation compared to other regularization methods on CIFAR-10 and CelebA. Moreover, Our consistency regularized GAN (CR-GAN) improves state- of-the-an art FID scores for conditional generation from 14.73 to 11.48 on CIFAR-10 and from 8.73 to 6.66 on ImageNet-2012.
INTRODUCTION
Generative Adversarial Networks (GANs) (Goodfellow et al.] 2014) have recently demonstrated impressive results on image-synthesis benchmarks (Radford al. [2066 Zhang al. 2017 Miyato & Koyama| 2018 Zhang et al. 2018 Brock et al. [0188] Karras et al. 2019). In the original setting, GANs are composed two neural networks trained with competing goals: the generator trained to synthesize realistic samples to fool the discriminator and the discriminator is trained to distinguish real samples from fake ones produced by the generator.
One major problem with GANs is the instability of the training procedure and the general sensitivity of the results to various hyperparameters (Salimans et al.) 2016). Because GAN training implicitly requires finding the Nash equilibrium of non-convex game in continuous and high dimensional parameter space, it is substantially more complicated than standard neural network training. fact, formally characterizing the convergence properties of the GAN training procedure is mostly open problem (Odena 2019). Previous work (Arjovsky& & Bottou 2017 Miyato et al. 2018a Odena et al.) 2017 Chen et al. 2019 Wei et al. 2018) has shown that interventions focused on the discriminator can mitigate stability issues. Most successful interventions fall into two categories, normalization and regularization. Spectral normalization is the most effective normalization method, in which weight matrices in the discriminator are divided by an approximation of their largest singular value. For regularization, Gulrajaai al. (2017) penalize the gradient norm of straight lines between real data and generated data. Roth et al. (2017) propose to directly regularize the squared gradient norm for both the training data and the generated data. DRAGAN (Kodali al., 2017) introduces another form of gradient penalty where the gradients at Gaussian perturbations of training data are penalized. One may anticipate simultaneous regularization and normalization could improve sample quality. However, most of these gradient based regularization methods either provide marginal gains or fail to introduce any improvement when normalization is used (Kurach et al. 2019). which is also observed in our experiments. These regularization methods and spectral normalizaton are motivated by controlling Lipschitz constant of the discriminator. We suspect this might be the reason that applying both does not lead to overlaid gain.

---

Published as a conference paper at ICLR 2020
Image space
Manifold space
Semantic feature space
Before consistency
After consistency
An illustration of consistency regularization for GANs. Before consistency regularization, the zoomed-in dog and the zoomed-in cat (hottom left) can be closer than they are to their original images in feature space induced by the GAN discriminator. This is illustrated in the upper right (the semantic feature space), where the purple dot is closer to the blue dot than to the red dot, and so forth. After we enforce consistency regularization based on the implicit assumption that image augmentation preserves the semantics we care about, the purple dot pulled closer to the red dot.
Figure 1:
In this paper, we examine a technique called consistency regularization (Bachman et al. 2014 Saj-
jadi et 2016; Laine & Aila 2016 Zhai al. 2019 Xie al. 2019 Hu et 2017) in contrast to gradient-based regularizers. Consistency regularization is widely used in semi-supervised learning to ensure that the classifier output remains unaffected for an unlabeled example even it augmented in semantic-preserving ways. In light of this intuition, we hypothesize well-trained discrimina- tor should also be regularized to have the consistency property, which enforces the discriminator to be unchanged by arbitrary semantic- semantic-preserving perturbations and to focus more semantic and structural changes between real and fake data. Therefore, we propose a simple regularizer to the dis- criminator of GAN: augment images with pemantic-reserrving augmentations before they fed into the GAN discriminator and penalize the sensitivity of the discriminator to those augmentations.
This technique simple to and surprisingly effective. It is well less computationally expen- sive than prior techniques. More importantly, in our experiments, consistency regularization can always further improve the model performance when spectral normalization is used, whereas the performance gains of previous regularization methods diminish in such case. In extensive ablation studies, we show that works across a large range of GAN variants and datasets. We also show that simply applying this technique on top of existing GAN models leads to new state-of- oof-the-art results as measured by Frechet Inception Distance (Heusel al. 2017).
In summary, our contributions are summarized as follows:
We propose consistency regularization for GAN discriminators to yield simple, effective regularizer with lower computational cost than gradient-based regularization methods.
We conduct extensive experiments with different GAN variants demonstrate that our technique interacts effectively with spectral normalization. Our consistency regularized GAN (CR-GAN) achieves the best FID scores for unconditional image generation on both CIFAR-10 10 and CelebA.
We show that simply applying the proposed technique can further boost the performance of otate-of-the-art GAN models. We improve FID) scores for conditional image generation from 14.73 to 11.48 on CIFAR-10 and from 8.73 to 6.66 on ImageNet-2012.
METHOD
2.1 GANS
A GAN consists of generator network and discriminator network. The generator G takes latent variable 2 ~ p(z) sampled from a prior distribution and maps it to the observation space X' and produces a decision output over possible
X. The discriminator D takes an observation observation sources (either from G or from the empirical data distribution). In the standard GAN training procedure the generator G and the discriminator D are trained by minimizing the following
2

---

Published as a conference paper at ICLR 2020
objectives in an alternating fashion:
LD -Ba-pan [log D(x)] - Bzxp(=)) [] log (z)(
(1)
Le - -zp(()) log ([G()))),
where p(2) usually a standard normal distribution. This formulation originally proposed by Goodfellow et al.] (2014) as non-saturating (NS) GAN. A significant amount of research has been done on modifying this formulation in order to improve the training process. A notable example is the hinge-loss version of the adversarial Ioss (Lim & Ye 2017 Tran et al. 2017):
LD Berppan [min(0, -1 (xx)) E pp=(() [min -1 D(G(2)))
(2)
L = Pxp(())
[+((=)))]
Another commonly adopted GAN formulation is the Wassertein GAN (WGAN) (Arjovsky al. 2017), in which the authors propose clipping the weights of the discriminator in an attempt enforce that the GAN training procedure implicitly optimizes bound the Wassertein distance between the target distribution and the distribution given by the generator. The loss function of WGAN can
be written
Lp = Egg-pon [D()) + Essp(s) [D(G(=))),
LG = - Ez)p(=) [D(G(
Subsequent work has refined this technique in several ways (Gulrajani et al. 2017 Miyato et al. 2018a Sa Zhang al.) 2019), and the current widely-used practice is enforce spectral normaliza- tion Miyato et al. 2018a) on both the generator and the discriminator.
2.2
CONSISTENCY REGULARIZATION
Consistency regularization has emerged as gold-standard technique (Sajjadi et al. 2016 Laine & Aila 2016 Zhai et al. 2019; Xie et al. 2019 Oliver et al. 2018 Berthelot et al. 2019) for semi- supervised learning image data. The basic idea is simple: input image is perturbed in some semantics- prrenttcc---eeervinng ways and the sensitivity
the classifier to that perturbation is penalized. The perturbation take many forms: it can be image flipping. cropping. adversarial attacks. The regularization form is either the mean-squared between the model's output for perturbed and non-perturbed input the KI. divergence (Xie al. 2019
(Sajjadi et al., Laine Aila (016)
Miyato et al. 2018b) between the distribution over classes implied by the output logits.
2.3
CONSISTENCY REGULARIZATION FOR GANS
The goal of the discriminator in GANs is to distinguish real data from fake ones produced by the generator. The decision should be invariant to any valid Nomainn-specite data augmentations. For example, in the image domain, the image being real or not should not change if we flip the image horizontally or translate the image by a few pixels. However, the discriminator in GANs does not guarantee this property explicitly.
To resolve this, we propose a consistency regularization on the GAN discriminator during train- ing. In practice, randomly augment training images they are passed to the discriminator and penalize the sensitivity of the discriminator to those augmentations.
We use (x) to denote the output vector before activation of the jth layer of the discriminator given input z. T(x) denotes a stochastic data augmentation function. This function can be linear or nonlinear, but aims to preserve the semantics of the input. Our proposed regularization is given by
min Lar = min Σ -
(4)
where j indexes the layers, m is the starting layer and n is the ending layer that consistency is enforced. Xj is weight coefficient for jth layer and ||-I| denotes L norm of given vector. This consistency regularization encourages the discriminator to produce the same output for a data point under various data augmentations.
3

---

Published as a conference paper at ICLR 2020
Algorithm Consistency Regularized GAN (CR-GAN), We use X = 10 by default.
Input: generator and discriminator parameters 0g, Op, consistency regularization coefficient X, Adam hyperparameters ax B1, 82. batch size M, number of discriminator iterations per gen- erator iteration ND
1:
for number of training iterations do
2:
for f, 1, ND do
3:
for i = 1,..,, M do
4:
Sample p(z), x: Paat (z)
5:
Augment x to get (x))
6:
L D(x) (x)(]]
7:
DD D (xx)
end for 8:
9: P + Adam 1/2 EE (Z.)) AL 31,
10: end for
Sample a batch of latent variables 1 ~ p(2)
12: 0c t A-dam 1
M (-
13:
end for
In our experiments, we find that consistency regularization on the last layer of the discriminator before the activation function is sufficient. Ley can be rewritten as
Ler D(x) D(T()))]
(5)
where from now on we will drop the layer index for brevity. This cost is added to the discriminator loss (weighted by hhper-parametee A) when updating the diseriminator parameters. The generator update remains unchanged. Thus, the overall consistency regularized GAN (CR-GAN) objective is written as
L L.D + Xlar
L-= T.G.
Our design of Lep is general-purpose and thereby can work with any valid adversarial losses LG and LD for GANs (See Section 2.1 for examples). Algorithm illustrates D the details of CR-GAN with Wassertein loss as an example. In contrast to previous regularizers, our method does not increase much overhead. The only extra computational cost comes from feeding additional (third) image through the discriminator forward and backward when updating the discriminator parameters.
3 EXPERIMENTS
This section validates our proposed CR-GAN method. First we conduct large scale study to com- pare consistency regularization to existing GAN regularization techniques (Kodali et al. 2017: Gul- rajani et al. 2017 Roth et al., 2017) for several GAN architectures, loss functions and other hyper- parameter settings. We then apply consistency regularization state-of-the-ar GAN model (Brock) et al. 2018) and demonstrate performance improvement, Finally, we conduct ablation studies to in- vestigate the importance of various design choices and hyper-parameters. All our experiments
based on the open-soure code from Compare GAN (Kurach et al. 2019). which is available at gan.
http://gihubb.ooogggollleco/
3.1 DATASETS AND EVALUATION METRICS
We validate our proposed method on three datasets: CIFAR-10 Krizhevsky 2009 CELEBA-HQ-
128 (Karras et al. 2018). and ImageNet-2012 (Russakovsky et al. 2015). We follow the procedure
in Kurach et al. (2019) to prepare datasets. CiFAR-10 consists of 60K of 32 32 images in 10 classes; 50K for training and 10K for testing. CELEBA-HQ-I23 (CelebA) contains 30K images of faces at resolution of 128 x 128. We 3K images for testing and the rest of images for training. ImageNet-2012 contains roughly 1.2 million images with 1000 distinct categories and we down-sample the images to 128 128 in experiments.
We adopt the Fréchet Inception distance (FID) (Heusel et al. 2017) as primitive metric for quantita- tive evaluation, as FID has proved be more consistent with human evaluation. In experiments the
4

---

Published as a conference paper at ICLR 2020
True| Hinge
23
21
40
20
18
W/O
13R
Oura
/5A
Oura
W00
DR
Ours
(8)
(b)
(0)
SN = True loss - Hinge
SN = True | loss NS
SN True [lass WAS
28
250
22
150
50
WH GP
DR
150t
Our
WID
GP
DR
Our
V/0
D4
J5R
Oun
[c)
(1)
Figure 2: Comparison of our method with existing regularization techniques under different GAN losses. Techniques include regularization (W/O), Gradient Penalty (GP) (Gulrajani et al. 2017)) DRAGAN (DR) (Kodali et al. (2017) and JS- IS-Regularizer (JSR) (Roth et al. 2017). Results (a-c) are for CIFAR-10 10 and results (d-f) are for CelebA.
FID is calculated on the test dataset. In particular, we use 10K generated images vs. 10K test images on CIFAR-10, 3K vs. 3K on CelebA and 50K vs. 50K on ImageNet. We also provide the Inception Score (Salimans et al. (0166) for different methods in the Appendix [ for supplementary results. By default, the augmentation used in consistency regularization a combination of randomly shifting the image by a few pixels and randomly flipping the image horizontally. The shift size is 4 pixels for CIFAR-I0 10 and CelebA and 16 for ImageNet.
3.2 COMPARISON WITH OTHER GAN REGULARIZATION METHODS
In this section, compare our methods with three GAN regularization techniques, Gradient Penalty (GP) (Gulrajani et al., 2017). DRAGAN Regularizer (DR) (Kodali et al. 2017) and JS-Regularizer (ISR) (Roth et al. 2017) CIFAR-10 and CelebA.
Following the procedures from (Kurach et al. 2019 Lucic et al. 2018), we evaluate these methods across different optimizer parameters, loss functions, regularization coefficient and neural architec- tures. For optimization, we use the Adam optimizer with batch size of 64 for all our experiments. We stop training after 200k generator update steps for CIFAR-10 10 and 100k steps for CelebA. By default, spectral normalization (SN) (Miyato et al. (018a) is used in the discriminator, as this is the most effective normalization method for GANs (Kurach et al., 2019) and is becoming the standard for 'modern' GANs (Zhang et al. 2019 Brock et al. 2018). Results without spectral normalization can be seen in the Appendix
3.2. IMPACT OF LOSS FUNCTION
In this section, we discuss how each regularization method performs when the loss function is changed. Specifically, we evaluate regularization methods using three loss functions: the non- saturating loss (NS) (Goodfellow al. [2014), the Wasserstein loss (WAS) (Arjovsky et al., 2017). and the hinge loss (Hinge) (Lim & 2017 Tran et al. 2017). For each loss function, we evaluate over pyper-parametts settings of the Adam optimizer (more details in Section Af the appendix). For each configuration, we run each model 3 times with different random seeds. For the regulariza- tion coefficient, we use the best value reported in the corresponding paper. Specifically is set to be 10 for both GP, DR and our method and 0.1 for JSR. In this experiment, network architecture (Miyato et al.) [2018a] 8a) for simplicity. In the end, similar as Kurach et al. (2099) aggregate all runs and report the FID distribution of the top 15% of trained models.
use the SNDCGAN
The results are shown in Figure 2 The consistency regularization improves the baseline across all different loss functions and both datasets. Other techniques have more mixed results: For example,
5

---

Published as a conference paper at ICLR 2020
Setting CIFAR-10 (SNDCGAN)
W/O
GP
DR
JSR
Ours (CR-GAN)
24.73
25.83
25.08
25.17
18.72
CIFAR-10 10 (ResNet)
19.00
19.74
18.94
19.59
14.56
CelebA (SNDCGAN)
25.95
22.57
21.91
22.17
16.97
Table 1: Best FID scores for unconditional image generation on CIFAR-10 10 and CelebA
CIFAR-10
CalebA
Lambda
10
10
1.0
10.0
100.0
10
30
Ours
JSR
Figure 3: Comparison of FID scores with different values of the regularization coefficient À on CIFAR-10 and CelebA. The dotted line is model without regularization.
GP and DR can marginally improve the performance for settings (d) and (c) but lead worse results for settings (a) and (b) (which is consistent with findings from Kurach et al. (2019). In all cases, our consistency reeisttancyyruullaiieeeee GAN models have the lowest (best) FID.
This finding is especially encouraging, considering that the consistency regularization has lower computational cost (and simpler to implement) than the other techniques. In our experiments, the consistency regularization is around 1.7 times faster than gradient based regularization techniques, including DR, GP and JSR, which need to compute the gradient of the gradient norm [[V]l Please see Table Cl in the appendix for the actual training speed.
3.2.2 IMPACT OF THE REGULARIZATION COEFFICIENT
Here we study the sensitivity of GAN regularization techniques the regularization coefficient X. We train SNDCGANs with non-saturating losses and fix the other hyper-parameters. À is chosen among (0.1. 1, 10, 100). The results are shown Figure From this figure, we can see consistency regularization is more robust to changes in À than other GAN regularization techniques (it also has the best FID for both datasets). The results indicate that consistency regularization can used as plug-and-play technique to improve GAN performance in different settings without much hyper- parameter tuning.
3.2.3 IMPACT OF NEURAL ARCHITECTURES
To validate whether the above findings hold across different neural architectures, we conduct exper- iments on CIFAR-ID 10 using a ResNet (He al. [016 Gulrajani al. (2077) architecture instead of an SNDCGAN. All other experimental settings are same as in Section 3.2.1 The FID values are presented in Figure
By comparing results Figure + and Figure 2 can see that results on SNDCGAN and results on ResNet are comparable, though consistency regularization favors even better in this case: In sub-plot (c) of Figure we can see that consistency regularization is the only regularization method that can generate satisfactory samples with reasonable FID score (The FID scores for other methods are above 100). Please see Figure [3 for the actual generated samples in this setting. As in Section 3.2. consistency regularization has the best FID for each setting.
In Table we show FID scores for the best-case settings from this section. Consistency regulariza- tion improves on the baseline by a large margin and achieves the best results across different network architectures and datasets. In particular, achieves an FID 14.56 on CIFAR-10 16.97 on CelebA. In fact, FID score of 14.56 on CIFAR-10 for unconditional image generation is even lower than the 14.73 reported in [Brock et al. (2018) for condition amage-synthene with a much larger network architecture and much bigger batch size.

---

Published as a conference paper at ICLR 2020
SN Thus (ssss Hinge
5h = True los =5
155-
SN - Thus [sess WA5
24-
140
22
125-
16
16
450
DK
14
W/
W00
0ars
(e]
(b)
(c)
Figure 4: Comparison of FID scores with ResNet structure on different loss settings CIFAR-10.
3.3 COMPARISON WITH STATE-OF-THE-ART GAN MODELS
In this section, we add consistency regularization to the state-of-the-ee BigGAN model (Brock al. 2018) and perform class conditional image-synthesis on CIFAR-10 and ImageNet. Our model has exactly the same architecture and is trained under the same settings as BigGAN". the open-source implementation of BigGAN from Kurach et al. (2019). The only difference is that our model uses consistency regularization. In Table 2 we report the original FID scores without noise truncation. Consistency regularization improves the FID score of BigGAN on CIFAR-I0 from 20.42 to 11.48. In addition, the FID on ImageNet is improved from 7.75 to 6.66.
Generated samples for CIFAR-10 and ImageNet with consistency regularized models and baseline models are shown in Figures E1 and E3 in the appendix.
Dataset
SNGAN
SAGAN
BigGAN
BigGAN CR-BigGAN"
CIFAR-10 10
17.5
14.73
20.42
11.48
ImageNet 27.62
18.65
8.73
7.75
6.66
Table 2: Comparison of our technique with of GAN models including SNGAN (Miy- ato & Koyamal 2018), SAGAN (Zhang et al.] 2019) and BigGAN (Brock et al. 2018) for class conditional image generation on CIFAR-10 10 and ImageNet in terms of FID. BigGAN the Big- GAN implementation of Kurach et al. (2019). CR-BigGAN" has the exactly same architecture as BigGAN and is trained with the same settings. The only difference is CR-BigGAN adds consis- tency regularization.
4 ABLATION STUDIES AND DISCUSSION
4.1 How MUCH DOES AUGMENTATION MATTER BY ITSELF?
Our consistency regularization technique actually has two parts: we perform data augmentation on inputs from the training data, and then consistency is enforced between the augmented data and the original data. We are interested in whether the performance gains shown in Section 3 are merely due data augmentation, since data augmentation reduces the over-fitting of the discriminator to the input data. Therefore, we have designed an experiment to answer this question. First, we train three GANs: (1) a GAN trained with consistency regularization, as in Algorithm [ (2) a baseline GAN trained without augmentation or consistency regularization, and (3) a GAN trained with only data augmentation and no consistency regularization. We then plot (Figure 5 both their FID and the test accuracy of their discriminator on a held-out test set. The FID tells us how 'good' the resulting GAN is. and the discriminator test accuracy tells us how much the GAN discriminator over fits. Interestingly, we find that these two measures are not well correlated in this case. The model trained with only data augmentation over-fits substantially less than the baseline GAN, but has almost the same FID. The model trained with consistency regularization has the same amount of over-fitting as the model trained with just data augmentation, but a much lower FID.
This suggests an interesting hypothesis, which is that the mechanism by which the consistency regu- larization improves GANs is not simply discriminator generalization (in terms of classifying images into real vs fake). We believe that the main reason for the impressive gain from the consistency regularization is due to learning more semantically meaningful representation for the discriminator. More specifically, data augmentation will simply treat all real images and their transformed images
7

---

Published as a conference paper at ICLR 2020
0.7
GAY Gooiad Aug
DAN Gr Aag
Gaiclel Cose Rog.
GrA Reg.
0.4 1200
2000 3000
GANAel
4000
0.0
1230
2000
Eoodaa
3000
4330
1200
2320 3003
Epoda
4000
Figure 5: A study of how much data augmentation matters by itself. Three GANs were trained on CIFAR-10. 10: one baseline GAN, one GAN with data augmentation only, and one GAN with consis- tency regularization. (Lefft) Training accuracy of the GAN discriminator. (Middle) Test accuracy of the GAN discriminator on the held out test set. The accuracy is low for the baseline GAN, which in- dicates it suffered from over-fitting The accuracy for the other two is basically indistinguishable for each other. This suggests that augmentation by itself is enough to reduce discriminator over- fittingg,,,, and that consistency regularization by itself does little to address over-fitting. (Right) FID scores of the three settings. The score for the GAN with only augmentation is not any better than the score for the baseline, even though its discriminator is not over-fitting. The score for the GAN with consis- tency regularization is better than both of the others, suggesting that the consistency regularization acts on the score through some mechanism other than by reducing discriminator over-fitting.
Metric Gaussian Noise FID
Random shift & flip
Cutout
Cutout w/ random shift & flip
21.91
16.04.1..17
17.10.0.29
19.46:10.26
FID scores on CIFAR-10 10 for different types of image augmentation. Gaussian noise is the worst, and random shift and flip is the best, consistent with general consensus on the best way to perform image optimization on CIFAR-10 (Zagoruyko & Komodakis 2016).
Table 3:
with the same label as real without considering semanties, whereas our consistency regularization further enforces learning implicit manifold structure in the discriminator that pulls semantically similar images (i.e., original real image and the transformed image) to closer the discriminator representation space.
4.2 How DOES THE TYPE OF AUGMENTATION AFFECT RESULTS?
To analyze how different types of data augmentation affect our results, we conduct an ablation study on the CIFAR-10 dataset comparing the results of using four different types of image augmentation: (1) adding Gaussian noise to the image in pixel-space, (2) randomly shifting the image by a few pixels and randomly flipping it horizontally, (3) applying cutout Vries & Taylor 2017) trans- formations to the image, and (4) cutout and random shifting and flipping. As shown in Table 3 random flipping and shifting without cutout gives the best results (FID 16.04) among all four meth- ods. Adding Gaussian noise in pixel-space gives the worst results. This result empirically suggests that adding Gaussian noise is not good semantic preserving transformation the image manifold. It's 's also noteworthy that the most extensive augmentation (random flipping and shifting with cutout) did not perform the best. One possible reason is that the generator sometimes also generates samples with augmented artifacts (e.g., cutout). If such artifacts do not exist in the real dataset, it might lead to worse FID performance.
CONCLUSION
In this paper, we propose simple, effective, and computationally cheap method consistency reg- ularization to improve the performance of GANs. Consistency regularization is compatible with spectral normalization and results in improvements in all the many contexts in which we evaluated it. Moreover, we have demonstrated consistency regularization is more effective than other regular- ization methods under different loss functions, neural architectures and optimizer settings. We have also shown simply applying consistency regularization on top of state-of the-art GAN models can further greatly boost the performance. Finally, we have conducted a thorough study on the design choices and parameees of consistency regularization.
8

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
We thank Colin Raffel for feedback on drafts of this article. We also thank Marvin Ritter, Michael Tschannen and Mario Lucic for answering our questions of using compare GAN codebase for large scale GAN evaluation.
REFERENCES
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. In ICLR, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein GAN. arXiv:1701.07875. 2017.
Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. In NeurlPS, 2014.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Paperno, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. arXiv:/905.02249, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint 1xXiv:/809.11096 2018.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In CVPR, 2019.
Terrance DeVries and Graham W Taylor. Improved regularization convolutional neural networks 2017.
with cutout. arXiv preprint
lan J. Goodfellow, Jean Pouget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Ishaan Guirajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Im- proved training of wasserstein GANs. In NewrIPS, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
Weihua Hu, Takeru Miyato, Seiya Tokui, Eiichi Matsumoto, and Masashi Sugiyama. Learning discrete representations via information maximizing self-augmented training. ICML, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for improved quality, stability, and variation. In ICLR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. CVPR, 2019.
Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. On convergence and stability of gans. arXiv preprint arXiv:/705.001 705.00725, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. A large-scale study on regularization and normalization in gans. In ICML, 2019.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised leariin. arXiv preprint arXiv:1610.02242, 2016.
Jae Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv: 1705.02944, 2017.

---

Published as a conference paper at ICLR 2020
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? A large-scale study. In NeurlPS, 2018.
Takeru Miyato and Masanori Koyama. cGANS with projection discriminator. In ICLR, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018a.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions pattern analysis and machine intelligence, 2018b.
Augustus Odena. Open questions about generative adversarial networks. Distill, 2019. doi: 10. 23915/distill.00015
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In /CML, 2017.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and lan Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In NeurIPS, pp. 3235-3246, 2018.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016.
Kevin Roth, Aurélien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In NeurlPS, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. IJCV, 115(3):211-255 2015.
Mehdi Sajaadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transforma- tions and perturbations for deep semi-supervised learning. In NewrlPS, 2016.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurlPS, 2016.
Dustin Tran, Rajesh Ranganath, and David M. Blei. Deep and hierarchical implicit models. arXiv:1702.08896, 2017.
Xiang Wei, Boqing Gong. Zixia Liu, Wei Lu, and Liqiang Wang. Improving the improved training of wasserstein gans: A consistency term and its dual effect. In ICLR, 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904. 12848. 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. s Self-supervised semi- supervised learning, arXiv preprint arXiv:1905.03670 2019.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. StackGAN: Text to photo-realistic image synthesis with stacked generative adversarial networks. In ICCV, 2017.
Han Zhang, Ian J. Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In ICML, 2019.
Zizhao Zhang, Yuanpu Xie, and Lin Yang. Photographic text-to-image synthesis with hierarchically-nesee adversarial network. In CVPR, 2018
10

---

Published as a conference paper at ICLR 2020
APPENDIX
A HYPERPARAMETER SETTINGS OF OPTIMIZER
Setting
br
52
Naiis
A
0.0001
0.5
0.9
0.0001
0.5
0.999
C
0.0002
0.5
0.999
1
DERD
D
0.0002
0.5
0.999
0.001
0.5
0,9
5
0.001
0.5
0.999
G
0.001
0.9
0.999
5
Table A1: Hlyper-parametet of the optimizer used in our experiments.
Here, similar the experiments in Miyato al. (018a): Kurach et al. (2099)
evaluate all reg-
ularization methods across different hyperparameters settings for (1) learning rate Ir (2) first and second order momentum parameters of Adam 31, 32 (3) number of the updates of the discriminator per generator update, Ndis- The details of all the settings are shown in Table AT Among all these settings, A-D the "good" hyperparameters used in previous publications (Radford al. 2016 Gulrajani et al. 2017 Kurach et al.) 2019): E, F. G are the "aggressive" hyperparameter settings in- troduced by Miyato et al. (2018a) to test model performance under noticeably large learning rate or disruptively high momentum. In practice, we find setting C generally works the best for SNDCGAN and setting D is the optimal setting for ResNet. These two settings are also the default settings in the Compare GAN codebase for the corresponding network architectures.
CIFAR-10
CelebA
50
Setting
20
5
0
W/O
GP
DR
JSR
Ours
W/O
JSR
Ours
Figure A1: Comparison of FID scores with different optimizer settings.
Figure Al displays the FID score all methods with 7 settings A-G. We can observe that con- sistency regularization is fairly robust even for some of the aggressive hyperparameter settings. In general, the proposed consistency regularization can generate better samples with different optimizer settings compared with other regularization methods.

---

Published as a conference paper at ICLR 2020
B COMPARISON OF DIFFERENT REGULARIZATION METHODS WHEN SPECTRAL NORMALIZATION IS NOT USED
CIFAR-10 SNDCGAN
80
SN m False| | loss Hinge
SN
False| loss NS
250
SN False loss WAS
70
150
40
30
50
M/O
GP
DR
15A
Ours
WD
GP
DR (b)
/SA
Our
1/0
GP
DR
J5A
Outs
(c)
CIFAR-10 10 ResNet
Falce| lass
False|
False
WAS
35
200
30
25
300
20
30
15
Ours
16.
w/O
DR
JSR
Ours
JSA
(e)
(1)
CelebA SNDCGAN
False | loss Hinge
SN False loss
False loss WAS
55
30
50-
200
150
30-
25
JSR
Our
V/O
dP
(h)
(0)
Figure B1: Comparison of FID scores when SN is not used.
Here, we compare different regularization methods when spectral normalization (SN) is not used. As shown in Figure B1 our consistency regularization always improves the baseline model (W/O). It also achieves the best FID scores in most of the cases, which demonstrates that consistency reg- ularization does not depend on spectral normalization. By comparing with the results in Figure 2 and Figure we find adding spectral normalization will further boost the results. More importantly, the consistency regularization is only method that improve on top of spectral normalization without exception. The other regularization methods do not have this property.
C TRAINING SPEED
Here we show the actual training speed of discriminator updates for SNDCGAN on CIFAR-10 with NVIDIA Tesla V100. Consistency regularization is around 1.7 times faster than gradient based regularization techniques.
Method Speed (step/s)
W/O
GP
DR
JSR
Ours (CR-GAN)
66.3
29.7
29.8
29.2
51.7
Table C1: Training speed of discriminator updates for SNDCGAN on CIFAR-10.
12

---

Published as a conference paper at ICLR 2020
D GENERATED SAMPLES FOR UNCONDITIONAL IMAGE GENERATION
Real images
Ours (FID:16.97)
DR (FID: 21.91)
JBR(FID: 17)
(FID: 22.57)
WO (FID: 26 25.95)
Figure DI: Comparison of generated samples of CelebA
Real Images
Ours (FID:14.56)
DR (FID:18.94)
W/O (FID:19.00)
JSR (FID: 19.59)
GP(FID: 19.74)
Figure D2: Comparison of generated samples for unconditional image generation on CIFAR-10 with ResNet architecture.
13

---

Published as a conference paper at ICLR 2020
W/O
DR
Ours (CR-GAN)
GP
JSR
Figure D3: Comparison of unconditional generated samples on CIFAR-I0 with a ResNet architec- ture, Wasserstein loss and spectral normalization. This is hard hyperparameter setting where the baseline and previous regularization methods fail to generate reasonable samples. Consistency Reg- ularization is the only regularization method that can generate satisfactory samples in this setting. FID scores are shown in sub-plot (c) of Figure
E GENERATED SAMPLES FOR CONDITIONAL IMAGE GENERATION
CR-BigGAN" (FID: 11.67)
BigGAN" (FID: 20.42)
Figure El: Comparison of generated samples for conditional image generation on CIFAR-10. Each row shows the generated samples of one class.
14

---

Published as a conference paper at ICLR 2020
Figure E2: Comparison of conditionally generated samples of BigGAN and CR-BigGAN* ImageNet. (Left)) Generated samples of CR-BigGAN* (Right) Generated samples of BigGAN*
15

---

Published as a conference paper at ICLR 2020
HOUDD
HLL
Figure E3: More results for conditionally generated samples of BigGAN and CR-BigGAN on ImageNet. (Left) Generated samples of CR-BigGAN* (Right) Generated samples of BigGAN*.
16

---

Published as a conference paper at ICLR 2020
F COMPARISON WITH INCEPTION SCORE
Inception Score (IS) another GAN evaluation metric introduced by Salimans et al. (216)). Here, we compare the Inception Score of the unconditional generated samples on CIFAR-10. As shown in Table FT] Figure FT and Figure F2 consistency regularization achieves the best IS result with both SNDCGAN and ResNet architectures.
Setting
W/O
GP
DR
JSR
Ours (CR-GAN)
CIFAR-IO (SNDCGAN)
7.54
7.54
7.54
7.52
7.93
CIFAR-10 (ResNet)
8.20
8.04
8.09
8.03
8.40
Table Fl: Best Inception Score for unconditional image generation on CIFAR-10.
7.5
SN Falso| loss
Hingo
SN Falso looss NS
SN False loss WAS
7.0
6.0
7.3
5.5
7.2-
5.0
7.1
4.5
7.0
W/O
6P
DR
J5R
Ours
W/0
GP
DR
JSR
Ours
w/0
06
JSR
Ours
8.0
Hinge
True
8.0
True loss WAS
8.0
7.5
7.9
7.9
7.0
7.8
7.8
6.5
7.7
6.0
7.7
7.5
5.5
7.4
4.5
7.5
7.3
7.4
7.2
3.5
W/O
DR
ISR
DK
JSR
Js
Ours
Figure F1: Comparison of IS with a SNDCGAN architecture on different loss settings. Models are trained on CIFAR-10.
Hinge
8.8
SN
WAS
8.2
5.6
8.4
8.0-
8.2
18
7.8
7.4-
7.4
7.2
wio
DR
JSR
Curs
7.2
JSR
Ours
2
Wio
Hinge
8.4
loss
6.2
8.1
8.0
8.0
7.8
7.7
7.4
7.6
7.2
wi
JSR
JSht
Figure F2: Comparison of IS with a ResNet architecture on different loss settings. Models are trained on CIFAR-10.
17

---

Published as a conference paper at ICLR 2020
G EFFECT OF THE NUMBER OF LAYERS REGULARIZED IN DISCRIMINATOR
Here, we examine the effect of the number of layers regularized in discriminator. In this experiment, SNDCGAN architecture with NS loss on the CIFAR-IO dataset. There are intermediate layers in the discriminator. To start, we add consistency only to the last layer (0 intermediate lay- ers). Then we gradually enforce consistency for more intermediate layers. We use two weighting variations to combine the consistency loss across different layers. In the first setting, the weight of each layer is the inverse feature dimension d, in that layer, which corresponds X; 1/d; in Equation In the second setting, we give equal weight to each layer, which corresponds to A = 1. The results for both settings are shown in Figure GT] In both settings, we observe that consistency regularization on the final layer achieves reasonably good results. Adding the consistency to first few layers the discriminator harms the performance. For simplicity, we only add consistency regularization in the final layer of the discriminator in the rest of our experiments.
25
500
400
300
(a)
(b)
Figure G1: Comparison consistency regularization on different number of intermediate layers: (a) first weight setting, where the weight for each layer is the inverse of its feature dimension (b) second weight setting, where each layer has equal weight.
H CONSISTENCY REGULARIZATION ON THE GENERATED SAMPLES
In this section, we investigate the effect of adding consistency regularization for the generated sam- ples. We compare four settings, no consistency regularization (W/O), regularization only on the real samples (CR-Real), consistency regularization only on the fake samples produced by the genera- tor (CR-Fake) and regularization on both real and fake samples (CR-All). CR-Real presented in Algorithm D CR-Fake has similar computational cost CR-Real and CR-All doubles the compu- tational cost, since both the augmented real and fakes samples need to be fed into the discriminator to calculate the consistency loss. As shown in Figure HT CR-Real, CR-Fake and CR-All are always better than the baseline without consistency regularization. In addition, CR-Real is consistently better than CR-Fake. It is interesting note that CR-All is not always better than CR-real given the extra computational costs and stronger regularization. For example, CR-All improves FID from 20.21 of CR-Real to 15.51 for SNDCGAN, but also gives slightly worse results for ResNet (14.93 vs 15.07) and for CR-BigGAN* (1 (11.48 vs 12.51). We observe that enforcing additional consistency on the generated samples gives more performance gain when the model capacity is small and that gain decreases when model capacity increases. For computational efficiency and simplicity of the training algorithm, we use consistency regularization on real samples for the rest of our experiments.
18

---

Published as a conference paper at ICLR 2020
30
20
25
15-
15
10
5
W/O
CR-Real
CR-Fake
CR-AIl
W/O
CR-Real
CR-Fake
CR.AIl
(b)
30
15
10
5-
W/O
CR-Real
CR-Fake
CR-AIl
Figure HI: Comparison of FID scores with no consistency regularization (W/O), regularization only on the real samples (CR-Real), consistency regularization only on the fake samples produced by the generator (CR- Fake) and regularization on both real and fake samples (CR-All) for (a) unconditional image generation on CIFAR-10 10 with SNDCGAN, (b) unconditional image generation CIFAR-10 with ResNet, (c) conditional image generation on CIFAR- 10 with CR-Bi CR-BigGAN
19


Reviews:
[{"review_id": "rkli38WvcH", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper presents a new regularization technique termed consistency regularization for training GANs. The idea is the following: the authors propose to penalize the sensitivity of the last layer of the discriminator to augmented images. This idea is simple yet efficient: it is easy to implement, a regularization term is gradient-free, and its computation is up to 1.8 times faster than standard gradient-based regularization techniques. The authors tested different augmentation techniques and concluded that simple ones behave better (e.g., shifting and flipping). The experimental results show an impressive gain in FID measure, renewing the current state-of-the-art score for class conditional image generation on CIFAR-10 dataset. \n\nPros: \nThe proposed technique is very simple and intuitive; it easy to implement, and it is computationally cheap. The experiments were held for three runs with different random seeds, supporting its consistency.  The paper is overall clearly written and easy to understand. \n\nCons:\nThe reported experimental results are held only for BigGAN architecture while not considering different networks to ensure the stability of the proposed regularization. Also, the paper would benefit from a clear experiment description on CelebA dataset (e.g., adding the results to Table 1). \n\nQuestions:\n-Have you tried other transforms, which potentially keep images on the manifold, including zoom, resize, rotation, brightness adjustment, etc.?\n-How the number of layers in $L_{cs}$ (formulas 2-3) affects FID? \n-Have you considered an unconditional setting? \n\nMinor comments: \n-It would be more convenient if the authors explicitly numerate subplots; e.g., in Figure 2, it is confusing to refer the subplots labeled by (a)-(f) as written in caption. \n-Additionally, it would be nice to include say best FID scores over different loss functions (from Figure 2) to Table 1.\n- In section 4.3, you wrote that you tried different $\\lambda$ values:  {0,1,10, 100}, but Figure 4 does not cover all of them. \n- It would be nice to add implementation details (e.g., optimizer, learning rate parameters, steps per discriminator, etc.) for better reproducibility.\n-The paper would benefit from illustrations of generated samples.\n-Please check the spelling of the penultimate article name in references (Zhai et al., 2019). \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJg_aihN9S", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use Consistency Regularization for training GANs, a technique known to work well in unsupervised learning. The technique consists in applying a transformation to real images and enforcing that the features of the discriminator between the transformed inputs and the original inputs are similar. The author show that using this technique enables them to improve the performance of a standard GAN significantly on CIFAR10. They also carry an ablation study studying the influence of the different part of the proposed technique.\n\nOverall I'm in favor of accepting this paper. The paper is well written, with convincing experiments and an interesting ablation study. However I have several minor issues that I think could greatly improve the paper if addressed.\n\nMinor comments:\n- I think an idea which is somewhat related but hasn't been mentioned in the paper, is the idea of adding noise to the input when training GANs [1]. I think this is worth mentioning in the related work.\n\n- Related to the previous point, why penalizing features and not directly output ? What about also trying to classify the transformed images as real ? Also you say that penalizing the last layer, I think including the influence of m (eq 2) in the ablation study would be interesting.\n\n- The authors provide some measure of standard deviation on some experiments but not on all of them. It would be nice to systematically report the standard deviation for every experiments.\n\n- In figure 1 the author make the hypothesis that the discriminator will output very different score to images semantically close together. Did the author verify this hypothesis experimentally ?\n\n- Also why penalizing only the samples from the real distribution and not from the generator ? have you tried both ?\n\n- When the test accuracy of the discriminator is low, it could also be that the discriminator is under-fitting, it would be nice to also report the train accuracy for the discriminator.\n\n- I think the conclusion about the effect of consistency regularization vs data augmentation is a bit vague since consistency regularization has no sense without data-augmentation. \n\n- It's quite interesting but also disappointing that combining transformations doesn't give that much of an improvement. Do the author have any intuition why this is the case ? and why learning them one after the other would work ?\n\nReferences:\n[1] Arjovsky and Bottou. \"Towards Principled Methods for Training Generative Adversarial Networks.\" (ICLR 2017)", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1eKjX745B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The topic of this paper is out of the reviewer's domain (Bayesian optimization, RL, and neuroscience). The reviewer has been reviewing ICLR for several years. Such mismatches had not happened in the past.\n\nThe reviewer doesn't think this paper reached the bar of a good ICLR paper but hesitates to reject.\n\n\nThis work proposed a training stabilizer for GANs based on the notion of Consistency Regularization. Experimentally, the authors had augmented data passed into the GAN discriminator and penalize the sensitivity of the ultimate layer of the discriminator to these augmentations.\n\nThe authors claimed \"We conduct a series of ablation studies to demonstrate that the\nconsistency regularization is compatible with various GAN architectures and loss\nfunctions. Moreover, the proposed simple regularization can consistently improve\nthese different GANs variants significantly. \"\n\n\n\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a simple and effective way to stabilize training by adding consistency term to discriminator. Given the stochastic augmentation procedure $T(x)$ the loss is just a penalty on $D$. The main unsolved question why it help to make discriminator "smoother" in the consistency case for a standard GAN (since typically, no constraints are enforced). Nevertheless, at the moment this a working heuristics that gives new SOTA, and that is the main strength. The reviewer all agree to accept, and so do I.
Decision: Accept (Poster)