Training GANs with Stronger Augmentations via Contrastive Discriminator
Authors: Jongheon Jeong
Jinwoo Shin
Venue: ICLR 2021 Poster (2021)
Keywords: generative adversarial networks
contrastive learning
data augmentation
visual representation learning
unsupervised learning

Abstract: Recent works in Generative Adversarial Networks (GANs) are actively revisiting various data augmentation techniques as an effective way to prevent discriminator overfitting. It is still unclear, however, that which augmentations could actually improve GANs, and in particular, how to apply a wider range of augmentations in training. In this paper, we propose a novel way to address these questions by incorporating a recent contrastive representation learning scheme into the GAN discriminator, coined ContraD. This "fusion" enables the discriminators to work with much stronger augmentations without increasing their training instability, thereby preventing the discriminator overfitting issue in GANs more effectively. Even better, we observe that the contrastive learning itself also benefits from our GAN training, i.e., by maintaining discriminative features between real and fake samples, suggesting a strong coherence between the two worlds: good contrastive representations are also good for GAN discriminators, and vice versa. Our experimental results show that GANs with ContraD consistently improve FID and IS compared to other recent techniques incorporating data augmentations, still maintaining highly discriminative features in the discriminator in terms of the linear evaluation. Finally, as a byproduct, we also show that our GANs trained in an unsupervised manner (without labels) can induce many conditional generative models via a simple latent sampling, leveraging the learned features of ContraD. Code is available at https://github.com/jh-jeong/ContraD.

Published as a conference paper at ICLR 2021
TRAINING GANS WITH STRONGER AUGMENTATIONS VIA CONTRASTIVE DISCRIMINATOR
Jongheon Jeong' & Jinwoo Shin
School of Electrical Engineering *Graduate School of Al Korea Advanced Institute of Science and Technology (KAIST)
Daejeon 34141, South Korea
{jongheonj, jinwoos] @kaist. ac.
ABSTRACT
Recent works in Generative Adversarial Networks (GANs) are actively revisiting various data augmentation techniques as an effective way to prevent discriminator overfitting. It is still unclear, however, that which augmentations could actually improve GANs, and in particular, how to apply a wider range of augmentations in training. In this paper, we propose novel way to address these questions by incorporating a recent contrastive representation learning scheme into the GAN discriminator, coined ContraD. This "fusion" enables the discriminators to work with much stronger augmentations without increasing their training instability, thereby preventing the discriminator overfitting issue GANs more effectively. Even better, we observe that the contrastive learning itself also benefits from our GAN training, i... by maintaining discriminative features between real and fake samples, suggesting strong coherence between the two worlds: good contrastive representations are also good for GAN discriminators, and vice versa. Our ex- perimental results show that GANs with ContraD consistently improve FID and IS compared to other recent techniques incorporating data augmentations, still maintaining highly discriminative features in the discriminator in terms of the lin- ear evaluation. Finally, as a byproduct, we also show that our GANs trained in unsupervised manner (without labels) can induce many conditional generative models via simple latent sampling, leveraging the learned features of ContraD. Code available at https: //g /gi thub. com/ jh- jen-jjenn/Coora....
INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al., 2014) have become one of the most prominent approaches for generative modeling with a wide range of applications (Ho Ermon, 2016; Zhu et al., 2017; Karras et al., 2019; Rott Shaham et al., 2019). In general, GAN is defined by a minimax game between two neural networks: a generator network that maps a random vector into the data domain, and discriminator network that classifies whether given sample is real (from the training dataset) or fake (from the generator). Provided that both generator and discriminator attain their optima at each minimax objective alternatively, is theoretically guaranteed that the generator implicitly converges to model the data generating distribution (Goodfellow et al., 2014).
Due to the non-convee/stationaayy nature of the minimax game, however, training GANs in practice is often very unstable with an extreme sensitivity to many hyperparameters (Salimans et al., 2016; Lucic et al., 2018; Kurach et al., 2019). Stabilizing the GAN dynamics has been extensively studied in the literature (Arjovsky et al., 2017; Gulrajani et al., 2017; Miyato et al., 2018; Wei et al., 2018; Jolicoeur-No Marttinea,,, 2019; Chen et al., 2019; Schonfeld et al., 2020), and the idea of incorporat- ing data augmentation techniques has recently gained particular attention on this line of research: more specifically, Zhang et al. (2020) have shown that consistency regularization between discrimi- nator outputs of clean and augmented samples could greatly stabilize GAN training, and Zhao et al. (2020c) further improved this idea. The question of which augmentations are good for GANs has been investigated very recently in several works (Zhao et al., 2020d: Tran et al., 2021; Karras al., 2020a; Zhao et al., 2020a), while they unanimously conclude only a limited range augmentations (e.g., flipping and spatial translation) were actually helpful for the current form of training GANs.

---

Published as a conference paper at ICLR 2021
Meanwhile, not only for GANs, data augmentation has also been played key role in the literature of self-supervised representation learning (Doersch et al., 2015; Gidaris et al., 2018; Wu et al., 2012) 8), especially with the recent advances in contrastive learning (Bachman et al., 2019; Oord et al., 2018; Chen et al., 2020a;b; Grill et al., 2020): egg., Chen et al. (2020a) have shown that the performance gap between supervised- and unsupervised learning can be significantly closed with large-scale con- trastive learning over strong data augmentations. In this case, contrastive learning aims to extract the mutual information shared across augmentations, so good augmentations for contrastive learn- ing should keep information relevant to downstream tasks (e-g.. classification), while discarding nuisances for generalization. Finding such augmentations is still challenging, yet in some sense, it is more tangible than the case of GANs, as there are some known ways to formulate the goal rigourously, e-g., InfoMax (Linsker, 1988) or InfoMin principles (Tian et al., 2020).
Contribution. In this paper, we propose Contrastive Discriminator (Contral)), a new way of train- ing discriminators of GAN that incorporates the principle of contrastive learning. Specifically, in- stead of directly optimizing the discriminator network for the GAN loss, ContraD uses the network mainly to extract a contrastive representation from a given set of data augmentations and (real or generated) samples. The actual discriminator that minimizes the GAN loss is defined independently upon the contrastive representation, which turns out that a simple 2-layer network is sufficient to work a complete GAN. By design, ContraD can be naturally trained with augmentations used in the literature of contrastive learning, e-g., those proposed by SimCLR (Chen et al., 2020a), which are in fact much stronger than typical practices in the context of GAN training (Zhang et al., 2020; Zhao et al., 2020e;a; Karras et al., 2020a). Our key observation here is that, the task of contrastive learning (to discriminate each of independent real samples) and that of GAN discriminator (to discriminate fake samples from the reals) benefit each other when jointly trained with shared representation.
self-super learning. including contrastive learning, have been recently applied in GAN as an auxiliary task upon the GAN loss (Chen et al., 2019; Tran et al., 2019; Lee et al., 2021; Zhao et al., 2020d), mainly in attempt to alleviate catastopic forgetting in discriminators (Chen et al., 2019). For conditional GANs, Kang& & Park (2020) have proposed a contrastive form of loss to efficiently incorporate a given conditional information into discriminators. Our work can be differentiated to these prior works in sense that, to the best of our knowledge, it is the first method that success- fully leverage contrastive learning alone to incorporate a wide range of data augmentations in GAN training. Indeed, for example, Zhao et al. (2020d) recently reported that simply regularizing auxil- iary SimCLR loss (Chen et al., 2020a) improves GAN training, but could not outperform existing methods based on simple data augmentations, egg., bCR (Zhao et al., 2020c).
BACKGROUND
Generative adversarial networks. We consider a problem of learning generative model Pg from given dataset [.,,.V where 2; Paaa and x; E X. To this end, generative adversarial network (GAN) (Goodfellow et al., 2014) considers two neural networks: (a) a generator network G Z -> X' that maps latent variable
p(z) into X, where p(z) is specific prior distribution, and (b) a discriminator network D : X' + [0, 1] that discriminates samples from Pdata and those from the implicit distribution Pa derived from G(z). The primitive form of training G and D is the following:
min max (G, D) = E Ex- (xx)) + E (arppte)oog(( (1- ((G())))
(1)
D
For fixed G. the inner maximization objective (1) with respect to D leads to the following optimal discriminator Dū, and consequently the outer minimization objective with respect to G becomes to minimize the Although this formulation (1) theoretically guarantees P Pasa as the global optimum, the non- saturating loss (Goodfellow et al., 2014) is more favored in practice for better optimization stability:
divergence between Pdata and py: Dū maxp V(G,
pan Pr
max LDD == VVG, D), and min L(G) == EE -:o:===-(((DD Eollo((((((())) D>
(2)
G
Here, compared to (1), G is now optimized to let D to classify G(z) as 1, i.e., the "real".
Contrastive representation learning. Consider two random variables (1) and ((2), which are often referred as views. Generally speaking, contrastive learning aims to extract a useful repre- sentation of (1) and (2) from learning a function that identifies whether a given sample is from
2

---

Published as a conference paper at ICLR 2021
p(v(())(((()) i.e. whether two views are dependent or not. More specif- ically, the function estimates the mutual information v(2)) between the two views. To this end, Oord et al. (2018) proposed to minimize InfoNCE loss, which turns out to maximize a lower bound of I(v(1); (()).. Formally, for given v. p(v(1)) and vf p(v(2)/o))) |o!" while assum-
(pv())((v()) ())) ()
ing of²) ~ p(v(2)) for j = 1, 1..... Kk, the InfoNCE loss is defined by:
exp(s(u,)) 0. (3(
LUCE (v; (1 1 ;))))))))) == ---log log Σ exp(s(u, (1) , ((
(3)
where -) is the score function that models the log-density ratio of p(v(2) (1() to p(v(2), possi- bly including some parametrized encoders for (²) and v
Many of recent unsupervised representation learning methods are based on this general framework of contrastive learning (Wu et al., 2018; Bachman et al., 2019; Hénaff' et al., 2020; He et al., 2020; Chen al., 2020a). In this paper, we focus on the one called SimCLR (Chen al., 2020a), that adopts wide range of independent data augmentations to define views: specifically, for given set of data samples x [x:][=,, SimCLR applys two independent augmentations, namely t and t2, to the given data to obtain vl)) and v(2), i.e., (v(1), v!2)) == (x), (x)). The actual loss of SimCLR slightly different to InfoNCE, mainly due to practical considerations for sample efficiency:
N
( (2()) = 2N
Σ (Ixce(o)") [0(2): Ssincia) + TAxz (2) ; (2]],
SSincLa)
(4)
where v-, == v. {o:}. For SsıaCIa. SimCLR specifies to use (a) an encoder network f X 7 R:de, (b) a small neural network called projection head h : Rdd + Rddd, and (c) the normalized temperature-scaled cross entropy (NT-Xent). Putting altogether, SsincLa is defined by:
SstnCIR (v v (2) (. = M(((()))) : A(f(v(2))
(5)
where is temperature hyperparameter. Once the training is done with respect LSiuOLa. the projection head h is discarded and f is served as the learned representation for downstream tasks.
CONTRAD: CONTRASTIVE DISCRIMINATOR FOR GANS
We aim to develop a training scheme for GANs that is capable to extract more useful information under stronger data augmentation beyond the existing yet limited practices, e.g., random transla- tions up to few pixels (Zhang et al., 2020; Zhao et al., 2020c;a). For examples, one may consider to apply augmentations introduced in SimCLR (Chen et al., 2020a) for training GANs, i.e., stochas- tic composition of random resizing, cropping, horizontal flipping, color distortion, and Gaussian blurring. Indeed, existing approaches that handle data augmentation in GANs not guranteed to work best in this harsh case (as also observed empirically in Table 5), and some of them are even expected to harm the performance: egg., applying consistency regularization (Zhang et al., 2020; Zhao et al., 2020c), which regularizes a discriminator to be invariant to these augmentations, could overly restrict the expressivity of the discriminator learn meaningful features.
Our proposed architecture of Contrastive Discriminator (ContraD) is designed to alleviate such potential risks from using stronger data augmentations by incorporating the contrastive learning scheme of SimCLR (Chen et al., 2020a), which arguably works with those augmentations, and its design practices into the discriminator. More concretely, the main objective of Contral not to minimize the discriminator loss of GAN, but to learn a contrastive representation that is compatible to GAN, This means that the objective does not break the contrastive learning, while the representa- tion still contains sufficient information to discriminate real and fake samples, that a small neural network discriminator is enough to perform its task upon the representation. We describe how to leam such a representation of ContraD in Section 3.1, and how to incorporate this in the actual GAN training in Section 3.2. Finally, Section 3.3 introduces a natural application of ContraD for deriving conditional generative models from unconditionally trained GAN. Figure illustrates ContraD.
3

---

Published as a conference paper at ICLR 2021
augmentation
representation
projection
contrastive
(1)
con
Pdata
(2)
D
Giz)
Vr.
h.
"real"
G
ag
Lain
"fake"
Figure I: An overview of Contrastive Discrininator (ContraD). Overall, the representation of Con- traD is not learned from the discriminator loss (Lais), but from two contrastive losses L and Lean. each is for the real and fake samples, respectively, Here, ag(-) denotes the stop-gradient operation.
3.1 CONTRASTIVE REPRESENTATION FOR GANS
Recall the standard training of GANs, ie., we train G 2 - and D : [0, 1] via optimizing (2) from given dataset {x, } and let T be family of possible transforms for data augmentation, which we assume in this paper to be those used in SimCLR (Chen et al., 2020a). In order to define start by re-defining the scalar-valued D to have vector-valued
the training objective of ContraD, outputs, ie., D X' -> Rd=, to represent an encoder network of contrastive learning. Overall, the encoder network D of ContraD is trained by minimizing two different contrastive losses: (a) the SimCLR loss (4) on the real samples, and (b) the supervised contrastive loss (Khosla et al., 2020) on the fake samples, which will be explained one-by-one in what follows.
Loss for real samples. By default, ContraD is built upon the SimCLR training: in fact, the training becomes equivalent SimCLR if the loss (b) missing, i.e., without considering the fake sam- ples. Here, we attempt to simply follow the contrastive training scheme for real samples to keep open the possibility improve the method by adopting other self-supervised learning methods with data augmentations (Chen et al., 2020b; Grill et al., 2020). More concretely, we first compute two (2) independent views of vta) = t1 (x), (x) for real samples using tı, t2 T, and minimize:
I (D, bx) = IiuCCL (v v
(
(2); DD,h),
(6)
where hr IRd + Rd is projection head for this loss.
Loss for fake samples. Although the representation learned via L (6) may enough, e-g., for discriminating two independent real samples, it does not give any guarantee that this representation would still work for discriminating real and fake samples, which needed as a GAN discrimina- We compensate this by considering auxiliary loss T. that regularizes the encoder to keep necessary information to discriminate real and fake samples. Here, several design choices can be possible: nevertheless,
observe that one should choose the loss deliberately, otherwise the con- trastive representation from L could be significantly affected. For example, we found using the original GAN loss (2) for L could completely negate the effectiveness of ContraD.
In this respect, we propose to use the supervised contrastive loss (Khosla et al., 2020) over fake sam- ples to this end, an extended version of contrastive loss to support supervised learning by allowing more than one view to be positive, so that views of the same label can be attracted to each other in the embedding space. In our case, we assume all the views from fake samples have the same label
against those from real samples. Formally, for each (1) let (2) be subset of (2) that represent
the positive views for v Then, the supervised contrastive loss is defined by:
exp(SsincLa) ()) (.2(2)))
Lgupcoo (v (2) (2)
log
(7)
(2)
Σ, exp( (xsp(Ssinca((e (2)
(2) (2)
4

---

Published as a conference paper at ICLR 2021
Using the notation, we define the ContraD loss for fake samples as follows:
Le (D, [1p) ==
Lisupcon (Ve,é, (Ve, -ee,-15 of of (2 [vr, D.:;DDD,,, h+),
(8)
where again 1r := La (Z))) is a random view of fake samples, and -= V. {oi). Remark that we use an independent projection header h4 Rd + Rd instead of h in (6) for this loss.
There can be other variants for Leo.. as long as (a) it leads D to discriminate real and fake samples, while (b) not compromising the representation from L For example, one can incorporate another view of fake samples to additionally perform SimCLR similarly to L (6). Nevertheless, we found the current design (8) is favorable among such variants considered, both in terms of performance under practice of GANs, and computational efficiency from using only a single view.
To sum up. ContraD learns its contrastive representation by minimizing the following loss:
Leon hy, h+) Lo (D, h2) +
where Acan > 0 a hyperparameter. Nevertheless, we simply use Acon in our experiments.
3.2 TRAINING GANS WITH CONTRAD
The contrastive loss defined in (9) is jointly trained under the standard framework of GAN, e,g., we present in this paper the case of non-saturating loss (2), for training the generator network G. To obtain (scalar) discriminator score needed to define the GAN loss, simply use an additional discriminator head ha Rd - R upon the contrastive representation of D. Here, key difference of ContraD training from other GANs is that ContraD only optimizes the parameters of ha for mini- mizing the GAN loss: in practical situations of using stochastic gradient descent for the training, this can be implemented by stopping gradient before feeding inputs to ha- Therefore, the discriminator loss of ContraD is defined as follows:
(10) where sg()(v;)) and rr- = sg()(v:)). Here, (-)) denotes the sigmoid function, V. and v random views of real and fake samples augmented via T, respectively, and sg(-) is the stop- gradient operation. Combined with the contrastive training loss (9). the joint training loss Lp for single discriminator update the following:
Lase == - Er, (og(((((a((())) - Er, [log(1 (ha(r))))
LD = Leon + Adie L + Aeoa + Aaie
(11)
where Aais 0 is hyperparameter. Again, as like Acon. we simply use dais in our experiments.
Finally, the loss for the generator G can be defined simply like other standard GANs using the diseriminator score, except that we also augment the generated samples G(zz) to vz, in a similar way to DiffAug (Zhao et al., 2020a) or ADA (Karras et al., 2020a): namely, in case of the non-saturating loss (Goodfellow et al., 2014), we have:
LG == Ev. [og((((h
(12)
Note that the form (12) is equivalent to the original non-saturating loss (2) if we regard ((hh(DD()) : X' [0, 1] single discriminator. Algorithm in Appendix A describes concrete training procedure GANs with ContraD using Adam optimizer (Kingma & Ba, 2014).
3.3 SELF-CONDITIONAL SAMPLING WITH CONTRAD
Apart from the improved training of GANs, the learned contrastive representation of ContraD could offer some additional benefits in practical scenarios compared to the standard GANs. In this section, we present an example of how one could further utilize this additional information derive a con- ditional generative model from an conndditionaly-trainee GAN with ContraD. More specifically, here we consider a variant of discriminator-drrinn incorporate a given representation vector from the ContraD encoder as a conditional information. Originally. DDLS attempts improve the sample quality of a pre-trained GAN via Langevin sampling (Welling & Teh, 2011) 1) on the latent space of the following form:
latent sampling (DDLS) (Che et al., 2020) to
Ze+1 Ze 2
Vn, E(zz) + Ven. ~ N0 1),
(13)
5

---

Published as a conference paper at ICLR 2021
Table 1: Comparison of the best FID score and IS on unconditional image generation of CIFAR-10 and CIFAR-100. Values the rows marked by are from those reported in its reference.
CIFAR-10
CIFAR-100
Architecture
Method
Augment.
FID
FID
f
26.6
7.38
28.5
7.25
(Zhang et al., 2020)
HFlip, Trans
19.5
7.87
22.2
7.91
G: SNDCGAN
bCR (Zhao et al., 2020c)
HFlip. Trans
14.0
8.35
19.2
8.46
D: SNDCGAN
DiffAug (Zhao et al., 2020a)
Trans, CutOut
22.9
7.64
27.0
7.47
ContraD (ours)
SimCLR
10.9
8.78
15.2
9.09
41.3
6.33
52.3
5.24
CR (Zhang al., 2020)
HFlip, Trans
32.1
7.08
36.5
6.55
G: SNDCGAN
bCR (Zhao al., 2020c)
HFlip. Trans
22.8
7.29
7.30
D: SNResNet-18
DiffAug (Zhao et al., 2020a)
Trans, CutOut
59.5
5.62
58.7
5,39
ContraD (ours)
SimCLR
9.86
9.09
15.0
9.56
G: StyleGAN2 StyleGAN2
11.1
9.18
16.5
9.51
DiffAug* (Zhao et al., 2020a)
Trans, CutOut
9.89
9.40
15.2
10.0
ContraD (ours)
SimCLR
9.80
9.47
14.1
10.0
Table 2: Comparison of the best FID score and IS on unconditional image generation of CelebA-
HQ-128 128 with SNDCGAN. We report the mean and standard deviation of best scores across trials.
CelebA-HQ W/O
Hinge
CR
bCR
ContraD (ours)
FID IS f
2..8
26.44132
20.9
19.5
17.1+1..8 2.44
2.06+0.6
+.05
2.11
2.21 +0.10
where E(z) loz) d(G(z)) and d(-) denotes the logit of D. In this manner, for our ContraD, we propose to use the following "conditional" version of energy function E(z, u) for an arbitrary vector v E Rd in the ContraD representation space, called conditional DDLS (cDDLS):
E(z, v) == - log p(z) ha (a(((())) - A(e D(G(Z))).
(14)
In experiments, for instance but not limited show that using the learned weights obtained from linear evaluation as conditional vectors successfully recovers class-conditional generations from an unconditional ContraD model.
4 EXPERIMENTS
We verify the effectiveness of our ContraD in three different aspects: (a) its performance on image generation compared to other techniques for data augmentations in GANs, (b) the quality of repre- sentations learned from ContraD in terms of linear evaluation, and (c) its ability of self-conditional sampling via cDDLS. Overall, we constantly observe integrating contrastive learning into GANs and vice versa have positive effects to each other: (a) for image generation, ContraD enables a simple SNDCGAN outperform a state-of-the-art architecture of StyleGAN2 on CIF1R-10/100; (b) in case of linear evaluation, on the other hand, we observe the representation from ContraD could out- perform those learned from SimCLR. Finally, we perform an ablation study to further understand each of the components we propose. We provide the detailed specification on the experimental setups, egg., architectures, training configurations and hyperparameters Appendix F.
We consider variety of datasets including CIFAR- 10/100 (Krizhevsky, 2009), CelebA-HQ-12 (Lee et al., 2020), AFHQ (Choi et al., 2020) and ImageNet (Russakovsky et al., 2015) our ex- periments, mainly with three well-known GAN architectures: SNDCGAN (Miyato et al., 2018), StyleGAN2 (Karras et al., 2020b) and BigGAN (Brock et al., 2019).' We measure Fréchet Incep- tion distance (FID) (Heusel et al., 2017) and Inception score (IS) (Salimans al., 2016) as quanti- tative metrics to evaluate generation quality.² We compute FIDs between the rest set and generated samples of the same size. We follow the best hyperparameter practices explored by Kurach et al. (2019) for SNDCGAN models on CIFAR and CelebA-HQ-1 HQ8 datasets. For StyleGAN2, on the
Results on AFHQ and ImageNet can be found in Appendix B and C, respectively. We use the official Inception V3 model in TensorFlow compute both FID and IS in experiments.

---

Published as a conference paper at ICLR 2021
Table 3: Comparison of classification accuracy under linear evaluation protocol CIFAR-10 and CIFAR-100. We report the mean and standard deviation across 3 runs of the evaluation.
Dataset
Training
SNDCGAN
SNResNet-18
StyleGAN2
SimCLR (Aeon
Adis
0)
12.
80.3
86,2
CIFAR-10
ContraD (ours)
77.5
85.72000
88.6
SimCLR (Aean
Adia
0)
30.8
41.2+00.6
61.1 60.06
CIFAR-100
ContraD (ours)
37.41006
51.1
68.1 10007
other hand, we follow the training details of Zhao et al. (2020a) in their CIFAR experiments. All the training details are shared across methods per experiment, but the choice of batch size for training ContraD: we generally observe that ContraD greatly benefits from using larger batch (Appendix E), similarly to SimCLR (Chen et al., 2020a), but in contrast to the common practice of GAN training. Therefore, in our experiments, consider to use 2x to 8x larger batch sizes for training ContraD, where the details are specified in Appendix F.
4.1 RESULTS
Image generation on CIFAR-10/100. CIFAR-I0 and CIFAR-100 (Krizhevsky, 2009) consist of 60K images of size 32 x 32 in 10 and 100 classes, respectively, 50K for training and 10K for testing. We start by comparing ContraD on CIFAR-10/100 with three recent methods that applies data aug- mentation in GANs: (a) Consistency Regularization (CR) (Zhang et al., 2020), (b) the "balanced" Consistency Regularization (bCR) (Zhao et al., 2020c), and (c) Differentiable Augmentation (Dif- fAug) (Zhao et al., 2020a) as baselines." We follow their best pratices which data augmentations to use: specifically, we use a combination of horizontal flipping (HFlip) and random translation up to 4 pixels (Trans) for CR and bCR, and Trans + CutOut (De Vries & Taylor, 2017) for DiffAug for CIFAR datasets. We mainly consider SNDCGAN and StyleGAN2 for training. but in addition, further consider scenario that the discriminator architecture of SNDCGAN is scaled up into SNResNet-18, which is identical to ResNet- 18 (He et al., 2016) without batch normalization (loffe & Szegedy, 2015) but with spectral normalization (Miyato et al., 2018), in order see that how each training method behaves when the capacity of generator and discriminator are significantly different.
Table shows the results: overall, we observe our ContraD uniformly improves the baseline GAN training by a large margin, significantly outperforming other baselines as well. Indeed, our results on SNDCGAN already achieve better FID than those from the baseline training of StyleGAN2. More- over, when the discriminator of SNDCGAN is increased to SNResNet-18, ContraD could further is partieularly remark-
improve its results, without even modifying any of the hyperparameters. able that all the other baselines could not improve their results in this setup: A further ablation study presented in Section 4.2 reveals that stopping gradients before the discriminator head in our design is an important factor to benefit from SNResNet-18. 18. Finally, our ContraD can also be applied to larger, sophisticated StyleGAN2, further improving both FID and IS compared to DiffAug.
Image generation CelebA-HQ-128. We also perform experiments on CelebA-HQ-122 (Lee et al., 2020), which consist of 30K human faces of size 128 x 128, to see that our design of ContraD still applies well for high-resolution datasets. We split 3K images out of CelebA-HQ- 128 for testing, and use the remaining for training. We use SNDCGAN for this experiment, and compare FID and IS to (a) the default non-saturating loss (W/O) (Goodfellow et al., 2014), (b) the hinge loss (Hinge) (Lim & Ye, 2017; Tran et al., 2017), (c) CR and (d) bCR. Similarly to the CIFAR experiments, we use the "HFlip+ " augmentation for both CR and bCR, but for CelebA-HQQ-128 we allow them to translate up to 16 pixels, following (Zhang et al., 2020). Again, as shown in Table 2, we confirm that ContraD could still improve training of GANs compared to other methods, even with this high-resolution, yet with limited samples and a straightforward choice of architecture.
Linear evaluation. Recall that the training objective of ContraD we propose in (11) can be simply reduced into the SimCLR loss (Chen et al., 2020a) without considering the fake samples, i.e., by letting Acon Aasa = 0. A natural question from here is whether those fake samples could help to
$ 'Here, we notice that ADA (Karras et al., 2020a). a concurrent work to DiffAug, also offers a highly similar method to DiffAug, but with an additional heuristic of tuning augmentations on-the-fly with validation dataset. We consider this addition is orthogonal to ours, and focus comparing DiffAug for elearer comparison in the CIFAR experiments. more direct comparison with ADA, nevertheless, can be found in Appendix B.
7

---

Published as a conference paper at ICLR 2021
Table 4: Comparison of FID (lower is better) from the class-wise subsets of CIFAR-10 test set K images per class). "Random" indicates unconditional generation, and "Train- "K" indicates 1K random subsamples from the CIFAR-I0 train set of the same class.
D Arch.
Sampling
Plane
Car
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Mean
Random
110.6
125.9
103.8
104.4
107 0
121.0
134.1
120.1
128.8
130.0
118.6
SNDCGAN
+ cDDLS
65,3
72.1
63,9
69.4
57.5
69.0
107.0
61.5
58.2
53.6
67.8
SNResNet-18
Random
117.6
136.3
100.5
99.6
98.6
115.0
128.2
111.7
140.9
140.3
118.9
+ cDDLS
67.1
65.8
59.8
63.8
59.7
61.9
68.8
59.9
58.9
51.6
61.7
Train- IK
38.9
28.4
41.4
49.7
36.4
40.2
41.7
33.6
30.7
24.6
36.6
Comparison of the best FID and IS on CIFAR-10 with SND- CGAN when stronger (bCR and DiffAug) or weaker augmentations
Table 5:
Table 6:
Comparison of the best FID score and IS on CIFAR-10 10 for ablations of our proposed components. All the models are trained with batch size 256. For the ablation of "MLP ha", we replace ha with a linear model. "sg(-)" indicates the use of stop-gradient operation.
(ContraD) are used for each method.
Method
Augment.
FID 4.
18
D: SNDCGAN
D: SNResNet-18
bCR
HFlip. Trans
14.0
8.35
MLP ha $g()
Loan
Lon
FID
IS
FID J
IS
Aug.
SimCLR
20.6
7.44
V
/
11.1
8.62
10.6
8.99
DiffAug
Aug.
Trans, CutOut
SimCLR
22,9
21.9
7.42
7.64
x
/
x
/
/
/
/
11.6
185
3.43
8.61
28.0
274
2.09
7.52
ContraD
SimCLR
10.9
8.78
/
x
/
11.9
8.45
182
2.02
Aug.
HRlipp Trans
13.7
8.54
/ /
x
1.93
232
2.56
improve the representation of SimCLR: we verify this by comparing the linear evaluation perfor- mance (Kolesnikov al., 2019; Chen et al., 2020a; Grill et al., 2020) of the learned representation of ContraD to those from SimCLR: specifically, linear evaluation measures the test accuracy that linear classifier the top of given (frozen) representation could maximally achieve for given (downstream) labeled data. Following this protocol, we compare the linear evaluation of a Con- traD model with its ablation that Acon and Aaie are set to 0 during its training. We consider three discriminator architectures and two datasets for this experiments: namely, we train SNDCGAN, SNResNet- 18, and StyleGAN2 models on CIFAR-10 and CIFAR-100. In this experiment, we use batch size 256 for training all the SNDCGAN and SNResNet- 18 models, while in case of Style- GAN2 we use batch size 64 instead. Table summarizes the results, and somewhat interestingly, it shows that ContraD significantly improves the SimCLR counterpart in terms of linear evaluation for all the models tested: other words, we observe that keeping discriminative features between real and fake samples in their representation, which is what ContraD does in addition to SimCLR, can be beneficial to improve the contrastive learning itself. Again, our observation supports a great potential of ContraD not only in the context of GANs, but also in the contrastive learning.
Conditional DDLS with ContraD. We further evaluate the performance of conditional DDLS (cD- DLS) (14) proposed for ContraD, with task of deriving given (unconditionally--taia -rriinne GAN model. Here, we measure the performance of this conditional generation by computing the class-wise FIDs to the test set, i.e., FIDs computed for each subsets of the same class in the test set. We test cDDLS on SNDCGAN and SNResNet-18 18 ContraD models trained on CIFAR-10. 10. To perform conditional generation for a specific class, we simply take learned linear weight obtained from the linear evaluation protocol for that class. The results summa- rized in Table show that applying eDDLS upon unconditional generation significantly improves class- wise FIDs for all the classes in CIFAR-10. Some of the actual samples conditionally generated from an SNResNet-18 ContraD model can be found in Figure of Appendix D.
generative models for
4.2 ABLATION STUDY
We conduct an ablation study for a more detailed analysis on the proposed method. For this section, unless otherwise specified, we perform experiments CIFAR-10 10 with SNDCGAN architecture.
Stronger augmentations for other methods. One of the key characteristics of ContraD compared to the prior works is the use of stronger data augmentations for training GANs, e.g., we use the augmentation pipeline of SimCLR (Chen et al., 2020a) in our experiments. In Table 5, we examine the performance bCR (Zhao al., 2020c) and DiffAug (Zhao et al., 2020a) when this stronger
8

---

Published as a conference paper at ICLR 2021
augmentation is applied, and shows that the SimCLR augmentation could not meaningfully improve FID or IS for both methods. Unlike these methods, our ContraD effectively handles the SimCLR augmentations without overly regularizing the discriminator, possibly leveraging many components of contrastive learning: e.g. the normalized loss, or the use of separate projection heads.
Weaker augmentations for ContraD. On the other side, also consider case when Contral is rather trained with a weaker data augmentation, here we consider "HFlip, Trans" instead of "Sim- CLR", in Table 5: even in this case, we observe ContraD is still as good (or better in terms of IS) a strong baseline of bCR. The degradation in FID (and IS) compared to "ContraD + SimCLR" possibly due to that SimCLR requires strong augmentations learn good representation (Chen et al., 2020a), e.g., we also observe there is degradation in the linear evaluation performances, 77.5 + 72.9, when "HFlip, Trans" is used.
MLP discriminator head. In our experiments, we use 2-layer network for the discriminator head ha- As shown Table 6, this choice can be minimal: replacing ha with linear model severely breaks the training. Conversely, although not presented here, we have also observed that MLPs of more than two layers neither give significant gains on the performance in our experimental setup.
Stopping gradients. We use the stop-gradient operation sg(-) before the discriminator head ha in attempt to decouple the effect of GAN loss to the Contral representation. Table 6 also reports the ablation of this component: overall, not using sg(-) does not completely break the training, but it does degrade the performance especially when a deeper discriminator is used, e.g., SNResNet-18 18.
Contrastive losses. Recall that the representation of ContraD is trained by two losses, namely L and Leon (Section 3. From an ablation study on each loss, as presented in Table 6, we observe that both of the losses are indispensable for training work in a larger discriminator architecture.
Separate projection headers. As mentioned in Section 3.1, we use two independent projection headers, namely hr and he, define I.o00 and L respectively. We have observed this choice is empirically more stable than sharing them: e.g., under h = hg, we could not obtain a reasonable performance when only D is increased to SNResNet-18 on CIFAR-10 with SNDCGAN. Intuitively, an optimal embedding (after projection) from Leon can be harmful to SimCLR, as it encourages the embedding of real samples to degenerate into a single point, i.e., the opposite direction of SimCLR.
5 CONCLUSION AND FUTURE WORK
In this paper, we explore a novel combination of GAN and contrastive learning, two important, yet seemingly different paradigms in unsupervised representation learning. They both have been independently observed the crucial importance of maintaining consistency to data augmentations in their representations (Zhang al., 2020; Tian et al., 2020), and here we further observe that these two representations complement each other when combined upon the shared principle of view- invariant representations: although we have put more efforts this work to verify the effectiveness of contrastive learning on GANs, we do observe the opposite direction through our experiments, and scaling up our method to compete with oftc-of-thhe-r self-supervised learning benchmarks (Chen et al., 2020b; Grill et al., 2020; Caron et al., 2020) would be an important future work.
We believe our work also suggests several interesting ideas to explore in future research in both sides of GAN and contrastive learning: for example, our new design of introducing small header to minimize the GAN loss upon other (e.g., contrastive) representation is a promising yet unexplored way of designing new GAN architecture. For the SimCLR side, on the other hand, suggest new idea of incorporating "fake" samples for contrastive learning, which is also an interesting direc- tion along with other recent attempts to improve the efficiency of negative sampling in contrastive learning, egg., via hard negative mining (Kalantidis et al., 2020; Robinson et al., 2021).
ACKNOWLEDGMENTS
This work was supported by Samsung Advanced Institute of Technology (SAIT). This work was also partly supported by Institute of Information & Communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (N0.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST)). The authors would like to thank Minkyu Kim for helping addi- tional experiments in preparation of the samera-ready revision, and thank the anonymous reviewers for their valuable comments improve our paper.

---

Published as a conference paper at ICLR 2021
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214-223, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:// proceedings mlm www...mww........mmm..mmmmmmee.e www.pprssssv00m jowsssyyoooolylloooo 7a html.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems 32, pp- 15545_
Curran Associates, Inc., 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - Mining discriminative com- ponents with random forests. In Proceedings of the European Conference on Computer Vision, pp. 446-461. Springer, 2014.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
https://cpenzeview nnt/ forum?idde Forum?id-Blxsqj00
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33, 2020.
Tong Che, Ruixiang Zhang, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your GAN is secretly an energy-based model and you should use discriminator driven latent sampling. In H. Larochelle, M. Ranzato, Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 12275-12287 Curran As- sociates, Inc., 2020. URL wwwptttpppse..o http:::ssp//oooeeedeee ings www.ieirisp..... psw.ips.... cc wwtpppppp....... /ttpapprr/2///111/// 90525e70b 0111778222555555551555882
pdf.
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised GANs via auxiliary rotation loss. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12154-12163, 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research. PMLR, 2020a.
Xinlci Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXiv preprint wwXiv:2003.04297, 2020b.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor Woo Ha. StarGAN v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, June 2020.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Terrance DeVrics and Graham W Taylor. Improved regularization convolutional neural networks with Cutout, arXiv preprint arxiv:/70ss 04552, 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 1422-1430 2015.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. URL https: //tp//eeeeeeeee/eee/eeeeee/eeeee net wwtp/eeeurrtteee forum? di v1421100000
10

---

Published as a conference paper at ICLR 2021
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Pro- cessing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014. URL ptstt://papere: cpssccc/paper//// happer//423--0 2014422-geneaatiii ve- adversarie aatilll-nttttt. pdf.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 21271-21284. Curran Associates, Inc., 2020. URL, https: /ptoceed ngs neurips. (appaper222// file/ weeaaaaaa05ceeeee 111000/2211111122222222
Paper.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of Wasserstein GANs. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems 30, pp. 5767-5777. Curran Associates, Inc., 2017. URL http: / pwpp/spppessee//////eep/eee nips. cc/ wa s se wwwseerreeeeeeeeeee waaeeeiinnnnnnggae
paper/7159 1mp improved- t srraiingg-er
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, June 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. PMLR, 13-18 Jul 2020. URL http: httpp//oceeedeeeecc ngg. www.l.ppress...... vv11 99 www.ahfaaffffaffmoo html.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems 30, pp. 6626-6637 Curran Associates, Inc., 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, pp.
Curran Associates, Inc., 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448 456, Lille, France, 07-09 Jul 2015. PMLR. URL htth:::: httptttpppp/ppoooeed..ooo //pppceeeedngg.. m1 mlr.prres/vv3l ////////ffffeee offel5 html.
Alexia Jolicoeur-Martineall The relativistic discriminator: a key element missing from stan- dard GAN. In International Conference on Learning Representations, 2019. URL https: // //openzeview. net wttp:e/ww.nn.ooummmmeeeee
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 21798-21809. Curran Associates, Inc., 2020. URL https: httpt:tttt///oeetteee http://pooooeeeeeeeee/eoe ngs. neurips. cc/pappr/20200
Minguk Kang and Jaesik Park. ContraGAN: Contrastive learning for conditional image gen- eration. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 21357-21369, Curran As- sociates, Inc., 2020. URL https: htttpp::/ppooceednngg www.eeeeeddii....enieeeeeeeeeeee cctprips..ee.eee...ee.eeoooeeeee f490c742ccm 8b8eeódca 63f wttp:rppppscccppppp////////..... http//ppppe/222/1112222
11

---

Published as a conference paper at ICLR 2021
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019.
Tero Karras, Miia Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12104-12114, Curran Associates, Inc., 2020a. URL https; / cc/ htc//paper/202/// e 8d30aa 9 6e 72047744477555 4bd230 6a1 PaaedddPape..pee
httptt:ttp/pooooeeeeeootoot /ptpoeeeiigggs...eoeeppppo neurips.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of StyleGAN. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020b.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Ad- vances in Neural Information Processing Systems, volume 33, pp. 18661-18673, Curran As- sociates, Inc., 2020. URL https: httpss:/pproceedingg nwwrips. wttpcc/ppppppe/eeee/// d89a66c7cs a 2 9b1 bdbabo
Paaper pdf.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2014.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recog- nition, pp. 1920-1929, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Depart- ment of Computer Science, University of Toronto, 2009.
Karol Kurach, Mario Lučić, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. A large-scale study regularization and normalization in GANs. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Pro- ceedings of Machine Learning Research, pp. 3581-3590, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http: http///ooccediiieee/e/ee mlr. wwwppprsssssee. v /kurach1 9a html.
Cheng-Hie Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. MaskGAN; Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020.
Kwot Sin Lee, Ngoc-Trung Tran, and Ngai-Man Cheung. InfoMax-GAN Improved adversarial image generation via information maximization and contrastive learning. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 3942 2952, January 2021.
Jac Hyun Lim and Jong Chul Ye. Geometric GAN. arXiv preprint arXiv: 2017.
R. Linsker. Self-organization in a perceptual network. Computer, 21(3):105-117, 1988.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? a large-scale study. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bisa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 700-709. Curran Associates, Inc., 2018. URL http: /aapers.pppppp/p nips. ca/paper/ 7350-ares garee--aas---e created -a a- la aarge- wwagge-ccllle wesaacllleeeee studle-ssuuupppeee
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for GANs do actually converge? In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Re- search, pp. 3481-3490, Stockholmsmässan Stockholm Sweden, 10-15 Jul 2018. PMLR. URI. http:/ /ttf:/prrooe http///ooceeecooooooo ngs .m prwss/eessueee vB0 mesccederrrrr haam..
Takeru Miyato and Masanori Koyama. eGANs with projection discriminator. In International Conference on Learning Representations, 2018, URL https: http///ppeeeee.ce////eeee net forum? M--BBySVVpRRR
12

---

Published as a conference paper at ICLR 2021
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018. URL
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In The Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding, arXiv preprint arXiv: 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8026-8037, 2019.
Joshua David Robinson, Ching-Yas Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021.
URL httpt:ttp///e /ttps/////ppereeeeeee//em net forum? cerrumidd-cRioooOOC
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. SinGAN: Learning generative model from a single natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, October 2019.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211 152, 2015. doi: 10000//11263-01--088881
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training GANs. In D. D. Lee, M. Sugiyama, V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, Curran Associates, Inc., 2016. URL http: /twpppaa.s...//e/ nips cc paper/ pp. wwwrroovedteeiiiiiieeeeeee ques for training gans. .df.
6125- improved
Edgar Schonfeld, Bernt Schiele, and Anna Khoreva. U-Net based discriminator for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8207-82 8216, 2020.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6827-6839 Curran Associates, Inc., 2020. URL https: http://pcooeeen...cceece neurips..... htspps.c/////.ee ce/ httee///22///ieee//eeeeeeeeeeee 00e.a91122009999959555555888
Dustin Tran, Rajesh Ranganath, and David Blei. Hicrarchical implicit models and Miclihooddfrre variational inference. In Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL httpp::/pprcceee/// file/6f1d0705cc 0tt.eee2222221111111110120303030 Paperr.-p..... .pdf.
neurips cc/paper/2017/
Ngoe-Trung Tran, Viet-Hung Tran, Ngoc-Bao Nguyen, and Ngai-Man Cheung. An improved self- supervised GAN via adversarial training. 2019.
Ngoe-Truun Tran, Viet-Hung Tran, Agoc-Bao Nguyen, Trung-Kien Nguyen, and Ngai-Man Che- ung. On data augmentation for GAN training. IEEE Transactions on Image Processing, 30: 1882-1897, 2021. doi: 00.1009/1P22222334493466
Xiang Wei, Zixia Liu, Liqiang Wang, and Boqing Gong. Improving the improved training of Wasserstein GANs. In International Conference on Learning Representations, 2018. URL.
.nt/ httpps://ppenrevaa wttpriewwwwe/tttuueneeeme worumoiddd--oo..o 1orummidd-xxx90000
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, pp. 681-688 2011.
13

---

Published as a conference paper at ICLR 2021
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742. 2018.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to 20o. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1485-3492 IEEE, 2010.
Han Zhang, Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In International Conference on Learning Reprexentations, 2020.
URL www.hhtps//oote http///ppeeeeeeeeeeeeeeeeeeeeeee net forum?
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun-Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7559-7570. Cur- Associates, Inc., 2020a. URL httpt: http::sss///oooeedieneee neurips wwwerriss.ccpaaaeeeeeeeeeeeeeeee pttpp//ppppp.//////// BLLL:////5500000311111100222222
Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen. Feature quantization im- proves GAN training. Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research. PMLR, 2020b.
Zhengli Zhao, Sameer Singh, Honglak Lee, Zizhao Zhang, Augustus Odena, and Han Zhang. Im- proved consistency regularization for GANs, 2020c.
Zhengli Zhao, Zizhao Zhang, Ting Chen, Sameer Singh, and Han Zhang. Image augmentations for GAN training. arXiv preprint arXiv:2006.022. 02595, 022509 2020d.
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, Oct 2017.
14

---

Published as a conference paper at ICLR 2021
A TRAINING PROCEDURE OF CONTRAD
Algorithm GANs with Contrastive Discriminator (Contral))
Require: generator G, discriminator D, Adam hyperparameters o B2, number of D updates per G update ND. family of data augmentations T > 0.
1: for training iterations do
2: fort= to ND do
3: Sample æ ~ Pdata (x) and p(z) Sample tı, t2, t3 ~
4:
(1) v. 1t (ax), 12 (z) L/3 (G()))
5:
Leon t LSiinLa( 21 (2) 12h
6:
7:
Tronn N Eisl Lusppoon (1t [e, -1;
1; 18
(1), of [,, D, h4)
8:
Pr, + sg(D (n( )) sg D(m+))
9:
Lais f- -Er, -In,,,oog (oogll )))l - Er, log(1 - o(ha(rr)))
10:
LD Lion + NoonLeon + daieLais
11:
D, D,h,, 122 f- Adam (D, (D, M1 h4, hal, Lpi a, 32)
12:
end for
13:
LG f- -E,, -,,,,,,,,ooo (hg
14:
G Adam(G, Lo; a, Bi, 82)
15:
end for
B RESULTS ON AFHQ DATASETS
We evaluate our training method on the Animal Faces-HQ (AFHQ) dataset (Choi et al., 2020), which consists of ~ 15,000 samples of animal-face images at 512x512 resolution, to further verify the ef- fectiveness of ContraD on higher- regouttresolu,ion yet Limited-sized datasets. We partition the dataset by class labels into three sub-datasets, namely AFHQ-Dog (4,739 samples), AFHQ-Cat (5,155 sam- ples) and AFHQ- (4,738 samples), and evaluate FIDs on these datasets. In this experiment, compare our method with ADA (Karras et al., 2020a), another recent data augmentation scheme for GANs (concurrently to DiffAug (Zhao et al., 2020a) considered in Section 4), which includes a dynamic adaptation of augmentation pipeline during training. We follow the training details of ADA particularly for this experiment: specifically, we train StyleGAN2 with batch size 64 for 25M train- ing samples, using Adam with (a, B1, B2) = (0.0025, 0.0, 0.99), Rí regularization with 7 = 0.5, and exponential moving average on the generator weights with half-life of 20K samples. For Con- traD training on AFHQ datasets, we consider to additionally use CutOut VDe & Taylor, 2017) of p = 0.5 upon the SimCLR augmentation, which we observe an additional gain in FID. We also follow the way of computing FIDs as per Karras et al. (2020a) here for fair comparison, i.e., by comparing 50K of generated samples and all the training samples (~5K per dataset).
Table compares our ContraD models with the results reported by Karras et al. (2020b) where the models are available at https: //tpt/ggttbb//// coo/NVlaab hww.aaa/stllllaaaaeeeeaee stylegann-aaaa Overall, we consistently observe that ContraD improves the baseline StyleGAN2, even achieving significantly better FIDs than ADA on AFHQ-Dog and AFHQ-Wild. These results confirms the effectiveness of ContraD on resouurresolttion datasets, especially under the regime of limited data. Qualitative comparisons can be found in Figure 6, 7, and 8 of Appendix D.
Table 7: Comparison of the best FID score (lower is better) on unconditional image generation of AFHQ datasets (Choi et al., 2020) with StyleGAN2. Values in the rows marked by are from those reported in its reference. We set our results bold-faced whenever the value improves the StyleGAN2 baseline ("Bascline"). Underlined indicates the best score among tested for each dataset.
Architecture
Method
DoG
CAT
WILD
Baseline* (Karras et al., 2020b)
19.4
5.13
3.48
StyleGAN2
ADA* * (Karras et al., 2020a)
7.40
3.55
3.05
ContraD (ours)
7.16
3.82
2.54
15

---

Published as a conference paper at ICLR 2021
C RESULTS ON IMAGENET DATASET
Table 8: Comparison of the best FID score and IS on conditional generation of ImageNet (64x64)
with BigGAN architecture. Values in the rows marked by * from those reported in its reference.
ImageNet
FID
IS
BigGAN (Brock et al., 2019)
10.6
25,43
FQ-BigGAN* (Zhao al., 2020b)
9.67
25,961104
ContraD-BigGAN (ours)
8.32
26.94
Table 9: Comparison linear evaluation and transfer learning performance across 6 natural image classification datasets for BigGAN discriminators pretrained on ImageNet (64 x 64). We report the top-1 accuracy except for ImageNet and SUN397, which we instead report the top-5 accuracy.
Training (BigGAN)
ImageNet
CIFARIO
CIFARIO0
DTD
SUN397
Flowers
Food
Supervised (ImageNet)
63.5
76.1
55.2
45.4
31.7
78.1
44.5
SimCLR (Aeon = Aaia 0)
43.4
81.2
55.3
43,9
37.6
69.8
38.8
ContraD (ours)
51.5
84.5
61.1
50.6
44.4
78.6
44.5
We also apply ContraD to the -fate-of-the-art BigGAN (Brock et al., 2019) architecture on ImageNet (Russakovsky et al., 2015), to examine the effectiveness of our method on this large-scale, class- conditional GAN model. For this experiment, we exactly follow the training setups done in FQ- BigGAN (Zhao et al., 2020b), a recent work showing that feature quantization in discriminator could improve state-of-the-art GANs, which is based on the official PyTorch (Paszke et al., 2019) implementation of BigGAN.* Following Zhao et al. (2020b). we down-scale the ImageNet dataset into 64 x 64, and reduce the default channel width of 96 into 64. We train the model with batch size 512 for 100 epochs (~2500K generator steps). Unlike other experiments, we compute FIDs between the training set and 50K of generated samples for fair comparison the baseline scores reported by Zhao et al. (2020b). In Table 8, we show that ContraD further improves the baseline BigGAN training and FQ-BigGAN (Zhao et al., 2020b) both in FID and IS under the same training setups.
Next, we evaluate the representation of the learned BigGAN encoder D from ContraD under lin- ear evaluation (on ImageNet) and several transfer learning benchmarks of natural image classifica- tion: namely, we consider CIFAR- 10 and CIFAR- 100 (Krizhevsky, 2009), the Describable Textures Dataset (DTD) (Cimpoi et al., 2014), SUN397 (Xiao et al., 2010), Oxford 102 Flowers (Flowers) (Nilsback & Zisserman, 2008), and Food-101 (Bossard et al., 2014). As done in Table 3, we com- pare ContraD with an ablation when Aeon = Adie = 0, which equivalent to the SimCLR (Chen et al., 2020a) training. In addition, we also compare against the supervised baseline, where the rep- resentation is pre-trained on ImageNet with standard cross-entropy loss using the same architecture of BigGAN discriminator. Similarly to the ContraD training, this baseline is also trained with the SimCLR augmentation, using Adam optimizer (Kingma & Ba, 2014) for 120 epochs.
Table summarizes the results. We first note that the supervised training achieves 63.5% top-5 accuracy on ImageNet, while the SimCLR baseline achieves 43.4% (in linear evaluation). Again, as also observed in Table 3, Contral significantly improves the linear evaluation performance from the baseline SimCLR, namely to 51.5%. Although the results are somewhat far behind compared to those reported in the original SimCLR (Chen et al., 2020a), this is possibly due to the use of model with much limited capacity, i.e., the BigGAN discriminator: its essential depth is only 100 and it also uses spectral normalization (Miyato al., 2018) for every layer unlike the ResNet-50 architec- ture considered by Chen et al. (2020a). Nevertheless, we still observe similar trend in the accuracy gap between supervised and (unsupervised) contrastive learning. On the other hand, the remaining results in Table 9, those from the transfer learning benchmarks, confirm clear advantage of Con- traD, compared to both the SimCLR and supervised baselines: we found the two baselines perform similarly on transfer learning, while our ContraD training further improves SimCLR consistently.
'https: / /i thub. com/ jorck Bi gGAN- Pyrch
'e notice here that, although BigGAN is class- conditionl architecture. i.e., it uses the label information in training, the encoder part 1) in ContraD is still trained in unsupervised manner, as put the labels only after stopping gradients, i.e. in the discriminator header hass-
16

---

Published as a conference paper at ICLR 2021
D QUALITATIVE RESULTS
Real Images
ContraD (FID: 10.9)
bCR (FID: 14.0)
CR (FID: 19.5)
DiffAug (FID: 22.9)
W/O (FID: 26.6)
Figure 2: Qualitative comparison of unconditionally generated samples from GANs with different training methods. All the models are trained on CIFAR-10 with SNDCGAN architecture.
Real Images
ContraD (FID: 14.8)
bCR (FID: 19.2)
CR (FID: 22.2)
DiffAug (FID: 27.0)
W/O (FID: 28.5)
Figure 3: Qualitative comparison of unconditionally generated samples from GANs with different training methods. All the models are trained on CIFAR-100 with an SNDCGAN architecture.
17

---

Published as a conference paper at ICLR 2021
W/O (FID: 24.8)
CR (FID: 20.9)
bCR (FID: 19.5)
ContraD (FID: 17.1)
Figure 4: Qualitative comparison of unconditionally generated samples from GANs with different training methods. All the models are trained on CelebA-HQ-128 with an SNDCGAN architecture.
Plane
Car
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Figure 5: Visualization of conditionally generated samples via conditional DDLS (Section 3.3) with ContraD. We use an SNDCGAN generator trained with an SNResNet-Ie 18 ContraD for the generation.
18

---

Published as a conference paper at ICLR 2021
Real Images
StyleGAN2 Untruncated (FID: 19.4)
ADA / Untruncated (FID: 7.40)
ContraD / Untruncated (FID: 7.16)
ADA/ / Truncated (v 0.7)
ContraD Truncated (v = 0.7)
Figure 6: Qualitative comparison of unconditionally generated samples from GANs with different training methods, along with real images from the training set. All the models are trained on AFHQ- Dog (4,739 images) with StyleGAN2. We apply the truncation trick (Karras et al., 2019) with 0.7 to produce the images the bottom row. We use the pre-trained models officially released by the authors to obtain the results of the baseline StyleGAN2 and ADA.
19

---

Published as a conference paper at ICLR 2021
Real Images
StyleGAN2 / Untruncated (FID: 5.13)
ADA / Untruncated (FID: 3.55)
ContraD / Untruncated (FID:3.82 3.82)
ADA/ / Truncated (vp = 0.7)
ContraD / Truncated (v = 0.7)
Figure 7: Qualitative comparison of unconditionally generated samples from GANs with different training methods, along with real images from the training set. All the models are trained AFHQ- Cat (5,153 images) with StyleGAN2. We apply the truncation trick (Karras et al., 2019) with i = 0.7 to produce the images the bottom row. We use the pre-trained models officially released by the authors to obtain the results of the baseline StyleGAN2 and ADA.
20

---

Published as a conference paper at ICLR 2021
Real Images
StyleGAN2 Untruncated (FID: 3.48)
ADA / Untruncated (FID: 3.05)
ContraD / Untruncated (FID: 2.54)
ADA/ / Truncated (v 0.7)
ContraD Truncated (v = 0.7)
Figure 8: Qualitative comparison of unconditionally generated samples from GANs with different training methods, along with real images from the training set. All the models are trained on AFHQ- Wild (4,738 images) with StyleGAN2. We apply the truncation trick (Karras et al., 2019) with 0.7 to produce the images the bottom row. We use the pre-trained models officially released by the authors to obtain the results of the baseline StyleGAN2 and ADA.
21

---

Published as a conference paper at ICLR 2021
E
EFFECT OF USING LARGER BATCH SIZES
64
128
256
512
FID
10
W/O
bCR
ContraD
Figure 9: Comparion of the FID distribution of the top 25% of trained models (Kurach et al., 2019) on CIFAR-10 (SNDCGAN) for different batch sizes.
Contrary to existing practices for training GANs, we observe in our experiments that ContraD typ- ically needs larger batch size in training to perform best. We further confirm this effect of batch sizes in Figure 9: overall, we observe the stanard training ("W/O") is not significantly affected by partieular batch size, while ContraD offers much better FIDs on larger batch sizes. We also ob- serve bCR (Zhao et al., 2020c) slightly benefits from larger batches, but not as much as ContraD does. This observation is consistent with Chen al. (2020a) that contrastive learning benefits from using larger batch sizes, and it also supports our key hypothesis on the close relationship between contrastive and discriminator representations.
F EXPERIMENTAL DETAILS
F.1 ARCHITECTURE
Overall, we consider four architectures in our experiments: SNDCGAN (Miyato et al., 2018), StyleGAN-2 (Karras al., 2020b), and BigGAN (Brock et al., 2019) for GAN architectures, and additionally SNResNet-18 for discriminator architecture only. For all the architectures considered, we have modified the last linear layer into a multi-layer perceptron (MLP) having the same input and output size, for fair comparison to ContraD that requires such MLP for the discriminator header ha- When a GAN model is trained via ContraD, we additionally introduce two 2-layer MLP projection heads of the output size 128 upon the penutimate representation in the discriminator to stand h, and h4. All the models are implemented in PyTorch (Paszke et al., 2019) framework.
SNDCGAN. We adopt a modified version of SNDCGAN architecture by Kurach al. (2019) fol- lowing the experimental setups of other baselines (Zhang et al., 2020; Zhao et al., 2020c). The official implementation can be found in compare-gan codebase," and we have re-implemented this TensorFlow implementation into PyTorch framework. The detailed structures of the generator and discriminator of SNDCGAN are summarized in Table 10 and 11, respectively.
SNResNet-11. We modify the ResNet-18 (He et al., 2016) architecuture, where PyTorch imple- mentation available from torchvision' library, not include batch normalization (Ioffe & Szegedy, 2015) layers inside the network. Instead, we apply spectral normalization (Miyato et al., 2018) for all the convolutional and linear layers. We made such a modification to adapt this model as a stable GAN discriminator, batch normalization layers often harm the GAN dynamics in discriminator practice.
StyleGAN2. We follow the StyleGAN2 architecture used in the DiffAug baseline (Zhao et al., 2020a). Specifically, we generally follow the official StyleGAN2, but the number of channels at 32x32 resolution is reduced to 128 and doubled at each coarser level with maximum of 512 channels, in order to optimize the model for CIFAR datasets, as suggested by Zhao et al. (2020a).
htttps:/ //www///wwww////o/////////... www..gitii git.ub. www.hh..co//oooooooooooooooo e compare_gan "https:// / thub. www.///wtwtc////////ooooooo https///////////a////////ootoo//
22

---

Published as a conference paper at ICLR 2021
Table 10: SNDCGAN generator
Table 11: SNDCGAN discriminator
Layer
Kernel
Output
Layer
Kernel
Output
Latent & Linear, BN, ReLU
128
Conv, LeI
3,3, 1]
h
w 64
h/8 x w/8 x 512
Conv, LeakyRel.U
1.4.2
h/2 x w x 128
DeConv, BN, ReI.U
1,
h/4
X w x 256
Conv, LeakyReLU
3,3.
h/2 x w/2 x 128
DeConw, BN, ReLU
4,4,2
h/2
w
x 128
Conv, LeakyRel.J
4,4,2
h/4 x w//4 x 256
DeConv, BN, ReLUU
1,4,2
h
w
x 64
Conv, Lew.kyeLRRLUU
[3,3
h/4 x w/4 x 256
DeConv, Tanh
3,3, 1
X w
X
Conv, LeakyReLU
4,4,2
h/8 w/ x 512
Cony, LeakyReL.U
3,3,
h/8 x w/8 x 512
BigGAN. We use the official PyToch implementation of BigGAN (https://github. com/ a jjbrock/ / Bi gCAN-PyTorch), In order to adapt ContraD into the class-conditional BigGAN architecture, we apply the goggggraientt operation not only on the penultimate representation before computing has but also before applying the class- conditional projection (Miyato & Koyama, 2018).
F.2 TRAINING DETAILS
SNDCGAN. Overall, we follow the best hyperparameter practices explored by Kurach et al. (2019) for SNDCGAN models on CIFAR and CelebA-HQ-IE datasets: We use Adam (Kingma & Ba, 2014) with (ar, 32) ) = (0.0002, 0.5, 0.999) for optimization with batch size of 64, while in case of ContraD on CIFAR datasets the batch size of 512 is used instead. For CelebA-HQ- 128, we keep the default batch size of 64 even for ContraD. All models but the "Hinge" baselines in Table 2 are trained with the non-saturating loss (Goodfellow et al., 2014). We stop training after 200K generator updates for CIFAR- and 100K for CelebA-HQ-122 When CR bCR is used, we set their regularization strengths = 10 for both real and fake images.
StyleGAN2. We follow the training details of DiffAug (Zhao al., 2020a) in their CIFAR
periments: we use Adam with (ar, (,,,,,33) = (0.002, 0.0, 0.99) for optimization with batch size of 32, but 64 for ContraD models. We use non-saturating loss for training, and use RS regularization (Mescheder et al., 2018) with = 0.1. We do not use, however, the path length regularization and the lazy regularization (Karras et al., 2020b) in training. We take exponential moving average on the generator weights with half-life of 10 We stop training after 800K generator updates. When CR or bCR is used, we set their regularization strengths X = 10 for both real and fake images.
Linear evaluation and transfer learning. For a single linear evaluation (and transfer learning) run, we train a linear classifier upon a given (frozen) representation. The results in Table are trained via stochastic gradient descent for 100 epochs: the initial learning rate of 0. and it is decayed by 0.1 at 60, 75, and 90-th epoch. In case of Table 9, on the other hand, we use Adam optimizer (Kingma & Ba, 2014) for 90 epochs: the initial learning rate if 0.0002, and it is decayed by 0.3 at 60 and 75-th epoch. We augment every training dataset via random crop, resizing and horizontal flipping.
0.1 for 1,000 iterations. We follow all the practical considerations proposed by Che et al. (2020) to improve the sample quality of DDLS: specifically, (a) we separately set the standard deviation of the Gaussian noise (13) 0.1, and (b) to alleviate mode dropping issues, instead of rumning DDLS on G itself, we run the sampling with G* (z, x') == G(z) + ez', where z' is also jointly updated via Langevin dynamics.
Conditional DDLS. We run Langevin sampling of step size
Hyperparameters in ContraD. Unless otherwise specified, we use Xean = Aais = 1, and r = 0.1 in our experiments, where 7 is the temperature hyperparameter defined in (5). We use hidden layer of size 512 for fyy, hr. ha in SNDCGAN and StyleGAN2, and 1024 in SNResNet- 18 and BigGAN. For ContraD models, we use linear warm-up strategy on learning rate up to 3K steps of training for both generator and discriminator, following practices in contrastive learning (Chen et al., 2020a).
SimCLR augmentations. We use the augmentation pipeline proposed in SimCLR (Chen et al., 2020a) for training ContraD. Specifically, for CIFAR datasets, we sequentially transform given image by: (a) random crop and resizing. (b) horizontal flipping, (c) color jittering with probability of p = 0.8, and (d) graying the image with probability of p 0.2. We additionally apply (f) Gaussian blurring with probability of p = 0.5 for higher resolution images such as CelebA-HQ-112 and ImageNet. We ensure that every transformation in this pipelines are differentiably implemented, so that the augmentation still can be used for training the generator of GAN.
23


Reviews:
[{"review_id": "wWtYkekwup", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors suggest using the contrastive loss to improve the training of the discriminator and further stabilize the GAN training process. More specifically, the proposed method incorporates the self-supervised simCLR contrastive loss on a pair of transformed real images and supervised contrastive loss on the fake ones. The proposed method is evaluated on the image synthesis task on CIFAR10/100 and CelebA-HQ-128 images and over several different GAN models.\n\nStrengths:\n* The idea of using self-supervised learning for improving the training dynamics of the discriminator makes sense and is an interesting exploration area.\n* Empirical evaluations show a consistent and significant advantage for the proposed methods and ablation studies verify the contributions of the different proposed components.\n\nweaknesses:\n* The proposed method is rather a careful ensembling of existing components, e.g. simCLR self-supervised or supervised contrastive loss in the right context, rather than a radically novel methodological contribution.\n* The proposed method is only tested on relatively low-resolution datasets, namely CIFAR10/100 and CelebA-HQ-128. It would have been interesting to also demonstrate the contributions on the more challenging higher resolution datasets.\n\nDetailed comments:\n* Equation 7: If I am not mistaken, it doesn't seem quite the same as in the referred supervised contrastive loss from Khosla et al.; more specifically, I think the order between log and \\sum_{v_{i+}^{(2)} \\in V_{i+}^{(2)}} need to be reversed. Please clarify.\n* An ablation study I was missing was having the proposed method without the extensive simCLR augmentations and see how it compares to the other methods.\n* It would have been interesting to also compare with some other recent and relevant work, e.g. ADA (Karras et al.).\n* \"Remark that we use an independent projection header h_f instead of h_r\" => Is this making a significant difference? Are there ablation studies showing this?\n* Typo: \"approaches that handles\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "sYACn1zjun8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\nThe manuscript proposes ContraD - a method that incorporates the recent SimCLR self-supervised learning method for images into the GAN training framework. Experimental results show that the proposed method consistently improves on strong baselines in terms of FID scores.\n\n**Score justification**\nThe paper is well-written, the introduction carefully synthesizes a lot of recent work on self-supervised representation learning and GANs; results are convincing (albeit mostly obtained on smaller-scale datasets) and ablations support paper claims.\n\n\n**Major comments**\n* Ablation of $L_{con}^{+}$ in Table 6 tells a mixed story - it breaks down for the SNResNet discriminator, but seems to work well for the SNDCGAN discriminator. It's hard to conclude something definitive from the failed SNResNet experiment (could have worked with more hyper-parameter tuning?), but the successful SNResNet experiment suggests that the $L_{con}^{+}$ may not be necessary at all, which is at odds with the authors' narrative. Would be great to see more ablations showing that this is not the case, including a combined ablation of $L_{con}^{+}$ , $L_{con}^{-}$ and stop gradients.\n\n* It would be interesting to see linear separation and transfer to other datasets (as in the SimCLR paper) for the ImageNet models. Strong results there would support the authors' narrative of the synergy GANs and SimCLR training.\n* I am not fully convinced by the strong claims of strong coherence between GANs and contrastive representation learning (e.g. \"[...] these two representations greatly complement each other [...]\", \"[...] a good contrastive representation is good for GANs and vise versa\"). In particular it seems to be that GAN training benefits from contrastive learning (it seems to be a good auxiliary loss in presence of strong augmentation), but the claims of benefit to SimCLR don't seem fully substantiated.\n\n**Minor comments**\n* Multiple citations do not have correct capitalization (e.g. \"gan\")\n* Results presented in Table 3 (linear evaluation on CIFAR-10) can be a bit misleading - at first glance they seem to suggest that * ContraD improves on SimCLR results from the original paper (model pre-trained on ImageNet and transferred to CIFAR-10). Making it clear that this is not the case (e.g. by including the original SimCLR results and/or supervised performance) would be helpful to the readers.\n* Similarly, Table 4 would be easier to take in in the context of FID scores that are achieved by class-conditional GANs.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "gbroPcaasof", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to improve GAN training by incorporating augmentations from contrastive learning. Specifically, a new contrastive discriminator, named ContraD, is proposed for GANs; with ContraD, the encoder part of the discriminator is trained with (two) contrastive learning losses, while the left discriminator head and the GAN generator are trained as usual. The authors argue that this specific fusion of GAN and contrastive learning signiﬁcantly stabilizes GAN training and, moreover, the fused two research fields could benefit each other.\n\nThe paper is well written overall. The idea is interesting, and the technical details may be sound. However, I am not fully convinced because of the concerns listed below.\n \nAlthough the approach is proposed to stabilize GAN training, this aspect was not highlighted in the experiments.  \n \nIn Section 3.1, the discriminator encoder is trained with two specifically-chosen self-supervised-learning losses, and the experiments show that such a specific combination is essential for a good performance; otherwise, “the GAN loss could completely negate the effectiveness of ContraD. ” So why would that happen? How to choose those losses in practice? Detailed discussions are necessary here.\n \nThe first sentence of Section 3.2 is contradictory with Eq (12) and Algorithm 1.\n \nIn Eq (14), how to choose the v for a specific class? Was the label information used to get the v?\n \nIn Table 1, it seems the performance of StyleGAN2 is much lower than that reported in the original paper. Please elaborate on that.\n \nIn the experiments, larger batch sizes are used for the proposed method. Will that affect the fairness? Discussions are necessary.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fqyvlqreIbH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "==== Summary ====\n\nThis paper improves upon state-of-the-art GANs by incorporating recent advances of contrastive representation learning into the training of discriminator. In particular, the discriminator loss function consists of three terms: (1) the original SimCLR loss on the multi-view real data pairs; (2) the supervised contrastive loss (Khosla et al, 2020) that assigns high scores among the fake sample pairs and giving lower score among the real data pairs; (3) the usual discriminator loss in GAN training. While each of these terms alone is not entirely new, the author proposes several tricks to make the training of GANs together with the contrastive loss works. Empirically, the proposed method outperforms other GAN methods trained with auxiliary data augmentation techniques, and demonstrates good representations under the linear classifier probing setup. \n\nPros:\n\n(1) Writing is clear and easy to follow\n\n(2) Strong empirical performance in both image generation and classifier probing \nSolid experiment designs with thorough ablation studies\n\nCons:\n\n(1) Rather limited novelty in terms of technical contributions\n\n==== Technical Questions ====\n\nQ1: Regarding the supervised contrastive loss for fake samples (i.e., L_{con}^{-} in Figure 1).\nI am wondering if it is beneficial to also consider data augmentation on the fake sample produced by generators. In that case, the loss function matrix in Figure 1 will look closer to the L_{con}^{+} part, which encourages different views of the same fake sample to have similar representations.\n\nQ2: I feel like there’s one ablation setting missing in Table 1: using “HFlip, Trans” data augmentation for the ContraD. This way, we can see the true benefit of combining those three losses.\n\nQ3: For the linear evaluation results in Table 3, SimCLR results seem not consistent with their original paper? What causes the difference?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims to improve the training of generative adversarial networks (GANs) by incorporating the principle of contrastive learning into the training of discriminators in GANs. Unlike in an ordinary GAN which seeks to minimize the GAN loss directly, the proposed GAN variant with a contrastive discriminator (ContraD) uses the discriminator network to first learn a contrastive representation from a given set of data augmentations and real/generated examples and then train a discriminator based on the learned contrastive representation. It is noticed that a side effect of such blending is the improvement in contrastive learning as a result of GAN training. The resulting GAN model with a contrastive discriminator is shown to outperform other techniques using data augmentation.

**Strengths:**
  * It proposes a new way of training the discriminators of GANs based on the principle of contrastive learning.
  * The paper is generally well written to articulate the main points that the authors want to convey.
  * The experimental evaluation is well designed and comprehensive.

**Weaknesses:**
  * Even though the proposed learning scheme is novel, the building blocks are based on existing techniques in GAN and contrastive learning.
  * The claim that GAN helps contrastive learning is not fully substantiated.
  * It is claimed in the paper that the proposed contrastive discriminator can lead to much stronger augmentations *without catastrophic forgetting*. However, this “catastrophic forgetting” aspect is not really empirically validated in the experiments.
  * The writing has room for improvement.

Despite its weaknesses, this paper explores a novel direction of training GANs that would be of interest to the research community.

Decision: Accept (Poster)