Conditional GANs with Auxiliary Discriminative Classifier
Authors: Liang Hou
Qi Cao
Huawei Shen
Xueqi Cheng
Venue: ICLR 2022 Submitted (2022)
Keywords: conditional generative adversarial networks
conditional image generation

Abstract: Conditional generative models aim to learn the underlying joint distribution of data and labels, and thus realize conditional generation. Among them, auxiliary classifier generative adversarial networks (AC-GAN) have been widely used, but suffer from the problem of low intra-class diversity on generated samples. In this paper, we point out that the fundamental reason is that the classifier of AC-GAN is generator-agnostic, and therefore cannot provide informative guidance to the generator to approximate the target distribution, resulting in minimization of conditional entropy that decreases the intra-class diversity. Motivated by this observation, we propose a novel conditional GAN with auxiliary \textit{discriminative} classifier (ADC-GAN) to resolve the problem of AC-GAN. Specifically, the proposed auxiliary \textit{discriminative} classifier becomes generator-aware by recognizing the labels of the real data and the generated data \textit{discriminatively}. Our theoretical analysis reveals that the generator can faithfully replicate the target distribution even without the original discriminator, making the proposed ADC-GAN robust to the hyper-parameter and stable during the training process. Extensive experimental results on synthetic and real-world datasets demonstrate the superiority of ADC-GAN on conditional generative modeling compared to competing methods.

Under review as a conference paper at ICLR 2022
CONDITIONAL GANS WITH AUXILIARY DISCRIMINA- TIVE CLASSIFIER
Anonymous authors Paper under double-blind review
ABSTRACT
Conditional generative models aim learn the underlying joint distribution of data and labels, and thus realize conditional generation. Among them, auxiliary classifier generative adversarial networks (AC-GAN) have been widely used, but suffer from the problem of low intra-class diversity on generated samples. In this paper, we point out that the fundamental reason that the classifier of AC- GAN is generator-agnostic, and therefore cannot provide informative guidance to the generator to approximate the target distribution, resulting in minimization of conditional entropy that decreases the intra-class diversity. Motivated by this observation, we propose a novel conditional GAN with auxiliary discriminative classifier (ADC-GAN) resolve the problem of AC-GAN. Specifically, the pro- posed auxiliary discriminative classifier becomes generator-aware by recognizing the labels of the real data and the generated data discriminativelly Our theoretical analysis reveals that the generator can faithfully replicate the target distribution even without the original discriminator, making the proposed ADC-GAN robust and stable during the training process. Extensive experi- mental results on synthetic and real-world datasets demonstrate the superiority of ADC-GAN on conditional generative modeling compared to competing methods.
to the
INTRODUCTION
Generative adversarial networks (GANs) (Goodfellow et al. 2014) have been gained great progress in learning high- digh-diimnsional,, complex data distribution such as natural images (Karras al. [019] 2020b at Brock et al., (019). Standard GANs consist of a generator network that transfers latent code sampled from a tractable distribution in the latent space to a data point the data space and a discriminator network that attempts to distinguish between the real data and the generated one. The generator trained in an adversarial game against the discriminator such that it can replicate the data distribution at the Nash equilibrium of the game. Remarkably, the training of GANs is notoriously unstable to reach the equilibrium, and thereby the generator prone to mode collapse (Salimans) et al. [016 Lin et al., 2018 Chen al. 2019). In addition, practitioners are interested in controlling the properties of the generated samples (Yan et al. (2015] Tan et al. 2020) in practical applications.
A key solution to address the above issues is conditioning, leading conditional GANs (Mirza & Osindero 2014).
Conditional GANs (cGANs) is family variant of GANs that leverages the side information from annotated labels of samples to implement and train a conditional generator, and therefore achieve conditional image generation from class-label (Odena al.) 2017 MMyato & Koyama 2018 Brock et al. 2019) or text (Reed et al. 2016)] Xu et al. 2018 et al. 2019). To implement the conditional generator, the common technique nowadays injects the conditional information via conditional batch normalization (de Vries et al. 2017). To train the conditional generator, a lot of efforts focus on effectively injecting the conditional information into the discriminator or classifier (Odenal 2016 Miyato & Koyama| 2018 Zhou et al., 2018 Kavalerov et al. [022]] Kang & Park 2ho0 et al. 2020). Among them, the auxiliary classifier generative adversarial network (AC-GAN) (Odena et al, 2017 has been widely used due to its simplicity and extensibility. Specifically, AC-GAN utilizes an auxiliary classifier that first attempts to recognize the label of data and then teaches the generator to produce label-consistent (classifiable) data. However, it has been reported that AC-GAN suffers from the low intra-class diversity problem on generated samples, especially on datasets with large number of classes (Odena et al. 2017 Shu al. 2017 Gong al. 2019).

---

Under review as a conference paper at ICLR 2022
In this paper, point out that the fundamental reason for the low intra-class diversity problem of AC-GAN is that the classifier is agnostic to the generated data distribution and thus cannot provide informative guidance to the generator in learning the target distribution. Motivated by this obser- vation, we propose a novel conditional GAN with an auxiliary discriminative classifier, namely ADC-GAN, resolve the problem of AC-GAN by enabling the classifier to be aware of the gen- erated data distribution. To this end, the discriminative classifier is trained to distinguish between the real and generated data while recognizing their labels. The discriminative property enables the classifier to provide the discrepancy between the real and generated data distributions analogy to the discriminator, and the classification property allows it to capture the dependencies between the data and labels. We show in theory that the generator of the proposed ADC-GAN can replicate the joint data and label distribution under the guidance of the discriminative classifier at the optima even without the discriminator, making our method robust to hyper-parameet
and stable on training. We also discuss the superiority of ADC-GAN compared to two most related works (TAC-GAN (Gong et al. 2019) and PD-GAN (Miyato & Koyama, 2018)) by analyzing their potential issues and lim- itations. Experimental results clearly show that the proposed ADC-GAN successfully resolves the problem of AC-GAN by faithfully learning the real joint data and label distribution. The advantages over competing eGANs in experiments conducted on both synthetic and real-world datasets verify the effectiveness of the proposed ADC-GAN in conditional generative modeling.
PRELIMINARIES AND OUR ANALYSIS
2.1 GENERATIVE ADVERSARIAL NETWORKS
Generative adversarial networks (GANs) (Goodfellow et al. 2014) consist of two types of neural networks: the generator G: Z + X that maps a latent code 2 Z endowed with an easily sampled distribution Pz to a data point x E X, and the discriminator D X' -> [0. 1] that distinguishes between real data that sampled from the real data distribution Px and fake data that sampled from the generated data distribution Qx G Pz implied by the generator. The goal of the generator is to confuse the discriminator by producing data that is as real as possible. Formally, the objective functions for the discriminator and the generator defined as follows:
min max D) = Ex [log [(z)] + E [log(1 - D(x))). D
(1)
Theoretically, the learning of generator under an optimal discriminator can be regarded as minimiz- ing the Jensen-Shannon (JS) divergence between the real data distribution and the generated data distribution, i.e., ming JS(Px x1Qxx. This would enable the generator to recover the real data distri- bution at its optima. However, the training of GANs is notoriously unstable, especially when lacking additional supervision such as conditional information. Moreover, the content of the generated im- ages of GANs cannot be specified in advance.
2.2 AC-GAN
Learning GANs with conditional information can not only improve the training stability and gen- eration quality of GANs but also achieve conditional generation, which has more practical value than unconditional generation in real-world applications. One of the most representative conditional GANs is AC-GAN (Odena et al. 2017). which utilizes an auxiliary classifier C : X' - ) to learn the dependencies between the real data x ~ Px and the label y Py and then enforce the condi- tional generator G 2 x ) + X' to synthesize classifiable data as much as possible. The objective functions for the discriminator D, the auxiliary classifier C, and the generator G of AC-GAN are defined as follows!
max V(G, D) + (E log Cly)=))). D.C
min V(G, D) - (En.y-xx. () (log C(ylz)))
where À > 0 is a thyer--paaaeetee and Qx, = G (Pz x Py) denotes the joint distribution of generated data and labels implied by the generator.
We follow the common practice in the literature to adopt the stable version instead of the original one.
2

---

Under review as a conference paper at ICLR 2022
share
share
share -
share
Dp
D
C
Cml
D
Ca
real
fake
real
real & fake
X (a) PD-GAN
(b) AC-GAN
(c) TAC-GAN
(d) ADC-GAN
Figure 1: Illustration of discriminatoos/laasiffeess of existing conditional GANs (PD-GAN Miyato & Koyama 2018), AC-GAN (Odena al. 2017). and TAC-GAN (Gong al. 2019) and the proposed ADC-GAN. indicates real (1 1) or fake (1 = 0) and y is the class-label of data x. ADC- GAN is different from PD-GAN with explicitly predicting the label and different with AC-GAN and TAC-GAN that the classifier Cj also distinguishes real from fake like the discriminator.
Proposition 1. The optimal classifier of AC-GAN outputs as follows:
(²(yxz) = p(x,, v)
(3)
p(x)
Theorem 1. Given the optimal classifier, at the equilibrium point, optimizing the classification task for the generator of AC-GAN is equivalent to:
min KL(Qxx -L(())))))))) G
where - (Z, y) log ((yll))dr dx is the conditional entropy of generated data.
The proofs of Proposition and Theorem I are referred Appendix A. and A.2 respectively.
Our Theorem [ exposes two shortcomings of AC-GAN. First, maximization of the KL divergence between the marginal generator distribution and the marginal data distribution maxç KL(Qx Pxx) contradicts the goal of conditional generative modeling that matches Qx.y with Px,y. Although this issue can be mitigated to some extent by the adversarial training objective between the discrimi- nator and the generator that minimizes the JS divergence between the two marginal distributions, we find that it still has negative impact on the training stability. Second, minimization of the entropy of label conditioned on data with respect to the generated distribution ming H (Y]X) will result in that the label generated data should be completely determined by the data itself. In other words, it will force the generated data of each class away from the classification hyper-plane, explaining the low intra-class diversity generated samples in AC-GAN especially when the distributions of different classes have negliiiie overlap, which is supported by the fact that state-of offthe-an clas- sifiers nor human cannot achieve 100% accuracy on real-world datasets (Russakovsky et al. 2015). Note that the original version of AC-GAN, whose classifier is trained by both real and generated samples, could also suffer from the same issue (see Appendix B
3 THE PROPOSED METHOD: ADC-GAN
The goal of conditional generative modeling is to faithfully approximate the joint distribution of real data and labels regardless of the shape of the target joint distribution (whether there is overlap between distributions of different classes). Note that the learning of the generator in AC-GAN is affected by the classifier. In other words, the reason for the consequence of Theorem D originates from Proposition I] which indicates that the optimal classifier of AC-GAN is agnostic the density of the generated (marginal or joint) distribution (q(x) or q(z, y)). Therefore, the classifier cannot provide the discrepancy between the target distribution and the generated distribution, resulting in
3

---

Under review as a conference paper at ICLR 2022
Table Comparison of objective of the generator under the optimal discriminator and classifier.
Method
Objective of the generator under the optimal discriminator and classifier
AC-GAN
ming JS( (Sx Qx) + (KL(Qx,y
Px,y)
Kl.(Qx Px) + Ho(VIx))
TAC-GAN
ming JS( Px Qx) + X-
(KL(Qx,Y
Px,y)
KL Px))
ADC-GAN
ming JS Qx) + (KL(Qx. Px,)))
PD-GAN
ming (Ixx))
biased learning objective to the generator. Recall that the optimal discriminator (* p(a) +(a
p(a)
able to be aware of the real data density as well as the generated data density (Goodfellow et al,
2014), and thus can provide the discrepancy p(z)
g(z) [=D"(z) between the real data distribution
(a)
and the generated data distribution unbiasedly optimize the generator. Intuitively, the density- aware ability on both real and generated data is caused by the fact that the discriminator attempts to distinguish between real and fake samples. Motivated by this observation, we propose make the classifier to be distinguishable between real and fake samples, establishing discriminative classifier C2 : X' + V x {0. 1} that recognizes the label of real and fake samples discriminatively. Formally, the objective functions for the discriminator D, the discriminative classifier Co, and the generator G of the proposed ADC-GAN are defined as follows:
max D) + X- (Ez.y~Px,y [log Caly, 1|x)] Ex,y olg Ca C/())0/=)))= D,Ca
(5)
min V(G, D) X (Bn.y-Qx. [log Ca(y. [z=) - Ex,yQux [log (y
where Caly, 1/x) (reps. Cy,,, 0|z)) denotes the probability that data is classified the label and real (reps. fake) data simultaneously.
Proposition 2. For fixed generator, the optimal classifier ADC-GAN outputs as follows:
p(x, v)
q(x v)
Cy 11x) p(x) + ((x)
(g. p(x) + ((z)
(6)
The proof is referred to Appendix A.3 Proposition 2 confirms that the discriminative classifier be aware of the densities of the real and generated joint distributions, therefore it is able to provide the discrepancy p(z.g) ([z) to unbiasedly optimize the generator as we prove below.
((2,9) (s.0l))
Theorem 2. Given the optimal classifier, at the equilibrium point, optimizing the classification task
for the generator of ADC- GAN is equivalent to:
min KL(Qx,y Px,x))
The proof is referred to Appendix A.4 Theorem suggests that the classifier itself can guarantee the generator to replicate the real joint distribution in theory regardless of the shape of the joint distribution. In practice, we retain the discriminator to train the generator and share all layers but the head of the classifier with the discriminator illustrated in Figure [] and Equation 5 for faster convergence speed. Coupled with the adversarial training against the discriminator, the generator of the proposed ADC-GAN, under the optimal discriminator and classifier, can be regarded as min- imizing the following divergences: ming JS "S(Pxlll xx + A KL(Qx.y |
| Since the optimal solution of conditional generative modeling belongs to the optimal solution set of generative mod- eling, i.e., arg ming KL(Qx.y Px,x) C arg ming JS( x [x).. learning with the discriminator will not change the convergence point of the generator that approximates the joint distribution of real data and labels regardless of the value of hyper-parameter À > 0. Furthermore, the hyper-parameter provides the flexibility to adjust the weight conditional generative modeling.
DISCUSSION ON COMPETING METHODS
In this section, we analyze the drawbacks of the two competing methods, TAC-GAN (Gong et al.) 2019) and PD-GAN (Miyato & Koyama 2018). to demonstrate the superiority and rationality of ADC-GAN compared them. Before diving into the details, we show diagrams of the discriminator and classifier of these methods in Figure I and summarize the theoretical learning goal for the generator under the optimal discriminator and classifier of these methods in Table overview.
4

---

Under review as a conference paper at ICLR 2022
4.1 TAC-GAN
TAC-GAN (Gong et al., 2019) addresses the low intra-class diversity issue of AC-GAN by eliminat- ing the conditional entropy with respect the generated data distribution H)(Y)X) via learning of the generator with another classifier Cm X + V/ which is trained on the generated samples. The objective functions for the discriminator D, the twin classifiers C' and Coai. and the generator of TAC-GAN are defined as follows:
max V(G, D) + A Pxx Dog C(yla)) Ex, llg (mc(y/=)))
D,C,Cm.
(8)
min V(G, X (Em.y-ux.y [log C(ul2:)] Rx. (log (mi(y]z))) G
Theorem 3. Given the twin optimal classifiers, at the equilibrium point, optimizing the classification tasks for the generator of TAC-GAN is equivalent to:
min KL(Qx,v (I Px,))
KL(Qx (Px).
The proof is referred to Appendix A.3
Theorem reveals that the learning objective of the generator TAC--AN,, under the optimal classifier, can be regarded as minimizing contradictory divergences, Le., minimization between joint distributions but maximization between marginal distributions. Al- though theoretically the IS divergence or others (Nowozin et al., 2016: Arjovsky et al.) 2017) intro- duced through the adversarial training between the discriminator and the generator might remedy this issue, the optimal discriminator and classifier are difficult to obtain the practical optimization ensure that the contradiction is eliminated. We argue that the training instability of TAC-GAN reported in the literature (Kocaoglu et al. 2018: Han et al., 2020) and founded in our experiments can be explained by this analysis and interpretation.
4.2 PD-GAN
PD-GAN (Miyato & Koyama 2018) injects the conditional information into the projection dis- criminator Dp X
0, IJ via the inner-product between the embedding of label and the
representation of data to calculate the joint discriminative score of the data-label pair. In such way, PD-GAN inherits the property of convergence point similar to the standard GAN such that can avoid the low intra-class diversity issue of AC-GAN. The objective functions for the projection discriminator Dp and the generator G of PD-GAN are defined as follows:
min max Dp) Ex [log Dp (Ir, y) Eu,y [log(1 - Dp(x, v))l
(10)
Dr
Based on this minimax game, the optimal projection discriminator has the following form:
p(z, y)
Dz(r, u) = +xeepp-d (xsp(dd'ii d' (I,
+
p(x, v) + (( v)
(11)
d d' y) = log p( y)
log p(x)
+ log p(y|x)
= r(x) r(ylz), (y(x)
(xx, y)
(xx)
exp())-d(a)))
exp(u
where p(yla)) (z) = Σ x(z)) and ((yla)) = Σx exp(ef-p(x) with K = |>|| is the number of
labels. And they accordingly define:
r(x) = ())x)))
K
r(ylz) = (ug og)) 6(x))
log
exp (v. d(x)) - log exp (o x(x))
(12)
F(glz))
However, PD-GAN actually ignores the partition term of the projection discriminator in the form of:
@ Equation 12 and constructs the logit
d(z, y) = r(x) + f(g(=)) (p(x)) + by o(z),
(13)
"The authors mistakenly argue that @ can be merged into r(æ). However, r(æ) does not incorporate any label information (vP or n.), which should he considered by @ Therefore, it unreasonable to merge @ into r(z). PD-GAN actually discards @ in implementing the projection discriminator.
5

---

Under review as a conference paper at ICLR 2022
Data
Log loss
Hinge
PD-GAN
AC-GAN
TAC-GAN
ADC-GAN
Figure Distribution learning results on one-dimensional synthetic data.
with by = D oy is the difference between two learnable embeddings of label y defined in two implicit conditional probabilities p(ylz) and ((y) .r), o(-) the representation extractor, and (+)) outputs scalar based on the extracted representation. Discarding the partition term would make PD-GAN no longer belong probability models that model the conditional probabilities p(y]æ) and g(v x), resulting in losing the complete dependencies between data and labels. Moreover, the discriminator constructed according to the optimal form of the minimax GAN lacks theoretical guar- antee when applied on other loss functions such as the hinge loss (Lim & Ye 2017 Tran et al. 2017). which PD-GAN actually used, and may even limit its discriminative ability. The proposed ADC- GAN be flexibly applied to any version of the loss function V(G, D) do not require the specific form the original discriminator.
5 EXPERIMENTS
In this section, conduct extensive experiments both synthetic and real-world datasets to demonstrate the effectiveness of the proposed ADC-GAN, Specifically, ADC-GAN has the advan- tages of being robust the hyper-parameter X, stable during the training process, and capable of accurately modeling dependencies between data and labels, in addition to the comparable or even better performance on conditional generative modeling than existing eGANs.
5.1 SYNTHETIC DATA
We first experiment on a one-dimensional synthetic mixture of Gaussian to validate the distribution learning ability of methods. As shown in the left-top of Figure the real data distribution sists three classes which there non-negligible overlap between them. Both generator and discriminator multi-layer perceptrons with non-linearity of Tanh. In particular, three different settings on the GAN loss function while keeping the learning of the generator with the classifier fixed if it exists. The first row except the data part shows the learned distributions that are estimated by kernel density estimation (Parzen) 1962) on the generated data of AC-GAN, TAC-GAN, and ADC-GAN without the original GAN loss VIG, D). The second and the third rows show the results of these methods trained with the log loss (Goodfellow et al.) 2014) and the hinge loss (Lim Ye 2017 Tran et al. 2017). respectively. The poor performance of PD-GAN with hinge loss confirms that it is sensitive to the loss function. AC-GAN tends to generate classifiable data so that it decreases the intra-class diversity all settings, verifying the Theorem TAC-GAN without GAN loss cannot accurately reproduce the data distribution, verifying the Theorem 3 And the worse performance TAC-GAN with hinge loss compared with ADC-GAN confirms that the contradiction stated in Theorem is not easy to climinate by the discriminator. Expectedly, the pro- posed ADC-GAN can accurately replicate the data distribution even without the original GAN loss, verifying the Theorem 2 For more quantitative results, please refer to Appendix C.1
investigate

---

Under review as a conference paper at ICLR 2022
000000 000000001
666000
12200
002000 201.040211002200000000
= 100
02200022220000
AC-GAN
TAC-GAN
ADC-GAN
Figure 3: Hyper-parameter robustness results on overlapping MNIST.
5.2 OVERLAPPING MNIST
In this subsection, experiment on a constructed dataset to investigate the hyper-parameter ro- bustness of the proposed ADC-GAN compared to existing blsssifierbbsee methods, i.e., AC-GAN and TAC-GAN. We follow the practice of (Gong et al. 2019) to construct two-class handwritten equal number of digits of '0' and 'I' and the
digit dataset from MNIST. The first class contains second class contains an equal number of digits 'O' and '2'. In other words, the support regions of the two classes have overlap of digits '0'. We change the weight of classifier from À = 1 to = 100 with multiplicative step 10. As shown in Figure 3 AC-GAN mistakenly discards digit '0' in the first class when À 1. When À 10, the digit "O' in the generated data of AC-GAN totally disappears, indicating that its classifier encourages the generator to avoid learning the data in the overlapping region. In general, AC-GAN shows significant decrease intra-class diversity. TAC-GAN encounters mode collapse when X = 100, while ADC-GAN faithfully replicates the real data distribution regardless of the value of A. These results suggest that the proposed method has excellent robustness on typee--asameters compared to existing classifier- methods since the guidances the generator received from the discriminator and classifier are harmonious.
5.3 CIFAR-10, CIFAR-100, AND TINY-IMAGENET
We experiment three real-world datasets: CIFAR-10, CIFAR-100, and Tiny-Im CIFAR- 10 consists of 50k training images and 10k validation images with resolution of 32 x 32. CIFAR- 100 runs similar samples with CIFAR-10 but has 100 classes rather than 10 classes in CIFAR- 10. Tiny-ImageNer contains 200 classes where each class contains 500 training images and 50 validation images with resolution of 64 64. We implement all methods based on the BigGAN- PyTorch repository. The optimizer is Adam with learning rate of 2 x 10-4 for both the generator and discriminator on CEFARR11//00 respectively, on Iiny-ImageNet. We train all methods for 500 epochs with batch size of 50 on CIFAR-10/100 and 100 on Image The discriminatoor/llssiieee are updated 4 times per generator update step CIFAR- and 2 times Tiny ImageNet. We follow the practice of (Miyato & Koyama
10 and 10 for the generator and discriminator,
2018
Gong et al. 2019) to adopt the hinge loss (Lim & Ye! 2017 Tran
Table 2: FID and Intra-FII (the averaged intra-class FID) and Accuracy (%) comparison of different methods on CIFAR-10, CIFAR-100, and Tmny-ImageNet, respectively.
Datasets
Metrics
PD-GAN
AC-GAN
TAC-GAN
ADC-GAN
FID (L)
6.51
6.81
5.85
5.56
CIFAR-10
Intra-FID (L)
46.35
38.34
37.38
34.26
Accuracy (1)
62.21
84.56
88.05
89.19
FID (1)
8.47
11.63
11.37
7.98
CIFAR-100
Intra-FID (L)
136.79
161.39
158.15
132.27
Accuracy (t)
38.81
55.29
60.13
62.29
FID (L)
20.54
24.48
23.78
20.15
Tiny-ImageNet
Intra-FID (L)
94.23
154.82
124.68
97.69
Accuracy (1)
27.56
42.26
41.78
44.29

---

Under review as a conference paper at ICLR 2022
-0-
(a) CIFAR-10 FID on different X'
(b) FID curves on CIFAR-10
(c) PD-GAN
SCORE
(d) CIFAR- 100 FID on different X
(e) FID curves ZIFAR-100
(f) ADC-GAN
Figure 4: (a.d) FID comparison of methods with different X' on CIFAR-10 10 and CIFAR-100. The objective of competing methods is (1 (G, D) + XVeGG, C), where Ve C) is the task between the generator and classifier. (b.e) FID curves with GAN training iterations on CIFAR-10 10 and CIFAR- (c.f) T-SNE visualization of CIFAR-10 10 training data, using learned representations extracted from the penultimate layer in discriminators. Different colors represent different classes.
et al. 2017) as an implementation of (GG, We set the hyper-parameter of AC-GAN as À = 0.2 on all datasets it performs the best. As for TAC-GAN and ADC-GAN, the hyper-parameter is set as x = 1.0 on CIFAR- and À = 0.5 on Iny--mmgeNet.
We report the FID (Heusel et al., 2017) and Intra-FID (Miyato & Koyama 2018) (the FID for each class) results of all methods on CIFAR-10 10, CIFAR-100, and Viny-ImageNe in Table 2 AC- GAN obtains the worst results and diverges on all datasets (see Figure (b) (e 7). TAC-GAN also diverges on CIFAR-100 and Tiny- ImageNet and only achieves a relatively stable FID training curve on CIFAR-10. We here report their results at the best FID checkpoint. These unstable FID curves implicitly reveal the drawback of the existing classifier- divergences. To explicitly show this, we set the objective function classifier
cGANs that minimizes contradictory
methods as (1 (G, D) + XVV((G. C), where Vc(G, C) is the task between the generator and classifier. As shown in Figure 4(a) 4(d) ADC-GAN gains consistent FID results across different X' even for X. while AC-GAN and TAC-GAN performs substantially bad when X' becomes large. Table 2 also shows that ADC-GAN achieves the superior FID and Intra-FID on CIFAR- 10 and CIFAR- 100 and comparable scores with PD-GAN on Ting-ImageNet. The reason why PD-GAN obtains slightly better Intra-FID than ADC- GAN on Tiny-ImageNet that Tiny-ImageNet is a more coarse-graie coarse-grained dataset, in which data in different classes have weak correlations. Impressively, ADC-GAN performs much better training stability than PD-GAN in terms of the FID curves as shown in Figure 4(b) 4 We argue that the reason is that ADC-GAN is less prone to over-fitting than PD-GAN due to that the discriminative classifier of ADC-GAN solves more difficult task than the projection discriminator
X = 1.0 (i.e., without the discriminator), showing strong robustness
PD-GAN.
To investigate whether the model captures accurate dependencies between data and labels, we con- duct image classification experiments based on the learned representations extracted from the shared penultimate layer in the discriminator/Classinn classifier in seikit-learn library with default settings to compute the accuracy results. As reported in Table 2 ADC-GAN significantly outperforms competing methods all datasets, indicating that ADC-GAN effectively learns accurate dependencies between data and labels. The reason is that the discriminative classifier needs to distinguish between real and fake data while simultaneously recog-
Specifically, we utilize the logistical regression
8

---

Under review as a conference paper at ICLR 2022
Table 3: FID and IS comparison with competing methods on ImageNet. The results of competing methods are copied from TAC-GAN (Gong et al. 2019) and SAGAN (Zhang et al. 2019)
Datasets
Metrics
PD-GAN/BigGAN
AC-GAN
TAC-GAN
ADC-GAN
FID (1)
22.77
23.75 28.86 0.29
16.75
ImageNet
IS (T)
38.05 0.79
28.5
55.43 + 0.90
nizing the labels of samples, which forces the classifier to have a more powerful and robust ability of modeling data-to-class relations. Notice that PD-GAN obtains the worst accuracy results. By com- paring the CIFAR-10 10 T-SNE (Van der Maaten & Hinton| 2008) visualization results of PD-GAN and ADC-GAN using the learned features of validation data in Figure [ 4f) one clearly see that PD-GAN is incapable of learning accurate dependencies between data and labels.
5.4 IMAGENET
In this subsection, we compare the proposed ADC-GAN with competing methods on ImageNet (128 x 128) with 1,000 classes (Deng et al., 2009), each of which contains around 1, 1,300 images. We adopt the BigGAN with base channels of 64 as the backbone of ADC-GAN and train one step for the discriminator/llassfeer and one step for the generator, following the practices of TAC-GAN (Gong) ctt al, 2019). We follow the instruction of FQ-GAN (Zhao et al. 2020) to train ADC-GAN for 128k iterations. As shown in T the proposed ADC-GAN significantly outperforms competing cGANs in terms of both Inception Score (IS) (Salimans et al. 2016) and FID metrics, showing the effectiveness large-scale high- high-resolution image datasets.
6 RELATED WORK
Conditional generative adversarial networks (cGANs) (Mirza & Osindero, 2014) is a family of GANs, which is capable of generating novel data depended on the given label. The research on cGANs can be divided into two aspects. One is to study how to implement conditional generator network structurally, Approaches in this category are mainly concatenating the label vector with the noise vector (Mirza & Osindero 2014), conditional batch normalization (de Vries et al. 2017), and conditional convolution layers (Sagong et al., 2019). The other is to study how to train the condi- tional generator to generate label-dependent data. AC-GAN (Odena al. (2017) leveraged iliary classifier to determine the relationship between data and labels. MH-GAN (Kavalerov et al.] 2021 improved AC-GAN by replacing the cross-entropy loss of the classifier with the multi-hinge loss. Shu et al. (0177) analyzed that AC-GAN learns a biased distribution from a Lagrange view. 2018) extended the real class of the discriminator into K classes of the clas- sifier, forming a discriminator with K + 1 classes. Omni-GAN (Zhou et al. 2020) combined the dis- criminator with the classifier into K + dimensional multi-label discriminator. TAC-GAN (Gong et al. (019)) corrected the biased learning objective of AC-GAN by introducing another classifier, which was the multi-class version Anti-Labeler of CausalGAN (Kocaoglu et al. 2018). UAC- GAN (Han et al. (2200) improved the training stability of TAC-GAN via MINE (Belghazi et al.) 2018 PD-GAN (Miyato & Koyama| 2018) injected the label information into the discriminator via projection technique. In complementary
AM-GAN (Zhou
our work, ContraGAN (Kang & Park] (020) modeled the data-to-data relations as well as the data-to-class relations using a conditional contrastive loss.
7 CONCLUSIONS
In this paper, we present novel conditional generative adversarial network with an auxiliary dis- criminative classifier (ADC-GAN) to achieve faithful conditional generative modeling. The dis- criminative classifier can provide the discrepancy between the joint distribution of the real data and labels and that of the generated data and labels to the generator by discriminatively predicting the la- bel of the real and generated data. Therefore, the generator can faithfully learn the real joint data and label distribution at the Nash equilibrium. We also discuss the differences between ADC-GAN with competing cGANs and analyze their potential issues and limitations. Extensive experimental results validate the theoretical superiority of the proposed ADC-GAN compared to competing cGANs.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial net- works. In Doina Precup and Yee Whye Teh (eds.). Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214-223. PMLR, 06-11 Aug 2017. URL http:://rroceddngs...... arjovsky17a.htm
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Jennifer Dy and Andreas Krause (eds.), Proceedings the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 531-540. PMLR, 10-15 Jul 2018. URL bttps::/pprceet www.innnnss......n.mmmm......iem www.m.rrr...ese.... v be wwt.////tte..iiiae htt.ghaiiiiaaam..mm
Andrew Brock, Jeff Donahue, and Karen Simonyan.
Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
https:://ppeneviier net http:/ewwwwwee///.feeee.e... F
Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, and Neil Houlsby. Self-supervised gans via auxiliary rotation loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Harm de Vries, Florian Strub, Jeremie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso-
ciates, Inc., 2017. URL 6fab6e33a3424488 e3 htw.-aaeeddecoooc--aaaeeeeeee pddf
hwwp::/ww....................... meurips.... wttp://wss.cccpp///////////////. http/ppper2211//111e
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Mingming Gong, Yanwu Xu, Chunyuan Li, Kun Zhang, and Kayhan Batmanghelich. Twin auxi- lary classifiers gan. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buce Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019, URL httpttss:: http::::s//p/cceedingg.o newriieurpppp... cc/paperr2019/filee Papp:e/addbbbeeeeaaaae..ee. Paper.pds
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL Baa2899122FA1110655eeclbbaaaaaae
neurips. ce/ http://ppp...cpppp////11//// cc/ppapprr/211//11eeee pdf
Ligong Han, Anastasis Stathopoulos, Tao Xue, and Dimitris Metaxas. Unbiased auxiliary classifier gans with mine. arXiv preprint arXiv:2006. 200::0006.05677. 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by two time-scale update rule converge local nash equilibrium. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https:: httpps::/procceed.iitt newrisseriipp.... http://c.......o..paaaee///////. cctpppper/211////eeee.
Paper. pdf
Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image gen- eration. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 21357-21369. Curran As- sociates, Inc., 2020. URL http::ttp///tt h htw.::/eeeeddn......ii.eee....e. httpappps.cccpppe///////111 F490c742cd833 (wwe88888eeeee wa
pd
10

---

Under review as a conference paper at ICLR 2022
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila, Training generative adversarial networks with limited data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12104-12114. Curran Associates, Inc., 2020a. URL https:: neurips. cctpappe///2///////c f1le http////ee/B///aaaaaaaaaa Bd3 96e 2440759 040e440055-0002220001111aaa2222-
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020b.
Iya Kavalerov, Wojeiech Czaja, and Rama Chellappa. A multi-class hinge loss for conditional gans. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1290-1299, January 2021.
Murat Kocaoglu, Christopher Snyder, Alexandros G. Dimakis, and Sriram Vishwanath. Causal- GAN: Learning causal implicit generative models with adversarial training. In International Conference on Learning Reprexentations, 2018. URLhttps: httpss////peeeeeeeeeeeee net forum?
id=BJE 4xWOW
Jac Hyun Lim and Jong Chul Ye. Geometric gan. arXiv preprint 17X.::7705.02894, 2017.
Zinan Lin, Ashish Khetan, Giulia Fanti, and Sewoong Oh. Pacgan: The power of two samples in generative adversarial networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018 URL https: httttpp://proceeddngg..s. hew.oeeeedddgg...uu.pss...... nww.ips.e.pp.c. cc/ saper.pp..
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.178. 2014.
Takeru Miyato and Masanori Koyama, cGANs with projection discriminator. In International Conference Learning Representations, 2018. URLLtttps://ppeereviieee hwtp://w..ee.frrooeeeeeeeo forum?
1d=ByS1VpgRZ)
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran As- sociates, Inc., 2016. URL / cedebb6e 72 8c3 (at-F=9119874499---pp-- pddf
Augustus Odena. Semi-supervised learning with generative adversarial networks. arXiv preprint arXiv:1606.01583, 2016. 2016.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with aux- iliary classifier GANs. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th In- ternational Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2642-2651. PMLR, 06-11 Aug 2017. URL httt:://prcceedinggg seaeediiinnsm..........mmm .mlr. p2ess//70/odenal///5
Emanuel Parzen. On estimation of a probability density function and mode. The annals of mathe- matical statisties, 33(3):1065-1076, 1962.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1660-1069, New York, New York, USA, 20-22 Jun 2016. PMLR. URL htllhttp::/ htttt::/prrocediii ngs ml www.spressssee reed1 6.mtml

---

Under review as a conference paper at ICLR 2022
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision. 115(3):111252, 2015.
Min-Cheol Sagong, Yong-Goo Shin, Yoon-Jae Yeo, Seung Park, and Sung-Jea Ko. cgans with conditional convolution layer. arXiv preprint arXiv:1906.0000 1906000000, 2019.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, Cur- ran Associates, Inc., 2016, URL
nwwrips....... cttppcpeppp./1/////////// pdf
Fi File/ 02/3/B33333333 abe 7 22db2d8 76 0166666660630000662230
Rui Shu, Hung Bui, and Stefano Ermon. Ac-gan learns biased distribution. In NIPS Workshop on Bayesian Deep Learning, volume 8, 2017.
Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Lu Yuan, Sergey Tulyakov, and Nenghai Yu. Michigan: Aulti-input-condiioonne hair image generation for portrait editing. arXiv preprint arXiv:2010. 2020.
Dustin Tran, Rajesh Ranganath, and David Blei. Hierarchical implicit models and likelihood- free variational inference. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpttppp://ppcceeiiiiii Papp:aao00033---Pe.p.pp
neurips. cc/
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Xinchen Yan, Jimei Yang, Kihyuk Sohn, and Honglak Lee. Atrrbbuteimaage: Conditional image generation from visual attributes. arXiv preprint arXiv:1512.00550, 2015.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7354-7363. PMLR, 09-15 Jun 2019. URL http::ttpp/pett http:://rroceddings.
mlr. mlr.press/v2 0axzess/997/haag111 mtazizaaagggg1 html
Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen. Feature quantization im- proves GAN training. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th In- ternational Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 11376-11386. PMLR, 13-18 Jul 2020. URL htllatts/ //Lp//ecp///p mmr. press/vi19/ htw.1/99/hha22....h.o.
Peng Zhou, Lingxi Xie, Bingbing Ni, Cong Geng, and Qi Tian. Omni-gan: On the secrets of cgans and beyond. arXiv preprint 1.. 2020.
Zhiming Zhou, Han Cai, Shu Rong. Yuxuan Song, Kan Ren, Weinan Zhang, Jun Wang. and Yong Yu. Activation maximization generative adversarial nets. In International Conference Learning Representations, 2018 8.
net. forat//forumfii fo
Minfeng Zhu, Pingho Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative ad- versarial networks for -otttto-mmag synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ( (CVPR). June 2019.
12

---

Under review as a conference paper at ICLR 2022
A PROOFS
A.1 PROOF OF PROPOSITION I
Proposition 1. The optimal classifier of AC-GAN outputs as follows:
(*(la²) p(a,y)
(3)
p(x)
Proof.
max R Px. log C(ylz)) = Px By- Prix Bog C(ylar)
C
7 min Px Pr Cyylz)] = Ex [H(p()(z)) + (LLobba)))))))
C
+ (yloz) arg min KLlolloo p(ul>)) p(z,,) 3)
p(x)
A.2 PROOF OF THEOREM I
Theorem 1. Given the optimal classifier, at the equilibrium point, optimizing the classification task for the generator AC-GAN is equivalent to:
min KL(Qxx
where - (x, log ((y)x))d is the conditional entropy of generated data.
Proof.
log p(x, p(x)
p(z, v) ((x) q(2, v) -Ex log q(2, y) p(x) q(x)
maxE maxEz.y Rxxux [log (y(2=) = Ex
p(z,)) v)
log g(x)
Exyy log ((,,))
=E log v(x,8)
+ Ex
p(x)
((x)
min (L,,xXXyIIIX.
KL[(Q (a(Qx||Px)) Px) + (YIx)
A.3 PROOF OF PROPOSITION 2
Proposition 2. For fixed generator, the optimal classifier of ADC-GAN outputs as follows:
p(x, y)
(3. = q(x,y)
Cg.
(6)
+ +(z)+
Proof.
max Ex, Dog Cly. 1|x)) Eu,y ((o.) = max E I... Dog C(y, [[))]
with E (0,1) and ndh (x, 1) )0 1) +1" 5 v) + _ v). [[]) min Ez Ey pm
max Eew Eg [lgg
C [log
pm Pr |- log Cly, [x]]
min min Py H ( xx) KL (v. (1)
( 1dH (x.
C3,y, = arg min
= ([x)
C
1 (T)
Therefore, the optimal classifier of ADC-GAN has the form of C* (y, 1|x) = (2,0,1
p(x.))
(x))
p())+g(2)
and C* (v, 0|z) (2,x0))
((2.))
that concludes the proof.
(²)
p(2)+9(z)
13

---

Under review as a conference paper at ICLR 2022
A.4 PROOF OF THEOREM 2
Theorem 2. Given the optimal classifier, at the equilibrium point, optimizing the classification task for the generalor ADC-GAN is equivalent lo:
min (L(QXxx(XIxx))
G
Proof.
max Bxy. [log C" 1[r)] Qx., [log Cy( 0/z)]
min
- -x Dlg C* C* [x))] Bry-Qxx (log C (v, 0 0[))]
p(z,1 y)
Em log g( y) p(x) + ((x)
+ min Qxx
log p(x) + g(x))
G
log ((. y)
min Ez Qx.y
min KL(Qx.y||Pxxx
G
p(r,z) G
A.5 PROOF OF THEOREM
3
Proposition 3. For fixed generator, the twin optimal classifiers of TAC-GAN output as follows:
p(z, v)
C (yl()) 9(x, y)
C"
(14)
p(x)
((x))
Proof. The proof is similar to that of Proposition T in Appendix [.. by considering C and Cest as two independent classifiers with respect to distribution P and Q, respectively.
Theorem 3. Given the twin optimal classifiers, the equilibrium point, optimizing the classification tasks for the generator of TAC-GAN is equivalent to:
min KI((Q.y|IPxxvv - -
(9)
Proof.
max Eg Br Qxx [log C'(y|=)]
Ex Qx. flog (n)())
G
log p(x, v)
Eny-exx log g(x, v)
max En Ux +
G
p(x:)
((x))
log p(x,, y)
EaNQx log p(x)
+ max BuyyQxxx
G
((x, v)
((x))
min (L(XXXX G
[L(Qx|IPx)
B ISSUE OF THE ORIGINAL AC-GAN
In this section, show that original AC-GAN whose auxiliary classifier is trained with both real and fake samples still suffer from the issue proved in Theorem Formally, the full objective function of the original AC-GAN is formulated as follows.
max (G, + (Ex.y>Px, log C(yl=)] + Eu,y [log C(ylz)))
D.C
(15)
min (GG, D) - (En.y-ux. Dlog Cyylz))) G
The objective function for training the classifier can be rewritten
14

---

Under review as a conference paper at ICLR 2022
max Ex, yx, [log C(ylz)) + xx.. (log)(/)) + max Eax (log(( (16) C
with pr (z, v) (p(ar,) (x, v)) and p²) En (x, {((()) + g(x)). And can obtain the optimal classifier the following:
max Rx log (ogCl))) = min C
log log C(vlz))
min Bg
+ KL(P" (g))(()))
C
(17)
= C pr (vlx) pla, v) + (x,
p(z) + ((x))
Even though the conditional generator learns the joint real data and label distribution, i.e., ((, y) p(x, y) and ((x) = p(xx), the optimal classifier (y)) p(z.))+g(r_.)) p(a-v) will still provide p(#)+q(()
p(æ)
the objective stated in Theorem I to optimize the generator, which contains the conditional entropy of generated samples Ho! (Y ) that would reduce the intra-class diversity of generated samples. In other words, the discriminative classifier does not allow the generator to remain on the desired
distribution because it still provide momentum to update the generator, resulting biased learning objective for the generator.
15

---

Under review as a conference paper at ICLR 2022
C MORE RESULTS
C.I SYNTHETIC DATA
In this section, report more results on experiments conducted on the one-dimensional synthetic data and new two-dimensional synthetic data. The oinediimes data consists of three Gaussian components with (lo = 0. μ41 = 3, μ2 and σu 1, 2, 02 3, and the similar for the two- dimensional data. For implementing the generator, discriminator, and classifier, multi-layer perceptron with hidden size
three-layer and the Tanh non-linearity. The optimizer Adam with learning rate 0.002 and betas (31, 82) (0.5, 0.999). We train all methods for 40 epochs with batch size of 256. Table 4 reports the quantitative maximum mean discrepancy (MMD) results on the one-dimensional synthetic data conducted in Section 5.1 Lower MMD means better learning results. Figure 6 and Table show the qualitative and quantitative results, respectively, conducted on the two-dimensional synthetic Gaussian data. In general, the proposed ADC-GAN consistently replicates the data distribution under different loss function settings.
Data
wlo GAN loss
Hinge loss
PD-GAN
AC-GAN
TAC-GAN
ADC-GAN
Figure 5: Distribution learning results on the one-dimensional synthetic data.
Table 4: MMD (1) results of each method the onc-dimensional synthetic data.
GAN Loss
Class
PD-GAN
AC-GAN
TAC-GAN ADC-GAN
ClassO
1.83
0.05
0.06
Classl
27.32
5.99
0.06
No
Class2
27583.31
130.89
0.19
Marginal
2919.31
7.81
0.14
ClassO
0.02
0.13
0.01
0.06
Class
0.05
0.64
0.14
0.07
Log
Class2
0.10
639.55
0.90
0.19
Marginal
0.11
75.58
0.66
0.14
ClassO
11328.12
2.08
0.21
0.40
Class
10671.01
29.96
3.04
1.22
Hinge
Class2
8420.67
32011.93
93.08
8.55
Marginal
10218.60
3243.28
7.81
0.80
16

---

Under review as a conference paper at ICLR 2022
Dala
Wo GAN loss
Logloss
Hinge loss
PD-GAN
AC-GAN
TAC-GAN
ADC-GAN
Figure 6: Distribution learning results on the two-dimensional synthetic data.
Table 5: MMD (4) results of each method the two-dimensional synthetic data.
GAN Loss
Class
PD-GAN
AC-GAN
TAC-GAN
ADC-GAN
ClassO
1.28
1.55
0.07
Class
6.64
152.49
0.09
No
Class2
10491.93
671.89
0.11
Marginal
1085.61
166.03
0.02
ClassO
0.05
0.06
0.01
0.02
Classl
0.02
0.64
0.27
0.12
Log
Class2
0.73
119.91
0.52
0.86
Marginal
0.01
13.12
0.04
0.08
ClassO
1625.92
1.42
0.70
0.06
Classl
1138.02
1.43
5.75
0.23
Hinge
Class2
918.97
9440.54
33.34
0.06
Marginal
1203.51
1019.39
7.99
0.03
160
PD-GAN
140
TAC-GAN
ADC-GAN
120
50
100
150
200
250
GAN training itarations (x10")
Figure 7: FID curves with GAN training iterations Tmng-Im
17


Reviews:
[{"review_id": "sPLQC5gW_2w", "reviewer": "Reviewer_ebJs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n1. The paper presents interesting analysis of AC-GAN, TAC-GAN, PD-GAN. Especially the Theorem 3 reveals potential drawbacks of TAC-GAN.\n2. The experimental results on synthetic and real datasets demonstrate the superiority of ADC-GAN on conditional generative modeling tasks.\n\n## Weaknesses\n1. I don't fully agree with some claims made by the authors:\n   1. Page 5, footnote 2, it might be true that term (a) is \"ignored\" or set to zero, but it is not sufficient to say this inductive bias is a mistake.\n2. Equation 8, this is not the original form of TAC-GAN. The original TAC-GAN is built upon AC-GAN, so term (c) in Equation 3 in the TAC-GAN paper is missing. I notice that Equation 8 is consistent with the actual implementation of TAC-GAN, but I guess it is good to state this clearly in the paper.\n3. Implementations of ADC-GAN and TAC-GAN are the same? As I checked the provided code in supplementary, I think the proposed ADC-GAN is very similar to TAC-GAN (as defined in Equation 8): In fact, if spectral norm (SN) and bias are not used in the linear classification layer, they are exactly equivalent. This is because the weight of $C_d$ is just $C$ and $C_{mi}$ stacked together. I would consider this as an implementation difference. Note that Theorem 2 and 3 are different, I doubt the superior performance of ADC-GAN might come from the difference in SN or a different choice of hyperparameters. In such case, it would be helpful if the author could provide code for MoG experiment, which is cleaner, simpler, and no SN applied (if the code is borrowed from TAC-GAN). Please correct me if I am wrong, and I'm happy to amend my score accordingly.\n4. It would be helpful if the author could provide results of ADC-GAN on ImageNet at 256 resolution.\n5. Table 2, why not use the reported numbers in TAC-GAN paper? I checked Table 1 in TAC-GAN paper, and their FID on CIFAR100 is 7.22 which is lower than the reported 7.98, any explanation?\n6. It is nice to see ADC-GAN worked even without GAN loss, I am curious how the model performs (w/o GAN loss) on challenging datasets such as ImageNet? It is also surprising that in Supplementary Table 4, Hinge loss does worse than no GAN loss: if Theorem 2 holds, the solution set of ADC-GAN is a subset of (unconditional) GAN (say with hinge loss), so adding GAN loss wouldn't affect ADC-GAN training. Can the author explain this?\n7. Comparing Theorem 2 with a plain cGAN which minimizes $JS(Q_{X,Y}||P_{X,Y})$, does the reverse KL tend to cause mode collapse? (in theory it seems that JS is better than reverse KL? a typical yet imprecise point to be made is that KL causes mode averaging and reverse KL causes mode collapse. [1])\n\n[1] Zhao, Miaoyun, et al. \"Bridging Maximum Likelihood and Adversarial Learning via α-Divergence.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.", "summary_of_the_paper": "This paper proposes a new conditional GAN model that employs a discriminative classifier that predicts in the joint space of label and real/fake domain. The theoretical analysis shows the proposed ADC-GAN can minimize the reverse KL between joint $Q_{X,Y}$ and $P_{X,Y}$.", "summary_of_the_review": "I think the paper presents an interesting analysis of TAC-GAN, AC-GAN, and other cGAN methods. My concern is that the proposed method has the same actual implementation as existing method (TAC-GAN). The paper also lacks results on high-resolution generation. I am willing to raise my score if my concerns are resolved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LbdeIQnr2Df", "reviewer": "Reviewer_mZT7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n* The proposed discriminative classifiers seems interesting to resolve the biased issue of ACGAN. \n\nWeakness:\n\n* The key results seems to be Thm 2, which is based on Prop. 2. However, the proof shows pm(x, y, l) = pm(x, y, 1) + pm(x, y, 0) = 1/2 p(x, y) + 1/2q(x, y). at the very beginning. How do you get the second equation?  I guess it may not be a fatal error, but I can't tell the correctness at the moment. \n\n* The main criticism of AC-GAN is being generator-agnostic, which I think it's not fully appropriate. A common practical implementation of AC-GAN is also using generated data to train classifier, which I think it's a straightforward idea. Under this, whether it can simply resolve the generator-agnostic issue? However, by just doing so, the performance seems not as competitive as the reported numbers of the proposed method.  Could you comment on it? \n\n* The reported numbers looks not quite consistent with other works to me.  \n\n  - For CIFAR10 results, the reported FID are all below 7, however, it's not the case for most of the existing works. For example, all the models reported here https://github.com/POSTECH-CVLab/PyTorch-StudioGAN#cifar10-3x32x32 are all with FID > 7. \n\n - For CIFAR100, the numbers reported in TAC-GAN seems better than the 11.37 reported in the paper for the TAC-GAN and 7.98 for the proposed method. \n\n- For ImageNet, again the reported BigGAN is worse than it should be.  For example, see Table 2 in https://arxiv.org/pdf/2111.01118v1.pdf which provides a nice comparison. \n\n* Some descriptions are not accurate. For example, there are multiple sentences about \"GANs are notoriously unstable to train\", which is true back to 2014. However, there are already \"tons\" of papers working on it to resolve the issues. The authors should at least cite those and make a fair description.  Secondly, the below Eq (6), the authors mention they proposed method can be \"unbiasedly optimize\", which is not true under alternative and minibatch setting. Again, there are lots of works (check all the works on GAN optimization) discussing this issue. The authors should remove this incorrect claim. \n\n* Proposition 1 seems trivial. Also, Theorem 1 is quite similar to the analysis in TAC-GAN. I would highly suggest removing Proposition 1,  which you don't have to pretend to be a theoretical paper. Also, cite TAC-GAN for the analysis. We should not copy or redo the analysis from the predecessor. \n\n* I have the concern of the analysis. Most of the results rely on assuming sth components are optimal. However, in reality, they are not hold in reality, and there is no convergence analysis provided.  Could the authors comment on it?  \n\n* The biased issue of ACGAN is known.  For example, \n\n  - AC-GAN Learns a Biased Distribution, 2017\n - Unbiased Auxiliary Classifier GANs with MINE, 2020\n\n There are many more. The authors should provide a better overview for the progress of this direction.\n\n\nQuestion:\n\n* In Table 1, PD-GAN is optimizing JS(P||Q) while the proposed ADC-GAN is optimizing KL(P||Q).  To me, there should not much difference. Any insights why the proposed ADC-GAN, which optimizes an asymmeric loss, should be better? \n\nSuggestion:\n\n* There are some very recent works in NeurIPS and also highly relayed, \n  - Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training, NeurIPS 2021\n  - A Unified View of cGANs with and without Classifiers, NeurIPS 2021\n  Although they are posted online after ICLR deadline, I would strongly encourage the authors comment on the similarities and differences between the proposed work and these two, because they will be on public for a when the paper decision of ICLR is out anyway. Note that I won't judge the paper decision based on this, but I think it's great to have for the community. It's fine even though the ideas are overlapping. ", "summary_of_the_paper": "The paper is about improving conditional GANs. To be specifically, it also aims to resolve the bias issue of ACGAN by proposing a discriminative classifier. The discriminative classifier is a hybrid model of discriminator and classifier, where it has to not tell real or fake, but also the class.  Preliminary analysis of the proposed method are provided. Experiments are conducted on the standard benchmarks. ", "summary_of_the_review": "The main concerns are first on the analysis, which I couldn't tell the correctness at the moment. Second issue is the empirical results, which seems not consistent with other works. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Glk94RfuJ5", "reviewer": "Reviewer_uPwH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The problem of low intra-class diversity of classifier-based cGANs is important. \n    - Reasons: The projection-based cGAN, PD-GAN, does not suffer from the low intra-class diversity problem, but it converges more slowly than classifier-based cGANs (see Omni-GAN, arXiv:2011.13074).  The classifier-based cGANs converge faster but suffer from low intra-class diversity. Therefore, it is of great value to improve classifier-based cGANs so that we can completely abandon the PD-GAN converging slowly in practice. \n- This paper provides a theoretical perspective for analyzing the loss function of different cGANs. \n- The proposed ADC-GAN is very simple to implement without additional computational overhead.\n\nWeaknesses:\n\n- Please detail in the paper how the FID in Table 2 is calculated (for example, how many generated images are used, whether the training set or the validation set is used, and whether the inception model is from PyTorch or tensorflow). In addition, I also recommend including IS in Table 2.\n- In Equ. 5, the notations of $C_d(y,1|x)$ and $C_d(y,0|x)$ are a bit confusing. After checking the code in the supplementary material, I understood the meaning of the equation. In fact, $C_d(y,1|x)$ and $C_d(y,0|x)$ are implemented using a fully connected layer with output dimensions of num_classes * 2. I suggest that the author use much clearer notation to make it easier to understand. The author can refer to the notation of equation 8, which is clear. \n- In Figure 4e, I am surprised that as a classifier-based cGAN, ADC-GAN did not suffer from mode collapse. The author also does not seem to apply weight decay for the discriminator, as Omni-GAN does. I am not sure if the author has used other regularization techniques to stabilize the training. As far as I know, if do not add regularizations such as weight decay, other classifier-based cGANs will collapse earlier, such as AC-GAN, Multi-hinge GAN, and Omni-GAN. It would be better if the author could explain this phenomenon. \n- I know that there is an improved version of AC-GAN (ImAC-GAN), as discussed in section 3.3 of the Omni-GAN paper. ImAC-GAN has a clear performance gain compared to AC-GAN. It would be better if the author could discuss the relationship between ImAC-GAN and ADC-GAN. \n- In Figure 4 (c) and (f), the results of T-SNE visualization are not very convincing. In my opinion, the author should not use the discriminator to extract feature representations, because the PD-GAN discriminator is less supervised than ADC-GAN. I suggest that the author use a pre-trained classification model to extract features for a fair comparison.\n- In the caption of Figure 4, the author says that the T-SNE uses training data, but in the last paragraph of section 5.3, the author says that the T-SNE uses the validation data. \n", "summary_of_the_paper": "- This paper aims to solve the low intra-class diversity on generated images of AC-GAN, a classifier-based cGAN. \n- As far as I know, this is an important issue that limits classifier-based cGANs (the counterpart is the projection-based cGAN, i.e, PD-GAN).\n- The authors point out that the reason is that the classifier of AC-GAN is generator-agnostic and minimization of conditional entropy decreases the intra-class diversity. \n- The authors propose ADC-GAN (auxiliary discriminative classifier) to solve this problem, and theoretical analysis is also presented.\n", "summary_of_the_review": "The author proposes a simple but seemingly promising classifier-based cGAN. I hope the author can answer my questions above in detail. I will improve the score based on the author's answer. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F9V6r9VE3YK", "reviewer": "Reviewer_DPgR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n(+) The paper exactly points out the primitive problem of ACGAN from the optimization perspective. Since ACGAN is widely adopted in the machine learning area, analyzing problems of ACGAN is necessary and valuable.\n\n(+) The proposed auxiliary discriminative classifier is reasonable, and easy to implement. Also, ADC-GAN does not require much computational burden.\n\n(+) Section 4.2 is very interesting and the explanation of why projection discriminator fails to approximate the joint distribution in Figure 2 is reasonable.\n\nWeaknesses:\n\n(-) It seems that Theorem 1 has already been covered in TAC-GAN paper (paragraphs below eq.4 of the TAC-GAN paper [R1]). Although mathematical formulations are different from each other, the arguments of Theorem 1 and the paragraphs seem to be very similar. I think it is essential to clarify differences between two arguments.\n\n(-) It seems that all experiments were conducted once. It would be better to conduct experiments several times since GANs have been known to have a large performance variance. \n\n(-) The contribution that ADC-GAN can generate diverse images compared to ACGAN and TAC-GAN is not fully demonstrated. Although FID has been a widely used metric to measure fidelity and diversity of generated images, I think It is not enough. I recommend the authors to utilize the improved precision and recall [R2], classification accuracy score [R3], or density and coverage [R4] to quantify the ability of generating diverse images of ADC-GAN.\n\n(-) In section 5.1, the authors conducted the distribution learning experiment using one-dimensional conditional gaussians whose supports are overlapped. I accept that ADC-GAN can learn the joint distribution which consists of the one-dimensional conditional gaussians better than PD-GAN, AC-GAN, and TAC-GAN. However, what about a joint distribution which consists of conditional gaussians with disjoint supports? Can ADC-GAN learn the joint distribution better than other cGANs?", "summary_of_the_paper": "This paper proposes the Auxiliary Discriminative Classifier GAN (ADC-GAN) to eliminate a contractionary objective and conditional entropy in ACGAN generator training. Specifically, the authors mathematically demonstrate that training ACGAN without a discriminative label classifier causes minimizing an undesirable divergence (KL(q(x)||p(x))) which conflicts with the joint distribution matching (KL(q(x,y)||p(x,y))). Also, they insist that the lack of intra-class diversity of ACGAN results from the absence of generator guidance for training the discriminator. To resolve all these issues, they devise a new classifier, the auxiliary discriminative classifier and deploy the new classifier directly on the ACGAN framework. Experiments demonstrate that ADC-GAN can successfully learn the joint distribution whose conditional marginals have non-negligible support overlap using MoG dataset. In addition, they show the effectiveness of ADCGAN compared to ACGAN, projection discriminator, and TAC-GAN on four benchmark datasets (CIFAR10, CIFAR100, Tiny-ImageNet, and ImageNet) using IS, FID, iFID metrics. ", "summary_of_the_review": "The authors propose a new type of ACGAN named ADC-GAN to address an improper optimization process of ACGAN. They apply adversarial training not only for the discriminator but also for the auxiliary classifier to eliminate a contradictory divergence and conditional entropy in ACGAN training. In the experimental results, they prove the effectiveness of ADC-GAN using synthetic datasets and various benchmark datasets. However, I think Theorem 1 has already been addressed in TAC-GAN paper and experimental results do not fully demonstrate the effectiveness of the proposed method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a conditional generative adversarial network with an auxiliary discriminative classifier for conditional generative modeling. The auxiliary discriminative classifier can provide the discrepancy between the joint distribution of the real data and labels and that of the generated data and labels to the generator by discriminatively predicting the label of the real and generated data. Experiment results are provided to demonstrate the effectiveness of the proposed idea.  The current paper receives mixed ratings after rebuttal (5, 6, 5, 8). Except that one reviewer (the Reviewer uPwH) will champion the paper with a score of 8, the concerns of the other three reviewers remain. To be specific, even though Reviewer ebJs assigns a score of 6, he/she doesn’t champion the paper because additional experiments requested are not provided by the authors, including (i) training on more datasets or higher resolutions, (ii) visualizing feature norm and grad norm as done in ReACGAN, (iii) experiments on ADC-GAN without unconditional GAN loss. The Reviewer DPgR pointed out that the paper might have a novelty issue because it bears some similarities with other works but it lacks a discussion in the revision. Additionally, Reviewer mZT7 pointed out that the authors didn’t provide a revised paper during the rebuttal, thus leading to a difficulty to assess the quality of the final paper. As a result, AC thinks that the paper is not ready to publish at the current stage and recommends a rejection.  The AC urges the authors to revise their paper according to the comments provided by the reviewers, and resubmit their work in a future venue.
Decision: Reject