A Unified View of cGANs with and without Classifiers
Authors: Si-An Chen
Chun-Liang Li
Hsuan-Tien Lin
Venue: NeurIPS 2021 Poster (2021)
Keywords: generative model
generative adversarial network
GAN
conditional generation
conditional GAN
classifier

Abstract: Conditional Generative Adversarial Networks (cGANs) are implicit generative models which allow to sample from class-conditional distributions. Existing cGANs are based on a wide range of different discriminator designs and training objectives. One popular design in earlier works is to include a classifier during training with the assumption that good classifiers can help eliminate samples generated with wrong classes. Nevertheless, including classifiers in cGANs often comes with a side effect of only generating easy-to-classify samples. Recently, some representative cGANs avoid the shortcoming and reach state-of-the-art performance without having classifiers. Somehow it remains unanswered whether the classifiers can be resurrected to design better cGANs. In this work, we demonstrate that classifiers can be properly leveraged to improve cGANs. We start by using the decomposition of the joint probability distribution to connect the goals of cGANs and classification as a unified framework. The framework, along with a classic energy model to parameterize distributions, justifies the use of classifiers for cGANs in a principled manner. It explains several popular cGAN variants, such as ACGAN, ProjGAN, and ContraGAN, as special cases with different levels of approximations, which provides a unified view and brings new insights to understanding cGANs. Experimental results demonstrate that the design inspired by the proposed framework outperforms state-of-the-art cGANs on multiple benchmark datasets, especially on the most challenging ImageNet. The code is available at https://github.com/sian-chen/PyTorch-ECGAN.

A Unified View of cGANs with and without Classifiers
Si-An Chen National Taiwan University a09922007@csie attuuedu. tw
Chun-Liang Li Google Cloud AI Shunliang@google com
Hsuan-Tier Lin
National Taiwan University htlin@csie ntu. edu.
Abstract
Conditional Generative Adversarial Networks (cGANs) are implicit generative models which allow to sample from class-conditional distributions. Existing cGANs are based on wide range of different discriminator designs and training objectives. One popular design earlier works is to include a classifier during training with the assumption that good classifiers can help eliminate samples gen- erated with wrong classes. Nevertheless, including classifiers eGANs often comes with side effect of only generating casy-to-classify samples. Recently, some representative cGANs avoid the shortcoming and reach state-of-the-ar per- formance without having classifiers. Somehow it remains unanswered whether the classifiers can be resurrected to design better eGANs. In this work, we demonstrate that classifiers can be properly leveraged to improve eGANs. We start by using the decomposition of the joint probability distribution to connect the goals unified framework. The framework, along with
eGANs and classification classic energy model to parameterize distributions, justifies the use of classifiers for cGANs in principled manner. It explains several popular cGAN variants, such as ACGAN, ProjGAN, and ContraGAN, as special cases with different levels of approximations, which provides unified view and brings new insights to under- standing cGANs. Experimental results demonstrate that the design inspired by the proposed framework outperforms state-of-the-as cGANs on multiple benchmark datasets, especially on the most challenging ImageNet. The code is available at https://ge thtth.b.ccm com/si an whwn/PyTorchh ECGAN.
Introduction
Generative Adversarial Networks [GANs; 10] is a family of generative models that are trained from the duel of a generator and a discriminator. The generator aims to generate data from a target distribution, where the fidelity the generated data is "screened" by the discriminator. Recent studies on the objectives [2, 37, 29, 25, 36, 26, 38], backbone architectures [41, 50], and regularization techniques [13, 35, 51] for GANs have achieved impressive progress on image generation, making GANs the state-of-ther -ote---ftthhr-rrtt art approach to generate high fidelity and diverse images [3]. Conditional GANs (cGANs) extend GANs to generate data from class-conditionall distributions [33, 39, 34, 16].
The capability of conditional generation extends the application horizon of GANs to conditional image generation based on labels [39] or texts [43], speech enhancement [32], and image style transformation [18, 53].
One representative cGAN Auxiliary Classifier GAN [ACGAN: 39], which decomposes the condi- tional discriminator to classifier and unconditional discriminator. The generator of ACGAN is expected to generate images that convince the unconditional discriminator while being classified to
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the right class. The classifier plays pivotal role in laying down the law of conditional generation for ACGAN, making the very first cGAN that can learn to generate 1000 classes of ImageNet images [6]. That is, ACGAN used to be a leading cGAN design. While the classifier in ACGAN indeed improves the quality of conditional generation, deeper studies revealed that the classifier biases the generator to generate easier-to-classi images [45], which in term decreases the capability to match the target distribution.
Unlike ACGAN, most state-of-the-art cGANs are designed without a classifier. One representative cGAN without a classifier is Projection GAN [ProjGAN: 34], which learns an embedding for each class to form a projection- based conditional discriminator. ProjGAN not only generates higher-quality images than ACGAN, but also accurately generates images in target classes without relying on an explicit classifier. In fact, it was found that ProjGAN usually cannot be further improved by adding a classification loss [34]. The finding, along with the success of ProjGAN and other cGANs without classifiers [15, 4], seem to suggest that including classifier is not helpful for improving cGANs.
In this work, we challenge the belief that classifiers are not helpful for cGANs, with the conjecture that leveraging the classifiers appropriately can benefit conditional generation. We propose a framework that pins down the roles of the classifier and the conditional discriminator by first decomposing the joint target distribution with Bayes rule. We then model the conditional discriminator as an energy function, which is an unnormalized log probability. Under the energy function, we derive the corresponding optimization term for the classifier and the conditional discriminator with the help of Fenchel duality to form the unified framework. The framework reveals that a jointly generative model can be trained via two routes, from the aspect of the classifier and the conditional discriminator, respectively. We name our framework Energy- based Conditional Generative Adversarial Networks (ECGAN), which not only justifies the use of classifiers for cGANs in a principled manner, but also explains several popular cGAN variants, such as ACGAN [39], ProjGAN [34], and ContraGAN [16] as special cases with different approximations. After properly combining the objectives from the two routes the framework, we empirically find that ECGAN outperforms other cGAN variants across different backbone architectures on benchmark datasets, including the most challenging ImageNet.
We summarize the contributions of this paper as:
We justify the principled use of classifiers for cGANs by decomposing the joint distribution.
We propose a cGAN framework, Energy-based Conditional Generative Adversarial Net- works (ECGAN), which explains several popular cGAN variants in a unified view.
We experimentally demonstrate that ECGAN consistently outperforms other of cGANs across different backbone architectures on benchmark datasets.
The paper is organized as follows. Section 2 derives the unified framework that establishes the role of the classifiers for cGANs. The framework is used to explain ACGAN [39], ProjGAN [34], and ContraGAN [16] in Section 3. Then, we demonstrate the effectiveness of our framework by experiments in Section 4. We discuss related work in Section 5 before concluding in Section 6.
2 Method
Given a K-class dataset (x,, y) ~ Pa. where y E {1... .. K} the class of and pa is the underlying data distribution. Our goal is to train a generator G to generate sample G(2,8) v) following Pa (a(y))) where 2 is sampled from a known distribution such as (O, 1). To solved the problem, a typical cGAN framework can be formulated by extending an unconditional GAN as:
max min
Σ
E D(a,v) v) Pa(aly)
E D(G(z,v),)) p(2)
where G is the generator and D is a discriminator that outputs higher values for real data. The choice of D leads to different types of GANs [10, 2, 29, 8).
At first glance, there is no classifier in Eq. (1). However, because of the success of leveraging label information via classification, it is hypothesized that a better classifier can improve conditional generation [39]. Motivated by this, in this section, we show how we bridge classifiers to cGANs by Bayes rule and Fenchel duality.

---

2.1 Bridge Classifiers to Discriminators with Joint Distribution
classifier, when viewed from a probabilistic perspective, is a function that approximates Pa (3) the probability that z belongs to class y. On the other hand, a conditional discriminator, telling whether æ is real data in class y, can be viewed as a function approximate pa(zly). To connect pa(yla:) and pa(xly), an important observation is through the joint probability:
log y) p(æly) + log
(2)
= logp(y)>) + logp(x).
(3)
The observation illustrates that we can approximate log in two directions: one containing p(arly) for conditional discriminators and one containing p(y|=r) for elassifiers. The finding reveals that by sharing the parameterization, updating the parameters one direction may optimize the other implicitly. Therefore, we link the classifier the conditional discriminator by training both objectives jointly.
2.2 Learning Joint Distribution via Optimizing Conditional Discriminators
Since p(y) is usually known priori (e.g., uniform) or able to easily estimated (e.g.. empirical counting), we focus on learning p(æly) in E2) Specifically, since log p(x, y) E R, we parameterize it via fo(x), such neural network with K real value outputs, where exp(fe(x) [yl) x p(z, y) Similar parameterization is also used in exponential family [48] and energy based model [23]. Therefore, the log-likelihood log p(rly) can be modeled as:
exp (fo())lvl) = fo(()))) log (0),),
logpo(xlu) = log
(4)
(0)
where (y) J exp (fol dr'.
Optimizing Eq. (4) is challenging because of the intractable partition function (y) Here we introduce the Fenchel duality [48] the partition function (Q)
log (0) = max To
E 99 (x)
[N]( (1x))
where 4, a distribution conditioned on y and (4n) - Eeg (u)) [log by (x:) is the entropy
of Qy. The derivation is provided in Appendix A. By the Fenchel duality, we obtain our maximum likelihood estimation in Eq. (4) as:
[f(())v|]] + (1(a))) [1( (9)(x)
max
IE Pa(m.))
[fo()) - max 9x
E
(5)
To approximate the solution of qg. in additional to density models, we can train an auxiliary generator la as in cGANs to estimate (z)(() via sampling. That is, we can sample x from to by x (2(2, v), where 2 N(0 1). The objective (5) then becomes:
max min 9 6
Σ
[[(())[l]] E pa(xlo)
(2(( H H(g((: p(=)
(6)
which is almost the form of Eq (1) except the entropy (- 3)). We leave the discussion about the entropy estimation in Section 2.4. Currently, the loss function to optimize the objective without the entropy can be formulated as:
La, (x, z, y = [x) + fa(ga(z))lv) Cg. (2,31 = = fe v))lv)
2.3 Learning Joint Distributions via Optimizing Unconditional Discriminators & Classifiers
Following Eq. (3), we can approximate logp(as,)) v) by approximating log p(ylar) and logp(x). With our energy function fe. pe 2)) can be formulated as:
po(y|>) = pol2.v) exp(fo(x)(1))
po(x) En exp(fo(=)lbT)
3

---

which is equivalent to the y 'hth output of SOFTMAX([f Therefore, we can maximize the log- (y/z) æ)) by consider fo as softmax classifier minimizing the cross-entropy loss:
likelihood of Po (yl
Lar y: 0) = log (SOFTMAX (f)( [v])
the other hand, to maximize the log-likelihood of p(x), we introduce a reparameterization ho(x) log Ey exp(fo(x))x)):
Σ Po(2, = log Σ I. exp(fo(z))x) bl)
po()) = log
Eg exp(fo(<" da
exp(log(2 E, exp(fo(x)lu)l
exp(ho (x))) (xppho(('1) dx
log
log
J=. exp(log( Ey expl (x (x')l
ho)))) log((Z(()))
where (0) = l- exp((o(xx)) dx. Similar to Eq. (5), we can rewrite log (') by its Fenchel duality:
log Z'(0) max
[ho[x)]+ (())
(8)
[9(2))
where q is a distribution of and (g) is the entropy of
Combining Eq. (7) and Eq. (8) and reusing the generator Section 2.2, we obtain the optimization problem:
max min
E
[h)(x)] - E [ho(as(2, w))l - 11 (gs) p(a)
pa(m.))
Similar to Eq. (6), the objective of the unconditional discriminator is equivalent to typical GANs augmented with an entropy term. The loss function without considering the entropy can be formulated
La (2, (x: ho(g)(2)) f.g (z, v; = ho (o(g)(2,))
2.4 Entropy Approximation in eGANs
In Section 2.2 and Section 2.3, we propose two approaches to train cGANs with and without classification. Unsolved problems in Eq. (6) and Eq. (9) are the entropy terms H (ga(s, y)) and H H(g)). In previous work, various estimators have been proposed to estimate entropy or its gradient [46, 42, 21, 27J. One can frrely choose any approach to estimate the entropy in the proposed framework. In this work, we consider two entropy estimators, and we will show how they connect with existing cGANs.
The first approach is the naive constant approximation. Since entropy is always non-neen non-negatvee, we naturally have the constant zero as a lower bound. Therefore, we can maximize the objective by replacing the entropy term with its lower bound, which is zero in this case. This approach is simple but we will show its effectiveness in Section 4 and how it links our framework to ProjGAN and ContraGAN in Section 3.
The second approach is estimating a variational lower bound. Informally, given a batch of data { M1 ..., mm ym)) ). encoder function I, and class embedding function e(y), the negative 2C loss used in ContraGAN [16).
Lo(2s, ye; t) = log (d(((z)) + Em [m = vil d(I(a;). d(I(zz),e(z.)) + k [ * 1] d(((=)) (1xx))
(10)
an empirical estimate of a proper lower bound of (XX [40], where d(a, b) = exp(a Topp(Tb//t b/t) is distance function with temperature I.. We provide the proof in Appendix B.
The 2C loss heavily relies on the embeddings [xx)) and e(y). Although we only need to estimate the entropy of generated data in Eq. (6) and Eq. (9), still rely on true data to learn the embeddings in

---

Image
Label
Feature
Softmax CE Loss: Lelf
Emberdiing
Cond. Adv. Loss: Ldz Lug
Label
D2
Projector
Uncond. Adv. Loss: Ldg: fa
(dlo
Contrastive Loss: L Llae
O
K
Softmax CE Loss
Uncond. Adv. Loss
Contrastive Loss
Cond. Adv. Loss
Figure 1: The discriminator's design of ECGAN. Dj be any network backbone such DCGAN, ResGAN, BigGAN. D2 is linear layer with K outputs, where K is the number of classes.
practice. Therefore, the loss function of Eq. (6) can be written as:
LD. (x, 2, 0) La, (I, - XccC creal Co. (z, yi 6) S,, (z, vi + Acc lake
where Ae is hyperparameter controlling the weight of the contrastive loss, and 2 Clake contrastive loss calculated on batch of real data and generated data respectively.
the
Similarly, the loss function of Eq. (9) becomes:
LD, (2, 2. 0) Lug (s,, 2, Xchee
(2, (; L. (2r, yi ()) + Clake
The introduction of 2C loss allows accommodate ContraGAN into framework.
2.5 Energy-based Conditional Generative Adversarial Network
Previous work has shown that multitask training benefits representation learning [30] and training discriminative and generative models jointly outperforms their purely generative or purely discrimi- native counterparts [11, 28]. Therefore, we propose framework named Energy-based Conditional Generative Adversarial Network (ECGAN), which combines the two approaches in Section 2.2 and Section 2.3 to learn the joint distribution better. The loss function can be summarized as:
CD 2, y: 0) = La, (z, 2, y: 0) + aLd, (x, 2, y: 0) + XeCe + >ar y;
(11)
Ca(2,y: Φ) L. (2, ys Φ) + aLgz (2,3: v) + Ac flae
(12)
where a is a weight parameter for the unconditional GAN loss. The discriminator's design is illustrated in Fig 1.
Here we discuss the intuition of the mechanisms behind each component in Eq. (11). Ca, is loss function for conditional discriminator. It updates the y-th output when given data pair (ar, y). fad, guides to an unconditional discriminator. It updates all outputs according to whether x is real. Lar learns a classifier. It increases the y-th output and decreases the other outputs for data belonging to class y. Finally, Lreal and Caae play the roles to improve the latent embeddings by pulling the embeddings of data with the same class closer.
Previously, we derive the loss functions Ld, and Ld, as the loss in Wasserstein GAN [2]. In practice, we use the hinge loss as proposed in Geometric GAN [26] for better stability and convergence. We use the following combination of Ld, and Ldz:
Hinge( (T y) + a (T(aat)); Jo fo y) + a ho ho For more discussion of the implementation of hinge loss, please check Appendix C. The overall training procedure of ECGAN is presented in Appendix E.
(13)

---

Accommodation to Existing cGANs
In this section, we show that our framework covers several representative cGAN algorithms, including ACGAN [39], ProjGAN [35], and ContraGAN [16]. Through the ECGAN framework, we obtain unified view of cGANs, which allows to fairly compare and understand the pros and cons of existing cGANs. We name the ECGAN counterparts ECGAN-0, ECGAN-C, and ECGAN-E, corresponding to ProjGAN, ACGAN, and ContraGiAN, respectively. We summarize the settings in Table and illustrate the discriminator designs in Appendix F.
Existing cGAN
ECGAN Counterpart
a
Aal
A
ProjGAN
ECGAN-0
0
0
0
ACGAN ContraGAN
ECGAN-C
0
>
0
0
ECGAN-E
0
>
0
Table I: A summary of cGANs and their closest ECGAN counterpart.
3.1 ProjGAN
ProjGAN [34] is the most representative cGAN design that is commonly used in state-of-the-ar research [3, 50]. Let the output of the penultimate layer in the discriminator g(x). The output of ProjGAN's discriminator is:
D(r, v) = wh g(x) + bw + wy g(x) = (w,, + w,y) g(x)) + b..
(14)
where Was by are the parameters for the unconditional linear layer, and wy is the class embedding of On the other hand, the output of discriminator in ECGAN is:
Dar, v) = [=)[v] = (WT g(z) + b) = w.y g(x) by
(15)
where W, are the parameters the linear output layer in fe. As shown in Eq. (14) and Eq. (15), the architectures of ProjGAN and ECGAN almost equivalent. In addition, the loss function of ProjGAN can be formulated as:
LG D(((2))))) Cn = D(zz + D(G(2),
which is special case of ECGAN while >₁ = Aarr O. We name this case ECGAN-0, which the simplest version ECGAN. Compared with ProjGAN, ECGAN-0 has additional bias terms for the output of each class.
3.2 ACGAN
ACGAN [39] is the most well- knoln cGAN algorithm that leverages a classifier to achieve conditional generation. Given -llass dataset, the discriminator of ACGAN parameterized by network with K + 1 outputs. The first output, denoted as D(x)), is an unconditional discriminator distinguishing between real and fake images. The remaining K outputs, denoted as ((x), is classifier that predicts logits for every class. The loss function of ACGAN can be formulated as:
LG D(G((2) + AGLa)
Lp - D(x) + D(G(z)) + Xa(Car(a, yi C) Lar(G()),,
where G is the generator, Ay and du are hyperparameters to control the weight of cross-entropy loss.
The formulation of ACGAN is similar to our ECGAN when a = d and Aer > 0. We call the special case as ECGAN-C, with suffix 'C' for classification loss. ECGAN-C uses a conditional discriminator which plays the role of classifier at the same time. Hence the generator in ECGAN-C learns from the conditional discriminator rather than the cross-entropy loss which is biased for generative objectives.

---

# training data
Dataset
# training
# test
# classes
Resolution
per class
CIFAR-10
50,000
10,000
10
32 x
5,000
Tiny ImageNet
100,000
10,000
200
64 x 64
500
ImageNet
1,281,167
50,000
1,000
128
128
1,281
Table Datasets for evaluation.
3.3 ContraGAN
ContraGAN [16] proposed 2C loss, which we mentioned Eq. (10), to capture the data-to-data relationship and data-to-label relationship. The 2C loss is applied in both discriminator and generator achieve conditional generation. That is:
LG - -D(G())))) + Ao
LD = - D(x, v) + D(G(=). + A
The loss functions are similar to ones in ECGAN with datr 0 and Ae > O. We call it ECGAN- E, where 'E' means entropy estimation. The main difference between ContraGAN and ECGAN-E is the output layer of their discriminators. While ContraGAN uses single-output network, ECGAN uses a K- output network fe which has higher capacity.
We keep Eq. (11) and Eq. (12) as simple possible to reduce the burden of hyperparameter tuning. Under the simple equations of the current framework, ECGAN-C and ECGAN-E are the closest counterparts to ACGAN and ContraGAN. The subtle difference (in addition to the underlying network architecture) is that ACGAN uses La, instead of Ld; (ECGAN-C); ContraGAN uses L Lop instead of Lá far (ECGAN-E). One future direction is to introduce more hyperparameters in Eq. (11) and Eq. (12) to get closer counterparts.
Experiment
We conduct experiments on CIFAR-10 [20] and Tiny ImageNet [22] for analysis, and ImageNet [6] for large-scale empirical study. Table 2 shows the statistics of the datasets. All datasets are publicly available for research use. They not constructed for human-related study. We do not specifically take any personal information from the datasets in our experiments.
In our experiment, we use two common metrics, Frechet Inception Distance [FID; 14] and Inception Score [IS; 44], to evaluate our generation quality and diversity. Besides, we use Intra-FID, which is the average of FID for each class, to evaluate the performance of conditional generation.
4.1 Experimental Setup
We use StudioGAN [16] to conduct our experiments. StudioGAN PyTorch- based project distributed under the MIT license that provides implementation and benchmark of several popular GAN architectures and techniques. To provide reliable evaluation, we conduct experiments on 10 and Tiny ImageNet with 4 different random seeds and report the means and standard deviations for each metric. We evaluate the model with the lowest FID for each trial. The default backbone architecture is BigGAN [3]. We fix the learning rate for generators and discriminators to 0.0001 and 0.0004, respectively, and tune Aar in [1, 0.1, 0.05, 0.01). We follow the setting X = 1 in [16] when using 2C loss, and set a = I when applying unconditional GAN loss. The experiments take 1-2 days on single GPU (Nvidia Tesla V100) machines for CIFAR-10, Tiny ImageNet, and take 6 days on 8-GPU machines for ImageNet. More details described in Appendix D.
4.2 Ablation Study
We start our empirical studies by investigating the effectiveness of each component in ECGAN. We use symbols "U' to represent unconditional GAN loss, "C" to represent classification loss, and 'E'

---

Dataset
ECGAN Variant
(L)
IT(
Intra-FID (L)
ECGAN-0
$.049:0.00
9.759 = 0.061
41.708 0.2278
ECGAN-U
7
9.967 0.078
41.430 0.326
CIFAR-10
ECGAN-C
7.996 1166:000
9.870
41.715 0.307
ECGAN-UC
7.942 + 0.041
10.002
41.425 0.221
ECGAN-UCE
8.039 + 0.161
9.898 + 0.064
41.371 + 0.278
ECGAN-0
24.077 = 1.660
16.173 = 0.671
214.811
=3627
ECGAN-U
20.876 = 1.651
15.318 + 1.148
215.117 7.034
Tiny ImageNet
ECGAN-C
24.853 3.902
16.554 + 1.500
212.661
8.135
ECGAN-UC
18.919 0.774
18.442 1.036
203.373
5.101
ECGAN-UCE
24.728 + 0.974
17.935 + 0.619
209.547
1.968
Table 3: Ablation study of ECGAN on CIFAR-10 10 and Tiny ImageNet. ECGAN-O means the vanilla
version of ECGAN where Aar = >a 0. The label U stands for unconditional gan loss (a >> 0).
means classification loss (Aar > 0). E means entropy estimation loss via contrastive learning (Ae 0)-
to represent entropy estimation loss, which is 2C loss in our implementation. The concatenation of the symbols indicates the combination of losses. For example, ECGAN-UC means ECGAN with both unconditional GAN loss and classification loss (a 0 and Aar 0). Table shows the results of ECGAN from the simplest ECGAN-0 to the most complicated ECGAN-UCH On CIFAR-10, 10, ECGAN-O already achieves decent results. Adding unconditional loss, classification loss, or contrastive loss provides slightly better or on-par performance. On the harder Tiny Imagenet, the benefit of unconditional loss and classification loss becomes more significant. While ECGAN-U already shows advantages to ECGAN-O, adding classification loss to ECGAN-U further improves all metrics considerably. We also observe that directly adding classification loss is not sufficient to improve cGAN, which consistent to the finding in [34]. The fact reveals that the unconditional GAN loss is crucial component to bridge classifiers and discriminators in cGANs. We also find that adding contrastive loss does not improve ECGAN-UC. An explanation that the entropy estimation lower bound provided by the contrastive loss is too loose benefit the training. Furthermore, the additional parameters introduced by 2C loss make the optimization problem more complicated. As result, we the combination ECGAN-UC as the default option of ECGAN in the following experiments.
Comparison with Existing cGANs
We compare ECGAN to several representative cGANs including ACGAN [39], ProjGAN [34], and ContraGAN [16], with three representative backbone architectures: DCGAN [41], ResNet [13], and BigGAN [3]. Table 4 compares the results of each combinations of cGAN algorithms and backbone architectures. The results show that ECGAN-UC outperforms other cGANs significantly with all backbone architectures on both CIFAR-10 and Tiny ImageNet. We also noticed that ContraGAN, though achieves decent image quality and diversity, learns a conditional generator that interchanges some classes while generating, hence has low Intra-FIID. Overall, the experiment indicates that ECGAN-UC can be a preferred choice for cGAN in general situations.
4.4 Comparisons between Existing eGANs and their ECGAN Counterpart
Table 5 compares ProjGAN, ContraGAN, ACGAN to their ECGAN counterparts. As we described in Section 3, each these representative cGANs can be viewed as special cases under our ECGAN framework. As mentioned in Section 3, ECGAN-0 has additional bias terms in the output layer compared to ProjGAN. The results in Table 5 shows that the subtle difference still brings significant improvement the generation quality, especially the harder Tiny ImageNet.
Compared to ContraGAN, ECGAN-E has the same loss but different design in the discriminator's output layer. While the discriminator of ContraGAN has only single output, ECGAN-E has multiple outputs for every class. The difference makes ECGAN-E solve the label mismatching problem of ContraGAN mentioned in Section 4.3 and benefits generation on 10. but does not work well on Tiny ImageNet. is probably because of the scarcity training data in each class Tiny ImageNet. Only 50 data available for updating the parameters corresponding to each class.
8

---

Dataset
Backbone
method
HD)
IS
Intra-FID
ACGAN
32.507 + 2.174
7.621
0.088
129.603 1.212
ProjGAN
=
0.185
08.164
DCGAN
ContraGAN
28.310 1.761
7.637 0.125
153.730 + 9.965
ECGAN-UC
18.035 = 0.788
8.487 0.131
59.343 1.557
ACGAN
10.073 = 0.274
9.512
48.464 0.716
ProjGAN
10.195 0.203
9.268 0.139
46.598 + 0.070
CIFAR-10
ResGAN
ContraGAN
10,551 0.976
9.087 0.228
138.944 + 12.582
ECGAN-UC
9.244 1 0.062
9,651 0.098
43.876 1 0.384
ACGAN
1.114
2.742.0.00
45.243 1.
ProjGAN
8.145 0.156
9.840 0.080
1 42.110
BigGAN
ContraGAN
8,617 1 0.671
9.679 0.210
114.602 13.261
ECGAN-UC
0.041
=0.002
41.425 + 0.221
ACGAN
29.528 1 4.612
12.964 0.770
315.408 1.171
ProjGAN
28,451 + 2.242
12.213 0.624
242.332 11.447
Tiny ImageNet
BigGAN
ContraGAN
24.915 = 1.222
13.445 0.371
257.657
3,246
ECGAN-UC
18.780 = 1.291
17.475 = 1.052
204.830 + 5.648
Table 4: Comparison between cGAN variants with different backbone architectures on CIFAR-10
and Tiny ImageNet
Dataset
method
FID (L)
(f)
Intra-FII (L)
ProjGAN
8.145 0.156
9,840
0.080
ECGAN-0
8.049 + 0.092
9.759 + 0.061
41.708 + 0.278
ContraGAN
8.617 0.671
9.679 0.210
+
CIFAR-10
ECGAN-E
8.038 + 0.102
9.876 + 0.036
41.155 + 0.277
ACGAN
8.615 0.146
9.742 0.041
45.243 + 0.129
ECGAN-C
8.102 + 0.039
9.980 + 0.093
41.109 + 0.273
ProjGAN
28.451 2.242
12.213 = 0.624
242.332 = 11.447
ECGAN-0
24.077 + 1.660
16.173 + 0.671
214.811 + 3.627
ContraGAN
24.915 + 1.222
13.445 + 0.371
257.657 3.244
Tiny ImageNet
ECGAN-E
38.270 + 1.174
12.576
+ 0.405
239.184
+ -2.228
ACGAN
29.528 + 4.612
12.964
+ 0.770
315.408
+ 1.171
ECGAN-C
24.853 + 3.902
16.554
+ 1.500
212.661 :8.135
Table 5: Compare between representative eGANs and their ECGAN counterparts.
I Last, we compare ECGAN-C to ACGAN. Both of them optimize a GAN loss and classification loss. However, ECGAN-C combines the discriminator and the classifier, so the generator can directly optimize cGAN loss rather than the classification loss. As result, ECGAN-C demonstrates better performance on both CIFAR-10 and Tiny ImageNet. In sum, the comparisons show that through the unified view provided by ECGAN, we can improve the existing methods with minimal modifications.
4.5 Evaluation on ImageNet
We compare our ECGAN-UC and ECGAN-UCE with BigGAN [3) and ContraGAN [16] on ImageNet. We follow all configurations of BigGAN with batch size 256 in StudioGAN. The numbers in Table 6 are reported after 200,000 training steps if not specified. The results show that ECGAN-UCE outperforms other cGANs dramatically. The comparison between ECGAN-UC and ECGAN-UCE indicates that the 2C loss brings more significant improvement in the ECGAN framework than in ContraGAN. The proposed ECGAN-UCE achieves 8.49 FID and 80.69 inception score. To the best of our knowledge, this a ofate-of-the-eee result of GANs with batch size 256 on ImageNet. Selected generated images shown in Appendix G.
5 Related Work
The development of cGANs started from feeding label embeddings to the inputs of GANs or the feature vector some middle layer [33, 7]. To improve the generation quality, ACGAN [39] proposes to leverage classifiers and successfully generates high-resolution images. The use of classifiers

---

Method
FID(
IS(1)
BigGAN
24.08
28.63
ContraGAN
25.16
25.25
ECGAN-UC
30.05
26.47
ECGAN-UCE
12.16
56.33
ECGAN-UCE (400k step)
8.49
80.69
Table 6: Evaluation on ImageNet1 12x 128. (*: Reported by StudioGAN.)
in GANs studied Triple GAN [24] for semi-supervised learning and Triangle GAN [9] for cross-domain distribution matching. However, Shu [45] and Miyato and Koyama [34] pointed out that the auxiliary classifier in ACGAN misleads the generator to generate images that are easier to be classified. Thus, whether classifiers can help conditional generation still remains questionable.
In this work, we connect eGANs with and without classifiers via an energy model parameterization from the joint probability perspective. [12] use similar ideas but focus sampling from the trained classifier via Markov Chain Monte Carlo [MCMC; 1J. Our work is also similar to concurrent work [1]]. which improves [12] by introducing Fenchel duality to replace computationally- intensive MCMC. They use a variational approach [19] to formulate the objective for tractable entropy estimation. In contrast,
study the GAN perspective and the entropy estimation via contrastive learning. Therefore, the proposed ECGAN can be treated complements works compared with [12, 11] by studying GAN perspective. We note that the studied cGAN approaches also result in better generation quality than its variational alternative [11].
Last, [5] study the connection between exponential family and unconditional GANs. Different from [5], we study the conditional GANs with the focus to provide a unified view of common cGANs
and an insight into the role of classifiers in cGANs.
6 Conclusion
In this work, we present a general framework Energy-based Conditional Generative Networks (ECGAN) to train cGANs with classifiers. With the framework, including ACGAN, ProjGAN, and ContraGAN, in a unified view. The experiments demonstrate that ECGAN outperforms state-of-the-at eGANS benehmark datasets, especially the most challenging ImageNet. Further investigation can be conducted to find a better entropy approximation improve cGANs by advanced techniques for classifiers. We hope this work can pave the way to more advanced cGAN algorithms in the future.
explain representative cGANs,
7 Limitations and Potential Negative Impacts
There are two main limitations in the current study. One is the investigation on ImageNet. Ideally, more experiments and analysis on ImageNet can further strengthen the contribution. But training with such a large dataset is barely affordable for our computational resource, and we can only resort the conclusive findings in the current results. The other limitation is whether the metrics such as FID) truly reflect generation quality, but this limitation is considered an open problem to the community anyway.
As with any work on generative models, there is potential risk of the proposed model being misused to create malicious content, much like how misused technology can be used to forge bills. In this sense, more anti-forgery methods will be needed to mitigate the misuse in the future.
Acknowledgement
We thank the anonymous reviewers for valuable suggestions. This work is partially supported by the Ministry Science and Technology of Taiwan via grants MOST 177-262--E-022-00 1 110-222--E-002-00 We also thank the National Center for High- high-pperformance Computing (NCHC) of National Applied Research Laboratories (NARLabs) Taiwan for providing computational resources.
and
10

---

References
[1] Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. An introduction to mcmc for machine learning. Machine learning, 50(1):5-43, 2003. 10
[2] Martín Arjovsky, Soumith Chintala, and Léon Bottou. abs/1701.07875, 2017. 1,2,5,15
Wasserstein
GAN.
CoRR,
[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity
natural image synthesis. In ICLR, 2019. 7, 8,
[4] Grigorios G. Chrysos, Jean Kossaifi, and Stefanos Zafeiriou. Roegan: Robust conditional GAN. Int. J. Comput. Vis., 128(10):2665-2688 2020. 2
[5] Bo Dai, Zhen Liu, Hanjun Dai, Niao He, Arthur Gretton, Le Song, and Dale Schuurmans. Exponential family estimation via adversarial dynamics embedding, NeurIPS, 2019. 10
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 7
[7] Emily L. Denton, Soumith Chintala, Arthur Szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. In NeurIPS, 2015. 9
[8] William Fedus, Mihaela Rosca, Balaji Lakshminarayanan, Andrew M. Dai, Shakir Mohamed, and lan Goodfellow. Many paths to equilibrium: Gans do not need to decrease a divergence at every step. In ICLR, 2018. 2
[9] Zhe Gan, Liqun Chen, Weiyao Wang, Yunchen Pu, Yizhe Zhang, Hao Liu, Chunyuan Li, and Lawrence Carin. Triangle generative adversarial networks. In NeurIPS, 2017. 10
[10] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sher- jil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. CoRR, abs/1406.2661, 2014. 1,2
[11] Will Grathwohl, Jacob Kelly, Milad Hashemi, Mohammad Norouzi, Kevin Swersky, and David Duvenaud. No MCMC for me: Amortized sampling for fast and stable training of energy-based based models. CoRR, abs/2010.04230, 2020. 5, 10
[12] Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In ICLR, 2020. 10
[13] Ishaan Guirajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In NeurlPS, 2017. 1,8 8
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In NeurIPS, 2017.
(15) Mingqi Hu, Deyu Zhou, and Yulan He. Variational conditional GAN for fine-grained controllable image generation. In ACML, 2019.
[16] Minguk Kang and Jaesik Park. Contragan: Contrastive learning for conditional image generation. In NewrIPS, 2020. 1, 2, 4, 6, 7, 8, 9, 15, 16
[17] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. In ICIR, 2018. 16
[18] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In ICML, 2017.
[19] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015. 10, 16
[20] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009. 7

---

[21] Rithesh Kumar, Anirudh Goyal, Aaron C. Courville, and Yoshua Bengio. Maximum entropy generators for energy-based models. CoRR, abs/1901.08508, 2019. 4
[22] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7:7, 2015.
[23] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. A tutorial on energy-based learning. 2006. 3
[24] Chongxuan Li, Taufik Xu, Jun Zhu, and Bo Zhang. Triple generative adversarial nets. In NeurlPS, 2017. 10
[25] Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póczos. MMD GAN: towards deeper understanding of moment matching network. In NeurIPS, 2017. 1
[26] Jae Hyun Lim and Jong Chul Ye. Geometric GAN. CoRR, abs/1705.02894, 2017. 1,5, 15, 16
[27] Jae Hyun Lim, Aaron C. Courville, Christopher J. Pal, and Chin-Wei Huang. AR-DAE: towards unbiased neural entropy gradient estimation. In ICML, 2020.
[28] Hao Liu and Pieter Abbeel. Hybrid discriminaaive-generaiie training via contrastive learning. CoRR, abs/2007.09070, 2020. 5
[29] Xudong Mao, Qing Li, Haoran Xie, Raymond Y. K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In ICCV, 2017. 1,2
[30] Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. JMLR, 17:81:1-8111 1-:::32, 2016. 5
[31] Lars M. Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In ICML, 2018. 16
[32] Daniel Michelsanti and Hua Tan. Conditional generative adversarial networks for speech enhancement and noise-robust speaker verification. In Interspeech, 2017.
[33] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. CoRR, 1.s/1411.1784. 2014. 1,9
[34] Takeru Miyato and Masanori Koyama. cgans with projection discriminator. In ICLR, 2018. 1, 6, 8, 10
[35] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In ICLR, 2018. 1, 6, 16
[36] Youssef Mrouch and Tom Sercu. Fisher gan. In NeurlPS, 2017.
[37] Youssef Mroueh, Tom Sercu, and Vaibhava Goel. matching gan. In /CML, 2017.
Megan: Mean and covariance feature
[38] Youssef Mrouch, Chun-Liang Li, Tom Sercu, Anant Raj, and Yu Cheng, Sobolev gan. In ICLR, 2018.
[39] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. Doina Precup and Yee Whye Teh, editors, ICML, 2017. 1, 2, 8,
[40] Ben Poole, Sherjil Ozair, Aäron van den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, ICMI., 2019. 4, 15
[41] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR, 2016. 1, 8
[42] Rajesh Ranganath, Dustin Tran, and David M. Blei. Hierarchical variational models. In ICML, 2016.
[43] Scott E. Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak
Lee. Generative adversaria text to image synthesis. In ICML, 2016.
12

---

[44] Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung. Alec Radford, and Xi Chen. Improved techniques for training gans. In NeurIPS, 2016.
[45] Rui Shu. Ac-gan learns a biased distribution. 2017. 2, 10
[46] Shashank Singh and Barnabás Póczos. Analysis of k-nearest neighbor distances with application entropy estimation. CoRR, abs/1 2016. 4
[47] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs/1807 0aos//807.0374, 2018. 15
[48] Martin J. Wainwright and Michael I. Jordan. Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn, 1(1-2):1-305, 2008.
[49] Yasin Yazici, Chuan-Sheng Foo, Stefan Winkler, Kim-Hui Yap, Georgios Piliouras, and Vijay Chandrasekhar. The unusual effectiveness of averaging in GAN training. In ICLR, 2019. 16
[50] Han Zhang, Ian J. Goodfellow, Dimitris N. Metaxas, and Augustus Odena. Self-attention
generative adversarial networks. /CML, 2019. 1, 6, 16
[51] Han Zhang. Zizhao Zhang, Augustus Odena, and Honglak Lee. Consistency regularization for generative adversarial networks. In ICLRR 2020.
[52] Shengyu Zhao, Zhijian Liu, Ji Lin, Jun Yan Zhu, and Song Han. Differentiable augmentation for data-efficient GAN training. In NeurlPS, 2020. 16
[53] Jun-Yar Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV, 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] Section
(b) Did you describe the limitations of your work? [Yes] Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Section 7
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Section 4, Section 4.1, Appendix D
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] Table 3, Table 4, Table
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Section 4, Section 4.1
(b) Did you mention the license of the assets? [Yes] Section 4, Section 4.1
(c) Did you include any assets either in the supplemental material
URL? [Yes]
Supplemental material
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] Section Section 4.1. All assets distributed under the MIT license.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] Section
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "kP24RKy5KE", "reviewer": "Reviewer_gBbd", "summary": "The authors target to leverage classifiers in a principled manner and unify them in cGAN, by using the decomposition of joint probability distribution and a classic energy model to parameterize the distribution. Experiments show new state-of-the-art generation quality in terms of FID and IS, over several datasets (CIFAR-10, Tiny ImageNet) on several GAN implementations (DCGAN, ResGAN, BigGAN) and against several baselines (ACGAN, ProjGAN, ContraGAN).", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "********************************************************************\nStrengths\n********************************************************************\n+ Clear exposition: good writing and good result demos.\n+ The research problem is clearly defined.\n+ Reproducible.\n+ Experiments are dense and results are informative to set the new state-of-the-art.\n\n********************************************************************\nWeaknesses\n********************************************************************\n\n- Experiments can be more convincing if not only on toy datasets like CIFAR-10 or Tiny ImageNet. Repeat the quantitative comparisons on the entire ImageNet dataset with the full resolution.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "HE_K8k4zSzJ", "reviewer": "Reviewer_YHsT", "summary": "This paper introduced a general framework for conditional image generation. And existing cGAN methods such as ACGAN, ProjGAN and ContraGAN are inclued in the proposed algorithm. Experimental results in Cifar and Tiny ImageNet verified the effectiveness of different components of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This workd presented a detailed discussion among the differences on existing condiontal image generation methods. The unified view of cGAN is interesting. However, the authors did not prpose new loss functions for cGANs. \n\n\nThanks for the author feedback. Some issues about the experimental evaluation have been addressed welll. However, I still concern about the originality of this work which is also indicated by Reviewer gvcB. In addition, the new results on ImageNet are also confusing and not satisfactory. According to the table below, your reinplementations of ACGAN, ProjGAN and ContraGAN are better. Then the comparisons with BigGAN, ContraGAN based on StudioGAN are not convincing.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The experimental results are not convicing. The authors should empoly datasets with higher resolution such as $128\\times128$, $256\\times256$. The qualitative results in Fig. 1 and Fig. 2 are very blurry.\n2. It seems that the reported FID results of existing methods are low compared with the PyTorch-StudioGAN. Please give more explanation.\n3. The performance of different variants in Tab. 2 is not consistent. Could the classification loss and entropy estimation loss help the conditional image generation? ", "review_text": "", "extra_scores": {"needs_ethics_review": "Yes", "ethics_review_area": ["Privacy and Security (e.g., consent)"], "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "CrkGcqbqQLU", "reviewer": "Reviewer_gvcB", "summary": "This submission proposes to analyze the most popular variations of conditional GANs (ACGAN, ProjGAN, ContraGAN) under a unified, energy-based, formulation (ECGAN).\n\nSpecifically, ECGAN is composed of terms derived from two different decompositions of the joint probability p(x,y).\nThe paper then links each term to one of the popular cGAN approaches.\n\nECGAN is evaluated against the traditional models and an extensive ablation study tests the impact of each component of the proposed model.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe paper builds very heavily on existing work. The ECGAN formulations are only marginally different from previous works.\n\nQuality:\nThe submission seems technically sound, supported by adequate theoretical framework and experiments.\nIt is however difficult to draw conclusive results on only CIFAR-10 and Tiny ImageNet.\n\nClarity: \nThe paper is clear and well organized. \nThe association between the different cGANs and their corresponding ECGAN variants could be debatable, however (see questions below).\n\nSignificance: \nThe proposed formulation provides a unified view of the different losses terms used in conditional GANs that allows for a valuable principled evaluation.\nMoreover, considering that ECGAN-UC has very encouraging reported performances and that its implementation is only a slight departure from previous cGAN's, it could become a staple for conditional GANs if future works confirm its good performances.\n\nOverall, I believe the paper is interesting to read and of good quality. \nI still have some questions and remarks that I hope can be clarified:\n\n- Since ProjGAN sums both an unconditional and a conditional output, wouldn't it make more sense to compare it to ECGAN-U instead of ECGAN-0? Especially since the used Hinge Loss implementation, described in appendix A, sums the conditional and unconditional terms before the Hinge operation, just like in ProjGAN.\nI would appreciate it if the authors could comment and discuss the differences of model and performances between ProjGAN and ECGAN-U.\n\n- Similarly, ACGAN is compared to ECGAN-C in section 3.2. But in section 2.2, ACGAN is already viewed as an unnamed variant of ECGAN (without the ECGAN-0 term and alpha=1 instead).\nCould the authors comment on why ACGAN is not directly considered a specific variant of ECGAN?\nConsidering both previous points, my opinion is that the comparisons in section 3 seem forced and actually make the paper less clear.\n\n- Additional experiments on diverse datasets would strengthen the results, even if performed only with ECGAN-UC and a baseline.\n\n\n-------------------------------------\nPost-rebuttal:\nI'd like to thank the authors for their detailled responses and for providing additional results.\nAfter reading the different reviews and responses, I believe the proposed framework is promising and the comparison between the different variants can provide interesting insights for GANs.\nHowever, I also believe the paper needs to be very clear about exactly how the classical cGAN fit or don't fit into the ECGAN formulation.\nWithout the ability to assess a revised version, I remain unconvinced that small modifications would be sufficient in that regard.\n\nI therefore keep my previous rating (6: Marginally above the acceptance threshold).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and negative impacts are adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "CiO7WkGGH9g", "reviewer": "Reviewer_hEZ7", "summary": "The authors discuss that existing methods on including classifiers in a cGAN biases the generator in generating easy to classify images. Therefore, they propose a way to include classifiers in a cGAN to improve its performance in a principled manner. To do so, they decompose the joint probability distribution by the Bayes rule that results in linking classifiers to conditional discriminators. The proposed formulation shows that a joint generator model can be trained from two directions: a conditional discriminator and an un-conditional discriminator with a classifier.\nThey combine the formulation of these two routes and propose a new method called Energy-based cGAN (ECGAN). ECGAN shows how to use a classifier for cGANs, and it explains other variants of cGANs such as ContraGAN, ACGAN, and ProjGAN. They empirically show that ECGAN outperforms existing cGANs by achieving higher FID (and similar ones) score on two sets of datasets (CIFAR10, Tiny ImageNet).\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives\n\n•\tThe paper's motivation and the overall idea are interesting. \n\n•\tPrincipled way to use classifiers in conditional GAN. The authors present a new view to explore cGANs and use classifiers in a way that is beneficial to cGANs. \n\n•\tThe presented framework covers and explains other variants of cGANs (ContraGAN, ProjGAN, ACGAN).\n\n•\tExperimental results confirm previous results such as adding a classifier directly to ProjGAN does not result in improvements. \n\n•\tBased on the experiments, the ECGAN method achieves higher scores compared to the existing methods. \n\n•\tThe experimental findings are consistent across different GAN architectures. \n\nNegatives:\n\n•\tThe additional of the cross entropy loss to equation (11) appears adhoc to cover ACGAN as one of the variants of ECGAN. \n\n•\tThe authors may add more explanation on the general architecture of the method. Are there two Discriminators with a \n classifier and a generator? There is some confusion regarding whether the classifier and the conditional discriminator are the same neural network or separate? \n\n•\tThe authors explain how they link conditional discriminators and classifiers. It is, however, unclear why the loss components are helping? what kinds of behaviors are promoted by the loss components?\n\n•\tIt would be beneficial if the authors also present training stability analysis (similar to figure 3 of ContraGAN [1]). This analysis can reveal how the formulation and losses are helpful.\n\n•\tIt would be great if the authors can visualize and compare the diversity of generated images across different existing cGANs (ContraGAN, ProjGAN, ACCGAN).\n\n•\tWhat is the computational cost of this method compared to the ContraGAN and ProjGAN?\n\n•\tSince the results between the various approaches are not significantly different for one of the 2 datasets used (CIFAR-10), the authors could experiment with more challenging datasets (e.g., ImageNet) to demonstrate that their findings are consistent.\n\n\n[1] Kang, Minguk, and Jaesik Park. \"Contragan: Contrastive learning for conditional image generation.\" arXiv preprint arXiv:2006.12681 (2020).\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are adequately discussed in the paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "8XGkmO7yIHU", "reviewer": "Reviewer_iBCw", "summary": "The authors of the paper propose a unified view of conditional generative adversarial networks (cGANs). To this end, they analyze the log of joint probability distribution p(x, y) through two different perspective of views; via a conditional discriminator using Eq.(2), and via an unconditional discriminator and classifier using Eq.(3). Inspired by a multitask learning, they suggest Energy-based Conditional Generative Adversarial Networks (ECGAN) where two approaches are combined together to approximate the joint probability distribution. Under this framework, previous cGANs such as ProjGAN, ACGAN and ContraGAN can be closely explained using the variants of ECGAN with the right choice of hyperparameters, namely, ECGAN-0, ECGAN-C and ECGAN-E, respectively. \nTheir experiments on CIFAR-10 and Tiny ImageNet show that these ECGAN variants outperform ProjGAN, ACGAN and ContraGAN in most cases of experiments. Furthermore, ECGAN-UC outperforms these variants of ECGAN as well as previous cGANs with various backbone structures such as DCGAN, ResGAN, and BigGAN. Through this result, the authors conclude that adding a classifier helps improving cGANs with the help of the unconditional discriminator as opposed to the previous findings in Shu et al. [42] and ProjGAN [34].\t", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nPrevious cGANs have taken one of approaches explained in section 2.2 and 2.3 in a broad perspective. By combining them, this paper indeed provides a unified picture of cGANs. Although the concurrent work [11] is somewhat similar in that they both try to solve Eq.(5), the tasks they are trying to solve and the approaches they are taking are quite different as mentioned in the related work.\n\nQuality:\n- Line 249 ~ 250 - The authors mentioned that the entropy estimation lower bound provided by the contrastive loss is too loose to benefit the training. But, doesn’t ECGAN-UC correspond to using 0 as a lower bound for entropy (1st approach mentioned to estimate entropy) which is a looser bound for entropy? \n- Line 247 ~ 248 - The authors claimed that the unconditional GAN loss is a crucial component to bridge classifiers and discriminators in cGANs according to the performance results provided in Table2. Could authors provide any idea why it is the case other than the empirical evidence?\n- Their experiments are somewhat limited as they did not provide the experiment results on ImageNet which were used by all of ProjGAN, ACGAN and ContraGAN.\n\nClarity:\n- Line 101 - It would be better to specify a section of the book or provides a proof in the appendix as done in appendix A.1 of [11].  \n- Line 141 - Could authors elaborate why Eq.(10) is an empirical estimate of a proper lower bound of an entropy?\n- Line 164 ~ 165 - It would be better if the authors can provide experiment results that compare the performance with the original wassertein loss versus hinge loss since it is hard to capture how much improvement is originated from using the hinge loss.\n- Line 197 ~ 198 - The authors said the main difference between ACGAN and ECGAN-C is that ECGAN-C uses a conditional discriminator. But, if my understanding on the equations in section 3.2 is correct, another difference is that ACGAN has additional cross entropy loss for the generated images. Do authors think it is negligible? Also, wouldn’t ECGAN-C closer to ACGAN if unconditional discriminator is used instead of conditional discriminator? According to the claim in line 247 ~ 248 that unconditional GAN loss is crucial component to bridge classifiers and discriminators in cGANs, I am not sure why ECGAN-C is set in that way.\n- Typos:\n    * It seems the denominator part of Eq.(10) is missing an indicator function of (k != I) according to the Eq.(8) in ContraGAN.\n    * Line 161 - increase -> increases, decrease -> decreases\n    * Eq.(14) - Third term in the middle h(x) -> g(x)\n    * Table 4 - bolding is missing for some cells. \n    * Citations [11] and [12] are mixed up in the related work section.\n\nSignificance:\n- Although there are some remaining questions as specified above and experiments are somewhat limited, this work can benefit the community by providing a unified framework for cGANs. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some limitations that the authors did not recognize are provided above. ", "review_text": "", "extra_scores": {"ethical_concerns": "The authors acknowledged that the proposed algorithm can be potentially misused through creating malicious contents. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a framework for thinking about several existing class-conditional GAN approaches in a unified way, which also leads to a new suggested algorithm. The algorithm outperforms existing approaches on TinyImageNet, and some initial results are shown in the rebuttal that it also does so on ImageNet. I think the framework is sensible and, from what we can tell, the new algorithm is probably an improvement over existing ones. Please ensure that you address the various concerns raised by reviewers, as well as adding the new results, in your final version.
Decision: Accept (Poster)