A Unified View of cGANs with and without Classifiers
Year: 2022

Abstract:
Conditional Generative Adversarial Networks (cGANs) are implicit generative models which allow to sample from class-conditional distributions. Existing cGANs are based on a wide range of different discriminator designs and training objectives. One popular design in earlier works is to include a classifier during training with the assumption that good classifiers can help eliminate samples generated with wrong classes. Nevertheless, including classifiers in cGANs often comes with a side effect of only generating easy-to-classify samples. Recently, some representative cGANs avoid the shortcoming and reach state-of-the-art performance without having classifiers. Somehow it remains unanswered whether the classifiers can be resurrected to design better cGANs. In this work, we demonstrate that classifiers can be properly leveraged to improve cGANs. We start by using the decomposition of the joint probability distribution to connect the goals of cGANs and classification as a unified framework. The framework, along with a classic energy model to parameterize distributions, justifies the use of classifiers for cGANs in a principled manner. It explains several popular cGAN variants, such as ACGAN, ProjGAN, and ContraGAN, as special cases with different levels of approximations, which provides a unified view and brings new insights to understanding cGANs. Experimental results demonstrate that the design inspired by the proposed framework outperforms state-of-the-art cGANs on multiple benchmark datasets, especially on the most challenging ImageNet. The code is available at https://github.com/sian-chen/PyTorch-ECGAN.

heading: 1 Introduction
text: Generative Adversarial Networks [GANs; 10] is a family of generative models that are trained from the duel of a generator and a discriminator. The generator aims to generate data from a target distribution, where the fidelity of the generated data is “screened” by the discriminator. Recent studies on the objectives [2, 37, 29, 25, 36, 26, 38], backbone architectures [41, 50], and regularization techniques [13, 35, 51] for GANs have achieved impressive progress on image generation, making GANs the state-of-the-art approach to generate high fidelity and diverse images [3]. Conditional GANs (cGANs) extend GANs to generate data from class-conditional distributions [33, 39, 34, 16]. The capability of conditional generation extends the application horizon of GANs to conditional image generation based on labels [39] or texts [43], speech enhancement [32], and image style transformation [18, 53].
One representative cGAN is Auxiliary Classifier GAN [ACGAN; 39], which decomposes the conditional discriminator to a classifier and an unconditional discriminator. The generator of ACGAN is expected to generate images that convince the unconditional discriminator while being classified to
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
the right class. The classifier plays a pivotal role in laying down the law of conditional generation for ACGAN, making it the very first cGAN that can learn to generate 1000 classes of ImageNet images [6]. That is, ACGAN used to be a leading cGAN design. While the classifier in ACGAN indeed improves the quality of conditional generation, deeper studies revealed that the classifier biases the generator to generate easier-to-classify images [45], which in term decreases the capability to match the target distribution.
Unlike ACGAN, most state-of-the-art cGANs are designed without a classifier. One representative cGAN without a classifier is Projection GAN [ProjGAN; 34], which learns an embedding for each class to form a projection-based conditional discriminator. ProjGAN not only generates higher-quality images than ACGAN, but also accurately generates images in target classes without relying on an explicit classifier. In fact, it was found that ProjGAN usually cannot be further improved by adding a classification loss [34]. The finding, along with the success of ProjGAN and other cGANs without classifiers [15, 4], seem to suggest that including a classifier is not helpful for improving cGANs.
In this work, we challenge the belief that classifiers are not helpful for cGANs, with the conjecture that leveraging the classifiers appropriately can benefit conditional generation. We propose a framework that pins down the roles of the classifier and the conditional discriminator by first decomposing the joint target distribution with Bayes rule. We then model the conditional discriminator as an energy function, which is an unnormalized log probability. Under the energy function, we derive the corresponding optimization term for the classifier and the conditional discriminator with the help of Fenchel duality to form the unified framework. The framework reveals that a jointly generative model can be trained via two routes, from the aspect of the classifier and the conditional discriminator, respectively. We name our framework Energy-based Conditional Generative Adversarial Networks (ECGAN), which not only justifies the use of classifiers for cGANs in a principled manner, but also explains several popular cGAN variants, such as ACGAN [39], ProjGAN [34], and ContraGAN [16] as special cases with different approximations. After properly combining the objectives from the two routes of the framework, we empirically find that ECGAN outperforms other cGAN variants across different backbone architectures on benchmark datasets, including the most challenging ImageNet.
We summarize the contributions of this paper as:
• We justify the principled use of classifiers for cGANs by decomposing the joint distribution.
• We propose a cGAN framework, Energy-based Conditional Generative Adversarial Networks (ECGAN), which explains several popular cGAN variants in a unified view.
• We experimentally demonstrate that ECGAN consistently outperforms other state-of-the-art cGANs across different backbone architectures on benchmark datasets.
The paper is organized as follows. Section 2 derives the unified framework that establishes the role of the classifiers for cGANs. The framework is used to explain ACGAN [39], ProjGAN [34], and ContraGAN [16] in Section 3. Then, we demonstrate the effectiveness of our framework by experiments in Section 4. We discuss related work in Section 5 before concluding in Section 6.

heading: 2 Method
text: Given a K-class dataset (x, y) ∼ pd, where y ∈ {1 . . .K} is the class of x and pd is the underlying data distribution. Our goal is to train a generator G to generate a sample G(z, y) following pd(x|y), where z is sampled from a known distribution such as N (0, 1). To solved the problem, a typical cGAN framework can be formulated by extending an unconditional GAN as:
max D min G ∑ y E pd(x|y) D(x, y)− E p(z) D(G(z, y), y) (1)
where G is the generator and D is a discriminator that outputs higher values for real data. The choice of D leads to different types of GANs [10, 2, 29, 8].
At first glance, there is no classifier in Eq. (1). However, because of the success of leveraging label information via classification, it is hypothesized that a better classifier can improve conditional generation [39]. Motivated by this, in this section, we show how we bridge classifiers to cGANs by Bayes rule and Fenchel duality.

heading: 2.1 Bridge Classifiers to Discriminators with Joint Distribution
text: A classifier, when viewed from a probabilistic perspective, is a function that approximates pd(y|x), the probability that x belongs to class y. On the other hand, a conditional discriminator, telling whether x is real data in class y, can be viewed as a function approximate pd(x|y). To connect pd(y|x) and pd(x|y), an important observation is through the joint probability:
log p(x, y) = log p(x|y) + log p(y) (2) = log p(y|x) + log p(x). (3)
The observation illustrates that we can approximate log p(x, y) in two directions: one containing p(x|y) for conditional discriminators and one containing p(y|x) for classifiers. The finding reveals that by sharing the parameterization, updating the parameters in one direction may optimize the other implicitly. Therefore, we link the classifier to the conditional discriminator by training both objectives jointly.

heading: 2.2 Learning Joint Distribution via Optimizing Conditional Discriminators
text: Since p(y) is usually known a priori (e.g., uniform) or able to easily estimated (e.g., empirical counting), we focus on learning p(x|y) in Eq.(2). Specifically, since log p(x, y) ∈ R, we parameterize it via fθ(x), such as a neural network with K real value outputs, where exp(fθ(x)[y]) ∝ p(x, y) . Similar parameterization is also used in exponential family [48] and energy based model [23]. Therefore, the log-likelihood log p(x|y) can be modeled as:
log pθ(x|y) = log ( exp (fθ(x)[y])
Zy(θ)
) = fθ(x)[y]− logZy(θ), (4)
where Zy(θ) = ∫ x′ exp (fθ(x ′)[y]) dx′.
Optimizing Eq. (4) is challenging because of the intractable partition function Zy(θ). Here we introduce the Fenchel duality [48] of the partition function Zy(θ):
logZy(θ) = max qy
[ E
qy(x) [fθ(x)[y]] +H(qy) ] where qy is a distribution of x conditioned on y and H(qy) = −Exf∼qy(x) [log qy(x)] is the entropy of qy. The derivation is provided in Appendix A. By the Fenchel duality, we obtain our maximum likelihood estimation in Eq. (4) as:
max θ
[ E
pd(x,y) [fθ(x)[y]]−max qy
[ E
qy(x) [fθ(x)[y]] +H(qy)
]] . (5)
To approximate the solution of qy , in additional to density models, we can train an auxiliary generator qφ as in cGANs to estimate Eqy(x) via sampling. That is, we can sample x from qφ by x = qφ(z, y), where z ∼ N (0, 1). The objective (5) then becomes:
max θ min φ ∑ y E pd(x|y) [fθ(x)[y]]− E p(z) [fθ(qφ(z, y))[y]]−H(qφ(·, y)), (6)
which is almost in the form of Eq (1) except the entropy H(qφ(·, y)). We leave the discussion about the entropy estimation in Section 2.4. Currently, the loss function to optimize the objective without the entropy can be formulated as:
Ld1(x, z, y; θ) = −fθ(x)[y] + fθ(qφ(z))[y] Lg1(z, y;φ) = −fθ(qφ(z, y))[y]

heading: 2.3 Learning Joint Distributions via Optimizing Unconditional Discriminators & Classifiers
text: Following Eq. (3), we can approximate log p(x, y) by approximating log p(y|x) and log p(x). With our energy function fθ, pθ(y|x) can be formulated as:
pθ(y|x) = pθ(x, y)
pθ(x) = exp(fθ(x)[y])∑ y′ exp(fθ(x)[y ′]) ,
which is equivalent to the y’th output of SOFTMAX(fθ(x)). Therefore, we can maximize the loglikelihood of pθ(y|x) by consider fθ as a softmax classifier minimizing the cross-entropy loss:
Lclf(x, y; θ) = − log (SOFTMAX (fθ(x)) [y])
On the other hand, to maximize the log-likelihood of p(x), we introduce a reparameterization hθ(x) = log ∑ y exp(fθ(x)[y]):
log pθ(x) = log (∑ y pθ(x, y) ) = log (∑ y exp(fθ(x)[y])∫ x′ ∑ y′ exp(fθ(x ′)[y′]) dx′ )
= log
( exp(log( ∑ y exp(fθ(x)[y])))∫
x′ exp(log( ∑ y′ exp(fθ(x ′)[y′]))) dx′
) = log ( exp(hθ(x))∫
x′ exp(hθ(x′)) dx′ ) = hθ(x)− log(Z ′(θ)), (7)
where Z ′(θ) = ∫ x exp(hθ(x)) dx. Similar to Eq. (5), we can rewrite logZ ′(θ) by its Fenchel duality:
logZ ′(θ) = max q [ E q(x) [hθ(x)] +H(q) ] (8)
where q is a distribution of x and H(q) is the entropy of q.
Combining Eq. (7) and Eq. (8) and reusing the generator in Section 2.2, we obtain the optimization problem:
max θ min φ E pd(x,y) [hθ(x)]− E p(z) [hθ(qφ(z, y))]−H(qφ) (9)
Similar to Eq. (6), the objective of the unconditional discriminator is equivalent to typical GANs augmented with an entropy term. The loss function without considering the entropy can be formulated as:
Ld2(x, z, y; θ) = −hθ(x) + hθ(qφ(z)) Lg2(z, y;φ) = −hθ(qφ(z, y))

heading: 2.4 Entropy Approximation in cGANs
text: In Section 2.2 and Section 2.3, we propose two approaches to train cGANs with and without classification. Unsolved problems in Eq. (6) and Eq. (9) are the entropy termsH(qφ(·, y)) andH(qφ). In previous work, various estimators have been proposed to estimate entropy or its gradient [46, 42, 21, 27]. One can freely choose any approach to estimate the entropy in the proposed framework. In this work, we consider two entropy estimators, and we will show how they connect with existing cGANs.
The first approach is the naive constant approximation. Since entropy is always non-negative, we naturally have the constant zero as a lower bound. Therefore, we can maximize the objective by replacing the entropy term with its lower bound, which is zero in this case. This approach is simple but we will show its effectiveness in Section 4 and how it links our framework to ProjGAN and ContraGAN in Section 3.
The second approach is estimating a variational lower bound. Informally, given a batch of data {(x1, y1), . . . , (xm, ym)}, an encoder function l, and a class embedding function e(y), the negative 2C loss used in ContraGAN [16],
LC(xi, yi; t) = log ( d(l(xi), e(yi)) + ∑m k=1 Jyk = yiK d(l(xi), l(xk))
d(l(xi), e(yi)) + ∑m k=1 Jk 6= iK d(l(xi), (l(xk))
) , (10)
is an empirical estimate of a proper lower bound of H(X) [40], where d(a, b) = exp(a>b/t) is a distance function with a temperature t. We provide the proof in Appendix B.
The 2C loss heavily relies on the embeddings l(x) and e(y). Although we only need to estimate the entropy of generated data in Eq. (6) and Eq. (9), we still rely on true data to learn the embeddings in
practice. Therefore, the loss function of Eq. (6) can be written as:
LD1(x, z, y; θ) = Ld1(x, z, y; θ) + λcLrealC LG1(z, y;φ) = Lg1(x, y;φ) + λcLfakeC ,
where λc is a hyperparameter controlling the weight of the contrastive loss, and LrealC ,LfakeC are the contrastive loss calculated on a batch of real data and generated data respectively.
Similarly, the loss function of Eq. (9) becomes:
LD2(x, z, y; θ) = Ld2(x, z, y; θ) + λcLrealC LG2(z, y;φ) = Lg2(x, y;φ) + λcLfakeC ,
The introduction of 2C loss allows us to accommodate ContraGAN into our framework.

heading: 2.5 Energy-based Conditional Generative Adversarial Network
text: Previous work has shown that multitask training benefits representation learning [30] and training discriminative and generative models jointly outperforms their purely generative or purely discriminative counterparts [11, 28]. Therefore, we propose a framework named Energy-based Conditional Generative Adversarial Network (ECGAN), which combines the two approaches in Section 2.2 and Section 2.3 to learn the joint distribution better. The loss function can be summarized as:
LD(x, z, y; θ) = Ld1(x, z, y; θ) + αLd2(x, z, y; θ) + λcLrealC + λclfLclf(x, y; θ) (11) LG(z, y;φ) = Lg1(z, y;φ) + αLg2(z, y;φ) + λcLfakeC (12)
where α is a weight parameter for the unconditional GAN loss. The discriminator’s design is illustrated in Fig 1.
Here we discuss the intuition of the mechanisms behind each component in Eq. (11). Ld1 is a loss function for conditional discriminator. It updates the y-th output when given a data pair (x, y). Ld2 guides to an unconditional discriminator. It updates all outputs according to whether x is real. Lclf learns a classifier. It increases the y-th output and decreases the other outputs for data belonging to class y. Finally, LrealC and L fake C play the roles to improve the latent embeddings by pulling the embeddings of data with the same class closer.
Previously, we derive the loss functions Ld1 and Ld2 as the loss in Wasserstein GAN [2]. In practice, we use the hinge loss as proposed in Geometric GAN [26] for better stability and convergence. We use the following combination of Ld1 and Ld2 :
Hinge(fθ(xreal, y) + α · hθ(xreal), fθ(xfake, y) + α · hθ(xfake)). (13) For more discussion of the implementation of hinge loss, please check Appendix C. The overall training procedure of ECGAN is presented in Appendix E.

heading: 3 Accommodation to Existing cGANs
text: In this section, we show that our framework covers several representative cGAN algorithms, including ACGAN [39], ProjGAN [35], and ContraGAN [16]. Through the ECGAN framework, we obtain a unified view of cGANs, which allows us to fairly compare and understand the pros and cons of existing cGANs. We name the ECGAN counterparts ECGAN-0, ECGAN-C, and ECGAN-E, corresponding to ProjGAN, ACGAN, and ContraGAN, respectively. We summarize the settings in Table 1 and illustrate the discriminator designs in Appendix F.

heading: 3.1 ProjGAN
text: ProjGAN [34] is the most representative cGAN design that is commonly used in state-of-the-art research [3, 50]. Let the output of the penultimate layer in the discriminator be g(x). The output of ProjGAN’s discriminator is:
D(x, y) = wTu g(x) + bu + w T y g(x) = (wu + wy) T g(x) + bu (14)
where wu, bu are the parameters for the unconditional linear layer, and wy is the class embedding of y. On the other hand, the output of a discriminator in ECGAN is:
D(x, y) = f(x)[y] = (WT g(x) + b)[y] = wTy g(x) + by (15)
where W,b are the parameters of the linear output layer in fθ. As shown in Eq. (14) and Eq. (15), the architectures of ProjGAN and ECGAN are almost equivalent. In addition, the loss function of ProjGAN can be formulated as:
LG = −D(G(z), y) LD = −D(x, y) +D(G(z), y),
which is a special case of ECGAN while α = λc = λclf = 0. We name this case ECGAN-0, which is the simplest version of ECGAN. Compared with ProjGAN, ECGAN-0 has additional bias terms for the output of each class.

heading: 3.2 ACGAN
text: ACGAN [39] is the most well-known cGAN algorithm that leverages a classifier to achieve conditional generation. Given aK-class dataset, the discriminator of ACGAN is parameterized by a network with K + 1 outputs. The first output, denoted as D(x), is an unconditional discriminator distinguishing between real and fake images. The remaining K outputs, denoted as C(x), is a classifier that predicts logits for every class. The loss function of ACGAN can be formulated as:
LG = −D(G(z)) + λgLclf(G(z), y;C) LD = −D(x) +D(G(z)) + λd(Lclf(x, y;C) + Lclf(G(z), y;C))
where G is the generator, λg and λd are hyperparameters to control the weight of cross-entropy loss.
The formulation of ACGAN is similar to our ECGAN when α = λc = 0 and λclf > 0. We call the special case as ECGAN-C, with a suffix ‘C’ for classification loss. ECGAN-C uses a conditional discriminator which plays the role of a classifier at the same time. Hence the generator in ECGAN-C learns from the conditional discriminator rather than the cross-entropy loss which is biased for generative objectives.

heading: 3.3 ContraGAN
text: ContraGAN [16] proposed 2C loss, which we mentioned in Eq. (10), to capture the data-to-data relationship and data-to-label relationship. The 2C loss is applied in both discriminator and generator to achieve conditional generation. That is:
LG = −D(G(z), y) + λcLfakeC LD = −D(x, y) +D(G(z), y) + λcLrealC
The loss functions are similar to ones in ECGAN with α = λclf = 0 and λc > 0. We call it ECGANE, where ‘E’ means entropy estimation. The main difference between ContraGAN and ECGAN-E is the output layer of their discriminators. While ContraGAN uses a single-output network, ECGAN uses a K-output network fθ which has higher capacity.
We keep Eq. (11) and Eq. (12) as simple as possible to reduce the burden of hyperparameter tuning. Under the simple equations of the current framework, ECGAN-C and ECGAN-E are the closest counterparts to ACGAN and ContraGAN. The subtle difference (in addition to the underlying network architecture) is that ACGAN uses Ld2 instead of Ld1 (ECGAN-C); ContraGAN uses Ld2 ,Lg2 instead of Ld1 ,Lg1 (ECGAN-E). One future direction is to introduce more hyperparameters in Eq. (11) and Eq. (12) to get closer counterparts.

heading: 4 Experiment
text: We conduct our experiments on CIFAR-10 [20] and Tiny ImageNet [22] for analysis, and ImageNet [6] for large-scale empirical study. Table 2 shows the statistics of the datasets. All datasets are publicly available for research use. They were not constructed for human-related study. We do not specifically take any personal information from the datasets in our experiments.
In our experiment, we use two common metrics, Frechet Inception Distance [FID; 14] and Inception Score [IS; 44], to evaluate our generation quality and diversity. Besides, we use Intra-FID, which is the average of FID for each class, to evaluate the performance of conditional generation.

heading: 4.1 Experimental Setup
text: We use StudioGAN1 [16] to conduct our experiments. StudioGAN is a PyTorch-based project distributed under the MIT license that provides implementation and benchmark of several popular GAN architectures and techniques. To provide reliable evaluation, we conduct experiments on CIFAR-10 and Tiny ImageNet with 4 different random seeds and report the means and standard deviations for each metric. We evaluate the model with the lowest FID for each trial. The default backbone architecture is BigGAN [3]. We fix the learning rate for generators and discriminators to 0.0001 and 0.0004, respectively, and tune λclf in {1, 0.1, 0.05, 0.01}. We follow the setting λc = 1 in [16] when using 2C loss, and set α = 1 when applying unconditional GAN loss. The experiments take 1-2 days on single GPU (Nvidia Tesla V100) machines for CIFAR-10, Tiny ImageNet, and take 6 days on 8-GPU machines for ImageNet. More details are described in Appendix D.

heading: 4.2 Ablation Study
text: We start our empirical studies by investigating the effectiveness of each component in ECGAN. We use symbols ‘U’ to represent unconditional GAN loss, ‘C’ to represent classification loss, and ‘E’
1https://github.com/POSTECH-CVLab/PyTorch-StudioGAN
to represent entropy estimation loss, which is 2C loss in our implementation. The concatenation of the symbols indicates the combination of losses. For example, ECGAN-UC means ECGAN with both unconditional GAN loss and classification loss (α > 0 and λclf > 0). Table 3 shows the results of ECGAN from the simplest ECGAN-0 to the most complicated ECGAN-UCE. On CIFAR-10, ECGAN-0 already achieves decent results. Adding unconditional loss, classification loss, or contrastive loss provides slightly better or on-par performance. On the harder Tiny Imagenet, the benefit of unconditional loss and classification loss becomes more significant. While ECGAN-U already shows advantages to ECGAN-0, adding classification loss to ECGAN-U further improves all metrics considerably. We also observe that directly adding classification loss is not sufficient to improve cGAN, which is consistent to the finding in [34]. The fact reveals that the unconditional GAN loss is a crucial component to bridge classifiers and discriminators in cGANs. We also find that adding contrastive loss does not improve ECGAN-UC. An explanation is that the entropy estimation lower bound provided by the contrastive loss is too loose to benefit the training. Furthermore, the additional parameters introduced by 2C loss make the optimization problem more complicated. As a result, we use the combination ECGAN-UC as the default option of ECGAN in the following experiments.

heading: 4.3 Comparison with Existing cGANs
text: We compare ECGAN to several representative cGANs including ACGAN [39], ProjGAN [34], and ContraGAN [16], with three representative backbone architectures: DCGAN [41], ResNet [13], and BigGAN [3]. Table 4 compares the results of each combinations of cGAN algorithms and backbone architectures. The results show that ECGAN-UC outperforms other cGANs significantly with all backbone architectures on both CIFAR-10 and Tiny ImageNet. We also noticed that ContraGAN, though achieves decent image quality and diversity, learns a conditional generator that interchanges some classes while generating, hence has low Intra-FID. Overall, the experiment indicates that ECGAN-UC can be a preferred choice for cGAN in general situations.

heading: 4.4 Comparisons between Existing cGANs and their ECGAN Counterpart
text: Table 5 compares ProjGAN, ContraGAN, ACGAN to their ECGAN counterparts. As we described in Section 3, each of these representative cGANs can be viewed as special cases under our ECGAN framework. As mentioned in Section 3, ECGAN-0 has additional bias terms in the output layer compared to ProjGAN. The results in Table 5 shows that the subtle difference still brings significant improvement to the generation quality, especially on the harder Tiny ImageNet.
Compared to ContraGAN, ECGAN-E has the same loss but different design in the discriminator’s output layer. While the discriminator of ContraGAN has only single output, ECGAN-E has multiple outputs for every class. The difference makes ECGAN-E solve the label mismatching problem of ContraGAN mentioned in Section 4.3 and benefits generation on CIFAR-10, but does not work well on Tiny ImageNet. It is probably because of the scarcity of training data in each class in Tiny ImageNet. Only 50 data are available for updating the parameters corresponding to each class.
Last, we compare ECGAN-C to ACGAN. Both of them optimize a GAN loss and a classification loss. However, ECGAN-C combines the discriminator and the classifier, so the generator can directly optimize cGAN loss rather than the classification loss. As a result, ECGAN-C demonstrates better performance on both CIFAR-10 and Tiny ImageNet. In sum, the comparisons show that through the unified view provided by ECGAN, we can improve the existing methods with minimal modifications.

heading: 4.5 Evaluation on ImageNet
text: We compare our ECGAN-UC and ECGAN-UCE with BigGAN [3] and ContraGAN [16] on ImageNet. We follow all configurations of BigGAN with batch size 256 in StudioGAN. The numbers in Table 6 are reported after 200,000 training steps if not specified. The results show that ECGAN-UCE outperforms other cGANs dramatically. The comparison between ECGAN-UC and ECGAN-UCE indicates that the 2C loss brings more significant improvement in the ECGAN framework than in ContraGAN. The proposed ECGAN-UCE achieves 8.49 FID and 80.69 inception score. To the best of our knowledge, this is a state-of-the-art result of GANs with batch size 256 on ImageNet. Selected generated images are shown in Appendix G.

heading: 5 Related Work
text: The development of cGANs started from feeding label embeddings to the inputs of GANs or the feature vector at some middle layer [33, 7]. To improve the generation quality, ACGAN [39] proposes to leverage classifiers and successfully generates high-resolution images. The use of classifiers
in GANs is studied in Triple GAN [24] for semi-supervised learning and Triangle GAN [9] for cross-domain distribution matching. However, Shu [45] and Miyato and Koyama [34] pointed out that the auxiliary classifier in ACGAN misleads the generator to generate images that are easier to be classified. Thus, whether classifiers can help conditional generation still remains questionable.
In this work, we connect cGANs with and without classifiers via an energy model parameterization from the joint probability perspective. [12] use similar ideas but focus on sampling from the trained classifier via Markov Chain Monte Carlo [MCMC; 1]. Our work is also similar to a concurrent work [11], which improves [12] by introducing Fenchel duality to replace computationallyintensive MCMC. They use a variational approach [19] to formulate the objective for tractable entropy estimation. In contrast, we study the GAN perspective and the entropy estimation via contrastive learning. Therefore, the proposed ECGAN can be treated a complements works compared with [12, 11] by studying a GAN perspective. We note that the studied cGAN approaches also result in better generation quality than its variational alternative [11].
Last, [5] study the connection between exponential family and unconditional GANs. Different from [5], we study the conditional GANs with the focus to provide a unified view of common cGANs and an insight into the role of classifiers in cGANs.

heading: 6 Conclusion
text: In this work, we present a general framework Energy-based Conditional Generative Networks (ECGAN) to train cGANs with classifiers. With the framework, we can explain representative cGANs, including ACGAN, ProjGAN, and ContraGAN, in a unified view. The experiments demonstrate that ECGAN outperforms state-of-the-art cGANs on benchmark datasets, especially on the most challenging ImageNet. Further investigation can be conducted to find a better entropy approximation or improve cGANs by advanced techniques for classifiers. We hope this work can pave the way to more advanced cGAN algorithms in the future.

heading: 7 Limitations and Potential Negative Impacts
text: There are two main limitations in the current study. One is the investigation on ImageNet. Ideally, more experiments and analysis on ImageNet can further strengthen the contribution. But training with such a large dataset is barely affordable for our computational resource, and we can only resort to the conclusive findings in the current results. The other limitation is whether the metrics such as FID truly reflect generation quality, but this limitation is considered an open problem to the community anyway.
As with any work on generative models, there is a potential risk of the proposed model being misused to create malicious content, much like how misused technology can be used to forge bills. In this sense, more anti-forgery methods will be needed to mitigate the misuse in the future.

heading: Acknowledgement
text: We thank the anonymous reviewers for valuable suggestions. This work is partially supported by the Ministry of Science and Technology of Taiwan via the grants MOST 107-2628-E-002-008-MY3 and 110-2628-E-002-013. We also thank the National Center for High-performance Computing (NCHC) of National Applied Research Laboratories (NARLabs) in Taiwan for providing computational resources.

Review summaries:
The authors of the paper propose a unified view of conditional generative adversarial networks (cGANs). To this end, they analyze the log of joint probability distribution p(x, y) through two different perspective of views; via a conditional discriminator using Eq.(2), and via an unconditional discriminator and classifier using Eq.(3). Inspired by a multitask learning, they suggest Energy-based Conditional Generative Adversarial Networks (ECGAN) where two approaches are combined together to approximate the joint probability distribution. Under this framework, previous cGANs such as ProjGAN, ACGAN and ContraGAN can be closely explained using the variants of ECGAN with the right choice of hyperparameters, namely, ECGAN-0, ECGAN-C and ECGAN-E, respectively.  Their experiments on CIFAR-10 and Tiny ImageNet show that these ECGAN variants outperform ProjGAN, ACGAN and ContraGAN in most cases of experiments. Furthermore, ECGAN-UC outperforms these variants of ECGAN as well as previous cGANs with various backbone structures such as DCGAN, ResGAN, and BigGAN. Through this result, the authors conclude that adding a classifier helps improving cGANs with the help of the unconditional discriminator as opposed to the previous findings in Shu et al. [42] and ProjGAN [34].	

This submission proposes to analyze the most popular variations of conditional GANs (ACGAN, ProjGAN, ContraGAN) under a unified, energy-based, formulation (ECGAN).  Specifically, ECGAN is composed of terms derived from two different decompositions of the joint probability p(x,y). The paper then links each term to one of the popular cGAN approaches.  ECGAN is evaluated against the traditional models and an extensive ablation study tests the impact of each component of the proposed model. 

This paper introduced a general framework for conditional image generation. And existing cGAN methods such as ACGAN, ProjGAN and ContraGAN are inclued in the proposed algorithm. Experimental results in Cifar and Tiny ImageNet verified the effectiveness of different components of the proposed method.

The authors discuss that existing methods on including classifiers in a cGAN biases the generator in generating easy to classify images. Therefore, they propose a way to include classifiers in a cGAN to improve its performance in a principled manner. To do so, they decompose the joint probability distribution by the Bayes rule that results in linking classifiers to conditional discriminators. The proposed formulation shows that a joint generator model can be trained from two directions: a conditional discriminator and an un-conditional discriminator with a classifier. They combine the formulation of these two routes and propose a new method called Energy-based cGAN (ECGAN). ECGAN shows how to use a classifier for cGANs, and it explains other variants of cGANs such as ContraGAN, ACGAN, and ProjGAN. They empirically show that ECGAN outperforms existing cGANs by achieving higher FID (and similar ones) score on two sets of datasets (CIFAR10, Tiny ImageNet).