A Unified View of cGANs with and without Classifiers

1 Introduction . Generative Adversarial Networks [ GANs ; 10 ] is a family of generative models that are trained from the duel of a generator and a discriminator . The generator aims to generate data from a target distribution , where the fidelity of the generated data is “ screened ” by the discriminator . Recent studies on the objectives [ 2 , 37 , 29 , 25 , 36 , 26 , 38 ] , backbone architectures [ 41 , 50 ] , and regularization techniques [ 13 , 35 , 51 ] for GANs have achieved impressive progress on image generation , making GANs the state-of-the-art approach to generate high fidelity and diverse images [ 3 ] . Conditional GANs ( cGANs ) extend GANs to generate data from class-conditional distributions [ 33 , 39 , 34 , 16 ] . The capability of conditional generation extends the application horizon of GANs to conditional image generation based on labels [ 39 ] or texts [ 43 ] , speech enhancement [ 32 ] , and image style transformation [ 18 , 53 ] . One representative cGAN is Auxiliary Classifier GAN [ ACGAN ; 39 ] , which decomposes the conditional discriminator to a classifier and an unconditional discriminator . The generator of ACGAN is expected to generate images that convince the unconditional discriminator while being classified to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the right class . The classifier plays a pivotal role in laying down the law of conditional generation for ACGAN , making it the very first cGAN that can learn to generate 1000 classes of ImageNet images [ 6 ] . That is , ACGAN used to be a leading cGAN design . While the classifier in ACGAN indeed improves the quality of conditional generation , deeper studies revealed that the classifier biases the generator to generate easier-to-classify images [ 45 ] , which in term decreases the capability to match the target distribution . Unlike ACGAN , most state-of-the-art cGANs are designed without a classifier . One representative cGAN without a classifier is Projection GAN [ ProjGAN ; 34 ] , which learns an embedding for each class to form a projection-based conditional discriminator . ProjGAN not only generates higher-quality images than ACGAN , but also accurately generates images in target classes without relying on an explicit classifier . In fact , it was found that ProjGAN usually can not be further improved by adding a classification loss [ 34 ] . The finding , along with the success of ProjGAN and other cGANs without classifiers [ 15 , 4 ] , seem to suggest that including a classifier is not helpful for improving cGANs . In this work , we challenge the belief that classifiers are not helpful for cGANs , with the conjecture that leveraging the classifiers appropriately can benefit conditional generation . We propose a framework that pins down the roles of the classifier and the conditional discriminator by first decomposing the joint target distribution with Bayes rule . We then model the conditional discriminator as an energy function , which is an unnormalized log probability . Under the energy function , we derive the corresponding optimization term for the classifier and the conditional discriminator with the help of Fenchel duality to form the unified framework . The framework reveals that a jointly generative model can be trained via two routes , from the aspect of the classifier and the conditional discriminator , respectively . We name our framework Energy-based Conditional Generative Adversarial Networks ( ECGAN ) , which not only justifies the use of classifiers for cGANs in a principled manner , but also explains several popular cGAN variants , such as ACGAN [ 39 ] , ProjGAN [ 34 ] , and ContraGAN [ 16 ] as special cases with different approximations . After properly combining the objectives from the two routes of the framework , we empirically find that ECGAN outperforms other cGAN variants across different backbone architectures on benchmark datasets , including the most challenging ImageNet . We summarize the contributions of this paper as : • We justify the principled use of classifiers for cGANs by decomposing the joint distribution . • We propose a cGAN framework , Energy-based Conditional Generative Adversarial Networks ( ECGAN ) , which explains several popular cGAN variants in a unified view . • We experimentally demonstrate that ECGAN consistently outperforms other state-of-the-art cGANs across different backbone architectures on benchmark datasets . The paper is organized as follows . Section 2 derives the unified framework that establishes the role of the classifiers for cGANs . The framework is used to explain ACGAN [ 39 ] , ProjGAN [ 34 ] , and ContraGAN [ 16 ] in Section 3 . Then , we demonstrate the effectiveness of our framework by experiments in Section 4 . We discuss related work in Section 5 before concluding in Section 6 . 2 Method . Given a K-class dataset ( x , y ) ∼ pd , where y ∈ { 1 . . .K } is the class of x and pd is the underlying data distribution . Our goal is to train a generator G to generate a sample G ( z , y ) following pd ( x|y ) , where z is sampled from a known distribution such as N ( 0 , 1 ) . To solved the problem , a typical cGAN framework can be formulated by extending an unconditional GAN as : max D min G ∑ y E pd ( x|y ) D ( x , y ) − E p ( z ) D ( G ( z , y ) , y ) ( 1 ) where G is the generator and D is a discriminator that outputs higher values for real data . The choice of D leads to different types of GANs [ 10 , 2 , 29 , 8 ] . At first glance , there is no classifier in Eq . ( 1 ) . However , because of the success of leveraging label information via classification , it is hypothesized that a better classifier can improve conditional generation [ 39 ] . Motivated by this , in this section , we show how we bridge classifiers to cGANs by Bayes rule and Fenchel duality . 2.1 Bridge Classifiers to Discriminators with Joint Distribution . A classifier , when viewed from a probabilistic perspective , is a function that approximates pd ( y|x ) , the probability that x belongs to class y . On the other hand , a conditional discriminator , telling whether x is real data in class y , can be viewed as a function approximate pd ( x|y ) . To connect pd ( y|x ) and pd ( x|y ) , an important observation is through the joint probability : log p ( x , y ) = log p ( x|y ) + log p ( y ) ( 2 ) = log p ( y|x ) + log p ( x ) . ( 3 ) The observation illustrates that we can approximate log p ( x , y ) in two directions : one containing p ( x|y ) for conditional discriminators and one containing p ( y|x ) for classifiers . The finding reveals that by sharing the parameterization , updating the parameters in one direction may optimize the other implicitly . Therefore , we link the classifier to the conditional discriminator by training both objectives jointly . 2.2 Learning Joint Distribution via Optimizing Conditional Discriminators . Since p ( y ) is usually known a priori ( e.g. , uniform ) or able to easily estimated ( e.g. , empirical counting ) , we focus on learning p ( x|y ) in Eq. ( 2 ) . Specifically , since log p ( x , y ) ∈ R , we parameterize it via fθ ( x ) , such as a neural network with K real value outputs , where exp ( fθ ( x ) [ y ] ) ∝ p ( x , y ) . Similar parameterization is also used in exponential family [ 48 ] and energy based model [ 23 ] . Therefore , the log-likelihood log p ( x|y ) can be modeled as : log pθ ( x|y ) = log ( exp ( fθ ( x ) [ y ] ) Zy ( θ ) ) = fθ ( x ) [ y ] − logZy ( θ ) , ( 4 ) where Zy ( θ ) = ∫ x′ exp ( fθ ( x ′ ) [ y ] ) dx′ . Optimizing Eq . ( 4 ) is challenging because of the intractable partition function Zy ( θ ) . Here we introduce the Fenchel duality [ 48 ] of the partition function Zy ( θ ) : logZy ( θ ) = max qy [ E qy ( x ) [ fθ ( x ) [ y ] ] +H ( qy ) ] where qy is a distribution of x conditioned on y and H ( qy ) = −Exf∼qy ( x ) [ log qy ( x ) ] is the entropy of qy . The derivation is provided in Appendix A . By the Fenchel duality , we obtain our maximum likelihood estimation in Eq . ( 4 ) as : max θ [ E pd ( x , y ) [ fθ ( x ) [ y ] ] −max qy [ E qy ( x ) [ fθ ( x ) [ y ] ] +H ( qy ) ] ] . ( 5 ) To approximate the solution of qy , in additional to density models , we can train an auxiliary generator qφ as in cGANs to estimate Eqy ( x ) via sampling . That is , we can sample x from qφ by x = qφ ( z , y ) , where z ∼ N ( 0 , 1 ) . The objective ( 5 ) then becomes : max θ min φ ∑ y E pd ( x|y ) [ fθ ( x ) [ y ] ] − E p ( z ) [ fθ ( qφ ( z , y ) ) [ y ] ] −H ( qφ ( · , y ) ) , ( 6 ) which is almost in the form of Eq ( 1 ) except the entropy H ( qφ ( · , y ) ) . We leave the discussion about the entropy estimation in Section 2.4 . Currently , the loss function to optimize the objective without the entropy can be formulated as : Ld1 ( x , z , y ; θ ) = −fθ ( x ) [ y ] + fθ ( qφ ( z ) ) [ y ] Lg1 ( z , y ; φ ) = −fθ ( qφ ( z , y ) ) [ y ] 2.3 Learning Joint Distributions via Optimizing Unconditional Discriminators & Classifiers . Following Eq . ( 3 ) , we can approximate log p ( x , y ) by approximating log p ( y|x ) and log p ( x ) . With our energy function fθ , pθ ( y|x ) can be formulated as : pθ ( y|x ) = pθ ( x , y ) pθ ( x ) = exp ( fθ ( x ) [ y ] ) ∑ y′ exp ( fθ ( x ) [ y ′ ] ) , which is equivalent to the y ’ th output of SOFTMAX ( fθ ( x ) ) . Therefore , we can maximize the loglikelihood of pθ ( y|x ) by consider fθ as a softmax classifier minimizing the cross-entropy loss : Lclf ( x , y ; θ ) = − log ( SOFTMAX ( fθ ( x ) ) [ y ] ) On the other hand , to maximize the log-likelihood of p ( x ) , we introduce a reparameterization hθ ( x ) = log ∑ y exp ( fθ ( x ) [ y ] ) : log pθ ( x ) = log ( ∑ y pθ ( x , y ) ) = log ( ∑ y exp ( fθ ( x ) [ y ] ) ∫ x′ ∑ y′ exp ( fθ ( x ′ ) [ y′ ] ) dx′ ) = log ( exp ( log ( ∑ y exp ( fθ ( x ) [ y ] ) ) ) ∫ x′ exp ( log ( ∑ y′ exp ( fθ ( x ′ ) [ y′ ] ) ) ) dx′ ) = log ( exp ( hθ ( x ) ) ∫ x′ exp ( hθ ( x′ ) ) dx′ ) = hθ ( x ) − log ( Z ′ ( θ ) ) , ( 7 ) where Z ′ ( θ ) = ∫ x exp ( hθ ( x ) ) dx . Similar to Eq . ( 5 ) , we can rewrite logZ ′ ( θ ) by its Fenchel duality : logZ ′ ( θ ) = max q [ E q ( x ) [ hθ ( x ) ] +H ( q ) ] ( 8 ) where q is a distribution of x and H ( q ) is the entropy of q . Combining Eq . ( 7 ) and Eq . ( 8 ) and reusing the generator in Section 2.2 , we obtain the optimization problem : max θ min φ E pd ( x , y ) [ hθ ( x ) ] − E p ( z ) [ hθ ( qφ ( z , y ) ) ] −H ( qφ ) ( 9 ) Similar to Eq . ( 6 ) , the objective of the unconditional discriminator is equivalent to typical GANs augmented with an entropy term . The loss function without considering the entropy can be formulated as : Ld2 ( x , z , y ; θ ) = −hθ ( x ) + hθ ( qφ ( z ) ) Lg2 ( z , y ; φ ) = −hθ ( qφ ( z , y ) )

Summary: This submission proposes to analyze the most popular variations of conditional GANs (ACGAN, ProjGAN, ContraGAN) under a unified, energy-based, formulation (ECGAN).  Specifically, ECGAN is composed of terms derived from two different decompositions of the joint probability p(x,y). The paper then links each term to one of the popular cGAN approaches.  ECGAN is evaluated against the traditional models and an extensive ablation study tests the impact of each component of the proposed model.