Conditional GANs with Auxiliary Discriminative Classifier

1 INTRODUCTION . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have been gained great progress in learning high-dimensional , complex data distribution such as natural images ( Karras et al. , 2019 ; 2020b ; a ; Brock et al. , 2019 ) . Standard GANs consist of a generator network that transfers a latent code sampled from a tractable distribution in the latent space to a data point in the data space and a discriminator network that attempts to distinguish between the real data and the generated one . The generator is trained in an adversarial game against the discriminator such that it can replicate the data distribution at the Nash equilibrium of the game . Remarkably , the training of GANs is notoriously unstable to reach the equilibrium , and thereby the generator is prone to mode collapse ( Salimans et al. , 2016 ; Lin et al. , 2018 ; Chen et al. , 2019 ) . In addition , practitioners are interested in controlling the properties of the generated samples ( Yan et al. , 2015 ; Tan et al. , 2020 ) in practical applications . A key solution to address the above issues is conditioning , leading to conditional GANs ( Mirza & Osindero , 2014 ) . Conditional GANs ( cGANs ) is a family variant of GANs that leverages the side information from annotated labels of samples to implement and train a conditional generator , and therefore achieve conditional image generation from class-label ( Odena et al. , 2017 ; Miyato & Koyama , 2018 ; Brock et al. , 2019 ) or text ( Reed et al. , 2016 ; Xu et al. , 2018 ; Zhu et al. , 2019 ) . To implement the conditional generator , the common technique nowadays injects the conditional information via conditional batch normalization ( de Vries et al. , 2017 ) . To train the conditional generator , a lot of efforts focus on effectively injecting the conditional information into the discriminator or classifier ( Odena , 2016 ; Miyato & Koyama , 2018 ; Zhou et al. , 2018 ; Kavalerov et al. , 2021 ; Kang & Park , 2020 ; Zhou et al. , 2020 ) . Among them , the auxiliary classifier generative adversarial network ( AC-GAN ) ( Odena et al. , 2017 ) has been widely used due to its simplicity and extensibility . Specifically , AC-GAN utilizes an auxiliary classifier that first attempts to recognize the label of data and then teaches the generator to produce label-consistent ( classifiable ) data . However , it has been reported that AC-GAN suffers from the low intra-class diversity problem on generated samples , especially on datasets with a large number of classes ( Odena et al. , 2017 ; Shu et al. , 2017 ; Gong et al. , 2019 ) . In this paper , we point out that the fundamental reason for the low intra-class diversity problem of AC-GAN is that the classifier is agnostic to the generated data distribution and thus can not provide informative guidance to the generator in learning the target distribution . Motivated by this observation , we propose a novel conditional GAN with an auxiliary discriminative classifier , namely ADC-GAN , to resolve the problem of AC-GAN by enabling the classifier to be aware of the generated data distribution . To this end , the discriminative classifier is trained to distinguish between the real and generated data while recognizing their labels . The discriminative property enables the classifier to provide the discrepancy between the real and generated data distributions analogy to the discriminator , and the classification property allows it to capture the dependencies between the data and labels . We show in theory that the generator of the proposed ADC-GAN can replicate the joint data and label distribution under the guidance of the discriminative classifier at the optima even without the discriminator , making our method robust to hyper-parameter and stable on training . We also discuss the superiority of ADC-GAN compared to two most related works ( TAC-GAN ( Gong et al. , 2019 ) and PD-GAN ( Miyato & Koyama , 2018 ) ) by analyzing their potential issues and limitations . Experimental results clearly show that the proposed ADC-GAN successfully resolves the problem of AC-GAN by faithfully learning the real joint data and label distribution . The advantages over competing cGANs in experiments conducted on both synthetic and real-world datasets verify the effectiveness of the proposed ADC-GAN in conditional generative modeling . 2 PRELIMINARIES AND OUR ANALYSIS . 2.1 GENERATIVE ADVERSARIAL NETWORKS . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) consist of two types of neural networks : the generator G : Z → X that maps a latent code z ∈ Z endowed with an easily sampled distribution PZ to a data point x ∈ X , and the discriminator D : X → [ 0 , 1 ] that distinguishes between real data that sampled from the real data distribution PX and fake data that sampled from the generated data distribution QX = G ◦ PZ implied by the generator . The goal of the generator is to confuse the discriminator by producing data that is as real as possible . Formally , the objective functions for the discriminator and the generator are defined as follows : min G max D V ( G , D ) = Ex∼PX [ logD ( x ) ] + Ex∼QX [ log ( 1−D ( x ) ) ] . ( 1 ) Theoretically , the learning of generator under an optimal discriminator can be regarded as minimizing the Jensen-Shannon ( JS ) divergence between the real data distribution and the generated data distribution , i.e. , minG JS ( PX‖QX ) . This would enable the generator to recover the real data distribution at its optima . However , the training of GANs is notoriously unstable , especially when lacking additional supervision such as conditional information . Moreover , the content of the generated images of GANs can not be specified in advance . 2.2 AC-GAN . Learning GANs with conditional information can not only improve the training stability and generation quality of GANs but also achieve conditional generation , which has more practical value than unconditional generation in real-world applications . One of the most representative conditional GANs is AC-GAN ( Odena et al. , 2017 ) , which utilizes an auxiliary classifier C : X → Y to learn the dependencies between the real data x ∼ PX and the label y ∼ PY and then enforce the conditional generator G : Z × Y → X to synthesize classifiable data as much as possible . The objective functions for the discriminator D , the auxiliary classifier C , and the generator G of AC-GAN are defined as follows1 : max D , C V ( G , D ) + λ · ( Ex , y∼PX , Y [ logC ( y|x ) ] ) , min G V ( G , D ) − λ · ( Ex , y∼QX , Y [ logC ( y|x ) ] ) , ( 2 ) where λ > 0 is a hyper-parameter , and QX , Y = G ◦ ( PZ × PY ) denotes the joint distribution of generated data and labels implied by the generator . 1We follow the common practice in the literature to adopt the stable version instead of the original one . Proposition 1 . The optimal classifier of AC-GAN outputs as follows : C∗ ( y|x ) = p ( x , y ) p ( x ) . ( 3 ) Theorem 1 . Given the optimal classifier , at the equilibrium point , optimizing the classification task for the generator of AC-GAN is equivalent to : min G KL ( QX , Y ‖PX , Y ) − KL ( QX‖PX ) +HQ ( Y |X ) , ( 4 ) where HQ ( Y |X ) = − ∫ ∑ y q ( x , y ) log q ( y|x ) dx is the conditional entropy of generated data . The proofs of Proposition 1 and Theorem 1 are referred to Appendix A.1 and A.2 , respectively . Our Theorem 1 exposes two shortcomings of AC-GAN . First , maximization of the KL divergence between the marginal generator distribution and the marginal data distribution maxG KL ( QX‖PX ) contradicts the goal of conditional generative modeling that matches QX , Y with PX , Y . Although this issue can be mitigated to some extent by the adversarial training objective between the discriminator and the generator that minimizes the JS divergence between the two marginal distributions , we find that it still has a negative impact on the training stability . Second , minimization of the entropy of label conditioned on data with respect to the generated distribution minGHQ ( Y |X ) will result in that the label of generated data should be completely determined by the data itself . In other words , it will force the generated data of each class away from the classification hyper-plane , explaining the low intra-class diversity of generated samples in AC-GAN especially when the distributions of different classes have non-negligible overlap , which is supported by the fact that state-of-the-art classifiers nor human can not achieve 100 % accuracy on real-world datasets ( Russakovsky et al. , 2015 ) . Note that the original version of AC-GAN , whose classifier is trained by both real and generated samples , could also suffer from the same issue ( see Appendix B ) . 3 THE PROPOSED METHOD : ADC-GAN . The goal of conditional generative modeling is to faithfully approximate the joint distribution of real data and labels regardless of the shape of the target joint distribution ( whether there is overlap between distributions of different classes ) . Note that the learning of the generator in AC-GAN is affected by the classifier . In other words , the reason for the consequence of Theorem 1 originates from Proposition 1 , which indicates that the optimal classifier of AC-GAN is agnostic to the density of the generated ( marginal or joint ) distribution ( q ( x ) or q ( x , y ) ) . Therefore , the classifier can not provide the discrepancy between the target distribution and the generated distribution , resulting in a biased learning objective to the generator . Recall that the optimal discriminator D∗ ( x ) = p ( x ) p ( x ) +q ( x ) is able to be aware of the real data density as well as the generated data density ( Goodfellow et al. , 2014 ) , and thus can provide the discrepancy p ( x ) q ( x ) = D∗ ( x ) 1−D∗ ( x ) between the real data distribution and the generated data distribution to unbiasedly optimize the generator . Intuitively , the densityaware ability on both real and generated data is caused by the fact that the discriminator attempts to distinguish between real and fake samples . Motivated by this observation , we propose to make the classifier to be distinguishable between real and fake samples , establishing a discriminative classifier Cd : X → Y × { 0 , 1 } that recognizes the label of real and fake samples discriminatively . Formally , the objective functions for the discriminator D , the discriminative classifier Cd , and the generator G of the proposed ADC-GAN are defined as follows : max D , Cd V ( G , D ) + λ · ( Ex , y∼PX , Y [ logCd ( y , 1|x ) ] + Ex , y∼QX , Y [ logCd ( y , 0|x ) ] ) , min G V ( G , D ) − λ · ( Ex , y∼QX , Y [ logCd ( y , 1|x ) ] − Ex , y∼QX , Y [ logCd ( y , 0|x ) ] ) , ( 5 ) where Cd ( y , 1|x ) ( reps. Cd ( y , 0|x ) ) denotes the probability that a data x is classified as the label y and real ( reps. fake ) data simultaneously . Proposition 2 . For fixed generator , the optimal classifier of ADC-GAN outputs as follows : C∗d ( y , 1|x ) = p ( x , y ) p ( x ) + q ( x ) , C∗d ( y , 0|x ) = q ( x , y ) p ( x ) + q ( x ) . ( 6 ) The proof is referred to Appendix A.3 . Proposition 2 confirms that the discriminative classifier be aware of the densities of the real and generated joint distributions , therefore it is able to provide the discrepancy p ( x , y ) q ( x , y ) = C∗d ( y,1|x ) C∗d ( y,0|x ) to unbiasedly optimize the generator as we prove below . Theorem 2 . Given the optimal classifier , at the equilibrium point , optimizing the classification task for the generator of ADC-GAN is equivalent to : min G KL ( QX , Y ‖PX , Y ) . ( 7 ) The proof is referred to Appendix A.4 . Theorem 2 suggests that the classifier itself can guarantee the generator to replicate the real joint distribution in theory regardless of the shape of the joint distribution . In practice , we retain the discriminator to train the generator and share all layers but the head of the classifier with the discriminator as illustrated in Figure 1 and Equation 5 for faster convergence speed . Coupled with the adversarial training against the discriminator , the generator of the proposed ADC-GAN , under the optimal discriminator and classifier , can be regarded as minimizing the following divergences : minG JS ( PX‖QX ) + λ · KL ( QX , Y ‖PX , Y ) . Since the optimal solution of conditional generative modeling belongs to the optimal solution set of generative modeling , i.e. , argminG KL ( QX , Y ‖PX , Y ) ⊆ argminG JS ( PX‖QX ) , learning with the discriminator will not change the convergence point of the generator that approximates the joint distribution of real data and labels regardless of the value of hyper-parameter λ > 0 . Furthermore , the hyper-parameter λ provides the flexibility to adjust the weight of conditional generative modeling .

Summary: The paper is about improving conditional GANs. To be specifically, it also aims to resolve the bias issue of ACGAN by proposing a discriminative classifier. The discriminative classifier is a hybrid model of discriminator and classifier, where it has to not tell real or fake, but also the class.  Preliminary analysis of the proposed method are provided. Experiments are conducted on the standard benchmarks.