Conditional GANs with Auxiliary Discriminative Classifier

1 INTRODUCTION . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) have been gained great progress in learning high-dimensional , complex data distribution such as natural images ( Karras et al. , 2019 ; 2020b ; a ; Brock et al. , 2019 ) . Standard GANs consist of a generator network that transfers a latent code sampled from a tractable distribution in the latent space to a data point in the data space and a discriminator network that attempts to distinguish between the real data and the generated one . The generator is trained in an adversarial game against the discriminator such that it can replicate the data distribution at the Nash equilibrium of the game . Remarkably , the training of GANs is notoriously unstable to reach the equilibrium , and thereby the generator is prone to mode collapse ( Salimans et al. , 2016 ; Lin et al. , 2018 ; Chen et al. , 2019 ) . In addition , practitioners are interested in controlling the properties of the generated samples ( Yan et al. , 2015 ; Tan et al. , 2020 ) in practical applications . A key solution to address the above issues is conditioning , leading to conditional GANs ( Mirza & Osindero , 2014 ) . Conditional GANs ( cGANs ) is a family variant of GANs that leverages the side information from annotated labels of samples to implement and train a conditional generator , and therefore achieve conditional image generation from class-label ( Odena et al. , 2017 ; Miyato & Koyama , 2018 ; Brock et al. , 2019 ) or text ( Reed et al. , 2016 ; Xu et al. , 2018 ; Zhu et al. , 2019 ) . To implement the conditional generator , the common technique nowadays injects the conditional information via conditional batch normalization ( de Vries et al. , 2017 ) . To train the conditional generator , a lot of efforts focus on effectively injecting the conditional information into the discriminator or classifier ( Odena , 2016 ; Miyato & Koyama , 2018 ; Zhou et al. , 2018 ; Kavalerov et al. , 2021 ; Kang & Park , 2020 ; Zhou et al. , 2020 ) . Among them , the auxiliary classifier generative adversarial network ( AC-GAN ) ( Odena et al. , 2017 ) has been widely used due to its simplicity and extensibility . Specifically , AC-GAN utilizes an auxiliary classifier that first attempts to recognize the label of data and then teaches the generator to produce label-consistent ( classifiable ) data . However , it has been reported that AC-GAN suffers from the low intra-class diversity problem on generated samples , especially on datasets with a large number of classes ( Odena et al. , 2017 ; Shu et al. , 2017 ; Gong et al. , 2019 ) . In this paper , we point out that the fundamental reason for the low intra-class diversity problem of AC-GAN is that the classifier is agnostic to the generated data distribution and thus can not provide informative guidance to the generator in learning the target distribution . Motivated by this observation , we propose a novel conditional GAN with an auxiliary discriminative classifier , namely ADC-GAN , to resolve the problem of AC-GAN by enabling the classifier to be aware of the generated data distribution . To this end , the discriminative classifier is trained to distinguish between the real and generated data while recognizing their labels . The discriminative property enables the classifier to provide the discrepancy between the real and generated data distributions analogy to the discriminator , and the classification property allows it to capture the dependencies between the data and labels . We show in theory that the generator of the proposed ADC-GAN can replicate the joint data and label distribution under the guidance of the discriminative classifier at the optima even without the discriminator , making our method robust to hyper-parameter and stable on training . We also discuss the superiority of ADC-GAN compared to two most related works ( TAC-GAN ( Gong et al. , 2019 ) and PD-GAN ( Miyato & Koyama , 2018 ) ) by analyzing their potential issues and limitations . Experimental results clearly show that the proposed ADC-GAN successfully resolves the problem of AC-GAN by faithfully learning the real joint data and label distribution . The advantages over competing cGANs in experiments conducted on both synthetic and real-world datasets verify the effectiveness of the proposed ADC-GAN in conditional generative modeling . 2 PRELIMINARIES AND OUR ANALYSIS . 2.1 GENERATIVE ADVERSARIAL NETWORKS . Generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) consist of two types of neural networks : the generator G : Z → X that maps a latent code z ∈ Z endowed with an easily sampled distribution PZ to a data point x ∈ X , and the discriminator D : X → [ 0 , 1 ] that distinguishes between real data that sampled from the real data distribution PX and fake data that sampled from the generated data distribution QX = G ◦ PZ implied by the generator . The goal of the generator is to confuse the discriminator by producing data that is as real as possible . Formally , the objective functions for the discriminator and the generator are defined as follows : min G max D V ( G , D ) = Ex∼PX [ logD ( x ) ] + Ex∼QX [ log ( 1−D ( x ) ) ] . ( 1 ) Theoretically , the learning of generator under an optimal discriminator can be regarded as minimizing the Jensen-Shannon ( JS ) divergence between the real data distribution and the generated data distribution , i.e. , minG JS ( PX‖QX ) . This would enable the generator to recover the real data distribution at its optima . However , the training of GANs is notoriously unstable , especially when lacking additional supervision such as conditional information . Moreover , the content of the generated images of GANs can not be specified in advance . 2.2 AC-GAN . Learning GANs with conditional information can not only improve the training stability and generation quality of GANs but also achieve conditional generation , which has more practical value than unconditional generation in real-world applications . One of the most representative conditional GANs is AC-GAN ( Odena et al. , 2017 ) , which utilizes an auxiliary classifier C : X → Y to learn the dependencies between the real data x ∼ PX and the label y ∼ PY and then enforce the conditional generator G : Z × Y → X to synthesize classifiable data as much as possible . The objective functions for the discriminator D , the auxiliary classifier C , and the generator G of AC-GAN are defined as follows1 : max D , C V ( G , D ) + λ · ( Ex , y∼PX , Y [ logC ( y|x ) ] ) , min G V ( G , D ) − λ · ( Ex , y∼QX , Y [ logC ( y|x ) ] ) , ( 2 ) where λ > 0 is a hyper-parameter , and QX , Y = G ◦ ( PZ × PY ) denotes the joint distribution of generated data and labels implied by the generator . 1We follow the common practice in the literature to adopt the stable version instead of the original one . Proposition 1 . The optimal classifier of AC-GAN outputs as follows : C∗ ( y|x ) = p ( x , y ) p ( x ) . ( 3 ) Theorem 1 . Given the optimal classifier , at the equilibrium point , optimizing the classification task for the generator of AC-GAN is equivalent to : min G KL ( QX , Y ‖PX , Y ) − KL ( QX‖PX ) +HQ ( Y |X ) , ( 4 ) where HQ ( Y |X ) = − ∫ ∑ y q ( x , y ) log q ( y|x ) dx is the conditional entropy of generated data . The proofs of Proposition 1 and Theorem 1 are referred to Appendix A.1 and A.2 , respectively . Our Theorem 1 exposes two shortcomings of AC-GAN . First , maximization of the KL divergence between the marginal generator distribution and the marginal data distribution maxG KL ( QX‖PX ) contradicts the goal of conditional generative modeling that matches QX , Y with PX , Y . Although this issue can be mitigated to some extent by the adversarial training objective between the discriminator and the generator that minimizes the JS divergence between the two marginal distributions , we find that it still has a negative impact on the training stability . Second , minimization of the entropy of label conditioned on data with respect to the generated distribution minGHQ ( Y |X ) will result in that the label of generated data should be completely determined by the data itself . In other words , it will force the generated data of each class away from the classification hyper-plane , explaining the low intra-class diversity of generated samples in AC-GAN especially when the distributions of different classes have non-negligible overlap , which is supported by the fact that state-of-the-art classifiers nor human can not achieve 100 % accuracy on real-world datasets ( Russakovsky et al. , 2015 ) . Note that the original version of AC-GAN , whose classifier is trained by both real and generated samples , could also suffer from the same issue ( see Appendix B ) . 3 THE PROPOSED METHOD : ADC-GAN . The goal of conditional generative modeling is to faithfully approximate the joint distribution of real data and labels regardless of the shape of the target joint distribution ( whether there is overlap between distributions of different classes ) . Note that the learning of the generator in AC-GAN is affected by the classifier . In other words , the reason for the consequence of Theorem 1 originates from Proposition 1 , which indicates that the optimal classifier of AC-GAN is agnostic to the density of the generated ( marginal or joint ) distribution ( q ( x ) or q ( x , y ) ) . Therefore , the classifier can not provide the discrepancy between the target distribution and the generated distribution , resulting in a biased learning objective to the generator . Recall that the optimal discriminator D∗ ( x ) = p ( x ) p ( x ) +q ( x ) is able to be aware of the real data density as well as the generated data density ( Goodfellow et al. , 2014 ) , and thus can provide the discrepancy p ( x ) q ( x ) = D∗ ( x ) 1−D∗ ( x ) between the real data distribution and the generated data distribution to unbiasedly optimize the generator . Intuitively , the densityaware ability on both real and generated data is caused by the fact that the discriminator attempts to distinguish between real and fake samples . Motivated by this observation , we propose to make the classifier to be distinguishable between real and fake samples , establishing a discriminative classifier Cd : X → Y × { 0 , 1 } that recognizes the label of real and fake samples discriminatively . Formally , the objective functions for the discriminator D , the discriminative classifier Cd , and the generator G of the proposed ADC-GAN are defined as follows : max D , Cd V ( G , D ) + λ · ( Ex , y∼PX , Y [ logCd ( y , 1|x ) ] + Ex , y∼QX , Y [ logCd ( y , 0|x ) ] ) , min G V ( G , D ) − λ · ( Ex , y∼QX , Y [ logCd ( y , 1|x ) ] − Ex , y∼QX , Y [ logCd ( y , 0|x ) ] ) , ( 5 ) where Cd ( y , 1|x ) ( reps. Cd ( y , 0|x ) ) denotes the probability that a data x is classified as the label y and real ( reps. fake ) data simultaneously . Proposition 2 . For fixed generator , the optimal classifier of ADC-GAN outputs as follows : C∗d ( y , 1|x ) = p ( x , y ) p ( x ) + q ( x ) , C∗d ( y , 0|x ) = q ( x , y ) p ( x ) + q ( x ) . ( 6 ) The proof is referred to Appendix A.3 . Proposition 2 confirms that the discriminative classifier be aware of the densities of the real and generated joint distributions , therefore it is able to provide the discrepancy p ( x , y ) q ( x , y ) = C∗d ( y,1|x ) C∗d ( y,0|x ) to unbiasedly optimize the generator as we prove below . Theorem 2 . Given the optimal classifier , at the equilibrium point , optimizing the classification task for the generator of ADC-GAN is equivalent to : min G KL ( QX , Y ‖PX , Y ) . ( 7 ) The proof is referred to Appendix A.4 . Theorem 2 suggests that the classifier itself can guarantee the generator to replicate the real joint distribution in theory regardless of the shape of the joint distribution . In practice , we retain the discriminator to train the generator and share all layers but the head of the classifier with the discriminator as illustrated in Figure 1 and Equation 5 for faster convergence speed . Coupled with the adversarial training against the discriminator , the generator of the proposed ADC-GAN , under the optimal discriminator and classifier , can be regarded as minimizing the following divergences : minG JS ( PX‖QX ) + λ · KL ( QX , Y ‖PX , Y ) . Since the optimal solution of conditional generative modeling belongs to the optimal solution set of generative modeling , i.e. , argminG KL ( QX , Y ‖PX , Y ) ⊆ argminG JS ( PX‖QX ) , learning with the discriminator will not change the convergence point of the generator that approximates the joint distribution of real data and labels regardless of the value of hyper-parameter λ > 0 . Furthermore , the hyper-parameter λ provides the flexibility to adjust the weight of conditional generative modeling .

Summary: - This paper aims to solve the low intra-class diversity on generated images of AC-GAN, a classifier-based cGAN.  - As far as I know, this is an important issue that limits classifier-based cGANs (the counterpart is the projection-based cGAN, i.e, PD-GAN). - The authors point out that the reason is that the classifier of AC-GAN is generator-agnostic and minimization of conditional entropy decreases the intra-class diversity.  - The authors propose ADC-GAN (auxiliary discriminative classifier) to solve this problem, and theoretical analysis is also presented.