Class Balancing GAN with a Classifier in the Loop

1 INTRODUCTION . Image Generation witnessed unprecedented success in recent years following the invention of Generative Adversarial Networks ( GANs ) by Goodfellow et al . ( 2014 ) . GANs have improved significantly over time with the introduction of better architectures ( Gulrajani et al. , 2017 ; Radford et al. , 2015 ) , formulation of superior objective functions ( Jolicoeur-Martineau , 2018 ; Arjovsky et al. , 2017 ) , and regularization techniques ( Miyato et al. , 2018 ) . An important breakthrough for GANs has been the ability to effectively use the information of class conditioning for synthesizing images ( Mirza & Osindero , 2014 ; Miyato & Koyama , 2018 ) . Conditional GANs have been shown to scale to large datasets such as ImageNet ( Deng et al. , 2009 ) with 1000 classes ( Miyato & Koyama , 2018 ) . One of the major issues with unconditional GANs has been their inability to produce balanced distributions over all the classes present in the dataset . This is seen as problem of missing modes in the generated distribution . A version of the missing modes problem , known as the ‘ covariate shift ’ problem was studied by Santurkar et al . ( 2018 ) . One possible reason may be the absence of knowledge about the class distribution P ( Y |X ) 1 of the generated samples during training . Conditional GANs on the other hand , do not suffer from this issue since the class label Y is supplied to the GAN during training . However , it has been recently found by Ravuri & Vinyals ( 2019 ) that despite being able to do well on metrics such as Inception Score ( IS ) ( Salimans et al . ( 2016 ) ) and Frèchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) , the samples generated from the state-of-the-art conditional GANs lack diversity in comparison to the underlying training datasets . Further , we observed that although conditional GANs work well in balanced case , they suffer performance degradation in the imbalanced case . In order to address these shortcomings , we propose an orthogonal method ( with respect to label conditioning ) to induce the information about the class distribution P ( Y |X ) of generated samples in the GAN framework using a pre-trained classifier . We achieve this by tracking the class distribution of samples produced by the GAN using a pre-trained classifier . The regularizer utilizes the class distribution to penalize excessive generation of samples from the majority classes , thus enforcing 1Here Y represents labels and X represents data . the GAN to generate samples from minority classes . Our regularizer involves a novel method of modelling the forgetting of samples by GANs , based on the exponential forgetting observed in neural networks ( Kirkpatrick et al . ( 2017 ) ) . We infer the implications of our regularizer by a theoretical bound and empirically verify the same . We conduct empirical analysis of the proposed class balancing regularizer in two diverse and challenging scenarios : ( i ) Training GANs for image generation on long-tailed datasets : Generally , even in the long-tailed distribution tasks , the test set is balanced despite the imbalance in the training set . This is because it is important to develop Machine Learning systems that generalize well across all the support regions of the data distribution , avoiding undesired over-fitting to the majority ( or head ) classes . Hence , it is pertinent to train GANs that can faithfully represent all classes . ( ii ) Transferring the knowledge from a learnt classifier ( P ( Y |Xt ) ) to a GAN being trained on arbitrary prior distribution P ( Xp ) : This is a specific situation where the samples from target distribution Xt are unavailable . Instead , discriminative feature knowledge is indirectly available in the form of a trained classifier ( P ( Y |Xt ) ) . This is a perfect fit for crafting input-agnostic ( Universal ) adversarial perturbations in the data-free scenario . We show that the proposed regularizer can enable the generated samples to not only extract information about the target data with a trained classifier in the loop , but also represent its support to a greater extent . In summary , our contributions can be listed as follows : • We propose a ‘ class-balancing ’ regularizer that makes use of the statistics ( P ( Y |X ) ) of generated samples to promote uniformity while sampling from an unconditional GAN . The effect of our regularizer is depicted both theoretically ( Section 3 ) and empirically ( Section 4 ) . • We show that our regularizer enables GANs to learn uniformly across classes even when the training distribution is long-tailed . We observe gains in FID and accuracy of a classifier trained on generated samples . • We also show that by combining a pre-trained classifier ( i.e . P ( Y |Xt ) ) trained on a target dataset Xt , with an arbitrary distribution P ( Xp ) , our framework is capable of synthesizing novel samples related to the target dataset . We show that UAPs created on such novel samples generalize to real target data and hence lead to an effective data-free attack . This application is novel to our framework and can not be realized by conditional GANs . 2 BACKGROUND . 2.1 GENERATIVE ADVERSARIAL NETWORKS ( GANS ) . Generative Adversarial Networks ( GANs ) are formulated as a two player game in which the discriminator D tries to classify images into two classes : real and fake . The generator G tries to generate images ( transforming a noise vector z ∼ Pz ) which fool the discriminator ( D ) into classifying them as real . The game can be formulated by the following objective : min G max D Ex∼Pr [ log ( D ( x ) ) ] + Ez∼Pz [ log ( 1−D ( G ( z ) ) ] ( 1 ) The exact optimization for trainingD is computationally prohibitive in large networks and the GAN is trained by alternative minimization using loss functions . Multiple loss functions have been proposed for stabilizing the GAN training . In our work we use the relativistic loss function ( JolicoeurMartineau , 2018 ) which is formulated as : LrelD = −E ( x , z ) ∼ ( Pr , Pz ) [ log ( σ ( D ( x ) −D ( G ( z ) ) ) ] ( 2 ) LrelG = −E ( x , z ) ∼ ( Pr , Pz ) [ log ( σ ( D ( G ( z ) ) −D ( x ) ) ] ( 3 ) This unconditional GAN formulation does not have any class conditioning and produces different number of samples from different classes ( Santurkar et al. , 2018 ) . In other words , the distribution is not balanced ( uniform ) across different classes for the generated data . 2.2 CONDITIONAL GAN . The conditional GAN ( Mirza & Osindero , 2014 ) generates images associated with input label y using the following objective : min G max D Ex∼Pr [ log ( D ( x|y ) ) ] + Ez∼Pz [ log ( 1−D ( G ( z|y ) ) ] ( 4 ) The Auxillary Classifier GAN ( ACGAN ) ( Odena et al. , 2017 ) uses an auxiliary classifier for classification along with normal discriminator to enforce high confidence samples from the conditioned class y . Whereas cGAN with projection ( Miyato & Koyama , 2018 ) uses Conditional Batch Norm ( De Vries et al. , 2017 ) in the generator and uses a projection step in the discriminator to provide class information to the GAN . We refer to this method as cGAN in the subsequent sections . Possible issue with Conditional GAN in Long-tailed Setting : The objective in eq . ( 4 ) can be seen as learning a different G ( z|y ) and D ( x|y ) for each of the K classes . In this case the tail classes with fewer samples can suffer from poor generalization as they have very few samples . However , in practice there is parameter sharing among different class generators but still class specific parameters are also present in form of Conditional BatchNorm . We find that performance of conditional GANs degrade more in comparison to unconditonal GANs in the long-tailed scenario ( Section 4 ) . 3 METHOD . In our method we propose to introduce a pretrained classifier ( C ) to provide feedback to the generator about the label distribution P ( Y ) over the generated images . The proposed regularizer is added with the generator loss and trained using backpropogation . We first describe the method of modelling in Section 3.1 . The exact formulation of the regularizer and its theoretical properties are described in Section 3.2 . The overview of our method is presented in Figure ( 1a ) . 3.1 CLASS STATISTICS FOR GAN . GAN is a dynamic system in which the generator G has to continuously adapt itself in a way that it is able to fool the discriminator D. During the training , discriminator D updates itself , causing the objective for the generator G also to change . This change in objective can be seen as learning of different tasks for the generator G. In this context , we draw motivation from the seminal work on catastrophic forgetting in neural networks ( Kirkpatrick et al. , 2017 ) which shows that a neural network trained using SGD suffers from exponential forgetting of earlier tasks when trained on a new task . Based on this , we define effective class frequency N̂ tk of class k at cycle t as : N̂ tk = ( 1− α ) ˆN t−1k + c t−1 k ( 5 ) Here ct−1k is the number of samples of class k produced by the GAN in cycle ( t − 1 ) . The class of the sample is determined by the pretrained classifier C. Although D gets updated continuously , the update is slow and requires some iterations to change the form of D. Hence we update the statistics after certain number of iterations which compose a cycle . Here α is the exponential forgetting factor which is set to 0.5 in all our experiments . We normalize the class frequency N̂ tk to obtain discrete effective class distribution : N tk = N̂ tk∑ k N̂ t k ( 6 ) 3.2 REGULARIZER FORMULATION . The regularizer objective is defined as the maximization of the term ( Lreg ) below : max p̂ ∑ k p̂k log ( p̂k ) N tk ( 7 ) where p̂ = ∑n i=1 C ( G ( zi ) ) n . In other words , p̂ is the average softmax vector ( obtained from the classifier C ) over the batch of n samples and p̂k is its kth component corresponding to class k. zi corresponds to random noise vector sampled from Pz . If the classifier C recognizes the samples confidently with probability ≈ 1 , p̂k can be seen as the approximation to the ratio of the number of samples that belong to class k to the total number of samples in the batch n. TheN tk in the regularizer term is obtained through the update rule in Section 3.1 and is a constant during backpropagation . We want to emphasize that classifier C is not required to be trained on same data as the GAN , instead it can be trained in ways such as semi-supervised learning , few-shot learning , etc . For instance , in section 4.2 we show that a classifier trained in a semi-supervised scenario also enables the GAN to produce a balanced distribution . Hence our approach doesn ’ t specifically need labelled data which is in contrast to conditional GANs which require labels for each image while training . Proposition : The maximization of the proposed objective in ( 7 ) leads to the following bound on p̂k : p̂k ≤ e −K ( log ( K ) −1 ) N t k∑ k N t k −1 ( 8 ) where K is the number of distinct class labels produced by classifier C. Please refer to the appendix Section A.1 for proof of the same . Implications of the proposition : The bound on p̂k is inversely related to the exponent of the fraction of effective class frequency N tk/ ∑ kN t k for a given class k. In case of generating a balanced distribution , p̂k = 1/K which leads to the exponential average N tk = 1/K . Hence given sufficient iterations , the p̂k value will achieve the upper bound which signifies tightness of the same . To demonstrate effect of the regularizer empirically , we construct two extreme case examples based on the nature of the bound : • IfN tk N ti , ∀i 6= k , then the bound on p̂k would approach e−K ( log ( K ) −1 ) −1 . Hence the network is expected to decrease the proportion of class k samples . • If N tk N ti , ∀i 6= k , then the bound on p̂k will be e−1 . Hence the network is expected to increase the proportion of class k samples . We verified the two extreme cases above by training a SNDCGAN ( Miyato et al. , 2018 ) ( DCGAN with spectral normalization ) on CIFAR-10 and fixing N̂ tk ( unnormalized version of N t k ) across time steps and term it as Nk . Then we initialize Nk to a very large value and a very small value . Results presented in Figure ( 1b ) show that the GAN increases the proportion of samples of class k in case of low Nk and decreases the proportion of samples in case of large Nk . This shows the balancing behaviour of proposed regularizer .

Summary: The paper proposes a regularizer to force an unconditional GAN generator to produce samples that follow a uniform class distribution. To provide feedback to the generator about the class distribution over the generated images, the proposed method utilizes a pretrained classifier on the same (imbalanced) training dataset. Motivated by the exponential forgetting of earlier tasks in neural networks [1], the regularization term encourages the generator to increase the proportion of samples of an infrequent class after a certain number of iterations and vice versa. Empirical studies are performed to show the effectiveness of the regularization: 1) the paper shows that the proposed method enables generating samples with a uniform class distribution with a GAN trained on a dataset with a long-tailed class distribution and (2) that the method benefits in generating universal adversarial perturbations (UAPs) in the data-free scenario.