Intervention Generative Adversarial Nets

1 INTRODUCTION . As one of the most important advances in generative models in recent years , Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have been attracting great attention in the machine learning community . GANs aim to train a generator network that transforms simple vectors of noise to produce “ realistic ” samples from the data distribution . In the basic training process of GANs , a discriminator and a target generator are trained in an adversarial manner . The discriminator tries to distinguish the generated fake samples from the real ones , and the generator tries to fool the discriminator into believing the generated samples to be real . Though successful , there are two major challenges in training GANs : the instability of the training process and the mode collapse problem . To deal with these problems , one class of approaches focus on designing more informative objective functions ( Salimans et al. , 2016 ; Mao et al. , 2016 ; Kodali et al. , 2018 ; Arjovsky & Bottou ; Arjovsky et al. , 2017 ; Gulrajani et al. , 2017 ; Zhou et al. , 2019 ) . For example , Mao et al . ( 2016 ) proposed Least Squares GAN ( LSGAN ) that uses the least squares loss to penalize the outlier point more harshly . Arjovsky & Bottou discussed the role of the Jensen-Shannon divergence in training GANs and proposed WGAN ( Arjovsky et al. , 2017 ) and WGAN-GP ( Gulrajani et al. , 2017 ) that use the more informative Wasserstein distance instead . Other approaches enforce proper constraints on latent space representations to better capture the data distribution ( Makhzani et al. , 2015 ; Larsen et al. , 2015 ; Che et al. , 2016 ; Tran et al. , 2018 ) . A representative work is the Adversarial Autoencoders ( AAE ) ( Makhzani et al. , 2015 ) which uses the discriminator to distinguish the latent representations generated by encoder from Gaussian noise . Larsen et al . ( 2015 ) employed image representation in the discriminator as the reconstruction basis of a VAE . Their method turns pixel-wise loss to feature-wise , which can capture the real distribution more simply when some form of invariance is induced . Different from VAE-GAN , Che et al . ( 2016 ) regarded the encoder as an auxiliary network , which can promote GANs to pay much attention on missing mode and derive an objective function similar to VAE-GAN . A more detailed discussion of related works can be found in Appendix C. In this paper we propose a novel technique for GANs that improve both the training stability and the quality of generated images . The core of our approach is a regularization term based on the latent representations of real images provided by an encoder network . More specifically , we apply auxiliary intervention operations that preserve the standard Gaussian ( e.g. , the noise distribution ) to these latent representations . The perturbed latent representations are then fed into the generator to produce intervened samples . We then introduce a classifier network to identify the right intervention operations that would have led to these intervened samples . The resulting negative cross-entropy loss is added as a regularizer to the objective when training the generator . We call this regularization term the intervention loss and our approach InterVention Generative Adversarial Nets ( IVGAN ) . We show that the intervention loss is equivalent with the JS-divergence among multiple intervened distributions . Furthermore , these intervened distributions interpolate between the original generative distribution of GAN and the data distribution , providing useful information for the generator that is previously unavailable in common GAN models ( see a thorough analysis on a toy example in Example 1 ) . We show empirically that our model can be trained efficiently by utilizing the parameter sharing strategy between the discriminator and the classifier . The models trained on the MNIST , CIFAR-10 , LSUN and STL-10 datasets successfully generate diverse , visually appealing objects , outperforming state-of-the-art baseline methods such as WGAN-GP and MRGAN in terms of the Frèchet Inception Distance ( FID ) ( proposed in ( Heusel et al. , 2017 ) ) . We also perform a series of experiments on the stacked MNIST dataset and the results show that our proposed method can also effectively alleviate the mode collapse problem . Moreover , an ablation study is conducted , which validates the effectiveness of the proposed intervention loss . In summary , our work offers three major contributions as follows . ( i ) We propose a novel method that can improve GAN ’ s training as well as generating performance . ( ii ) We theoretically analyze our proposed model and give insights on how it makes the gradient of generator more informative and thus stabilizes GAN ’ s training . ( iii ) We evaluate the performance of our method on both standard real-world datasets and the stacked MNIST dataset by carefully designed expriments , showing that our approach is able to stabilize GAN ’ s training and improve the quality and diversity of generated samples as well . 2 PRELIMINARIES . Generative adversarial nets The basic idea of GANs is to utilize a discriminator to continuously push a generator to map Gaussian noise to samples drawn according to an implicit data distribution . The objective function of the vanilla GAN takes the following form : min G max D { V ( D , G ) , Ex∼pdata log ( D ( x ) ) + Ez∼pz log ( 1−D ( G ( z ) ) ) } , ( 1 ) where pz is a prior distribution ( e.g. , the standard Gaussian ) . It can be easily seen that when the discriminator reaches its optimum , that is , D∗ ( x ) = pdata ( x ) pdata ( x ) +pG ( x ) , the objective is equivalent to the Jensen-Shannon ( JS ) divergence between the generated distribution pG and data distribution pdata : JS ( pG‖pdata ) , 1 2 { KL ( pG‖ pG + pdata 2 ) +KL ( pdata‖ pG + pdata 2 ) } . Minimizing this JS divergence guarantees that the generated distribution converges to the data distribution given adequate model capacity . Multi-distribution JS divergence The JS divergence between two distributions p1 and p2 can be rewritten as JS ( p1‖p2 ) = H ( p1 + p2 2 ) − 1 2 H ( p1 ) − 1 2 H ( p2 ) , where H ( p ) denotes the entropy of distribution p. We observe that the JS-divergence can be interpreted as the entropy of the mean of the two distribution minus the mean of two distributions ’ entropy . So it is immediate to generalize the JS-divergence to the setting of multiple distributions . In particular , we define the JS-divergence of p1 , p2 , . . . , pn with respect to weights π1 , π2 , . . . , πn ( ∑ πi = 1 and πi ≥ 0 ) as JSπ1 , ... , πn ( p1 , p2 , . . . , pn ) , H ( n∑ i=1 πipi ) − n∑ i=1 πiH ( pi ) . ( 2 ) The two-distribution case described above is actually a special case of the ‘ multi-JS divergence ’ , where π1 = π2 = 12 . When πi > 0 ∀i , it can be found immediately by Jensen ’ s inequality that JSπ1 , ... , πn ( p1 , p2 , . . . , pn ) = 0 if and only if p1 = p2 = · · · = pn . 3 METHODOLOGY . Training GAN has been challenging , especially when the generated distribution and the data distribution are far away from each other . In such cases , the discriminator often struggles to provide useful information for the generator , leading to instability and mode collapse problems . The key idea behind our approach is that we construct auxiliary intermediate distributions that interpolate between the generated distribution and the data distribution . To do that , we first introduce an encoder network and combine it with the generator to learn the latent representation of real images within the framework of a standard autoencoder . We then perturb these latent representations with carefully designed intervention operations before feeding them into the generator to create these auxiliary interpolating distributions . A classifier is used to distinguish the intervened samples , which leads to an intervention loss that penalizes the dissimilarity of these intervened distributions . The reconstruction loss and the intervention loss are added as regularization terms to the standard GAN loss for training . We start with an introduction of some notation and definitions . Definition 1 ( Intervention ) . Let O be a transformation on the space of d-dimension random vectors and P be a probability distribution whose support is in Rd . We call O a P-intervention if for any d-dimensional random vector X , X ∼ P⇒ O ( X ) ∼ P. Since the noise distribution in GAN models is usually taken to be standard Gaussian , we use the standard Gaussian distribution as the default choice of P and abbreviate the P-intervention as intervention , unless otherwise claimed . One of the simplest groups of interventions is block substitution . Let Z ∈ Rd be a random vector , k ∈ N and k|d . We slice Z into k blocks so that every block is in R d k . A block substitution intervention Oi is to replace the ith block of Z with Gaussian noise , i = 1 , . . . , dk . We will use block substitution interventions in the rest of the paper unless otherwise specified . Note that our theoretical analysis as well as the algorithmic framework do not depend on the specific choice of the intervention group . Notation We use E , G , D , f to represent encoder , generator , discriminator and classifier , respectively . Here and later , preal means the distribution of the real data X , and pz is the prior distribution of noise z defined on the latent space ( usually is taken to be Gaussian ) . Let Oi , i = 1 , . . . , k denote k different interventions and pi be the distribution of intervened sample Xi created from Oi ( namely Xi = G ( Oi ( E ( X ) ) ) ) . Intervention loss The intervention loss is the core of our approach . More specifically , given a latent representation z that is generated by an encoder network E , we sample an intervention Oi from a complete group S = { O1 , . . . , Ok } and obtain the corresponding intervened latent variable Oi ( z ) with label ei . These perturbed latent representations are then fed into the generator to produce intervened samples . We then introduce an auxiliary classifier network to identify which intervention operations may lead to these intervened samples . The intervention loss LIV ( G , E ) is simply the resulting negative cross-entropy loss and we add that as a regularizer to the objective function when training the generator . As we can see , the intervention loss is used to penalize the dissimilarity of the distributions of the images generated by different intervention operations . Moreover , it can be noticed that the classifier and the combination of the generator and the encoder are playing a two-player adversarial game and we will train them in an adversarial manner . In particular , we define LIV ( G , E ) = −min f Vclass , where Vclass = Ei∼U ( [ k ] ) Ex′∼pi − eTi log f ( x′ ) . ( 3 ) Theorem 1 ( Optimal Classifier ) . The optimal solution of the classifier is the conditional probability of label y given X ′ , where X ′ is the intervened sample generated by the intervention operation sampled from S. And the minimum of the cross entropy loss is equivalent with the negative of the Jensen Shannon divergence among { p1 , p2 , ... , pk } . That is , f∗i ( x ) = pi ( x ) ∑k j=1 pj ( x ) and LIV ( G , E ) = JS ( p1 , p2 , ... , pk ) + Const . ( 4 ) The proof can be found in Appendix A.1 . Clearly , the intervention loss is an approximation of the multi-JS divergence among the intervened distributions { pi : i ∈ [ k ] } . If the intervention reaches its global minimum , we have p1 = p2 = · · · = pk . And it reaches the maximum log k if and only if the supports of these k distributions do not intersect with each other . This way , the probability that the ‘ multi ’ JS-divergence has constant value is much smaller , which means the phenomenon of gradient vanishing should be rare in IVGAN . Moreover , as shown in the following example , due to these auxiliary intervened distributions , the intervention is able to provide more informative gradient for the generator that is not previously available in other GAN variants . Example 1 ( Square fitting ) . Let X0 be a random vector with distribution U ( α ) , where α = [ − 12 , 1 2 ] × [ − 1 2 , 1 2 ] . And X1 ∼ U ( β ) , where β = [ a− 12 , a+ 1 2 ] × [ 1 2 , 3 2 ] and 0 ≤ a ≤ 1 . Assuming we have a perfect discriminator ( or classifier ) , we compute the vanilla GAN loss ( i.e . the JS-divergence ) and the intervention loss between these two distributions , respectively , • JS ( X0‖X1 ) = log 2 . • In order to compute the intervention loss we need figure out two intervened samples ’ distributions evolved from U ( α ) and U ( β ) . Y1 ∼ U ( γ1 ) ; γ1 = [ − 12 , 1 2 ] × [ 1 2 , 3 2 ] and Y2 ∼ U ( γ2 ) ; γ2 = [ a − 12 , a + 1 2 ] × [ − 1 2 , 1 2 ] . Then the intervention loss is the multi JS-divergence among these four distributions : LIV = JS ( X0 ; X1 ; Y1 ; Y2 ) = − ∫ Ac 1 4 log 1 4 dµ− ∫ A 1 2 log 1 2 dµ−H ( X0 ) = log 2 2 [ µ ( Ac ) +µ ( A ) ] = log 2 2 × 2 ( 2− a ) −H ( X0 ) = − ( log 2 ) a− Const . Here A is the shaded part in Figure 2 and Ac = { α ∪ β ∪ γ1 ∪ γ2 } \A . The most important observation is that the intervention loss is a function of parameter a and the traditional GAN loss is always constant . When we replace the JS with other f -divergence , the metric between U ( α ) and U ( β ) would still remain constant . Hence in this situation , we can not get any information from the standard JS for training of the generator but the intervention loss works well . Algorithm 1 Intervention GAN Input learning rate α , regularization parameters λ and µ , dimension d of latent space , number k of blocks in which the hidden space is divided , minibatch size n , Hadamard multiplier ∗ 1 : for number of training iterations do 2 : Sample minibatch zj , j = 1 , ... , n , zj ∼ pz 3 : Sample minibatch xj , j = 1 , ... , n , xj ∼ preal 4 : for number of inner iteration do 5 : wj ← E ( xj ) , j = 1 , ... , n 6 : Sample Gaussian noise 7 : Sample ij ∈ [ k ] , j = 1 , ... , n 8 : x′j ← G ( Oij ( wj ) ) 9 : Update the parameters of D by : 10 : θD ← θD − α2n∇θDLadv ( θD ) 11 : Update the parameters of f by : 12 : θf ← θf + αn∇θf n∑ j=1 log fij ( x ′ j ) 13 : Calculate LAdv and LIV 14 : Update the parameter of G by : 15 : θG ← θG + αn∇θG { L̂Adv + λL̂recon + µL̂IV } 16 : Update the parameter of E by : 17 : θE ← θE + αn∇θE { λL̂recon + µL̂IV } Reconstruction loss In some sense we expect our encoder to be a reverse function of the generator . So it is necessary for the objective function to have a term to push the map composed of the Encoder and the Generator to have the ability to reconstruct the real samples . Not only that , we also hope that the representation can be reconstructed from samples in the pixel space . Formally , the reconstruction loss can be defined by the ` p-norm ( p ≥ 1 ) between the two samples , or in the from of the Wasserstein distance between samples if images are regarded as a histogram . Here we choose to use the ` 1-norm as the reconstruction loss : Lrecon = EX∼preal‖G ( E ( X ) ) −X‖1 +Ei∼U ( [ k ] ) Ex , z∼preal , pz‖E ( G ( Oi ( z ) ) ) −Oi ( z ) ‖1 . ( 5 ) Theorem 2 ( Inverse Distribution ) . Suppose the cumulative distribution function of Oi ( z ) is qi . For any given positive real number , there exist a δ > 0 such that if Lrecon +LIV ≤ δ , then ∀i , j ∈ [ k ] , sup r ‖qi ( r ) − qj ( r ) ‖ ≤ . The proof is in A.2 . Adversarial loss The intervention loss and reconstruction loss can be added as regularization terms to the adversarial loss in many GAN models , e.g. , the binary cross entropy loss in vanilla GAN and the least square loss in LSGAN . In the experiments , we use LSGAN ( Mao et al. , 2016 ) and DCGAN ( Radford et al. , 2015 ) as our base models , and name the resulting IVGAN models IVLSGAN and IVDCGAN respectively . Now that we have introduced the essential components in the objective of IVGAN , we can write the loss function of the entire model : Lmodel = LAdv + λLrecon + µLIV , ( 6 ) where λ and µ are the regularization coefficients for the reconstruction loss and the intervention loss respectively . We summarize the training procedure in Algorithm 1 . A diagram of the full workflow of our framework can be found in Figire 3 .

Summary: The paper proposes a method for stabilizing the training of GAN as well as overcoming the problem of mode collapse by optimizing several auxiliary models. The first step is to learn a latent space using an autoencoder. Then, this latent space is "intervened" by a predefined set of $K$ transformations to generate a set of distributions $p_k$. A classifier is then taught to distinguish between $p_k$. Eventually, the weights of the classifier are shared with those of the discriminator network to produce the desired stabilization/diversification effect. In other words, the authors propose to stabilize GANs by intervening with the discriminator. This is done by sharing its weights with a classifier that trains on a perturbed latent distribution that is somehow related to the original problem via the prior assumption imposed.