Learning Gibbs-regularized GANs with variational discriminator reparameterization

1 INTRODUCTION . Although recent progress in GANs and variational methods have significantly advanced the capabilities of generative models for high-dimensional data , many issues still limit the practical application of such methods . In practice , GANs typically suffer from mode loss , whereas VAEs suffer from poor sample quality compared to GANs [ 40 , 14 , 3 ] . Several factors may be identified as contributing to these issues : first , the training loss may optimize for sample quality at the expense of mode coverage or vice-versa ; second , the variance of stochastic gradient estimates may be too high to admit efficient stochastic optimization ; finally , the models may not admit a good regularization scheme via the imposition of appropriate inductive biases . We advocate a novel approach to address these concerns , with a particular focus on the latter issue of regularization . Let p : RN → R+ denote the PDF of a continuous data distribution and q : RN → R+ denote the PDF of a learned model . Following recent work [ 31 ] , we advocate training a generative model to minimize the Jeffrey ( symmetric KL ) divergence minq KL ( p , q ) + KL ( q , p ) , representing q in a way that enables it to be efficiently evaluated at any point ( i.e. , by representing q as a pushforward distribution induced by an invertible warp [ 29 , 8 , 18 ] ) . In this work , we propose the key innovation of applying Fenchel-duality-based variational inference to KL ( q , p ) , which allows the latter to be optimized without having to explicitly evaluate p. This yields the following variational approximation of the Jeffrey divergence , which constitutes the training loss for our method : min q∈Q Ex̂∼p log p ( x̂ ) q ( x̂ ) + sup ν > 0 −Ex̂∼p q ( x̂ ) ν ( x̂ ) + Ex∼q log q ( x ) ν ( x ) , ( 1 ) where the variational parameters consist of the function ν . We now assert and later elaborate the following properties of the Jeffrey divergence and its variational approximation ( 1 ) : first , KL ( p , q ) essentially prevents mode loss , whereas KL ( q , p ) prevents the generation of any samples unlikely under the data [ 6 , 14 ] ; second , an exactly unbiased stochastic gradient of KL ( p , q ) may be obtained without variational inference ; finally , the optimal value of ν is p. The second key innovation of our work is to recognize that—in contrast to traditional GANs , which admit no comparable regularization principle—this model may be effectively regularized by imposing any domain-specific structure possessed by p on ν , which we thence interpret as a structured Gibbs distribution . These concepts are illustrated in Fig . 1 . Figure 1a shows the result of training a model q to optimize KL ( p , q ) , where q is represented as the pushforward of a Gaussian base distribution under an . autoregressive warp in two dimensions [ 29 , 8 , 18 ] . We observe that the generated samples effectively cover all the data modes , since a huge penalty is incurred if q ( x ) is low for any data point x ∼ p ; however , the generator additionally places mass outside the support of the data distribution p , because shifting half of q ’ s mass onto the support of p decreases KL ( p , q ) by no more than log 2 ; therefore , this objective effectively under-constrains q . We can resolve the ambiguity by adding the variationallyapproximated KL ( q , p ) term , which trains log ν to approximate log p while penalizing the generation of samples where log ν is low , as shown in Fig . 1b-d. By choosing different forms for log ν , we see that the ambiguity can be resolved in different ways : in Fig . 1b , log ν is chosen to have contours roughly matching the support of p , whereas Fig . 1c-d show the cases where log ν is represented as quadratic and as the output of a multi-layer perceptron , respectively . We see that matching the shape of log ν to the shape of p produces qualitatively good samples , whereas putting too little constraint on the shape of log ν ( as in Fig . 1d ) yields qualitatively worse results . Estimated cross-entropy values are also shown for each model , which again demonstrates how KL ( p , q ) is largely insensitive to the overall shape of the model q , as long as q covers the modes of p. Our method may also be viewed as incorporating a kind of f-GAN [ 26 ] optimizing a particular loss ( Jeffrey divergence ) , using a ( invertible ) generator that admits exact PDF evaluation , and reparameterizing the discriminator in a certain way . The Fenchel-variational view espoused in [ 26 ] reveals the optimal discriminator T to be a function of the odds ratio : T ∗ = h ( q/p ) for some function h determined by the particular f -divergence chosen . We observe that if q can be evaluated analytically , then we might as well represent the discriminator in terms of q and some function ν , where ν directly approximates p. This change of representation allows us to take any known regularities of p and impose them on ν ; e.g. , if p is known to be translation-invariant , then we can safely impose translational invariance on ν without imposing any undue constraints on our model . Note that we can not similarly impose p ’ s structure on T , which may be a complex function even if p and q are simple : for example , even if p and q are both bounded above , T may be unbounded , since it is a function of the ratio of the two . Fig . 2 illustrates how ν is structured for our application domain of vehicle trajectory forecasting : ν is represented as a sum of learned spatial rewards , which penalizes trajectories according to a learned function over spatial positions . Intuitively , this prevents trajectories from colliding with obstacles , while simultaneously learning the concept of an obstacle . 2 PUSHFORWARD REPRESENTATION AND IMPLEMENTATION . Optimizing Eq . ( 1 ) is straightforward except for one subtle point : we must be able to evaluate q pointwise and differentiate the expression Ex∼q log ( q ( x ) /ν ( x ) ) with respect to the parameters of q . Inspired by prior work [ 29 , 8 , 18 ] , we solve both these problems by representing q as the pushforward of a simple distribution under an invertible warp ( also known as a normalizing flow ) . Suppose µ is a distribution over a set Z and g : Z → X is a function ( the generator ) with domain Z . Then we can define a measure on X as the distribution of g ( z ) sampling z from µ—this distribution , denoted here by g|µ , is referred to as the pushforward of µ under g. Now suppose g is parameterized by θ and differentiable in x and θ . By representing q as qθ = gθ|µ , for some simple distribution µ , we can move the derivative wrt . θ inside the expectation by exploiting the property Ex∼gθ|µf ( x ) = Ez∼µf ( gθ ( z ) ) for all functions f : d dθ Ex∼q log q ( x ) ν ( x ) = d dθ Ex∼qθ|µ log q ( x ) ν ( x ) = Ex∼µ d dθ log qθ ( gθ ( x ) ) ν ( gθ ( x ) ) . ( 2 ) This is well-known as the reparameterization trick ; it allows us to obtain a low-variance , unbiased estimate of the parameter derivatives for learning with SGD . However , one problem remains : evaluating qθ ( x ) = ( gθ|µ ) ( x ) , which appears in both the first and last terms of ( 1 ) . This is solved by assuming that gθ is invertible : ẑ : = g−1θ ( x̂ ) . Thus , we have an analytic formula for the pushforward density : qθ ( gθ ( z ) ) = ( gθ|µ ) ( gθ ( z ) ) = µ ( z ) ∣∣ ( dgθ ) z∣∣−1 , ( 3 ) where | ( dgθ ) z| represents the determinant of the Jacobian of gθ evaluated at the point z . This finally allows us to rewrite Eq . ( 1 ) in the following explicit form , after performing some simplifications : −max θ inf φ Ex̂∼p log µ ( ẑ ) | ( dgθ ) ẑ| + µ ( ẑ ) | ( dgθ ) ẑ|νφ ( x̂ ) + Ez∼µ log µ ( z ) | ( dgθ ) z|νφ ( gθ ( z ) ) . ( 4 ) A pseudocode summary of our method is given in Algorithm 1 . X and Z denote batches of observed and latent samples , respectively , while subscripts D and G denote either data or generated samples . A few implementation issues are noted here . Applying the method to a particular problem requires the implementation of the invertible generator G ( · ; θ ) , the structured Gibbs energy log ν ( · ; φ ) , and the base distribution µ . In order to avoid numerical issues when q is not absolutely continuous wrt . ν ( i.e. , when ν is 0 but q is not ) , we reparameterize ν as ν ← αq + ν , where α is a small number . Given this assumption , the quantity q ( x ) / ( αq ( x ) + ν ( x ) ) can be rewritten in terms of the sigmoid σ. Optimization proceeds by alternating between minimizing the loss ( 1 ) in the generator parameters θ and maximizing it in the energy parameters φ . However , as noted in Alg . 1 , we may alternatively minimize a different objective for the generator : namely , −Ex∼p log q ( x ) + Ex∼q log ( q ( x ) /ν ( x ) ) . The rationale for the alternative objective is that , as noted in Sec . 3.1 , the inner minimization may be viewed as fitting ν to p—in which case , we may approximate KL ( q , p ) as Ex∼q log ( q ( x ) /ν ( x ) ) . The alternate generator objective was used for the toy experiment in Fig . 1 , whereas the original objective was used for the trajectory forecasting experiments . 3 DERIVATIONS AND INTERPRETATIONS . Equation ( 1 ) can be derived via a variational lower bound derived from Fenchel conjugacy , using a technique similar to [ 25 , 26 ] . Our approach of pairing this convex conjugate with the pushforward Algorithm 1 Pseudocode for C3PO implementation Require : XD : a batch of training data , ZG : batch of generator noise samples from µ 1 : ZD , det dg −1 XD ← G−1 ( XD ; θ ) . G−1 ( x ) returns g−1θ ( x ) and log det . of Jacobian of g −1 θ at x 2 : XG , det dgZG ← G ( ZG ; θ ) . G ( z ) returns gθ ( z ) and log det . of Jacobian of gθ at z 3 : log q ( XD ) ← logµ ( ZD ) + log |det dg−1XD | . Generator PDF at data samples 4 : log q ( XG ) ← logµ ( ZG ) + log |det dgZG | . Generator PDF at generator samples 5 : log q/ν ( XD ) ← logα+ log q ( XD ) − log ν ( XD ; φ ) 6 : log q/ν ( XG ) ← logα+ log q ( XG ) − log ν ( XG ; φ ) 7 : L← BatchMean ( − log q ( XD ) − α−1σ ( log q/ν ( XD ) ) + log σ ( log q/ν ( XG ) ) − logα ) 8 : for i← 1 . . . N do 9 : θ ← θ − β∇θL . Alternative generator loss : L : = − log q ( XD ) + log σ ( log q/ν ( XG ) ) 10 : for j ← 1 . . .M do 11 : φ← φ+ β∇φL direct density estimation motivates our method ’ s name : Convex Conjugate Coupled Pushforward Optimization ( C3PO ) . Observe that the Jeffrey divergence can be written as minq −Ex̂∼p log q ( x̂ ) + Ex∼q log q ( x ) − Ex∼q log p ( x ) . Since q can be sampled , evaluated , and differentiated ( Sec . 2 ) , but p can only be sampled , our goal in this section is to convert −Ex∼q log p ( x ) to something expressable in terms of expectations wrt . p and q . This is achieved by applying the Fenchel-Young inequality to the function f ( p ) : = − log p , which yields − log p ≥ supλ < 0 λp − ( −1 − log ( −λ ) ) . Substituting this inequality in place of − log p ( x ) yields the following lower bound of −Ex∼q log p ( x ) : ∫ X q ( x ) ( sup λ < 0 λp ( x ) + log ( −λ ) + 1 ) dx = 1 + sup λ < 0 ∫ X q ( x ) ( λ ( x ) p ( x ) + log ( −λ ( x ) ) ) dx , ( 5 ) where the sup on the right-hand side is taken over all functions λ : X → R− mapping the domain to a negative scalar . Observing that ∫ q ( x ) λ ( x ) p ( x ) dx can be written either as Ex∼qλ ( x ) p ( x ) or Ex̂∼pλ ( x̂ ) q ( x̂ ) , and making the substitution ν = −1/λ , we have the equivalent bound −Ex∼q log p ( x ) ≥ 1 + sup ν > 0 Ex̂∼p − q ( x̂ ) ν ( x̂ ) − Ex∼q log ν ( x ) . ( 6 ) Substituting this the Jeffrey divergence yields Eq . ( 1 ) .

Summary: This paper combines a number of ideas to train generative models with (deep) structured constraints. The general idea is similar to Flow-GAN, which learns a normalizing flow-based generator by optimizing the negative loglikelihood with an augmented GAN loss. However, It’s difficult to impose prior structure information in the GAN framework. To address this problem, the authors proposed to minimize a so-called Gibbs-regularized variational bound of Jeffery divergence, which is the summation of KL and reverse KL divergence. The authors provide some justification that the Jeffery divergence works by yielding good mass-covering and mode-seeing properties.