Training generative latent models  by variational f-divergence minimization

1 INTRODUCTION . Probabilistic modelling generally deals with the task of trying to fit a model pθ ( x ) parameterized by θ to a given distribution p ( x ) . To fit the model we often wish to minimize some measure of difference between pθ ( x ) and p ( x ) . A popular choice is the class of f -divergences1 ( see for example Sason & Verdú ( 2015 ) ) which , for two distributions p ( x ) and q ( x ) , is defined by Df ( p ( x ) ||q ( x ) ) = ∫ q ( x ) f ( p ( x ) q ( x ) ) dx ( 1 ) where f ( x ) is a convex function with f ( 1 ) = 0 . Many of the standard divergences correspond to simple choices of the function f , see table 1 . For example , for f ( u ) = u log u we have the `` forward '' KL divergence KL ( p ( x ) ||q ( x ) ) ; setting f ( u ) = − log u gives the `` reverse '' KL divergence KL ( q ( x ) ||p ( x ) ) . The divergence Df ( p ( x ) ||q ( x ) ) is zero if and only if p ( x ) = q ( x ) . However , for a constrained model pθ ( x ) fitted to a distribution p ( x ) by minimizing Df ( pθ ( x ) ||p ( x ) ) , the resulting optimal θ can be heavily dependent on the choice of the divergence function f ( Minka , 2005 ) . Whilst there is significant recent interest in using f -divergences to train complex probabilistic models ( Nowozin et al. , 2016 ; Goodfellow et al. , 2014 ) , the f -divergence is generally computationally intractable for such complex models . The main contribution of our paper is the introduction of an upper bound on the f -divergence . We show how this bound can be readily applied to training complex latent variable generative models based on only a modest departure from the standard Variational Autoencoder ( Kingma & Welling , 2013 ) . 2 BACKGROUND . 2.1 MAXIMUM LIKELIHOOD . For data x1 , . . . , xN drawn independently and identically from some unknown distribution , fitting an approximating distribution pθ ( x ) by minimizing the forward KL between pθ ( x ) and the empirical distribution p̂ ( x ) of the data we have 1The definition extends naturally to distributions on discrete x. p̂ ( x ) ≡ 1 N N∑ n=1 δ ( x− xn ) ⇒ KL ( p̂ ( x ) ||pθ ( x ) ) = − N∑ n=1 log pθ ( xn ) + const . ( 2 ) Minimizing KL ( p̂ ( x ) ||pθ ( x ) ) w.r.t . θ is therefore equivalent to maximizing the likelihood of the data , pθ ( x ) . Given the asymptotic guarantees of the efficiency of maximum likelihood ( Wolfowitz , 1965 ) , the forward KL is the standard divergence used in statistics and machine learning . 2.2 FORWARD VERSUS REVERSE KL . It is interesting to compare models trained by the forward and reverse KL divergences . For example , when pθ is Gaussian with parameters θ = ( µ , σ2 ) , then minimizing the forward KL gives arg min µ , σ2 KL ( p ( x ) ||pθ ( x ) ) ⇒ µ = ∫ p ( x ) xdx , σ2 = ∫ p ( x ) ( x− µ ) 2dx ( 3 ) so that the optimal setting is for µ to be the mean of p ( x ) and σ2 the variance . For an `` under-powered '' model ( a model which is not rich enough to have a small divergence ) pθ ( x ) and multi-modal p ( x ) this could result in pθ ( x ) placing significant mass on low probability regions in p ( x ) . This is the so-called `` mean matching '' behavior of KL ( p ( x ) ||pθ ( x ) ) that has been suggested as a possible explanation for the poor fidelity of images generated by models pθ ( x ) trained by forward KL minimization ( Goodfellow , 2016 ) . Conversely , when using the reverse KL objective KL ( pθ ( x ) ||p ( x ) ) , for a Gaussian pθ ( x ) and multi-modal p ( x ) with well separated modes , optimally µ and σ2 fit one of the local modes . This behavior is illustrated in figure 1 and is the so-called `` mode matching '' behavior of KL ( pθ ( x ) ||p ( x ) ) . For this reason , the reverse KL objective has been suggested to be useful if high quality data samples are preferable to coverage of the dataset ( Goodfellow , 2016 ) . This highlights the potentially significant difference in the resulting model that is fitted to the data , depending on the choice of divergence ( Minka , 2005 ) . In this sense , it is of interest to explore fitting generative models pθ ( x ) to a data distribution p ( x ) using f -divergences other than the forward KL divergence ( maximum likelihood ) . 2.3 LATENT GENERATIVE MODELS . For the model pθ ( x ) to have the power to generate complex datasets , we introduce a latent variable z , i.e . pθ ( x ) =∫ pθ ( x|z ) p ( z ) dz . Following standard practice we do not place any parameters on the prior for the latent z , though this would be straightforward . A standard approach to fitting a latent generative model to data x1 , . . . , xN is maximum likelihood . N∑ n=1 log pθ ( xn ) = N∑ n=1 log ∫ pθ ( xn|zn ) p ( zn ) dzn ( 4 ) In all but simple cases the integral above over the latent z is intractable and a lower bound is used instead N∑ n=1 log pθ ( xn ) ≥ N∑ n=1 ∫ qφ ( zn|xn ) [ log pθ ( xn|zn ) p ( zn ) − log qφ ( zn|xn ) ] dzn ≡ L ( θ , φ ) ( 5 ) where the so-called variational distribution qφ ( z|x ) is chosen such that the bound ( and its gradient ) is either computationally tractable or can be readily estimated by sampling ( Kingma & Welling , 2013 ) . The parameters φ of the variational distribution qφ ( z|x ) and parameters θ of the model pθ ( x ) are jointly optimized to increase the log-likelihood lower bound L ( θ , φ ) . This lower bound on the likelihood corresponds to an upper bound on the forward divergence KL ( p̂ ( x ) ||pθ ( x ) ) , where p̂ ( x ) is the empirical data distribution . Our interest is to train latent generative models using a different divergence from the forward KL . Whilst the above upper bound ( 5 ) on the forward divergence KL ( p̂ ( x ) ||pθ ( x ) ) is well known , an upper bound on other f -divergences seems to be unfamiliar ( Sason & Verdú , 2015 ) and we are unaware of any upper bound on general f -divergences that has been used within the machine learning community . Recently a lower bound on the f -divergence was introduced in Nowozin et al . ( 2016 ) by the use of the Fenchel conjugate . The resulting training algorithm is a form of minimax in which the parameters φ that tighten the bound are adjusted so as to push up the bound towards the true divergence , whilst the model parameters θ are adjusted to lower the bound . In Nowozin et al . ( 2016 ) the authors were then able to relate the Generative Adversarial Network ( GAN ) ( Goodfellow , 2016 ) training algorithm to the Fenchel conjugate lower bound on a corresponding f -divergence , see table 1 . In contrast , if the interest is purely on minimizing an f -divergence , it is arguably preferable to have an upper bound on the divergence since then standard optimization methods can be applied , resulting in a stable optimization procedure , see figure 2 . 2.4 f -DIVERGENCE BETWEEN DISTRIBUTIONS WITH DISJOINT SUPPORTS The f -divergence between two distributions , Df ( p ( x ) ||pθ ( x ) ) , is well defined when p ( x ) and pθ ( x ) have the same support . Many datasets that we are interested in , e.g . images , are generally believed to lie on low dimension manifolds embedded in high-dimensional space ( Narayanan & Mitter ( 2010 ) ) . Moreover , if our model is a mixture of delta functions , then the support of pθ ( x ) is contained in a countable union of manifolds of dimension at most dim Z . In this case , the supports of the data distribution and the model distribution are disjoint , so the f -divergence is not formally defined , we refer readers to Arjovsky & Bottou ( 2017 ) for details . To solve the problem , Sønderby et al . ( 2017 ) proposed the instance noise trick and applied it to stabilize the training of the discriminator in GANs . Barber et al . ( 2018 ) introduced and discussed the relevant properties of a divergence which is well defined on distributions with different supports . We extend this idea to construct a surrogate of f -divergence in section 3.1 and show how to train the new divergence using the auxiliary upper bound . 3 THE f -DIVERGENCE UPPER BOUND A central contribution of our paper is the following upper bound on Df ( p ( x ) ||q ( x ) ) between any two distributions p ( x ) and q ( x ) Df ( p ( x , z ) ||q ( x , z ) ) = ∫ q ( x , z ) f ( p ( x , z ) q ( x , z ) ) dxdz ≥ Df ( p ( x ) ||q ( x ) ) ( 6 ) where p ( x , z ) is a distribution with marginal ∫ p ( x , z ) dz = p ( x ) and similarly ∫ q ( x , z ) dz = q ( x ) . The bound corresponds to a generalization of the auxiliary variational method ( Agakov & Barber , 2004 ) and follows from a straightforward application of Jensen ’ s inequality : Df ( p ( x , z ) ||q ( x , z ) ) = ∫ q ( x ) ∫ q ( z|x ) f ( p ( x , z ) q ( x , z ) ) dzdx ≥ ∫ q ( x ) f ( ∫ q ( z|x ) p ( x , z ) q ( z|x ) q ( x ) dz ) dx = ∫ q ( x ) f ( p ( x ) q ( x ) ) dx = Df ( p ( x ) ||q ( x ) ) The result states that the divergence between two joint distributions is no less than the divergence between their marginals . Additional properties of the auxiliary f -divergence are given in section A of the supplementary material . We show that the bound is tight and reduces to Df ( p ( x ) ||q ( x ) ) when performing a full unconstrained minimization of the bound with respect to p ( z|x ) ( keeping q ( x , z ) fixed ) . 3.1 SPREAD f -DIVERGENCE Recently , Barber et al . ( 2018 ) proposed the spread divergence . We give a brief introduction and show how to apply the spread divergence to a f -divergence in order to alleviate the problem of disjoint supports between the data and model distributions , as discussed in section 2.4 . For q ( x ) and p ( x ) which have disjoint supports , we define new distributions q ( y ) and p ( y ) that have the same support . We let p ( y ) = ∫ x p ( y|x ) p ( x ) q ( y ) = ∫ x p ( y|x ) q ( x ) ( 7 ) where p ( y|x ) is a “ noise '' process designed such that p ( y ) and q ( y ) have the same support . For example , if we use a Gaussian p ( y|x ) = N ( y x , σ2 ) , then p ( y ) and q ( y ) both have support R. We thus define the spread f -divergence D′f ( q ( x ) ||p ( x ) ) = Df ( q ( y ) ||p ( y ) ) ( 8 ) This satisfies the requirements of a divergence , that is D′f ( q ( x ) ||p ( x ) ) ≥ 0 and D′f ( q ( x ) ||p ( x ) ) = 0 if and only if q ( x ) = p ( x ) . The auxiliary upper bound can be easily applied to the spread f -divergence D′f ( p ( x ) ||q ( x ) ) = Df ( p ( y ) ||q ( y ) ) ≤ Df ( p ( y , z ) ||q ( y , z ) ) ( 9 )

Summary: The paper proposes a method for training generative models with general f-divergences between the model and empirical distribution (the VAE and GAN objectives are captured as specific instantiations). The trick that leads to computational tractability involves utilizing a latent variable and optimizing the f-divergence between joint distributions which is an upper bound to the (desired) f-divergence between the marginal distributions. Distribution support issues are handled by convolving the data space with a blurring function. Empirical results on image datasets demonstrate that the additional training flexibility results in qualitatively different learned models. Specifically, optimizing the reverse KL (and/or Jensen-Shannon divergence) leads to sharper images, perhaps at the loss of some variance in the data distribution.