Metropolis-Hastings view on variational inference and adversarial training

1 INTRODUCTION . Bayesian framework and deep learning have become more and more interrelated during recent years . Recently Bayesian deep neural networks were used for estimating uncertainty ( Gal & Ghahramani , 2016 ) , ensembling ( Gal & Ghahramani , 2016 ) and model compression ( Molchanov et al. , 2017 ) . On the other hand , deep neural networks may be used to improve approximate inference in Bayesian models ( Kingma & Welling , 2014 ) . Learning modern Bayesian neural networks requires inference in the spaces with dimension up to several million by conditioning the weights of DNN on hundreds of thousands of objects . For such applications , one has to perform the approximate inference – predominantly by either sampling from the posterior with Markov Chain Monte Carlo ( MCMC ) methods or approximating the posterior with variational inference ( VI ) methods . MCMC methods provide the unbiased ( in the limit ) estimate but require careful hyperparameter tuning especially for big datasets and high dimensional problems . The large dataset problem has been addressed for different MCMC algorithms : stochastic gradient Langevin dynamics ( Welling & Teh , 2011 ) , stochastic gradient Hamiltonian Monte Carlo ( Chen et al. , 2014 ) , minibatch MetropolisHastings algorithms ( Korattikara et al. , 2014 ; Chen et al. , 2016 ) . One way to address the problem of high dimension is the design of a proposal distribution . For example , for the Metropolis-Hastings ( MH ) algorithm there exists a theoretical guideline for scaling the variance of a Gaussian proposal ( Roberts et al. , 1997 ; 2001 ) . More complex proposal designs include adaptive updates of the proposal distribution during iterations of the MH algorithm ( Holden et al. , 2009 ; Giordani & Kohn , 2010 ) . Another way to adapt the MH algorithm for high dimensions is combination of adaptive direction sampling and the multiple-try Metropolis algorithm as proposed in ( Liu et al. , 2000 ) . Thorough overview of different extensions of the MH algorithm is presented in ( Martino , 2018 ) . Variational inference is extremely scalable but provides a biased estimate of the target distribution . Using the doubly stochastic procedure ( Titsias & Lázaro-Gredilla , 2014 ; Hoffman et al. , 2013 ) VI can be applied to extremely large datasets and high dimensional spaces , such as a space of neural network weights ( Kingma et al. , 2015 ; Gal & Ghahramani , 2015 ; 2016 ) . The bias introduced by variational approximation can be mitigated by using flexible approximations ( Rezende & Mohamed , 2015 ) and resampling ( Grover et al. , 2018 ) . Generative Adversarial Networks ( Goodfellow et al. , 2014 ) ( GANs ) is a different approach to learn samplers . Under the framework of adversarial training different optimization problems could be solved efficiently ( Arjovsky et al. , 2017 ; Nowozin et al. , 2016 ) . The shared goal of ” learning to sample ” inspired the connection of GANs with VI ( Mescheder et al. , 2017 ) and MCMC ( Song et al. , 2017 ) . In this paper , we propose a novel perspective on learning to sample from a target distribution by optimizing parameters of either explicit or implicit probabilistic model . Our objective is inspired by the view on the acceptance rate of the Metropolis-Hastings algorithm as a quality measure of the sampler . We derive a lower bound on the acceptance rate and maximize it with respect to parameters of the sampler , treating the sampler as a proposal distribution in the Metropolis-Hastings scheme . We consider two possible forms of the target distribution : unnormalized density ( density-based setting ) and a set of samples ( sample-based setting ) . Each of these settings reveals a unifying property of the proposed perspective and the derived lower bound . In the density-based setting , the lower bound is the sum of forward and reverse KL-divergences between the true posterior and its approximation , connecting our approach to VI . In the sample-based setting , the lower bound admit a form of an adversarial game between the sampler and a discriminator , connecting our approach to GANs . The closest work to ours is of Song et al . ( 2017 ) . In contrast to their paper our approach ( 1 ) is free from hyperparameters ; ( 2 ) is able to optimize the acceptance rate directly ; ( 3 ) avoids minimax problem in the density based setting . Our main contributions are as follows : 1 . We introduce a novel perspective on learning to sample from the target distribution by treating the acceptance rate in the Metropolis-Hastings algorithm as a measure of sampler quality . 2 . We derive the lower bound on the acceptance rate allowing for doubly stochastic optimization of the proposal distribution in case when the target distribution factorizes ( i.e . over data points ) . 3 . For sample-based and density-based forms of target distribution we show the connection of the proposed algorithm to variational inference and GANs . The rest of the paper is organized as follows . In Section 2 we introduce the lower bound on the AR . Special forms of target distribution are addressed in Section 3 . We validate our approach on the problems of approximate Bayesian inference in the space of high dimensional neural network weights and generative modeling in the space of images in Section 4 . We discuss results and directions of the future work in Section 5 . 2 ACCEPTANCE RATE FOR THE METROPOLIS-HASTINGS ALGORITHM . 2.1 PRELIMINARIES . In MH algorithm we need to sample from target distribution p ( x ) while we are only able to sample from proposal distribution q ( x′ |x ) . One step of the MH algorithm can be described as follows . 1. sample proposal point x′ ∼ q ( x′ |x ) , given previously accepted point x 2. accept { x′ , if p ( x ′ ) q ( x | x′ ) p ( x ) q ( x′ | x ) > u , u ∼ Uniform [ 0 , 1 ] x , otherwise If the proposal distribution q ( x′ |x ) does not depend on x , i.e . q ( x′ |x ) = q ( x′ ) , the algorithm is called independent MH algorithm . The quality of the proposal distribution is measured by acceptance rate and mixing time . Mixing time defines the speed of convergence of the Markov chain to the stationary distribution . The acceptance rate of the MH algorithm is defined as AR = Eξ min { 1 , ξ } = ∫ dxdx′p ( x ) q ( x′ |x ) min { 1 , p ( x′ ) q ( x |x′ ) p ( x ) q ( x′ |x ) } , ( 1 ) where ξ = p ( x′ ) q ( x |x′ ) p ( x ) q ( x′ |x ) , x ∼ p ( x ) , x′ ∼ q ( x′ |x ) . ( 2 ) In case of independent proposal distribution we show that the acceptance rate defines a semimetric in distribution space between p and q ( see Appendix A.2 ) . 2.2 OPTIMIZING THE LOWER BOUND ON ACCEPTANCE RATE . Although , we can maximize the acceptance rate of the MH algorithm ( Eq . 1 ) directly w.r.t . parameters φ of the proposal distribution qφ ( x′ |x ) , we propose to maximize the lower bound on the acceptance rate . As our experiments show ( see Section 4 ) the optimization of the lower bound compares favorably to the direct optimization of the acceptance rate . To introduce this lower bound we first express the acceptance rate in terms of total variation distance . Theorem 1 For random variable ξ = p ( x ′ ) q ( x | x′ ) p ( x ) q ( x′ | x ) , x ∼ p ( x ) , x ′ ∼ q ( x′ |x ) Eξ min { 1 , ξ } = 1− 1 2 Eξ|ξ − 1| = 1− TV ( p ( x′ ) q ( x |x′ ) ∥∥∥∥p ( x ) q ( x′ |x ) ) , ( 3 ) where TV is the total variation distance . The proof of Theorem 1 can be found in Appendix A.1 . This reinterpretation in terms of total variation allows us to lower bound the acceptance rate through the Pinsker ’ s inequality AR ≥ 1− √ 1 2 · KL ( p ( x′ ) q ( x |x′ ) ∥∥∥∥p ( x ) q ( x′ |x ) ) . ( 4 ) The maximization of this lower bound can be equivalently formulated as the following optimization problem min φ KL ( p ( x′ ) qφ ( x |x′ ) ∥∥∥∥p ( x ) qφ ( x′ |x ) ) . ( 5 ) In the following sections , we show the benefits of this optimization problem in two different settings — when the target distribution is given in a form of unnormalized density and as a set of samples . In Appendix C.5 and C.1 we provide the empirical evidence that maximization of the proposed lower bound results in the maximization of the acceptance rate . 3 OPTIMIZATION OF PROPOSAL DISTRIBUTION . From now on we consider only optimization problem Eq . 5 but the proposed algorithms can be also used for the direct optimization of the acceptance rate ( Eq . 1 ) . To estimate the loss function ( Eq . 5 ) we need to evaluate the density ratio . In the density-based setting unnormalized density of the target distribution is given , so we suggest to use explicit proposal distribution to compute the density ratio explicitly . In the sample-based setting , however , we can not compute the density ratio , so we propose to approximate it via adversarial training ( Goodfellow et al. , 2014 ) . The brief summary of constraints for both settings is shown in Table 1 . The following subsections describe the algorithms in detail . 3.1 DENSITY-BASED SETTING . In the density-based setting , we assume the proposal to be an explicit probabilistic model , i.e . the model that we can sample from and evaluate its density at any point up to the normalization constant . We also assume that the proposal is reparameterisable ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ; Gal , 2016 ) . During the optimization of the acceptance rate we might face a situation when proposal collapses to the delta-function . This problem usually arises when we use Markov chain proposal , for example , qφ ( x′ |x ) = N ( x′ |x , σ ) . For such proposal we can obtain arbitrary high acceptance rate , making the σ small enough . However , this is not the case for the independent proposal distribution qφ ( x ′ |x ) = qφ ( x′ ) . In Appendix B.1 we provide more details and intuition on this property of acceptance rate maximization . We also provide empirical evidence in Section 4 that collapsing to the delta-function does not happen for the independent proposal . In this paper , we consider two types of explicit proposals : simple parametric family ( Section 4.2 ) and normalizing flows ( Rezende & Mohamed , 2015 ; Dinh et al. , 2016 ) ( Section 4.1 ) . Rich family of normalizing flows allows to learn expressive proposal and evaluate its density in any point of target distribution space . Moreover , an invertible model ( such as normalizing flow ) is a natural choice for the independent proposal due to its ergodicity . Indeed , choosing the arbitrary point in the target distribution space , we can obtain the corresponding point in the latent space using the inverse function . Since every point in the latent space has positive density , then every point in the target space also has positive density . Considering qφ ( x′ ) as the proposal , the objective of optimization problem 5 takes the form L ( p , qφ ) = KL ( p ( x′ ) qφ ( x ) ∥∥∥∥p ( x ) qφ ( x′ ) ) = E x ∼ p ( x ) x′ ∼ qφ ( x′ ) log p ( x ) qφ ( x ′ ) p ( x′ ) qφ ( x ) . ( 6 ) Explicit form of the proposal qφ ( x′ ) and the target p ( x ) distributions allows us to obtain density ratios qφ ( x ) /qφ ( x′ ) and p ( x′ ) /p ( x ) for any points x , x′ . But to estimate the loss in Eq . 6 we also need to obtain samples from the target distribution x ∼ p ( x ) during training . For this purpose , we use the current proposal qφ and run the independent MH algorithm . After obtaining samples from the target distribution it is possible to perform optimization step by taking stochastic gradients w.r.t . φ. Pseudo-code for the obtained procedure is shown in Algorithm 1 . Algorithm 1 Optimization of proposal distribution in density-based case Require : explicit probabilistic model qφ ( x′ ) Require : density of target distribution p̂ ( x ) ∝ p ( x ) while φ not converged do sample { x′k } Kk=1 ∼ qφ ( x′ ) sample { xk } Kk=1 ∼ p ( x ) using independent MH with current proposal qφ L ( p , qφ ) ' 1K ∑K k=1 log p ( xk ) qφ ( x ′ k ) p ( x′k ) qφ ( xk ) . approximate loss with finite number of samples φ← φ− α∇φL ( p , qφ ) . perform gradient descent step end while return optimal parameters φ Algorithm 1 could also be employed for the direct optimization of the acceptance rate ( Eq . 1 ) . Now we apply this algorithm for Bayesian inference problem and show that during optimization of the lower bound we can use minibatches of data , while it is not the case for direct optimization of the acceptance rate . We consider Bayesian inference problem for discriminative model on dataset D = { ( xi , yi ) } Ni=1 , where xi is the feature vector of ith object and yi is its label . For the discriminative model we know likelihood p ( yi |xi , θ ) and prior distribution p ( θ ) . In order to obtain predictions for some object xi , we need to evaluate the predictive distribution p ( yi |xi ) = Ep ( θ | D ) p ( yi |xi , θ ) . ( 7 ) To obtain samples from posterior distribution p ( θ | D ) we suggest to learn proposal distribution qφ ( θ ) and perform independent MH algorithm . Thus the objective 6 can be rewritten as L ( p ( θ | D ) , qφ ( θ ) ) = KL ( p ( θ′ | D ) qφ ( θ ) ∥∥∥∥p ( θ | D ) qφ ( θ′ ) ) ( 8 ) Note that due to the usage of independent proposal , the minimized KL-divergence splits up into the sum of two KL-divergences . KL ( p ( θ′ | D ) qφ ( θ ) ∥∥∥∥p ( θ | D ) qφ ( θ′ ) ) = KL ( qφ ( θ ) ∥∥∥∥p ( θ | D ) ) + KL ( p ( θ′ | D ) ∥∥∥∥qφ ( θ′ ) ) ( 9 ) Minimization of the first KL-divergence corresponds to the variational inference procedure . KL ( qφ ( θ ) ∥∥∥∥p ( θ | D ) ) = −Eθ∼qφ ( θ ) N∑ i=1 log p ( yi |xi , θ ) + KL ( qφ ( θ ) ‖p ( θ ) ) + log p ( D ) ( 10 ) The second KL-divergence has the only term that depends on φ . Thus we obtain the following optimization problem min φ [ − Eθ∼qφ ( θ ) N∑ i=1 log p ( yi |xi , θ ) + KL ( qφ ( θ ) ‖p ( θ ) ) − Eθ∼p ( θ | D ) log qφ ( θ ) ] . ( 11 ) The first summand here contains the sum over all objects in dataset D. We follow doubly stochastic variational inference and suggest to perform unbiased estimation of the gradient in problem 11 using only minibatches of data . Moreover , we can use recently proposed techniques ( Korattikara et al. , 2014 ; Chen et al. , 2016 ) that perform the independent MH algorithm using only minibatches of data . Combination of these two techniques allows us to use only minibatches of data during iterations of algorithm 1 . In the case of the direct optimization of the acceptance rate , straightforward usage of minibatches results in biased gradients . Indeed , for the direct optimization of the acceptance rate ( Eq . 1 ) we have the product over the all training data inside min function .

Summary: One of the main contributions of the paper is showing how maximizing the acceptance rate in Metropolis Hastings (MH) translates in minimizing the symmetric KL divergence of the target and the proposal distribution. The result aligns with the intuition that if this variance is 0, detailed balanced is satisfied and, hence, we can always accept the proposal. Also Equation 11 nicely fits the intuition that a good (independent) proposal should minimize the KL divergence between proposal and target (as in VI) under the constraint that the proposal has full support compared to the target distribution, which is enforced by the last term. Theorem 1 and its proof are great.