To Relieve Your Headache of Training an MRF, Take AdVIL

1 INTRODUCTION . Markov random fields ( MRFs ) find applications in a variety of machine learning areas ( Krähenbühl & Koltun , 2011 ; Salakhutdinov & Larochelle , 2010 ; Lafferty et al. , 2001 ) . In particular , one famous example is conditional random fields ( Lafferty et al. , 2001 ) , a conditional version of MRFs that was developed to address the limitations ( e.g. , local dependency and label bias ) of directed models for sequential data ( e.g. , hidden Markov models and other discriminative Markov models based on directed graphical models ) . However , the inference and learning of general MRFs are challenging due to the presence of a global normalizing factor , i.e . partition function , especially when latent variables are present . Extensive efforts have been devoted to developing approximate methods . On one hand , sample-based methods ( Neal , 1993 ) and variational approaches ( Jordan et al. , 1999 ; Welling & Sutton , 2005 ; Salakhutdinov & Larochelle , 2010 ) are proposed to infer the latent variables . On the other hand , extensive work ( Meng & Wong , 1996 ; Neal , 2001 ; Hinton , 2002 ; Tieleman , 2008 ; Wainwright et al. , 2005 ; Wainwright & Jordan , 2006 ) has been done to estimate the partition function . Among these methods , contrastive divergence ( Hinton , 2002 ) is proven effective in certain types of models . Most of the existing methods highly depend on the model structure and require model-specific analysis in new applications , which makes it important to develop black-box inference and learning methods . Previous work ( Ranganath et al. , 2014 ; Schulman et al. , 2015 ) shows the ability to automatically infer the latent variables and obtain gradient estimate in directed models . However , there is no black-box learning method for undirected models except the recent work of NVIL ( Kuleshov & Ermon , 2017 ) . NVIL introduces a variational distribution and derives an upper bound of the partition function in a general MRF , in the same spirit as amortized inference ( Kingma & Welling , 2013 ; Rezende et al. , 2014 ; Mnih & Gregor , 2014 ) for directed models . NVIL has several advantages over existing methods , including the ability of black-box learning , tracking the partition function during training and getting approximate samples efficiently during testing . However , NVIL also comes with two disadvantages : ( 1 ) it leaves the inference problem of MRFs unsolved1 and only trains simple MRFs with tractable ∗Dept . of Comp . Sci . & Tech. , BNRist Center , Institute for AI , THBI Lab , Tsinghua University , Beijing , 100084 , China †University of Amsterdam , and the Canadian Institute for Advanced Research ( CIFAR ) . 1NVIL ( Kuleshov & Ermon , 2017 ) presents a hybrid model . The “ inference ” in the title refers to directed part but not for an MRF . posteriors , and ( 2 ) the upper bound of the partition function can be underestimated ( Kuleshov & Ermon , 2017 ) , resulting in sub-optimal solutions on high-dimensional data . We propose Adversarial Variational Inference and Learning ( AdVIL ) to relieve some headache of learning an MRF model . AdVIL is a black-box inference and learning method that partly solves the two problems of NVIL and retains the advantages of NVIL at the same time . First , AdVIL introduces a variational encoder to infer the latent variables , which provides an upper bound of the free energy . Second , AdVIL introduces a variational decoder for the MRF , which provides a lower bound of the log partition function . The two variational distributions provide an estimate of the negative log-likelihood of the MRF . On one hand , the estimate is in an intuitive form of an approximate contrastive free energy , which is expressed in terms of the expected energy and the ( conditional ) entropy of the corresponding variational distribution . On the other hand , similar to GAN ( Goodfellow et al. , 2014 ) , the estimate is a minimax optimization problem , which is solved by stochastic gradient descent ( SGD ) in an alternating manner . Theoretically , our algorithm is convergent if the variational decoder approximates the model well . This motivates us to introduce an auxiliary variable to enhance the flexibility of the variational decoder , whose entropy is approximated by the third variational trick . We evaluate AdVIL in various undirected generative models , including restricted Boltzmann machines ( RBM ) ( Ackley et al. , 1985 ) , deep Boltzmann machines ( DBM ) ( Salakhutdinov & Hinton , 2009 ) , and Gaussian restricted Boltzmann machines ( GRBM ) ( Hinton & Salakhutdinov , 2006 ) , on several real datasets . We empirically demonstrate that ( 1 ) compared to the black-box NVIL ( Kuleshov & Ermon , 2017 ) method , AdVIL provides a tighter estimate of the log partition function and achieves much better log-likelihood results ; and ( 2 ) compared to contrastive divergence based methods ( Hinton , 2002 ; Welling & Sutton , 2005 ) , AdVIL can deal with a broader family of MRFs without model-specific analysis and obtain better results when the model structure gets complex as in DBM . 2 BACKGROUND . We consider a general case where the model consists of both visible variables v and latent variables h. An MRF defines the joint distribution over v and h as P ( v , h ) = e −E ( v , h ) Z , where E denotes the associated energy function that assigns a scalar value for a given configuration of ( v , h ) and Z is the partition function such that Z = ∫ v , h e−E ( v , h ) dvdh . Let PD ( v ) denote the empirical distribution of the training data . Minimizing the negative loglikelihood ( NLL ) of an MRF is a commonly chosen learning criterion and it is given by : L ( θ ) : =− EPD ( v ) [ log ∫ h e−E ( v , h ) Z dh ] , ( 1 ) where θ denotes the trainable parameters in E . Further , the gradient of θ is : ∇θL ( θ ) = EPD ( v ) [ ∇θF ( v ) ] − EP ( v ) [ ∇θF ( v ) ] , ( 2 ) where F ( v ) = − log ∫ h e−E ( v , h ) dh denotes the free energy and the gradient in Eqn . ( 2 ) is the difference of the free energy in two phases . In the first positive phase , the expectation of the free energy under the data distribution is decreased . In the second negative phase , the expectation of the free energy under the model distribution is increased . Unfortunately , both the NLL in Eqn . ( 1 ) and its gradient in Eqn . ( 2 ) are intractable in general for two reasons . First , the integral of the latent variables in Eqn . ( 1 ) or equivalently the computation of the free energy in Eqn . ( 2 ) is intractable . Second , the computation of the partition function in Eqn . ( 1 ) or equivalently the negative phase in Eqn . ( 2 ) is intractable . Variational inference . Extensive work introduces deterministic approximations for the intractability of inference , including the mean-field approximation ( Welling & Hinton , 2002 ; Salakhutdinov & Hinton , 2009 ) , the Kikuchi and Bethe approximations ( Welling & Sutton , 2005 ) and the recognition model approach ( Salakhutdinov & Larochelle , 2010 ) . In this line of work , the intractability of the partition function is addressed using Monte Carlo based methods . Contrastive free energy . Contrastive divergence ( CD ) ( Hinton , 2002 ) addresses the intractability of the partition function by approximating the negative phase in Eqn . ( 2 ) as follows : ∇θL ( θ ) = EPD ( v ) [ ∇θF ( v ) ] − EPCD ( v ) [ ∇θF ( v ) ] , ( 3 ) where PCD ( v ) denotes the empirical distribution obtained by starting from a data point and running several steps of Gibbs sampling according to the model distribution and the free energy F ( v ) is assumed to be tractable . Existing methods ( Welling & Hinton , 2002 ; Welling & Sutton , 2005 ) approximate F ( v ) using certain function G ( v ) and the gradient of θ is : ∇θL ( θ ) ≈ EPD ( v ) [ ∇θG ( v ) ] − EPCD ( v ) [ ∇θG ( v ) ] . ( 4 ) Although these generalized methods exist , it is nontrivial to extend CD-based methods to general MRFs because the Gibbs sampling procedure is highly dependent on the model structure . Black-box learning . The recent work of NVIL ( Kuleshov & Ermon , 2017 ) addresses the intractability of the partition function in a black-box manner via a variational upper bound of the partition function : Eq ( v ) [ P̃ ( v ) 2 q ( v ) 2 ] ≥ Z2 , ( 5 ) where P̃ ( v ) = e−F ( v ) is the unnormalized marginal distribution on v and q ( v ) is a neural variational distribution . As a black-box learning method , NVIL potentially allows application to broader model families and improves the capabilities of probabilistic programming systems ( Carpenter et al. , 2017 ) . Though promising , NVIL leaves the intractability of inference in an MRF unsolved , and the bound in Eqn . ( 5 ) is of high variance and is easily underestimated ( Kuleshov & Ermon , 2017 ) . 3 METHOD . As stated above , the black-box inference and learning of MRFs are still largely open . In this paper , we make a step towards solving the problems by a new variational approach . For simplicity , we focus on the resulting objective function in this section . See Appendix A for detailed derivation . 3.1 ADVERSARIAL VARIATIONAL INFERENCE AND LEARNING . First , we rewrite the NLL of the MRF ( See an illustration in Fig . 1 ) as follows : L ( θ ) = −EPD ( v ) [ −F ( v ) ] + logZ , ( 6 ) where the negative free energy and the log partition function are in the form of a logarithm of an integral . Naturally , we can apply the variational trick ( Jordan et al. , 1999 ) twice and approximate the two terms individually . Due to the presence of the minus before the first term in Eqn . ( 6 ) , the two variational tricks bound the two parts of the NLL in the opposite directions , detailed as below . Formally , on one hand , we introduce an approximate posterior for the latent variables Q ( h|v ) , which is parameterized as a neural variational encoder ( See an illustration in Fig . 1 ) , to address the intractability of inference as follows : L ( θ ) ≤ EPD ( v ) Q ( h|v ) [ E ( v , h ) + logQ ( h|v ) ] + logZ : = L1 ( θ , φ ) , ( 7 ) where φ denotes the trainable parameters in Q ( h|v ) . The upper bound is derived via applying the Jensen inequality and the equality holds if and only if Q ( h|v ) = P ( h|v ) for all v. In the bound , the first term is the expected energy , which encourages Q ( h|v ) to infer latent variables that have low values of the energy function E ( v , h ) , or equivalently high probabilities of P ( v , h ) . The second term corresponds to the negative conditional entropy of Q ( h|v ) , which increases the uncertainty of Q ( h|v ) . In the paper , we denote the conditional entropy of Q ( h|v ) asH ( Q ) : = −EPD ( v ) Q ( h|v ) [ logQ ( h|v ) ] . On the other hand , we introduce an approximate sampler q ( v , h ) , which is parameterized by a neural variational decoder ( See Fig . 1 ) , to address the intractability of the partition function as follows : L1 ( θ , φ ) ≥EPD ( v ) Q ( h|v ) energy term︷ ︸︸ ︷E ( v , h ) + entropy term︷ ︸︸ ︷logQ ( h|v ) ︸ ︷︷ ︸ Positive Phase −Eq ( v , h ) energy term︷ ︸︸ ︷E ( v , h ) + entropy term︷ ︸︸ ︷log q ( v , h ) ︸ ︷︷ ︸ Negative Phase : = L2 ( θ , φ , ψ ) , ( 8 ) where ψ denotes the trainable parameters in q ( v , h ) . The lower bound is derived via applying the Jensen inequality as well , and the equality holds if and only if q ( v , h ) = P ( v , h ) . It can be seen that the lower bound given by q ( v , h ) consists of the entropy ( denoted asH ( q ) ) and energy terms , which is similar to the upper bound in Eqn . ( 7 ) , and the overall objective is in the form of approximate contrastive free energy ( Hinton , 2002 ; Welling & Sutton , 2005 ) . Because the double variational trick bounds the NLL in opposite directions as above , we have a minimax optimization problem : min θ min φ max ψ L2 ( θ , φ , ψ ) . ( 9 ) The minimax formulation has been investigated in GAN ( Goodfellow et al. , 2014 ) and it is interpreted as an adversarial game between two networks . We name our framework adversarial variational inference and learning ( AdVIL ) following the well-established literature . Note that L2 ( θ , φ , ψ ) is neither an upper bound , nor a lower bound of L ( θ ) due to the double variational trick . However , we argue that solving the optimization problem in Eqn . ( 9 ) is reasonable because ( 1 ) it is equivalent to optimizing L ( θ ) under the nonparametric assumption , which is similar to GAN ( Goodfellow et al. , 2014 ) ; and ( 2 ) it converges to a stationary point of L1 ( θ , φ ) , which is an upper bound of L ( θ ) , under a weaker assumption , as stated in the following theoretical analysis .

Summary: The work proposes using variational distributions to model the model the inference of latent variables and model the partition function building on NVIL, thereby providing an algorithm that would work on general MRFs for both inference and learning. Since the two terms in the NLL are opposite in sign, it is a minimax operation and GAN like adversial training can be used. The paper shows providing tighter results to estimate the log partition function and comparisons on the digits dataset and Anneal importance sampling.