Unbiased Contrastive Divergence Algorithm for Training Energy-Based Latent Variable Models

1 INTRODUCTION . Energy-based latent variable models cover a broad class of generative models that are frequently used to characterize sophisticated distributions of high-dimensional data . Popular examples of this class include the restricted Boltzmann machines ( RBM , Smolensky , 1986 ; Hinton , 2012 ) , deep belief nets ( Hinton et al. , 2006 ) , and exponential family harmoniums ( Welling et al. , 2005 ) , among many others . Energy-based models are complementary to directed generative models such as the variational autoencoders ( Kingma & Welling , 2014 ) , and have gained great success in synthesizing realistic data samples ( Xie et al. , 2016 ; 2018b ) . They can also be combined with directed models to build more sophisticated structures ( Xie et al. , 2018a ) . In this article we focus on the energy-based latent variable model , whose general form can be expressed in terms of the joint distribution of a visible random vector , v ∈ V ⊂ Rp , and a hidden or latent random vector , h ∈ H ⊂ Rr , with the density function p ( v , h ; θ ) = 1 Z ( θ ) exp { −E ( v , h ; θ ) } , ( 1 ) where θ ∈ Θ is the unknown parameter vector , E ( v , h ; θ ) is the energy function , and Z ( θ ) is a normalizing constant to ensure that p ( v , h ; θ ) is a legitimate probability density or mass function . The model distribution , pv ( v ; θ ) , is defined to be the marginal distribution of p ( v , h ; θ ) . Similar to many other machine learning models , the standard approach to estimating the parameter vector θ is the maximum likelihood method . It can be shown that the derivative of the log-likelihood function can be expressed as the difference of two expectations , and hence Monte Carlo methods , especially the Markov chain Monte Carlo ( MCMC , Gilks et al. , 1995 ) , can be used to approximate the gradient . Various optimization techniques , such as the stochastic gradient method ( SG , Robbins & Monro , 1951 ; Bottou , 2010 ) , can then proceed to iteratively update the parameter estimate . This strategy , though elegant in theory , is not without limitations . In particular , MCMC estimators are typically consistent in the limiting case , but biased on finite steps , so one needs to run MCMC for a long time to obtain an accurate gradient , which would take tremendous amount of computing time . To reduce the computational complexity , Hinton ( 2002 ) proposed a simple and fast algorithm , called the contrastive divergence ( CD ) algorithm . The basic idea of CD is to truncate MCMC at the k-th step , and use the resulting approximate gradient to update θ , where k is a fixed integer as small as one . Such an approach is usually referred to as the CD-k algorithm . The simplicity and computational efficiency of CD makes it widely used in many popular energy-based models , and there was also numerous empirical evidence to illustrate the effectiveness of CD . More recently , the CD idea is applied to directed generative models ( Ruiz & Titsias , 2019 ) , where CD is used to define a loss function that combines variational inference and MCMC . However , the success of CD also raised a lot of questions regarding its convergence properties . Both theoretical and empirical results show that CD in general does not converge to a local minimum of the likelihood function ( Carreira-Perpiñán & Hinton , 2005 ) , and diverges even in some simple models ( Schulz et al. , 2010 ; Fischer & Igel , 2010 ) . The main issue of CD is that the truncation of MCMC produces a biased stochastic gradient for the log-likelihood function in every iteration , and such uncontrolled biases may be accumulated to distort the true ascent direction . Due to this reason , the training of energy-based models has been a longstanding challenge in machine learning research . In this article , we propose a new unbiased contrastive divergence ( UCD ) algorithm based on recent advances in unbiased MCMC theory , which offers new possibilities for solving the model training problem . In the seminal work Glynn & Rhee ( 2014 ) , the authors developed an unbiased estimator for the expectation with respect to the invariant distribution of a Markov chain . More recently , this estimator was further extended to the MCMC setting by Jacob et al . ( 2017 ) , using a technique called coupling . At a high level , by carefully designing the MCMC algorithm , one is able to get an unbiased MCMC estimator with only finite number of Markov transitions . Under the framework of Glynn & Rhee ( 2014 ) and Jacob et al . ( 2017 ) , the proposed UCD algorithm is a Gibbs-sampler-based training method for energy-based latent variable models . We prove that the stochastic gradient generated by UCD is unbiased with a finite variance , which implies the convergence of SG based on it . Similar to CD-k , UCD generates Markov chains to compute the gradient , but the chain in UCD stops at a random time , instead of a fixed one as in CD-k . The theoretical analysis indicates that the stopping time has a finite expectation , so on average the computation can be completed in finite time . Besides theoretical justifications , our numerical experiments show that UCD significantly improves existing training algorithms , suggesting that it is a promising approach with a solid convergence guarantee . The implementation of the UCD algorithm is available at https : //github.com/yixuan/cdtau . The highlights of this article are as follows : • We develop a new training algorithm for general energy-based latent variable models that include many popular models ( e.g . RBM ) as special cases . To our best knowledge , this is the first algorithm that has a solid convergence guarantee for such models . • The proposed algorithm resolves a longstanding problem of the CD algorithm , the bias in approximating the gradient . In particular , our method is completely unbiased , and theoretical justifications are developed to guarantee its convergence . • We have tailored a specialized algorithm for RBM , which is shown to significantly reduce the computational cost . 2 A BRIEF REVIEW OF CONTRASTIVE DIVERGENCE . In this section we briefly review the CD algorithm , and point out some of its weaknesses that have been studied in the existing literature . For a single observation v , the marginal data log-likelihood function is ` ( θ ; v ) = log { pv ( v ; θ ) } = log { ∫ p ( v , h ; θ ) dh } . Assume that E ( v , h ; θ ) is continuously differentiable for θ , and then with n data points D = ( v1 , . . . , vn ) , the derivative of the log-likelihood function ` ( θ ; D ) = ∑ ` ( θ ; vi ) , also known as the score function , can be written as ∂ ` ( θ ; D ) ∂θ = −n [ E ( v , h ) ∼p ( D ) p ( h|v ; θ ) { ∂E ( v , h ; θ ) ∂θ } − E ( v , h ) ∼p ( v , h ; θ ) { ∂E ( v , h ; θ ) ∂θ } ] , ( 2 ) where p ( D ) stands for the empirical distribution ofD , and p ( h|v ; θ ) is the conditional distribution of the latent variable h given v = v. A simple derivation of ( 2 ) can be found in Fischer & Igel ( 2014 ) . Throughout this article we denote x = ( v , h ) ∈ X : = V×H and f ( x ; θ ) = ∂E ( v , h ; θ ) /∂θ . Then the two expectations in ( 2 ) can be abbreviated as ED { f ( x ; θ ) } and EM { f ( x ; θ ) } , respectively , whereM : = p ( v , h ; θ ) is the complete model distribution . In many cases , for example the RBM model , ED { f ( x ; θ ) } has a closed form , so the major computational difficulty comes from the EM { f ( x ; θ ) } term . A common scheme to approximate this expectation is to run a Markov chain ξ0 → ξ1 → · · · with M as the invariant distribution , and then under mild conditions we have limt→∞ E { f ( ξt ; θ ) } = EM { f ( x ; θ ) } . Of course , such a limit can not be reached in finite steps , so the CD-k algorithm truncates the Markov chain at the k-th step , resulting in the following approximation : ∆ ( θ ) : = − [ ED { f ( x ; θ ) } − f ( ξk ; θ ) ] . ( 3 ) It is easy to see that ∆ ( θ ) ≈ n−1∂ ` ( θ ; D ) /∂θ is a stochastic approximation to the true gradient , so one can use SG to update θ via the iteration θi+1 = θi + αi∆ ( θi ) , where θi is the parameter estimate in the i-th iteration , and αi is the step size . Despite its simplicity , various research articles have pointed out the weaknesses of the CD-k algorithm . For instance , Sutskever & Tieleman ( 2010 ) gave an example to show that E { ∆ ( θ ) } is not the gradient of any objective function , and Schulz et al . ( 2010 ) ; Fischer & Igel ( 2014 ) studied numerical experiments in which CD-k does not converge at all for small k values . Carreira-Perpiñán & Hinton ( 2005 ) considered the fixed points of ∆ ( θ ) , the θ values such that E { ∆ ( θ ) } = 0 , and showed that they do not match the fixed points of ∂ ` ( θ ; D ) /∂θ in general . This implies that even if CD-k converges , the resulting parameter estimate may not be a local minimum of the likelihood function . Another variant of CD is the persistent contrastive divergence ( PCD , Tieleman , 2008 ; Tieleman & Hinton , 2009 ) , which has been reported to improve CD in many numerical experiments . However , it is still an approximation method , and its convergence properties are more difficult to analyze , as the stochastic gradients generated by PCD become correlated across iterations . In fact , Schulz et al . ( 2010 ) ; Fischer & Igel ( 2010 ) also gave examples in which PCD failed to converge . There are also some other training methods as extensions to CD , such as the multi-grid method ( Gao et al. , 2018 ) and the short-run MCMC ( Nijkamp et al. , 2019 ) , but all these methods inherit the bias of CD . To summarize , it is surprising that virtually none of the popular training methods for energy-based models , including CD and PCD , provide a solid convergence guarantee . The major defects of CD stem from the fact that ∆ ( θ ) is a biased estimator for the true log-likelihood gradient , and SG may fail with uncontrolled bias accumulation . To this end , the ultimate solution is to design a training algorithm that completely removes the bias of CD .

Summary: The paper proposes an algorithmic improvement that significantly simplifies training of energy-based models, such as the Restricted Boltzmann Machine. The key issue in training such models is computing the gradient of the log partition function, which can be framed as computing the expected value of f(x) = dE(x; theta) / d theta over the model distribution p(x). The canonical algorithm for this problem is Contrastive Divergence which approximates x ~ p(x) with k steps of Gibbs sampling, resulting in biased gradients. In this paper, the authors apply the recently introduced unbiased MCMC framework of Jacob et al. to completely remove the bias. The key idea is to (1) rewrite the expectation as a limit of a telescopic sum: E f(x_0) + \sum_t E f(x_t) - E f(x_{t-1}); (2) run two coupled MCMC chains, one for the “positive” part of the telescopic sum and one for the “negative” part until they converge. After convergence, all remaining terms of the sum are zero and we can stop iterating. However, the number of time steps until convergence is now random.