Contrastive Divergence Learning is a Time Reversal Adversarial Game

1 INTRODUCTION . Unnormalized probability models have drawn significant attention over the years . These models arise , for example , in energy based models , where the normalization constant is intractable to compute , and are thus relevant to numerous settings . Particularly , they have been extensively used in the context of restricted Boltzmann machines ( Smolensky , 1986 ; Hinton , 2002 ) , deep belief networks ( Hinton et al. , 2006 ; Salakhutdinov & Hinton , 2009 ) , Markov random fields ( Carreira-Perpinan & Hinton , 2005 ; Hinton & Salakhutdinov , 2006 ) , and recently also with deep neural networks ( Xie et al. , 2016 ; Song & Ermon , 2019 ; Du & Mordatch , 2019 ; Grathwohl et al. , 2019 ; Nijkamp et al. , 2019 ) . Fitting an unnormalized density model to a dataset is challenging due to the missing normalization constant of the distribution . A naive approach is to employ approximate maximum likelihood estimation ( MLE ) . This approach relies on the fact that the likelihood ’ s gradient can be approximated using samples from the model , generated using Markov Chain Monte Carlo ( MCMC ) techniques . However , a good approximation requires using very long chains and is thus impractical . This difficulty motivated the development of a plethora of more practical approaches , like score matching ( Hyvärinen , 2005 ) , noise contrastive estimation ( NCE ) ( Gutmann & Hyvärinen , 2010 ) , and conditional NCE ( CNCE ) ( Ceylan & Gutmann , 2018 ) , which replace the log-likelihood loss with objectives that do not require the computation of the normalization constant or its gradient . Perhaps the most popular method for learning unnormalized models is contrastive divergence ( CD ) ( Hinton , 2002 ) . CD ’ s advantage over MLE stems from its use of short Markov chains initialized at the data samples . CD has been successfully used in a wide range of domains , including modeling images ( Hinton et al. , 2006 ) , speech ( Mohamed & Hinton , 2010 ) , documents ( Hinton & Salakhutdinov , 2009 ) , and movie ratings ( Salakhutdinov et al. , 2007 ) , and is continuing to attract significant research attention ( Liu & Wang , 2017 ; Gao et al. , 2018 ; Qiu et al. , 2019 ) . Despite CD ’ s popularity and empirical success , there still remain open questions regarding its theoretical properties . The primary source of difficulty is an unjustified approximation used to derive its objective ’ s gradient , which biases its update steps ( Carreira-Perpinan & Hinton , 2005 ; Bengio & Delalleau , 2009 ) . The difficulty is exacerbated by the fact that CD ’ s update steps can not be expressed as the gradients of any fixed objective ( Tieleman , 2007 ; Sutskever & Tieleman , 2010 ) . In this paper , we present an alternative derivation of CD , which relies on completely different principles and requires no approximations . Specifically , we show that CD ’ s update steps are the gradients of an adversarial game in which a discriminator attempts to classify whether a Markov chain generated from the model is presented to it in its original or a time-reversed order ( see Fig . 1 ) . Thus , our derivation sheds new light on CD ’ s success : Similarly to modern generative adversarial methods ( Goodfellow et al. , 2014 ) , CD ’ s discrimination task becomes more challenging as the model approaches the true distribution . This keeps the update steps effective throughout the entire training process and prevents early saturation as often happens in non-adaptive methods like NCE and CNCE . In fact , we derive CD as a natural extension of the CNCE method , replacing the fixed distribution of the contrastive examples with an adversarial adaptive distribution . CD requires that the underlying MCMC be exact , which is not the case for popular methods like Langevin dynamics . This commonly requires using Metropolis-Hastings ( MH ) rejection , which ignores some of the generated samples . Interestingly , our derivation reveals an alternative correction method for inexact chains , which does not require rejection . 2 BACKGROUND . 2.1 THE CLASSICAL DERIVATION OF CD . Assume we have an unnormalized distribution model pθ . Given a dataset of samples { xi } independently drawn from some unknown distribution p , CD attempts to determine the parameters θ with which pθ best explains the dataset . Rather than using the log-likelihood loss , CD ’ s objective involves distributions of samples along finite Markov chains initialized at { xi } . When based on chains of length k , the algorithm is usually referred to as CD-k . Concretely , let qθ ( x′|x ) denote the transition rule of a Markov chain with stationary distribution pθ , and let rmθ denote the distribution of samples after m steps of the chain . As the Markov chain is initialized from the dataset distribution and converges to pθ , we have that r0θ = p and r ∞ θ = pθ . The CD algorithm then attempts to minimize the loss ` CD-k = DKL ( r 0 θ ||r∞θ ) −DKL ( rkθ ||r∞θ ) = DKL ( p||pθ ) −DKL ( rkθ ||pθ ) , ( 1 ) where DKL is the Kullback-Leibler divergence . Under mild conditions on qθ ( Cover & Halliwell , 1994 ) this loss is guaranteed to be positive , and it vanishes when pθ = p ( in which case rkθ = pθ ) . To allow the minimization of ( 1 ) using gradient-based methods , one can write ∇θ ` CD-k =EX̃∼rkθ [ ∇θ log pθ ( X̃ ) ] − EX∼p [ ∇θ log pθ ( X ) ] + dDKL ( r k θ ||pθ ) drkθ ∇θrkθ . ( 2 ) Here , the first two terms can be approximated using two batches of samples , one drawn from p and one from rkθ . The third term is the derivative of the loss with respect only to the θ that appears in r k θ , ignoring the dependence of pθ on θ . This is the original notation from ( Hinton , 2002 ) ; an alternative way to write this term would be ∇θ̃DKL ( rkθ̃ ||pθ ) . This term turns out to be intractable and in the original derivation , it is argued to be small and thus neglected , leading to the approximation ∇θ ` CD-k ≈ 1 n ∑ i ( ∇θ log pθ ( x̃i ) −∇θ log pθ ( xi ) ) ( 3 ) Here { xi } is a batch of n samples from the dataset and { x̃i } are n samples generated by applying k MCMC steps to each of the samples in that batch . The intuition behind the resulting algorithm ( summarized in App . A ) is therefore simple . In each gradient step θ ← θ− η∇θ ` CD-k , the log-likelihood of samples from the dataset is increased on the expense of the log-likelihood of the contrastive samples { x̃i } , which are closer to the current learned distribution pθ . Despite the simple intuition , it has been shown that without the third term , CD ’ s update rule ( 2 ) generally can not be the gradient of any fixed objective ( Tieleman , 2007 ; Sutskever & Tieleman , 2010 ) except for some very specific cases . For example , Hyvärinen ( 2007 ) has shown that when the Markov chain is based on Langevin dynamics with a step size that approaches zero , the update rule of CD-1 coincides with that of score-matching Hyvärinen ( 2005 ) . Similarly , the probability flow method of Sohl-Dickstein et al . ( 2011 ) has been shown to be equivalent to CD with a very unique Markov chain . Here , we show that regardless of the selection of the Markov chain , the update rule is in fact the exact gradient of a particular adversarial objective , which adapts to the current learned model in each step . 2.2 CONDITIONAL NOISE CONTRASTIVE ESTIMATION . Our derivation views CD as an extension of the CNCE method , which itself is an extension of NCE . We therefore start by briefly reviewing those two methods . In NCE , the unsupervised density learning problem is transformed into a supervised one . This is done by training a discriminator Dθ ( x ) to distinguish between samples drawn from p and samples drawn from some preselected contrastive distribution pref . Specifically , let the random variable Y denote the label of the class from which the variable X has been drawn , so that X| ( Y = 1 ) ∼ p and X| ( Y = 0 ) ∼ pref . Then it is well known that the discriminator minimizing the binary cross-entropy ( BCE ) loss is given by Dopt ( x ) = P ( Y = 1|X = x ) = p ( x ) p ( x ) + pref ( x ) . ( 4 ) Therefore , letting our parametric discriminator have the form Dθ ( x ) = pθ ( x ) pθ ( x ) + pref ( x ) , ( 5 ) and training it with the BCE loss , should in theory lead to Dθ ( x ) = Dopt ( x ) and thus to pθ ( x ) = p ( x ) . In practice , however , the convergence of NCE highly depends on the selection of pref . If it significantly deviates from p , then the two distributions can be easily discriminated even when the learned distribution pθ is still very far from p. At this point , the optimization essentially stops updating the model , which can result in a very inaccurate estimate for p. In the next section we provide a precise mathematical explanation for this behavior . The CNCE method attempts to alleviate this problem by drawing the contrastive samples based on the dataset samples . Specifically , each dataset sample x is paired with a contrastive sample x̃ that is drawn conditioned on x from some predetermined conditional distribution q ( x̃|x ) ( e.g . N ( x , σ2I ) ) . The pair is then concatenated in a random order , and a discriminator is trained to predict the correct order . This is illustrated in Fig . 2a . Specifically , here the two classes are of pairs ( A , B ) corresponding to ( A , B ) = ( X , X̃ ) for Y = 1 , and ( A , B ) = ( X̃ , X ) for Y = 0 , and the discriminator minimizing the BCE loss is given by Dopt ( a , b ) = P ( Y = 1|A = a , B = b ) = q ( b|a ) p ( a ) q ( b|a ) p ( a ) + q ( a|b ) p ( b ) . ( 6 ) Therefore , constructing a parametric discriminator of the form Dθ ( a , b ) = q ( b|a ) pθ ( a ) q ( b|a ) pθ ( a ) + q ( a|b ) pθ ( b ) = ( 1 + q ( a|b ) pθ ( b ) q ( b|a ) pθ ( a ) ) −1 , ( 7 ) and training it with the BCE loss , should lead to pθ ∝ p. Note that here Dθ is indifferent to a scaling of pθ , which is thus determined only up to an arbitrary multiplicative constant . CNCE improves upon NCE , as it allows working with contrastive samples whose distribution is closer to p. However , it does not completely eliminate the problem , especially when p exhibits different scales of variation in different directions . This is the case , for example , with natural images , which are known to lie close to a low-dimensional manifold . Indeed if the conditional distribution q ( ·|· ) is chosen to have a small variance , then CNCE fails to capture the global structure of p. And if q ( ·|· ) is taken to have a large variance , then CNCE fails to capture the intricate features of p ( see Fig . 3 ) . The latter case can be easily understood in the context of images ( see Fig . 2a ) . Here , the discriminator can easily distinguish which of its pair of input images is the noisy one , without having learned an accurate model for the distribution of natural images ( e.g. , simply by comparing their smoothness ) . When this point is reached , the optimization essentially stops . In the next section we show that CD is in fact an adaptive version of CNCE , in which the contrastive distribution is constantly updated in order to keep the discrimination task hard . This explains why CD is less prone to early saturation than NCE and CNCE .

Summary: This paper presents a way to view contrastive divergence (CD) learning as an adversarial learning procedure where a discriminator is tasked with classifying whether or not a Markov chain, generated from the model, has been time-reversed. Beginning with the classic derivation of CD and its approximate gradient, noting relevant issues regarding this approximation, the authors present a way to view CD as an extension of the conditional noise contrastive estimation (CNCE) method where the contrastive distribution is continually updated to keep the discrimination task difficult. Specifically, when the contrastive distribution is chosen such that the detailed balance property is satisfied, then the CNCE loss becomes exactly proportional the CD-1 update with the derivation further extended to CD-k. Practical concerns regarding lack of detailed balance are mitigated through the use of Metropolis-Hastings rejection or an adaptive weighting that arises when deriving the gradient of their time-reversal classification loss.  A toy example providing empirical support for correcting the lack of detailed balance is included.