Contrastive Divergence Learning is a Time Reversal Adversarial Game
Authors: Omer Yair
Tomer Michaeli
Venue: ICLR 2021 Spotlight (2021)
Keywords: Unsupervised learning
energy based model
adversarial learning
contrastive divergence
noise contrastive estimation

Abstract: Contrastive divergence (CD) learning is a classical method for fitting unnormalized statistical models to data samples. Despite its wide-spread use, the convergence properties of this algorithm are still not well understood. The main source of difficulty is an unjustified approximation which has been used to derive the gradient of the loss. In this paper, we present an alternative derivation of CD that does not require any approximation and sheds new light on the objective that is actually being optimized by the algorithm. Specifically, we show that CD is an adversarial learning procedure, where a discriminator attempts to classify whether a Markov chain generated from the model has been time-reversed. Thus, although predating generative adversarial networks (GANs) by more than a decade, CD is, in fact, closely related to these techniques. Our derivation settles well with previous observations, which have concluded that CD's update steps cannot be expressed as the gradients of any fixed objective function. In addition, as a byproduct, our derivation reveals a simple correction that can be used as an alternative to Metropolis-Hastings rejection, which is required when the underlying Markov chain is inexact (e.g., when using Langevin dynamics with a large step).

Published as a conference paper at ICLR 2021
CONTRASTIVE DIVERGENCE LEARNING IS A TIME REVERSAL ADVERSARIAL GAME
Yair Omer
Tomer Michaeli
Department of Electrical Engineering Technion Israel Institute of Technology Haifa, Israel
Department of Electrical Engineering
Technion Israel Institute of Technology Haifa, Israel t tomer m@ee. www.nien....c......
omeryai & gmail com
ABSTRACT
Contrastive divergence (CD) learning is a classical method for fitting unnormal- ized statistical models to data samples. Despite its wide-spread use, the conver- gence properties of this algorithm are still not well understood. The main source of difficulty is unjustified approximation which has been used to derive the gradient of the loss. In this paper, we present an alternative derivation CD that does not require any approximation and sheds new light the objective that is actually being optimized by the algorithm. Specifically, we show that CD is an adversarial learning procedure, where discriminator attempts to classify whether a Markov chain generated from the model has been time-reversed. Thus. although predating generative adversarial networks (GANs) by more than decade, CD is. in fact, closely related to these techniques. Our derivation settles well with pre- vious observations, which have concluded that CD's update steps cannot be ex- pressed as the gradients of any fixed objective function. In addition, as a byprod- uct, our derivation reveals a simple correction that can be used as an alternative to Metropolis-Hasting: rejection, which is required when the underlying Markov chain is inexact (e.g., when using Langevin dynamics with a large step).
INTRODUCTION
Unnormalized probability models have drawn significant attention over the years. These models arise, for example, in energy based models, where the normalization constant is intractable to com- pute, and are thus relevant to numerous settings. Particularly, they have been extensively used the context of restricted Boltzmann machines (Smolensky 1986; Hinton 2002), deep belief networks
(Hinton et al. 2006; Salakhutdinoy & Hinton 2009). Markov random fields (Carreira-Perpine &
Hinton 2005
Hinton & Salakhutdinov/ 2006)) and recently also with deep neural networks (Xie
et al. 2016 Song & Ermon 2019 Du & Mordatch 2019 Grathwohl et al. 2019: Nijkamp et al.
2019
Fitting an unnormalized density model to a dataset is challenging due to the missing normalization constant of the distribution. naive approach is to employ approximate maximum likelihood esti- mation (MLE). This approach relies on the fact that the likelihood's gradient can be approximated using samples from the model, generated using Markov Chain Monte Carlo (MCMC) techniques. However, good approximation requires using very long chains and is thus impractical. This dif- ficulty motivated the development of a plethora of more practical approaches, like score matching (Hyvärinen) 2005). noise contrastive estimation (NCE) (Gutmann & Hyvärinen] 2010), and condi- tional NCE (CNCE) (Ceylan & Gutmann] 2018), which replace the log-likelihood loss with objec- tives that do not require the computation of the normalization constant or its gradient.
Perhaps the most popular method for learning unnormalized models is contrastive divergence (CD) (Hinton 2002). CD's advantage over MLE stems from its use of short Markov chains initialized at the data samples. CD has been successfully used in a wide range of domains, including modeling images (Hinton al. 2006). speech (Mohamed & Hinton 2010), documents (Hinton & Salakhut- dinov] (2009), and movie ratings (Salakhutdinov et al. (007), and is continuing to attract significant
research attention (Liu & Wang 2017 Gao et al. 2018 Qiu et al. 2019).

---

Published as a conference paper at ICLR 2021
Learned model
MCMC (genarator)
Original order
Dalasel sample
Discriminator Dj
Reversed
i) Generator update step
ii) Discriminator update step
Contrastive divergence as an adversarial process. In the first step, the distribution model is used generate MCMC process which is used generate chain of samples. In the second step the distribution model is updated using a gradient descent step, using the MCMC transition rule.
Figure 1:
Despite CD's popularity and empirical success, there still remain open questions regarding its theo- retical properties. The primary source of difficulty is an unjustified approximation used to derive its objective's gradient, which biases its update steps (Carreira-Perpinar & Hinton (0055 [Bnngi & De-
lalleau 2009). The difficulty is exacerbated by the fact that CD's 's update steps cannot be expressed the gradients of any fixed objective (Tieleman) 2007 Sutskever Tieleman 2010).
In this paper, we present an alternative derivation of CD, which relies on completely different princi- ples and requires no approximations. Specificallyy we show that CD's update steps are the gradients of an adversarial game in which a discriminator attempts to classify whether a Markov chain gen- erated from the model presented to in its original or a time-reversed order (see Fig. D. Thus, our derivation sheds new light on CD's success: Similarly to modern generative adversarial meth- al., [014), CD's discrimination task becomes more challenging as the model
ods (Goodfellow approaches the true distribution. This keeps the update steps effective throughout the entire train- ing process and prevents early saturation as often happens non-adaptive methods like NCE and CNCE. In fact, derive CD natural extension of the CNCE method, replacing the fixed distri- bution of the contrastive examples with an adversarial adaptive distribution.
CD requires that the underlying MCMC be exact, which is not the case for popular methods like (MH) rejection, which ignores some of the generated samples. Interestingly, derivation reveals an alternative correction method for inexact chains, which does not require rejection.
Langevin dynamics. This commonly requires using Metropolis-
BACKGROUND
2.1
THE CLASSICAL DERIVATION OF CD
Assume we have unnormalized distribution model pa. Given dataset of samples {2;} inde- pendently drawn from some unknown distribution p. CD attempts to determine the parameters with which po best explains the dataset. Rather than using the log-likelihood loss, CD's objective involves distributions of samples along finite Markov chains initialized (æ;)- When based on chains of length k, the algorithm is usually referred as CD-A.
Concretely, let qo (x'(x) denote the transition rule of a Markov chain with stationary distribution pa. and let ri denote the distribution of samples after steps of the chain. As the Markov chain is initialized from the dataset distribution and converges to pa. we have that r = p and r = Pa- The
CD algorithm then attempts minimize the loss
lCD. = DKI. (ru((ellrgg)) - DKI.(Fillrz))
DKL. (Pk((p(pp)) DEL (po))
where DRL. is the Kullback- Leibler divergence. Under mild conditions on qo (Cover & Halliwell 1994] this loss is guaranteed to be positive, and it vanishes when pa (in which case r = Po).
2

---

Published as a conference paper at ICLR 2021
To allow the minimization of ①
using gradient-base based methods, can write
VolCD.E =B =Bx (o log 10--Ex--- Vooo-p/Voooooo/53 + dDLL dr
V
(2)
Here, the first two terms can be approximated using two batches of samples, one drawn from p and one from rÿ. The third term is the derivative of the loss with respect only to the that appears in rz. ignoring the dependence of po O. This is the original notation from (Hinton 2002); alternative way to write this term would be V2DKL (rgllp)). This term turns out to be intractable and in the original derivation, it is argued to be small and thus neglected, leading to the approximation
VolcD. ~
n2 E
(Vo log Vo log
(3)
Here {a:} is batch of samples from the dataset and (#:
samples generated by applying
MCMC steps to each of the samples in that batch. The intuition behind the resulting algorithm (sum- marized in App. A is therefore simple. In each gradient step 0 Ve the log- log-likelihood of samples from the dataset is increased on the expense of the log-likelihodd of the contrastive samples } which are closer the current learned distribution po.
Despite the simple intuition, it has been shown that without the third term, CD's update rule (2) generally cannot be the gradient of any fixed objective (Tieleman 2007; Sutskever & Tieleman, 2010) except for some very specific cases. For example, [Hyvärinen (207)) has shown that when the Markov chain is based on Langevin dynamics with step size that approaches zero, the update rule of CD-1 coincides with that of sctre-matching Hyvärinen 2005). Similarly, the probability flow method of Sohl-Dicsste et al. (201I) has been shown to be equivalent to CD with a very unique Markov chain. Here, we show that regardless of the selection of the Markov chain, the update rule in fact the exact gradient a particular adversarial objective, which adapts to the current learned model in each step.
2.2 CONDITIONAL NOISE CONTRASTIVE ESTIMATION
Our derivation views CD as an extension of the CNCE method, which itself is an extension of NCE.
We therefore start by briefly reviewing those two methods.
In NCE, the unsupervised density learning problem is transformed into a supervised one. This is done by training discriminator Dy(x) to distinguish between samples drawn from p and samples drawn from some preselected contrastive distribution Pret- Specifically, let the random variable Y denote the label of the class from which the variable X has been drawn, that X 1) p and X|(Y = 0) ~ Pref- Then it is well known that the discriminator minimizing the binary cross-entropy (BCE) loss given by
p(z) p(x) + (x)
Dop (x) = P(Y = 1|X = x)
(4)
Therefore, letting our parametric discriminator have the form
Da(x) (o(x) po(x) Pre
(5)
and training it with the BCE loss, should in theory lead Do(x) =ppx( p(z). In practice, however, the convergence of NCE highly depends on the selection of pre-- If significantly deviates from p. then the two distributions can easily discriminated even when the learned distribution P4 is still very far from p. At this point, the optimization essentially stops updating the model, which can result in a very inaccurate estimate for p. In the next section we provide precise mathematical explanation for this behavior.
Dxp and thus to (o
The CNCE method attempts to alleviate this problem by drawing the contrastive samples based on the dataset samples. Specifically, each dataset sample x: is paired with contrastive sample ă that is drawn conditioned on z from some predetermined conditional distribution (()22) (e.g. (x I)).. The pair is then concatenated in random order, and diseriminator is trained to predict the correct
3

---

Published as a conference paper at ICLR 2021
P
q' fixed
P
go Based on
(a,b)
(a,
Do(z)
Dz(a)
(a) CNCE
(b) CD-1
Figure 2: From CNCE CD-1. (a) In CNCE, each contrastive sample is generated using fixed conditional distribution q(-l-) (which usually corresponds to additive noise). The real and fake sam- ples are then concatenated and presented to a discriminator in a random order, which is trained to predict the correct order. (b) CD-1 can be viewed as CNCE with g(-l-) that corresponds to the transition rule of a Markov chain with stationary distribution pg. Since q depends on pe (hence the subscript 0), during training the distribution of contrastive samples becomes more similar to that of the real samples, making the discrimination task harder.
Specifically, here the two classes are of pairs (A, B) corre- sponding to (A, B) = (X, X) for Y = 1, and (A, B) = XX for Y = 0, and the discriminator minimizing the BCE loss is given by
order. This is illustrated in Fig.
2a
{(bla)((a) Dape(a, b) = PPY = -IAA a, B ((bla)p(a) + (abl))()))
(6)
Therefore, constructing parametric discriminator of the form
(bbla)po(a) D) ((bla)po(a) ((a
(1 g(alb)po(b) (bbl))oo(a)
and training with the BCE loss, should lead Pe x p. Note that here Da is indifferent to a scaling of po, which is thus determined only up an arbitrary multiplicative constant.
CNCE improves upon NCE, it allows working with contrastive samples whose distribution closer to p. However, it does not completely eliminate the problem, especially when p exhibits different scales of variation in different directions. This the case, for example, with natural images, which are known to lie close to a low-dimensional manifold. Indeed if the conditional distribution q(-l-) is chosen to have a small variance, then CNCE fails to capture the global structure of p. And if q(-l.) is taken to have large variance, then CNCE fails to capture the intricate features of p (see Fig. 3). The latter case can be easily understood in the context of images (see Fig. 2a). Here, the discriminator having learned an accurate model for the distribution natural images (e.g., simply by comparing their smoothness). When this point reached, the optimization essentially stops.
easily distinguish which of its pair of input images is the noisy one, without
In the next section we show that CD is in fact an adaptive version of CNCE, in which the contrastive distribution is constantly updated in order to keep the discrimination task hard. This explains why CD is less prone to early saturation than NCE and CNCE.
AN ALTERNATIVE DERIVATION OF CD
We now present our alternative derivation of CD. In Sec. 3.1 we identify a decomposition of the CNCE loss, which reveals the term that is responsible for early saturation. In Sec. 3.2 we then present a method for adapting the contrastive distribution in a way that provably keeps this term bounded away from zero. Surprisingly, the resulting update step turns out to precisely match that of CD-1, thus providing a new perspective on CD learning. In Sec. 3.3 we extend our derivation to include CD-k (with k > 1).

---

Published as a conference paper at ICLR 2021
First iteration
Last iteration
Weights (Ap)
Learned model
10
3D view (Xy-
CNCE large
10
The X plane
10r4
ONIE smar
16m
D-13
The Xy plane
50002
10
10a
0
L
50000
GT dersit Samples
Samples
Contrastvve samples
from model
(a) The toy model
(b) Comparing CNCE with CD
Figure 3: A toy example illustrating the importance of the adversarial nature of CD. Here, the data lies close to 2D spiral embedded in 10-dimensional space. (a) The training samples in the first dimensions. (b) Three different approaches for learning the distribution: CNCE with large contrastive variance (top), CNCE with small contrastive variance (middle), and CD based on Langevin dynamics MCMC with the weight adjustment described in Sec. 3.4 (bottom). As can be seen in the first columns, CD adapts the contrastive samples according the data distribution, whereas CNCE does not. Therefore, CNCE with large variance fails to learn the distribution because the vast majority of its contrastive samples are far from the manifold and quickly become irrelevant (as indicated by the weights ag in the third column). And CNCE with small variance fails to learn the global structure the distribution because its contrastive samples are extremely close to the dataset samples. CD, on the other hand, adjusts the contrastive distribution during training, so to generate samples that are close to manifold yet traverse large distances along it.
3. REINTERPRETING CNCE
Let us denote
wa(a, b) e ((alb)po(b) q(b ( (a
(8)
so that we can write CNCE's discriminator 7 as
Do(a, b) (1 + we wa b))
Then we have the following observation (see proof in App. B Observation 1. The gradient of the CNCE loss be expressed
[oo(X (Vo loggo((( - Vologro(X))]
VoleNCE
Xmp 8|X~g
(10)
where
a0(2, F) A (1 wo (x. () -11--
(11)

---

Published as a conference paper at ICLR 2021
Note that (T0) is similar in nature to the (approximate) gradient of the CD loss (3) Particularly, as in CD, the term Ve log (X) Ve log (>() causes each gradient step to increase the logllikelihood of samples from the dataset on the expense of the log- logliikei
the contrastive samples. How-
ever, opposed to CD, here we also have the coefficient ag (2:, .), which assigns weight between O and 1 to each pair of samples (I, .). To understand its effect, observe that
orp (1, .) = Do(a, 3) Do .))
(12)
Namely, this coefficient is precisely the probability that the discriminator assigns the incorrect order of the pair. Therefore, this term gives low weight to "easy" pairs (i.e., for which (z æ)) is close to 1) and a high weight to "hard" ones.
This weighting coefficient is of course essential for ensuring convergence to For example, it prevents log py from diverging =00 when the discriminator presented with the same samples over and over again. The problem is that discriminator can often correctly discriminate all training pairs, even with Po that is still far from p. In such cases, ag becomes practically zero for all pairs and the model stops updating. This shows that good contrastive distribution is one which keeps the discrimination task hard throughout the training. As we show next, there a particular choice which provably prevents og from converging zero, and that choice results in the CD method.
3.2 FROM CNCE TO CD-1
To bound 0g away from 0, and thus avoid the early stopping of the training process, we now extend the original CNCE algorithm by allowing the conditional distribution q to depend on pe (and thus to change from one step to the next). Our next key observation is that in this setting there exists a particular choice that keeps ag constant.
Observation 2. If qqq is chosen to be the transition probability of a reversible Markov chain with stationary distribution pg. then
1
c0(2, 2) 2'
Vx,
(13)
Proof. A reversible chain with transition q and stationary distribution ps, satisfies the detailed bal- ance property
Vz, R.
(14)
Substituting
14 into 08 leads to wo (ze, .) 1, which from 11 implies oo (z, T)
D
This observation directly links CNCE to CD. First, the suggested method for generating the con- trastive samples is precisely the one used in CD-1. Second, this choice of q leads to org (2., ē) 1.
it causes the gradient of the CNCE loss (10) to become
V VoLCNCE = 1 X~p
Ve log Ve log po
(15)
X|X~a
We have thus obtained an alternative derivation of CD-1. Namely, rather than viewing CD-1 learning as an approximule gradient descent process for the loss I we can view each step as the exact gradient the CNCE discrimination loss, where the reference distribution is adapted to the current learned model pa- This is illustrated in Fig. 2b
which is exactly proportional the CD-1 update 3..
Since q is chosen based on Pa, the overall process is in fact an adversarial game. Namely, the opti- mization alternates between updating g, which acts as generator, and updating ps. which defines the discriminator. As pe approaches p. the distribution of samples generated from the MCMC also becomes closer to p. which makes the discriminator's 's task harder and thus prevents early saturation.
should be noted that formally, since depends on pe. it also indirectly depends on 0, so that a more appropriate notation would be qa. However, during the update of pe we fix qa (and vise versa), so that the gradient in the discriminator update does not consider the dependence of go on O. This is why 15) does not involve the gradient of X which depends on qo-
The reason for fixing qe comes from the adversarial nature of the learning process. Being part of the chain generation process, the goal of the transition rule qo is to generate chains that appear to be time- reversible, while the goal of the classifier, which is based on the model po, is to correctly classify

---

Published as a conference paper at ICLR 2021
whether the chains were reversed. Therefore, we do not want the optimization of the classifier to affect qo. This is just like in GANs, where the generator and discriminator have different objectives, and when updating the discriminator the generator is kept fixed.
3.3 FROM CD-1 CD-k
To extend our derivation to CD-k with an arbitrary k 2 1, let us now view the discrimination problem of the previous section as special case of more general setting. Specifically, the pairs of samples presented to the discriminator in Sec. 3.2] can be viewed as Markov chains of length two (comprising the initial sample from the dataset and one extra generated sample). It is therefore natural to consider also Markov chains of arbitrary lengths. That is, assume we initialize the MCMC at sample 2; from the dataset and run it for lt steps to obtain a sequence (a:(0) ,,(1), ... (*)),,
where x (0) = We can then present this sequence to a discriminator either in its original order, or time-reversed, and train the discriminator to classify the correct order. We coin this a reversal classification task. Interestingly, in this setting, we have the following.
Observation 3. When using a reversible Markov chain of length k + 1 with stationary distribution po, the gradient of the BCE loss of the time-reversal classification task is given by
VoleNCE 1 Vo log po (x)))) Vo Tologpoo (ogooo((((0)))
(16)
which is exactly identical to the CD-k update
up multiplicative factor of 3
This constitutes an alternative interpretation of CD-k. That is, CD-k can be viewed as a time-reversal adversarial game, where each step, the model pv is updated so as allow the discriminator to better distinguish MCMC chains from their time-reversed counterparts.
Two remarks are in order. First, it is interesting to note that although the discriminator's task is to classify the order of the whole chain, its optimal strategy is to examine only the endpoints of the chain, x (0) and (*(), Second, it is insightful to recall that the original motivation behind the CD-k loss ( was that when pe equals p, the marginal probability of each individual step in the chain is also Our derivation, however, requires more than that. To make the chain indistinguishable from its time-reversed version, the joint probability of all samples the chain must be invariant flip of the order. When ps = p, this is indeed the case, due to the detailed balance property [14)
Proof of Observation 3 We provide the outline of the proof (see full derivation in App. C Let (A A1)) with Y = corresponding to (A(0), (1)) ....,.(k)) A(0)) = (X11), x(1), ..., x(() and Y = to
(k) denote the input to the discriminator and let indicate the order of the chain,
(A(O), A(1)... ...,,... (0)) (X((), (( x()). The discriminator that minimizes the BCE
loss is now given by
D(ao, 01 .......,... ox) P(Y = 11 an. (1)) a1.. AA = ax)
1 + a(
((aola)) (0olla)) ... 1(ax=1lax)xe) )((a)) -1
g(ax|ax>) 1) ((a1la))p(10
wo a:)
(17)
The CNCE paradigm thus defines discriminator Do having the form of 17] but with p replaced by pa. Recall that despite the dependence of the transition probability q on the current learned model pe. it is regarded as fixed within each discriminator update step. We therefore omit the subscript 0 from y here. Similarly to the derivation of (10). explicitly writing the gradient of the BCE loss of our discriminatrion task, gives
wa (X(i-1)) x(()))) [
log Vo log (ogp)((r1o))
Volehain = E
1
E (x(() ...... x())
vo loggo)
(18)
7

---

Published as a conference paper at ICLR 2021
CD+MH
ACD
dansity
MCMC samples
Figure 4: Here, we use different CD configurations for learning the model of Fig. All configu- rations use Langevin dynamics as their MCMC process, but with different ways of compensating for the lack of detailed balance. From left to right we have the ground truth density, CD w/o any correction, CD with Metropolis Hetropolis-Hastinnss rejection, and CD with our proposed adjustment.
where we now defined
wo (a; (a;
Ord
(19)
Note that 10) is special case of (18) corresponding to k 1, where X and X in 10) are ( and
(1) in (8). As before, when satisfies the detailed balance property (14) we obtain Wi = 1 and
consequently the weighting term ag again equals 5. Thus, the gradient (18 reduces (6 which
is exactly proportional to the CD-k update
3.4 MCMC PROCESSES THAT DO NOT HAVE DETAILED BALANCE
In our derivation, we assumed that the MCMC process reversible, and thus exactly satisfies the detailed balance property (14). This assumption ensured that wy = and thus ag = 2.2 In practice, however, commonly used MCMC methods satisfy this property only approximately. For example, the popular discrete Langevin dynamics process obeys detailed balance only in the limit where the step size approaches zero. The common approach to overcome this is through Metropolis-Hasting (MH) rejection (Hastings 1970), which guarantees detailed balance by accepting only a portion of the proposed MCMC transitions. In this approach, the probability of accepting transition from closely related to the weighing term Wa, and is given by
A(x, 2) min (1, (x,
(20)
Interestingly, our derivation reveals an alternative method for accounting for lack of detailed balance.
Concretely, we saw that the general expression for the gradient of the BCE loss (before assuming detailed balance) is given by 18). This expression differs from the original update step of CD-k only in the weighting term (2 (0) ................................ (+)). Therefore, all that is required for maintaining correctness x (k) (see Alg. in App. [p). Note that in this case, the update depends not only on the end-points of the chains, but rather also on their intermediate steps. As can seen in Fig. this method performs just as well as MH. and significantly better than vanilla CD without correction.
in the absence of detailed balance, is to weigh each chain by its "hardness" (p))(())....
4 ILLUSTRATION THROUGH A TOY EXAMPLE
To illustrate our observations, we now conclude with simple toy example (see Fig. Our goal here is not to draw general conclusions regarding the performance of CNCE and CD, but rather merely highlight the adversarial nature of CD and its importance when the data density exhibits different scales of variation along different directions.
We take data concentrated around a 2-dimensional manifold embedded in 10-dimensional space. Specifically, let (1)) (10)) denote the standard basis in 10 Then each data sample is generated by adding Gaussian noise to a random point along 2D spiral lying in the (1)) plane. The STD times larger than that in the other axes. Figure 3a shows the projections of the the data samples onto the the first dimensions. Here, use
of the noise in the e!!) and e?2) directions
8

---

Published as a conference paper at ICLR 2021
multi-layer perceptron (MLP) parametric model, log pe. and train it using several different learning configurations (for the full details see App. D)
Figure 3b visualizes the training well as the final result achieved by each configuration. The first two rows show CNCE with Gaussian contrastive distributions of two different STDs. The third row its MCMC process. As can be seen, for CNCE with a large STD, the contrastive samples are able to explore large areas around the original samples, but this causes their majority to lie relatively far from the manifold (see their projections onto the e()) plane). In this case, ag decreases quickly, causing the learning process to ignore most samples at a very early stage of the training. When using CNCE with a small STD, the samples remain relevant throughout the training, but this comes at the price of inability to capture the global structure of the distribution. CD, on the other hand, is able to enjoy the best of both worlds as adapts the contrastive distribution over time, Indeed, as the learning progresses, the contrastive samples move closer to the manifold to maintain their relevance. Note that since we use the adjusted version of CD, the weights in this configuration are not precisely We chose the step size of the Langevin Dynamics so that the median of the weights is approximately 10-?
shows the adjusted CD described in Sec. 3.4 with Langevin Dynamics
Figure shsww the results achieved by different variants of CD. As can be seen, without correcting for the lack of detailed balance, CI) fails to estimate the density correctly. When using MH rejec- tion to correct the MCMC, or our adaptive CD (ADC) to correct the update steps, the estimate is significantly improved.
CONCLUSION
The classical CD method has seen many uses and theoretical analyses over the years. The original derivation presented the algorithm as an approximate gradient descent process for a certain loss. However, the accuracy of the approximation has been matter of much dispute, leaving it unclear what objective the algorithm minimizes in practice. Here, we presented an alternative derivation of CD's update steps, which involves no approximations. Our analysis shows that CD is in essence an ad adversarial learning procedure, where discriminator is trained to distinguish whether a Markov chain generated from the learned model has been fieeflipped or not. Therefore, although predating GANs by more than a decade, CD in fact belongs to the same family of techniques. This provides possible explanation for its empirical success.
Acknowledgement
This research supported by the Technion Ollendorff Minerva Center.
REFERENCES
Yoshua Bengio and Olivier Delalleau. Justifying and generalizing contrastive divergence. Neural Computation, 2106):100111611, 2009.
Miguel A Carreira-Perpinaa and Geoffrey E Hinton. On contrastive divergence learning. In Society for Artificial Intelligence and Statistics, volume 10, pp. 33-40, 2005.
Ciwan Ceylan and Michael U Gutmann. Conditional noise-contrastive estimation of unnormalised models. In International Conference on Machine Learning, pp. 726-734, 2018
Thomas M Cover and J Halliwell. Which processes satisfy the second law, Physical Origins of Time Asymmetry. pp. 98-107, 1994.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. In Advances in Neural Information Processing Systems, volume 32, pp. 3608-3618 2019.
Ruiqi Gao, Yang Lu, Junpei Zhou, Song-Chun Zhu, and Ying Nian Wu. Learning generative con- vnets via multi-grid modeling and sampling. In Conference on Computer Vision and Pattern Recognition, pp. 9155-9164, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems, pp. 2672-2680, 2014.

---

Published as a conference paper at ICLR 2021
Will Grathwohl, Kuan-Chiee Wang, Whern-Herrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference Learning Representations, 2019.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Society for Artificial Intelligence and Statistics, pp. 297- 304, 2010.
W Keith Hastings. Monte Carlo sampling methods using markov chains and their applications.
Biometrika, 57(1):97-109, 1970.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-1000) 2002.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 3 313(5786)5504-507, 2006.
Geoffrey E Hinton and Russ Salakhutdinov. Replicated softmax: an undirected topic model. In Advances in Neural Information Processing Systems, pp. 1607-1614, 2009.
Geoffrey E Hinton, Simon Osindero, and Wee- Teh. A fast learning algorithm for deep belief' nets. Neural Computation, 18(7):1227-1554, 2006.
Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6 6(A5r))695-709, 2005.
Aapo Hyvärinen. Connections between score matching, contrastive divergence, and pseudolikeli-
hood for continuous-valued variables. IEEE Transactions Neural Networks, 1855)1529-1531, 2007.
Qiang Liu and Dilin Wang. Learning deep energy models: Contrastive divergence vs. amortized mle. arXiv preprint arXiv:1707.00797, 2017.
Abdel-rahman Mohamed and Geoffrey Hinton. Phone recognition using restricted boltzmann ma- chines. In International Conference on Acoustics, Speech and Signal Processing, pp. 4354 4357. IEEE, 2010.
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy-based model. Advances Neural Information Pro- cessing Systems, pp. 5232-5242, 2019.
Yixuan Qiu, Lingsong Zhang, and Xiao Wang. Unbiased contrastive divergence algorithm for train- ing energy-basee latent variable models. In International Conference Learning Representa- tions, 2019.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep Boltzmann machines. In Artificial Intelligence and Statistics, pp. 448-455, 2009.
Ruslan Salakhutdinov, Andriy Mnih, and Geoffrey Hinton. Restricted Boltzmann machines for collaborative filtering. In International Conference on Machine Learning, pp. 791-798, 2007.
Paul Smolensky. Information processing in dynamical systems: Foundations of harmony theory. Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986.
Jascha Sohl-Dickstein, Peter Battaglino, and Michael DeWeese. Minimum probability flow learn- ing. In ICML, 2011.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In Advances in Neural Information Processing Systems, pp. 11918-1 19988-11930,,,1111111111111 2019.
Ilya Sutskever and Tijmen Tieleman. On the convergence properties of contrastive divergence. In International Conference Artificial Intelligence and Statistics, pp. 789-795, 2010.
Tijmen Tieleman. Some investigations into energy-based models. PhD thesis, University Toronto, 2007.
Jianwen Xic, Yang Lu, Song-Chin Chun Zhu, and Yingnian Wu. A theory of generative convnet. In International Conference Machine Learning, pp. 2635-2644. 2016.
10

---

Published as a conference paper at ICLR 2021
A
ALGORITHMS
Below we summarize the algorithms of the classical CD and the proposed adjusted version described in Sec. 3.4
Algorithm 1: Contrastive Divergence k
parametric model pe, MCMC transition rule (-(-))) with stationary distribution po, step size 1), chain length k. while not converged do Sample batch {or)== from the dataset
Require:
Initialize { } to be copy of the batch
for 1= to m:
for jorj=11 to & do
Draw sample x' from ge(-)ā,)
end
Vo (og Vo log)o(x;)
9:
end
0 + 0 / Σi 9:
end
Algorithm 2: Adjusted Contrastive Divergence
Require: parametric model pa, MCMC transition rule (-1.)) whose stationary distribution is Pg. step size 1), chain length k. while not converged do Sample batch {<,} from the dataset Initialize {= to copy of the batch for i=1 to m
w
for j=1 to 1: do
Draw sample x' from (-())a))
wit + win (po(""))
pp
I; x
end
a; t (1 + 1/w
Di Va log po (T:) Vo log)o(
end
0 - 0 / E,
end
B DERIVATION OF CNCE'S GRADIENT
Proof of Observation
The BCE loss achieved by the CNCE discriminator
7 is given by
lCNCE
-11 1E Anp [log(D((A,B))
E B~p
[log(1 - Do(A, B))]
B|Aru
AlBay
=- X~p X|Xwg
(Xo (x)))]
(21)
11

---

Published as a conference paper at ICLR 2021
where we used the fact that 1 - Do(a, b) = Da) Now, substituting the definition of Dg form g.. the gradient of (2T) can be expressed as
[Vo log (1 + 1oo (x,,
V VoLENCE
E X~p 8|Xm0
(1+ (x))
Vowro (x (x, x3]
E X-p X|Xma
wo (
=E X
1 w+ (Xx) x))
V Vowey (X,,
X|X<g
wo (X, X)
+ 1o (x, x)
V Vows(X (X, x)
E Xmp X|Xmqq
(X( X)
ux)(X,
E Xmpp X|X
+ wxx x)-1)
Vo logg
E Xap
o0 xo(X, X
Vo loggoo - Vo Vologro(x)) (x))
(22)
X|x-q
where we used the fact that Vowg we V (w) and the definition of ag from
C
DERIVATION OF THE GRADIENT OF CNCE WITH MULTIPLE MC STEPS
We here describe the full derivation of the gradient in [8] following the same steps as in BCE loss achieved by the discriminator in 17 is given by
The
Lchain E log( (x
(0) X(a)
x()))]]
(23)
where we again used the fact that 1 - (D 0n .a)) = Do(an, .k.......00 Now, substituting the definition of Dg form (7), the gradient of (3 can be expressed
II wo (X xx(0))
V o Velhnni
E
V log
1
wo (x ((11)
Wg (X (x11)) ((01
E
1
V
[1 wim ( (X (.xX-ix
wo (x (((0)
I oom (X 11, x(i)
I Jom (1) ((1)
k II
wo (x(6-1) 1) S (oaa
1 wol wo (x((i-1) x(6))-1 (x())1))(()))111
Vo log
E 1oo (X
x(t)) (vo log po (
Vo log (oogpo(X(0) (x)
(24)
where we used the definition of og from
19
D TOY EXPERIMENT AND TRAINING DETAILS
We here deseribe the full details of the toy model and learning configuration, which we used to produce the results in the paper. The code for reproducing the results is available at - (for the blind review the code will be available in the supplementary material).
The toy model used in the paper consists of a distribution concentrated around a 2D spiral embedded in 10 dimensional space. Denoting the 10 orthogonal axes of the standard basis in this space by
12

---

Published as a conference paper at ICLR 2021
FC x0 512
Leaky ReLUU
FC
Laaky ReLL
FC x12
Leaky ReLU
FC x12 512
Lecky ReLU
512 512
Locky ReLU
FC 512
Leaky RaLU
512
Lasky RaLU
FC (1212xx55 x
Figure 5: The architecture.
......eee20)) the spiral lies in the e plane and is confined to [-1, 1] in each of these two axes. The samples of the model are produced by selecting random points along the spiral and adding Gaussian noise them. In order keep the samples close to the e(2)_g(2) plane used
isotropic noise with STD of 0.05 in the e(1) and e(2) directions, and STD of 0.01 in the directions e(3) " (200)
As parametric model for po(),, we used an 8-layer multi-layer perceptron (MLP) width 512 with skip connections, illustrated in Fig. 5
Throughout the paper we referred to the results of five different learning configurations.
1. CNCE with optimal (small) variance. This configuration additive Gaussian noise as its contrastive distribution. We found 0.0075 to be the STD of the Gaussian which produces the best results.
2. CNCE with large variance. This configuration is similar the previous except for the STD of the Gaussian which was set to 0.3 in order to illustrate the problems of using a conditional distribution with large variance.
3. CD without any MCMC correction. For the MCMC process used 5 steps of Langevin dynamics, where we did not employ any correction for the inaccuracy which results from using Langavin dynamics with finite step size. We found 0.0075 to be the step size (multiplying the standard Gaussian noise term) which produces the best results.
4. CD with MH correction. This configuration similar to the previous one except for MH rejection scheme which was used during the MCMC sampling. In this case we found the step size of 0.0125 to produce the best results.
5. Adjusted CD. This configuration similar to the previous one except that used the method from Sec. Sec. instead of MH rejection. Similarly to the previous configuration, we found the step size of 0.0125 to produce the best results.
The optimization of all configurations exponential decaying learning rate. Except for the training of the third configuration, the learning rate ran down from 10 10 over 100000 optimization steps. For the third configuration had to reduce the learning rate by a factor of 10 in order to prevent the optimization from diverging.
preformed using SGD with momentum of 0.9 and
In order select the best step size / variance for each of the configurations we ran parameter sweep around the relevant value range. The results of this sweep are shown in Fig. 6
For the selection of the number of training steps, have iteratively increased the number of steps until the results stopped improving for all configurations. These results are presented in Fig.
13

---

Published as a conference paper at ICLR 2021
Step size 0.005
Stop size 0,0075
Stop s220 0.01
Stop sbe = 0,0125
Step 526 0.015
Stop size 0,0175
CNCE
L
CDw coneeciinn
CD +
r
ACD
L
Figure 6: The step size sweep. In the case of CNCE, the step size is in fact the STD the conditional distribution. For the case of CD, the training has diverged for large step sizes even after the learning rate has been significantly reduced. The highlighted figures indicate the selected step sizes.
14

---

Published as a conference paper at ICLR 2021
# of Steps = 10000
of Steps 20000
# of Steps 50000
# of Steps 100000
of Steps 200000
ONCE
L
L
L
L
CD wio correction
L
CD + MH
L
L
L
ACD
Figure 7: Selecting the number of training steps.
15


Reviews:
[{"review_id": "yNU4iv7jxr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nTo implement the contrastive divergence (CD) algorithm in practice, an intractable term is typically omitted from the gradient. This leads to an approximation. This work shows that the resulting algorithm can in fact be viewed as an exact algorithm targeting a different, adversarial objective. The derivation in this paper also shows how Markov chains which are not reversible w.r.t. the posterior distribution of interest can be employed within the algorithm. Effectively, this assigns an importance weight to each sample which akin to the acceptance ratio which would be needed for a Metropolis--Hastings type correction.\n\n\n\nStrengths and novelty\n\nTo my knowledge, the derivation of the relationship between CD and conditional noise contrastive estimation (CNCE) is novel. Making it clear how these algorithms are related is a contribution worth publishing.\n\n\nWeaknesses\n\nPerhaps having an additional non-toy example would have been a nice illustration. However, since the paper's main focus is on establishing theoretical connections between CD and CNCE, I do not believe that the lack of further numerical examples should preclude publication.\n\n\n\nMinor comments\n\n- A number of entries in the bibliography have typos such as missing capitalisation of proper nouns inconsistent use of capital letters in journal and conference names.\n\n- I don't understand the last term in Eq. 2. This needs to be more rigorously written.\n\n- Section 3.2 extablishes that the CD-1 gradient is a special case of the CNCE gradient. This would mean that both CD-1 and CNCE lead to computationally the same algorithm. However, this appears not to be the case in the toy example. For readers not familiar with CNCE, please make it more clear in what way CD-1 and CNCE differ in practice if both use the same reversible Markov chain.\n\n- In Section 2.1, it is stated that CD does not use log-likelihood loss. However, it seems to me that for $p_\\theta$-reversible Markov chains, if the chain is either fast mixing or the number of iterations $k$ sufficiently large, the gradient in Eq. 3 is proportional to the gradient of the log-likelihood (multiplied by $-1$) in expectation because in this case, the first term has expectation zero.", "extra_scores": {}, "extra_text": {}}, {"review_id": "t6EYOQ2Q-2Y", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper presents a way to view contrastive divergence (CD) learning as an adversarial learning procedure where a discriminator is tasked with classifying whether or not a Markov chain, generated from the model, has been time-reversed. Beginning with the classic derivation of CD and its approximate gradient, noting relevant issues regarding this approximation, the authors present a way to view CD as an extension of the conditional noise contrastive estimation (CNCE) method where the contrastive distribution is continually updated to keep the discrimination task difficult. Specifically, when the contrastive distribution is chosen such that the detailed balance property is satisfied, then the CNCE loss becomes exactly proportional the CD-1 update with the derivation further extended to CD-k. Practical concerns regarding lack of detailed balance are mitigated through the use of Metropolis-Hastings rejection or an adaptive weighting that arises when deriving the gradient of their time-reversal classification loss.  A toy example providing empirical support for correcting the lack of detailed balance is included.\n\nStrengths:\nThe paper is well written. From \"first principles\" through the CD-CNCE link, the paper was straightforward to follow without technical issues and with appropriate references. The results of this work appear novel, and proofs seem correct. The ability to use the weighting to address detailed balance in practice is neat, and the experiments, though limited, show promise.\n\nConcerns:\nI understand performance comparisons and experiments were not the focus of the paper. However, considering the newly presented link between CNCE and CD, it would have been exciting to see some evaluation metrics. Perhaps even just a simple experiment from the original NCE or CNCE papers. \nFor the MCMC process, appendix D mentions that 5 steps of Langevin dynamics were used. How was 5 selected? Was there any significant gain or degradation when it varied? More generally, what kind of impact does chain length have on the discriminator’s classification ability? Does chain length affect the behavior of CD with MH correction and adjusted CD similarly?\n\nMinor:\nIn Figure (3b), it is said that CD is based on Langevin dynamics MCMC adjusted with the method of Sec. 3.4 yet both MH and the weight adjustment are included there. Which one is in Fig. (3.4) \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8cBLeu0qZC", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-rebuttal update: Thank you for your response.  My concerns are relatively minor and I believe this work is above the acceptance threshold.\n\n### Summary\n\nThis paper formulates CD-k training as an adversarial game, where the EBM parameterizes a discriminator which tries to classify whether a k-step Markov chain is reversed or not.\n\n### Reasons for Score\n\nPros:\n\n+ While there have been analyses of CD training under different restrictive scenarios, to my knowledge this is the first paper whose formulation applies to the CD-k algorithm used in practice.\n\n+ The derivations seem correct.\n\nCons:\n\n- While it is nice to have a justification for CD-k, the manuscript could be improved if there is more discussion about practical implications of this formulation. For example, is it possible to include a discussion on the commonly used tempered Langevin dynamics (Nijkamp et al, 2019, Du & Mordatch, 2019), which seems justifiable with the current formulation?\n\n- The impact of the work is still limited by the fact that it doesn't explains the PCD algorithm, which is more common in large-scale settings.\n\n### Minor Comments\n\n- It is probably better to change the notations to signify the fact that the gradient of $\\tilde{X}$ or $X^{(k)}$ wrt $\\theta$ will not be accounted, e.g., by using $\\partial_\\theta$ instead of $\\nabla_\\theta$, or $\\text{stop\\_gradient}$. \n\n- It is technically incorrect to say that CD's update steps don't correspond to the gradient of any objective. CD-1 with infinitesimal step-size corresponds to various well-defined objectives, see the references below.\n\n- In Eq.(15), the subscript in $E_{q_\\theta}$ should probably be dropped, to be consistent with the rest of the paper.\n\n### References\n\nHyvarinen, Aapo. \"Connections between score matching, contrastive divergence, and pseudolikelihood for continuous-valued variables.\" IEEE Transactions on neural networks 18.5 (2007): 1529-1531.\n\nSohl-Dickstein, Jascha, Peter Battaglino, and Michael R. DeWeese. \"Minimum probability flow learning.\" arXiv preprint arXiv:0906.4779 (2009).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6FJuWMBxSp", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# General statements\nThis paper has a special flavour, in the sense that it provides new light on a very established training method for energy-based models: contrastive divergence. Its core contribution is to provide a theoretically grounded understanding of CD as it is widely used, avoiding the common assumption that this algorithm stems out of a simplifying assumption.\n\nThis is done through a connection between CD and adversarial training. On their way, the authors show how some minor corrections suggested by their interepretation may dramatically improve performance of CD, at least on their toy example.\n\nSince CD is a widely accepted method, I feel that the deliberate choice of restricting their experiments on toy data is legitimate.\n\n\nAll in all, I would say that the paper is a very nice read, and its english usage is good, as well as the references that are appropriate.\nI think that it is appropriate for presentation at ICLR, since it may stimulate new research on CD.\n\n\n# Detailed comments\nBelow are some minor comments in chronological order\n## Introduction\n* \"Thus, Our\": uppercase\n\n## Toy example\n* In figure 4, you probably mean \"from left to right\"\n* To be extra sure, are you effectively disabling gradient recording when computing \\tilde{x} as I assume you do ? I'm asking because \\tilde{x} actually appears as a function of x, parameterized by \\theta, i.e. as \\tilde{x}_\\theta(x), since it involves the transition kernel q_theta for its computation. As you write below eq. (17), you are considering that the kernel q as kept fixed, explaining such a choice. \nhowever, and if I'm not mistaken, it should not be too difficult with autograd mechanics to include this dependency in the updates. Did you try it ? Did it break the algorithm ? \n* I would appreciate more steps in your derivations (22) and (24): I don't follow easily the transitions to lines 2 and 4 of each.\n* The neural net used for the toy data looks impressively large (8 layers of FC+leakyReLU with 512 hidden size). Was it really necessary ?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper reveals a novel interpretation of the well-established CD for energy-based model training as an adversarial game through conditional NCE. The paper could be potential impactful for the community of EBMs.

There are several points should be addressed in final version:

1, Based on such an interpretation, the number of steps becomes a tunable parameters, rather than in vanilla understaning in CD-family (the larger, the better in terms of approximation, by with more computation cost).

2, It is okay to stop the gradient when solving an adversarial game as the paper discussed. However, propagating the gradient through the component is also another choice, which leads to the algorithm proposed in [1].

It will be interesting to discuss these in the paper.

[1] Sohl-Dickstein, Jascha, Peter Battaglino, and Michael R. DeWeese. "Minimum probability flow learning." arXiv preprint arXiv:0906.4779 (2009).


Decision: Accept (Spotlight)