Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: # General statements
This paper has a special flavour, in the sense that it provides new light on a very established training method for energy-based models: contrastive divergence. Its core contribution is to provide a theoretically grounded understanding of CD as it is widely used, avoiding the common assumption that this algorithm stems out of a simplifying assumption.

This is done through a connection between CD and adversarial training. On their way, the authors show how some minor corrections suggested by their interepretation may dramatically improve performance of CD, at least on their toy example.

Since CD is a widely accepted method, I feel that the deliberate choice of restricting their experiments on toy data is legitimate.


All in all, I would say that the paper is a very nice read, and its english usage is good, as well as the references that are appropriate.
I think that it is appropriate for presentation at ICLR, since it may stimulate new research on CD.


# Detailed comments
Below are some minor comments in chronological order
## Introduction
* "Thus, Our": uppercase

## Toy example
* In figure 4, you probably mean "from left to right"
* To be extra sure, are you effectively disabling gradient recording when computing \tilde{x} as I assume you do ? I'm asking because \tilde{x} actually appears as a function of x, parameterized by \theta, i.e. as \tilde{x}_\theta(x), since it involves the transition kernel q_theta for its computation. As you write below eq. (17), you are considering that the kernel q as kept fixed, explaining such a choice. 
however, and if I'm not mistaken, it should not be too difficult with autograd mechanics to include this dependency in the updates. Did you try it ? Did it break the algorithm ? 
* I would appreciate more steps in your derivations (22) and (24): I don't follow easily the transitions to lines 2 and 4 of each.
* The neural net used for the toy data looks impressively large (8 layers of FC+leakyReLU with 512 hidden size). Was it really necessary ?
review_title: A new theoretical understanding of contrastive divergence as adversarial training

final_score: rating: 7: Good paper, accept
review_content: Summary

To implement the contrastive divergence (CD) algorithm in practice, an intractable term is typically omitted from the gradient. This leads to an approximation. This work shows that the resulting algorithm can in fact be viewed as an exact algorithm targeting a different, adversarial objective. The derivation in this paper also shows how Markov chains which are not reversible w.r.t. the posterior distribution of interest can be employed within the algorithm. Effectively, this assigns an importance weight to each sample which akin to the acceptance ratio which would be needed for a Metropolis--Hastings type correction.



Strengths and novelty

To my knowledge, the derivation of the relationship between CD and conditional noise contrastive estimation (CNCE) is novel. Making it clear how these algorithms are related is a contribution worth publishing.


Weaknesses

Perhaps having an additional non-toy example would have been a nice illustration. However, since the paper's main focus is on establishing theoretical connections between CD and CNCE, I do not believe that the lack of further numerical examples should preclude publication.



Minor comments

- A number of entries in the bibliography have typos such as missing capitalisation of proper nouns inconsistent use of capital letters in journal and conference names.

- I don't understand the last term in Eq. 2. This needs to be more rigorously written.

- Section 3.2 extablishes that the CD-1 gradient is a special case of the CNCE gradient. This would mean that both CD-1 and CNCE lead to computationally the same algorithm. However, this appears not to be the case in the toy example. For readers not familiar with CNCE, please make it more clear in what way CD-1 and CNCE differ in practice if both use the same reversible Markov chain.

- In Section 2.1, it is stated that CD does not use log-likelihood loss. However, it seems to me that for $p_\theta$-reversible Markov chains, if the chain is either fast mixing or the number of iterations $k$ sufficiently large, the gradient in Eq. 3 is proportional to the gradient of the log-likelihood (multiplied by $-1$) in expectation because in this case, the first term has expectation zero.
review_title: Establishes an interesting theoretical link

final_score: rating: 7: Good paper, accept
review_content: Summary:
This paper presents a way to view contrastive divergence (CD) learning as an adversarial learning procedure where a discriminator is tasked with classifying whether or not a Markov chain, generated from the model, has been time-reversed. Beginning with the classic derivation of CD and its approximate gradient, noting relevant issues regarding this approximation, the authors present a way to view CD as an extension of the conditional noise contrastive estimation (CNCE) method where the contrastive distribution is continually updated to keep the discrimination task difficult. Specifically, when the contrastive distribution is chosen such that the detailed balance property is satisfied, then the CNCE loss becomes exactly proportional the CD-1 update with the derivation further extended to CD-k. Practical concerns regarding lack of detailed balance are mitigated through the use of Metropolis-Hastings rejection or an adaptive weighting that arises when deriving the gradient of their time-reversal classification loss.  A toy example providing empirical support for correcting the lack of detailed balance is included.

Strengths:
The paper is well written. From "first principles" through the CD-CNCE link, the paper was straightforward to follow without technical issues and with appropriate references. The results of this work appear novel, and proofs seem correct. The ability to use the weighting to address detailed balance in practice is neat, and the experiments, though limited, show promise.

Concerns:
I understand performance comparisons and experiments were not the focus of the paper. However, considering the newly presented link between CNCE and CD, it would have been exciting to see some evaluation metrics. Perhaps even just a simple experiment from the original NCE or CNCE papers. 
For the MCMC process, appendix D mentions that 5 steps of Langevin dynamics were used. How was 5 selected? Was there any significant gain or degradation when it varied? More generally, what kind of impact does chain length have on the discriminator’s classification ability? Does chain length affect the behavior of CD with MH correction and adjusted CD similarly?

Minor:
In Figure (3b), it is said that CD is based on Langevin dynamics MCMC adjusted with the method of Sec. 3.4 yet both MH and the weight adjustment are included there. Which one is in Fig. (3.4) 

review_title: This paper is acceptable, and the ICLR community may benefit from the contributions this paper brings to light. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Post-rebuttal update: Thank you for your response.  My concerns are relatively minor and I believe this work is above the acceptance threshold.

### Summary

This paper formulates CD-k training as an adversarial game, where the EBM parameterizes a discriminator which tries to classify whether a k-step Markov chain is reversed or not.

### Reasons for Score

Pros:

+ While there have been analyses of CD training under different restrictive scenarios, to my knowledge this is the first paper whose formulation applies to the CD-k algorithm used in practice.

+ The derivations seem correct.

Cons:

- While it is nice to have a justification for CD-k, the manuscript could be improved if there is more discussion about practical implications of this formulation. For example, is it possible to include a discussion on the commonly used tempered Langevin dynamics (Nijkamp et al, 2019, Du & Mordatch, 2019), which seems justifiable with the current formulation?

- The impact of the work is still limited by the fact that it doesn't explains the PCD algorithm, which is more common in large-scale settings.

### Minor Comments

- It is probably better to change the notations to signify the fact that the gradient of $\tilde{X}$ or $X^{(k)}$ wrt $\theta$ will not be accounted, e.g., by using $\partial_\theta$ instead of $\nabla_\theta$, or $\text{stop\_gradient}$. 

- It is technically incorrect to say that CD's update steps don't correspond to the gradient of any objective. CD-1 with infinitesimal step-size corresponds to various well-defined objectives, see the references below.

- In Eq.(15), the subscript in $E_{q_\theta}$ should probably be dropped, to be consistent with the rest of the paper.

### References

Hyvarinen, Aapo. "Connections between score matching, contrastive divergence, and pseudolikelihood for continuous-valued variables." IEEE Transactions on neural networks 18.5 (2007): 1529-1531.

Sohl-Dickstein, Jascha, Peter Battaglino, and Michael R. DeWeese. "Minimum probability flow learning." arXiv preprint arXiv:0906.4779 (2009).

review_title: Interesting formulation, practical impact seems limited

Meta-review: This paper reveals a novel interpretation of the well-established CD for energy-based model training as an adversarial game through conditional NCE. The paper could be potential impactful for the community of EBMs.

There are several points should be addressed in final version:

1, Based on such an interpretation, the number of steps becomes a tunable parameters, rather than in vanilla understaning in CD-family (the larger, the better in terms of approximation, by with more computation cost).

2, It is okay to stop the gradient when solving an adversarial game as the paper discussed. However, propagating the gradient through the component is also another choice, which leads to the algorithm proposed in [1].

It will be interesting to discuss these in the paper.

[1] Sohl-Dickstein, Jascha, Peter Battaglino, and Michael R. DeWeese. "Minimum probability flow learning." arXiv preprint arXiv:0906.4779 (2009).


Decision: Accept (Spotlight)