Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper describes training a sequence of conditional EBMs (inspired by  Ho et al. (2020)) instead of training unconditional EBMs.  Each conditional energy describes the probability of recovering x, given its noisy version \hat{x}. The noisy version of x can be described as a normal distribution centered at x, so the condition EBM has an additional term ||x - \hat{x}||^2, which constrains the Langevin dynamics to remain in the vicinity of \hat{x}, so it converges faster!
The inference is that starting at white noise x_0, it defines a conditional EBMs given x_0, runs Langaving dynamics to sample from P(x|x_0) defined by the conditional EBMs, and then use the sample as the evidence of another conditional EBMs.

I assume the main advantage of this training over Ho et al. (2020) is having an energy-based model that can be used in other applications, so that would be nice to see the performance of trained EBMs on some applications other than image generation such as image inpainting and robust classification as discussed in Du and Mordatch (2019).

In general, I believe that this is an exciting paper and an important step towards training better EBMs.

In connection to score matching, Saremi et al. (2018) and Saremi and Hyvarinen (2019) should be cited as well.

Saremi et al. (2018), Deep Energy Estimator Networks.
Saremi and Hyvarinen (2019), Neural Empirical Bayes.

typo: "score-based based methods" in Section 4.1

Alg2: for t \gets T - 1 to 0 do
review_title: Interesting approach for training EBMs as a sequence of conditional EBMs

final_score: rating: 7: Good paper, accept
review_content: The paper proposed a novel method to train EBMs based on diffusion recovery likelihood. It constructs a sequence of noisy version of data and learn a conditional between consecutive noisy pairs. Compare to working with the likelihood directly, doing so makes the training much easier. Besides, even using a potentially non-convergent MCMC for gradient estimation, it still leads to a well-behaved energy potential, unlike EBMs trained via maximising the likelihood directly.

## Pros

1. The paper is well-written and easy to understand. 
2. The method is novel and solves the training difficulty issue for EBMs neatly.
3. The analysis of the normal approximation and how it leads to choose step sizes in the Langevin dynamics is neat.
4. The paper discusses related work well while motivating the method and also makes interesting connections to VI and score matching in Section 3.4.

## Cons

1. Only applications on image generations are conducted.

## Questions

1. Does the method work only with the specific architecture here, especially the Transformer sinusoidal position embedding?
2. It says T = 6 and K = 10 leads to unstable training in Section 4.1. Does the optimization diverge or something? It would be good to see some trace plots for the training procedure.
3. Why do you use the specific HMC setup for Section 4.2? Specifically, using a leapfrog step of 2 can lead to issues in which the MCMC only explore local modes. I would like to see how a long-run NUTS works here. A well-behaved potential should work in both settings.

## Discussions

1. I'm quite glad to see Section 4.2 that shows the long-run MCMC works on the trained EBMs, even with (potentially) biased gradients for training. Is it because we only need relative short-run MCMC for convergence when using recovery likelihood so that the bias is actually small?
2. How does the method relate to amortized MCMC?

## Related work

There are a few more related works that the authors may consider including.

[1] proposed a similar idea of making the modelling task easier using a sequence of targets (for EBMs), but achieved by density ratio estimation.
[2] uses coupled MCMC to resolve the biased gradient issue when using CD.

[1] Rhodes, Benjamin, Kai Xu, and Michael U. Gutmann. "Telescoping Density-Ratio Estimation." Advances in Neural Information Processing Systems. 2020.
[2] Qiu, Yixuan, Lingsong Zhang, and Xiao Wang. "Unbiased Contrastive Divergence Algorithm for Training Energy-Based Latent Variable Models." International Conference on Learning Representations. 2019.
review_title: Very nice method!

final_score: rating: 6: Marginally above acceptance threshold
review_content: #### Summary of the paper :
The authors propose to learn the recovery likelihood of a sequence of Energy-Based Models (EBM) trained on increasingly noisy version of the datasets. The authors demonstrate that optimizing the recovery likelihood is more convenient than learning the actual marginal likelihood, as it leads to more stable MCMC sampling and provide high-fidelity samples. The authors show competitive generation performances on CIFAR-10, Celeb1 and LSUN datasets.

#### Pros : 
* The problem of non-convergent MCMC sampling in EBM trained with marginal likelihood is well motivated and well illustrated.
* The authors proposition is well grounded in theory, and comes with rigorous and strong mathematical derivation.
* The paper is clear, well written, and well structured

#### Cons:
* The quantitative comparison (IS and FID) is performed only on the CIFAR-10 datasets (see detailed comments).
* The experimental evidences are not convincing.

#### Recommandation:
Given the strength of the theoretical derivation, I would tend to accept the paper (marginally above threshold). However, stronger experimental evidences / discussion concerning the comparison with [1] & [2] would be beneficial for a better recommandation.

#### Detailed comments:
* While the authors show qualitative sampling results on LSUN, CIFAR-10, and CelebA they are quantitatively comparing the proposed framework on CIFAR-10 only. Even if CIFAR is challenging for generative models, I think the quantitative comparison on other databases is missing. In particular the comparison with [1] and [2] is necessary as the main claim of the paper is to show improvement using recovery likelihood.

* The strength of the authors proposition is to provide a framework that could leverage less diffusion time steps than [2] while keeping good generation performance. Even if the authors clearly state such a theoretical advantage in paragraph 3.4, the experimental evidence is not convincing. Less diffusion time step (e.g. the case T=6, K=50) results in degraded FID and IS compared to [2]. In addition, the comparison between  T=1000, K=0 (DSM) versus [2] show a significant advantage in favor of [2] (they also use 1000 time step in [2]). What is the reason ? It suggests that optimizing the recovery likelihood is less efficient than optimizing the marginal one in diffusion problem...

* Comparable setup (i.e T=1000, K=0 (DSM) versus T=1000, K=0) lead also to degraded performance : Why score-matching approach leads to better performance than diffusion recovery ? The quality of the paper would be greatly improved with such a discussion.

#### Typos and suggestions to improve the paper
* The initials 'MLE' are not explicitly defined
* The Eq 3 should come with a demonstration (in Annex) or at least with a reference .
* Eq 9 should be an approximate sign as it is a Taylor expansion.
* The transition from Eq.9 to Eq.10 should be demonstrated (in Annex)
* Eq 13 : N —> \matcal{N}
* Algorithm 2, line 2: It seems that there is a typo : t <- T-1 to T? Is it : t <- T-1 to 0

[1] : Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. On learning non-convergent short- run mcmc toward energy-based model. arXiv preprint arXiv:1904.09770, 2019b. 

[2]: Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arXiv:2006.11239, 2020.
review_title: Strong theory, experimental part is weaker

Meta-review: The paper presents an interesting model to sample from the Gibbs distribution using diffusion based method. The theory is interesting and it is related well to the current research in the field. All reviewers agree that this is a noteworthy contribution to ICLR. 
Decision: Accept (Poster)