Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: 
The authors extensively analyze the training of Restricted Boltzmann Machines (RBMs).
They show that there is definite room for improvement in training RBMs in terms of

- *Evaluation:* The authors show that despite its popularity, log likelihood is not the only metric which should be tracked while training the RBM and show that bad machines can have a deceptively good log likelihood but not be able to generate good samples and vice versa.
- *Methods:* Despite their popularity, both CD-k and PCD-k can be outperformed in terms of the quality of generated samples by Rdm-k which uses chains started at uniformly sampled states to compute the negative statistics

Moreover they empirically show that for Rdm-k, smaller values of k lead to models which memorize data (Figure 2A $\Delta S$) and a larger k leads to generalization (likelihood of generated samples is close to that of the data) (Figure 2A $\mathcal{L}$).
main_review: ### Strengths

- The authors conducted extensive experiments to analyze what had been thought to be an un-interesting and known aspect of RBM training and have demonstrated surprising results.
- The entire paper (especially the introduction) is well written, motivation is well presented and the experimental evaluation is well explained.
- The paper proposes and summarizes multiple very useful metrics which can be used by practitioners to track their RBM training.

### Weaknesses

- [Clarified in rebuttal] The evaluation and the methods proposed, while highly informative and useful, are computationally expensive.
  - The authors mention this in their supplementary material. 
- While the mixing time estimation is interesting a comparison with theoretical bounds, in particular [1] would have been more interesting.
- Possible citations:
  - [2] could be cited as a more recent survey of training methods along with the citation on line 33
  - [3] was recently published in ICML and proposes a novel method to train RBMs which is unrelated to CD-k, a similar analysis of that method compared to Rdm-k would be very interesting.
  - [4] proposed a method to train RBMs using random walk tours which provide an almost unbiased estimate of the negative statistics
    - they additionally observed a similar effect where samples from shorter tours (k) demonstrated memorization and longer tours (k) demonstrated generalization, which is corroborated by the observations in the current paper.
  - [5] proposed using the up and coming unbiased couplings method to train EBMs including RBMs, since this method is scalable to larger RBMs it could be the subject of such a study.

### Other criteria

- Originality : Existing problem, marginally novel solution, exciting and novel results.
- Quality : Extremely high quality of writing.
- Clarity : Experiments are well justified,  motivations are well explained, results are well elucidated.
- Significance : High significance in terms of insight, prohibitive computational cost will hamper utility.

[1] - Tosh, Christopher. "Mixing rates for the alternating Gibbs sampler over restricted Boltzmann machines and friends." International Conference on Machine Learning. PMLR, 2016.

[2] - Song, Yang, and Diederik P. Kingma. "How to train your energy-based models." arXiv preprint arXiv:2101.03288 (2021).

[3] - Grathwohl, Will, et al. "Oops I Took A Gradient: Scalable Sampling for Discrete Distributions." arXiv preprint arXiv:2102.04509 (2021).

[4] - Savarese, Pedro, Mayank Kakodkar, and Bruno Ribeiro. "From monte carlo to las vegas: Improving restricted boltzmann machine training through stopping sets." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

[5] - Qiu, Yixuan, Lingsong Zhang, and Xiao Wang. "Unbiased contrastive divergence algorithm for training energy-based latent variable models." International Conference on Learning Representations. 2019.
limitations_and_societal_impact: Adequately addressed
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper discusses about some important factors affecting the results of RBMs.

1. The choice of mixing time.

2. The mixing time and the number of steps(k) affect two regimes of RBMs, namely equilibrium and out-of-equilibrium.

3. How to choose the number of steps(k). (A) short k for convincing samples in short learning times (B) large k for learning the correct equilibrium distribution of the RBM.
main_review: 1. The main problem of this paper is lack of innovation. The author compares the results of 3 different recipes (CD, PCD, Rdm) proposed by others, and puts forward that the number of MCMC steps has a dramatic effect on the model. This paper does not improve the previous algorithms a lot and not give a specific method or function concerning how to choose number of MCMC steps.

2. All the conclusions and results are proved empirically, not theoretically. However, the model has only been applied on two datasets, and deals with only binary nodes. The conclusions should be more convincing by training on more datasets.

3. The algorithm and results in section 2-5 can be presented in a clearer way by adding flow chart of the algorithm and tables of different results.

==============
After reading the response of authors, I update my comments (see below).
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors demonstrate that many RBM models are learned in an unexpected out-of-equilibrium regime where the distribution of short run MCMC samples used to update the model differs greatly from the long run equilibrium distribution of the model. The claims are supported with diagnostics to measure out-of-equilibrium behavior and visualizations of samples after across MCMC updates.
main_review: The main objective of this paper is to demonstrate that RBM training has an unexpected but widespread behavior where samples generated by short run sampling using common initialization techniques like CD and PCD are typically out of equilibrium. Despite this, learning can remain stable and generate short run images that are very similar to data images, and the apparent success of learning a generative model can lead to the mistaken conclusion that the RBM has learned a valid equilibrium distribution. The authors question common practice of evaluating RBM's exclusively using short-run samples, which are not indicative of the steady-state, or negative samples from the end of training, which might not even be configurations which are attainable by the model once the weights are frozen. To examine the equilibrium distribution of RBM's, the authors use long run MCMC samples and find that equilibrium states are poor samples across many learning configurations. The points raised by the authors are crucial from the perspective of rigorous probabilistic learning of an RBM and this work sheds light on important but under-recognized phenomena in generative modeling.

The main weakness of this work is that the authors do not sufficiently address the related work [a] which covers many of the same ideas. The work [a] identifies a convergent and non-convergent learning phase for the Energy-Based Model (EBM) that directly correspond to the equilibrium and out-of-equilibrium phases identified by the authors for RBM training. It appears that this phenomenon presented by the authors occurs in energy models with different kinds of potentials. Given the high degree of similarity between these two works, I recommend against acceptance until a discussion of the connections to [a] is made clear by the authors in a revised version.

The authors present several diagnostics to identify equilibrium and out-of-equilibrium behavior. The most convincing diagnostics to me were the auto-correlation measures over the course of training. The other diagnostics seemed to be somewhat ad-hoc but nonetheless they appear to be useful indicators. One diagnostic that that authors did not include but possibly should is a measurement of the energy of the visible states over the course of training. Typically one would expect that the energy of short run MCMC samples should be about the same as data samples. The work [a] shows that when the short run samples are out of equilibrium, then long-run samples typically have much lower energy than short run or data samples. Successful learning should occur when short run, long run, and data energy values remain in the same spectrum, and it might be possible to diagnose out of equilibrium behavior for RBMs by examining whether long run visible energy values are in the same spectrum as short run and/or data samples. 

The scale and complexity of the experiments is somewhat limited because the authors exclusively examine binary pixel states rather than continuous and multi-channel pixels. The authors clearly demonstrate that out-of-equilibrium learning occurs for RBM models. I am a little uncertain over the author's exact conclusion regarding the method for learning an equilibrated model. The abstract claims that increasingly large k can be used to learn a valid equilibrium, but experiments with Rmd-k show even for k=10,000, non-equilibrium effects appear after around 100k MCMC updates. On the other hand, PCD learns a more reasonable equilibrium even for smaller k, which is consistent with what is reported in [a]. It would be good for the authors to clarify strategies for learning a correct equilibrium. 

**Overall Summary**: This work identifies an important and under-recognized outcome of out-of-equilibrium RBM learning. Bringing attention to this phenomenon is useful and valuable. The major weakness of this work is that it does not include a related work [a] that presents very similar findings. The experimental complexity could also be increased by using continuous pixels and color images. I recommend against accepting this work until relations to prior work are made clear. I am willing to change my score based on author feedback and revisions.

[a] On the anatomy of MCMC-based maximum likelihood learning of energy-based models. E. Nijkamp, M. Hill, T. Han, Y.N. Wu, S.C. Zhu. AAAI 2020. https://arxiv.org/abs/1903.12370
limitations_and_societal_impact: The authors have adequately addressed limitations and impact of their work.
review_title: null

Meta-review: This experimental paper has been well received by the reviewers, who are overall fairly confident that the paper deserves to be accepted. They also unanimously point out the quality of writing and presentation. The reviewers have done a great job, and I trust the authors to implement the promised revisions before publication. The main weakness appears to be a lack of originality. But given the timeliness of the models studied here and the overall quality of presentation, this papers complements well existing literature pointing at the non-equilibrium properties of energy based generative models. Moreover the reviewers acknowledge mostly convincing responses to their comments. In particular, I think it is important that, as promised by the authors, they implement the numerical experiments on more datasets, especially given the pure experimental nature of the paper. Second, a reviewer commented: "Can authors provide a reference protocol for training RBM in applications based on results of this paper? This would be helpful for readers." I completely back-up that point. I think that this would add to the paper. Last but not least: it is of crucial importance that the authors extensively discuss and compare their results to reference https://arxiv.org/abs/1903.12370 , and explain in details how their contributions complements it and confirms further its initial findings in a different context.






Decision: Accept (Poster)