Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work revisits the GRBM to improve its learning and sampling process. One key innovation is introducing Langevin sampling to GRBMs. Two variants are proposed: sampling the visible marginal potential, and hybrid Gibbs-Langevin sampling that trades off between Gibbs updates of hidden units and Langevin updates of visible units. The other key innovation is initializing MCMC samples from noise so that the model can generate samples from noise once training is over. Experiments show that the learning method can successfully train GRBMs that can generate reasonable images from scratch. 
strength_and_weaknesses: STRENGTHS:

1. The work improves on GRBM learning and introduces the first RBM method that can generate images from scratch.
2. Sample quality is reasonable given the restrictions of the potential energy.

WEAKNESSES:

1. The motivation of using GRBMs is unclear. From one perspective, the visible GRBM is simply a deep EBM with a restricted architecture. The hidden units distinguish the GRBM and RBM family, but the applications do not depend on the hidden representations. A clear motivation for the use of a GRBM would greatly strengthen the paper.
2. The presentation of the modified CD initialization is missing reference to related EBM works such as [1] that also use noise-initialized MCMC for both training and testing. It might be useful to perform an ablation study to investigate the effect of the proportion of data samples and noise initialized samples. It is possible that the data samples might not be needed at all.

[1] Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model
https://arxiv.org/pdf/1904.09770.pdf
clarity,_quality,_novelty_and_reproducibility: The novelty of this work is somewhat lacking because it uses an established energy function and established methods such as Langevin sampling and noise-initialized MCMC from the EBM literature. The quality of results are good for the RBM family but very limited compared to virtually all contemporary generative model families.
summary_of_the_review: The work shows improvements within the GRBM family, but this potential is known to be very restrictive and even in ideal circumstances will likely be unable to match a wide variety of contemporary generative models. Without a clear motivation for using the GRBM, I recommend against accepting.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces many new insights that ease the training of and push the capabilities of Gaussian-Bernoulli Restricted Boltzmann Machines (GRBMs). In particular, they propose a hybrid Gibbs-Langevin sampling algorithm for inference, as well as a modified CD algorithm using two Markov chains (paired with gradient clipping) that allows for training GRBMs without many of the empirical hacks necessary to stabilize training in practice. The authors demonstrate the effectiveness of their approach on synthetic and benchmark image datasets (e.g. MNIST, CelebA).
strength_and_weaknesses: Strengths:
- The theoretical insights in the paper were sound and important.
- This is also the first time that GRBMs can generate images unconditionally. Although the sample qualities can be improved, the fact that this could be done is impressive even with a small model architecture is impressive.

Weaknesses:
- Although I don’t expect the FIDs for samples generated from other datasets (e.g. CelebA) to be competitive, would the authors comment on the gap between GRBMs and existing (maybe simple) baselines? 
- It would have also been nice to have additional experimental details (e.g. does the VAE architecture match that of the GRBM?).

clarity,_quality,_novelty_and_reproducibility: - Quality: The paper was high quality and clear.
- Novelty: The techniques and theoretical insights provided in the paper were novel, and significantly advance the potential for the practical usefulness of GRBMs.
- Reproducibility: No code was attached with the submission.

summary_of_the_review: The paper provides novel and interesting theoretical insights improving the performance of GRBMs in practice, as well as several proof of concept experiments for how they can be leveraged for generative modeling. This paper will be of interest to the community, and I imagine this encouraging new follow-up works.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work aims to train Gausssian-Bermoulli restricted Boltzmann machines (GRBMs) for generative modeling. For inference, it proposes to use a hybrid sampling that combines Gibbs sampling steps with Langevin samplers for GRBMs instead of the vanilla Gibbs sampling such that the gradient information of the log density can be leveraged. For learning, a modified contrastive divergence is proposed that encourages the model to generate samples from noises. The proposed training framework is evaluated on both synthetic and real-world datasets: Gaussian Mixtures, MNIST, FashionMNIST, and CelebA.

strength_and_weaknesses: The main contribution of this work is the proposed Gibbs-Langevin sampling for GRBMs and the modified contrastive divergence that allows the GRBMs to generate images from noise and to be compared with other existing deep generative models. It is interesting to see GRBMS, models with relatively simple architectures, are capable of generating reasonable images. All the proposed methods are nicely explained and seem technically sound. Still, here are some concerns/confusions:
- I find the motivation on why using GRBMs for generative modeling and when they are preferable to other models is somewhat lacking. Even though the paper claims that an important motivation is that a GRBM can convert real-valued data to stochastic binary data, it is not something unique to GRBMs since a VAE with discrete latent space can do the same thing. In terms of empirical evaluations, the FID scores of GRBMs are still a lot worse than the existing generative models.
- In related work section, it says that the proposed learning algorithm allows to learn a much lower variances compared with other methods. I wonder why smaller learned variances are a good thing. Would they lead to a less diverse model distribution?
- In Section 3.1, paragraph Gibbs-Langevin sampling, the authors claim that it performs better than generic Gibbs due to the leverage of gradient information of log density. Similar comparison between Gibbs-Langevin and Langevin is missing. I wonder why in Table one it is much better than Langevin. Any intuition why.
- In experimental section, it is vague what does adjustment refer to. Do you mean gradient clipping, modified CD or something else? This greatly harms the readability of empirical results.
clarity,_quality,_novelty_and_reproducibility: This paper is overall well-written and nicely polished. It makes a reasonable attempt to provide sufficient background for the readers to understand the proposed methods. One minor issue is that the font size in fig 1 is too tiny to read.
summary_of_the_review: Overall, nice improvements on challenging GRBMs.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Strength:

---" The work improves on GRBM learning and introduces the first RBM method that can generate images from scratch. "
---" Sample quality is reasonable given the restrictions of the potential energy. "
--- "The theoretical insights in the paper were sound and important. "
--- "This is also the first time that GRBMs can generate images unconditionally. Although the sample qualities can be improved, the fact that this could be done is impressive even with a small model architecture is impressive."
--- some reviewers also acknowledge from their own experience that it's tough to make GRBM work. 

Weakness

---" I find the motivation on why using GRBMs for generative modeling and when they are preferable to other models is somewhat lacking. "
--- Below is AC's summary based on some discussions with the reviewer. One way to phrase the critical question is: how do we interpret the claim "the first time that GRBMs can generate images unconditionally"? Does GRBMs sound anywhere plausible for some future applications? We have to ask these questions because if one restricts to a sufficient small scope (in this case, only use GRBMs), then it's perhaps always possible to be the first. On the other hand, we also want to be open-minded enough to potential progress and realize that many breakthroughs didn't sound very promising at the early stage of the research. Unfortunately, I think the AC and the reviewers were generally not sufficiently convinced here. 
The authors bring up some nice hypothetical possibilities enabled by their work but didn't have any solid applications. It appears to the AC and the reviewers that it would be good to have some solid (maybe new) applications that cannot be achieved with other EBMs or generative models. 
--- there were other discussions on the (lack of) novelty of the work. However, it's not the major consideration for the decision. 

justification_for_why_not_higher_score: Weakness

---" I find the motivation on why using GRBMs for generative modeling and when they are preferable to other models is somewhat lacking. "
--- Below is AC's summary based on some discussions with the reviewer. One way to phrase the critical question is: how do we interpret the claim "the first time that GRBMs can generate images unconditionally"? Does GRBMs sound anywhere plausible for some future applications? We have to ask these questions because if one restricts to a sufficient small scope (in this case, only use GRBMs), then it's perhaps always possible to be the first. On the other hand, we also want to be open-minded enough to potential progress and realize that many breakthroughs didn't sound very promising at the early stage of the research. Unfortunately, I think the AC and the reviewers were generally not sufficiently convinced here. 
The authors bring up some nice hypothetical possibilities enabled by their work but didn't have any solid applications. It appears to the AC and the reviewers that it would be good to have some solid (maybe new) applications that cannot be achieved with other EBMs or generative models. 
--- there were other discussions on the (lack of) novelty of the work. However, it's not the major consideration for the decision. 

justification_for_why_not_lower_score: n/a
Decision: Reject