Gaussian-Bernoulli RBMs Without Tears
Authors: Renjie Liao
Simon Kornblith
Mengye Ren
David J. Fleet
Geoffrey Hinton
Venue: Submitted to ICLR 2023 (2023)
Keywords: Gaussian-Bernoulli Restricted Boltzmann Machines
Restricted Boltzmann Machines
Langevin Monte Carlo Sampling
Contrastive Divergence

Abstract: We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations. We propose a novel Gibbs-Langevin sampling algorithm that outperforms existing methods like Gibbs sampling.
We propose a modified contrastive divergence (CD) algorithm so that one can generate images with GRBMs starting from noise.
This enables direct comparison of GRBMs with deep generative models, improving evaluation protocols in the RBM literature. 
Moreover, we show that modified CD and gradient clipping are enough to robustly train GRBMs with large learning rates, thus removing the necessity of various tricks in the literature.
Experiments on Gaussian Mixtures, MNIST, FashionMNIST, and CelebA show GRBMs can generate good samples, despite their single-hidden-layer architecture. 

Under review as a conference paper at ICLR 2023
GAUSSIAN-BERNOULI RBMs WITHOUT TEARS
Anonymous authors Paper under double-blind review
ABSTRACT
We revisit the challenging problem of training GGassian-Bernouile restricted Boltzmann machines (GRBMs), introducing two innovations. We propose a novel Gibbs-Langevin sampling algorithm that outperforms existing methods like Gibbs sampling. We propose a modified contrastive divergence (CD) algorithm so that one can generate images with GRBMs starting from noise. This enables direct comparison of GRBMs with deep generative models, improving evaluation pro- tocols in the RBM literature. Moreover, we show that modified CD and gradient clipping are enough to robustly train GRBMs with large learning rates, thus re- moving the necessity of various tricks in the literature. Experiments on Gaussian Mixtures, MNIST, FashionMNIST, and CelebA show GRBMs can generate good samples, despite their single-hidden Madden-layer architecture.
INTRODUCTION
Restricted Boltzmann machines (RBMs) (Smolensky 1986 Freund & Haussler 1991] Hinton 2002) are generative energy-based models (EBMs) with stochastic binary units. A variant of Boltz- mann machines (Ackley et al. 1985), they have bipartite graphical structure that enables effi- cient probabilistic inference, and they can be stacked to form deep belief networks (DBNs) (Hinton) & Salakhutdinov/ (2006; [Bengio al. 2006 [Hinton et al., 2006) and deep Boltzmann machines (DBMs) (Salakhutdinov & Hinton] 2009) Cho et al.) 2013). Gaussian-Bernout RBMs (GRBMs) (Weling et 2004, Hinton & Salakhutdinoy 2006) extend RBMs to model continuous data by replacing the binary visible units of the RBM with Gaussian random variables.
GRBMs remain challenging to learn, however, despite many proposed modifications to the model or training algorithm. For instance, et (2007) add a regularization term to encourage sparsely activated binary hidden units. Krizhevsky et al. (2009) attribute the difficulties learning to high- frequency noise present in natural images. Factorized high-order terms were introduced in (Ranzato) & Hinton 2010 Ranzato et al., 2010) to allow GRBMs to explicitly learn the covariance structure among pixels. Nair & Hinton 2010) suggest that binary hidden units are problematic, and pro- posed model variants with real-valued hidden units. [Cho et al.] (201 Ta 2013) advocate the use of parallel tempering sampling (Earl & Deem 2005). adaptive learning rate, and enhanced gradient (Cho et al. 201Tb) to improve GRBM learning. |Melchirr et al. (2017) conclude that difficulties in GRBM training due to training algorithms rather than the model itself; they advocate the use of gradient clipping, specialized weight initialization, and contrastive divergence (CD) (Hinton 2002) rather than persistent CD (Tieleman] 2008). Tramel et al.] (2018) propose the truncated Gaussian visible units and employ the Sowulless-Anndesonn Palmer (TAP) mean-field approximation for infer- ence and learning. (Upadhya & Sastry (202T) propose a stochastic difference of convex functions programming (S-DCP) algorithm to replace CD in training GRBMs.
An important motivation for improving GRBM learning is so that can be used as a front-end to convert real-valued data to stochastic binary data. This would enable research on modelling real- valued data via DBMs/DBNs, which are more expressive due to their deep architectures. This class of models are of special interest: their learning algorithm involves only local computation, and thus they are more biologically plausible than EBMs trained using backprop. As GRBMs are perhaps the simplest hybrid (including both continuous and discrete random variables) EBMs, investigating the inference and learning algorithms of GRBMs would lay the foundation and inspire more future research deep hybrid EBMs, which are useful for many applications like generating (continuous) images and their (discrete) scene graphs. Finally, RBMs and GRBMs are actively studied in quantum computing and physics (Melko et al. 2019
Ajagekar & You. 2020) since they naturally fit the

---

Under review as a conference paper at ICLR 2023
problem formulation (e.g., Ising models) required by many quantum computing devices. Progress on RBMs/GRBMs could potentially benefit such interdisciplinary research.
To this end, we propose improved GRBM learning methods for image data. First, we propose hybrid Gibbs-Langevin sampling algorithm that outperforms predominant use of Gibbs sampling. To the best of our knowledge this is the first use of Langevin sampling for GRBM training (with or without Metropolis adjustment). Second, we propose a modified CD algorithm so that one can generate images with learned GRBMs starting from Gaussian noise. This enables fair and direct comparison of GRGGB BMS with deep generative models, something beyond the reach of existing GRBM learning methods. Third, We show that the modified CD with gradient clipping sufficient to train GRBMs, thus removing the need for heuristics that have been crucial for existing approaches. At last, we empirically show that GRBMs can generate good samples on Gaussian Mixtures, MNIST, MNshionMNiST, and CelebA, despite they have single hidden layer.
2 RELATED WORK
Learning the variances There are two variances to be estimated in GRBM modelling. One the intrinsic variance of the data, e.g., the variance of image intensities, which is fixed once the data observed. The other is the (extrinsic) variance parameter in GRBMs, which governs the level of additional Gaussian noises added to visible units. Learning the extrinsic variance is thus necessary for generating sharp and realistic images. But small variance parameters tend to cause the energy function and its gradient to have large values, thus making the stochastic gradient estimates returned by CD numerically unstable. Most existing methods fix the variance (e.g.. to one) to avoid this issue. Krizhevsky et al. 22099;; Cho et al. (201 la) consider learning the variance using a smaller learning rate than for other parameters, obtamingg much better reconstruction, thus supporting the importance of learning variances. However, many of the learned filters are still noisy and point-like. [Melchior let al. (2017) learn a shared variance across all visible units, yielding improved performance, espe- cially with large numbers of hidden units. In this work, we learn one variance parameter per visible unit and achieve much lower learned variances than existing methods, e.g., approximately le on MNIST.
Stochastic gradient estimation and learning rate Due to the intractable log partition function of GRBMs, one often estimates the gradients of the log likelihood w.r.t. parameters via Monte Carlo. Gibbs sampling is predominant in CD learning due to its simplicity, but it mixes slowly in practice. One can refer to (Decelle et al., 202T for a detailed study on the mixing time of CD for RBMs. This yields noisy gradient estimates which often cause training instabilities and prohibits using large learning rates. [ho et al. (201 la> explore parallel tempering with adaptive learning rates to obtain better reconstruction. Cho et al. (2013) propose enhanced gradients that are invariant to bit- bit-llipping in hidden units. Melchior et al. (2017) show that gradient clipping and special weight initialization support robust CD learning with large learning rates. We advocate Langevin MC to improve gradients, and validate that gradient clipping does enable training with large learning rates.
Model capacity Theis et al. (2011) show that GRBMs are outperformed even by simple mixture models in estimating likelihoods for image data. Wang et al. (2012); Melchior et (2017) demon- strate that GRBMs can be expressed as either product of experts or a constrained Gaussian mixture in the visible domain, hinting that GRBMs need more hidden units than the true number of compo- nents to fit additive mixture densities well. Krause et al. (2013): Gu et al. (2022) provide theoretical guarantees on GRBMs for universal approximation of mixtures and smooth densities. Although this shows that GRBMs are expressive, they do not lead directly to practical GRBM learning algorithms.
Model Evaluation Like many deep generative models, evaluating GRBMs is difficult, as the log likelihood is intractable. To date, GRBMs have been evaluated by visually inspecting reconstructed images, filters and hidden activation (i.e., features), and sampled images during CD training. Quanti- tative metrics include reconstruction errors, and error rates of post-hoc trained classifiers on learned features. However, these metrics do not necessarily indicate if GRBMs are good generative mod- els (Melchior et al., 2017). Unlike existing work, we sample from learned GRBMs, starting from Gaussian noise, enabling direct comparisons with other generative models, qualitatively (visually inspecting samples) and quantitatively (e.g., Frechet Inception distance (FID) (Heusel et al. 2017). Note that similar noise-initialization strategy has been studied in EBMs (Nijkamp al. 2019)
2

---

Under review as a conference paper at ICLR 2023
Algorithm Langevin Sampling for GRBMs
1: Input: v(0), step size crg, total step T, burn-in step 7, adjust step
2: For / 1....,T
3:
a; = CosineScheduler((t, T.0))
4:
v = (tt-1) - az 8v + 22n ~ D. Use marginal energy in Eq. (5)
@E(v(t-1)
5:
If <= or (t > 1y and ~ ~0)1 < A(v, v(t1)))
v(t)
6:
7: Else
8: v(0) vl(-1)
9: Return: (v(T+I:T)}
Di:j indexes consecutive samples from i-th to j-th
GAUSSIAN- BAASSANN-ERRNOULI RESTRICTED BOLTZMANN MACHINES
Gaussian- Bernoulli -aunsian-Bernoulli Restricted Boltzmann Machine (GRBM) (Welling et al. 2004 Krizhevsky ct al., 22009] Cho al., 201 la Melchior et al., (017) is Markov Random Field (MRF) with contin- uous stochastic visible units and binary stochastic hidden units. Denoting N visible units as v E R and M hidden units as h {0, J,M, the energy function associated with GRBM is defined to be
Eo(v, h)
v
V
Wh wh-b'h b
(1)
2
with weight matrix W E RNM bias b E RM, mean E RR,, and variance σ² E REY where, unless stated otherwise, x denotes element-wise division between vectors x and y, as is convention in the GRBM literature. We denote the set of learnable parameters as 0 { b, μt, o²). To ensure the variance remains non-negative during learning, we adopt a reparameterization, directly learning log 2² rather than 2² or σr. Finally, given the energy function, one can define the Boltzmann distribution, over visible and hidden states, as
1 po (v, h) = 7 exp (-Eo(v, h)) where Z
Σ exp (-Ea(v, h))
the normalization constant, which is intractable for even moderately large M.
The underlying graphical model, like an RBM, is bipartite graph with edges only connecting visible units hidden units. This entails conditional independence of the form p(v/h) II, p(v. ()) and p(hlv) = II p(hllv). One can also derive the following conditional distributions for GRBMs,
p(h, = 1|v) = Sigmoid (WT o +b)],
p(v[h) = N (v|Wh++ + μi diag(o"))
(4)
where N (v|Wh + μa, (iag(o²)) ) is the multivariate Gaussian distribution with mean Wh + μ., and the diagonal covariance matrix diag(o²). Here, Sigmoid 1/(1 exp(-x)) is applied to the vector x in an element-wise manner, and [:ls denotes the j-th element of the corresponding vector.
The marginal distribution over visible units is p(v) = exp( (ap((EE((v)) Ev(v)) where
Σσ(v) 2
11
Softplus (WT + +bb +b)'1
(5)
and Softplus(x) = log( (1 (1+exp(x)) is applied in an element-wint manner, and 1 is the all-one vector of size M. We call (v( the marginal energy to distinguish from the GRBM energy in Eq. (I. We leave the derivation to Appendix A.I As shown in Melchior et al. (2017), one can also rewrite the marginal distribution p( (v) as a constrained Gaussian mixture.
INFERENCE
When performing probabilistic inference, often chooses between variational inference (Hinton & Van Camp 1993 Jordan et al., 1999) and Markov chain Monte Carlo (MCMC) methods (Neal
T993
Andrieu al. 2003). We focus MCMC common variational methods have been less

---

Under review as a conference paper at ICLR 2023
effective with RBMs and GRBMs (Gabrié et al.) 2013; Takahashi & Yasuda 2016). From the generative modelling perspective, we wish to draw samples of visible units during inference. There two natural approaches to this: 1) sample from the joint distribution in Eq. 2 and discard the samples of hidden units, or 2) directly sample from the marginal distribution.
Gibbs sampling (Geman & Geman 1984) is perhaps the predominant approach, due to its simplicity. In the context of GRBMs, one altemntes between sampling hidden units given visible units, and sampling visible units given hidden units. This produces samples from the joint distribution in Eq.
2
The detailed Gibbs sampling algorithm is given in Appendix A.2
Langevin Sampling. Langevin Monte Carlo (Grenander & Miller 1994 Roberts & Tweedie 1996
Welling & Teh, 201I) is class of MCMC methods that generate samples from a distribution of continuous random variables by simulating Langevin dynamics. Since GRBMs are hybrid EBMs, iee, comprising continuous and discrete random variables, we have at least two ways to leverage Langevin sampling. One is to directly apply Langevin sampling to the marginal distribution of visible units in Eq. (). Suppose at time step - 1, we have sample V;-1 and want to draw new sample Vt. The proposal distribution corresponding to one-step Langevin dynamics is given by
((vIv(--))=N - ( ot 0b(v((-1))
20x 1
(6)
Ov
where the gradient the of marginal energy E. w.r.t. the visible units is given in Appendix A.3 we use the Metropolis-Hastingg algorithm to accept or reject proposed samples, the acceptance probability of proposal v,, given the previous state, Vi-1. is (see Appendix A.3 for derivation):
(-Ea(v
1
| - (1) + axe
aB(
12)
exp
A(v v((-1)) = min 1,
-Foov((-2)) -
1 do1 |()) ((- + ar,
1B(
exp
8v
Alg. ssowws the Metropolis-adjusted Langevin Algorithm (MALA) for the marginal GRBM. Com- pared to generic MALA, it also includes an extra hyperparameter, namely, the adjust step v, If 1) is set to 0, then we perform a Metropolis adjustment at every sampling step, as prescribed in the generic MALA. If 7) is set to K > 0, then we skip the Metropolis adjustment for the first K steps. The adjust step effectively controls a trade-off between sampling accuracy and computational efficiency. Since we do not hope to see Gaussian noise in our final- imal-sampled images (i.e., beyond the level of intrinsic noise in the observations), is beneficial to decay the noise level, as in score-based models (Song & Ermon 2019). For certain step-size-annealle guarantees on the convergence of Langevin sampling (Durmus & Moulines] 2019). For simplicity, use the cosine scheduler and find it works well in practice. More details about the scheduler are provided in Appendix A..3
schedules and energy functions, there are theoretical
Gibbs-Langevin Sampling. We also introduce new hybrid sampler for GRBMs (see Alg. 2 Like the Gibbs sampler, it alternates between sampling hidden units conditioned on visible units, and sampling visible units given the hidden units. Unlike generic Gibbs, which directly samples from the Gaussian p(v (+).,, instead use Langevin MC to sample the continuous visible units given the hidden state. The use of Langevin MC may seem unnecessary because the Gaussian con- ditional permits one-step sampling algorithm. The subtlety comes from the fact that the finite-step Langevin sampler explicitly depends on the initial sample. Specifically, the proposal distribution of one complete outer-loop step in Alg. 2 e.g., at iteration t 1, can be expressed
q(v, hlv 10 ((1)) ((llv) g(vlv (t- (t"))),
(8)
where ghllv) is given by Eq. 44, and (tv) (((1)) is the proposal distribution of a K-step Langevin sampler (i.e. from the inner loop). This proposal distribution explicitly depends on the initial visible sample, (t--)) from iteration t t-1 1. By contrast, the generic Gibbs sampler does not have such dependence, i.e.. ((v)) h(t (t-1)) = (())) This dependence allows us to construct a persistent Markov chain in the space of visible units. Moreover, the Langevin sampler
Herr sampling accuracy means the closeness hetween the underlying distribution of samples and the target distribution measured in, e.g., total variation Wasserstein distances.
4

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Gibbs-Langevin Sampling for GRBMs
1: Input: v(), h step size og, total step T, burn-in step 7, adjust step 7) Langevin step
2: Function Langevin(v(!)) h, ag. K);
3: For l = ..... K
4:
0k = CosineSchedulerr(s. K, K,0
5:
((k) v(k-1) ank
(E(v(k-1)
+ 22aEs ~ ~(0,(
6:
Return (KK)
7:
8:
For 1 = 1. I.....T
9:
Langevin(v((--)) 11(h),(())) h(t-)))). aap, K)
10:
h p(hlv)
11:
If <= or f>
and u ~(0, < Á ((v, (v(l-1), h((-11)))
12:
(()) =v,
13:
Else
14:
v(1), ((t) v(t-1) (t-1)) Return: {(v(1+1:)) ((++1)))
leverages the informative gradient log density whereas Gibbs sampler does not. We find that our new sampler performs significantly better than the vanilla Gibbs sampler in practice.
The Metropolis adjustment for these Gibbs-Lanneviin proposals is, however, somewhat more in- volved. Following Alg. [] with (()) ((t-1) and (KK v. by marginalizing out the intermediate
states on the Markov chain, we obtain the proposal
g(v(K) h hnt-"))=]
II k=1
af (0ap)aan h(=1)) dv ....dd.--)))
(9)
The integrand in Eq. 9 comprises K one-step Langevin updates, each of which is given by g(v ((v/v(R- h (1- = =N ( (( ar
@E(E(K-10 (h h((-1))) 200
(10)
Ov
for which the energy gradient is given in Appendix A.4 Although the multiple integral in Eq. appears intractable, one can use reparameterization derive the following analytical form,
(9)
# + (t-I)
Σ 2ar.3R ))
(( h (1
(o +
Σ
BEAU
diag
(11)
where Bx 11 (1 ))), Vk E tO...., 1} and 3K Based this result, one can show that the acceptance probability for the Metropolis adjustment A((v (0))
(v (t-11) h0 h(=-)))==
(ov ( (1) g(h g(h ((1
Ev h())
exp
V20
min
1, --ooy((tt),nh--- h((-1))
(12)
v(t) - vit -aapt--))a(((t-ee -a(n+wh(e--) (²) o(hil0)/(tt)
exp
V20
Brol and 0² = Dx B..Bi. We 5
where a(h- = 1|v) = [iggmoid (W
oz + bb]
leave derivations to Appendix A.4
3.2 LEARNING
To learn GRBMs, we maximize the log likelihood of the observed data using stochastic gradient- based methods, e.g., contrastive divergence (CD). Depending on whether we use the joint (Eq. or the marginal (Eq. (5) distribution, we have two possible gradient estimators.
Learning with the Joint Distribution. When optimizing the GRBM with the joint distribution, one can express the general form the gradient of the log likelihood w.r.t. parameters
2 ( h)
eE h)
V0
(13)
09
80
5

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Modified CD Learning Algorithm for GRBMs with Joint Density
1: Input: CD-step K, burn-in step M, learning Rate 1) Langevin step size og. SGD step T
2: For 1,... T
3: vt -Vuuta
4: ht -(hly+)
5: V [
Compute Positive Gradient
86
6: VO ~ NN 1), hn ~ v(llvv)
D- Modified CD to start with noise
7:
(VM hārkk Sampler(Vo ,ū, a00)
D Alg. 5 or Alg.
o is current mean variance
8: VA = @E(v b
Compute Negative Gradient
09
9: 0 0= (v0 - V
D> Compute Update
Return 0
Here, following the notation in the RBM literature, we denote expectation under the data distribu- tion, i.e., po (h(v) Páma(v), as (-)a (po (v) [.]. Similarly, denote the expectation under the model distribution, po (v, h) as (+)m (xo ha) [-]. The expected gradients under the data and
model distributions are called positive and negative gradients respectively. Based on Eq. 13 we
can formulate the gradients of specific parameters follows.
VW
(14)
m
14
14
V/4
(15)
(V. )1)
Σj V; W hj
( - 14) -4:)
E,
2 V log 0
(16)
20
o
20
bb. (h:)d (h.)
(17)
Since the expectations in these gradients are generally intractable, we use Monte Carlo methods to approximate them. To sample from the joint density, we can use Gibbs or Gibbs-Langevin samplers as described in Sec. 3.1 The overall learning algorithm is outlined in Alg. An important detail is that we multiply the initial Langevin step size by the average variance at each gradient update step and then feed it to the sampler. Since the variance is decreasing (the energy function and its gradient are increasing) as learning goes on, keeping the step size roughly invariant to such scaling would make the sampling more effective.
Learning with the Marginal Distribution. Now we turn to learning the model under the marginal distribution in Eq. (5). Since have the marginal distribution of visible units, directly get the gradients of log likelihood w.r.t. model parameters 0 as,
(o((v) 00
F
V0-
(18)
00
Since the gradient dE((v 88
does not depend on h anymore, we have
(v)
FE 00
(og(v)
FFo(v)
= E prem (vmn (v)
Epo(v)
(19)
00
00
00
Based on above results, we can work out the detailed gradients which are the same those in Eq. 14 to Eq. (17) but with h replaced with Sigmoid (W -/2 + b). More details are left to Appendix
A.5
We use the Langevin sampler in Sec. 3.1 to sample from the marginal density to approximate the intractable expectation. The overall learning algorithm is outlined in Alg.
Modified Contrastive Divergence. The above two learning algorithms resemble CD if one ignores the specific sampler used. There exists subtle yet important difference however. For most deep generative models one generates samples starting from noise. But this does not work well for mod- els trained with CD, where sampling starts from observed data. This discrepancy of the starting sample between training and testing would be significant issue if the Markov chain does mix

---

Under review as a conference paper at ICLR 2023
Algorithm 4 Modified CD Learning Algorithm for GRBMs with Marginal Density
1: Input: CD-step K, burn-in step M, learning rate 7), Langevin step size cu. SGD step T
2:
For 1,... ,
3: vt Vduta
vo+ @Bivt)
4:
Compute Positive Gradient
30
5:
Vo
I)
> Modified CD to start with noise
6:
{v; j = M, ... .K} Sampler(va ,0o)
Alg. O is current mean variance
7: VO- = @Blv
Compute Negative Gradient
de
8:
0 = 0 v V
Compute Update
9:
Return
(a)
(b)
(C)
(d)
(c)
Figure 1: Density modelling using GRBMs data from a Gaussian mixtures with isotropic (rows and 2) and anisotropic variances (rows and 4). Rows and show normalized GMM densities and (unnormalized) negative energy values for GRBMs. Rows 2 and 4 show samples drawn un- der different models and methods; i.e., (a) Ground Truth; (b) Gibbs; (c) Langevin wo. Adjust; (d) Adjust; (f) Gibbs-Langevin Adjust.
Langevin Adjust: (e) Gibbs-Langevin
sufficiently quickly. We therefore modify CD by running two Markov chains to collect samples for positive and negative gradients respectively. The positive Markov chain is the same as in CD, i.e., starting from observed data. The negative Markov chain starts from sample of standard Nor- mal noise rather than the reconstructed data") Since the positive chain starting from data will usually stay close to the data distribution, this modification pushes the negative Markov chain, starting from noise, toward the data distribution. Moreover, the discrepancy between training and testing ceases to be important as we can start from standard Normal noise while sampling from the learned model.
4 EXPERIMENTS
We examine the empirical behavior of our new GRBM algorithms on benchmark image datasets, namely, MNIST, Fashion-MNIST (Xiao et al. 2017). and CelebA (Liu et al. 2015).
Implementation Details. We found that training with modified CD alone occasionally diverges, necessitating careful tuning of the learning rate. However, adding gradient clipping (e.g., clip gradi- ent norm to 10) enables stable training with all aforementioned sampling methods. We therefore set
TThe reconstructed data is typically obtained by running complete step of Gibbs sampler from the observed data, thus being highly likely close observed data.

---

learning rate to 0.01 for all experiments. Such large learning rate almost never works in the litera- ture. Melchior et al. (2017) used gradient clipping and similarly large learning rates, but they had to set the learning rate for the variances 100 times smaller than that for the weights and biases during Under review as a conference paper at ICLR 2023
CD training. But thanks to the modified CD and gradient clipping, we found this special treatment
of variances is unnecessary. We do not use momentum, weight decay, PCD, or other tricks.
4.1 MODELING GAUSSIAN MIXTURE DENSITIES
We first evaluate density modelling by GRBMs when the data density is known, i.e., Gaussian mixture models (GMMs) in our case. This is challenging for GRBMs as the marginal distribution of visible units GRBMs is essentially a constrained Gaussian mixture, i.e., the weights of mixture components depend on one another (Melchior et al. 2017 As such, the mixture components in GRBMs can not be freely placed in the visible domain so one actually needs more hidden units than the log of the number of mixture components to fit GMMs well. We consider the 2D case for simplicity and better visibility. We generate 1,000 samples from two types (isotropic and anisotropic variances) of GMMs with 3 components as shown in Fig. T and learn GRBMs using our modified CD with different sampling algorithms, from which we can draw samples. Here all samplers run for 100 steps during both CD training and testing (see Appendix [..] for more detail), Density plots and samples are shown in Fig. I case but fails the anisotropic case. Both Langevin and Gibbs-Langevin sampling collapse when the adjustment is absent. We believe the cosine step size schedule contributes to the collapse as it removes more stochasticity Langevin dynamics with small step sizes, thus making sampling more similar gradient descent. But there are more modes so that the sampling may collapse to different modes, and the diversity of images remains acceptable. Finally, both Langevin and Gibbs- Lange--n do recover all three modes with the adjustment, which shows the adjustment helps the mixing in this synthetic case.
Notice that Gibbs manages to recover the three modes in the isotropic
will see later, in image modelling, this may not be so severe;
4.2 IMAGE GENERATION
We learn GRBMs to fit image datasets including MNIST, FashionMNIST, and CelebA. To the best of our knowledge, this the first time that GRBMs have been shown to (unconditionally) generate good images. We provide the ablation study in Appendix B.2 and more results in Appendix B.3
Methods
FID
VAE
16.13
4
2sVAE Dai & Wipf 2019
12.60
PixeICNN++ (Slimans el al.)
11.38
WGAN (Arjovsky el al.
2017
10.28
NVAE (ahddtt & Kautz
2020
7.93
GRBMs
Gibbs
47.53
Langevin wo. Adjust
43.80
Langevin w. Adjust
41.24
Gibbs-Langevin wo. Adjust
17.49
Gibbs-Langevin w. Adjust
19.27
Intermediate samples from Gibbs- Langevin sampling.
Figure 2:
Table 1: Results on MNIST dataset.
MNIST. We train GRBMs with hidden size 4096 and 100 sampling steps on MNIST. We com- pare FID scores of GRBMs with other deep generative models in Table [[] We can see that Gibbs- Langevin family works significantly better than the Langevin family. The Metropolis adjustment ("w. Adjust" in Table[]p improves Langevin slightly but degrades Gibbs-Langevis slightly, which is different from what we observed on synthetic data. This is likely because the image distribution is so complicated (e.g., having significantly more modes) that the adjustment rejects proposed moves more frequently than before. Some sophisticated strategy may be needed to increase the acceptance probability. Nevertheless, GRBMs trained with Ganbs-Langevin without adjustment achieve FID scores comparable to other generative models, which is impressive given the manglehhidden-lycee architecture. The learning curve of (natural) log variance is shown in Fig. 3al The learned variance
8

---

Under review as a conference paper at ICLR 2023
2
4
4
0
6
2 / I 2 I 9 7
/
9
6
8
0
8
4
0
7
8
L"
0
7
Lng Maraace
0
5
5
O
Q
9
0
2
6
4
3 O
/
2
7
9
O
-30
9
O
4
3
7
6
500
3000
1500 Epoch
2030
2520
3300
6
2
8
9
5
(a)
(b)
(c)
Figure 3: (a) Learning curve of (natural) log variances, (b) learned filters, and (c) samples on MNIST.
(b)
(C)
Figure 4: Samples from GRBMs on (a) FashionMNIST, (b) -32, and (c) CK 64.
converges to around 1e-5 which is significantly smaller than those reported in the literature. The learned filters are shown in Fig. 3b Although some point-like filters still exist, stroke-like filters common, thus indicating GRBMs indeed learn meaningful features. We show samples drawn from the best GRBM in Fig. 3d The intermediate samples from Ginge-Langevin are shown in Fig. Since Gibbs-Langevin without adjustment works the best, we use it for remaining experiments.
FNSSi We then train GRBMs on FahsionMNIST which more challenging than MNIST. We set hidden size to 10,000 and the sampling step to 100. Samples drawn from learned GRBMs are shown in Fig. 4a GRBMs successfully learn the shapes of clothes, shoes, bags, and so on. However, they fail to capture fine textures. Since many images in this dataset look similar in shape but differ texture, the resulting samples look similar to each other.
CelebA. Last, we consider the even more challenging CelebA dataset. In particular, we explore two versions of this dataset: 1) CelebA-32 where we center-crop (140 x 140) and downsample images to 32 x 32; 2) SclebA-2K-66 where randomly select 2,000 images from the original CclebA and apply the same center crop and downsampling to 64 x 64. We set hidden size to 10,000 and explore the number of 100 and 200 sampling steps. Generated samples are shown in Fig. 4b and 4c From the figure, we can see that GRBMs can learn to generate reasonably good face images.
5 CONCLUSION
In this paper, we revisit learning Gaussian- Beriiullli restricted Boltzmann machines. We investigate Langevin Monte Carlo and propose novel Gibbs-Langevin sampling method. Furthermore, modify the contrastive divergence (CD) algorithm so that one can sample data from learned GRBMs starting from noise. Modified CD along with gradient clipping enables robust training of GRBMs with large learning rates. Finally, we show that GRBMs can unconditionally generate images with good qualities, despite its single-hidden layer architecture. In the future, would be beneficial to extend the current GRBMs to convolutional GRBMs which should be able to learn better localized filters. Meanwhile, would be interesting to explore Gaussian deep belief networks (GDBNs), which are deeper than GRBMs and should be superior. At last, investigating our Gibbs-Langevin sampling for hybrid deep energy based models could be a fruitful direction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for boltzmann machines. Cognitive science, 91):1477-169 1985.
Akshay Ajagekar and Fengqi You. Quantum computing assisted deep learning for fault detection and diagnosis in industrial process systems. Computers & Chemical Engineering, 143:107119, 2020.
Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. An introduction to meme for machine learning. Machine Learning. 50( 50(1):5-43, 2003.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214 123. PMLR, 2017.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer- wise training of deep networks. Advances in neural Information Processing Systems, 19, 2006.
Kyung Hyun Cho, Tapani Raiko, and Alexander Ilin. Gaussian-bermoull deep boltzmann machine. In The 2013 International Joint Conference on Neural Networks (IJCNN) pp. 1-7. IEEE, 2013.
KyungHyun Cho, Alexander Ilin, and Tapani Raiko. Improved learning of gaussian-berouuli re- stricted boltzmann machines. In International Conference on artificial Neural networks, pp. 10- 17. Springer, 2011a.
KyungHyun Cho, Tapani Raiko, and Alexander Ilin. Enhanced gradient and adaptive learning rate for training restricted boltzmann machines. In ICML, 2011b.
Bin Dai and David Wipf. Diagnosing and enhancing vae models. arXiv preprint arXiv: arXiv: 1903.0579, 2019.
Aurélien Decelle, Cyril Furtlehner, and Beatriz Seoane. Equilibrium and non-equilibrium regimes in the learning of restricted boltzmann machines. Advances in Neural Information Processing Systems, 34:5345-5359, 2021.
Alain Durmus and Eric Moulines. High-dimensional bayesian inference via the unadjusted langevin algorithm. Bernoulli, 25(44)):854-2882, 2019.
David Earl and Michael W Deem. Parallel tempering: Theory, applications, and new perspectives.
Physical Chemistry Chemical Physics, 7(23):3910-3916, 2005.
Yoav Freund and David Haussler. Unsupervised learning of distributions on binary vectors using two layer networks, Advances in Neural Information Processing Systems, 4, 1991.
Marylou Gabrié, Eric W Tramel, and Florent Krzakala. Training restricted boltzmann machine via the booules-andderson-paliine free energy. Advances in Neural Information Processing Systems, 28, 2015.
Stuart Geman and Donald Geman. Stochastic relaxation, gibbs distributions, and the bayesian restoration of images. IEEE Transactions Pattern Analysis and Machine Intelligence, (6); 721-741, 1984.
Ulf Grenander and Michael Miller. Representations of knowledge in complex systems. Journal of the Royal Statistical Society: Series (Methodological), 56(4):549-5 581, 1994.
Linyan Gu, Lihua Yang, and Feng Zhou. Approximation properties of gaussian-binary restricted boltzmann machines and gaussian-binary deep belief networks. Neural Networks, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in Neural Information Processing Systems, 30, 2017.
Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 1771 2002.
10

---

Under review as a conference paper at ICLR 2023
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-500 507, 2006.
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the de- scription length of the weights. In Proceedings of the sixth annual Conference on Computational Learning theory, pp. 5-13, 1993.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. fast learning algorithm for deep belief nets. Neural Computation, 18:1 18:1527-1554, 2006.
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine Learning, 37(2):183-233, 1999.
Oswin Krause, Asja Fischer, Tobias Glasmachers, and Christian lgel. Approximation properties of dbns with binary hidden units and real-valued visible units. In International Conference on Machine Learning, pp. 419-426. PMLR, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Honglak Lee, Chaitanya Ekanadham, and Andrew Ng. Sparse deep belief net model for visual area v2. Advances in Neural Information Processing Systems, 20, 2007.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Jan Melchior, Nan Wang, and Laurenz Wiskott. Gaussian-binary restricted boltzmann machines for modeling natural image statistics. PloS one, 12(2):e0110015, 2017.
Roger G Melko, Giuseppe Carleo, Juan Carrasquilla, and J Ignacio Cirac. Restricted boltzmann machines in quantum physics. Nature Physics, 1599)8887-82, 2019.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In leml, 2010.
Radford M Neal. Probabilistic inference using Markov chain Monte Carlo methods. Department of Computer Science, University of Toronto Toronto, ON, Canada, 1993.
Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non- persistent short-run meme toward energy- based model. Advances Neural Information Process- ing Systems, 32, 2019.
Mare' Aurelio Ranzato and Geoffrey E Hinton. Modeling pixel means and covariances using factor- ized third-order boltzmann machines. In 2010 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2551-2558. IEFE, 2010.
Mare' Aurelio Ranzato, Alex Krizhevsky, and Geoffrey Hinton. Factored 3-way restricted boltzmann machines for modeling natural images. In Proceedings of the thirteenth International Conference on Artificial Intelligence and Statistics, pp. 621-628. JMLR Workshop and Conference Proceed- ings, 2010.
Gareth Roberts and Richard Tweedie. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, pp. 341-363, 1996.
Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial Intelligence and Statistics, pp. 448-455 PMLR, 2009.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: A pixelenn imple- mentation with discretized logistic mixture. ICLR.
P Smolensky. Information processing in dynamical systems: foundations of harmony theory. In Par- allel distributed processing: explorations in the microstructure of cognition, vol. 1: foundations, pp. 191-281. 1986.
11

---

Under review as a conference paper at ICLR 2023
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Chako Takahashi and Muneki Yasuda. Mean-field inference in gaussian restricted boltzmann ma- chine. Journal of the Physical Society of Japan, 85(3):034001, 2016.
Lucas Theis, Sebastian Gerwinn, Fabian Sinz, and Matthias Bethge. all likelihood, deep belief is not enough. The Journal of Machine Learning Research, 12:3071-3096, 2011.
Tijmen Ticleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th International Conference on Machine Learning, pp. 1064- 1071,2005
Eric W Tramel, Marylou Gabrié, Andre Manoel, Francesco Caltagirone, and Florent Krzakala. De- terministic and generalized framework for unsupervised learning with restricted boltzmann ma- chines. Physical Review X, 8(4):041006, 2018.
Vidyadhar Upadhya and PS Sastry. Learning gaussian-beree bernoulli rbms using difference of convex functions optimization. IEEE Transactions Neural Networks and Learning Systems, 2021.
Arash Vahdat and Jan Kautz. Nvae: deep hierarchical variational autoencoder. Advances in Neural Information Processing Systems, 33:19667-19679, 2020.
Nan Wang, Jan Melchior, and Laurenz Wiskott. An analysis of gaussian-binane restricted boltzmann machines for natural images. In ESANN, 2012.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Mearning (ICML-II), pp. 681- 688. Citeseer, 2011.
Max Welling, Michal Rosen-Zvi. and Geoffrey E Hinton. Exponential family harmoniums with application to information retrieval. Advances in Neural Information Processing Systems, 17, 2004.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion ing machine learning algorithms, 2017.
Fashion-mmist: mnist: a novel image dataset for benchmark-
12

---

Under review as a conference paper at ICLR 2023
A
DERIVATIONS
A.1 MARGINAL PROBABILITY DISTRIBUTION OF VISIBLE UNITS OF GRBMS
We derive the marginal distribution of visible units as follows,
p(v) = Σ h)
1
Σ exp(- Eo,y,, h))
Z
1
T
Σ
exp ((/²)
Wh + bTh)
exp
Z
2
σ
h
1
T
4
+ exp ((xx) T W) + b..) b)))
=
Z
exp
T
Softplus ((x²) W + tb,) +b)))
II exp
= Z
exp
1
V
4
T
V
4
exp Softplus (WT o² + b) T
= exp
Z
2
9
σ
T
(WT 0² +b) T 1).
= exp
Z
Softplus
(20)
o
A.2 GIBBS SAMPLING
Algorithm Gibbs Sampling for GRBMs
1: Input: number of steps T, burn-in step T
2: v(0) ~N T)
3: For == =1...., T
4:
((9) ~ p(h(v(t-1))
following Eq. following Eq.
5: v(t) p0l><(([h@h ((v
6: Return: {(v(9, h(t)) = T +1, 1,..., T3
Gibbs sampling (Geman& & Geman 1984) is perhaps the most popular approach due its simplicity. In the context of GRBMs, we can alternate hetween sampling hidden units given visible units and sampling visible units given hidden units. Alg. [ blocked Gibbs sampler; it samples all visible units (a block of random variables) at once given all hidden units (the other block) and vice versa. Given the conditional independence in the bipartite graphical model, this block Gibbs sampler is equivalent to univariate Gibbs sampler that updates one variable at time given the others follow- ing some schedule. In fact, any schedule comprising a sequence of hidden units followed by all visible units or vice versa would make the equivalence hold. In other words, it preserves the conver- gence of the original univariate Gibbs sampler and runs as fast as blocked Gibbs sampler. Relying on this Gibbs sampler, Eq. We can then discard the samples within the burn-in stage and treat remaining ones as the final set of samples.
get samples of visible and hidden units from the joint distribution in
A.3 LANGEVIN SAMPLING
The gradient of the marginal energy w.r.t. visible units is
0
W Sigmoid (WT T 2x2 + b)
v
(21)
dv o²
13

---

Under review as a conference paper at ICLR 2023
The cosine scheduler for annealing the step size is, CosineScheduler((:, K, ao) = 200 1 + cos
k
(22)
K
where og is the k-th step size, or is the initial step size, and K is the total number of steps.
The derivation of the Metropolis adjustment for Langevin sampling is follows,
p(v(*) (0((((1))/(()))
A(v(") v(t-1))- min
1, p(v(((11))) ((((0(/(())/)(
exp
Ed((((t))
exp
1 (t 1) v(0) at ab(v(n)
8N
min
1,
(- -bo((((11))
1
v(0) v(l-1) + ate BF( (it
exp
exp
4ax
r((-1) v(t) + at 2 )
(o(((())
1
exp
Is.
min
-Eo((((-1))- - 4n,
1
v(0) v(t-1) 1) ar Bb(vit-13)
cxp
dv
(23)
A.4
SAMPLING
We now derive the Metropolis Adjustment for Gibbs-Langevin sampling. At time step t-1, the proposal distribution in Alg. 2 is
<(v, (t-1) D, (1(-1)) = g(hll))(v\v(( (( (((-))),
(24)
where
<(hlv) = Sigmoid (WT (3/) +b)
(25)
Denoting v(l-1) = ((0) and ((K), we have,
q(v/v((-1), (i(l-)))(((11)) = (//11)//(10) h(1-1))
K
= = -..../
II ((( 11i1 (.(l)...dv(kk. dv(k-1)
(26)
where
0E(v(k-1),h (t 2001
((v/b(1-1) (c(1)) =N (v/010-1) az
(27)
Ov
σg h) v Whh
(28)
Ov
2
The key question here is how to derive the analytical form of g(((K)|)(U) (1 (((())), The most straightforward way is to compute the multiple integral directly. By fixing all variables except for (A) in Eq. (8) we can integrate out ((A) analytically via the Gaussian integral trick, i.e., 1 exp(-ax² + bar + c)dr / exp( 6 one can ideally integrate out all ( ...., ((K-1) in an analytical manner. However, this process is quite involved due to the fact that the integral of (() depends on both ((++1) and (kk-1)
+ c). Then by applying the same trick recursively,
We instead resort the reparameterization trick. In particular, at the outer loop step f, the k-th inner loop step of Langevin sampling is as follows,
(P(v(t-1,,n(C-))
((6)_ = (k) ax
+ 22
dv 1(k-1) 1 (1-
-((-11) as
/20nEx
r²
1 + (1-1)
= =(1- ai
5(K-1)
+
+ /20
(29)
o²
14

---

Under review as a conference paper at ICLR 2023
where vk E {1....,K), K). E NN I). This discretization of Langevin dynamics gives sample
path of the distribution q(v(k) (7()) D)= We now show that this sample path could be repa- rameterized as simpler one which gives the desirable analytical form of q( q(v(k))v(0) h (t))). To simplify the derivation, we introduce Bx [[a==+1 (1 )x). Ak E {o...., 1} and 3x 1. Therefore, after K steps, we have,
#+ Wh(t-1)
v(k)
(1
0K σ2
v(K-1)
tak
+ V20KEK
σ²
II (1- (1- (1-2g)
Wh (-1)
1
02l
[0))
+
ax
σ²
20
km
Jmk+1
/ + (h(t-1)
Bbv(0)
Σ
BRaE
Bav 2a
(30)
Here {Exlk = 1,.... }} are independent random variables from the standard Normal distribution. Since we know that the linear combination of several independent Gaussian random variables leads to another Gaussian random variable, we have
q (((())):0) (t) = N (B0
Σ BeaR
# + Wh
2axB3
(31)
We can compute the acceptance probability,
A((v(() h ((),, (v((-1), h((-1))
((v((), h(9) (g(( h(t ((t)) h())
-min
1, p(v(t-1), h () h(0) (
w10 (ooh (o
1, p(v(1), h )g
min
p(v(t-1) 1) h ))
h([-)))
EE((v() h()))
w v(t-1) Bor!
al
gill (h1c-2)) ((t-11)
exp
v20
min
1,
(,((v(t-1,,h(t-)
viti Bo aau+ a(u+Wh(t-D))
((h(t) v(())
exp
V20
(32)
where q(h, 1|v) = (Sigmoid (WT ax + bb] a= 2. BLoc and Ø² = E 20
A.5 LEARNING
We derive the detailed gradients of the marginalized log likelihood of visible units w.r.t. model parameters below.
WW
35
V:
Sigmoid
W o²
+b)1,)
v: Sigmoid (WT o² + +b)l,)
(33)
V;
Vi Hi 2 o
V/
(34)
(V: 12
E, v. Sigmoid (WT 2/2 + +b]]
V log of-
20?
(v, 2
E3 Wi
Siggmoi (W N/2 b)],
(35)
20²
1a- Sigmoid (WT) 02 + +b)] m
Vb; Sigmoid
(WT o² +bb]]
[b)][1a-
>m
(36)
15

---

Under review as a conference paper at ICLR 2023
B MORE EXPERIMENTAL RESULTS
For all experiments, we set the initial variances of GRBMs to be 1, clip the gradient norm to be no larger than 10, and use SGD with neither momentum nor weight decay. We divide the total energy of mini-batch by the batch size so that we are minimizing the average negative log likelihood. We also decay the learning rate of SGD from the initial value 0.01 to using the same cosine scheduler as described in Eq. 122 The burn-in step in CD learning is set to 0, ie.. we do not discard any samples from any Markov chains. For all experiments involving images, we standardize the input image by subtracting the pixel-wise mean and dividing by the pixel-wise standard deviation. For color images, the subtraction and division is performed channel-wise. The details of all baselines on MNIST are follows.
VAE. We use encoder with 4 convolutional blocks (3 x 3 Conv+BN+ReLU along with a 2-layer MLP. The decoder contains a 2-layer MLP followed by 4 convolutional block (3 x 3 Cony+BN+ReLU) and 1-hidden-layee -laer MLP. The total number of parameters is around 3.98M.
2s For the encoder, we use 4 convolutional blocks (3 x 3 Conv+BN+ReLU), For the decoder, we use 2 convolutional blocks (3 x Conv+BN+ReL.U) followed by another convolutional blocks (3 x 3 Conv SonnTranspose-e Transpose+BN+ReL.U) BN+ReL.U). The total number of parameters is around 13.58M.
PixeICNN++, We have 8 masked convolutional blocks (3 Conv+BN+ReLU) which result in 1.41M parameters.
WGAN. For the discriminator, we have 4 convolutional blocks (3 x 3 Conv+BN+ReLU), For the generator, we have convolutional blocks (3 x Con/Transpoee+B-RRLL..) The total number of parameters is around 1.73M.
NVAE. We have 44 ResBlock and 42 ResBlock for the encoder and decoder respectively, which results in around 33.36M parameters.
Our GRBMs instead have a single hidden layer and about 3.21M parameters.
B.1 GAUSSIAN MIXTURE DENSITIES
The batch size and the hidden size are set to 100 and 256 respectively. We adjust at every step whenever Metropolis adjustmentment is used as the experiments with Gaussian mixture densities are fast. Although smaller hidden size could work, but we found this size makes learning converges stably for all sampling algorithms. To ensure a fair comparison, we train all GRBMs for 50K epochs and use the last model to draw the density plots and samples, despite the learning processes with most of inference algorithms converge within 5K to 10K epochs.
B.2 ABLATION STUDY ON MNIST
In this part, we perform ablation study on MNIST to investigate the effect of several important factors. In all experiments on MNIST, we set batch size to 512 and the number of epochs to 3000. First, we vary the CD step and the hidden size while fixing the other hyperparameters. The results are shown in Table We found that 4096 hidden size and 100 CD steps work the best on MNIST. More CD steps would potentially be better but take longer time train. Then turn to study the number of Langevin sampling steps, the adjust step size, the initial Langevin step size, and its annealing. Here annealing means we decay the initial Langevin step size 0 following the cosine scheduler as training goes on. The results are shown in Table 3 We can see that the larger the initial Langevin step size, the better the performance. But values larger than 0.04 would sometimes make the sampling numerically fail. The more the Langevin steps, the better the performance would be. Again, comes with more computational cost with more Langevin steps. We also find that may not be necessary to adjust at every step and annealing the initial step size slightly improves the performance of Gbbis-Langevin with adjustment. We further study the effect of standard Normal noise vs. data in the initialization of negative Markov chains in CD. In particular, given a mini-batch of data, we randomly draw a portion of them so that their negative Markov chains start with samples drawn from the standard Normal distribution, same as what we did in modified CD. But for the
16

---

Under review as a conference paper at ICLR 2023
remaining portion of data, start their negative Markov chains from the data, same what did for the positive Markov chains. This can help us understand the importance of noises in the initialization of modified CD. The results decreases with more data being used as the initialization. This is expected since the discrepancy between the initial distributions of the negative Markov chain during training and the Markov chain during testing is increasing. Note that if we use data as initialization for all samples, then the underlying learning method reduces to the original CD.
shown in Table 4 It is clear that the sample quality
B.3 MORE VISUAL. RESULTS
We train 3K epochs for experiments both FashionMNIST and CelebA-32 datasets. For CeleA- 2K-64, we train 4K epochs. The batch size on FashionMNIST and CelebA-32 is 512 whereas the batch size CeleA-2K-664 is 100.
We show the samples drawn from the best GRBMs learned with different sampling methods in Fig. 5 It is clear that samples corresponding to Gibbs-Langevin have better visual qualities than those from Langevin and Gibbs. We also show more results of GRBMs learned with Gings-Langevin in Fig.
Fig.
Fig. 8
and Fig.
Methods
CD Step
Hidden Size
FID
Gibbs-Langevin wo. Adjust
2048
32.33
Gibbs-Langevin wo. Adjust
50
4096
21.02
Gibbs-Langevin wo. Adjust
8192
22.05
Gibbs-Langevin Adjust
100
4096
17.49
Table 2: Ablation study of the hidden size and the number of CD steps on MNIST dataset.
Langevin Step K
Langevin Step Size
Adjust Step
Methods
Anneal on
FID
Gibbs-Langevin wo. Adjust
20
X
35.08
Gibbs-Langevin wo. Adjust
5
20
X
19.00
Gibbs-Langevin wo. Adjust
10
20
X
17.49
Gibbs-Langevin wo. Adjust
X
21.05
Gibbs-Langevin wo. Adjust
10
5
X
25.67
Gibbs-Langevin Adjust
20
X
21.31
Gibbs-Langevin Adjust
10
20
X
25
21.25
Gibbs-Langevin w. Adjust
20
X
50
20.64
Gibbs-Langevin Adjust
10
20
V
50
19.27
Table 3: Ablation study of the number Langevin steps K, the initial Langevin step size Okps annealing of the initial Langevin step size, and the Metropolis adjust step v on MNIST dataset. All runs use 100 CD steps.
Data (%)
Noise (%)
FID
0
100
17.49
25
75
19.02
50
50
21.03
75
25
53.20
Table 4: Ablation study of the initialization negative Markov chains.
17

---

Under review as a conference paper at ICLR 2023
7
9
4
2
2
7
4
5
6
2
4
4
4
1
2
2
4
2
4 /
4
/
0
3
0
5
0
0
5
6
9
7
8
8
9
0
8
1
5
3
3
7
8
3
7
3
6
6
4
6
8
2
5
4
0
7
/
0
3
2
7
0
0
?
6
/
9
3
9
5
4
4
O
3
(a) Gibbs
(b) Langevin
(c)
Langevin adjust
(d) Gibbs-Langevin
(e) Gibbs- Gibbs-Langevin w. adjust
Figure 5: Samples from GRBMs learned with different sampling algorithms on MNIST
8
7
2
0
5
9
5 0
7
6
8
6
3
0
3
7
7
5
7
9
0
9
4
2
4
4
7
2
2
2
4
7
3
9
8
4
I
3
7
4
7
0
/
_
8
7
G
5
9
6
/
/
6
9
8
6
3
4
5
3
6
8
3
0
S
4
5
O
4
2
8
7
7
3
7
4
6
2
9
9
7
O
7
6
O
5
7
9
0
5
9
3
7
6
7
2
0
S
4
5
7
2
9
7
6
/
9
/
3
3
9
5
/
0
7
7
8
9
7
7
8
7
4
0
Figure 6: More samples from the learned GRBM (Gibbs-Langevin) on MNIST.
Figure 7: More samples from the learned GRBM (Gibbs-Langevd) FaShionMNISST.
18

---

Under review as a conference paper at ICLR 2023
Figure More samples from the learned GRBM (Gibbs- Langevinn)) on CelebA-32.
Figure 9: More samples from the learned GRBM (Gibbs-Langevin) on CelebA-2K-64.
19


Reviews:
[{"review_id": "mSYtqDDNXVH", "reviewer": "Reviewer_Uikc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work revisits the GRBM to improve its learning and sampling process. One key innovation is introducing Langevin sampling to GRBMs. Two variants are proposed: sampling the visible marginal potential, and hybrid Gibbs-Langevin sampling that trades off between Gibbs updates of hidden units and Langevin updates of visible units. The other key innovation is initializing MCMC samples from noise so that the model can generate samples from noise once training is over. Experiments show that the learning method can successfully train GRBMs that can generate reasonable images from scratch. ", "summary_of_the_review": "The work shows improvements within the GRBM family, but this potential is known to be very restrictive and even in ideal circumstances will likely be unable to match a wide variety of contemporary generative models. Without a clear motivation for using the GRBM, I recommend against accepting.", "strength_and_weaknesses": "STRENGTHS:\n\n1. The work improves on GRBM learning and introduces the first RBM method that can generate images from scratch.\n2. Sample quality is reasonable given the restrictions of the potential energy.\n\nWEAKNESSES:\n\n1. The motivation of using GRBMs is unclear. From one perspective, the visible GRBM is simply a deep EBM with a restricted architecture. The hidden units distinguish the GRBM and RBM family, but the applications do not depend on the hidden representations. A clear motivation for the use of a GRBM would greatly strengthen the paper.\n2. The presentation of the modified CD initialization is missing reference to related EBM works such as [1] that also use noise-initialized MCMC for both training and testing. It might be useful to perform an ablation study to investigate the effect of the proportion of data samples and noise initialized samples. It is possible that the data samples might not be needed at all.\n\n[1] Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model\nhttps://arxiv.org/pdf/1904.09770.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cTMfSB7Wae8", "reviewer": "Reviewer_SJ4K", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces many new insights that ease the training of and push the capabilities of Gaussian-Bernoulli Restricted Boltzmann Machines (GRBMs). In particular, they propose a hybrid Gibbs-Langevin sampling algorithm for inference, as well as a modified CD algorithm using two Markov chains (paired with gradient clipping) that allows for training GRBMs without many of the empirical hacks necessary to stabilize training in practice. The authors demonstrate the effectiveness of their approach on synthetic and benchmark image datasets (e.g. MNIST, CelebA).", "summary_of_the_review": "The paper provides novel and interesting theoretical insights improving the performance of GRBMs in practice, as well as several proof of concept experiments for how they can be leveraged for generative modeling. This paper will be of interest to the community, and I imagine this encouraging new follow-up works.", "strength_and_weaknesses": "Strengths:\n- The theoretical insights in the paper were sound and important.\n- This is also the first time that GRBMs can generate images unconditionally. Although the sample qualities can be improved, the fact that this could be done is impressive even with a small model architecture is impressive.\n\nWeaknesses:\n- Although I don’t expect the FIDs for samples generated from other datasets (e.g. CelebA) to be competitive, would the authors comment on the gap between GRBMs and existing (maybe simple) baselines? \n- It would have also been nice to have additional experimental details (e.g. does the VAE architecture match that of the GRBM?).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RRVnsI_moD2", "reviewer": "Reviewer_sW5s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to train Gausssian-Bermoulli restricted Boltzmann machines (GRBMs) for generative modeling. For inference, it proposes to use a hybrid sampling that combines Gibbs sampling steps with Langevin samplers for GRBMs instead of the vanilla Gibbs sampling such that the gradient information of the log density can be leveraged. For learning, a modified contrastive divergence is proposed that encourages the model to generate samples from noises. The proposed training framework is evaluated on both synthetic and real-world datasets: Gaussian Mixtures, MNIST, FashionMNIST, and CelebA.\n", "summary_of_the_review": "Overall, nice improvements on challenging GRBMs.", "strength_and_weaknesses": "The main contribution of this work is the proposed Gibbs-Langevin sampling for GRBMs and the modified contrastive divergence that allows the GRBMs to generate images from noise and to be compared with other existing deep generative models. It is interesting to see GRBMS, models with relatively simple architectures, are capable of generating reasonable images. All the proposed methods are nicely explained and seem technically sound. Still, here are some concerns/confusions:\n- I find the motivation on why using GRBMs for generative modeling and when they are preferable to other models is somewhat lacking. Even though the paper claims that an important motivation is that a GRBM can convert real-valued data to stochastic binary data, it is not something unique to GRBMs since a VAE with discrete latent space can do the same thing. In terms of empirical evaluations, the FID scores of GRBMs are still a lot worse than the existing generative models.\n- In related work section, it says that the proposed learning algorithm allows to learn a much lower variances compared with other methods. I wonder why smaller learned variances are a good thing. Would they lead to a less diverse model distribution?\n- In Section 3.1, paragraph Gibbs-Langevin sampling, the authors claim that it performs better than generic Gibbs due to the leverage of gradient information of log density. Similar comparison between Gibbs-Langevin and Langevin is missing. I wonder why in Table one it is much better than Langevin. Any intuition why.\n- In experimental section, it is vague what does adjustment refer to. Do you mean gradient clipping, modified CD or something else? This greatly harms the readability of empirical results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject