Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
Authors: Aurélien Decelle
Cyril Furtlehner
Beatriz Seoane
Venue: NeurIPS 2021 Poster (2021)
Keywords: Restricted Boltzmann Machine
Out of Equilibrium
Monte Carlo mixing time
Unsupervised learning
Energy Based Model
Disordered Systems

Abstract: Training Restricted Boltzmann Machines (RBMs) has been challenging for a long time due to the difficulty of computing precisely the log-likelihood gradient. Over the past decades, many works have proposed more or less successful recipes but without studying systematically the crucial quantity of the problem: the mixing time i.e. the number of MCMC iterations needed to sample completely new configurations from a model. In this work, we show that this mixing time plays a crucial role in the behavior and stability of the trained model, and that RBMs operate in two well-defined distinct regimes, namely equilibrium and out-of-equilibrium, depending on the interplay between this mixing time of the model and the number of MCMC steps, $k$, used to approximate the gradient.  We further show empirically that this mixing time increases along the learning, which often implies a transition from one regime to another as soon as $k$ becomes smaller than this time.
In particular, we show that using the popular $k$ (persistent) contrastive divergence approaches, with $k$ small, the dynamics of the fitted model are extremely slow and often dominated by strong out-of-equilibrium effects. On the contrary, RBMs trained in equilibrium display much faster dynamics, and a smooth convergence to dataset-like configurations during the sampling.
Finally, we discuss how to exploit in practice both regimes depending on the task one aims to fulfill: (i) short $k$s can be used to generate convincing samples in short learning times, (ii) large $k$ (or increasingly large) must be used to learn the correct equilibrium distribution of the RBM. Finally, the existence of these two operational regimes seems to be a general property of energy based models trained via likelihood maximization.

Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
Aurélien Decelle 12.
Cyril Furtlehner
Beatriz Scoane
'Departamento de Física Téorica, Universidad Complutense, 28040 Madrid, Spain. Université Paris-Saclay, CNRS, INRIA Tau team, LISN, 91190, Gif-sur- Yvette, France. aadecelle@ucm.e.. "bseoane@ucm.es
Abstract
Training Restricted Boltzmann Machines (RBMs) has been challenging for long time due to the difficulty of computing precisely the log-likelihood gradient. Over the past decades, many works have proposed recipes but without studying the crucial quantity of the problem: the mixing time, i.e. the number of Monte Carlo iterations needed sample from a model. In this work, we show that this mixing time plays a crucial role in the dynamics and stability the trained model, and that RBMs operate in well-defined regimes, namely equilibrium and out-of-equilibrimm, depending on the interplay between this mixing time of the model and the number of steps, used to approximate the gradient. We further show empirically that this mixing time increases with the learning, which often implies transition from one regime to another as soon as k becomes smaller than this time. In particular, we show that using the popular (persistent) contrastive divergence approaches, with l small, the dynamics of the learned model are extremely slow and often dominated by strong -utuo--qqullirrium effects. On the contrary, RBMs trained equilibrium display faster dynamics, and smooth convergence dataset-like configurations during the sampling. Finally we discuss how to exploit practice both regimes depending the task one aims to fulfill: (i) short & can be used to generate convincing samples in short learning times, (ii) large & (or increasingly large) is needed to learn the correct equilibrium distribution of the RBM. Finally, the exis- tence of these two operational regimes seems be general property of energy based models trained via likelihood maximization.
less successful training
configurations
Introduction
Restricted Boltzmann Machines (RBM) are one of the generative stochastic neural networks that have been available for decades [T] and are renowned for their capacity to learn complex datasets 2 and draw similar new data. Furthermore, the hidden correlations found in the data are easily ac- cessible
5. which particularly interesting for scientific use of Big Data. Despite these positive features, RBMs still remain hard to train and evaluate, and hence to use in practical prob- lems. In comparison, Generative Adversarial Network (GAN) 6 (for which is much harder to find a working architecture) are more commonly used, not only because of the advantages given by convolutional layers, but also because the generative process does not depend on costly sampling procedure with an uncontrolled convergence time.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Both training and data generation of RBMs are tricky and unpredictable in inexperienced hands. One of the main difficulties is that hard evaluate the learning is progressing or not. Yet, many works in the past twenty years have proposed recipes for good practices in RBM training [ Unfortunately, the evaluation of these recipes relies only on the comparison of the values reached by the log-likelihood (LL.) [8] 7 9 a quantity that cannot be monitored in tractable times during the learning process. Studies properly characterizing the quality, independence, or stability of the generated samples using the different recipes are nearly absent in the Literature. In fact, many works just show a set of new samples (for eye evaluation) that were either obtained after a short Markov Chain Monte Carlo (MCMC) sampling initialized at the dataset, or borrowed from the permanent chain (negative particle) at the end of the training [10]. In other studies, a reconstruction error after one several MCMC steps is used to determined if the machine
correctly trained [[I]. However, none of these measures guarantees that the trained model can generate from scratch new samples nor that the dataset are typical samples of that model. At most these tests assure you that samples of the dataset are locally stable w.r.t few MCMC steps, unless proper decorrelation from the initial conditions or a link equilibrium properties are investigated. Finally, more recent works use a classification score on the activation of the latent variables [12. 13], which reflects that the relevant features were learned, but silent about the generative power of the machine.
RBMs are receiving an increasing attention in life and pure sciences during the recent years 5 14 13 [6. 17
19 20 21 given their potential for interpretability of the pattems learned by the machine (a rare feature in machine learning). Indeed, the RBM formalism enables the extraction of the (unknown) probability distribution function of the dataset. Yet, such an approach is meaningful if the dataset is related to the equilibrium properties of the trained model. Given these new and exciting prospectives for the RBMs, it is more than ever important to establish reproducible protocols and evaluation tools to guarantee not only that the model generates good enough data, but also that the dataset are typical equilibrium samples of the model.
18,
In this work we demonstrate that the classical training procedures can lead to two distinct regimes of the RBM: an equilibrium and an out-of-qquilbbrium (OOE) one. A finding that had been recently discussed in the Literature under the name of convergent or non-convergent regimes in a different family of generative energy based models (EBMs), those whose energy function is ConvNet (22]. Here, at variance to previous approaches, we show that the cquilibrium regime is observed the number of MCMC steps, k. used to estimate the LL gradient, is larger than the algorithm mixing time (MT). The latter emerges when falls below this MT. In particular, we show that for a classical dataset such as MNIST, the MT is always rather large and grows with the learning time. This implies that learning an "equilibrium" model (i.e. a RBM that operates in the equilibrium regime) is costly, and that most of the works in the Literature operate without any doubt in the OOE regime. In this regime, the equilibrium distribution of the machine at the end of the learning is significantly differ- ent from that of the dataset. Yet. despite this handicap, we show the OOE regime can be exploited to generate good samples at short training times, an observation that have already been exploited in practice [23] for convolution EBM. All our conclusions rely on the study of 9 datasets: MNIST (24), whose results are discussed in the main-tex: FashionMNIST (25), Caltech101 Silhouettes small- NORB dataset (26). human genome dataset (27 (with similar dimensions as MNIST but more structured), the high-quality CelebA (28) projected in black and white, and in low definition but in color, and CIFAR (29. The analysis of most of the last datasets are discussed only in the SM and show similar behavior to the one observed MNIST.
Our paper is organized follows. RBMs, their general principles and learning equations are defined in sec. [ We review previous related works in sec. 3 In sec. [ we define the different observables used monitor the training and sampling. Finally, we discuss experimental results in sec. 5
2 Definition of the model
An RBM is Markov random field with pairwise interactions defined on a bipartite graph of two non con-intrracting layers of variables: the visible nodes, v = Ug = 1,.., No., represent the data, while the hidden ones, h h = 1, N,, are the latent representations of this data and are there to build arbitrary dependencies between the visible units (see sketch in fig. [- Usually, the nodes are vinary-valued in (0. yet Gaussian or more arbitrary distributions on real-valued bounded support are also used, ultimately making RBMs adaptable for more heterogeneous datasets. Here, we deal only with binary (0, 1} variables for both the visible and hidden nodes. Other approaches
2

---

Gibbs sampling MCMC steps
A
Rdm10'
Log- pre-likklihood
-100
Rdm10
Learning
CD10
-150
Hidden layer
1ug
Generation sampling
Parameter updates f
-200
PCD10
Visible layer
MCMC steps
10a
10'
10"
10
10
10
fax: Iparmmtter updates]
Ar 10
D
MNIST dataset
E
CD10
PCD
Rdm10
334 3331
10
6-6 P 2 7
2
5
13 4
10
/ +
WK5R
4C65490089
1110314371
L 10
K.
8
15000005000
7
27254 1250
4
60:
386326499
326043294
1 55
/
6
10
6
3 3/3 0
00300
6 2 144040000 40/003
10
0
6
D
020002200000
Sketch of RBM in A and of the pipeline of our analysis in B. C Evolution of the LL of the dataset during the learning for different RBMs trained using either a very long MCMC sampling to estimate the gradient (k 10*. Rdm-10*) or short samplings & 10 following different recipes for the initialization of the MC chains 10D-10, PCD-10 10 and Rdm-10). The large symbols mark the LL of the RBMs used generate database, and E a subset of the samples drawn by the & = 10 RBMs marked in C through a MCMC sampling of the equilibrium Gibbs measure initialized random and after 10,10 1100.100..... .0 and 10 MCMC steps. Note that in our setup the MC chain are initialized at random, and we can observe that RBMs trained with CD-10 do converge any digit despite reaching higher value of the LL than Rdm 10 which instead manage to create digits quite fast. Also note the lack of diversity of Rdm 10 for large sampling time.
Figure 1:
samples in E. In D we show subset of the MNIST
hidden units 30), giving a ReLu type of activation functions for the hidden layer do work well, but in our experiments observed qualitatively the similar dynamical behavior and therefore we will stick to binary hidden units for the rest the article. The energy function of an RBM is taken
using
Elu, b,c. - -
-
-n ho - Σberr biu: - Ecahar
(1)
with w the weight matrix and b, c the visible and hidden biases, respectively. The Boltzmann distribution is then given by
1llu,h h w, b, cl) with Z
plu, h/w, b.
Σ (v.h)
Elv.k]
Z
being the partition function the system. RBMs are usually trained using gradient ascent of the function of the training dataset D {x(1), (o())..... .... (M)) being the LL defined
M
M
C(w, b, c(D) = M-1
Σ
Imp(v = o (m)) ww, b) M
Σ
Efu(m) b.w.b.e. In
(3)
m=11
(h)
The gradient is then composed of two terms: the first accounting for the interaction between the RBM's response and the training set, and same for the second, but using the samples drawn by the machine itself. The expression of the LL gradient w.r.t. all the parameters is given by
Øc = (wzhaa (viha)
dc
(vi) (De' and dc dc.
(ha) - (ha)H:
(4)
0b where (f(v, h)) = M-'E E(,) f(v(m), (1)((())))) denotes an average over the dataset, and (f H.) the average over the Boltzmann measure in Eq.
Swin

---

A large part of the literature on RBMs focuses on schemes to approximate the r.h.s. of Eqs. In theory, these terms can be computed up to an arbitrary precision using parallel MCMC simulations as long as the number of MC steps are large enough to ensure proper sampling of the equilib- rium configuration space. In other words, sampling times should be larger than the MCMC MT. A naive way to implement this idea is to initialize each of these parallel Markov chains on random conditions, perform & MCMC steps, and use the final configurations to estimate the gradient. We call this scheme Rdm-k. This scheme is not used in practice because it would require too many steps if the MT is too large. Many recipes have been proposed to shorten this sampling and ap- proximate the negative term of the gradient, but since the introduction of the so-called contrastive divergence (CD) by Hinton [1I, only a limited number of schemes are used in practice. The first one, CD-k, initializes the MCMC simulation from the same data of the mini-batch used to com- pute the gradient, and performs k ~ ((1) sampling steps. This approximation relies on the idea that the dataset must be a good approximation of the equilibrium samples of well trained RBM, something that is not true for a poorly trained machine, whose typical samples are quite distant from the dataset. In the second commonly used method, the last configurations of the Markov chains saved from one parameter update to the other, and then used to initialize the subsequent chain. As in the other methods, k MCMC steps are used estimate the gradient for each update. This method is known as persistent-CD: PCD-k (32), and takes ad advantage of the fact that the RBM's parameters change smoothly during the learning and the same is expected for the models' equilibrium distribu- tior Finally. other approximations, such as mean-fiel -ield or TAP equations [[0], or more elaborate approaches, such as the parallel tempering technique [33] 34), . can be used. common feature of all these recipes is that they rely on very few simulation steps to sample the Boltzmann distribution, and the trained model often generates rather bad configurations.
To illustrate this last comment, let us discuss a provocative experiment shown on fig. THE where we try to sample new digits just by sampling the equilibrium measure of our trained RBMs using a MCMC initialized uniformly at random. Let us stress that this is not the standard set-up used in the field to generate new samples, but it is strong test of the generation power of the trained model. We compare the outcome of the generated samples (as function of the sampling time) for three RBMs trained each following different recipes to compute the negative term of the gradient: CD-10, PCD- 10 and Rdm-10. 10. At this point, k- = 10 might seem a very small number. It is however the typical order of magnitude used in the Literature. The details of the learning protocol are discussed later in the text and summarized in the SM. The first striking point on fig. E, is the inability of the CD-10 RBM to properly sample digits in this set-up. This problem has been reported before 35 but since CD is part of the standard methods, we want to insist on this point. In addition, PCD- trained RBM generates proper digits but only after a very long sampling time, while with Rdm-10, quite surprisingly, digits are properly generated after only 10 MCMC steps. Yet, samples obtained after much longer MCMC steps have unbalanced digits representations, like the being over-represented. In fact, we observe that poorly trained models generate samples that do not respect the correct balance between the 10 digits present in the original database (see fig. D-D). We already mentioned the 1s in the Rdm-10 10 run, but it happens the same (to a lesser extent) with the O in the PCD-10 case. Alternatively to these generated samples, we show the evolution of the LLI during the learning of these same three RBMs in fig. D-C. By definition of the loss function, the LL intended to be maximized, so it constitutes a traditional measure of the quality of the training. The LL of the models used for the sampling of fig. E are highlighted in large symbols. This plot illustrates the lack of reliability of the LL to monitor the quality of the learning. For instance, CD-10 reaches quite high values of LL and yet, it cannot generate a single digit. For the Rdm-10 is the other way around, it generates decent samples but with a very poor LL. Finally, we observe that the LL of PCD-10 reaches very high values, comparable to RBMs trained using up to k= 10 MCMC steps to estimate the gradient. We will come back to this point. Our experiment highlights, first, that the LL cannot be used all alone to quantify the generative power of an RBM, and second, that the quality of the generated samples may variate in a non-monotonic way during the generation sampling.
'This adiabatic scheme is thus valid as long as the relaxation time for the RBM parameters during the learning is much larger than the MCMC mixing times.

---

3 Related works
Recent years have witnessed a large number of studies trying to understand and control the learning of RBMs. In these works, meta-parameters are varied at the same time that an approximation of the LL 88 36 of the train/test set is monitored throughout the learning process. The stability of the evolution of the LL and the maximum value achieved, are then used to compare the different recipes and parameters. For instance, in Ref. [8] the LL of different learning schemes are evaluated and the samples inspected visually. Unfortunately, no estimator is used to assess the quality of the samples, neither is clear the precise protocol used to generate these samples. In Ref. [9], the authors investigate the stability of the learning of RBMs with binary input by comparing the learned fea- tures. They show (using mostly the LL and an eye inspection of the learned features) that a centered version of the gradient can be much stabler. In Ref. [7], the authors study, in a systematic way, the convergence properties of CD, PCD and PT on several small toy models that can be analyzed exactly, that is, where the LL can be computed by brute-force. We can find more recent works 37 38 39] 40] improving the learning scheme for RBMs and yet, still not giving much information about the quality of the generated samples nor the equilibrium properties of the trained models. In our results below, will show that without putting on the table this information, the comparison between methods or tuning of parameters, becomes extremely unstable. Finally, we want to mention that it is particularly surprising that the CD recipe (with short k) is still used 41 42 43 39]. As shown, RBMs trained with CD are not able to generate proper samples from scratch. And this be- havior is true even if doing k = (100) MC steps at each update. The classical explanation for this is that CD is creating local energy well around the datapoint which are separated by large energy barriers 35 since the dynamics do not have time to relax when A is small.
In the rest of the paper, we explain how, by characterizing the MT, we are able to explain the capricious behavior of RBMs with respect to the different usual schemes. Furthermore, we will see that our results are totally consistent to what is observed in other families of EBMs such as ConvNet generative networks 23 p or Boltzmann Machines 44 45) when trained using short k negative chains, which suggests that this explanation applies to EBMs in general.
4 Monitoring the learning performance of RBMs
In this work we wish to address the following questions: (i) How can one quantify that an RBM has correctly learned a dataset? (ii) has it Iearned the equilibrium model? (iii) when should one stop the learning and the sampling of new configurations? We already argued in sec. 2 that, alone, the LII of the dataset is not precise enough to address these questions. For this reason, we have considered several additional observables to quantify the generated samples' quality. We introduce shortly below the different metrics used in our analysis, the precise definitions of all of them can be found in the SM. All the observables are computed and averaged using sets of N₂ = 1000 true or generated samples.
Log-likelihood L: On the contrary other machine learning methods (such classification), the LL cannot be exactly evaluated in an RBM without an exhaustive search of the configuration space. Thus we estimate the partition function using the Annealed Importance Sampling (AIS) method [46). Details of our implementation can be found in SM. We call _ the LL of the (original) dataset with respect to the model, and CRBM the LL. of the generated samples with respect to the model.
Error of the second moment (2); we compute the mean square error (MSE) between the covari- ance matrix computed with a set of samples generated by the RBM and one of the dataset.
Error in the power spectral density PSD: For images, we compute the log average power spec- trum function of radial wave number distance for both a subset of the dataset and the generated set, and then compute the MSE between the two functions. In practice, we compute the error (be- tween true and generated samples) on the logarithm of the squared module of the image's discrete DD-Fourier transform elements Ag at a fixed distance d in Fourier space
P(d) log ((llAs))) II (1000))))))))))=))
(5)
"I has been observed in MNIST that by flipping the input 0 ++ 1, the usual gradient ascent was much slower in one case than in the other.
5

---

Error in the Adversarial Accuracy Indicator (AAI) EAAT; The AAI was recently introduced in Ref. [47] with the goal of quantifying the resemblance and the "privacy" of data drawn by a generative model with respect to the training set. This metric is based on the idea of data nearest neighbors. We begin by defining two sets containing, each one, N₂ samples: (i) "target" set containing only generated samples T = {URBM N and (ii) a "source" with samples from the dataset S = { (m) 1 Jm=1- Then, for each sample m in groups {T. S}. we compute the closest sample using the euclidean distance- to the set groups {T, S} For instance, the closest samples in { for sample m in {T' would be drs(m) = min, VRRM - -vb [m)
||
We can compute in the same way dsr (m), dms(m) and (mr(m). Once we have all these 4 distances, we can estimate how often the "nearest neighbor" (the sample at the shortest distance) of each sample is contained in the same starting set, or in the other one. When the generated samples are statistically indistinguishable from real ones, both frequencies should converge the value of a random guess 0.5 when N₂ large. We therefore compute the MSE of these two frequencies with respect to the asymptotic value 0.5, which defines gAAI, See SM for more details.
Relative entropy AS: We approximate the entropy of given set of data by its byte size when compressed using gzip [48). In particular, compute the entropy of a group of N₂ random samples and the entropy of another set of data of the same size, called S composed of the dataset, in equal portions by samples of the train and generated set. Finally, we compute the relative size of this cross set with respect to the dataset's 's one, minus the expected value for identical sets, i.e. AS = Scross/ SS 1. Then, a large AS indicates that the generated set is less "ordered" than the dataset. On the contrary, a small AS reflects that the generated samples lack diversity.
In the SM, we show the evolution of other quality images observables along the generation sampling (such as the error in the third moment correlations, the error in the energy, or the Frechet Inception Distance score [49 for images). All them showing very similar trends to the ones shown here.
5 Results
Equipped with this set of quality measures, we now want to monitor and evaluate the learning process. To this end, we need to control precisely the learning dynamics and its interplay with the cquilibration time. To do so, we try to isolate all the different factors affecting the learning, and for this reason, at first stage, we avoid using the common tricks to speed up the estimation of the r.h.s. of the gradient of Eqs during the learning. With this goal, we focus our first experiments on the Rdm-k schemes, and discuss the interplay with standard methods such as CD or PCD later on. The Rdm-k scheme is very practical to study the convergence to equilibrium for two reasons: (i) the MCMC chains are initialized from configurations that are uncorrelated with the dataset, and (ii) the sampling protocol used during the learning identical to the one used to generate new samples from scratch with the learned RBM. The pipe-line of our experiments is summarized in Fig. [ B. We split the analysis in two separated stages: learning and generation sampling. During the learning, we update the parameters for 2-1 times using the gradient estimated following the Rdm-k scheme (with i = 10, 50, 100, 500, 10 and 10*). We save the RBM parameters at different numbers of updates equally spaced in logarithmic scale. The number of updates used to train particular RBM will be referred as its age, tage- During the generation stage, we sample the equilibrium Gibbs measure of each RBM (of age tage) with MCMC initialized at random. We evaluate the quality of the samples generated at Gibbs sampling times La equally spaced in logarithmic scale. Unless something else is mentioned, we will consider RBMs with No, 550 hidden nodes and trained with a fixed learning rate of 1j = 0.01, hence the different RBMs will differ only by the value of k and tage. Further details on the learning parameters can be found in SM.
Evolution of the quality of the generated samples with the sampling time-
We show in Fig. - the evolution of the quality estimators of the generated samples throughout the generation sampling for RBMs of different ages (i.e. following the different color lines in Fig. B), and trained with three different k. We recall that perfect samples, in the sense that they are indis- tinguishable from the dataset, would have a score 0 all the quality observables except for the LL, where CRBM should match L2 (LP) is marked as a horizontal line). For all l and early learning times (ages below S 10 updates in red-yellowish colors), all the metrics describe a constant behavior with lg at roughly the same values obtained using completely untrained RBM, and generally far away

---

lags [perameterr updates]
Rdm 10
A
B
201
582
1683
4866
14063
40642
117456
342
990
2862
8272
23907
69091
199675
10
Rdm 10 (4=10)
Rdm 500 (A=500)
Rdm 10* (A=10*)
10
-7
10-3
CD10
10
225
300
C
RBM
2ND
10-1
10
10
PCDIO
10-3
10-2
100
10
10
10-3
0.1
0.0
10-3
random
datasel
10
10
10a
10
10
-0.1
[MCCC steps]
C
GAAI
10-
AS
gSD
8
1h [MCMC steps]
10
10
10a
10
10'
10
10'
10"
10
10'
10
10
10'
10
10
10
10"
10'
10?
10t
[MCMC steps]
to IMCMC steps]
IMCMC
steps]
& |MCMC steps)
D
tG [MCMC steps
[08]
FX1
2151
1600
PX54
RE43
PRRREESSR LOOOEE LOOAGC
MNIST
8
8
8
8
0
FashionMNIST
NORB REDUX
CELEBA COLOR
CELEBA COLOR
CELEBA B&W HQ
CIFAR
CALTECH 101 Silhouettes
Figure 2: Evolution of the quality of the generated samples during the data generation sampling
(along the color dashed lines of Fig. BB) In A we show the LI. and the of quality estimators (introduced in Sect. 2 as function of the sampling time, lG. for chains initialized at random and using RBMs trained for increasing number of updates fage (age is coded by colors) using the Rdm-k scheme with k = 10, 500, 10a. In general, the closer & and AS are to zero, the more similar the generated and original datasets are. For the LL, we show the value computed with the generated set (CRBM in continuous growing lines) and with the original dataset (C", in dashed horizontal lines). We observe that these two measures cross at roughly the same to at which all reach minimum and AS crosses zero. B We show the evolution of cPSD for 3 RBM Ms trained using the Rdm-10, CD-10 and PCD-10 recipes, for the same ages as in A. We consider two possible initializations of the Markov chain (random in full lines and dataset in dashed lines). C We plot the position of the best performance peak and the time at which AS = O, as function of the number of MCMC steps k used for estimating the negative term of the gradient during the training. D We show an example of the images generated as function of lg. using RBMs trained with Rdm-100 and with different datasets. The sampling time to that matches is is hightlighted in an orange frame.
7

---

from O. For an intermediate range of ages (pale blue colors), we observe an important improvement of the metries with the RBM's age, up to certain value (dark blue lines) above which time evolution curves start to collapse. Furthermore, for k = 10, only young ages converge to constant curves in tG, something that extends to intermediate ages for k 500 and covers the majority of the ages for k = 10 In all the cases, the sampling of the oldest machine (lage ~ 200k updates) describe a non-monotonous behavior in the quality measures, with a clear peak of best quality at tG ~ k. This is also the point where L2 and CRBM intersect and where AS crosses 0. In Fig. C for each learning process using a different k we plot the position of this theat (obtained for each quality metries) against the value of k used to train the machine. We observe perfect match between both times. At this point we want to stress that we obtain exactly the same qualitative behavior with the other considered datasets, as shown in the SM. Fig. 2 reflects that these RBMs "memorize" the number of MCMC steps used to estimate the gradient during the learning phase. Memory effects are clear signal of non-equilibrium effects (50) and in fact, this effect is not present long as k is long enough to equilibrate the system during the learning. Indeed, we see that the larger k, the longer (in terms of number of updates) it takes observe memory effects (for i- 10 we only appreciate -itoo---quilirium effects above 100k updates, while they appear around tage Z 10 updates in the case of k 10). These two regimes (equilibrium and OQE) are also reproduced when using the CD and PCD schemes during the learning. We show in Fig. 2-B the evolution of EPSD of the generated samples as function of tG for RBMs trained with the 3 schemes and k 10 steps (higher values of k are shown in the SM). We compare results with MCMC chains initialized either at random (continuous lines) or at the dataset (dashed lines). The sampling of the RBMs trained with CD-10 is also highly complex in time, and much slower than that of the Rdm- 10 considering that the values of EPSD measured in runs from random initialization and from the dataset never converge to the same values for large fagee The situation is rather different for 10, where the error sampling curves describe a very smooth behavior in time and a very slow convergence towards an equilibrium value. Yet According to the minimum values reached in CPSD the equilibrium samples of that model are significantly worse than the best samples created using the Rdm scheme (even working in the OOE regime). This difference is observed also in the rest of quality estimators and at much higher values of k, as shown in Fig. 5 of the SM. Indeed, for MNIST, one needs to reach PCD-10" to start to match the qualities achieved with the Rdm-10 10 recipe when exploiting the OOE regime.
Quantifying the mixing time: At this point, we try to quantify the MT or relaxation time of the MCMC dynamics of our RBM as a function of tage. to rationalize figs. 2
To do so, we compute the
function, p. of the averaged visible variables
p(t) = Cm (t) Cm (0) where Cm No
1
E (m,,(t) - mm,)(m - -m)
(6)
where mt(t) = sigmoid( Wiah (t)+ (b)+ and m, is the equilibrium magnetization! 3 At this point, let us stress that in a perfectly trained RBM, mi would be equal to 1/N, Em n. (.()) Yet, this is not the case in a poorly trained machine. For this reason, we estimate these equilibrium values from a long 10th MCMC steps simulation. The value of p(t) will be close t are very similar, and to O if they are completely decorrelated. In order to measure properly the MT, we need to compute the autocorrelation function from equilibrium configurations. Therefore, we measure P with Eq. (6) but after discarding the first MCMC ~ 10t step We show the time- correlation curves obtained for our best trained machine (using Rdm and k = 10+) in Fig. B for different values of fage. Clearly, the older the machine, the slower the relaxation. Such study is harder with the poorly trained RBMs because equilibration is achieved at times to > 10 for most of the fages, being particularly dramatic for Rdm-10 where relaxations are extremely slow, see SM. The decay being typically exponential, we can easily extract the MT (or the exponential relaxation time), f. from fit of p(t) to A exp((t/tt.) We show the dependence of to with lage in Fig. B C. This ta (for the slowest observable) controls the time-scale of the relaxations in the system, the time necessary to sample independent (uncorrelated) configurations, which means that the number of MCMC iterations necessary to reach equilibrium from the beginning of MCMC simulation is expected to scale with it. Typically, the thermalization time, ttherm= is some few times this la 51. This means that, in order to guarantee sampling of equilibrium configurations,
if the data at t 0 and
[4
'The time-autocorrelation function the visible, hidden or averaged hidden units are qualitatively the same. In practice, we discard as many MCMC steps needed to stop detecting dependence on the t 0 point.
8

---

A
1.00
fige
[updates]
B 10
(t- (:00-00000-000 - (f= 10*)
(k= - (k=
+ 201
8272
0.75
(x= -. 10=)0²)
342
14063
582
23907
10
IMMCC steps
10
0.50
990
40642
0.25
1683
69091
10-
random start dataset start Iem merge
10
102
2862
117456
101
0.00
4866
199675
10"
10
10th
10
10"
10
10
10
10a
10"
10a
10a
102
10
10"
to [MCMC stops]
1G
[MCMC steps]
Syp Lupdates]
Figure 3: A The time-outocorrelttiion function of the visible mean magnetizations, Pm (see Eq. 6 computed during the sampling of an RBM trained with Rdm-10 for different number of updates lage- This function carries the information about the number of MCMC steps that are needed (in equilibrium) to forget a initial configuration of the MC chain. Clearly, the time needed to decorrelate grows with the age of the RBM. In B, for the same RBMs, we show the evolution of the generated samples' quality through the grsD observable, as function of the sampling time tc. and for two different initializations: from the dataset and from random. We mark the times ętherm beyond which both runs converge to the same constant value. C We compare the mixing times, far obtained from a fit of the curves Pm (t) in A to the form exp(-t/ta), with the convergence sereeecee-t-eepee -euuiiiiibiumm times pherm obtained in B for RBMs trained with different values of k. We see that the equilibration time therm follow the same trend for various values of k:
one needs Gibbs samplings longer than this 4herm (which scales with 4.). Note that leherm (the distance equilibrium) might depend on the MC starting point while la should not, yet 4herm will grow proportionally with la for any learning scheme. In fact, we can estimate independently the thermalisation time for the Rdm-k runs, therm- by comparing results from two independent simulations with different initializations: random and dataset, and wait until they converge to the same equilibrium expectation value. This means that one can estimate ftherm (for each tage) by the number of MCMC steps needed to collapse the measures of the two trajectories to a constant value. See for instance Fig. BB where we show the evolution of EPSD with to for the two runs, The merging point for each fage is highlighted with a square. We show the feherm values we obtain with this procedure in Fig. 3 C for the different values of k.
As a result, these two times, la and 4herm- provide us with a lower and a safe upper bound (for instance, when initializing random) for the minimum number of MCMC steps (for the learning's Gibbs sampling), k. needed to learn good equilibrium RBM model. As shown in Fig. 3-C, these times grow progressively with fage. and in particular, falling always out of the range of iterations used to train the h 10 runs (once the RBM starts to learn the dataset for tage 2 100 updates), and falling out the range of scheme Rdm-500 for lage 2 10 and for the last 2 lage for the Rdm-104 scheme (i.c. for tage Z 10t updates). In other words, at those ages, the learning enters in the OOE regime and the RBM starts to learn the dynamical process rather than the equilibrium properties. In fact, these ages mark the crossover between smooth and mon-montt evolution of the observables -LA. A. This reinforces our previous claim that the apparition of the "best performance peak" in the error estimators is a purely non-equilibrium effect. A similar comment can be made for the LL. See for instance Fig. - first row, for each lages we compare the LL measured on the generated CRBM and original dataset L as function of its generation sampling time 4G. We distinguish, again, two distinct behaviours associated to the equilibrium or OOE regime. In the first case, the LL of the generated samples converges to the value of the dataset and remain there even at large to- Whereas, in the OOE regime, _ crosses LD at to R k and continues growing afterwards. This highlights the problem of using the LL to evaluate the sample generation power. While the fact that the likelihood increases during the learning is in general good sign, it is not necessarily linked with the equilibrium properties of the model. In fact, the observation that CBM tends to reach much higher values than L should be rather interpreted
in Fig. Fig.2-A
sign of poor learning, as shown in Fig.
As a direct consequence of the results discussed above, we can see that even in the case where sampling is openly OOF, such as with the Rdm- 10 scheme, one can generate fairly good samples as long as one limits the generation sampling to LG ~ k. as shown in Fig. [ for k = 10: digits are reproduced with uniform probability, and they look OK on visual inspection. For longer samplings

---

RBMs for practitioners
Always control the quality of the generated samples
Reproducible non-informative
Same non informative
Data generation
initialization pureon
initialization
k=(10) MCMC steps to estimate the gradient
Learning phase
Sample for the same & steps with the same dynamics
Generation phase
out-of-equllibrium regime Equilibrium regime
Goal
"Keep permanent chain
longer better model
Fit a model
fret, check stabdayy sampling to ong enought
[nitialization irrelevant
probability distribution
Learn (unnormalized)
*Any initialization jostter informatiin)
ghough better nan informorttteee ve is control the mnstionit
Dataset are typical confs. of the model
Pattern extraction
+Run MCMC steps R-MTIC
Sample equilibrium
tage theck online the MT)
configurations using 'a > O(MT)
Update RBM parameters
(smppingg stable from
long you can
Figure 4: Flow chart of the algorithm proposed to train and sample properly RBM depending on the final goal of the training.
we start to get clear unbalanced distributions. It is also quite remarkable that when matching to with & in the OOE regime of Rdm schemes, we get better quality samples than with the standard CD PCD schemes even at very long sampling times, as we can see from the comparison of EPSD in Fig. 2-B. This is also true for other quality observables or larger values of as shown around Fig. 5 of the SM. In particular, for NMIST, the OOE recipe always generates better samples than the equilibrium regime reached with PCD trained with ks < 10x steps.
Conclusions
In this work, we show that the training of an RBM is richer and more complex than what previously thought. We show that the choice of the number of MCMC steps, k. used to estimate the gradient during the learning has a dramatic impact on the learned model's dynamics and equilibrium states. In practice, we find two regimes: (i) the OOE, when & is much lower than the MCMC mixing time, the trained RBM model encodes the dynamical process followed to train it, resulting in a highly non- linear, unstable and slow evolution of the quality of the generated samples with the sampling time, and a convergence towards poor and strongly biased samples. And (ii) the equilibrium regime, when k overpasses largely the mixing time, the marginal equilibrium probability distribution of the trained model (over the visible layer) matches that of the dataset. In such case, the quality of the generated samples improves smoothly in time converging faster to the equilibrium values, converging much faster than in the OOE regime. While regime (ii) is needed to obtain good trained models, we show that regime (i) is particularly well suited for sample generation because the data sampling non- lincarities can be controlled in the Rdm-k approach, and exploited to generate high-quality data in short training times. We further show that the MC mixing time increases significantly as the learning progresses, being always much larger than the standard ks used in the Literature. We summarize the best strategy to train and exploit RBMs depending on their purpose in Fig.
The existence of a crucial competing mechanism in the training process had already been noted and exploited in EBMs with more complex energy functions (those defined with deep convolutional neural networks) [2 23), and after our results, both regimes were also found in the simpler brother of the RBMs. the Bolzmann Machines 44 45
This evidences that the OOE regime, which was widely unnoticed in the Literature for decades, is a general consequence of the MCMC based like- lihood maximization methods, so it is expected to be reproduced in many other popular models like Deep Boltzmann Machine (DBM). Interestingly, our results should simplify their training and use, least in the OOE regime. More generally, this work provides a set of key points be checked when working with EBMs, and new fundamentally distinctive attribute to theses generative mod- els: equilibrium or OOE We leave for future works systematic comparisons between the models trained in each of these regimes, and in particular, the effect on the features. For instance, we ob- served that RBMs trained with low k suffer strong aging effects extremely similar to those observed on the spin glasses [52). Effects which are not present in equilibrium RBMs.
10

---

Acknowledgments and Disclosure of Funding
A.D. and B.S. were supported by the Comunidad de Madrid and the Complutense University of Madrid (Spain) through the Atracción de Talento program (Refs. 2019-T1/TIC-13298 and Ref. 2019-T1/TIC-12776) BS was also partially supported by MINECO (Spain) through Grant No. PGC20188-996684-BCCZ) (partially funded by FEDER).
References
[1] Paul Smolensky. Information processing in dynamical systems: Foundations of harmony the- ory. Technical report, Colorado Univ at Boulder Dept of Computer Science, 1986.
[2] Nicolas Le Roux and Yoshua Bengio. Representational power of restricted boltzmann ma- chines and deep belief networks. Neural computation,
2008.
[3] R.D. Hjelm, V.D. Calhoun, R. Salakhutdinov, E.A. Allen, T. Adali, and S.M. Plis. Restricted Boltzmann machines for neuroimaging: an application in identifying intrinsic networks. Neu- rolmage, 96:245-200 260, 2014.
[4] X. Hu, H. Huang, B. Peng, J. Han, N. Liu, J. Lv, L. Guo, C. Guo, and Liu. Latent source mining in fmri via restricted Boltzmann machine. Human brain mapping, 396))2368-2388, 2018.
[5] Jérôme Tubiana, Simona Cocco, and Rémi Monasson. Learning protein constitutive motifs from sequence data. Elife, 8:e39397, 2019.
[6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[7] Asja Fischer and Christian Igel. Training restricted boltzmann machines: An introduction. Pattern Recognition, 47(1):25-39, 2014. Number:
[8] Ruslan Salakhutdinov and lain Murray. On the quantitative analysis of deep belief networks. In Proceedings of the 25th international conference on Machine learning, pages 372-879, 2008.
[9] Jan Melchior, Asja Fischer, and Laurenz Wiskott. How to center deep boltzmann machines. The Journal of Machine Learning Research, 17(1):338733477, 2016.
[10] Marylou Gabrié, Eric W Tramel, and Florent Krzakala. Training restricted Boltzmann machine via the Thouless-AndeersonPPliile free energy. In Advances in neural information processing systems, pages 640-648, 2015.
[11] Peter de Boves Harrington. Feature expansion by continuous restricted boltzmann machine for near-infrared spectrometric calibration. Analytica chimica acta, 1010:20-28, 2018
[12] Huihui Shen and Hongwei Li. A gradient approximation algorithm based weight momentum for restricted boltzmann machine. Neurocomputing, 361:40-49, 2019.
[13] Ramasamy Savitha, ArulMurugan Ambikapathi, and Kanagasabai Rajaraman. Online rbm: Growing restricted boltzmann machine the fly for unsupervised representation. Applied Soft Computing, 92:106278, 2020.
[14] Guido Montúfar. Restricted boltzmann machines: Introduction and review. In Information Geometry and Its Applications IV, pages 75-115. Springer, 2016.
[15] Jing Chen, Song Cheng, Haidong Xie, Lei Wang, and Tao Xiang. Equivalence of restricted boltzmann machines and tensor network states. Physical Review B, 9788)085104, 2018.
[16] Aurélien Decelle, Giancarlo Fissore, and Cyril Furtlehner. Thermodynamics of re- stricted Boltzmann machines and related learning dynamics. Journal of Statistical Physics, 17766))1576-1666 2018

---

[17] Kai Shimagaki and Martin Weigt. Selection of sequence motifs and generative hopfield-pote models for protein families. Physical Review E, 100(3):032128, 2019.
[18] Roger G Melko. Giuseppe Carleo, Juan Carrasquilla, and Ignacio Cirac. Restricted boltzmann machines quantum physics. Nature Physics, 15())8887-892, 2019.
[19] Moshir Harsh, Jérôme Tubiana, Simona Cocco, and Remi Monasson. 'place-cell' emergence and learning of invariant data with restricted Boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space. Journal of Physics A: Mathematical and Theoretical, 53(17):174002, 2020.
[20] Burak Yelmen, Aurélien Decelle, Linda Ongaro, Davide Marnetto, Corentin Tallec, Francesco Montinaro, Cyril Furtlchner, Luca Pagani, and Flora Jay. Creating artificial human genomes using generative neural networks. PLoS genetics, 17(2)e1009303, 2021.
[21] Barbara Bravi, Jérôme Tubiana, Simona Cocco, Rémi Monasson, Thierry Mora, and Alcksan- dra M Walczak. Rbm-mhe: A semi-supervised machine-learingg method for sample-specific prediction of antigen presentation by hla-i alleles. Cell systems, 12(2):195-202 2021.
[22] Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 5272 5280, 2020.
[23] Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run meme toward energy-basee model. Advances in Neural Information Processing Systems, 32:5232-5244 2019.
[24] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(12))2 1998.
[25] Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for bench- marking machine learning algorithms. CoRR, abs/ 2017.
[26] Yann LeCun, Fu Jie Huang, and Leon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages 11-104. IEEE, 2004.
[27] Vincenza Colonna, Qasim Ayub. Yuan Chen, Luca Pagani, Pierre Luisi, Marc Pybus, Erik Garrison, Yali Xue, and Chris Tyler-Smith. Human genomic regions with exceptionally high levels of population differentiation identified from 911 whole-genome sequences. Genome biology, 15(6):1-14, 2014.
[28] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CoRR,
2017.
[29] Alex Krizhevsky, Geoffrey Hinton, et Learning multiple layers features from tiny images. Technical Report TR-2009, University of Toronto, Toronto., 2009.
[30] V. Nair and G.E. Hinton. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807- 814, 2010.
[31] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8)17771-1800, 2002.
[32] Tijmen Tieleman. Training restricted Boltzmann machines using approximations to the like- lihood gradient. In Proceedings of the 25th international conference on Machine learning, pages 1064-1071,0 2008
[33] Koji Hukushima and Koji Nemoto. Exchange monte carlo method and application to spin glass simulations. Journal of the Physical Society of Japan, 6546):160411688, 1996.
12

---

[34] Russ R Salakhutdinov. Learning in markov random fields using tempered transitions. In Y. Bengio. D. Schuurmans, Lafferty. C. Williams, and A. Culotta, editors. Advances in Neural Information Processing Systems, volume 22. Curran Associates, Inc., 2009.
[35] Guillaume Desjardins, Aaron Courville, Yoshua Bengio, Pascal Vincent, and Olivier Delal- leau. Tempered markov chain monte carlo for training of restricted boltzmann machines. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 145-152. JMLR Workshop and Conference Proceedings, 2010.
[36] Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Accurate and conservative estimates of mrf log-likelihood using reverse annealing. In Artificial Intelligence and Statistics, pages 102-110. PMLR, 2015.
[37] Roger Grosse and Ruslan Salakhudinov. Scaling up natural gradient by sparsely factorizing the inverse fisher matrix. In International Conference on Machine Learning, pages 2304-2313. PMLR, 2015.
[38] Oswin Krause, Asja Fischer, and Christian lgel. Copulaatoo-cootrasiie-ddddrenrre Does con- sistency help with rbm training? Pattern Recognition Letters, 102:1-7, 2018.
[39] Enrique Romero, Ferran Mazzanti, Jordi Delgado, and David Buchaca. Weighted contrastive divergence. Neural Networks, 14:147-156, 2019.
[40] Vidyadhar Upadhya and PS Sastry. Efficient learning of restricted boltzmann machines using covariance estimates. In Asian Conference on Machine Learning, pages 836-851. PMLR, 2019.
[41] Ken-Ichi Aoki and Tamao Kobayashi. Restricted boltzmann machines for the long range ising models. Modern Physics Letters B, 30(34): 1650401, 2016
[42] Alan Morningstar and Roger G Melko. Deep learning the ising model near criticality. arXiv preprint arXiv: 1708. 04622, 2017.
[43] David Yevick and Roger Melko. The accuracy of restricted boltzmann machine models of ising systems. Computer Physics Communications, 258: 107518, 2021.
[44] Pierre Barrat-Charlan Anna Paola Muntoni, Kai Shimagaki, Martin Weigt, and Francesco Zamponi. Sparse generative modeling via parameter reduction of boltzmann machines: appli- cation to protein-sequenee families. Physical Review E, 104(2):024407, 2021.
[45] Anna Paola Muntoni, Andrea Pagnani, Martin Weigt, and Francesco Zamponi. adab- mdea: Adaptive boltzmann machine learning for biological sequences. arXiv preprint arXiv:2109.04105. 2021.
[46] Oswin Krause, Asja Fischer, and Christian Igel. Algorithmss for estimating the partition func- tion of restricted boltzmann machines. Artificial Intelligence, 278: 103195, 2020.
[47] Andrew Yale, Saloni Dash, Ritik Dutta, Isabelle Guyon, Adrien Pavao, and Kristin P Ben- nett. Generation and evaluation of privacy preserving synthetic health data. Neurocomputing, 416:244-255, 2020.
[48] Andrea Baronchelli, Emanuele Caglioti, and Vittorio Loreto. Measuring complexity with zip- pers. European journal of physics, 26(5):S69, 2005.
[49] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[50] Nathan C Keim, Joseph D) Paulsen, Zorana Zeravcic, Srikanth Sastry, and Sidney R Nagel. Memory formation in matter. Reviews of Modern Physics, 91(3):035002, 2019.
[51] Alan Sokal. Monte carlo methods in statistical mechanics: foundations and new algorithms. In Functional integration, pages 131-192. Springer, 1997.
[52] Mare Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications, volume 9. World Scientific Publishing Company, 1987.
13

---

7 Paper checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the pa- per's contributions and scope? [Yes] our claims have been checked with experimental results over three distinct datasets. We expect them to be valid in general.
(b) Did you describe the limitations of your work? [Yes] Even though our main contri- bution is to explain a practical and general mechanism of the RBM's learning, we explicitly specify the practical impact can have. In partieular, the relaxation time of RBMs can be very large what limits strongly the expectations of learning good equi- librium models or very high-dimensiona datasets. We alert this could have an impact of the reliability of the features extracted from OOE machine.
(c) Did you discuss any potential negative societal impacts of your work? [No] do not discuss negative societal impact since we do not believe it applies for this work. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] have read the ethics review guidelines and our paper conforms them or/and our work do not produce any specific risk on these lines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] Our work do not include theoretical results
(b) Did you include complete proofs of all theoretical results? [N/A] Our work do not include theoretical results
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We released a version of the code on github with some scripts reproduced part of the main exper- imental results. The dataset are public.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] The training details are specified in the supplementary material. (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] and no... the curves that are of interest for their trend have been made without error bar. These curves have been however averaged over 10x realization in order make the curve smooth enough. The auto-correlation curves have error bar in order to demonstrate clearly our point on the relaxation time, and errors are also computed for the computation of the relaxation time.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] We put in the SM estimate of the training time of our experiments and the used material.
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] cite the original paper associate to each of the dataset.
(b) the rest does not apply for us
(c) Did you mention the license of the assets? [N/A]
(d) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(e) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(f) Did you discuss whether the data you are using/curating contains personally identifi- able information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
14

---

(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "A2Bgeqmgcn1", "reviewer": "Reviewer_g5Pg", "summary": "The authors demonstrate that many RBM models are learned in an unexpected out-of-equilibrium regime where the distribution of short run MCMC samples used to update the model differs greatly from the long run equilibrium distribution of the model. The claims are supported with diagnostics to measure out-of-equilibrium behavior and visualizations of samples after across MCMC updates.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main objective of this paper is to demonstrate that RBM training has an unexpected but widespread behavior where samples generated by short run sampling using common initialization techniques like CD and PCD are typically out of equilibrium. Despite this, learning can remain stable and generate short run images that are very similar to data images, and the apparent success of learning a generative model can lead to the mistaken conclusion that the RBM has learned a valid equilibrium distribution. The authors question common practice of evaluating RBM's exclusively using short-run samples, which are not indicative of the steady-state, or negative samples from the end of training, which might not even be configurations which are attainable by the model once the weights are frozen. To examine the equilibrium distribution of RBM's, the authors use long run MCMC samples and find that equilibrium states are poor samples across many learning configurations. The points raised by the authors are crucial from the perspective of rigorous probabilistic learning of an RBM and this work sheds light on important but under-recognized phenomena in generative modeling.\n\nThe main weakness of this work is that the authors do not sufficiently address the related work [a] which covers many of the same ideas. The work [a] identifies a convergent and non-convergent learning phase for the Energy-Based Model (EBM) that directly correspond to the equilibrium and out-of-equilibrium phases identified by the authors for RBM training. It appears that this phenomenon presented by the authors occurs in energy models with different kinds of potentials. Given the high degree of similarity between these two works, I recommend against acceptance until a discussion of the connections to [a] is made clear by the authors in a revised version.\n\nThe authors present several diagnostics to identify equilibrium and out-of-equilibrium behavior. The most convincing diagnostics to me were the auto-correlation measures over the course of training. The other diagnostics seemed to be somewhat ad-hoc but nonetheless they appear to be useful indicators. One diagnostic that that authors did not include but possibly should is a measurement of the energy of the visible states over the course of training. Typically one would expect that the energy of short run MCMC samples should be about the same as data samples. The work [a] shows that when the short run samples are out of equilibrium, then long-run samples typically have much lower energy than short run or data samples. Successful learning should occur when short run, long run, and data energy values remain in the same spectrum, and it might be possible to diagnose out of equilibrium behavior for RBMs by examining whether long run visible energy values are in the same spectrum as short run and/or data samples. \n\nThe scale and complexity of the experiments is somewhat limited because the authors exclusively examine binary pixel states rather than continuous and multi-channel pixels. The authors clearly demonstrate that out-of-equilibrium learning occurs for RBM models. I am a little uncertain over the author's exact conclusion regarding the method for learning an equilibrated model. The abstract claims that increasingly large k can be used to learn a valid equilibrium, but experiments with Rmd-k show even for k=10,000, non-equilibrium effects appear after around 100k MCMC updates. On the other hand, PCD learns a more reasonable equilibrium even for smaller k, which is consistent with what is reported in [a]. It would be good for the authors to clarify strategies for learning a correct equilibrium. \n\n**Overall Summary**: This work identifies an important and under-recognized outcome of out-of-equilibrium RBM learning. Bringing attention to this phenomenon is useful and valuable. The major weakness of this work is that it does not include a related work [a] that presents very similar findings. The experimental complexity could also be increased by using continuous pixels and color images. I recommend against accepting this work until relations to prior work are made clear. I am willing to change my score based on author feedback and revisions.\n\n[a] On the anatomy of MCMC-based maximum likelihood learning of energy-based models. E. Nijkamp, M. Hill, T. Han, Y.N. Wu, S.C. Zhu. AAAI 2020. https://arxiv.org/abs/1903.12370", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed limitations and impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "4Nb91NEwwJq", "reviewer": "Reviewer_zU2d", "summary": "\nThe authors extensively analyze the training of Restricted Boltzmann Machines (RBMs).\nThey show that there is definite room for improvement in training RBMs in terms of\n\n- *Evaluation:* The authors show that despite its popularity, log likelihood is not the only metric which should be tracked while training the RBM and show that bad machines can have a deceptively good log likelihood but not be able to generate good samples and vice versa.\n- *Methods:* Despite their popularity, both CD-k and PCD-k can be outperformed in terms of the quality of generated samples by Rdm-k which uses chains started at uniformly sampled states to compute the negative statistics\n\nMoreover they empirically show that for Rdm-k, smaller values of k lead to models which memorize data (Figure 2A $\\Delta S$) and a larger k leads to generalization (likelihood of generated samples is close to that of the data) (Figure 2A $\\mathcal{L}$).", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- The authors conducted extensive experiments to analyze what had been thought to be an un-interesting and known aspect of RBM training and have demonstrated surprising results.\n- The entire paper (especially the introduction) is well written, motivation is well presented and the experimental evaluation is well explained.\n- The paper proposes and summarizes multiple very useful metrics which can be used by practitioners to track their RBM training.\n\n### Weaknesses\n\n- [Clarified in rebuttal] The evaluation and the methods proposed, while highly informative and useful, are computationally expensive.\n  - The authors mention this in their supplementary material. \n- While the mixing time estimation is interesting a comparison with theoretical bounds, in particular [1] would have been more interesting.\n- Possible citations:\n  - [2] could be cited as a more recent survey of training methods along with the citation on line 33\n  - [3] was recently published in ICML and proposes a novel method to train RBMs which is unrelated to CD-k, a similar analysis of that method compared to Rdm-k would be very interesting.\n  - [4] proposed a method to train RBMs using random walk tours which provide an almost unbiased estimate of the negative statistics\n    - they additionally observed a similar effect where samples from shorter tours (k) demonstrated memorization and longer tours (k) demonstrated generalization, which is corroborated by the observations in the current paper.\n  - [5] proposed using the up and coming unbiased couplings method to train EBMs including RBMs, since this method is scalable to larger RBMs it could be the subject of such a study.\n\n### Other criteria\n\n- Originality : Existing problem, marginally novel solution, exciting and novel results.\n- Quality : Extremely high quality of writing.\n- Clarity : Experiments are well justified,  motivations are well explained, results are well elucidated.\n- Significance : High significance in terms of insight, prohibitive computational cost will hamper utility.\n\n[1] - Tosh, Christopher. \"Mixing rates for the alternating Gibbs sampler over restricted Boltzmann machines and friends.\" International Conference on Machine Learning. PMLR, 2016.\n\n[2] - Song, Yang, and Diederik P. Kingma. \"How to train your energy-based models.\" arXiv preprint arXiv:2101.03288 (2021).\n\n[3] - Grathwohl, Will, et al. \"Oops I Took A Gradient: Scalable Sampling for Discrete Distributions.\" arXiv preprint arXiv:2102.04509 (2021).\n\n[4] - Savarese, Pedro, Mayank Kakodkar, and Bruno Ribeiro. \"From monte carlo to las vegas: Improving restricted boltzmann machine training through stopping sets.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.\n\n[5] - Qiu, Yixuan, Lingsong Zhang, and Xiao Wang. \"Unbiased contrastive divergence algorithm for training energy-based latent variable models.\" International Conference on Learning Representations. 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1txw6S1RAN2", "reviewer": "Reviewer_35Jj", "summary": "The paper discusses about some important factors affecting the results of RBMs.\n\n1. The choice of mixing time.\n\n2. The mixing time and the number of steps(k) affect two regimes of RBMs, namely equilibrium and out-of-equilibrium.\n\n3. How to choose the number of steps(k). (A) short k for convincing samples in short learning times (B) large k for learning the correct equilibrium distribution of the RBM.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The main problem of this paper is lack of innovation. The author compares the results of 3 different recipes (CD, PCD, Rdm) proposed by others, and puts forward that the number of MCMC steps has a dramatic effect on the model. This paper does not improve the previous algorithms a lot and not give a specific method or function concerning how to choose number of MCMC steps.\n\n2. All the conclusions and results are proved empirically, not theoretically. However, the model has only been applied on two datasets, and deals with only binary nodes. The conclusions should be more convincing by training on more datasets.\n\n3. The algorithm and results in section 2-5 can be presented in a clearer way by adding flow chart of the algorithm and tables of different results.\n\n==============\nAfter reading the response of authors, I update my comments (see below).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This experimental paper has been well received by the reviewers, who are overall fairly confident that the paper deserves to be accepted. They also unanimously point out the quality of writing and presentation. The reviewers have done a great job, and I trust the authors to implement the promised revisions before publication. The main weakness appears to be a lack of originality. But given the timeliness of the models studied here and the overall quality of presentation, this papers complements well existing literature pointing at the non-equilibrium properties of energy based generative models. Moreover the reviewers acknowledge mostly convincing responses to their comments. In particular, I think it is important that, as promised by the authors, they implement the numerical experiments on more datasets, especially given the pure experimental nature of the paper. Second, a reviewer commented: "Can authors provide a reference protocol for training RBM in applications based on results of this paper? This would be helpful for readers." I completely back-up that point. I think that this would add to the paper. Last but not least: it is of crucial importance that the authors extensively discuss and compare their results to reference https://arxiv.org/abs/1903.12370 , and explain in details how their contributions complements it and confirms further its initial findings in a different context.






Decision: Accept (Poster)