Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines

Training Restricted Boltzmann Machines ( RBMs ) has been challenging for a long time due to the difficulty of computing precisely the log-likelihood gradient . Over the past decades , many works have proposed more or less successful training recipes but without studying the crucial quantity of the problem : the mixing time , i.e . the number of Monte Carlo iterations needed to sample new configurations from a model . In this work , we show that this mixing time plays a crucial role in the dynamics and stability of the trained model , and that RBMs operate in two well-defined regimes , namely equilibrium and out-of-equilibrium , depending on the interplay between this mixing time of the model and the number of steps , k , used to approximate the gradient . We further show empirically that this mixing time increases with the learning , which often implies a transition from one regime to another as soon as k becomes smaller than this time . In particular , we show that using the popular k ( persistent ) contrastive divergence approaches , with k small , the dynamics of the learned model are extremely slow and often dominated by strong out-of-equilibrium effects . On the contrary , RBMs trained in equilibrium display faster dynamics , and a smooth convergence to dataset-like configurations during the sampling . Finally we discuss how to exploit in practice both regimes depending on the task one aims to fulfill : ( i ) short k can be used to generate convincing samples in short learning times , ( ii ) large k ( or increasingly large ) is needed to learn the correct equilibrium distribution of the RBM . Finally , the existence of these two operational regimes seems to be a general property of energy based models trained via likelihood maximization . 1 Introduction . Restricted Boltzmann Machines ( RBM ) are one of the generative stochastic neural networks that have been available for decades [ 1 ] and are renowned for their capacity to learn complex datasets [ 2 ] and draw similar new data . Furthermore , the hidden correlations found in the data are easily accessible [ 3 , 4 , 5 ] , which is particularly interesting for a scientific use of Big Data . Despite these positive features , RBMs still remain hard to train and evaluate , and hence to use in practical problems . In comparison , Generative Adversarial Network ( GAN ) [ 6 ] ( for which it is much harder to find a working architecture ) are more commonly used , not only because of the advantages given by convolutional layers , but also because the generative process does not depend on a costly sampling procedure with an uncontrolled convergence time . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Both training and data generation of RBMs are tricky and unpredictable in inexperienced hands . One of the main difficulties is that it is hard to evaluate if the learning is progressing or not . Yet , many works in the past twenty years have proposed recipes for good practices in RBM training [ 7 ] . Unfortunately , the evaluation of these recipes relies only on the comparison of the values reached by the log-likelihood ( LL ) [ 8 , 7 , 9 ] , a quantity that can not be monitored in tractable times during the learning process . Studies properly characterizing the quality , independence , or stability of the generated samples using the different recipes are nearly absent in the Literature . In fact , many works just show a set of new samples ( for eye evaluation ) that were either obtained after a short Markov Chain Monte Carlo ( MCMC ) sampling initialized at the dataset , or borrowed from the permanent chain ( negative particle ) at the end of the training [ 10 ] . In other studies , a reconstruction error after one or several MCMC steps is used to determined if the machine was correctly trained [ 11 ] . However , none of these measures guarantees that the trained model can generate from scratch new samples nor that the dataset are typical samples of that model . At most these tests assure you that samples of the dataset are locally stable w.r.t few MCMC steps , unless a proper decorrelation from the initial conditions or a link to equilibrium properties are investigated . Finally , more recent works use a classification score on the activation of the latent variables [ 12 , 13 ] , which reflects that the relevant features were learned , but is silent about the generative power of the machine . RBMs are receiving an increasing attention in life and pure sciences during the recent years [ 5 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] given their potential for interpretability of the patterns learned by the machine ( a rare feature in machine learning ) . Indeed , the RBM formalism enables the extraction of the ( unknown ) probability distribution function of the dataset . Yet , such an approach is meaningful if the dataset is related to the equilibrium properties of the trained model . Given these new and exciting prospectives for the RBMs , it is more than ever important to establish reproducible protocols and evaluation tools to guarantee not only that the model generates good enough data , but also that the dataset are typical equilibrium samples of the model . In this work we demonstrate that the classical training procedures can lead to two distinct regimes of the RBM : an equilibrium and an out-of-equilibrium ( OOE ) one . A finding that had been recently discussed in the Literature under the name of convergent or non-convergent regimes in a different family of generative energy based models ( EBMs ) , those whose energy function is a ConvNet [ 22 ] . Here , at variance to previous approaches , we show that the equilibrium regime is observed if the number of MCMC steps , k , used to estimate the LL gradient , is larger than the algorithm mixing time ( MT ) . The latter emerges when k falls below this MT . In particular , we show that for a classical dataset such as MNIST , the MT is always rather large and grows with the learning time . This implies that learning an “ equilibrium ” model ( i.e . a RBM that operates in the equilibrium regime ) is costly , and that most of the works in the Literature operate without any doubt in the OOE regime . In this regime , the equilibrium distribution of the machine at the end of the learning is significantly different from that of the dataset . Yet , despite this handicap , we show the OOE regime can be exploited to generate good samples at short training times , an observation that have already been exploited in practice [ 23 ] for convolution EBM . All our conclusions rely on the study of 9 datasets : MNIST [ 24 ] , whose results are discussed in the main-text , FashionMNIST [ 25 ] , Caltech101 Silhouettes , smallNORB dataset [ 26 ] , a human genome dataset [ 27 ] ( with similar dimensions as MNIST but more structured ) , the high-quality CelebA [ 28 ] projected in black and white , and in low definition but in color , and CIFAR [ 29 ] . The analysis of most of the last datasets are discussed only in the SM and show a similar behavior to the one observed on MNIST . Our paper is organized as follows . RBMs , their general principles and learning equations are defined in sec . 2 . We review previous related works in sec . 3 . In sec . 4 we define the different observables used to monitor the training and sampling . Finally , we discuss experimental results in sec . 5 . 2 Definition of the model . An RBM is a Markov random field with pairwise interactions defined on a bipartite graph of two non-interacting layers of variables : the visible nodes , v = vi , i = 1 , ... , Nv , represent the data , while the hidden ones , h = hj , j = 1 , ... , Nh , are the latent representations of this data and are there to build arbitrary dependencies between the visible units ( see sketch in fig . 1–A ) . Usually , the nodes are binary-valued in { 0 , 1 } , yet Gaussian or more arbitrary distributions on real-valued bounded support are also used , ultimately making RBMs adaptable for more heterogeneous datasets . Here , we deal only with binary { 0 , 1 } variables for both the visible and hidden nodes . Other approaches using truncated-Gaussian hidden units [ 30 ] , giving a ReLu type of activation functions for the hidden layer do work well , but in our experiments we observed qualitatively the similar dynamical behavior and therefore we will stick to binary hidden units for the rest of the article . The energy function of an RBM is taken as E [ v , h ; w , b , c ] = − ∑ ia viwiaha − ∑ i bivi − ∑ a caha , ( 1 ) with w the weight matrix and b , c the visible and hidden biases , respectively . The Boltzmann distribution is then given by p [ v , h|w , b , c ] = exp ( −E [ v , h ; w , b , c ] ) Z with Z = ∑ { v , h } e−E [ v , h ] , ( 2 ) being the partition function of the system . RBMs are usually trained using gradient ascent of the LL function of the training dataset D = { v ( 1 ) , · · · , v ( M ) } , being the LL defined as L ( w , b , c|D ) = M−1 M∑ m=1 ln p ( v = v ( m ) |w , b , c ) = M−1 M∑ m=1 ln ∑ { h } e−E [ v ( m ) , h ; w , b , c ] − lnZ . ( 3 ) The gradient is then composed of two terms : the first accounting for the interaction between the RBM ’ s response and the training set , and same for the second , but using the samples drawn by the machine itself . The expression of the LL gradient w.r.t . all the parameters is given by ∂L ∂wia = 〈viha〉D − 〈viha〉H , ∂L ∂bi = 〈vi〉D − 〈vi〉H and ∂L ∂ca = 〈ha〉D − 〈ha〉H , ( 4 ) where 〈f ( v , h ) 〉D = M−1 ∑ m ∑ { h } f ( v ( m ) , h ) p ( h|v ( m ) ) denotes an average over the dataset , and 〈f ( v , h ) 〉H , the average over the Boltzmann measure in Eq . 2 . A large part of the literature on RBMs focuses on schemes to approximate the r.h.s . of Eqs . 4 . In theory , these terms can be computed up to an arbitrary precision using parallel MCMC simulations as long as the number of MC steps are large enough to ensure a proper sampling of the equilibrium configuration space . In other words , sampling times should be larger than the MCMC MT . A naive way to implement this idea is to initialize each of these parallel Markov chains on random conditions , perform k MCMC steps , and use the final configurations to estimate the gradient . We call this scheme Rdm-k . This scheme is not used in practice because it would require too many k steps if the MT is too large . Many recipes have been proposed to shorten this sampling and approximate the negative term of the gradient , but since the introduction of the so-called contrastive divergence ( CD ) by Hinton [ 31 ] , only a limited number of schemes are used in practice . The first one , CD-k , initializes the MCMC simulation from the same data of the mini-batch used to compute the gradient , and performs k ∼ O ( 1 ) sampling steps . This approximation relies on the idea that the dataset must be a good approximation of the equilibrium samples of a well trained RBM , something that is not true for a poorly trained machine , whose typical samples are quite distant from the dataset . In the second commonly used method , the last configurations of the Markov chains are saved from one parameter update to the other , and then used to initialize the subsequent chain . As in the other methods , k MCMC steps are used to estimate the gradient for each update . This method is known as persistent-CD : PCD-k [ 32 ] , and takes advantage of the fact that the RBM ’ s parameters change smoothly during the learning and the same is expected for the models ’ equilibrium distribution1 . Finally , other approximations , such as mean-field or TAP equations [ 10 ] , or more elaborate approaches , such as the parallel tempering technique [ 33 , 34 ] , can be used . A common feature of all these recipes is that they rely on very few simulation steps to sample the Boltzmann distribution , and the trained model often generates rather bad configurations . To illustrate this last comment , let us discuss a provocative experiment shown on fig . 1–E where we try to sample new digits just by sampling the equilibrium measure of our trained RBMs using a MCMC initialized uniformly at random . Let us stress that this is not the standard set-up used in the field to generate new samples , but it is a strong test of the generation power of the trained model . We compare the outcome of the generated samples ( as function of the sampling time ) for three RBMs trained each following different recipes to compute the negative term of the gradient : CD-10 , PCD10 and Rdm-10 . At this point , k = 10 might seem a very small number . It is however the typical order of magnitude used in the Literature . The details of the learning protocol are discussed later in the text and summarized in the SM . The first striking point on fig . 1–E , is the inability of the CD-10 RBM to properly sample digits in this set-up . This problem has been reported before [ 35 , 8 ] , but since CD is part of the standard methods , we want to insist on this point . In addition , PCD-10 trained RBM generates proper digits but only after a very long sampling time , while with Rdm-10 , quite surprisingly , digits are properly generated after only 10 MCMC steps . Yet , samples obtained after much longer MCMC steps have unbalanced digits representations , like the 1 being over-represented . In fact , we observe that poorly trained models generate samples that do not respect the correct balance between the 10 digits present in the original database ( see fig . 1–D ) . We already mentioned the 1s in the Rdm-10 run , but it happens the same ( to a lesser extent ) with the 0 in the PCD-10 case . Alternatively to these generated samples , we show the evolution of the LL during the learning of these same three RBMs in fig . 1–C . By definition of the loss function , the LL is intended to be maximized , so it constitutes a traditional measure of the quality of the training . The LL of the models used for the sampling of fig . 1–E are highlighted in large symbols . This plot illustrates the lack of reliability of the LL to monitor the quality of the learning . For instance , CD-10 reaches quite high values of LL and yet , it can not generate a single digit . For the Rdm-10 is the other way around , it generates decent samples but with a very poor LL . Finally , we observe that the LL of PCD-10 reaches very high values , comparable to RBMs trained using up to k=104 MCMC steps to estimate the gradient . We will come back to this point . Our experiment highlights , first , that the LL can not be used all alone to quantify the generative power of an RBM , and second , that the quality of the generated samples may variate in a non-monotonic way during the generation sampling . 1This adiabatic scheme is thus valid as long as the relaxation time for the RBM parameters during the learning is much larger than the MCMC mixing times .

Summary:  The authors extensively analyze the training of Restricted Boltzmann Machines (RBMs). They show that there is definite room for improvement in training RBMs in terms of  - *Evaluation:* The authors show that despite its popularity, log likelihood is not the only metric which should be tracked while training the RBM and show that bad machines can have a deceptively good log likelihood but not be able to generate good samples and vice versa. - *Methods:* Despite their popularity, both CD-k and PCD-k can be outperformed in terms of the quality of generated samples by Rdm-k which uses chains started at uniformly sampled states to compute the negative statistics  Moreover they empirically show that for Rdm-k, smaller values of k lead to models which memorize data (Figure 2A $\Delta S$) and a larger k leads to generalization (likelihood of generated samples is close to that of the data) (Figure 2A $\mathcal{L}$).