Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines

Training Restricted Boltzmann Machines ( RBMs ) has been challenging for a long time due to the difficulty of computing precisely the log-likelihood gradient . Over the past decades , many works have proposed more or less successful training recipes but without studying the crucial quantity of the problem : the mixing time , i.e . the number of Monte Carlo iterations needed to sample new configurations from a model . In this work , we show that this mixing time plays a crucial role in the dynamics and stability of the trained model , and that RBMs operate in two well-defined regimes , namely equilibrium and out-of-equilibrium , depending on the interplay between this mixing time of the model and the number of steps , k , used to approximate the gradient . We further show empirically that this mixing time increases with the learning , which often implies a transition from one regime to another as soon as k becomes smaller than this time . In particular , we show that using the popular k ( persistent ) contrastive divergence approaches , with k small , the dynamics of the learned model are extremely slow and often dominated by strong out-of-equilibrium effects . On the contrary , RBMs trained in equilibrium display faster dynamics , and a smooth convergence to dataset-like configurations during the sampling . Finally we discuss how to exploit in practice both regimes depending on the task one aims to fulfill : ( i ) short k can be used to generate convincing samples in short learning times , ( ii ) large k ( or increasingly large ) is needed to learn the correct equilibrium distribution of the RBM . Finally , the existence of these two operational regimes seems to be a general property of energy based models trained via likelihood maximization . 1 Introduction . Restricted Boltzmann Machines ( RBM ) are one of the generative stochastic neural networks that have been available for decades [ 1 ] and are renowned for their capacity to learn complex datasets [ 2 ] and draw similar new data . Furthermore , the hidden correlations found in the data are easily accessible [ 3 , 4 , 5 ] , which is particularly interesting for a scientific use of Big Data . Despite these positive features , RBMs still remain hard to train and evaluate , and hence to use in practical problems . In comparison , Generative Adversarial Network ( GAN ) [ 6 ] ( for which it is much harder to find a working architecture ) are more commonly used , not only because of the advantages given by convolutional layers , but also because the generative process does not depend on a costly sampling procedure with an uncontrolled convergence time . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Both training and data generation of RBMs are tricky and unpredictable in inexperienced hands . One of the main difficulties is that it is hard to evaluate if the learning is progressing or not . Yet , many works in the past twenty years have proposed recipes for good practices in RBM training [ 7 ] . Unfortunately , the evaluation of these recipes relies only on the comparison of the values reached by the log-likelihood ( LL ) [ 8 , 7 , 9 ] , a quantity that can not be monitored in tractable times during the learning process . Studies properly characterizing the quality , independence , or stability of the generated samples using the different recipes are nearly absent in the Literature . In fact , many works just show a set of new samples ( for eye evaluation ) that were either obtained after a short Markov Chain Monte Carlo ( MCMC ) sampling initialized at the dataset , or borrowed from the permanent chain ( negative particle ) at the end of the training [ 10 ] . In other studies , a reconstruction error after one or several MCMC steps is used to determined if the machine was correctly trained [ 11 ] . However , none of these measures guarantees that the trained model can generate from scratch new samples nor that the dataset are typical samples of that model . At most these tests assure you that samples of the dataset are locally stable w.r.t few MCMC steps , unless a proper decorrelation from the initial conditions or a link to equilibrium properties are investigated . Finally , more recent works use a classification score on the activation of the latent variables [ 12 , 13 ] , which reflects that the relevant features were learned , but is silent about the generative power of the machine . RBMs are receiving an increasing attention in life and pure sciences during the recent years [ 5 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] given their potential for interpretability of the patterns learned by the machine ( a rare feature in machine learning ) . Indeed , the RBM formalism enables the extraction of the ( unknown ) probability distribution function of the dataset . Yet , such an approach is meaningful if the dataset is related to the equilibrium properties of the trained model . Given these new and exciting prospectives for the RBMs , it is more than ever important to establish reproducible protocols and evaluation tools to guarantee not only that the model generates good enough data , but also that the dataset are typical equilibrium samples of the model . In this work we demonstrate that the classical training procedures can lead to two distinct regimes of the RBM : an equilibrium and an out-of-equilibrium ( OOE ) one . A finding that had been recently discussed in the Literature under the name of convergent or non-convergent regimes in a different family of generative energy based models ( EBMs ) , those whose energy function is a ConvNet [ 22 ] . Here , at variance to previous approaches , we show that the equilibrium regime is observed if the number of MCMC steps , k , used to estimate the LL gradient , is larger than the algorithm mixing time ( MT ) . The latter emerges when k falls below this MT . In particular , we show that for a classical dataset such as MNIST , the MT is always rather large and grows with the learning time . This implies that learning an “ equilibrium ” model ( i.e . a RBM that operates in the equilibrium regime ) is costly , and that most of the works in the Literature operate without any doubt in the OOE regime . In this regime , the equilibrium distribution of the machine at the end of the learning is significantly different from that of the dataset . Yet , despite this handicap , we show the OOE regime can be exploited to generate good samples at short training times , an observation that have already been exploited in practice [ 23 ] for convolution EBM . All our conclusions rely on the study of 9 datasets : MNIST [ 24 ] , whose results are discussed in the main-text , FashionMNIST [ 25 ] , Caltech101 Silhouettes , smallNORB dataset [ 26 ] , a human genome dataset [ 27 ] ( with similar dimensions as MNIST but more structured ) , the high-quality CelebA [ 28 ] projected in black and white , and in low definition but in color , and CIFAR [ 29 ] . The analysis of most of the last datasets are discussed only in the SM and show a similar behavior to the one observed on MNIST . Our paper is organized as follows . RBMs , their general principles and learning equations are defined in sec . 2 . We review previous related works in sec . 3 . In sec . 4 we define the different observables used to monitor the training and sampling . Finally , we discuss experimental results in sec . 5 . 2 Definition of the model . An RBM is a Markov random field with pairwise interactions defined on a bipartite graph of two non-interacting layers of variables : the visible nodes , v = vi , i = 1 , ... , Nv , represent the data , while the hidden ones , h = hj , j = 1 , ... , Nh , are the latent representations of this data and are there to build arbitrary dependencies between the visible units ( see sketch in fig . 1–A ) . Usually , the nodes are binary-valued in { 0 , 1 } , yet Gaussian or more arbitrary distributions on real-valued bounded support are also used , ultimately making RBMs adaptable for more heterogeneous datasets . Here , we deal only with binary { 0 , 1 } variables for both the visible and hidden nodes . Other approaches using truncated-Gaussian hidden units [ 30 ] , giving a ReLu type of activation functions for the hidden layer do work well , but in our experiments we observed qualitatively the similar dynamical behavior and therefore we will stick to binary hidden units for the rest of the article . The energy function of an RBM is taken as E [ v , h ; w , b , c ] = − ∑ ia viwiaha − ∑ i bivi − ∑ a caha , ( 1 ) with w the weight matrix and b , c the visible and hidden biases , respectively . The Boltzmann distribution is then given by p [ v , h|w , b , c ] = exp ( −E [ v , h ; w , b , c ] ) Z with Z = ∑ { v , h } e−E [ v , h ] , ( 2 ) being the partition function of the system . RBMs are usually trained using gradient ascent of the LL function of the training dataset D = { v ( 1 ) , · · · , v ( M ) } , being the LL defined as L ( w , b , c|D ) = M−1 M∑ m=1 ln p ( v = v ( m ) |w , b , c ) = M−1 M∑ m=1 ln ∑ { h } e−E [ v ( m ) , h ; w , b , c ] − lnZ . ( 3 ) The gradient is then composed of two terms : the first accounting for the interaction between the RBM ’ s response and the training set , and same for the second , but using the samples drawn by the machine itself . The expression of the LL gradient w.r.t . all the parameters is given by ∂L ∂wia = 〈viha〉D − 〈viha〉H , ∂L ∂bi = 〈vi〉D − 〈vi〉H and ∂L ∂ca = 〈ha〉D − 〈ha〉H , ( 4 ) where 〈f ( v , h ) 〉D = M−1 ∑ m ∑ { h } f ( v ( m ) , h ) p ( h|v ( m ) ) denotes an average over the dataset , and 〈f ( v , h ) 〉H , the average over the Boltzmann measure in Eq . 2 . A large part of the literature on RBMs focuses on schemes to approximate the r.h.s . of Eqs . 4 . In theory , these terms can be computed up to an arbitrary precision using parallel MCMC simulations as long as the number of MC steps are large enough to ensure a proper sampling of the equilibrium configuration space . In other words , sampling times should be larger than the MCMC MT . A naive way to implement this idea is to initialize each of these parallel Markov chains on random conditions , perform k MCMC steps , and use the final configurations to estimate the gradient . We call this scheme Rdm-k . This scheme is not used in practice because it would require too many k steps if the MT is too large . Many recipes have been proposed to shorten this sampling and approximate the negative term of the gradient , but since the introduction of the so-called contrastive divergence ( CD ) by Hinton [ 31 ] , only a limited number of schemes are used in practice . The first one , CD-k , initializes the MCMC simulation from the same data of the mini-batch used to compute the gradient , and performs k ∼ O ( 1 ) sampling steps . This approximation relies on the idea that the dataset must be a good approximation of the equilibrium samples of a well trained RBM , something that is not true for a poorly trained machine , whose typical samples are quite distant from the dataset . In the second commonly used method , the last configurations of the Markov chains are saved from one parameter update to the other , and then used to initialize the subsequent chain . As in the other methods , k MCMC steps are used to estimate the gradient for each update . This method is known as persistent-CD : PCD-k [ 32 ] , and takes advantage of the fact that the RBM ’ s parameters change smoothly during the learning and the same is expected for the models ’ equilibrium distribution1 . Finally , other approximations , such as mean-field or TAP equations [ 10 ] , or more elaborate approaches , such as the parallel tempering technique [ 33 , 34 ] , can be used . A common feature of all these recipes is that they rely on very few simulation steps to sample the Boltzmann distribution , and the trained model often generates rather bad configurations . To illustrate this last comment , let us discuss a provocative experiment shown on fig . 1–E where we try to sample new digits just by sampling the equilibrium measure of our trained RBMs using a MCMC initialized uniformly at random . Let us stress that this is not the standard set-up used in the field to generate new samples , but it is a strong test of the generation power of the trained model . We compare the outcome of the generated samples ( as function of the sampling time ) for three RBMs trained each following different recipes to compute the negative term of the gradient : CD-10 , PCD10 and Rdm-10 . At this point , k = 10 might seem a very small number . It is however the typical order of magnitude used in the Literature . The details of the learning protocol are discussed later in the text and summarized in the SM . The first striking point on fig . 1–E , is the inability of the CD-10 RBM to properly sample digits in this set-up . This problem has been reported before [ 35 , 8 ] , but since CD is part of the standard methods , we want to insist on this point . In addition , PCD-10 trained RBM generates proper digits but only after a very long sampling time , while with Rdm-10 , quite surprisingly , digits are properly generated after only 10 MCMC steps . Yet , samples obtained after much longer MCMC steps have unbalanced digits representations , like the 1 being over-represented . In fact , we observe that poorly trained models generate samples that do not respect the correct balance between the 10 digits present in the original database ( see fig . 1–D ) . We already mentioned the 1s in the Rdm-10 run , but it happens the same ( to a lesser extent ) with the 0 in the PCD-10 case . Alternatively to these generated samples , we show the evolution of the LL during the learning of these same three RBMs in fig . 1–C . By definition of the loss function , the LL is intended to be maximized , so it constitutes a traditional measure of the quality of the training . The LL of the models used for the sampling of fig . 1–E are highlighted in large symbols . This plot illustrates the lack of reliability of the LL to monitor the quality of the learning . For instance , CD-10 reaches quite high values of LL and yet , it can not generate a single digit . For the Rdm-10 is the other way around , it generates decent samples but with a very poor LL . Finally , we observe that the LL of PCD-10 reaches very high values , comparable to RBMs trained using up to k=104 MCMC steps to estimate the gradient . We will come back to this point . Our experiment highlights , first , that the LL can not be used all alone to quantify the generative power of an RBM , and second , that the quality of the generated samples may variate in a non-monotonic way during the generation sampling . 1This adiabatic scheme is thus valid as long as the relaxation time for the RBM parameters during the learning is much larger than the MCMC mixing times .

Summary: The paper discusses about some important factors affecting the results of RBMs.  1. The choice of mixing time.  2. The mixing time and the number of steps(k) affect two regimes of RBMs, namely equilibrium and out-of-equilibrium.  3. How to choose the number of steps(k). (A) short k for convincing samples in short learning times (B) large k for learning the correct equilibrium distribution of the RBM.