Cascade of phase transitions in the training of energy-based models
Authors: Dimitrios Bachtis
Giulio Biroli
Aurélien Decelle
Beatriz Seoane
Venue: NeurIPS 2024 poster (2024)
Keywords: Restricted Boltzmann Machine
Generative model
Phase transition
statistical physics
Energy-based model
Primary area: generative_models
TL;DR: We show theoretically and numerically that the training of Energy-based models undergoes several phase transitions.

Abstract: In this paper, we investigate the feature encoding process in a prototypical energy-based generative model, the Restricted Boltzmann Machine (RBM). We start with an analytical investigation using simplified architectures and data structures, and end with numerical analysis of real trainings on real datasets. Our study tracks the evolution of the model’s weight matrix through its singular value decomposition, revealing a series of thermodynamic phase transitions that shape the principal learning modes of the empirical probability distribution. We first describe this process analytically in several controlled setups that allow us to fully monitor the training dynamics until convergence. We then validate these findings by training the Bernoulli-Bernoulli RBM on real data sets. By studying the phase behavior over data sets of increasing dimension, we show that these phase transitions are genuine in the thermodynamic sense. Moreover, we propose a mean-field finite-size scaling hypothesis, confirming that the initial phase transition, reminiscent of the paramagnetic-to-ferromagnetic phase transition in mean-field ferromagnetism models, is governed by mean-field critical exponents.

Cascade of phase transitions in the training of Energy-based models
Dimitrios Bachtis'
Giulio Biroli
Aurélien D
Beatriz Scoane2.3
Laboratoire de Physique de I'Fcole Normale Supérieure, ENS, Université PSL, CNRS, Sorbonne Université, Université Paris Cité, F-75005 Paris, France. DDepattamento de Física Téórica I, Universidad Complutense, 28040 Madrid, Spain. 3 Université Paris-Saclay, CNRS, INRIA Tau team, LISN, 91190, Gif-sur-Y Yvette, France.
Abstract
In this paper, we investigate the feature encoding process in a prototypical energy- based generative model, the Restricted Boltzmann Machine (RBM). We start with an analytical investigation using simplified architectures and data structures. and end with numerical analysis of real trainings on real datasets. Our study tracks the evolution of the model's s weight matrix through its singular value decomposition, revealing series of phase transitions associated to progressive learning of the principal modes of the empirical probability distribution. The model first learns the center of mass of the modes and then progressively resolve all modes through a cascade of phase transitions. We first describe this process analytically in a controlled setup that allows us to study analytically the training dynamics. We then validate our theoretical results by training the Binary-Binary RBM on real datasets. By using datasets of increasing dimension, to sharp phase transitions in the high-dimensional limit. Morcover, we propose and test mean-field finite-size scaling hypothesis. This shows that the first phase transition is the same universality class of the one we studied analytically, and which is reminiscent of the mean-field paramagnetic-to-ferrmmaag phase transition.
show that learning indeed leads
Introduction
In recent years, we have witnessed impressive improvements of unsupervised models capable of generating more and more convincing artificial samples [ 2 B. Although energy-based models [ and variational approaches (5) have been use for decades, the emergence generative adversarial networks (6), followed by diffusion models (7]. has significantly improved the quality of outputs. Generative models are designed learn the empirical distribution datasets high-dimensional space, where the dataset is represented as Dirac-delta pointwise distribution. While different types of difficulties are encounter when training these models, there is a general lack understanding of how the learning mechanism is driven by the considered dataset. This article explores the dynamics of learning in neural networks, focusing on pattern formation. Understanding how this process shapes the learned probability distribution is complex. Previous studies g 9 the Restricted Boltzmann Machine (RBM) [10] showed that the singular vectors of the weight matrix initially evolve align with the principal directions of the dataset, with similar results in 3-layer Deep Boltzmann Machine [T]] Additionally, an analysis using data from the ID Ising model explained weight formation in an RBM with single hidden node as a reaction-diffusion process (112). The main contribution of this work is to demonstrate that the RBM undergoes a series of second-order
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

phase transitions during learning, each corresponding to the acquisition of new data features. This is shown theoretically with a simplified model and on correlated patterns; and confirmed numerically with real datasets, revealing progressive segmentation of the learned probability distribution into distinct parts and exhibiting second order phase transitions.
2 Related work
The learning behavior of neural networks has been explored in various settings. Early work on deep linear neural networks demonstrated that even simple models exhibit complex behaviors during training, such as exponential growth in model parameters [3 14] Using singular value decomposition (SVD) of the weight matrix, researchers revealed a hierarchical learning structure with rapid transitions to lower error solutions. Linear regression dynamics later showed a connection
between the SVD of the dataset and the double-descent phenomenon [15.. Similar dynamics were found in Gaussian- Gaasssan RBMs [9], where learning mechanisms led to rapid transitions for the modes of the model's weight matrix. In this context, the variance of the overall distribution is adjusted to that of the principal direction of the dataset, while the singular vectors of the weight matrix are aligned to that of the dataset. Unlike linear models, non- linear netrolsn supervised unsupervised ones, not exhibit partition of the input's space. Yet, linear model in general can not provide a multimodal partition of the input space, should it be in supervised or unsupervised context, difference with non-linear ones.
was then shown that the most common binary-binary RBMs exhibit very similar patterns at the beginning of learning. transitioning from paramagnetic to condensation phase in which the learned distribution splits into a multimodal distribution whose modes are linked to the SVD of the weight matrix [8] The description of this process motivated the use of RBMs to perform unsupervised hierarchical clustering of data 17]. The succession of phase transitions had been previously observed in the process of training Gaussian mixture
[18 19 20), and in the analysis of (eacher-student models using statistical mechanics [2I 22 The latter cases are casier to understand analytically due to the simplicity of the Gaussian mixture. Nevertheless, the learned features somewhat simpler, as they are mainly represented by the means and variances of the individual clusters. Recently, sequences of phase transitions have been used to explain the mechanism with which diffusion model are hierarchically shaping the mode of the reverse diffusion process [23] 25 and due to a spontaneous broken symmetry (26) after a linear phase converging toward central fixed-point. The common observation is that the learning of a distribution is, in many cases, obtained by succession of creation of modes performed through second order process where the variance in one direction first grow before splitting into two parts, and then the mechanism is repeated. This procedure in particular demonstrate a hierarchical splitting, where the system refined finer and finer scale of features as it adjust its parameters on a given dataset.
3 Definition of the model
An RBM is Markov random field with pairwise interactions a bipartite graph consisting of two layers of variables: visible nodes (v = [v,, i = ,N .N.))
representing the data, and hidden nodes (h- = { = L...., N/h) )) representing latent features that create dependencies between visible units. Typically, both visible and hidden nodes are binary (f0, 1)), though they can also be Gaussian 22 or other real-valued distributions, such as truncated Gaussian hidden units [28]. For our analytical computations, we use symmetric representation (+) for both visible and hidden nodes to avoid handling biases. However, in numerical simulations, we revert the standard (fo, 1)) representation. The energy function is defined as follows:
(1) with W the weight matrix and the visible and hidden biases. respectively. The Boltzmann distri- bution is then given by p(v, h W ,,, c] Z exp(-EE(e FE,, h: W, W. b,) with Z Σ(v,h) e Ele being the partition function of the system. RBMs are usually trained using gradient ascent of the log likelihood (LL) function of the training dataset D (2(1), (000). the LL is then defined
Elv, h: W, b, c) - -io Wiaih E, - Eo caha.
M
M
<(W, b, c(D) = M
In p(p = (1()) ( b. c) = M-1
In
e E(p(m) h.W.b.c]] In
(h}
2

---

The computation of the gradient is straightforward and made two terms: the first accounting for the interaction between the RBM's response and the training set, also called postive term, and same for the second, but using the samples drawn by the machine itself, also called negative term. The expression of the L.I gradient w.r.t. all the parameters given by
DC (ugha)D (veha) Re
ac ab;
(v.) (vihu and
ac 8eg
-ha)D (ha)
(2)
Oheia
where (f(v, D E Enn) f(u(m), h))(hp((m) denotes an average over the dataset, and (f, h)) the average the Boltzmann distribution p(v. h; W, a c). Most of the challenges in training RBMs stem from the intractable negative term, which has a computational complexity and lacks efficient approximations. Typically, Monte Carlo Markov Chain (MCMC) methods are used to estimate this term, but their mixing time is uncontrollable during practical learning, leading to potentially out-of -itl-f--quulibrium training (29].
of ~ O(2min(M..)))))))
This work focuses on the initial phase of learning and the emergence of modes in the learned distribution from the gradient dynamics given by Eq. (2). In the following section, we first analytically characterize the early dynamics in simple setting, showing how it undergoes multiple second-order phase transitions. We then numerically investigate these effects on real datasets.
Theory of learning dynamics for simplified high-dimensional models of data
We develop the theoretical analysis by focusing on simplified high-dimensional probability distribu- tions that concentrate around different regions, or lumps, in the space of visible variables. Our aim is to analyze how the RBM learns the positions of these lumps, which represent, in simplified setting, the features present in the data. In order to simplify the analysis, we will consider the Sinary-Gaussian RBM (BG-RBM) defined below, yet the same results can be derived for the Binary-Binary RBM (BB-RBM) as shown in the SI B
4.1 Learning two features through a phase transition
We consider the following simplified setting: hidden nodes and put the biases to zero b = 0 and c = 0. As model of data, we consider a Mattis model with preferred direction & for the ground state, following the distribution
will be using vú #1 visible nodes, Gaussian
B
No
2
PMattis
exp
Σ
Mittis
2N,
where 3 = 11/T the inverse temperature and & = +1 represents pattern encoded in the model as Mattis state B00 31. In a Mattis model, represents a preferred direction of the model for large values of B, and in this simple case (with only one pattern) there is no need to specify distribution as long as its elements are + The Mattis model presents high-temperature phase with a single mode centred over zero magnetization m N. >. (w;) = for 3 B, (where (.) is the average w.r.t. the Boltzmann distribution) while in the low- now-emmpertture regime, 3 >>> Be, the model exhibits phase transition between two symmetric modes m = = (3) (3g- = 1). Henceforth, we shall focus on the regime 3 > 3e where the data distribution is concentrated on two lumps. From the analytical point of view, we can compute all interesting quantities in the thermodynamic limit N, - co. In order to keep the computation simple, we will characterize here the dynamics of the system when performing the learning using a BG-RBM [27] with one single hidden node. In our setting we assume that the distribution of the hidden node is centered in zero (i.e. there is no hidden bias) and that the variance is of = 1/N₂ (we discuss the reason for the scaling in Ab. The distribution is then
Σ vghw, - h22 N₂ 2
[E u,w,)² 2N,
PBG (h, v) = ZBG
exp
(1) =
Z exp
Using this model for the learning, the time evolution of the weights is given by the gradient. With BG- RBM we have that (H:h) N, Ej wy (1(()))) where the last average is taken over a distribution PBG (v). We can now easily compute the positive and negative term of the gradient w.r.t. the weight
'The special case in which all elements & = 1 is the well-known Curie- Weiss model in ferromagnetism.
3

---

matrix, For the positive term we obtain that (izey)D = f:Eym² where m = tanh (8m). The negative term can also be computed in the thermodynamic limit (1:2)) tanh(h wx) tanh (h* wy) with h" = n Σx Wi tanh(h" wrg.. If we take the limit of a very small learning rate, we can convert the parameter update rule using the gradient into a time differential equation for the parameters of the RBM, where is the learning time:
dwa = E
h" tanh(h w.))
dt
with the learning rate We analyze two distinct regimes for the dynamics. First, assuming that the weights are small at the beginning of the learning, we get that h* = O. We can then solve the Eq. (3) in this regime obtaining the evolution of the weights toward the direction & by projecting the differential equation on this preferred direction. Defining Ug N-122 Σ3 Erwy, we obtain
dUg m Ug. thus Ugg U
di
This illustrates that the weights are growing in the direction of & while the projection on any orthogonal direction stays constant. As the weights grow larger, the solution for h" will depart from zero. Then the correlation between the RBM visible variables starts to grow
1 withh exp (-N (uEe))RBM z
N.h² + Σx
W.W, No (1-Ex wg/N.)'
which means that the susceptibilityy X Eis (,tr( RBM, that is, the response of the system w.r.t an external perturbation, diverges when N. Ex w ~ 1, thus exhibiting a second order phase transition during the learning. Interestingly, x diverges y = (where N Ex Wā plays here then the role of the inverse temperature in the standard physical models) thus corresponding
power law with (critical) exponent
the mean-field universality class [32]. Finally, we can study the regime where the weights are not small. In that case, we can first observe that the evolution of the directions orthogonal to & cancel when the weights W align totally with the & at the end of the training. Finally, taking Wi = E, the gradient projected along & at stationarity imposes
wm h² tanh (w) and thus w /B and h" -Bm.
We confirm the main results of this section numerically in Fig. D showing they hold accurately even for moderate values of N,. The sum of the weights grows exponentially, following the magnetization squared (considering the learning rate), and the weights align with the direction E, while the norm of the weight vector converges towards vB. Additional analysis details and extended computations for the binary-binary RBM case, which is slightly more involved, are provided in the SI.
4.2 Learning multiple features though cascade of phase transitions
We consider now the case in which the data are characterized by more than two features. For concreteness, we focus on the case in which the data is drawn from the probability distribu- tion of the Hopfield model [31) with two patterns & and e², using the Hamiltonian H [u] =
8 E
( No
2 2 The generalization a larger (but finite) number of patterns is straight-
2
2N,
forward. Following 33 we consider the case in which the patterns are correlated and defined as: gl = =nn + '72 and g² = ny n²² n7' is a vector whose first N2 1+ 2 components are equal #1 with equal probability, and the remaining ones are zero (0 < x < 1). Whereas ry² is a vector whose last N, 1- components are equal to +1 with equal probability, and the remaining ones are zero When T ~ x this model is in a symmetry broken phase in which the measure is supported by four different lumps centred in = and = Analogously to what was done previously, we now consider a BG-RBM with a number of hidden nodes equal to the number of patterns where again both hidden nodes are centred in zero and have variance of 1/N.. The Hamiltonian is then given by H(e, h] - Σ in u.h, Wio + Σ. N/,, which corresponds to Hopfield model [II] with patterns w and w². The analysis presented in the previous section can be generalized to this case (see SI for more details) and one finds the dynamical equations for the evolution of the patterns:
dwf 1 No
Σ (wws)Duu
Σ (DrU, (1:U))RBMuS N.
1
dt
PIn the rest of the derivation, will remove it since it be absorb in redefinition of the time.

---

10-
1.20-
0.8
12
Mc
(6" 6
Llas
100
0.75
w.. (8 5
My 18" 6
Data wili bame
0.50
2-14 Ny e eaeu
DalS Ny Soau
022
Equesrid squoait
0.25
53
0.00
500
1000
1900
9000
Number of epachs
Number of spochs
Learning behavior of the BG-RBM with one hidden node, using data from the Mattis model at different inverse temperaturss system sizes and learning rates B, Ny, e. The argument of the exponential curves is set to m²cN. cN,, where is the learning rate. Inser: (top) behavior of the susceptibility X (bottom) magnetization h" of the learning RBM. The vertical line marks the point at which the susceptibility diverges, indicating the onset of spontaneous magnetization. Right: Learning curves for RBMs learning two correlated patterns. The dashed curves represent the weights of the two hidden nodes projected onto ç 5², while the dashed-dotted curves are projected onto &² <². Inset: Exponential growth during the two phases: top shows growth in the direction e + E² at a rate (1 + x2/2, and bottom shows growth in the direction ç _² rate p² (1 - k2/22. The arguments of the exponentials are not adjusted.
Figure 1:
As shown in the SI C (Dru))D =r'n + +²nnin where p, are function of B (and 3-1 K. r p). Note that this factorization of the correlation matrix is precisely its spectral eigendecomposition, which means that 17' and n7² are nothing but the principal directions of the standard principal component analysis (PCA). the beginning of the training dynamies the RBM in its high-temperaturee disordered phase, hence the second term of the RHS of Eq. (4) is zero. The weights w and w² have therefore an exponential growth the directions 17' and 1f", whereas the other components do not evolve. If the initial condition for the weights is very small, as we assume for simplicity, one can then write:
wh (1)
172
a a= 1,2
where we have neglected the small remaining components; 2nd and 2d are the projections of the initial condition along the directions 17 and v Since r > p, on the timescale (log NN./ (+²(1++ + k)) the component of the whs along 17" becomes of order one whereas the one over n² is still negligible. In this regime, the RBM is just like the one consider in the previous section with single pattern: the system will align with a single pattern that is given in that case by n7' x E + g², and it has a phase transition at the time ty:
2. 2.2 ( 1
No
At lr, the RBM learns that the data can be splitted in two groups centred in 1,7 but it does not have yet learned that each one of these two groups consist two lumps centred in 5² and g² (and respectively - and - The training dynamics after L can also be analyzed: the components of the weight vectors along 77' evolve and settle on timescales of order one to a value which is dependent on the initial condition (see the eq. in the SI). In the meanwhile, the components along ny keep growing; at a timescale (log (p² -i)) (quite larger than t in the limit N, - 20) they become of order In order analyze easily this regime, let's consider first the simple case in which the initial condition on the weights is such that w (0) n7² =²(() n and w (0) n7' = =²( n In this case, RBM is a Hopfield model with log likelihood:
write w A(t)m² + B(t) and w² A(t)m B(t)m? The corresponding
Σ Emwu)) 2N, 2A(1)2 (2
2N. + 2B(22 2N,
At t,, when (1+x A(t?)² one has the first transition which the RBM measure breaks in two lumps pointing in the direction i-7², as we explained above. this regime B(t) still negligible but keeps increasing with exponential rate. Using the results of [33, one finds that
5

---

W singulaa values
Jeft singula veciors
1.0
10-40
7.5
0.0
1t2 upcA e|
5.0
0.5
2.5
0.2
Aperiica
0.0
0.2
0.0 MCA
0.2
20000
4000
5000
epochs
epochs
spochet0)
spodu=6511
(podu=00)
opoche=200
spoche60011
0.2
0.0
-0.2
0.00
0:25-0.25
0.00
0.25-0.25
0.00
0.25-0.25
0.00
0.25-0.25
0.00
0.25- 0.25
0.00
0.25
201
m
Figure 2: Human genome dataset. Progressive coding of the main directions of the dataset when training an RBM with the human genome dataset (44. A, we show the dataset projected along the first two principal components of the dataset na with with (d)) referring the different entries in the dataset, i.e. an human individual. Points are colored according to the individual continental origin. In B, we show the evolution of the singular values Wa of the RBM weight matrix W as function of the number of training epochs, and in we show the scalar product of the corresponding singular vectors u, with the corresponding PCA component na. In D, we show the magnetization of the samples generated by the model different epochs, projected along the first two eigenvectors of W, which shows that the specialization of the model occurs through the progressive encoding of the main modes of the data in W.
1, and mCA 7h xd() /VV.
when 1 1=x = 1, second phase transition takes place. This defines time L77 which the probability measure of the RBM breaks from two lumps to four lumps, each one centred around one of the four directions = = We have considered special initial condition, but the phenomenon we found is general. In fact, for any initial condition one can show that the dynamical equations have instability on the timescale tiı, which generically induces the second symmetry breaking transition. On Fig. 1 right panel, we illustrate the exponential growth as described by the theory, show how these phase transitions agreement with previous work 99 and how the phase space is split during training time. At the end of the training, the patterns spanned by 1::2 since the likelihood is invariant by rotations in this subspace. In fact, we often found that the patterns are not perfectly aligned because we are not forcing the weights be binary. This analysis can naturally extend to more than two patterns, typically resulting in a cascade of phase transitions. In this process, the RBM progressively learns the data features, starting from coarse- coarse-grained grained version (just the center of mass) and gradually refining until all patterns are learned. The analysis done on the BG-RBM can of course be repeated on the BB-RBM (as done for the case with one mode in the SI B The main difference at that level between the two models is that in order have a retrieval phase, the BB-RBM needs to encode the patterns on an extensive number of hidden nodes (proportional to N.), while the BG-RBM needs only as many patterns as hidden nodes. Both models can match perfectly the dataset in the limit N, - oc, but we might encounter discrepancies for finite size. However, when dealing with real datasets, by construction, the BG-RBM can not reproduce higher-order correlations and therefore is less interesting than the BB case.
toward the two directions. In the SI 4.2
in very good
given by w = and w² g² modulo rotation in the subspace
Numerical Analysis
In the previous sections, we examined the learning process in simplified setups, in order to able to develop analytical treatment. In particular, have shown analytically in simple setting how the

---

weight matrix is shaped by the patterns present in the dataset and how the learning process dynamics triggered by the PCA components (the r7' and r7² in the previous example) and not by the learning of the encoded patterns (5¹ and {²). Moreover, we have shown that each time the RBMs learn new direction, the susceptibility of the system diverges with a precise power law everytime the RBM learns new direction, which is also associated with the development of new modes in the probability measure. In this section we will also show that the insights gained from this simplified analysis are also applicable to understanding the learning process of a Sinary-Binan RBM (BB-BBM) with many hidden nodes trained with real data sets. The details about the training procedure are given in SI E. For this purpose, we will consider 3 real data sets: (i) The Human Genome Dataset (HGD), (i) MNIST and (iii) CelebA, see details in the SI[D To show the occurrence of bonafide phase transitions, it is important to show the effect of increasing the system size (which transforms cross-overs in sharp transitions in the large N₂ limit). We will therefore resize these data sets in different dimensions by adjusting their resolution, i.e. by changing N,, while maintaining comparable statistical properties. Detailed information about the scaling process can be found in the SI
In real training processes, the machine is expected to gradually learn different patterns 88, from the data, as described in the previous sections. However, the identification of these patterns and their relationship to the statistical properties of the dataset remains unclear. Previous research [9 g 35 has shown that RBM training begins with the stepwise encoding of the most significant principal components of the dataset, (17*), ). which are the cigenvectors of the sample covariance matrix with the highest eigenvalues, on the SVD decomposition of its weight matrix Wia = Ea uaz i2 where u E IR No and t E IR., denote the left and right singular vectors corresponding to the singular value Wa- These vectors form orthonormal bases in R N. and IK N respectively, where the index ranges from to (N,, Nn) and the singular values Wa are arranged in descending order. At the beginning of the learning process, the left singular vectors, u", gradually align a-to-a to 17"'. This is consistent with the analytical results in our simple setting in the previous sections. In analogy to the mean-field magnetic models proposed in the previous section, the role of decreasing temperature is played by the increasing magnitude of the singular value wa associated with each mode o, and should lead to a series of phase transitions where the RBM measure splits into increasingly more and more modes. We show in Fig. 2 that these phenomena are at play by focusing on the evolution of the SVD of the RBM weight matrix when trained with the HGD dataset.
In panel A we show the first two principal components of the dataset, which highlights its strong multimodal structure, several distant clusters appear (in this case, they related to the continental origin of the individuals at hand). In Fig. D we show the sharp and sudden increases of the singular values Wa. as expected from our theoretical analysis, and in Fig. 2-C the evolution of the scalar product between u, and n, as a function of the number of training epochs. Different colors indicate different values of a. As expected, the modes are progressively expressed during training, and the first two singular vectors match the two principal directions of the dataset for a while. This last figure also shows us that the alignment with the PCA is only temporary (a limitation of current theoretical approaches), as the machine finds better patterns to encode the data as training progresses.
The progressive splitting of the RBM measure during the training dynamics is shown in Fig. D, for which we use N₂ = 1000 independent samples generated with the model trained up to a different number of epochs (the colors refer to the same epochs highlighted with vertical lines in Figs. 2 B and C). For visualization, we show the samples projected onto the right singular vectors of W. the magnetizations m.,- v u / /N,, with a 1, 2. At the beginning of training, the data points essentially Gaussian distributed, and the growth of wo over 4 is related to the splitting of the data into two different clusters on the mi axis, and the emergence of Wa is related to a second splitting on the m2 axis. At this stage of training, the projections along all subsequent directions are Gaussian distributed as they are the result of sum of random numbers (fixed by the random initialization of the weight matrix). This progressive splitting is crucial to express the diversity of the dataset shown in Fig. 2 A, and can be successfully used extract relational trees to cluster data points, as recently shown in Ref. [16]. The details about the numerical analysis are given in
At the beginning of training, when only a singular value has been expressed, and thus W R wunl the transition of the feature encoding process is analogous to the phase transition from the paramagnetic to the ferromagnetic phase in the Mattis model mentioned above with pattern u. The detailed justification can be found in SI[F] Our analysis allows us to define an effective temperature, linked to the eigenmode of W as B w 16. Now, since the critical temperature of the Mattis model is 3. = 1, we can show that the BB-RBM will condensate when the first eigenmode of the model
7

---

N,
3
7
14
10
28
30
56
10
N₂
28
10
Xoa yider
10y
2.5
10-1
1/(w]
wi)
10
1000
(wff
w wN)
109
ZPh
0.05//1/ /11 1/
iP
N,
28
10
104
Toon [MCMC stepe.
10-1
109
102
109
10-1
4/0M
10"
6
3
0
1000
2000
3000
4000
epocls
Figure 3: Traning with the MNIST dataset. A we show the evolution of the singular values of the RBM's coupling matrix W function of the training time. In B we show the evolution of the susceptibilities associated with the magnetizations along the right singular vectors of W, m. (u, v) /Nv. In both figures, we consider the standard N. 28 MNIST dataset, different colors refer to different modes. C we show the susceptibility associated with the overlaps q and between visible and hidden variables. In D we show the susceptibility of the first mode function of the first singular value w/j obtained with trainings on MNIST data scaled to different system sizes above and below L = 28. The numerical curves
compared with the theoretical expectation using the Mattis model in Eq. (6) using W1.c = 4.45. The same data are shown in E, scaled using the mean-field finite-size scaling ansatz of Eq. 7 In F, show the first 10 modes' susceptibilities Xm. as a function of their corresponding singular value Wa and compare them with the theoretical curve in D. In G, we show the MCMC relaxation time of the machines trained with different N,, datasets as a function of w1, together with the theoretical expectation for local moves in dashed lines.
reaches w. = 4, see S In a real training, we also have visible and hidden bias e which could easily change the model towards a random field Mattis model, which leads us to expect a slightly higher critical point but a very similar ferromagnetic phase transition, and in particular, it should not change the transition's mean-field universality class.
To show that there a cascade of transitions and that what was found for the HGD also holds for other datasets, now train the RBM with the MNIST dataset. In Fig, B plot the evolution of the singular values Wa along the training, which clearly show the progressive encoding of patterns. The progressive splitting of the RBM measure into clusters and the presence of phase transition can be monitored by measuring the variance of the distribution of the visible magnetizations m, along the a-th mode the analogous hidden magnetizations m. h- ū" / //NN, obtained using the hidden units. The variance of the magnetization multiplied by the number of variables used to compute it and 3, is related to the magnetic susceptibility via the fluctuation dissipation theorem, which means that
Xm = Nv (mm²)-(m)) (m)²² T' (m) /dh.
(5)
here (-) refers to the equilibrium measure with respect to RBM's Gibbs measure P( h), in practice estimated as the average over N₂ independent MCMC runs. It is well known that the magnetic susceptibilityy should diverge only limited by the overall system size N VN,, Nr in finite systems. These phenomena indeed takes place also in the RBM. We show in Fig. B the evolution of the X obtained using the magnetizations obtained along the different modes of W. As anticipated, the susceptibility Xm
vicinity of a second order phase transition and that such growth in

---

A
10a
Xmn, N N-ddpp
10
B
E
100
10
1287
(m)
0.00
109
////0
vf)
10-2
0.25
2bb =4.4
loop speed
3
5
500
0.25
10/56
10
(w]
(m))
0.00
100/5A
1000/66
109
0.25
1100000/56
10
Xm, N-1/
0.25
100
10-1
(m/))
0.00
60G
-0.25
3
9
15
1000
N
0.11
0.0 0.1
Figure 4: Training with the CELEBA and HGD datasets: In A, we plot the hidden susceptibility for different system sizes in the CELEBA dataset, with dashed lines indicating the expected divergence Wi,e- 4. In B, we show the mean-field FFS associated with the first transition using mean-field exponents. In C and D, we present the visible susceptibility for the first phase transition in the HGD dataset, using Wi.e 5.25 for scaling. E. typical hysteresis the low-temperature phase is illustrated for CELEBA (128 x 128), similar to the mean-fiehe Ising model in external fields.
associated to the magnetization m along the first mode, sharply grows as wi approaches 4, but it is more remarkable that this behavior not only restricted to the first mode, but it also reproduced by the subsequent modes in a step-wise process. According to the mapping between the low-rank RBM and the Mattis (or equivalently, the Heirs) model, we should expect that our Xms at least for the first mode a = 1 should behave as
4
Xm
(6)
3.
W. w
when approaching the critical point, which is equivalent to stating that the critical exponent is 7
Here, the factor 4 in the numerator is related the fact that the susceptibility obtained with {0,
variables is 4 times the standard one obtained with Ising spins, and that / = w²/16, /16. In Fig. 3-D. we function of wi using RBMs trained with
3g.33
show the susceptibility associated with the first mode MNIST data rescaled to different dimensions. As mentioned carlier, the growth of the susceptibility limited by the system size Ny. However, if we look increasingly larger sizes, we can observe the growth over several decades. This shows that at the transition we observe the Mattis/Curie- Weiss behavior of Eq. 6 as shown in the black dashed line, where the only adjustable parameter was the critical point Wi,c 4.45 (i.e. there is no adjustable fac-fo)
One of the crucial tests to ensure that finite-size transition is bona fide phase transition is study its behavior by changing the number of degrees of freedom. One of the standard tools to do this is make use of the so-called finite-size scaling (FSS) ansatz, motivated by renormalization group arguments [36] 37 38). Mean-fiet field models follow a modified FSS ansatz which was first studied
in 139 In particular, the FSS ansatz for the susceptibility is
x = N vid, Φ (N pid |B Bel).
with d6-) a size-independent scaling function, N = NN,N, is the effective size of our model and y 1, 1/2 and d,, = expected in the mean-field universality class. We test this ansatz in Fig. E showing that it does succeed to scale the finite-size data in the critical region, especially in the largest system sizes, which confirms both the mean-fin mean-field universality class and the prevalence of the transition in the thermodynamic limit. In Fig. A and B, and Fig. CC and D, we show that the indicators of a phase transitions- growth of the susceptibility and holds for CelebA and HGD datasets. Finally, final piece of evidence of the existence of a phase transition is presented in Fig. AE F, where we show that after the continuous transition has taken place, one can induce a discontinuous transition and hysteresis effects by applying field in the direction of
field finite size scaling-also

---

the learned pattern, in full agreement with what observed for standard phase transitions, the details of the analysis can be found in SIIE and further theoretical insights into the relationship between hysteresis and discontinuous transitions can be found in G
All the discussion so far has been mainly concerned with the first phase transition, when the RBM learns the first mode. But we have discussed in Fig. 3-B that an entire sequence of step-wise phase transitions occurred in the rest of the W- modes. In Fig. B-F we show each of these mode susceptibilities Xm, as a function of their corresponding singular value. They show extremely similar divergent behavior with respect to the mode 1, with an apparent slight variation of the critical point for each mode, although all appear to remain close to the predicted Wa ~ 4, suggesting that the subsequent transitions may be of similar mean-fiell nature. Exceeding second-order phase transitions has very strong impact on the overall quality of the training, in particular on the quality of the log- ligliikllhoodd gradient estimated by MCMC dynamics. Indeed, second-order transitions are associated with a well-known arresting effect, known critical slowing-down behavior, by which the thermalization times diverge with the correlation length < x |/3 - 8el-ve, where is the dynamical critical exponent, which is 2 for local and non-conserved order parameter moves in mean-field, making the thermalization of large systems extremely difficult in practice. We show that our exponential relaxation times diverge exactly as predicted in Fig. B-G. This has significant impact on the quality of models trained with maximum likelihood approaches, as these methods rely on MCMC to estimate gradients. It is therefore expected that MCMC mixing times increase sharply each time a mode is coded, which can be prohibitive for clustered and high-dimensional datasets. Recent studies have shown that pre- pre-training a low-rank RBM using other methods (and thus bypassing the initial phase transitions) can be very effective in improving the models in clustered datasets 440). However, we emphasize that the cascade of phase transitions described in this paper occurs regardless of the training scheme or whether the Markov chains reach equilibrium. This is discussed further in SI H
Extensions and limitations- All these results can be studied in detail for RBMs thanks to the fact that we can analytically deal with the Hamiltonian. However, our results can be extended to the case of Deep Boltzmann Machines, where previous works have also computed the phase diagram, which also based the SVD decomposition of the weighing matrices [1]]. but also in diffusion models where phase transitions linked to the learning has also been described [24] 26]. It therefore stands to reason that similar phenomena occur with even more complex models such Convolutional EBM, but where it is not clear how the parameters of the model can be decomposed. A first test would be to see what the projection of the generated data would look like in the different phases of learning.
Conclusions
In this paper, first characterized the learning mechanism of RBMs using simplified setting with dataset provided by a simple teacher model. We used two examples: one with two symmetric clusters and another with four correlated clusters. Our results show that the learning dynamics identify modes by exponential growth in the directions of the clusters dominated by the variances of these clusters. The theory predicts the timing of the first phase transitions and agrees well with [8]. Numerically, we have confirmed the existence of a cascade of phase transitions associated with the growing modes w, and accompanied by divergent susceptibility. Finite-size scaling suggests that these transitions are critical and fall into the class of mean-field universality. This set of phase transitions likely goes beyond RBMs and offers insights into learning mechanisms, particularly for generative models. These transitions have significant implications for both training and understanding the learned features. During training, each transition is associated with divergent MCMC relaxation time, which requires careful handling to properly train the model. In addition, the hysteresis phenomenon ensures that the learning trajectory involves second-orded phase transitions, which are beneficial for tracking the emergence of modes in the learned distribution. However, changing parameters (such as the local bias) could lead to first-order transitions that are detrimental to sampling and could explain the ineffectiveness of parallel tempering in the presence of temperature changes. In practice, our analysis shows that the principal directions of the weight matrix contain valuable information for understanding the learned model.
10

---

7 acknowledgments
The authors would like to thank Moreno-Gordo for his work and discussions during the initial phase of this project. The authors (A.D. and B.S.) acknowledge financial support by the Comu- nidad de Madrid and the Complutense University of Madrid (UCM) through the Atracción de Talento program (Refs. (019-TI/TIC-13298) and to project PID2021-125506NA, by the Ministerio de Economía y Competitividad, Agencia Estatal de Investigación (MICIU/AEI /10. 13039/51 10001 1033), and the Fondo Europeo de Desarrollo Regional (FEDER, UE).
financed
References
[1] Christian Ledig. Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham. Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681-4690, 2017.
[2] Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.
[3] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[4] Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energy- based learning. Predicting structured data, 1(0), 2006.
[5] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
[6] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 13:11)):3991444) 2020.
[7] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265 PMLR, 2015.
[8] Aurélien Decelle, Giancarlo Fissore, and Cyril Furtlehner. Thermodynamics of restricted boltzmann machines and related learning dynamics. Journal of Statistical Physics, 172:1576- 1608, 2018
[9] Aurélien Decelle, Giancarlo Fissore, and Cyril Furtlehner. Spectral dynamics of learning in restricted boltzmann machines. Europhysics Letters, 1 11966);0001,, 2017.
[10] Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statistics, pages 448-455. PMLR, 2009.
[11] Yuma Ichikawa and Koji Hukushima. Statistical-mechniie study of deep boltzmann machine given weight parameters after training by singular value decomposition. Journal of the Physical Society of Japan, 91 1):1 14001. 2022.
[12] Moshir Harsh, Jérôme Tubiana, Simona Cocco, and Remi Monasson. 'place-cell' emergence and learning of invariant data with restricted boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space. Journal of Physics A: Mathematical and Theoretical, 53(17):17400)) 2020.
[13] Andrew M Saxe, James I. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning deep linear neural networks. arXiv preprint arXiv:1312.6122, 2013.
[14] Andrew M Saxe, James L McClellans, and Surya Ganguli. Learning hierarchical categories in deep neural networks. Proceedings the Annual Meeting the Cognitive Science Society, volume 35, 2013.
11

---

[15] Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428-444 4466, 2020.
[16] Aurélien Decelle, Beatriz Scoane, and Lorenzo Rosset. Unsupervised hierarchical clustering us- ing the learning dynamics of restricted boltzmann machines. Physical Review E, 108(1):014)) 2023.
[17] Jorge Cernandez-De-Coe Ohe-CossiooDiaae -Daz, Thomas Tulinski, Simona Cocco, and Rémi Monasson. Replica symmetry breaking and clustering phase transitions in undersampled restricted boltz- mann machines. 2024.
[18] Kenneth Rose, Eitan Gurewitz, and Geoffrey C Fox. Statistical mechanics and phase transitions in clustering. Physical review letters, 65(8):945, 1990.
[19] David Miller and Kenneth Rose. Hierarchical, unsupervised learning with growing via phase transitions. Neural Computation, 8(2):425 450, 1996.
[20] Tony Bonnaire, Aurélien Decelle, and Nabila Aghanim. Cascade of phase transitions for multiscale clustering. Physical Review E, 103(1):012105, 2021..
[21] N Barkai and Haim Sompolinsky. Statistical mechanics of the maximun-likelibod density estimation. Physical Review E, 5763):1766, 1994.
[22] Thibault Lesieur, Caterina De Bacco, Jess Banks, Florent Krzakala, Cris Moore, and Lenka Zdeborová. Phase transitions and optimal algorithms in nigh-dimensional gaussian mixture clus- tering. In 2016 54th Annual Allerton Conference Communication, Control, and Computing (Allerton), pages 601-608 IEFE, 2016.
[23] Giulio Biroli and Mare Mézard. Generative diffusion in very large dimensions. 2023(9): 2023.
[24] Giulio Biroli, Tony Bonnaire, Valentin de Bortoli, and Mare Mézard. Dynamical regimes of diffusion models. arXiv preprint arXiv:2402.1844 18491, 2024.
[25] Antonio Sclocchi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data. arXiv preprint arXiv:2402.16991, 2024.
[26] Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[27] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[28] V. Nair and G.E. Hinton. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pages 807- 814, 2010.
[29] Aurélien Decelle, Cyril Furtlehner, and Beatriz Scoane. Equilibrium and con-equilibrium regimes in the learning of restricted boltzmann machines. Advances in Neural Information Processing Systems, 14:5345-5350 2021.
[30] DC Mattis. Solvable spin systems with random interactions. Physics Letters A, 56(5):421-422 1976.
[31] John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554 -25585588588 1982.
[32] Giorgio Parisi and Ramamurti Shankar. Statistical field theory. 1988.
[33] Francisco A Tamarit and Evaldo MF Curado. Pair-correlated patterns in hopfield model of neural networks. Journal of statistical physics, 62:473-480, 1991.
[34] 1000 Genomes Project Consortium al. global reference for human genetic variation. Nature, 526(757 1)66, 2015.
12

---

[35] Aurélien Decelle and Cyril Furtlehner. Restricted boltzmann machine: Recent advances and mean-field theory. Chinese Physics B, 30(4)2040202, 2021.
[36] Kurt Binder. Finite size scaling analysis ising model block distribution functions. Zeitschrift für Physik B Condensed Matter, 43:119-1400 1981.
[37] Daniel Amit and Victor Martin-Mayon Field theory, the renormalization group, and critical phenomena: graphs to computers. World Scientific Publishing Company, 2005.
[38] John Cardy. Finite-size scaling. Elsevier, 2012.
[39] E Brézin. An investigation of finite size scaling. Journal de Physique, 43(1):15-22 1982.
[40] Nicolas Béreux, Aurélien Decelle, Cyril Furtlehner, Lorenzo Rosset, and Beatriz Seoane. Fast, accurate training and sampling of restricted boltzmann machines. arXiv preprint urXiv:2405.15376 2024.
[41] Mare Mézard. Mean-field message-passing passsng equations in the hopfield model and its generaliza- tions. Physical Review E, 95(2):022117, 2017.
[42] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233 1998.
[43] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CoRR, abs/1710.10196. 2017.
[44] Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likeli- hood gradient. In Proceedings of the 25th international conference on Machine learning, pages 1064-1071, 2008.
[45] Frank den Hollander. Metastability under stochastic dynamics. Stochastic Processes and their Applications, 114(1):1-26, 2004.
[46] Anton Bovier. Metastability, pages 177-221. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009.
[47] Paul M Chaikin, Tom C Lubensky, and Thomas A Witten. Principles of condensed matter physics, volume 10. Cambridge university press Cambridge, 1995.
[48] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1418:1777)-880, 2002.
[49] Erik Nijkamp. Mitch Hill, Song-Cle Cong-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run meme toward energy-based model. Advances in Neural Information Processing Systems, 32:5232-5242 2019.
13

---

A Binary-Gauss RBM
We add some technical details to the derivation of the dynamical process. first recall the definition of general BG-RBM first. BG-RBMs consist of a bipartite model where the visible nodes binary (in our case 1) and the hidden nodes are Gaussians. The Hamiltonian follows the same expression the with variance o. In our analysis where we use only one or few hidden nodes, it is important that the variance scales as the inverse of the system's size of. 1/N, in order for the Hamiltonian of system to be an extensive property (proportional to N₂). This scaling is crucial for the analysis: if the energy term scales as system's size, then it is possible have two distinct phases: for small weight magnitude, the system is disordered (it does not polarize in any particular direction) while for large values of w the system will polarize toward one the direction encoded in the weight matrix. A more detailed description can be found in 30
Eq. | The hidden nodes are Gaussian random variables centred in zero
Now, recall that we consider the Mattis model, biased toward a pattern & for generating the dataset
No
3
(M)
exp
Σ
ZMattii
2.V,
where 3 the inverse temperature 1/T and 5: = +1 represents a potential pattern direction. The Mattis model presents high-temperature phase with single model centred over magnetization O for 3 < B. while in the low-temperature regime, 3 > Be. the model exhibits
E, N phase transition between two symmetric modes = + From the analytical point of view, we can compute all interesting quantities in the thermodynamies limit N, + cc. The RBM's distribution is given by
Σ v.hw; h22 Nr
PRBMI (h, v) = ZRBM
exp
2
[E );w;)?
(RBM(U) Z exp
2N,
Using this model for the learning, the time evolution of the weights given by the gradient. With BG-RBM we have that
/ dhaphp(((o)pp())
(u,h) Σ
dhv,hp(v,h) =
=
(8)
{w}
{0}
1
uyp(D)/
1 N
Σ
my (vrry)p
(9)
N
where the last average is taken over a distribution p(v). We can now easily compute the positive and negative term of the gradient w.rt. the weight matrix. For the positive term, assuming that 3 >> 1, obtain that
Znattii / dm
m² 13N,
Σ
2 + m3 Σ Ekuk
B. exp {w}
Zssattis /
m² + Σ log cosh (3)m)) -BN,
dm (anh((Emm) (8n exp
Evaluating the saddle point of the argument of the exponential (which is the same the one for the partition function) we have that
(mur)) Ei,m² where tanh (3m)
14

---

The negative term can also be computed in the thermodynamic limit (D:U))RBM =
N2
dh
Villy exp Σ
vchwox
ZUBM
2
= dh
Nr
tanh (hwr)) tanh(hur,) exp
Σ
log (2 cosh(hwk))
ZRBM
2
k
= tanh(h' (h" w;) tanh(h wy) with h"
Σ
wh tanh(h w))
Nv
where the last line is obtain by taking the saddle point the integral over h, (h" corresponding to the extremum). We can now express the gradient
dwa dt
E
Σ m b
Σ
Wy tanh(h ww) fanh(h"w;;)
Nv
N.
k
1
E Σ Nv
Eowym? h² tanh(h
Assuming first that the weights are small we get that h We can solve the gradient's equations in this regime. In such case, the only solution for the saddle point equation of the RBM is given by
h" 0 and we can see that the solution the evolution the weight is global toward the direction
f by projecting the differential equation on the preferred direction. Defining Ug we obtain
N, 1/2 Σ, E,W,,
dUe m2Ue thus Ug U@em²..
dt
This shows that the weights growing in the direction of & while the projection any orthogonal direction 6 stays constant:
Φe dw m N,
(06 E)( {) = 0 since d² E 0
di
When the weights grow larger, the solution for will depart from zero. The correlation of the learning RBM then starts to grow
1 / f
N₂h²
(DUU))RBM
Whh wayj exp
= wowl N Ex N.)
2
2
Σ
Σ
Lew;
No (1 E,
therefore exhibiting second order phase transition during the learning. Finally, we can study the regime where the weights are not small. In that case, can first observe that the evolution of the directions orthogonal to &, d² are given by
and diverges when No E, 3n
dw, di
m
Σ EkWu Edgh" tanh(h"
Σ of tanl(h""
which will cancel if the weight aligns totally with the g. Finally, taking we = E,w, the gradient projected along & at stationarity imposes
wm h² tanh(h" w) and thus / and h² /Bm
B Binary-Binary RBM
The RBM sharing both discrete binary variables the visible and hidden nodes is by far the most commonly used. In particular, using binary nodes in the hidden layer instead of the Gaussian distribution allows the model to potentially fit any order correlations of the dataset. In this section,
15

---

review how the learning dynamics translate to this case, using for simplicity binary {=1} variables. In order to obtain an interesting behavior in this phase of the learning, is important to consider particular parametrization of the RBM. We consider that all hidden nodes share the same weight.
This is important to be able to have a recall phase transition in the model. We therefore have the following Hamiltonian
H
Nh Σ U:W, Σ ha
(10)
where N₁ a:N, is the number of hidden nodes ha of the system and the vector W correspond to the weight shared across all the hidden nodes. In this model, we can now compute the positive and negative of the gradient. The first one is given by
1 Nn.
(0 Σ
DajD
Σ
dmwr exp f
3m3 Nv +
m3
Σ Ejuj
tanh
NE
Σ Wjyj
ZMattii
2
Nh
Σ) dmdru; exp
Bmm
Nv
+ m33 Σ Eg4g
5(T No Σ wyll)) tanh(r)
Mattis
2
Σ I
drdfdmwy exp
Bm2NN Nv
+ m3 Σ EgUs
(anh(T)e"*++1ie ' wyy
Maatis
[
dmdddde" 3m 2 N2/2tirr
tanh tanh (5:mB iN, Fw;) tanh(r)
Z
ΣLog log cosh [5, iN
'N,'ruuy Fwyy]
x exp
finding the saddle point of the argument in the exponential, obtain
(u:
Σ
ha) & tan tanh
N₁ Σ E,wy
&.m tanh
N Σ En
NA
The same type of computation can be done for the negative term, we found that
N u
had RBM E tanh (WTr)
tanh
Σ my tanh (wyr)
Nn
Again, in the small coupling regime (or at the beginning of the learning), when N₁ E, wz < we have that = O. In such case, the gradient over the weight matrix is given by
dw;
E.m tanh
m
Σ Enw
dt
No.
following the same approach as in the main text, we project the weights the unit vector un E/VN,, Ug = u W, which gives
dUg / Nym tanh
Mv
Uee
dt
Nn
We can integrate this equation, obtaining the solution
sinh
(/N. Ug(t)
m
sinh
VN.a
m
Ue(0)
exp
m 2²
m2
Ut(t)= = (0) exp
16

---

2.0
1.5
10-1
1.0
1.4 with My. 4000 c 00033
Ng =700, =0.04
=1000 =0.05
0..
expl m//o)
0.0
0 100
200
300
400
100
200
300
400
Number of epochs
Number epochs
Figure 5: Left: learning behavior of the Binary-Binary RBM, using data from the Mattis model. The different curves correspond systems of size N,- 900 inverse temperature
learning rate e 0.03, 0.04, 0.05 and N₁ = 400, 700, 1000 respectively. The argument of the exponential curves are adjusted but set m²e/a. Right: we illustrate the RBM's s dynamics in the Hinary-binary case with 3 = 1.4 and N, 900, Nn. 400. First the eigenvector == aligns itself with the pattern& &. Then, the eigenvalue wa=1 grows exponentially until reaching saturation and when it crosses the value 1, the system develops spontaneous magnetization.
where the second line is obtained the very large N, limit. Again we have an exponential growth in the first steps of the learning. At the end of the learning, the weights again align in the direction of This can be checked by the fact that the positive term of always orthogonal to any vector orthogonal &, and thus the simplest option for the gradient projected in those direction is to be orthogonal Taking W u we obtain
m tanh (mw/a) = T tanh(wr)
T tanh sanh(wr)/a)
The solution can be found numerically by solving the fixed point equation on and measuring the magnetization the dataset. In Fig[5 we illustrate our results in the same dataset in the section 4.1 taking the Mattis model with N, = 900, , = 1.4. varying the learning rate and the number of hidden nodes.
C Learning with correlated patterns
In this part we detail how the learning goes when considering a pair of correlated patterns. As described in 4.2 the pairs of patterns are defined as
+ n72 and E² m7¹ n72
where n7² is a vector whose first Nr 1tx components are equal to +1 with equal probability and the remaining ones are zero. The other vector r7" has its last N, -يب components equal to =1 with equal probability and the rest are O; we also have that K E [0, 1]. When x = 1, both patterns 1 are equal, while otherwise different but correlated. In particular LE'''. syp I ] N.k. Following the results of [33] it possible to compute the saddle point equations for the magnetization. The general form is given by
Σ E tanh (am1c) + Bmm2ft
=
Nv
1
Σ & tanh (Bm + Bmzz?)
m2 N
This system has been solved in 3 and exhibits the following properties. When 7>> + Ks the system is in the paramagnetic regime and my m2 0. When the temperature lowered and lies
17

---

in < <
< + R. the solution is given by the pair retrieval = m2 m 1 tanh(2,mm)
Finally, when T 1 R, the system condensates on the following solution
k tanh (3(m1 + +z)) +
K
m1
tanh (8(m1 - m2))
2
2g = 2
tanh (3(m1 + m=2)
tanh (8(m) - .22)
2
where basically the system either condensates toward one of the pattern e while the other magneti- zation has some non-zero value due to the correlation.
We can use the thermodynamics properties of this model to study how the learning of the RBM should behave the regime T < 1 K. order to use this model as generating the dataset, we need to compute the correlations (818))- The model presents four fixed points, all equally probable:
(m,, m2) (m+ m and its symmetric case (m1, m2) (-m+, mm
(m,1, m2) (m""m" .m+) and its symmetric case (m11, m2) = (-m""=m")
where m > m we have that
0. Therefore, writing r tanh(3(m m m))) and tanh(B(mm m m=))
(1rtz)
Σ
[int v tanh(3(m1 m2))] [(n] + nz²) tanh(3(m) + ma))]
(m)))
17 r²² + 1/, WE11P
because the cross terms 17. n3 are canceled when changing (m1 = m m2 = m to (m1 = m m2 m²) At this point, it is possible to write the gradient at the linear order and project it toward both direction 17' and 17² Denoting U n7 w and U. = 172 w", we get
dUee
2
+ K
U
dt
2
dU
7
p
k
U
dl
2
Using this form, we end up with the following solution of the weight matrix
(t) (0) U (0) x
exp (, ( 2
- -1]
n U. (0)
1 xr t) - ) (11)
exp
p²
2
We therefore understand the following. At the beginning of the learning, since r p. what is learned first is the mode toward the direction r7² x t E², in a timescale that is given by time t ~ 1/r². At different timescale, the part that is aligned with 17² will grow as well as discussed in the main text. Following the dynamics of the weights, as in eq. [ we can infer the moment where the phase transition occurs. When considering an Hopfield like model, we know that the transition happens when
2
B
Σ UiEe
Σ vi
2NN
2N,
that is, the critical temperature is 3. = Following dynamics the weights of eq. neglecting the terms that are not aligned with 17a we can write
11 and
1
Σ
viEi
2
U (0)
exp
+
2
Σ with
2
2N,
(1+ (1+1)/2
k)/2
2N,
US. (0)
1 +
By (t)
exp
-1
(1 (1+
where we can identify a sort of dynamical temperature associated the pattern ry' Now, we need to be careful since by definition, the pattern 17' is made of random +1 components its (1 + x)/2
18

---

10
(E(()
1=501
a5(t)
8
10-2
8
10
0
200
400
600
800
1000
1200
Number of epachs
hj
h1
Figure 6: Left: the empirical dynamics of the eigenvalues of the weight matrix, here denotes iha in blue. In green, the predicted dynamics as eq. adjusting only the initial conditions U (0) and (0). We see that the curves cross the line energy in the plane (h1, h2), the order parameters of the model. For different value of the weights during learning, we reconstruct the free energy of the system. We clearly see how the RBM first creates two minima, in the direction of r and then, split again to obtain the four fixed points.
at the same moments l, and ltr. Right: the free
elements and zero elsewhere. This rescales the critical temperature by factor (1 m2/2. Therefore we need to look when
1 ++ 3,p (t,)
and the same kind of argument can be used for the second transition with this time
B,,(() ( ( U (0) ))
2
exp p 121-1 - ]
x)/x (11-k)22
2
(t1) ~
We show in Fig. 6 left panel how the times t, and tys compare with the moment where the eigenvalues wa of the weight matrix cross the value one, which correspond to the phase transition following statistical mechanics approach [ We observe that both indicators are crossing the line = at the same moment. In Fig. 6 right panel,
plot the behavior of the free energy (in the plane (h,, h)))). We see that at the moment of the transition, the free energy opens in the direction corresponding to the transition. Projecting the dataset (black dots) in the same direction h1 (resp. h2), can how the system correctly positioned the minima once fully trained.
D The datasets and the rescaling
In this work, illustrated our results on three datasets:
1. The Human Genome Dataset (HGD) 34 containing binary vectors, each representing selection of 805 genes from human individual, where or Os indicate the presence or absence of gene mutations relative to reference sequence.
2. The MNIST dataset 442. containing 28 x 28 pixel black and white images of digitized handwritten digits.
3. The CelebA faces.
43 dataset, in black-and-wiin white, with 128 x 128 pixel images of celebrities
The datasets MNIST and CELEBA, were either downscaled or upscaled in order to create dataset of various sizes. practice, the function resize from the python library skimage was used either to increase or decrease the image size. The dataset HGD is geometrically a one-dimensional structure. In order to reduce its size, we took the convolution of each sample with kernel of size s 3. The output is one if the sum of the three input values (that discrete variables in {0, 1}) of the kernel
19

---

dataset
N
N₁
N
MNIST
14 X 14
250
0.005
500
MNIST
19 19
500
0.005
500
MNIST
28 x 28
1000
0,005
500
MNIST
39 x 39
2000
0.005
500
MNIST
56 x 56
4000
0,001
500
CelebA
45 x 45
125
4 10
500
CelebA
64 x
250
4 10
500
CelebA
90 x 90
500
4 10-00
500
CelebA
128 x 128
1000
4 10-5
500
HGD
201
25
0.002
4500
HGD
402
50
0.002
4500
HGD
805
100
0.002
4500
Table I: Hyperparameters of the RBMs analyzed in the main-text.
is above the threshold 2 and zero otherwise. A stride of has been chosen such that the resulting samples has its size reduced by a factor two.
E
Details on the training and the numerical analysis
E.1
Training
All RBMs analyzed in the main text were trained with the Persistent Contrastive Divergence (PCD) method [4] and == 100 MCMC sampling steps per parameter update, to approximate the negative term of the log- ligeeikoo gradient in Eq. (2) In this scheme, the last configurations reached the MCMC process to compute the previous update are used as initialization of the new chains used to compute the subsequent update. This scheme tends favor the equilibrium regime (2). The results with other training schemes is discussed in section H Moreover, usual, we keep Nehains independent parallel Markov chains. used to estimate the positive term of the gradient. The code to reproduce the experiments is freely available https:://iihb wtw.ul/oooo//uuueeneeeeellloo Torlll//oorhBBMMM The hyperparameters used for each training (no. of visible and hidden units N, and Nn. respectively, learning rate minibatch size Nms) are given in Table
simplicity, Nehains chosen match the minibatch size
E.2
Numerical analysis
E.2.1 Susceptibility
Part of the analysis in section of the main text is based on sampling the equilibrium configurations of models trained at different epochs extract the moments of the distribution of magnetizations ma. iec the projection of the samples along the different a-th singular vectors of W. For this purpose, we automatically selected 10 models uniformly in logarithmic scale in training time and annealed the N₂ 1000 independent samples from the least trained model to the most trained model, following the hot-to-cold thermal analogy, i.e. we perform Nmestr alternate Gibbs sampling MCMC steps each set of model parameters and use the last achieved configurations as a starting point to initialize the run the next set of model parameters. The less trained model initialized randomly. In parallel, we also consider the reverse scheme, where we consider a heating annealing. We start with the most trained machine, where all visible units initialized to 1 (to force all initial configurations to be in single cluster), and move backwards it in training time. We systematically checked that both analyses gave the same results for Nmest: 1000 the region of interest (the critical region) in Fig. and
addition to the sampling procedure, we computed the SWD of the matrix W at each new parameter set and projected each visible configuration along each of the left singular vectors ua to obtain m. = Uua w)/ /N., and each hidden configuration along the right singular Ua vector, to obtain m. (u., VN. The distribution moments are later estimated using the sample mean and variance of these N₂ measures.
20

---

E.2.2 Hysteresis loop
To investigate the hysteresis behavior between different lumps at or above the phase transition, we first select the training epochs which the first singular values of W, W1, take the values 4, 4.4 and 5. Note that for MNIST Wi,e e is approximately 4. With the model parameters at each of these three selected number of updates, we perform N₂ independent MCMC runs with the tilted Hamiltonian from Eq. G In these runs, the external field la is gradually varied to trace loop: starting from h = O, we slowly increase it to hmax: then decrease it to mmax and finally bring back to O. Again, the last configurations reached at certain h are used initialization for the next one. In practice, have chosen maax/ /N0-55 the increment in the field as 8h x /max//Nooo and Moop = 50. We modulate the speed of the loop by performing different number of MCMC steps & at each value of h. As shown in the figure, we consider & = 10, 100, 1000 and 10 The results shown in the main text were obtained using the RBM trained with the original CELEBA dataset (i.e. N2 128 x 128), but the results obtained with other sizes and datasets are completely analogous.
F Link between the low-rank RBM and Mattis model
Let us consider a low-rank Ising-Ising RBM in which the W matrix has single non-zero singular value w. with left and right singular vectors u and ú, and visible and hidden Ising variables (let's call these variables W, s and T to distinguish them from the binary 0, 1 version, which would be W and and h). In this case, the energy function of the RBM (if we ignore the biases for now) is
E h) = ww v)(ū h),
which leads marginal energy on the visible
1 {(s) = Σ log cosh [ V.uuna = - Now?m + O(m²),
(12)
where we have defined m u v/v Ny as the magnetization of the spins along the direction u and have exploited the fact that E. u² = because it is a unit vector. One can obtain an analogous expression for the marginal energy on the hidden units, formulated in terms of the hidden magnetization m = ú T/ T/VNn. These energy functions, for small m or m. are formally equal to those of the Mattis model for 3 w² which means that our RBM should manifest critical phase transition at 3.- = T-1 _² = I, with mean-field critical exponents. Standard RBMs are not formulated as Ising +1 1 variables, but in the form of binary {0, 1} variables where we have the equivalence 4W = W between the couplings matrices. This results in a critical point at we = 4 and an effective inverse temperature 3 w² /16.
G Hysteresis in discontinuous transitions
The hysteresis phenomenon shown in Fig. 4-E is a classical measure in statistical physics and is unique signature developed and observed in statistical physics reveal that high-dimensional probability measure had phase transition where it splits into two distinct lumps. The procedure is to tilt the probability measure by introducing a contribution in the energy function that favors one lump over the other. In the present case, we use the learned preferred direction associated with the first phase transition u. which also the direction in which the probability measure splits, and we add magnetic field to break the symmetry between the two modes created by the learning process. Therefore, we tilt the measure by adding
tilted
RBMM
Σ U.e.
The tilted Hamiltonian favors one of the two lumps depending on the value of the local bias h. When the measure is actually concentrated on two distinct lumps, one of them leads sudden discontinuous transition ("first-order transition" in physics). In our case, the lumps associated with the learned patterns and this additional contribution consists of the scalar product between the visible variables and the times of the learned patterns (the field that controls the strength of the tilting). In the presence of first-order phase transition, one usually finds the phenomenon of hysteresis, i.e. the transition from one clump to another can be delayed due to metastability, leading to the characteristic
21

---

N. 28 CD-10
N. 28 Rdm-11
10A
109
15 100
10
2 x 10"
10"
1/P
10-1
x 10-2
3 10-1
2000
4101
2000 opochs
400X)
opochs
Figure 7: We reproduce Figs. (right).
A and C but for RBMs trained with CD-10 (left) and Rdm-10
hysteresis loops that we show Fig. E (see e.g.
46 for a rigorous treatment and for physical treatment). This figure thus provides direct evidence for the decomposition of the measure into distinct lumps corresponding the second-order phase transition that occurs during learning. The details about the numerical implementation are given in Section E.2.2
45
the learned patterns, and that this decomposition occurs at
H
Results with other training schemes
All the machines analyzed in the main text were trained using the PCD-100 scheme, which involves initializing the chains with the PCD method and performing 100 MCMC steps per parameter update. This approach ensures that obtain models in good equilibrium, avoiding the non-monotonic behav- ior in sample quality typical of out-of-equilirrumm regimes (2)). However, it is more computationally expensive than standard methods, where only l- strategies like Contrastive Divergence (CD) [48]. where chains are initialized using the minibatch samples at each update, or the fully -oulliiiii regime (Rdm). where chains are always initialized randomly [29 49).
10 steps used, or alternative initialization
In this appendix, dataset. While the time evolution differs -often degrading susceptibility along learning directions- the overall picture of the cascade of phase transitions remains unchanged. We show the equivalent to Figs. BAC with these two new trainings in Fig. 7
analyze RBMs trained with CD-10 and Rdm-10 strategies the MNIST
22

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA] is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
23

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specinf well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The theoretical analysis relies on theoretical physics development. In such context, the full set of assumptions is not state clearly, but the results is expected to be correct within the range of application. As such no theorem or lemma is provided.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The method used here quite standard (RBM training in a simple case), all the rest is clearly specified.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.

---

If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ htppllc/ggidss///e
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.cc cc/ http://lllgggg..eee/eeeeeeuueeee http:///oooosuuussssssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
25

---

Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experiment perform are sufficiently precise within the claim of the paper so they not need further error bars.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: the experiments do need particular resources and can be trained personal laptop.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
26

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that there no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
27

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
28

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "VcIegkvi6z", "reviewer": "Reviewer_pfhu", "summary": "This work examines the relation between the internal representations of an RBM and the data that it is trained on as the training procedure goes forward. The work begins with a theoretical exposition for Binary-Gaussian RBMs trained CW-like models with 1 or 2 preferred encoding patterns. The theoretical analysis shows that the weights develop along directions that aggregate the directions of data modes before further settling into directions that capture each of the data modes. The analysis is extended to Binary-Binary RBMs. Several empirical results are presented. For simple models, theoretical findings are numerically validated. RBMs are also learned for HGD, MNIST, and CelebA dataset. The behavior of these learning situations mirrors certain behaviors of the simple cases. The RBM first learns patterns along the largest principal components of the data before undergoing a phase transition and settling into more nuanced representations.", "questions": "Are there conclusions from this work which can be extended to more powerful generative models such as diffusion models, deep EBMs, etc.?", "limitations": "Limitations are not discussed in detail.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The dynamics of generative model training and the relations between the data and internal representations of generative models is an important area of study. I appreciate that this work undertakes the challenging task of unraveling these behaviors.\n* This work offers interesting evidence that, across simple and more complex datasets, RBM learning has somewhat predictable behavior. In particular, RBM weights will first learn to represent the data directions with the highest variation before learning more refined representations that better capture modal structure.\n* Extensive derivations and empirical results are included.", "weaknesses": "* A major weakness of this work is the scope. While broad claims are made about implications for other generative models, the analysis presented seems specific to the RBM architecture. Studying more recent generative models with stronger capacity would greatly increase the impact of this work, although such analysis is very difficult.\n* The work is quite dense, and it can often be difficult to follow the lines of reasoning and conclusions based on derivations. While the topic is complex and this is to some extent inevitable, I recommend the authors make efforts where possible to streamline and clarify the presentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for carefully reading the manuscript and for their interest in it.\n\nWe agree with the reviewer that the next natural update of this project will be to investigate these features in more complex generative models. Furthermore, we still believe that it is important to establish secure foundations to build on, and we feel that this work is already very dense and compact to include further analysis on a different type of models, which would require different techniques and settings. \n\nNevertheless, we believe that the applicability of these results goes beyond RBMs for several reasons:\n  * It is very likely that the same kind of transitions are also present in the early stages of deep Boltzmann machine learning, since the same perturbative analysis of refs. [8,9] performed for RBMs was recently reproduced for deep Boltzmann machines in Ref. [Ichikawa, Y., & Hukushima, K. (2022). Statistical mechanical study of deep boltzmann machine given weight parameters after training by singular value decomposition. Journal of the Physical Society of Japan, 91(11), 114001.]. \n  * Cascades of phase transitions have also been observed in other generative models, e.g. in the training of the Gaussian mixture model [18-22] or very recently in the sample generation of diffusion models [23-25]. For the latter, it is conceivable that during the training dynamics the model of the score first learns the most prominent features, which needs less data (and hence less training time, at least in an online learning schedule) and then later the finer ones.\n\nIf we have the opportunity, we will discuss this point in a new \"Limitations\" section and try to make the paper more accessible by taking advantage of the additional page.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8KonvgSX7z", "reviewer": "Reviewer_mUPE", "summary": "This paper analyses the phenomenon of phase transitions in the training dynamics of restricted Boltzmann machines (RBM) in theory and practice. In the theory part, the phase transition is characterised for a toy model: The RBM with a hidden node per feature is fitted to data characterised by one feature vector $\\xi$ or two correlated feature vectors $\\xi_1, \\xi_2$. The phase transitions are detected as diverging correlation functions between visible states of the RBM and via exponential fitting of the RBM weights to the target features. \n\nIn the empirical part, the phase transitions are explored in the training of a RBM on the Human Genome Dataset, MNIST, and CelebA. It is observed that the singular values of the weight matrix and the susceptibility undergo sudden changes during training, indicating a phase transition.", "questions": "- Do the projections and projected weights in the theoretical section correspond exactly to the discussed singular value decomposition described in the empirical evaluation? Is a theoretical study of the susceptibility in the theoretical section feasible? \n- Can the phase transitions be detected during training to, for example, adjust for the longer thermalisation time in the vicinity of the phase transition? \n- When running a gradient based Langevin sampler during training, do your observations suggest spiking parameter updates in the vicinity of the phase transition due to second order discontinuities, thus creating potential training instabilities?", "limitations": "The work analyses the phenomenon of phase transitions in RBM training dynamics, and is limited to this setting by definition. The scope is clearly defined in the paper. However, the authors checked the limitations box in the check list without providing further justification in the main text or in the checklist. A brief sentence about the scope of the work or a reference to lines in the main text would be appropriate.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The theoretical evaluation seems sound and in general the paper is well written.\n- The paper explores an interesting phenomenon in training dynamics of RBMs. The experimental evaluation of the phase transition is compelling and highlights the emergence of phase transitions well. Similar studies may be applicable to other types of generative models.", "weaknesses": "- It seemed to me that the notation switches between $v$ and $s$ to denote visible states/spins/sites.\n- I think the paper could be improved by making a stronger connection between theory and empirical evaluation. For example, do the projections and projected weights in the theoretical section correspond exactly to the discussed singular value decomposition described in the empirical evaluation? Can the susceptibility be analysed in the theoretical section? More consistent nomenclature of the terms described would help with reading and understanding the paper.\n- The numerical analysis would benefit from paragraphs or subsections.\n- The analysed dynamics of the RBM weights don't fully reflect the actual training dynamics. In practice, RBM training requires self-sampling from the model which causes biases in the estimated negative term. I would expect that the sampler is adversely affected by the phase transitions in the model (e.g. sudden drop in mixing time). Considering that sampling in an RBM is relatively tractable (compared to other types of EBM) I am wondering if the training dynamics of e.g. one step contrastive divergence can be analysed in a similar fashion. However, the authors do address the effect of the phase transition on the MCMC sampling in the experimental section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and for their careful reading of our paper. First of all, we will of course revise the notation to keep it consistent between the two parts and also make the transition and connections between the two parts clearer. We thank the reviewer for their advice.\n\n*Weaknesses:*\n\n* we will ensure that the notations are homogeneous in the whole paper for the final version\n* Yes, the projections and projected weights in the theoretical section correspond to the discussed singular value decomposition described in the empirical evaluation. We agree that the paper would clearly improve by discussing and making explicit this connection. We will do it in the revised version. The susceptibility is analyzed in the theoretical section, see line 127. We show how the susceptibility diverges when the quantity $N^{-1} \\sum_i w_i^2 \\sim 1$ showing explicitly that the transition is of second order. We will add in the appendix a comment on the critical exponent.\n* We will add structure to the numerical part in the final version, where an additional page can be added. It is true that the negative term is where the difficulty relies on when training the RBM. In practice, at the beginning of the training the sampling is very easy (the chains mix very quickly) and very crude algorithms such as one-step CD can be used to compute the gradient. When the first phase transition occurs the mixing time suddenly jumps and at that point it is important to correctly compute the negative term. In this case, the sampling can introduce a strong bias, the theoretical analysis of which is not easy since the weights are not small and, for instance, the 1-step CD would introduce non-linearity in the terms. Yet, we can check in practice, that the picture discussed in the paper does not change much when training with CD-1, as we show in the attached PDF for MNIST. Eigenvalues still turn on one by one, the first eigenvectors are very similar to those of the PCA and the growth of the eigenvalues is associated with a growth of the susceptibility along these directions. We propose to add a small subsection or appendix in the new version of the paper discussing out-of-equilibrium effects in more detail.  \n\n*Questions:*\n* The directions onto which we project the gradient in the theoretical section indeed correspond to the principal directions of the dataset. In the case with one mode it is trivial (since the weight matrix is only a vector). In the multiple features case, it is easy to see by looking at the equation right after line 491, decomposing the correlation matrix of the dataset onto the vectors $\\eta$ and $\\bar{\\eta}$. We will make it clearer in the final version of the work. Furthemore, the divergence of the sucesptibility in the theoretical section is already done, see line 126-127 and the equation in-between. We will be more clear about the exponent with which it diverges and how it relates directly with a second-order phase transition.\n* The first phase transitions can be easily detected by monitoring the eigenvalues of the weight matrix (the singular values). In the model where the variables are $\\\\{0,1\\\\}$, the phase transitions occur when the singular values become higher than 4, while when the variables are $\\\\{\\pm 1\\\\}$, the transition occurs when the singular values become higher than 1. In that sense, it is easy to detect the first phase transition. In practice, it is then possible to adapt the number of Monte Carlo steps nearby the transitions. The next phase transitions can also be detected by monitoring the second,third, … eigenvalues of the weight matrix, although it is not clear how many of them their will be.\n*We do not use a Langevin sampler. Since we work with discrete variables, we rather rely on Markov Chain Monte Carlo and Gibbs parallel sampling. To answer the reviewer's question: We do not observe spikes when updating the parameters unless we set a high learning rate. In this case, we do see spikes in the eigenvalues around the transition, but not in the experiences discussed in the current version of the manuscript.\n\n*Limitations:*\n\nWe will add a new section Limitations, in which we will discuss the limitations of the current analysis: (i) the extent to which these results are applicable to more complicated models, (ii) the limitations of the analytical setup, and (iii) the limitations of the numerical setup (problems with insufficient MCMC sampling).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LUo166dxE4", "reviewer": "Reviewer_ZU37", "summary": "This paper presents an analysis of the phase transitions of RBMs through an analysis of the weight matrix. They find the dynamics tend toward the center, then diversity into modes. The theoretical results are supported by empirics on 3 ML datasets.", "questions": "- How do you imagine these transitions changing with respect to sampling convergence? Perhaps out of scope of this work, but would be interesting to see all of these plots generate but now with x axis being sample steps and see how that looks\n\n- What would be the biggest takeaways/generalization to be made from RBM phase transitions to more complex ML models?", "limitations": "The authors sufficiently addressed societal limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This expands the important field of exploring the dynamics of energy based models\n\n- The figures are generally well presented (conditioned on note below)\n\n- The authors offer a compelling argument for the scalability of these phase transitions, and the pipeline theory to CelebA is well done", "weaknesses": "- This paper would be right at home in a physical review journal, but given the audience of NeurIPS, I would recommend increasing the accessibility to the very ML focused audience (e.g. matching the notation used in ML world, things like labeling axes with descriptive words then putting the equations in text, numbering all equations, etc.)\n\n- 199 paragraph is too long", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper and for their comments. We do not entirely agree with the reviewer regarding the target audience of this article.\n\nFirst, the RBM itself is a model introduced by Hinton and Sejnowski for the purposes of machine learning. This model has been used in ML for decades, it was used as a building block of deep neural networks for a while, and is still used for ML tasks today, notably for interpretability, and in particular in low-data regimes (e.g. in neuroscience, population genetics,...). . While it is true that our analysis relies on phenomena more commonly studied in physics, our work shows that understanding these phenomena is crucial to fully understand the training of RBM, and how the weights are shaped during training. To name just a few elements: the second-order phase transition implies a sudden increase in mixing time which directly affects the quality of the training of RBMs, knowing that RBMs learn the direction of PCA helps to monitor training and to understand the structure of the weights, and we show how the exponential growth of the eigenvalues of the weight matrix is determined by the variances of the clusters.\n\n*Questions:*\n\n* It is not clear how the transitions would change in general. One should expect the first singular values to still correctly match the PCA components, because at the beginning of the learning process the mixing time is very small, which means that the estimation of the negative term by the CD-1 scheme (only one step initialized in the examples in the minibatch) is roughly sufficient. This also means that the first transition should be very similar to that of the PCD-100 studied in this article. It is not yet clear what should happen after the transitions start to follow each other, as the mixing times significantly exceed the 1 step used by the algorithm. To test this, we repeated the analysis for training with MNIST using the CD-1 algorithm. We show the equivalent of Fig.3 A,B,C in Fig.1 of the attached PDF file. We find that the image of the cascade of transitions remains unchanged, but the overall behavior is different because the susceptibility increases sharply with the growth of the corresponding eigenvalue, but then decreases again. In Fig. 2 in the attached PDF file, we compare the first 5 singular vectors with the first 5 PCA components of the data set, which show extremely similar patterns (within an overall sign). This picture is very similar to what we observe when analyzing the HGD dataset, where the mixing times become so large that even PCD-100 is no longer able to thermalize properly. We will include a subsection in the paper where we briefly discuss the impact of MCMC estimation of the gradient on the transitions.\n\n* Our analysis suggests that similar phenomena probably occur in more complex ML models. First, in Deep Boltzmann Machine, a previous work [Ichikawa, Y., & Hukushima, K. (2022). Statistical mechanical study of deep boltzmann machines given weight parameters after training by singular value decomposition. Journal of the Physical Society of Japan, 91(11), 114001] shows that the SVD of the weight matrix is crucially important as well. Second, in more general EBMs, it is also highly probable that such phase transitions occur and that the mixing time increases. Finally, the cascade of phase transitions that are observed here are very akin to the ones recently observed in the sample generation of diffusion models [23-25]. For the latter, it is conceivable that during the training dynamics the model of the score first learns the most prominent features, which needs less data (and hence less training time, at least in an online learning schedule) and then later the finer ones.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "17mwbfr1dG", "reviewer": "Reviewer_vpPo", "summary": "This paper analytically demonstrates a cascade of second-order phase transitions in the training of a simple RBM model and numerically validates this theory in the training on real datasets.", "questions": "Major\n- While the authors consider a single Gaussian hidden node, they do not provide any statistical details of the Gaussian distribution, such as the mean and variance. I assume the authors consider a Gaussian distribution with zero mean and $1/N$ variance, but I am uncertain about the meaning of $N$ (I guess $N \\equiv N_v$), why the authors choose this setup, and how $N$ relates to the statistics of $xi_i$.\n- The meaning of the equation between lines 161 and 162 is unclear, especially when the authors mention that the equation “is analogous to the one studied in the previous section.”\n- While the authors analytically investigate the BG-RBM with a single hidden node in Section 4, the numerical analysis in Section 5 is conducted on the BB-RBM with multiple hidden nodes. The authors should discuss the expected differences due to this gap and adequately clarify the limitations of their analysis.\n- I am not sure why “the projections along all subsequent directions are Gaussian” as stated in line 231 on page 6.\n- It is unclear how the authors obtained Figure 4 E and what they aim to demonstrate. Does the $h$ in Fig. 4 E indicate the hidden variable?\n- The direct relationship between Section 4 and Section 5 is hard to capture. As I understand it, the main part of this paper introduces an underlying preferred direction $\\xi_i$, but I am not sure which vectors correspond to $\\xi_i$ in Sec. 5. Please describe the correspondence between Section 4 and Section 5 and clarify the differences from the results in Ref. [8].\n\nMinor\n- While the authors mention the Mattis model to describe $\\xi$, the explanation of $\\xi$ is insufficient. It is unclear whether $\\xi$ is frozen, the distribution of $\\xi$, etc. \n- In Fig. 1, the x-labels of the left and right panels differ. The term \"susceptibility\" is not defined, each line is indistinguishable, and the meanings of the left and right vertical axes are not indicated in the inset of the left panel. There is no indication of the black solid lines in the insets of the right panel.\n- In lines 127 and 439, the condition for diverging the correlation seems to be a typo. $N^{-1}\\sum_{i} w_i^2 \\approx 1$ may be correct.\n- Please introduce the learning rate $\\epsilon$ in Eq. (3) because the authors provide results with respect to the learning rate in Fig. 1.\n- Please provide references and detailed descriptions for the BG-RBM primarily considered in Section 4.\n- The authors should correct numerous typos in the main text and the appendix. For instance, the authors often use expressions such as $s_i$ and $N$, which are not defined in this paper. In line 107 on page 3 and line 416 on page 12, there are typos in the definition of magnetization $m$. In line 414 on page 12, there is a typo in the definition of $\\xi_i$. In Eq. (3), the authors do not clarify what $t$ indicates.\n- The authors should consistently use expressions throughout the paper. In lines 136 and 194, the word \"appendix\" is used, but in other cases, the term \"SI\" is used. The title of Appendix B is \"Binary-Binary RBM,\" but the model is often referred to as \"Bernoulli-Bernoulli RBM\" in the paper.", "limitations": "The authors should discuss the limitations of their work more thoroughly.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- This paper theoretically analyzes the learning dynamics of weight parameters in the Binary (or Bernoulli)-Gaussian RBM (BG-RBM) through the lens of statistical physics.\n- The authors demonstrate that the RBM undergoes a series of second-order phase transitions during learning, both theoretically in the BG-RBM and numerically in the Binary-Bernoulli RBM (BB-RBM).", "weaknesses": "- There is a gap between the theoretical and numerical analysis in this paper. Specifically, the BG-RBM with a single hidden node is utilized in the theoretical analysis in Section 4, while the BB-RBM is used in the numerical analysis in Section 5.\n- Although the main results of this paper may merit acceptance into NeurIPS, I believe the authors should revise the manuscript to improve readability and clarity. Some suggestions are provided in the Questions section below. If the authors address these issues and answer my questions, I can strongly recommend accepting this paper for NeurIPS; however, I cannot recommend it for acceptance in its current form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for reading our paper and for their comments. We think that they are instrumental in improving the quality of our work.\nWe think that the gap between the theoretical part and the experiments highlighted by the reviewer is partly due to a gap in the presentation (which we will revise accordingly) and not in the analysis. Indeed, for simplicity, we have chosen to present the binary-Gaussian RBM in the main text, but we do analyze the binary-binary RBM in the Appendix. There is a clear relationship between the projections and projected weights in the theoretical section and the singular value decomposition described in the empirical evaluation. We think that this has not been clarified in the current version but it can be easily explained. We will discuss this and make it explicit in the revised version. Apart from that, the discrepancy between the theoretical and the experimental part is due to simplifications in the theoretical part, e.g. all hidden nodes have the same weights, which was done to keep the analysis tractable.\n\nWe answer the question into details in the official comments while commenting them very briefly:\n\n*Questions:*\n1. The referee is right, we should have been more careful when defining the quantity. The distribution of the hidden nodes is indeed of zero mean and variance of order $1/N_v$. The value is indeed $N=N_v$. This is the correct scaling to have a large dimensional (large $N_v$) limit. We provide a detailed explanation in the official comment.\n2. The sentence at lines 161-162, refers to the type of phase transition occurring in the system. In the case with multiple features, the first phase transition is very similar to the one with one feature, in the sense that it is of the same “nature”. In this regime, corresponding to the first phase transition, everything happens as if there were just one main direction, which is  $\\xi^1 + \\xi^2$. We will correct this sentence to make it clearer in the final version.\n3. The theory is written for the binary-Gaussian RBM in the main text and for the binary-binary RBM in the Appendix. In the latter case, we show that in order to observe an interesting behavior we need to have an extensive (large) number of hidden nodes. On this aspect the gap is not strong. The main difference between theoretical and experimental parts relies on the dataset which is much simpler in the former case. Based on the article [8], we expect that even if the dataset present various modes (small compared to the number of visible/hidden nods), the system can still be described in a mean-field setting and, consequently, the phenomenology observed for one or two modes can be extended to many modes (but only a small number). This is confirmed by our experiments. Now we do not claim that this description is valid all along the training process, and we can see that at large training time the mean-field theory description is not as accurate anymore. We will comment on this in the final version.\n4. This sentence might be indeed unclear and thus will be rewritten in the final version. More details are provided in the official comments.\n5. We thank the referee for pointing out that we should clarify further the panel E of Fig. 4. The aim of this figure was to show the “hysteresis” phenomenon which is a clear signature, devised and observed in statistical physics, to unveil that a high-dimensional probability measure had a phase transition at which it split in two different lumps. A complete explanation is provided in the official comments.\n6. A detailed comparison between ref [8] and our work is provided in the official comments and will be better discussed in our final work. Concerning the relationship between sec 4 and 5, the reviewer’s comments helped us to understand that we have to clarify the relationship between the SVD decomposition used in Sec. 5 with the projections used in Sec. 4. There is a direct connection between the PCA and the preferred directions ($\\xi^1 + \\xi^2$ and $\\xi^1 -\\xi^2$) used in the theoretical analysis, but it was not clear enough in the current version. We put a detailed explanation again in the official comments and we will try to make it clearer in the final version of our work.\n\n*Minor:*\n1. In the Mattis model $\\xi$ correspond to a preferred direction (pattern) of the model, which is a frozen variable. In the Mattis model, $\\xi$ can be any fixed binary vector. We will add a section in the appendix to describe with more detail the statistics of the model, as well as the conditions for $\\xi$.\n2. We thank the referee for pointing out these issues with Fig 1, we will correct them and define properly the quantities.\n3. The referee is right, we will correct the typographical error.\n4. The learning rate comes in front of the r.h.s. of eq. 3. We will add it in the final version. The learning rate naturally impacts the learning behavior, the crucial ingredient is to keep the ratio learning_rate/N small.\n5. We plan, as mentioned before, to add first a section describing the Mattis model, and then another one more specific explaining the BG-RBM in a more general context.\n6. We agree with the referee, we will carefully read the whole work to avoid confusion in the definitions and correct the typographical errors.\n7. We thank the referee, we will check for this problem in the final version.\n\n*Limitations:*\nWe will add a new section Limitations, in which we will discuss the limitations of the current analysis: (i) the extent to which these results are applicable to more complicated models, (ii) the limitations of the analytical setup, and (iii) the limitations of the numerical setup (problems with insufficient MCMC sampling).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewer, 

we join the pdf bringing more details on the learning of MNIST when using Contrastive Divergenge with one Monte Carlo step (CD-1).

Meta-review: I have read all comments and responses. Reviews appear to be consentaneous, with four positive scores of 7, 6, 6, and 6. All concerns of reviewers have been fixed. It is recommended to accept this manuscript.
Decision: Accept (poster)