Title: Cascade of phase transitions in the training of energy-based models
Venue: NeurIPS 2024
Accepted: yes
Keywords: Restricted Boltzmann Machine

Generative model

Phase transition

statistical physics

Energy-based model
Abstract: In this paper, we investigate the feature encoding process in a prototypical energy-based generative model, the Restricted Boltzmann Machine (RBM). We start with an analytical investigation using simplified architectures and data structures, and end with numerical analysis of real trainings on real datasets. Our study tracks the evolution of the model’s weight matrix through its singular value decomposition, revealing a series of thermodynamic phase transitions that shape the principal learning modes of the empirical probability distribution. We first describe this process analytically in several controlled setups that allow us to fully monitor the training dynamics until convergence. We then validate these findings by training the Bernoulli-Bernoulli RBM on real data sets. By studying the phase behavior over data sets of increasing dimension, we show that these phase transitions are genuine in the thermodynamic sense. Moreover, we propose a mean-field finite-size scaling hypothesis, confirming that the initial phase transition, reminiscent of the paramagnetic-to-ferromagnetic phase transition in mean-field ferromagnetism models, is governed by mean-field critical exponents.

Cascade of phase transitions in the training of Energy-based models

Abstract

In this paper, we investigate the feature encoding process in a prototypical energybased generative model, the Restricted Boltzmann Machine (RBM). We start with an analytical investigation using simplified architectures and data structures, and end with numerical analysis of real trainings on real datasets. Our study tracks the evolution of the model's weight matrix through its singular value decomposition, revealing a series of phase transitions associated to a progressive learning of the principal modes of the empirical probability distribution. The model first learns the center of mass of the modes and then progressively resolve all modes through a cascade of phase transitions. We first describe this process analytically in a controlled setup that allows us to study analytically the training dynamics. We then validate our theoretical results by training the Binary-Binary RBM on real datasets. By using datasets of increasing dimension, we show that learning indeed leads to sharp phase transitions in the high-dimensional limit. Moreover, we propose and test a mean-field finite-size scaling hypothesis. This shows that the first phase transition is in the same universality class of the one we studied analytically, and which is reminiscent of the mean-field paramagnetic-to-ferromagnetic phase transition.

1 Introduction

In recent years, we have witnessed impressive improvements of unsupervised models capable of generating more and more convincing artificial samples [1, 2, 3]. Although energy-based models [4] and variational approaches [5] have been in use for decades, the emergence of generative adversarial networks [6], followed by diffusion models [7], has significantly improved the quality of outputs. Generative models are designed to learn the empirical distribution of datasets in a high-dimensional space, where the dataset is represented as a Dirac-delta pointwise distribution. While different types of difficulties are encounter when training these models, there is a general lack of understanding of how the learning mechanism is driven by the considered dataset. This article explores the dynamics of learning in neural networks, focusing on pattern formation. Understanding how this process shapes the learned probability distribution is complex. Previous studies [8, 9] on the Restricted Boltzmann Machine (RBM) [10] showed that the singular vectors of the weight matrix initially evolve to align with the principal directions of the dataset, with similar results in a 3-layer Deep Boltzmann Machine [11]. Additionally, an analysis using data from the 1D Ising model explained weight formation in an RBM with a single hidden node as a reaction-diffusion process [12]. The main contribution of this work is to demonstrate that the RBM undergoes a series of second-order

phase transitions during learning, each corresponding to the acquisition of new data features. This is shown theoretically with a simplified model and on correlated patterns; and confirmed numerically with real datasets, revealing a progressive segmentation of the learned probability distribution into distinct parts and exhibiting second order phase transitions.

2 Related work

The learning behavior of neural networks has been explored in various settings. Early work on deep linear neural networks demonstrated that even simple models exhibit complex behaviors during training, such as exponential growth in model parameters [13, 14]. Using singular value decomposition (SVD) of the weight matrix, researchers revealed a hierarchical learning structure with rapid transitions to lower error solutions. Linear regression dynamics later showed a connection between the SVD of the dataset and the double-descent phenomenon [15]. Similar dynamics were found in Gaussian-Gaussian RBMs [9], where learning mechanisms led to rapid transitions for the modes of the model's weight matrix. In this context, the variance of the overall distribution is adjusted to that of the principal direction of the dataset, while the singular vectors of the weight matrix are aligned to that of the dataset. Unlike linear models, non-linear neural-networks, supervised or unsupervised ones, can not exhibit partition of the input's space. Yet, linear model in general can not provide a multimodal partition of the input space, should it be in supervised or unsupervised context, at difference with non-linear ones.

It was then shown that the most common binary-binary RBMs exhibit very similar patterns at the beginning of learning, transitioning from a paramagnetic to a condensation phase in which the learned distribution splits into a multimodal distribution whose modes are linked to the SVD of the weight matrix [8]. The description of this process motivated the use of RBMs to perform unsupervised hierarchical clustering of data [16, 17]. The succession of phase transitions had been previously observed in the process of training a Gaussian mixture [18, 19, 20], and in the analysis of teacher-student models using statistical mechanics [21, 22]. The latter cases are easier to understand analytically due to the simplicity of the Gaussian mixture. Nevertheless, the learned features are somewhat simpler, as they are mainly represented by the means and variances of the individual clusters. Recently, sequences of phase transitions have been used to explain the mechanism with which diffusion model are hierarchically shaping the mode of the reverse diffusion process [23, 24, 25] and due to a spontaneous broken symmetry [26] after a linear phase converging toward a central fixed-point. The common observation is that the learning of a distribution is, in many cases, obtained by a succession of creation of modes performed through a second order process where the variance in one direction first grow before splitting into two parts, and then the mechanism is repeated. This procedure in particular demonstrate a hierarchical splitting, where the system refined at finer and finer scale of features as it adjust its parameters on a given dataset.

3 Definition of the model

An RBM is a Markov random field with pairwise interactions on a bipartite graph consisting of two layers of variables: visible nodes ( v = { v i , i = 1 , . . . , N v } ) representing the data, and hidden nodes ( h = { h j , j = 1 , . . . , N h } ) representing latent features that create dependencies between visible units. Typically, both visible and hidden nodes are binary ( { 0 , 1 } ), though they can also be Gaussian [27] or other real-valued distributions, such as truncated Gaussian hidden units [28]. For our analytical computations, we use a symmetric representation ( {± 1 } ) for both visible and hidden nodes to avoid handling biases. However, in numerical simulations, we revert to the standard ( { 0 , 1 } ) representation. The energy function is defined as follows:

E [ v , h ; W , b , c ] = -∑ ia v i W ia h a -∑ i b i v i -∑ a c a h a , (1) with W the weight matrix and b , c the visible and hidden biases, respectively. The Boltzmann distribution is then given by p [ v , h | W , b , c ] = Z -1 exp( -E [ v , h ; W , b , c ]) with Z = ∑ { v , h } e -E [ v , h ] being the partition function of the system. RBMs are usually trained using gradient ascent of the log likelihood (LL) function of the training dataset D = { v (1) , · · · , v ( M ) } , the LL is then defined as

The computation of the gradient is straightforward and made two terms: the first accounting for the interaction between the RBM's response and the training set, also called postive term , and same for the second, but using the samples drawn by the machine itself, also called negative term . The expression of the LL gradient w.r.t. all the parameters is given by

where ⟨ f ( v , h ) ⟩ D = M -1 ∑ m ∑ { h } f ( v ( m ) , h ) p ( h | v ( m ) ) denotes an average over the dataset, and ⟨ f ( v , h ) ⟩ H , the average over the Boltzmann distribution p [ v , h ; W , a , c ] . Most of the challenges in training RBMs stem from the intractable negative term, which has a computational complexity of ∼ O (2 min( N h ,N v ) ) and lacks efficient approximations. Typically, Monte Carlo Markov Chain (MCMC) methods are used to estimate this term, but their mixing time is uncontrollable during practical learning, leading to potentially out-of-equilibrium training [29].

This work focuses on the initial phase of learning and the emergence of modes in the learned distribution from the gradient dynamics given by Eq. (2). In the following section, we first analytically characterize the early dynamics in a simple setting, showing how it undergoes multiple second-order phase transitions. We then numerically investigate these effects on real datasets.

4 Theory of learning dynamics for simplified high-dimensional models of data

We develop the theoretical analysis by focusing on simplified high-dimensional probability distributions that concentrate around different regions, or lumps , in the space of visible variables. Our aim is to analyze how the RBM learns the positions of these lumps, which represent, in a simplified setting, the features present in the data. In order to simplify the analysis, we will consider the Binary-Gaussian RBM (BG-RBM) defined below, yet the same results can be derived for the Binary-Binary RBM (BB-RBM) as shown in the SI B.

4.1 Learning two features through a phase transition

We consider the following simplified setting: we will be using v i = ± 1 visible nodes, Gaussian hidden nodes and put the biases to zero b = 0 and c = 0 . As a model of data, we consider a Mattis model with a preferred direction ξ for the ground state, following the distribution where β = 1 /T is the inverse temperature and ξ i = ± 1 represents a pattern encoded in the model as a Mattis state [30, 31]. In a Mattis model, ξ represents a preferred direction of the model for large values of β , and in this simple case (with only one pattern) there is no need to specify its distribution as long as its elements are ± 1 1 . The Mattis model presents a high-temperature phase with a single mode centred over zero magnetization m = N -1 v ∑ i ξ i ⟨ v i ⟩ = 0 for β < β c (where ⟨ . ⟩ is the average w.r.t. the Boltzmann distribution) while in the low-temperature regime, β > β c , the model exhibits a phase transition between two symmetric modes m = ± m 0 ( β ) ( β c = 1 ). Henceforth, we shall focus on the regime β > β c where the data distribution is concentrated on two lumps. From the analytical point of view, we can compute all interesting quantities in the thermodynamic limit N v →∞ . In order to keep the computation simple, we will characterize here the dynamics of the system when performing the learning using a BG-RBM [27] with one single hidden node. In our setting we assume that the distribution of the hidden node is centered in zero (i.e. there is no hidden bias) and that the variance is σ 2 h = 1 /N v (we discuss the reason for the scaling in SI A). The distribution is then

Using this model for the learning, the time evolution of the weights is given by the gradient. With BGRBM we have that ⟨ v i h ⟩ H = N -1 v ∑ j w j ⟨ v i v j ⟩ H where the last average is taken over a distribution p BG ( v ) . We can now easily compute the positive and negative term of the gradient w.r.t. the weight

matrix. For the positive term we obtain that ⟨ v i v j ⟩ D = ξ i ξ j m 2 where m = tanh( βm ) . The negative term can also be computed in the thermodynamic limit ⟨ v i v j ⟩ RBM = tanh( h ∗ w i ) tanh( h ∗ w j ) with h ∗ = 1 N ∑ k w k tanh( h ∗ w k ) . If we take the limit of a very small learning rate, we can convert the parameter update rule using the gradient into a time differential equation for the parameters of the RBM, where t is the learning time:

with ϵ the learning rate 2 . We can analyze two distinct regimes for the dynamics. First, assuming that the weights are small at the beginning of the learning, we get that h ∗ = 0 . We can then solve the Eq. (3) in this regime obtaining the evolution of the weights toward the direction ξ by projecting the differential equation on this preferred direction. Defining U ξ = N -1 / 2 ∑ i ξ i w i , we obtain dU ξ dt = m 2 U ξ , thus U ξ = U 0 ξ e m 2 t .

This illustrates that the weights are growing in the direction of ξ while the projection on any orthogonal direction stays constant. As the weights grow larger, the solution for h ∗ will depart from zero. Then the correlation between the RBM visible variables starts to grow

which means that the susceptibility χ = ∑ i,j ξ j ξ i ⟨ v i v j ⟩ RBM , that is, the response of the system w.r.t. an external perturbation, diverges when N -1 v ∑ k w 2 k ∼ 1 , thus exhibiting a second order phase transition during the learning. Interestingly, χ diverges as a a power law with a (critical) exponent γ = 1 (where N -1 v ∑ k w 2 k plays here then the role of the inverse temperature in the standard physical models) thus corresponding to the mean-field universality class [32]. Finally, we can study the regime where the weights are not small. In that case, we can first observe that the evolution of the directions orthogonal to ξ cancel when the weights W align totally with the ξ at the end of the training. Finally, taking w i = ξ i w, the gradient projected along ξ at stationarity imposes

We confirm the main results of this section numerically in Fig. 1, showing they hold accurately even for moderate values of N v . The sum of the weights grows exponentially, following the magnetization squared (considering the learning rate), and the weights align with the direction ξ , while the norm of the weight vector converges towards √ β . Additional analysis details and extended computations for the binary-binary RBM case, which is slightly more involved, are provided in the SI.

4.2 Learning multiple features though a cascade of phase transitions

We consider now the case in which the data are characterized by more than two features. For concreteness, we focus on the case in which the data is drawn from the probability distribution of the Hopfield model [31] with two patterns ξ 1 and ξ 2 , using the Hamiltonian H H [ v ] =

-β 2 N v ∑ 2 a =1 ( ∑ N v i =1 ξ a i v i ) 2 . The generalization to a larger (but finite) number of patterns is straightforward. Following [33] we consider the case in which the patterns are correlated and defined as: ξ 1 = η 1 + η 2 and ξ 2 = η 1 -η 2 ; η 1 is a vector whose first N v 1+ κ 2 components are equal to ± 1 with equal probability, and the remaining ones are zero ( 0 < κ < 1 ). Whereas η 2 is a vector whose last N v 1 -κ 2 components are equal to ± 1 with equal probability, and the remaining ones are zero. When T < 1 -κ this model is in a symmetry broken phase in which the measure is supported by four different lumps centred in ± ξ 1 and ± ξ 2 . Analogously to what was done previously, we now consider a BG-RBM with a number of hidden nodes equal to the number of patterns where again both hidden nodes are centred in zero and have variance σ 2 h = 1 /N v . The Hamiltonian is then given by H [ v , h ] = -∑ ia v i h a w ia + ∑ a h 2 a N v / 2 , which corresponds to a Hopfield model [31] with patterns w 1 and w 2 . The analysis presented in the previous section can be generalized to this case (see SI for more details) and one finds the dynamical equations for the evolution of the patterns:

0

20

40

60

80

100

Number of epochs

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

N

-

1

v

∑

i

ξiwi

β

=1.3,

N

v

=100,

/epsilon1

=2.0e-03

β

=1.4,

N

v

=100,

/epsilon1

=2.0e-03

β

=1.5,

N

v

=900,

/epsilon1

=2.5e-04

Exponential

∝

exp(

/epsilon1m

2

t

)

w

=

√

β

0

5

10

χ

40

60

80

100

Nb of epochs

0

1

m

0

500

1000

1500

2000

Number of epochs

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

0

200

400

10

-

4

10

-

3

10

-

2

10

-

1

10

0

∝

exp(

/epsilon1r

2 (1+

κ

)

2

t

)

w

1

·

(

ξ

1

+

ξ

2

)

w

2

·

(

ξ

1

+

ξ

2

)

w

1

·

(

ξ

1

-

ξ

2

)

w

2

·

(

ξ

1

-

ξ

2

)

500

750

1000

1250

Nb of epochs

10

-

2

10

-

1

10

0

∝

exp(

/epsilon1p

2 (1

-

κ

)

2

t

)

As shown in the SI C, ⟨ v i v j ⟩ D = r 2 η 1 i η 1 j + p 2 η 2 i η 2 j where r, p are a function of β (and β -1 = T < 1 -κ , r > p ). Note that this factorization of the correlation matrix is precisely its spectral eigendecomposition, which means that η 1 and η 2 are nothing but the principal directions of the standard principal component analysis (PCA). At the beginning of the training dynamics the RBM is in its high-temperature disordered phase, hence the second term of the RHS of Eq. (4) is zero. The weights w 1 and w 2 have therefore an exponential growth in the directions η 1 and η 2 , whereas the other components do not evolve. If the initial condition for the weights is very small, as we assume for simplicity, one can then write:

where we have neglected the small remaining components; z a and ˜ z a are the projections of the initial condition along the directions η 1 and η 2 . Since r > p , on the timescale (log N v ) / ( r 2 (1 + κ ) ) the component of the w a s along η 1 becomes of order one whereas the one over η 2 is still negligible. In this regime, the RBM is just like the one we consider in the previous section with a single pattern: the system will align with a single pattern that is given in that case by η 1 ∝ ξ 1 + ξ 2 , and it has a phase transition at the time t I :

At t I , the RBM learns that the data can be splitted in two groups centred in ± η 1 , but it does not have yet learned that each one of these two groups consist in two lumps centred in ξ 1 and ξ 2 (and respectively -ξ 1 and -ξ 2 ). The training dynamics after t I can also be analyzed: the components of the weight vectors along η 1 evolve and settle on timescales of order one to a value which is dependent on the initial condition (see the eq. in the SI). In the meanwhile, the components along η 2 keep growing; at a timescale (log N v ) / ( p 2 (1 -k )) (quite larger than t I in the limit N v →∞ ) they become of order one. In order to analyze easily this regime, let's consider first the simple case in which the initial condition on the weights is such that w 1 (0) · η 2 = -w 2 (0) · η 2 and w 1 (0) · η 1 = w 2 (0) · η 1 . In this case, one can write w 1 = A ( t ) η 1 + B ( t ) η 2 and w 2 = A ( t ) η 1 -B ( t ) η 2 . The corresponding RBM is a Hopfield model with log likelihood:

At t I , when ( 1+ κ 2 ) A ( t I ) 2 = 1 , one has the first transition in which the RBM measure breaks in two lumps pointing in the direction ± η 1 , as we explained above. In this regime B ( t ) is still negligible but keeps increasing with an exponential rate. Using the results of [33], one finds that

A

B

C

D

0

4-5-6-7-8-9-10

Wsingularvalues

Wleftsingularvectors

1.0

7.5

lovodn onl

0.0

5.0

Wα

0.5

European

2.5

-0.2

South Asian

East Asian

American

African

0.0

0.0

-0.2

0.0

0.2

0

2000

4000

0

5000

mPCA

epochs

epochs

epoch=401

epoch=651

epoch=901

epoch=2501

epoch=3001

epoch=5001

0.2

1

0.0

-0.2

-0.25 0.000.25-0.25 0.000.25-0.25 0.000.25-0.25 0.000.25-0.25 0.000.25-0.25 0.00

0.25

m1

m1

m1

m1

m1

m1

when 1 -κ 2 B ( t II ) 2 = 1 , a second phase transition takes place. This defines a time t II at which the probability measure of the RBM breaks from two lumps to four lumps, each one centred around one of the four directions ± ξ 1 , ± ξ 2 . We have considered a special initial condition, but the phenomenon we found is general. In fact, for any initial condition one can show that the dynamical equations have an instability on the timescale t II , which generically induces the second symmetry breaking transition. On Fig. 1, right panel, we illustrate the exponential growth as described by the theory, toward the two directions. In the SI 4.2, we show how these phase transitions are in very good agreement with previous work [9, 8] and how the phase space is split during training time. At the end of the training, the patterns are given by w 1 = ξ 1 and w 2 = ξ 2 modulo a rotation in the subspace spanned by ξ 1 , 2 , since the likelihood is invariant by rotations in this subspace. In fact, we often found that the patterns are not perfectly aligned because we are not forcing the weights to be binary. This analysis can naturally extend to more than two patterns, typically resulting in a cascade of phase transitions. In this process, the RBM progressively learns the data features, starting from a coarse-grained version (just the center of mass) and gradually refining until all patterns are learned. The analysis done on the BG-RBM can of course be repeated on the BB-RBM (as is done for the case with one mode in the SI B). The main difference at that level between the two models is that in order to have a retrieval phase, the BB-RBM needs to encode the patterns on an extensive number of hidden nodes (proportional to N v ), while the BG-RBM needs only as many patterns as hidden nodes. Both models can match perfectly the dataset in the limit N v →∞ , but we might encounter discrepancies for finite size. However, when dealing with real datasets, by construction, the BG-RBM can not reproduce higher-order correlations and therefore is less interesting than the BB case.

5 Numerical Analysis

In the previous sections, we examined the learning process in simplified setups, in order to be able to develop an analytical treatment. In particular, we have shown analytically in a simple setting how the

weight matrix is shaped by the patterns present in the dataset and how the learning process dynamics is triggered by the PCA components (the η 1 and η 2 in the previous example) and not by the learning of the encoded patterns ( ξ 1 and ξ 2 ). Moreover, we have shown that each time the RBMs learn a new direction, the susceptibility of the system diverges with a precise power law everytime the RBM learns a new direction, which is also associated with the development of new modes in the probability measure. In this section we will also show that the insights gained from this simplified analysis are also applicable to understanding the learning process of a Binary-Binary RBM (BB-RBM) with many hidden nodes trained with real data sets. The details about the training procedure are given in SI E.1. For this purpose, we will consider 3 real data sets: (i) The Human Genome Dataset (HGD), (ii) MNIST and (iii) CelebA, see details in the SI D. To show the occurrence of bonafide phase transitions, it is important to show the effect of increasing the system size (which transforms cross-overs in sharp transitions in the large N v limit). We will therefore resize these data sets in different dimensions by adjusting their resolution, i.e. by changing N v while maintaining comparable statistical properties. Detailed information about the scaling process can be found in the SI D.

In real training processes, the machine is expected to gradually learn different patterns ξ α , from the data, as described in the previous sections. However, the identification of these patterns and their relationship to the statistical properties of the dataset remains unclear. Previous research [9, 8, 35] has shown that RBM training begins with the stepwise encoding of the most significant principal components of the dataset, { η α } , which are the eigenvectors of the sample covariance matrix with the highest eigenvalues, on the SVD decomposition of its weight matrix W ia = ∑ α w α u α i ¯ u α a , where u α ∈ R N v and ¯ u α ∈ R N h denote the left and right singular vectors corresponding to the singular value w α . These vectors form orthonormal bases in R N v and R N h respectively, where the index α ranges from 1 to min( N v , N h ) and the singular values w α are arranged in descending order. At the beginning of the learning process, the left singular vectors, u α , gradually align α -toα to η α . This is consistent with the analytical results in our simple setting in the previous sections. In analogy to the mean-field magnetic models proposed in the previous section, the role of decreasing temperature is played by the increasing magnitude of the singular value w 2 α , associated with each mode α , and should lead to a series of phase transitions where the RBM measure splits into increasingly more and more modes. We show in Fig. 2 that these phenomena are at play by focusing on the evolution of the SVD of the RBM weight matrix when trained with the HGD dataset.

In panel A we show the first two principal components of the dataset, which highlights its strong multimodal structure, as several distant clusters appear (in this case, they are related to the continental origin of the individuals at hand). In Fig. 2-B we show the sharp and sudden increases of the singular values w α , as expected from our theoretical analysis, and in Fig. 2-C the evolution of the scalar product between u α and η α as a function of the number of training epochs. Different colors indicate different values of α . As expected, the modes are progressively expressed during training, and the first two singular vectors match the two principal directions of the dataset for a while. This last figure also shows us that the alignment with the PCA is only temporary (a limitation of current theoretical approaches), as the machine finds better patterns to encode the data as training progresses.

The progressive splitting of the RBM measure during the training dynamics is shown in Fig. 2-D, for which we use N s = 1000 independent samples generated with the model trained up to a different number of epochs (the colors refer to the same epochs highlighted with vertical lines in Figs. 2-B and C). For visualization, we show the samples projected onto the right singular vectors of W , the magnetizations m α = v · u α / √ N v with α = 1 , 2 . At the beginning of training, the data points are essentially Gaussian distributed, and the growth of w 1 over 4 is related to the splitting of the data into two different clusters on the m 1 axis, and the emergence of w 2 is related to a second splitting on the m 2 axis. At this stage of training, the projections along all subsequent directions are Gaussian distributed as they are the result of a sum of random numbers (fixed by the random initialization of the weight matrix). This progressive splitting is crucial to express the diversity of the dataset shown in Fig. 2-A, and can be successfully used to extract relational trees to cluster data points, as recently shown in Ref. [16]. The details about the numerical analysis are given in E.2.

At the beginning of training, when only a singular value has been expressed, and thus W ≈ w u ¯ u ⊤ , the transition of the feature encoding process is analogous to the phase transition from the paramagnetic to the ferromagnetic phase in the Mattis model mentioned above with pattern u . The detailed justification can be found in SI F. Our analysis allows us to define an effective temperature, linked to the eigenmode of W as β = w 2 / 16 . Now, since the critical temperature of the Mattis model is β c = 1 , we can show that the BB-RBM will condensate when the first eigenmode of the model

A

B

C

F

G

E

D

Nv

3

9

142 —192 —2

282—392—562

4

6

10

N、 = 282

101

a"p//-Nrux

10-1

5.0

lux

100

2.5

10-1

-

4/(w2,。 - w²)

10-2

0.0

0

2

4

6

0

1000

W1

(w2,c - w)N1/dauv

100

ma

-- 0.05/(1/w² - 1/w,c)

ma

Nv = 282

101

[sps NN] d

104

10-1

MM

一

q

oux

100

102

一

100

10-1

4/(w²,c — w²)

100

0

2

4

6

3

4

5

6

7

0

1000

2000

3000

4000

Wα

W1

epochs

reaches w c = 4 , see SI F. In a real training, we also have visible b and hidden bias c which could easily change the model towards a random field Mattis model, which leads us to expect a slightly higher critical point but a very similar ferromagnetic phase transition, and in particular, it should not change the transition's mean-field universality class.

To show that there is a cascade of transitions and that what was found for the HGD also holds for other datasets, we now train the RBM with the MNIST dataset. In Fig. 3-A we plot the evolution of the singular values w α along the training, which clearly show the progressive encoding of patterns. The progressive splitting of the RBM measure into clusters and the presence of a phase transition can be monitored by measuring the variance of the distribution of the visible magnetizations m α along the α -th mode or the analogous hidden magnetizations ¯ m α = h · ¯ u α / √ N h obtained using the hidden units. The variance of the magnetization multiplied by the number of variables used to compute it and β , is related to the magnetic susceptibility via the fluctuation dissipation theorem, which means that here ⟨·⟩ refers to the equilibrium measure with respect to RBM's Gibbs measure p ( v , h ) , in practice estimated as the average over N s independent MCMC runs. It is well known that the magnetic susceptibility should diverge in the vicinity of a second order phase transition and that such growth in only limited by the overall system size N = √ N v N h in finite systems. These phenomena indeed takes place also in the RBM. We show in Fig. 3-B the evolution of the χ m s obtained using the magnetizations obtained along the different modes α of W . As anticipated, the susceptibility χ m 1

A

B

A

C

D

E

wo=4

102

ap/l-1

0.25

10l

X

Z

lux

10-1

1282

642

0.00

100

--4/(w²,c —w²)

10-2

902

452

-0.25

wo =4.4

loop speed

3

4

5

-500

0

500

0.25

-10/8h

(w2,c - w)N1/dauv

W1

0.00

-100/8h

--1000/8h

-0.25

100

10000/8h

101

=0m

lux

0.25

100

lux

10-1

402

+805

m

0.00

--4/(w²。-w²)

-0.25

0

3

691215

0

500

1000

-0.10.00.1

W1

(w2,c — w})N1/duv

h

associated to the magnetization m 1 along the first mode, sharply grows as w 1 approaches 4, but it is more remarkable that this behavior is not only restricted to the first mode, but it is also reproduced by the subsequent modes in a step-wise process. According to the mapping between the low-rank RBM and the Mattis (or equivalently, the Curie-Weiss) model, we should expect that our χ m , at least for the first mode α = 1 should behave as

when approaching the critical point, which is equivalent to stating that the critical exponent is γ = 1 . Here, the factor 4 in the numerator is related to the fact that the susceptibility obtained with { 0 , 1 } variables is 4 times the standard one obtained with Ising spins, and that β = w 2 / 16 . In Fig. 3-D, we show the susceptibility associated with the first mode as a function of w 1 using RBMs trained with MNIST data rescaled to different dimensions. As mentioned earlier, the growth of the susceptibility is limited by the system size N v . However, if we look at increasingly larger sizes, we can observe the growth over several decades. This shows that at the transition we observe the Mattis/Curie-Weiss behavior of Eq. 6, as shown in the black dashed line, where the only adjustable parameter was the critical point w 1 , c = 4 . 45 (i.e. there is no adjustable pre-factor).

One of the crucial tests to ensure that a finite-size transition is a bona fide phase transition is to study its behavior by changing the number of degrees of freedom. One of the standard tools to do this is to make use of the so-called finite-size scaling (FSS) ansatz, motivated by renormalization group arguments [36, 37, 38]. Mean-field models follow a modified FSS ansatz which was first studied in [39]. In particular, the FSS ansatz for the susceptibility is with ϕ ( · ) a size-independent scaling function, N = √ N v N h is the effective size of our model and γ = 1 , ν = 1 / 2 and d u = 4 as expected in the mean-field universality class. We test this ansatz in Fig. 3-E showing that it does succeed to scale the finite-size data in the critical region, especially in the largest system sizes, which confirms both the mean-field universality class and the prevalence of the transition in the thermodynamic limit. In Fig. 4-A and B, and Fig. 4-C and D, we show that the indicators of a phase transitions-growth of the susceptibility and its mean-field finite size scaling-also holds for CelebA and HGD datasets. Finally, a final piece of evidence of the existence of a phase transition is presented in Fig. 4-E, where we show that after the continuous transition has taken place, one can induce a discontinuous transition and hysteresis effects by applying a field in the direction of

the learned pattern, in full agreement with what observed for standard phase transitions, the details of the analysis can be found in SI E.2.2, and further theoretical insights into the relationship between hysteresis and discontinuous transitions can be found in G.

All the discussion so far has been mainly concerned with the first phase transition, when the RBM learns the first mode. But we have discussed in Fig. 3-B that an entire sequence of step-wise phase transitions occurred in the rest of the W -matrix modes. In Fig. 3-F we show each of these mode susceptibilities χ m α as a function of their corresponding singular value. They show extremely similar divergent behavior with respect to the mode α = 1 , with an apparent slight variation of the critical point for each mode, although all appear to remain close to the predicted w α ∼ 4 , suggesting that the subsequent transitions may be of similar mean-field nature. Exceeding second-order phase transitions has a very strong impact on the overall quality of the training, in particular on the quality of the log-likelihood gradient estimated by MCMC dynamics. Indeed, second-order transitions are associated with a well-known arresting effect, known as critical slowing-down behavior, by which the thermalization times diverge with the correlation length ξ z ∝ | β -β c | -νz , where z is the dynamical critical exponent, which is 2 for local and non-conserved order parameter moves in mean-field, making the thermalization of large systems extremely difficult in practice. We show that our exponential relaxation times diverge exactly as predicted in Fig. 3-G. This has a significant impact on the quality of models trained with maximum likelihood approaches, as these methods rely on MCMC to estimate gradients. It is therefore expected that MCMC mixing times increase sharply each time a mode is coded, which can be prohibitive for clustered and high-dimensional datasets. Recent studies have shown that pre-training a low-rank RBM using other methods (and thus bypassing the initial phase transitions) can be very effective in improving the models in clustered datasets [40]. However, we emphasize that the cascade of phase transitions described in this paper occurs regardless of the training scheme or whether the Markov chains reach equilibrium. This is discussed further in SI H.

Extensions and limitationsAll these results can be studied in detail for RBMs thanks to the fact that we can analytically deal with the Hamiltonian. However, our results can be extended to the case of Deep Boltzmann Machines, where previous works have also computed the phase diagram, which are also based on the SVD decomposition of the weighing matrices [11], but also in diffusion models where phase transitions linked to the learning has also been described [24, 26]. It therefore stands to reason that similar phenomena occur with even more complex models such as Convolutional EBM, but where it is not clear how the parameters of the model can be decomposed. A first test would be to see what the projection of the generated data would look like in the different phases of learning.

6 Conclusions

In this paper, we first characterized the learning mechanism of RBMs using a simplified setting with a dataset provided by a simple teacher model. We used two examples: one with two symmetric clusters and another with four correlated clusters. Our results show that the learning dynamics identify modes by exponential growth in the directions of the clusters dominated by the variances of these clusters. The theory predicts the timing of the first phase transitions and agrees well with [8]. Numerically, we have confirmed the existence of a cascade of phase transitions associated with the growing modes w α and accompanied by divergent susceptibility. Finite-size scaling suggests that these transitions are critical and fall into the class of mean-field universality. This set of phase transitions likely goes beyond RBMs and offers insights into learning mechanisms, particularly for generative models. These transitions have significant implications for both training and understanding the learned features. During training, each transition is associated with a divergent MCMC relaxation time, which requires careful handling to properly train the model. In addition, the hysteresis phenomenon ensures that the learning trajectory involves second-order phase transitions, which are beneficial for tracking the emergence of modes in the learned distribution. However, changing parameters (such as the local bias) could lead to first-order transitions that are detrimental to sampling and could explain the ineffectiveness of parallel tempering in the presence of temperature changes. In practice, our analysis shows that the principal directions of the weight matrix contain valuable information for understanding the learned model.

7 acknowledgments

The authors would like to thank J. Moreno-Gordo for his work and discussions during the initial phase of this project. The authors (A.D. and B.S.) acknowledge financial support by the Comunidad de Madrid and the Complutense University of Madrid (UCM) through the Atracción de Talento program (Refs. 2019-T1/TIC-13298), and to project PID2021-125506NA-I00 financed by the Ministerio de Economía y Competitividad, Agencia Estatal de Investigación (MICIU/AEI /10.13039/501100011033), and the Fondo Europeo de Desarrollo Regional (FEDER, UE).

References

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681-4690, 2017.

Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. Advances in Neural Information Processing Systems, 32, 2019.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energybased learning. Predicting structured data, 1(0), 2006.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.

Aurélien Decelle, Giancarlo Fissore, and Cyril Furtlehner. Thermodynamics of restricted boltzmann machines and related learning dynamics. Journal of Statistical Physics, 172:15761608, 2018.

Aurélien Decelle, Giancarlo Fissore, and Cyril Furtlehner. Spectral dynamics of learning in restricted boltzmann machines. Europhysics Letters, 119(6):60001, 2017.

Ruslan Salakhutdinov and Geoffrey Hinton. Deep boltzmann machines. In Artificial intelligence and statistics, pages 448-455. PMLR, 2009.

Yuma Ichikawa and Koji Hukushima. Statistical-mechanical study of deep boltzmann machine given weight parameters after training by singular value decomposition. Journal of the Physical Society of Japan, 91(11):114001, 2022.

Moshir Harsh, Jérôme Tubiana, Simona Cocco, and Remi Monasson. 'place-cell'emergence and learning of invariant data with restricted boltzmann machines: breaking and dynamical restoration of continuous symmetries in the weight space. Journal of Physics A: Mathematical and Theoretical, 53(17):174002, 2020.

Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.

Andrew M Saxe, James L McClellans, and Surya Ganguli. Learning hierarchical categories in deep neural networks. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 35, 2013.

Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428-446, 2020.

Aurélien Decelle, Beatriz Seoane, and Lorenzo Rosset. Unsupervised hierarchical clustering using the learning dynamics of restricted boltzmann machines. Physical Review E, 108(1):014110, 2023.

Jorge Fernandez-De-Cossio-Diaz, Thomas Tulinski, Simona Cocco, and Rémi Monasson. Replica symmetry breaking and clustering phase transitions in undersampled restricted boltzmann machines. 2024.

Kenneth Rose, Eitan Gurewitz, and Geoffrey C Fox. Statistical mechanics and phase transitions in clustering. Physical review letters, 65(8):945, 1990.

David Miller and Kenneth Rose. Hierarchical, unsupervised learning with growing via phase transitions. Neural Computation, 8(2):425-450, 1996.

Tony Bonnaire, Aurélien Decelle, and Nabila Aghanim. Cascade of phase transitions for multiscale clustering. Physical Review E, 103(1):012105, 2021.

N Barkai and Haim Sompolinsky. Statistical mechanics of the maximum-likelihood density estimation. Physical Review E, 50(3):1766, 1994.

Thibault Lesieur, Caterina De Bacco, Jess Banks, Florent Krzakala, Cris Moore, and Lenka Zdeborová. Phase transitions and optimal algorithms in high-dimensional gaussian mixture clustering. In 2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 601-608. IEEE, 2016.

Giulio Biroli and Marc Mézard. Generative diffusion in very large dimensions. 2023(9):093402, oct 2023.

Giulio Biroli, Tony Bonnaire, Valentin de Bortoli, and Marc Mézard. Dynamical regimes of diffusion models. arXiv preprint arXiv:2402.18491, 2024.

Antonio Sclocchi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data. arXiv preprint arXiv:2402.16991, 2024.

Gabriel Raya and Luca Ambrogioni. Spontaneous symmetry breaking in generative diffusion models. Advances in Neural Information Processing Systems, 36, 2024.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

V. Nair and G.E. Hinton. Rectified linear units improve restricted Boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807814, 2010.

Aurélien Decelle, Cyril Furtlehner, and Beatriz Seoane. Equilibrium and non-equilibrium regimes in the learning of restricted boltzmann machines. Advances in Neural Information Processing Systems, 34:5345-5359, 2021.

DC Mattis. Solvable spin systems with random interactions. Physics Letters A, 56(5):421-422, 1976.

John J Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8):2554-2558, 1982.

Giorgio Parisi and Ramamurti Shankar. Statistical field theory. 1988.

Francisco A Tamarit and Evaldo MF Curado. Pair-correlated patterns in hopfield model of neural networks. Journal of statistical physics, 62:473-480, 1991.

1000 Genomes Project Consortium et al. A global reference for human genetic variation. Nature, 526(7571):68, 2015.

Aurélien Decelle and Cyril Furtlehner. Restricted boltzmann machine: Recent advances and mean-field theory. Chinese Physics B, 30(4):040202, 2021.

Kurt Binder. Finite size scaling analysis of ising model block distribution functions. Zeitschrift für Physik B Condensed Matter, 43:119-140, 1981.

Daniel J Amit and Victor Martin-Mayor. Field theory, the renormalization group, and critical phenomena: graphs to computers. World Scientific Publishing Company, 2005.

John Cardy. Finite-size scaling. Elsevier, 2012.

E Brézin. An investigation of finite size scaling. Journal de Physique, 43(1):15-22, 1982.

Nicolas Béreux, Aurélien Decelle, Cyril Furtlehner, Lorenzo Rosset, and Beatriz Seoane. Fast, accurate training and sampling of restricted boltzmann machines. arXiv preprint arXiv:2405.15376, 2024.

Marc Mézard. Mean-field message-passing equations in the hopfield model and its generalizations. Physical Review E, 95(2):022117, 2017.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. CoRR, abs/1710.10196, 2017.

Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning, pages 1064-1071, 2008.

Frank den Hollander. Metastability under stochastic dynamics. Stochastic Processes and their Applications, 114(1):1-26, 2004.

Anton Bovier. Metastability, pages 177-221. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009.

Paul M Chaikin, Tom C Lubensky, and Thomas A Witten. Principles of condensed matter physics, volume 10. Cambridge university press Cambridge, 1995.

Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800, 2002.

Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems, 32:5232-5242, 2019.

A Binary-Gauss RBM

We add some technical details to the derivation of the dynamical process. We first recall the definition of general BG-RBM first. BG-RBMs consist of a bipartite model where the visible nodes v are binary (in our case ± 1 ) and the hidden nodes are Gaussians. The Hamiltonian follows the same expression as the BB-RBM, Eq. 1. The hidden nodes are Gaussian random variables centred in zero with variance σ 2 h . In our analysis where we use only one or few hidden nodes, it is important that the variance scales as the inverse of the system's size σ 2 h = 1 /N v in order for the Hamiltonian of the system to be an extensive property (proportional to N v ). This scaling is crucial for the analysis: if the energy term scales as the system's size, then it is possible to have two distinct phases: for small weight magnitude, the system is disordered (it does not polarize in any particular direction) while for large values of w the system will polarize toward one of the direction encoded in the weight matrix. A more detailed description can be found in [30, 41].

Now, recall that we consider the Mattis model, biased toward a pattern ξ for generating the dataset

where β is the inverse temperature β = 1 /T and ξ i = ± 1 represents a potential pattern direction. The Mattis model presents a high-temperature phase with a single model centred over zero magnetization m = N -1 N v ∑ i v i = 0 for β < β c while in the low-temperature regime, β > β c , the model exhibits a phase transition between two symmetric modes m = ± m 0 ( β ) . From the analytical point of view, we can compute all interesting quantities in the thermodynamics limit N v →∞ . The RBM's distribution is given by

Using this model for the learning, the time evolution of the weights is given by the gradient. With BG-RBM we have that

where the last average is taken over a distribution p ( v ) . We can now easily compute the positive and negative term of the gradient w.r.t. the weight matrix. For the positive term, assuming that β > 1 , we obtain that

Evaluating the saddle point of the argument of the exponential (which is the same as the one for the partition function) we have that

The negative term can also be computed in the thermodynamic limit

where the last line is obtain by taking the saddle point of the integral over h , ( h ∗ corresponding to the extremum). We can now express the gradient as

Assuming first that the weights are small we get that h ∗ = 0 . We can solve the gradient's equations in this regime. In such case, the only solution for the saddle point equation of the RBM is given by h ∗ = 0 and we can see that the solution of the evolution of the weight is global toward the direction ξ by projecting the differential equation on the preferred direction. Defining U ξ = N -1 / 2 v ∑ i ξ i w i , we obtain

This shows that the weights are growing in the direction of ξ while the projection on any orthogonal direction ϕ α stays constant:

When the weights grow larger, the solution for h ∗ will depart from zero. The correlation of the learning RBM then starts to grow

and diverges when N -1 v ∑ i w 2 i ∼ 1 , therefore exhibiting a second order phase transition during the learning. Finally, we can study the regime where the weights are not small. In that case, we can first observe that the evolution of the directions orthogonal to ξ , ϕ α are given by

which will cancel if the weight W aligns totally with the ξ . Finally, taking w i = ξ i w, the gradient projected along ξ at stationarity imposes

B Binary-Binary RBM

The RBM sharing both discrete binary variables on the visible and hidden nodes is by far the most commonly used. In particular, using binary nodes in the hidden layer instead of the Gaussian distribution allows the model to potentially fit any order correlations of the dataset. In this section, we

review how the learning dynamics translate to this case, using for simplicity binary {± 1 } variables. In order to obtain an interesting behavior in this phase of the learning, it is important to consider a particular parametrization of the RBM. We consider that all hidden nodes share the same weight. This is important to be able to have a recall phase transition in the model. We therefore have the following Hamiltonian where N h = αN v is the number of hidden nodes h a of the system and the vector W correspond to the weight shared across all the hidden nodes. In this model, we can now compute the positive and negative of the gradient. The first one is given by

finding the saddle point of the argument in the exponential, we obtain

The same type of computation can be done for the negative term, we found that

Again, in the small coupling regime (or at the beginning of the learning), when N -1 h ∑ j w 2 j ≪ 1 , we have that τ = 0 . In such case, the gradient over the weight matrix is given by following the same approach as in the main text, we project the weights on the unit vector u 1 = ξ / √ N v , U ξ = u 1 W , which gives

We can integrate this equation, obtaining the solution

0

100

200

300

400

Number of epochs

10

-

2

10

-

1

10

0

1

N

h

∑

wia

N

h

=400,

/epsilon1

=0.03

N

h

=700,

/epsilon1

=0.04

N

h

=1000,

/epsilon1

=0.05

∝

exp(

m

2

t/α

)

w

∗

0

100

200

300

400

Number of epochs

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

w

α

=1

u

α

=1

·

ξ

τ

∗

where the second line is obtained in the very large N v limit. Again we have an exponential growth in the first steps of the learning. At the end of the learning, the weights again align in the direction of ξ . This can be checked by the fact that the positive term of always orthogonal to any vector orthogonal to ξ , and thus the simplest option for the gradient projected in those direction is to be orthogonal to ξ . Taking W = u 1 w , we obtain

The solution can be found numerically by solving the fixed point equation on τ , and measuring the magnetization of the dataset. In Fig.5 we illustrate our results in the same dataset as in the section 4.1, taking the Mattis model with N v = 900 , β = 1 . 4 , varying the learning rate and the number of hidden nodes.

C Learning with correlated patterns

In this part we detail how the learning goes when considering a pair of correlated patterns. As described in 4.2, the pairs of patterns are defined as

where η 1 is a vector whose first N v 1+ κ 2 components are equal to ± 1 with equal probability and the remaining ones are zero. The other vector η 2 has its last N v 1 -κ 2 components equal to ± 1 with equal probability and the rest are 0; we also have that κ ∈ [0 , 1] . When κ = 1 , both patterns ξ 1 , 2 are equal, while otherwise different but correlated. In particular E η 1 , η 2 [ ξ 1 ξ 2 ] = N v κ . Following the results of [33], it is possible to compute the saddle point equations for the magnetization. The general form is given by

This system has been solved in [33] and exhibits the following properties. When T > 1 + κ , the system is in the paramagnetic regime and m 1 = m 2 = 0 . When the temperature is lowered and lies

in 1 -κ < T < 1 + κ , the solution is given by the pair retrieval m 1 = m 2 = m = 1+ κ 2 tanh(2 βm ) . Finally, when T < 1 -κ , the system condensates on the following solution

where basically the system either condensates toward one of the pattern ξ 1 , 2 , while the other magnetization has some non-zero value due to the correlation.

We can use the thermodynamics properties of this model to study how the learning of the RBM should behave in the regime T < 1 -κ . In order to use this model as generating the dataset, we need to compute the correlations ⟨ s i s j ⟩ . The model presents four fixed points, all equally probable:

where m + > m -> 0 . Therefore, writing r = tanh( β ( m + + m -)) and p = tanh( β ( m + -m -)) we have that

because the cross terms η 1 i η 2 j are canceled when changing ( m 1 = m + , m 2 = m -) to ( m 1 = m -, m 2 = m + ) . At this point, it is possible to write the gradient at the linear order and project it toward both direction η 1 and η 2 . Denoting U a η 1 = η 1 · w a and U a η 2 = η 2 · w a , we get

Using this form, we end up with the following solution of the weight matrix

We therefore understand the following. At the beginning of the learning, since r > p , what is learned first is the mode toward the direction η 1 ∝ ξ 1 + ξ 2 , in a timescale that is given by time t ∼ 1 /r 2 . At a different timescale, the part that is aligned with η 2 will grow as well as discussed in the main text. Following the dynamics of the weights, as in eq. 11, we can infer the moment where the phase transition occurs. When considering an Hopfield like model, we know that the transition happens when that is, the critical temperature is β c = 1 . Following the dynamics of the weights of eq. 11, and neglecting the terms that are not aligned with η 1 we can write

where we can identify a sort of dynamical temperature associated to the pattern η 1 . Now, we need to be careful since by definition, the pattern η 1 is made of random ± 1 components on its (1 + κ ) / 2

24

0

200

400

600

800

1000

1200

Number of epochs

10

-

3

10

-

2

10

-

1

10

0

t

I

t

II

1+

κ

2

A

(

t

)

2

1

-

κ

2

B

(

t

)

2

˜

w

2

1

(

t

)

˜

w

2

2

(

t

)

-

1

0

1

h

2

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

2

.

0

2

.

0

2

.

0

2

.

0

2

.

2

2

.

2

2

.

2

2

.

2

2

.

4

2

.

4

2

.

4

2

.

4

2

.

6

2

.

6

2

.

6

2

.

6

2

.

8

2

.

8

2

.

8

2

.

8

3

.

0

3

.

0

3

.

0

3

.

0

3

.

2

3

.

2

3

.

2

3

.

2

3

.

4

3

.

4

3

.

4

3

.

4

t=3

0

.

00

0

.

00

0

.

15

0

.

30

0

.

45

0

.

60

0

.

75

0

.

75

0

.

90

0

.

90

1

.

05

1

.

05

1

.

20

1

.

20

1

.

35

1

.

35

1

.

50

1

.

50

1

.

65

1

.

65

1

.

80

1

.

80

1

.

95

1

.

95

1

.

95

1

.

95

2

.

10

2

.

10

2

.

10

2

.

10

2

.

25

2

.

25

2

.

40

2

.

40

t=401

-

1

0

1

h

1

-

1

0

1

h

2

-

0

.

15

-

0

.

15

0

.

00

0

.

15

0

.

30

0

.

30

0

.

45

0

.

45

0

.

60

0

.

60

0

.

75

0

.

75

0

.

90

0

.

90

1

.

05

1

.

05

1

.

20

1

.

20

1

.

35

1

.

35

1

.

50

1

.

50

1

.

65

1

.

65

1

.

65

1

.

65

1

.

80

1

.

80

1

.

95

1

.

95

1

.

95

1

.

95

t=501

-

1

0

1

h

1

-

0

.

24

-

0

.

24

-

0

.

18

-

0

.

18

-

0

.

12

-

0

.

12

-

0

.

06

-

0

.

06

0

.

00

0

.

00

0

.

06

0

.

06

0

.

12

0

.

12

0

.

18

0

.

18

0

.

0

.

24

0

.

30

0

.

30

0

.

36

0

.

36

0

.

42

0

.

42

0

.

48

0

.

48

0

.

48

0

.

48

0

.

54

0

.

54

0

.

54

0

.

54

0

.

60

0

.

60

0

.

66

0

.

66

0

.

66

0

.

66

0

.

72

0

.

72

0

.

78

0

.

78

t=1401

elements and zero elsewhere. This rescales the critical temperature by a factor (1 + κ ) / 2 . Therefore we need to look when

and the same kind of argument can be used for the second transition with this time

Weshow in Fig. 6, left panel how the times t I and t II compare with the moment where the eigenvalues w α of the weight matrix cross the value one, which correspond to the phase transition following a statistical mechanics approach [8]. We observe that both indicators are crossing the line y = 1 at the same moment. In Fig. 6, right panel, we plot the behavior of the free energy (in the plane ( h 1 , h 2 ) ). We see that at the moment of the transition, the free energy opens in the direction corresponding to the transition. Projecting the dataset (black dots) in the same direction as h 1 (resp. h 2 ), we can see how the system correctly positioned the minima once fully trained.

D The datasets and the rescaling

In this work, we illustrated our results on three datasets:

The Human Genome Dataset (HGD) [34] containing binary vectors, each representing a selection of 805 genes from a human individual, where 1s or 0s indicate the presence or absence of gene mutations relative to a reference sequence.

The MNIST dataset [42], containing 28 × 28 pixel black and white images of digitized handwritten digits.

The CelebA [43] dataset, in black-and-white, with 128 × 128 pixel images of celebrities faces.

The datasets MNIST and CELEBA, were either downscaled or upscaled in order to create dataset of various sizes. In practice, the function resize from the python library skimage was used either to increase or decrease the image size. The dataset HGD is geometrically a one-dimensional structure. In order to reduce its size, we took the convolution of each sample with a kernel of size s = 3 . The output is one if the sum of the three input values (that are discrete variables in { 0 , 1 } ) of the kernel

is above the threshold 2 and zero otherwise. A stride of 2 has been chosen such that the resulting samples has its size reduced by a factor two.

E Details on the training and the numerical analysis

E.1 Training

All RBMs analyzed in the main text were trained with the Persistent Contrastive Divergence (PCD) method [44] and k =100 MCMCsampling steps per parameter update, to approximate the negative term of the log-likelihood gradient in Eq. (2). In this scheme, the last configurations reached in the MCMC process to compute the previous update are used as initialization of the new chains used to compute the subsequent update. This scheme tends to favor the equilibrium regime [29]. The results with other training schemes is discussed in section H. Moreover, as usual, we keep N chains independent parallel Markov chains. For simplicity, N chains is chosen to match the minibatch size used to estimate the positive term of the gradient. The code to reproduce the experiments is freely available in https://github.com/AurelienDecelle/TorchRBM. The hyperparameters used for each training (no. of visible and hidden units N v and N h , respectively, learning rate ϵ or minibatch size N ms ) are given in Table 1.

E.2 Numerical analysis

E.2.1 Susceptibility

Part of the analysis in section 5 of the main text is based on sampling the equilibrium configurations of models trained at different epochs to extract the moments of the distribution of magnetizations m α , i.e. the projection of the samples along the different α -th singular vectors of W . For this purpose, we automatically selected 10 3 models uniformly in logarithmic scale in training time and annealed the N s = 1000 independent samples from the least trained model to the most trained model, following the hot-to-cold thermal analogy, i.e. we perform N mesfr alternate Gibbs sampling MCMC steps on each set of model parameters and use the last achieved configurations as a starting point to initialize the run on the next set of model parameters. The less trained model is initialized randomly. In parallel, we also consider the reverse scheme, where we consider a heating annealing. We start with the most trained machine, where all visible units are initialized to 1 (to force all initial configurations to be in a single cluster), and move backwards it in training time. We systematically checked that both analyses gave the same results for N mesfr = 1000 in the region of interest (the critical region) in Fig. 3 and 4.

In addition to the sampling procedure, we computed the SWD of the matrix W at each new parameter set and projected each visible configuration v along each of the left singular vectors u α , to obtain m α = ( u α · v ) / √ N v , and each hidden configuration h along the right singular ¯ u α vector, to obtain ¯ m α = (¯ u α · v ) / √ N v . The distribution moments are later estimated using the sample mean and variance of these N s measures.

E.2.2 Hysteresis loop

To investigate the hysteresis behavior between different lumps at or above the phase transition, we first select the training epochs in which the first singular values of W , w 1 , take the values 4, 4.4 and 5. Note that for MNIST w 1 , c is approximately 4 . With the model parameters at each of these three selected number of updates, we perform N s independent MCMC runs with the tilted Hamiltonian from Eq. (G). In these runs, the external field h is gradually varied to trace a loop: starting from h = 0 , we slowly increase it to h max , then decrease it to -h max and finally bring it back to h = 0 . Again, the last configurations reached at a certain h are used as initialization for the next one. In practice, we have chosen h max /N 0 . 75 v , the increment in the field as δh = 2 × h max /N loop and N loop = 50 . We can modulate the speed of the loop by performing a different number of MCMC steps k at each value of h . As shown in the figure, we consider k = 10 , 100, 1000 and 10 4 . The results shown in the main text were obtained using the RBM trained with the original CELEBA dataset (i.e. N v = 128 × 128 ), but the results obtained with other sizes and datasets are completely analogous.

F Link between the low-rank RBM and Mattis model

Let us consider a low-rank Ising-Ising RBM in which the W matrix has a single non-zero singular value ω , with left and right singular vectors u and ¯ u , and visible and hidden Ising variables (let's call these variables W , s and τ to distinguish them from the binary 0 , 1 version, which would be W and v and h ). In this case, the energy function of the RBM (if we ignore the biases for now) is

which leads to a marginal energy on the visible

where we have defined m = u · v / √ N v as the magnetization of the spins along the direction u and have exploited the fact that ∑ a ¯ u 2 a = 1 because it is a unit vector. One can obtain an analogous expression for the marginal energy on the hidden units, formulated in terms of the hidden magnetization ¯ m = ¯ u · τ / √ N h . These energy functions, for small m or ¯ m , are formally equal to those of the Mattis model for β = ω 2 , which means that our RBM should manifest a critical phase transition at β c = T -1 c = ω 2 c = 1 , with mean-field critical exponents. Standard RBMs are not formulated as Ising ± 1 variables, but in the form of binary { 0 , 1 } variables where we have the equivalence 4 W = W between the couplings matrices. This results in a critical point at w c = 4 and an effective inverse temperature β = w 2 / 16 .

G Hysteresis in discontinuous transitions

The hysteresis phenomenon shown in Fig. 4-E is a classical measure in statistical physics and is a unique signature developed and observed in statistical physics to reveal that a high-dimensional probability measure had a phase transition where it splits into two distinct lumps. The procedure is to tilt the probability measure by introducing a contribution in the energy function that favors one lump over the other. In the present case, we use the learned preferred direction associated with the first phase transition u , which is also the direction in which the probability measure splits, and we add a magnetic field to break the symmetry between the two modes created by the learning process. Therefore, we tilt the measure by adding

The tilted Hamiltonian favors one of the two lumps depending on the value of the local bias h . When the measure is actually concentrated on two distinct lumps, one of them leads to a sudden discontinuous transition ('first-order transition' in physics). In our case, the lumps are associated with the learned patterns and this additional contribution consists of the scalar product between the visible variables and the times of the learned patterns (the field that controls the strength of the tilting). In the presence of a first-order phase transition, one usually finds the phenomenon of hysteresis, i.e. the transition from one clump to another can be delayed due to metastability, leading to the characteristic

Nv = 282 CD-10

Nv = 282 Rdm-10

N5

100

0

100

10-1

ma

mα

101

2 ×100

100

以

100

b

6 × 10-1

b

4 × 10-1

b

3×10-1

0

2000

4000

0

2000

4000

epochs

epochs

hysteresis loops that we show in Fig. 4-E (see e.g. [45, 46] for a rigorous treatment and [47] for a physical treatment). This figure thus provides direct evidence for the decomposition of the measure into distinct lumps corresponding to the learned patterns, and that this decomposition occurs at the second-order phase transition that occurs during learning. The details about the numerical implementation are given in Section E.2.2.

H Results with other training schemes

All the machines analyzed in the main text were trained using the PCD-100 scheme, which involves initializing the chains with the PCD method and performing 100 MCMC steps per parameter update. This approach ensures that we obtain models in good equilibrium, avoiding the non-monotonic behavior in sample quality typical of out-of-equilibrium regimes [29]. However, it is more computationally expensive than standard methods, where only k = 1 -10 steps are used, or alternative initialization strategies like Contrastive Divergence (CD) [48], where chains are initialized using the minibatch samples at each update, or the fully out-of-equilibrium regime (Rdm), where chains are always initialized randomly [29, 49].

In this appendix, we analyze RBMs trained with CD-10 and Rdm-10 strategies on the MNIST dataset. While the time evolution differs -often degrading susceptibility along learning directionsthe overall picture of the cascade of phase transitions remains unchanged. We show the equivalent to Figs. 3A,B,C with these two new trainings in Fig. 7.

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: .

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: .

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The theoretical analysis relies on theoretical physics development. In such context, the full set of assumptions is not state clearly, but the results is expected to be correct within the range of application. As such no theorem or lemma is provided.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The method used here is quite standard (RBM training in a simple case), all the rest is clearly specified.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The experiment perform are sufficiently precise within the claim of the paper so they do not need further error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: the experiments do not need particular resources and can be trained on personal laptop.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: .

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: .

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: .

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: .

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: .

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: .

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analytically demonstrates a cascade of second-order phase transitions in the training of a simple RBM model and numerically validates this theory in the training on real datasets.\n\nSTRENGTHS:\n- This paper theoretically analyzes the learning dynamics of weight parameters in the Binary (or Bernoulli)-Gaussian RBM (BG-RBM) through the lens of statistical physics.\n- The authors demonstrate that the RBM undergoes a series of second-order phase transitions during learning, both theoretically in the BG-RBM and numerically in the Binary-Bernoulli RBM (BB-RBM).\n\nWEAKNESSES:\n- There is a gap between the theoretical and numerical analysis in this paper. Specifically, the BG-RBM with a single hidden node is utilized in the theoretical analysis in Section 4, while the BB-RBM is used in the numerical analysis in Section 5.\n- Although the main results of this paper may merit acceptance into NeurIPS, I believe the authors should revise the manuscript to improve readability and clarity. Some suggestions are provided in the Questions section below. If the authors address these issues and answer my questions, I can strongly recommend accepting this paper for NeurIPS; however, I cannot recommend it for acceptance in its current form.\n\nQUESTIONS:\nMajor\n- While the authors consider a single Gaussian hidden node, they do not provide any statistical details of the Gaussian distribution, such as the mean and variance. I assume the authors consider a Gaussian distribution with zero mean and $1/N$ variance, but I am uncertain about the meaning of $N$ (I guess $N \\equiv N_v$), why the authors choose this setup, and how $N$ relates to the statistics of $xi_i$.\n- The meaning of the equation between lines 161 and 162 is unclear, especially when the authors mention that the equation “is analogous to the one studied in the previous section.”\n- While the authors analytically investigate the BG-RBM with a single hidden node in Section 4, the numerical analysis in Section 5 is conducted on the BB-RBM with multiple hidden nodes. The authors should discuss the expected differences due to this gap and adequately clarify the limitations of their analysis.\n- I am not sure why “the projections along all subsequent directions are Gaussian” as stated in line 231 on page 6.\n- It is unclear how the authors obtained Figure 4 E and what they aim to demonstrate. Does the $h$ in Fig. 4 E indicate the hidden variable?\n- The direct relationship between Section 4 and Section 5 is hard to capture. As I understand it, the main part of this paper introduces an underlying preferred direction $\\xi_i$, but I am not sure which vectors correspond to $\\xi_i$ in Sec. 5. Please describe the correspondence between Section 4 and Section 5 and clarify the differences from the results in Ref. [8].\n\nMinor\n- While the authors mention the Mattis model to describe $\\xi$, the explanation of $\\xi$ is insufficient. It is unclear whether $\\xi$ is frozen, the distribution of $\\xi$, etc. \n- In Fig. 1, the x-labels of the left and right panels differ. The term \"susceptibility\" is not defined, each line is indistinguishable, and the meanings of the left and right vertical axes are not indicated in the inset of the left panel. There is no indication of the black solid lines in the insets of the right panel.\n- In lines 127 and 439, the condition for diverging the correlation seems to be a typo. $N^{-1}\\sum_{i} w_i^2 \\approx 1$ may be correct.\n- Please introduce the learning rate $\\epsilon$ in Eq. (3) because the authors provide results with respect to the learning rate in Fig. 1.\n- Please provide references and detailed descriptions for the BG-RBM primarily considered in Section 4.\n- The authors should correct numerous typos in the main text and the appendix. For instance, the authors often use expressions such as $s_i$ and $N$, which are not defined in this paper. In line 107 on page 3 and line 416 on page 12, there are typos in the definition of magnetization $m$. In line 414 on page 12, there is a typo in the definition of $\\xi_i$. In Eq. (3), the authors do not clarify what $t$ indicates.\n- The authors should consistently use expressions throughout the paper. In lines 136 and 194, the word \"appendix\" is used, but in other cases, the term \"SI\" is used. The title of Appendix B is \"Binary-Binary RBM,\" but the model is often referred to as \"Bernoulli-Bernoulli RBM\" in the paper.\n\nLIMITATIONS:\nThe authors should discuss the limitations of their work more thoroughly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper presents an analysis of the phase transitions of RBMs through an analysis of the weight matrix. They find the dynamics tend toward the center, then diversity into modes. The theoretical results are supported by empirics on 3 ML datasets.

STRENGTHS:
- This expands the important field of exploring the dynamics of energy based models

- The figures are generally well presented (conditioned on note below)

- The authors offer a compelling argument for the scalability of these phase transitions, and the pipeline theory to CelebA is well done

WEAKNESSES:
- This paper would be right at home in a physical review journal, but given the audience of NeurIPS, I would recommend increasing the accessibility to the very ML focused audience (e.g. matching the notation used in ML world, things like labeling axes with descriptive words then putting the equations in text, numbering all equations, etc.)

- 199 paragraph is too long

QUESTIONS:
- How do you imagine these transitions changing with respect to sampling convergence? Perhaps out of scope of this work, but would be interesting to see all of these plots generate but now with x axis being sample steps and see how that looks

- What would be the biggest takeaways/generalization to be made from RBM phase transitions to more complex ML models?

LIMITATIONS:
The authors sufficiently addressed societal limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyses the phenomenon of phase transitions in the training dynamics of restricted Boltzmann machines (RBM) in theory and practice. In the theory part, the phase transition is characterised for a toy model: The RBM with a hidden node per feature is fitted to data characterised by one feature vector $\\xi$ or two correlated feature vectors $\\xi_1, \\xi_2$. The phase transitions are detected as diverging correlation functions between visible states of the RBM and via exponential fitting of the RBM weights to the target features. \n\nIn the empirical part, the phase transitions are explored in the training of a RBM on the Human Genome Dataset, MNIST, and CelebA. It is observed that the singular values of the weight matrix and the susceptibility undergo sudden changes during training, indicating a phase transition.\n\nSTRENGTHS:\n- The theoretical evaluation seems sound and in general the paper is well written.\n- The paper explores an interesting phenomenon in training dynamics of RBMs. The experimental evaluation of the phase transition is compelling and highlights the emergence of phase transitions well. Similar studies may be applicable to other types of generative models.\n\nWEAKNESSES:\n- It seemed to me that the notation switches between $v$ and $s$ to denote visible states/spins/sites.\n- I think the paper could be improved by making a stronger connection between theory and empirical evaluation. For example, do the projections and projected weights in the theoretical section correspond exactly to the discussed singular value decomposition described in the empirical evaluation? Can the susceptibility be analysed in the theoretical section? More consistent nomenclature of the terms described would help with reading and understanding the paper.\n- The numerical analysis would benefit from paragraphs or subsections.\n- The analysed dynamics of the RBM weights don't fully reflect the actual training dynamics. In practice, RBM training requires self-sampling from the model which causes biases in the estimated negative term. I would expect that the sampler is adversely affected by the phase transitions in the model (e.g. sudden drop in mixing time). Considering that sampling in an RBM is relatively tractable (compared to other types of EBM) I am wondering if the training dynamics of e.g. one step contrastive divergence can be analysed in a similar fashion. However, the authors do address the effect of the phase transition on the MCMC sampling in the experimental section.\n\nQUESTIONS:\n- Do the projections and projected weights in the theoretical section correspond exactly to the discussed singular value decomposition described in the empirical evaluation? Is a theoretical study of the susceptibility in the theoretical section feasible? \n- Can the phase transitions be detected during training to, for example, adjust for the longer thermalisation time in the vicinity of the phase transition? \n- When running a gradient based Langevin sampler during training, do your observations suggest spiking parameter updates in the vicinity of the phase transition due to second order discontinuities, thus creating potential training instabilities?\n\nLIMITATIONS:\nThe work analyses the phenomenon of phase transitions in RBM training dynamics, and is limited to this setting by definition. The scope is clearly defined in the paper. However, the authors checked the limitations box in the check list without providing further justification in the main text or in the checklist. A brief sentence about the scope of the work or a reference to lines in the main text would be appropriate.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work examines the relation between the internal representations of an RBM and the data that it is trained on as the training procedure goes forward. The work begins with a theoretical exposition for Binary-Gaussian RBMs trained CW-like models with 1 or 2 preferred encoding patterns. The theoretical analysis shows that the weights develop along directions that aggregate the directions of data modes before further settling into directions that capture each of the data modes. The analysis is extended to Binary-Binary RBMs. Several empirical results are presented. For simple models, theoretical findings are numerically validated. RBMs are also learned for HGD, MNIST, and CelebA dataset. The behavior of these learning situations mirrors certain behaviors of the simple cases. The RBM first learns patterns along the largest principal components of the data before undergoing a phase transition and settling into more nuanced representations.\n\nSTRENGTHS:\n* The dynamics of generative model training and the relations between the data and internal representations of generative models is an important area of study. I appreciate that this work undertakes the challenging task of unraveling these behaviors.\n* This work offers interesting evidence that, across simple and more complex datasets, RBM learning has somewhat predictable behavior. In particular, RBM weights will first learn to represent the data directions with the highest variation before learning more refined representations that better capture modal structure.\n* Extensive derivations and empirical results are included.\n\nWEAKNESSES:\n* A major weakness of this work is the scope. While broad claims are made about implications for other generative models, the analysis presented seems specific to the RBM architecture. Studying more recent generative models with stronger capacity would greatly increase the impact of this work, although such analysis is very difficult.\n* The work is quite dense, and it can often be difficult to follow the lines of reasoning and conclusions based on derivations. While the topic is complex and this is to some extent inevitable, I recommend the authors make efforts where possible to streamline and clarify the presentation.\n\nQUESTIONS:\nAre there conclusions from this work which can be extended to more powerful generative models such as diffusion models, deep EBMs, etc.?\n\nLIMITATIONS:\nLimitations are not discussed in detail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}