Title: Fast, accurate training and sampling of Restricted Boltzmann Machines
Venue: NeurIPS 2024
Accepted: no
Keywords: Restricted Boltzmann Machine

Fast Sampling

Multimodal learning

training algorithm
Abstract: Thanks to their simple architecture, Restricted Boltzmann Machines (RBMs) are powerful tools for modeling complex systems and extracting interpretable insights from data. However, training RBMs, as other energy-based models, on highly structured data poses a major challenge, as effective training relies on mixing the Markov chain Monte Carlo simulations used to estimate the gradient. This process is often hindered by multiple second-order phase transitions and the associated critical slowdown. In this paper, we present an innovative method in which the principal directions of the dataset are integrated into a low-rank RBM through a convex optimization procedure. This approach enables efficient sampling of the equilibrium measure via a static Monte Carlo process. By starting the standard training process with a model that already accurately represents the main modes of the data, we bypass the initial phase transitions.  Our results show that this strategy successfully trains RBMs to capture the full diversity of data in datasets where previous methods fail. Furthermore, we use the training trajectories to propose a new sampling method, {\em parallel trajectory tempering}, which allows us to sample the equilibrium measure of the trained model much faster than previous optimized MCMC approaches and a better estimation of the log-likelihood. We illustrate the success of the training method on several highly structured datasets.

Abstract

Thanks to their simple architecture, Restricted Boltzmann Machines (RBMs) are powerful tools for modeling complex systems and extracting interpretable insights from data. However, training RBMs, as other energy-based models, on highly structured data poses a major challenge, as effective training relies on mixing the Markov chain Monte Carlo simulations used to estimate the gradient. This process is often hindered by multiple second-order phase transitions and the associated critical slowdown. In this paper, we present an innovative method in which the principal directions of the dataset are integrated into a low-rank RBM through a convex optimization procedure. This approach enables efficient sampling of the equilibrium measure via a static Monte Carlo process. By starting the standard training process with a model that already accurately represents the main modes of the data, we bypass the initial phase transitions. Our results show that this strategy successfully trains RBMs to capture the full diversity of data in datasets where previous methods fail. Furthermore, we use the training trajectories to propose a new sampling method, parallel trajectory tempering , which allows us to sample the equilibrium measure of the trained model much faster than previous optimized MCMCapproaches and a better estimation of the log-likelihood. We illustrate the success of the training method on several highly structured datasets.

1 Introduction

Energy-based models (EBMs) are a classic approach to generative modeling that has been studied for decades. They were introduced using the Restricted Boltzmann Machine formulation by Smolensky [1] and later further developed by Sejnowski et al. [2]. They provide a straightforward method for modeling effective interactions within complex data distributions and for sufficiently simple energy functions, such as the Boltzmann machine (BM) [3], it is also possible to interpret and infer the underlying constituent rules from the observed data. This inference strategy is often associated with the inverse Ising problem and pairwise interaction models [4], and it has found a great variety of applications in fields such as neuroscience [5] or computational biology [6]. A recent work has proposed replacing the use of pairwise models with the Restricted Boltzmann Machine (RBM) [7], as it allows the same direct interpretation of its energy function as an explicit many-body interaction model while greatly extending the expressive power of the model. RBMs are also very useful for grouping data into hierarchical families [8]. On the diametrically opposite side (on interpretability) are generative ConvNets [9, 10], where the energy function is formulated as a deep neural network, which are capable of synthesizing photorealistic images but are almost impossible to interpret as a physical model.

The applications of simple EBMs in science are very diverse. For example, they are often used today 35

to encode the Hamiltonian of physical many-body systems, such as Quantum wave functions [11] 36

or the accurate determination of ground state wave functions of strongly interacting and entangled 37

45

46

47

48

49

50

51

52

53

54

55

56

Despite the appealing modeling properties of RBMs, they are notoriously difficult to train, a challenge common to EBMs in general. The main difficulty arises from the computation of the log-likelihood gradient, which requires an ergodic exploration of a dynamically evolving and potentially complex free energy landscape using Markov Chain Monte Carlo (MCMC) processes. Recent studies have shown that models trained with non-convergent MCMC processes suffer from out-of-equilibrium dynamic memory effects [19, 20, 21]. This dynamical behavior can be explained analytically using moment-matching arguments [19, 22]. While exploiting these effects can yield fast and accurate generative models, even for highly structured data [23] or high-quality images with RBMs [24], this approach results in a sharp separation between the model's Gibbs-Boltzmann distribution and the dataset distribution, thereby undermining the interpretability of the model parameters [22, 7]. Thus, to extract meaningful information from datasets using RBMs, it is essential to ensure proper mixing of the chains during training, in short, one needs equilibrium models .

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

Both the ability to train an RBM in equilibrium and to generate convincing new samples from its equilibrium measure strongly depend on the dataset in question. For typical image datasets such as MNIST or CIFAR-10, good RBMs can be obtained by increasing the number of MCMC steps. However, this approach is no longer feasible for highly structured datasets [25]. Datasets from which one seeks scientific insights are often highly structured, such as genomics/proteomics data or low-temperature many-body physical systems. These datasets typically exhibit distinct clusters, identifiable via principal component analysis (PCA) which form distant groups of similar entries. We show an example of the PCA of 4 clustered dataset we will be studying in this work in Fig. 1; details about these datasets are given in the caption and in the Supplemental Information (SI). During training, the model must evolve from an initial normal distribution to an increasingly multimodal distribution. Sampling from multimodal distributions is particularly challenging because the mixing times are determined by the transition times between modes. But this is not the only difficulty. These distant modes are encoded by second-order phase transitions during training [26, 27, 28], leading to diverging mixing times in these regions - a phenomenon known as critical slowdown -, which means that mixing times are expected to grow with a power of their system size. This sampling challenge not only hinders the training process, but also limits the model's ability to generate new samples. Obtaining new and independent configurations would require an impractically large number of sampling steps.

2 Related work

Training EBMs by maximizing log-likelihood has long been a challenge in the community [29, 10]. 76 EBMs gained popularity with the introduction of the contrastive divergence algorithm [30], in which 77 a set of parallel chains is initialized on independent examples in the minibatch and the MCMC 78 process iterates for a few steps. Despite its widespread use, this algorithm yields models with poor 79 equilibrium properties that are ineffective as generative models [31, 32, 21]. An improvement is the 80 persistent contrastive divergence (PCD) algorithm [33], which maintains a permanent chain in which 81 the last configurations used to estimate the previous gradient update are reused. PCD acts like a 82 slow annealing process improving gradient estimation quality. However, it often fails on clustered 83 data as the statistical properties of the permanent chain quickly move away from the equilibrium 84 measure and degrade the model [25]. This problem, which is primarily related to phase coexistence, 85 can be addressed with constrained MCMC methods if appropriate order parameters are identified. 86 For RBMs, these order parameters are related to the singular value decomposition of the model 87 coupling matrix, which enables efficient reconstruction of multimodal distributions [25]. Although 88 this method is effective for evaluating model quality, it is too computationally intensive to be used 89 in training, even if it leads to models with good equilibrium properties. Other optimized MCMC 90 methods, such as the Parallel Tempering (PT) [34] algorithm, simulate multiple models at different 91 temperatures, facilitating mixing through temperature exchange [35, 32]. However, PT is costly and 92

GLYPH<0>

0

:

5

0

:

0

PC1

GLYPH<0>

0

:

4

GLYPH<0>

0

:

2

0

:

0

0

:

2

PC2

0

:

2

0

:

4

PC1

0

:

2

0

:

3

0

:

4

0

:

5

PC2

GLYPH<0>

0

:

25

0

:

00

0

:

25

PC1

GLYPH<0>

0

:

2

0

:

0

PC2

C -

Human

Genome

B -

Dataset

Low-rank RBM generation

0

:

2

0

:

4

PC1

0

:

2

0

:

4

0

:

6

PC2

A -

MNIST01

E

F

D

European

South Asian

East Asian

American

African

0.0

2

P

-0.2

-0.25

0.00

0.25

PC1

D

D

0.10

0

1

0.05

PC2

0.00

0.0

0.1

PC1

often ineffective, especially because EBMs undergo first-order phase transitions at the temperature 93 where PT typically fails because one needs too many temperatures to make the moves accepted. We 94 will see below that a more appropriate approach exchanges the models at different training times, 95 which only implies crossing second-order phase transitions. 96

97

98

99

100

101

102

103

104

105

106

107

108

The population annealing algorithm, which reweights parallel chains during learning based on their relative weight changes during parameter updates, was proposed as an alternative [36]. Similarly, reweighting chains using non-equilibrium physics concepts such as the Jarzynski equality has been proposed [37]. Both approaches struggle with highly structured data sets. To prevent the different chains to get too correlated around the training phase transitions, one must either increase the number of sampling steps or decrease the learning rate, which in practice means very long training processes to ensure a proper equilibrium training. Another strategy is to use EBMs as corrections for straightforward-to-sample flow-based models [38]. This simplifies sampling and learning, but sacrifices the interpretability of the energy function, which was our goal. An evolving flow model can be used as a fast sampling moves proposer for the EBM [39] objective. This method requires the training of two different networks in parallel and may result in the drop of the move acceptancy as the EBM becomes specialized.

109

110

111

112

For RBMs, a recent method called 'stacked tempering" [40] dramatically speeds up sampling by training smaller RBMs with latent variables from previous models, allowing fast updates to be proposed using a PT like algorithm. Authors also showed that this algorithm was much faster than the standard PT. While effective, it is too cumbersome for use in training. Also for RBMs, it has

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

recently been shown that it is possible to train a low-rank RBM that accurately reproduces the statistics of the data projected along the d first data principal directions through a convex and very fast optimization process (see [41] and the discussion below). This low-rank model can be seen as a good approximation to the correct RBM needed to describe the data, and has the nice property that it can be efficiently sampled via a static Monte Carlo process.

In this paper, we will show how to drastically reduce training times by starting the RBM training process at this low-rank RBM, as this means that the first and strongest dynamic effects associated with them are directly bypassed. We also show that one can exploit the training trajectory to develop an effective sampling method, the parallel trajectory tempering (PTT) that outperforms the 'stacked tempering" [40] and only requires saving a reduced number of models during the training. This strategy also allows to obtain reliable estimations for the log-likelihood in well-trained models, much better than those obtained with the standard Annealing Important Sampling (AIS) techniques [42]. Using both strategies, we show that we are able to train and evaluate methods that accurately represent the different modes in the dataset, where standard methods lead to mode collapse effects.

3 The Restricted Boltzmann Machine

The RBM is composed by N v visible nodes and N h hidden nodes. In our study, we primarily use binary variables { 0 , 1 } or ± 1 for both layers. The two layers (visible and hidden) interact via a weight matrix w , with no direct couplings within a given layer. Variables are also adjusted by visible and hidden local biases, θ and η , respectively. The Gibbs-Boltzmann distribution for this model is expressed as

where Z is the partition function of the system. As with other models containing hidden variables, the training objective is to minimize the distance between the empirical distribution of the data, p D ( v ) , and the model's marginal distribution over the visible variables, p ( v ) = ∑ h exp[ -H ( v , h )] /Z = exp[ -H ( v )] /Z . Minimizing the Kullback-Leibler divergence is equivalent to maximizing the likelihood of observing the dataset in the model. Thus, the log-likelihood L = ⟨-H ( v ) ⟩ D -log Z can be maximized using the classical stochastic gradient ascent. For a training dataset D = { v ( m ) } m =1 ,...,M , the log-likelihood gradient is given by

where ⟨·⟩ D denotes the average with respect to the entries in the dataset, and ⟨·⟩ RBM with respect to p ( v , h ) . Since Z is intractable, the model averages in the gradient are typically estimated using N s independent MCMC processes, and observable averages ⟨ o ( v , h ) ⟩ RBM are replaced by ∑ R r =1 o ( v ( r ) , h ( r ) ) /R , with ( v ( r ) , h ( r ) ) being the last configurations reached with each of the r = 1 , . . . , R parallel chains. To obtain reliable estimates, it should be ensured that each of the Markov chains mix well before each parameter update. However, ensuring equilibrium at each update is impractical, slow and tedious. The common use of non-convergent MCMC processes is the cause of most difficulties and weird dynamical behaviors encountered in training RBMs [21].

Typical MCMC mixing times in RBMs are very small at the beginning of the training and grow as it progresses [21], suffering with sharp increases every-time the training trajectory crosses each of the critical transitions that give birth to new modes [28]. In order to minimize out-of-equilibrium effects, it is often useful to keep R permanent (or persistent) chains, which means that the last configurations reached with the MCMC process used to estimate the gradient at a given parameter update t , P t ≡ { ( v ( r ) t , h ( r ) t ) } R r =1 , are used to initialize the chains of the subsequent update t +1 . This algorithm is typically referred as PCD. In this scheme, the process of training can be mimicked to a slow cooling process, only that instead of varying a single parameter, the temperature, a whole set of parameters Θ t = ( w t , θ t , η t ) are updated at every step to Θ t +1 = Θ t + γ ∇ L t with ∇ L t being the gradient in Eq. (2) estimated using the configurations in P t , and γ being the learning rate.

158

159

160

161

162

163

4 The low-rank RBM pretrained

In Ref. [41], it was shown that it is possible to train exactly (i.e. by direct numerical integration instead of MCMC sampling) an RBM containing a reduced number of modes in the weight matrix W by exploiting a mapping between the RBM and a Restricted Coulomb Machine and solving a convex optimization problem, see the SI. In other words, it is possible to train a RBM with a coupling matrix of this simplified form

and where the right singular vectors { u α } d α =1 correspond exactly to the first d principal directions 164 of the data set. Under this assumption, it is possible to write p ( v ) only as a function of d order 165 parameters given by the magnetizations along each of the u α components, m α ( v ) = u α · v / √ N v , 166 and in particular, 167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

where m = ( m 1 , . . . , m α ) . As proposed in [41], the optimal parameters of such a model can basically be determined by solving a regression problem. We describe this method in details in the SI. This means that once the model is trained, we obtain a probability p ( m ) defined on a much lower dimension than the original p ( v ) . Such a probability can be straightforwardly sampled using inverse transform sampling . Since this method requires a discretization of the m -space both for training and generation, we cannot consider intrinsic space dimension d > 4 dimensions in practice. These low-rank RBMs are then trained to reproduce the statistics of the dataset projected in its first d principal components. Despite their simplicity, the low-rank models are already able to generate an approximate version of the dataset, as shown in Fig. 1-D-F for the 4 datasets previously presented.

In the initial stage of the standard learning process, the model encodes the strongest PCA components of the data through multiple critical transitions [26, 27, 28]. Pre-training with the low-rank construction allows us to bypass these transitions and avoid out-of-equilibrium effects caused by critical slowing down associated to these transitions. Once the main directions are incorporated, training can efficiently continue with standard algorithms like PCD, as the mixing times of pre-trained machines tend to be much shorter. In particular, in the PCD-100 training with MNIST01, relaxation times for the visible variables' time correlation reach 5 · 10 5 MCMCsteps at the first three transitions, coinciding with the growth of singular values in the model weight matrix W . In contrast, the pre-trained machine has a much shorter relaxation time of ∼ 10 3 , allowing us to safely restart the PCD process from a set of equilibrium samples generated by static sampling of the low-rank RBM.

Overcoming these transitions has dramatic implications for the quality of the models we can train and how accurately they reproduce the statistics of the data. In Fig. 2, we show for 3 datasets the equilibrium samples drawn from 3 RBMs trained with identical number of samples, minibatch size, k = 100 Gibbs steps, and learning rate γ = 0 . 01 , but different training strategies. In particular, we consider 2 RBMs trained from scratch with the standard PCD [33] and the recently proposed Jarzynski reweighing method [23] (see SI for our specific implementation in the RBM), and a final machine trained with PCD and pre-trained with a low-rank RBM. In all cases, the quality of the generated samples is significantly better when pre-training is used. For the Mickey dataset, neither JarRBM nor normal PCD are able to generate convincing data. For the MNIST01 dataset, all 3 methods are able to generate convincing data, but only Pretrain+PCD is able to correctly balance all modes, as can be seen in Fig. 3, where we compare the histograms of the generated data projected onto the first 3 PCA directions with those of the dataset and a random selection of the generated samples. We see that the pre-training+PCD training perfectly balances the different modes (here we show the first 3 directions, but it goes much further), unlike the other 2 methods, and also generates more diverse images. We can also compare the log-likelihood of all 3 models and find that the pre-trained RBM achieves higher values. At this point, it is important to emphasize that in order to properly quantify the increase in log-likelihood, we need to use the PTT algorithm (see section 5) to correctly thermalize in these well-trained machines. For comparison, we show our PTT measure in dark and solid lines, while the standard AIS [42] estimate is shown in light dashed lines.

Already from the scatter plots we see that the pre-training has a dramatic effect in obtaining models where all modes are properly balanced, but also has important effects in the maximum test-likelihood

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

Dataset

PCD

pre-train+PCD

MNIST-01

HGD

we can achieve. In all cases, these equilibrium samples are drawn using the trajectory PT algorithm that will be explained in the next section, and the log-likelihood obtained using the equilibrium configurations obtained at different epochs as a result of the trajectory PT flow.

5 Standard Gibbs sampling vs. Parallel Trajectory Tempering (PTT)

One major challenge with structured datasets is quantifying the model's quality, since sampling the equilibrium measure of a well-trained model is often too time-consuming. This affects the reliability of generated samples and indirect measures as log-likelihood's estimation through Annealing Importance Sampling (AIS) [42], making them inaccurate and meaningless.

To illustrate this problem, let us consider the MNIST01 and the HGD datasets. MNIST01 dataset is bimodal and the HGD highly multimodal as shown in their PCA in Figs. 1-A and C. Let us consider that we want to sample the equilibrium measure of the RBMs trained using low-rank RBM pretraining. In order to draw new samples from these models, one would typically run MCMC processes from random initialization and iterate them until convergence. The mixing time is controlled by the jumping time between clusters. To accurately estimate the relative weight between modes, the MCMC processes must be ergodic, requiring many back-and-forth jumps. However, as shown in Figs. 4-A and C for the MNIST01 and HGD datasets, Gibbs sampling dynamics are extremely slow, rarely producing jumps even after 10 4 MCMCsteps. The yellow curves in Figs. 4-B and D show the mean number of jumps over 100 independent chains as a function of MCMC steps, indicating that a proper equilibrium generation would require at least 10 6 -10 7 MCMCsteps.

One effective way to accelerate the dynamics is to exploit the training trajectory, where the model progressively specializes through second-order phase transitions. To achieve this, we save RBMs trained at various epochs and propose swaps between configurations of similarly trained models. We

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

pre-train+PCD

JarRBM

PCD

A

B

-

0

.

4

-

0

.

2

0

.

0

PC1

0

.

0

2

.

5

5

.

0

7

.

5

10

.

0

12

.

5

Dataset

PCD

pre-train+PCD

-

0

.

1

0

.

0

0

.

1

0

.

2

PC2

0

1

2

3

4

5

-

0

.

1

0

.

0

0

.

1

0

.

2

PC3

0

2

4

6

10

1

10

3

Training time (epochs)

-

200

-

150

-

100

-

50

LL (nats)

LL AIS on the MNIST-01 dataset

PCD

JarRBM

RCM+PCD

C

call this the Parallel Trajectory Tempering (PTT) algorithm. Unlike the standard Parallel Tempering (PT) algorithm, which attempts swaps configurations between different temperatures, the PTT swaps between model parameters with different degrees of specialization. This approach is more natural for this problem because it involves crossing only second-order transitions, unlike the first-order transitions occurring in temperature annealing. And in fact, we show in Figs. 4-A and C, that this approach allows us to sharply accelerate the dynamics, as opposed to the standard PT algorithm (studied in detail for the MNIST dataset in [40]).

In the PTT algorithm, the configurations x = ( v , h ) of neighboring machines indexed by t and t -1 are interchanged with the probability

This move satisfies detailed balance with our target equilibrium distribution p ( x ) = exp( -H ( x )) /Z , ensuring that the moves lead to the same equilibrium measure. As 'nonspecialized" models mix very quickly, either because the distribution is essentially Gaussian at the initialisation of a standard training, or because the low-rank RBM can be sampled with a static Monte Carlo process (yielding independent configurations each time), the trajectory flow significantly accelerates convergence to equilibrium. The time interval between successive machines is selected in such a way that the probability of accepting interchanges between neighboring machines remains around 0.3. Pre-trained machines require a significant fewer number of models to be effective, because most selected models are positioned at the most prominent phase transitions. We give the number of machines used for each sampling process in the SI. We also provide there a specific and detailed description of the algorithm used.

In the red curves in Fig. (4)-B and D, we show the number of jumps between clusters as a function of the number of elementary MCMC steps, which in the PTT scheme refer to 1 Gibbs sampling step + one swap proposal. For the DNA dataset, we have two measures corresponding to jumps along the two principal component directions. We observe at 10 4 MCMCsteps an increase of the number of jumps by a factor of 80 for MNIST01 and by a factor of 1350 for the HGD in this machine, although

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

C)

D)

A)

B)

we achieve higher factors in other machines, as we show in the SI. The sampling of RBMs training on the MNIST01 dataset was the subject of the study of the 'stacked tempering" algorithm in [40]. If we compare the numbers with their work, we see that we achieve a 3-4 times higher speedup factor, where our model has the advantage that it does not need additional training, but simply uses the stored machines correctly.

Another desirable advantage of our PTT algorithm is that we can easily use it to compute an improved estimate of the AIS log-likelihood, except that in our case we consider the training trajectory instead of a cooling process and use the equilibrium samples obtained for each of the models to compute the model averages. In Figs. 3-C 5-A we compare the log-likelihood estimates obtained with our method (AIS-PTT) in full and dark lines and in light and dashed lines the AIS estimate (AIS). We see that both measures coincide for most parts of the training and that they split when the sampling becomes too long to thermalize along the temperature annealing curve in AIS. This effect is particularly evident for the JarRBM run in 5-A, where AIS takes a long time to recognize that the model suffers from a strong mode-collapse effect.

6 Overfitting and privacy loss as quality indicators

In this section, we examine the quality of the samples generated, regarding overfitting and privacy 270 criteria which have been defined for genomic data in particular. We look at this on the models trained 271 with PCD with and without pre-training. We do not include the Jarzysnki method here, as this method 272 fails to obtain a reliable model as clearly shown in the evolution of the Log-likelihood in Fig. 5. We 273 focus on the human genome dataset, as shown in Fig. 1-C, to evaluate the ability of various state274 of-the-art generative models to generate realistic fake genomes while minimizing privacy concerns 275 (i.e., reducing overfitting). Recent studies [17, 18] have thoroughly investigated this for a variety of 276 generative models. Both studies concluded that the RBM was the most accurate method for generating 277 high-quality and private synthetic genomes. The comparison between models relies primarily on the 278 Nearest Neighbor Adversarial Accuracy ( AA TS ) and privacy loss indicators, introduced in Ref. [43], 279 which quantify the similarity and the level of "privacy" of the data generated by a model w.r.t. the 280 training set. We have AA TS = 1 2 ( AA True + AA Synth ) where AA True [resp. AA Synth ] are two 281

10

2

10

3

10

4

Training time (epochs)

-

500

-

475

-

450

-

425

-

400

LL (nats)

LL AIS on the HGD dataset

PCD

pre-train+PCD

A

PCD

pre-train+PCD

Model

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

Privacy Loss

PCD

pre-train+PCD

Model

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

Value

AAtruth

AAsyn

B

C

quantities in [0 , 1] obtained by merging two sets of real and synthetic data of equal size N s and 282 measuring respectively the frequency that a real [rep. synthetic] has a synthetic [resp. real] as 283 nearest neighbor. If the generated samples are statistically indistinguishable from real samples, both 284 frequencies AA True and AA Synth should converge to 0.5 at large N s . AA TS can be evaluated both 285 with train or test samples and the privacy loss indicator is defined as Privacy loss = AA test TS -AA train TS 286 and is expected to be strictly positive. Fig. 5 shows the comparison of AA TS and privacy loss values 287 obtained with our two models, demonstrating that the pre-trained RBM clearly outperforms the 288 other model, and even achieves better results ( AA TS values much closer to 0.5) than those discussed 289 in [17, 18]. 290

291

292

293

294

295

296

297

298

299

300

301

302

7 Conclusions

We have shown that the strategy of initiating the training on a pre-trained low-rank RBM is an extremely effective strategy to obtain high quality models for structured datasets that accurately represent all the modes in the datasets and with significantly higher log-likelihoods. We have also shown that the models obtained in that way are: (i) better generative models than those obtained with standard trainings, both, in the sense that they over-fit less at the same time they are more indistinguishable from the test samples, (ii) they display faster relaxational dynamics.

We have also proposed a new fast sampling method that exploits the progressive learning of features in the training of RBMs to design an efficient trajectory PT strategy that allows accelerating the parallel Gibbs sampling dynamics by many orders of magnitude and overcome the performance of recent efficient sampling methods without adding any extra cost than saving models during the training.

Both strategies for training and sampling are very general, and could be generalized to more complex 303 EBMs. In this sense, the low-rank RBM model could be used as a more efficient pre-initialisation 304 in deeper structures, and the trajectory PT algorithm is suitable to be directly used in any EBM no 305 matter how complex it is. 306

307

8 Code availability

The code and datasets are available at https://github.com/nbereux/fast-RBM . 308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

References

Paul Smolensky. In Parallel Distributed Processing: Volume 1 by D. Rumelhart and J. McLelland , chapter 6: Information Processing in Dynamical Systems: Foundations of Harmony Theory. 194-281. MIT Press, 1986.

David H Ackley, Geoffrey E Hinton, and Terrence J Sejnowski. A learning algorithm for Boltzmann machines. Cognitive science , 9(1):147-169, 1985.

Geoffrey E Hinton and Terrence J Sejnowski. Optimal perceptual inference. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition , volume 448, pages 448-453. Citeseer, 1983.

H Chau Nguyen, Riccardo Zecchina, and Johannes Berg. Inverse statistical problems: from the inverse ising problem to data science. Advances in Physics , 66(3):197-261, 2017.

John Hertz, Yasser Roudi, and Joanna Tyrcha. Ising models for inferring network structure from spike data. 2011.

Simona Cocco, Christoph Feinauer, Matteo Figliuzzi, Rémi Monasson, and Martin Weigt. Inverse statistical physics of protein sequences: a key issues review. Reports on Progress in Physics , 81(3):032601, 2018.

Aurélien Decelle, Cyril Furtlehner, Alfonso de Jesús Navas Gómez, and Beatriz Seoane. Inferring effective couplings with restricted boltzmann machines. SciPost Physics , 16(4):095, 2024.

Aurélien Decelle, Beatriz Seoane, and Lorenzo Rosset. Unsupervised hierarchical clustering using the learning dynamics of restricted boltzmann machines. Phys. Rev. E , 108:014110, Jul 2023.

Jianwen Xie, Yang Lu, Song-Chun Zhu, and Yingnian Wu. A theory of generative convnet. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 2635-2644, New York, New York, USA, 20-22 Jun 2016. PMLR.

Yang Song and Diederik P Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288 , 2021.

Giuseppe Carleo and Matthias Troyer. Solving the quantum many-body problem with artificial neural networks. Science , 355(6325):602-606, 2017.

Roger G Melko, Giuseppe Carleo, Juan Carrasquilla, and J Ignacio Cirac. Restricted boltzmann 339 machines in quantum physics. Nature Physics , 15(9):887-892, 2019. 340

Koji Hashimoto, Sotaro Sugishita, Akinori Tanaka, and Akio Tomiya. Deep learning and the 341 ads/cft correspondence. Physical Review D , 98(4):046019, 2018. 342

Koji Hashimoto. Ads/cft correspondence as a deep boltzmann machine. Physical Review D , 343 99(10):106017, 2019. 344

Jérôme Tubiana, Simona Cocco, and Rémi Monasson. Learning protein constitutive motifs 345 from sequence data. Elife , 8:e39397, 2019. 346

347

348

349

Juan Rodriguez-Rivas, Giancarlo Croce, Maureen Muscat, and Martin Weigt. Epistatic models predict mutable sites in sars-cov-2 proteins and epitopes. Proceedings of the National Academy of Sciences , 119(4):e2113118119, 2022.

350

351

352

353

354

355

Burak Yelmen, Aurélien Decelle, Linda Ongaro, Davide Marnetto, Corentin Tallec, Francesco Montinaro, Cyril Furtlehner, Luca Pagani, and Flora Jay. Creating artificial human genomes using generative neural networks. PLoS genetics , 17(2):e1009303, 2021.

Burak Yelmen, Aurélien Decelle, Leila Lea Boulos, Antoine Szatkownik, Cyril Furtlehner, Guillaume Charpiat, and Flora Jay. Deep convolutional and conditional neural networks for large-scale genomic data generation. PLOS Computational Biology , 19(10):e1011584, 2023.

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

Erik Nijkamp, Mitch Hill, Song-Chun Zhu, and Ying Nian Wu. Learning non-convergent non-persistent short-run mcmc toward energy-based model. Advances in Neural Information Processing Systems , 32, 2019.

Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of mcmc-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 5272-5280, 2020.

Aurélien Decelle, Cyril Furtlehner, and Beatriz Seoane. Equilibrium and non-equilibrium regimes in the learning of restricted boltzmann machines. Advances in Neural Information Processing Systems , 34:5345-5359, 2021.

Elisabeth Agoritsas, Giovanni Catania, Aurélien Decelle, and Beatriz Seoane. Explaining the effects of non-convergent sampling in the training of energy-based models. arXiv preprint arXiv:2301.09428 , 2023.

Alessandra Carbone, Aurélien Decelle, Lorenzo Rosset, and Beatriz Seoane. Fast and functional structured data generators rooted in out-of-equilibrium physics. arXiv preprint arXiv:2307.06797 , 2023.

Renjie Liao, Simon Kornblith, Mengye Ren, David J Fleet, and Geoffrey Hinton. Gaussianbernoulli rbms without tears. arXiv preprint arXiv:2210.10318 , 2022.

Nicolas Béreux, Aurélien Decelle, Cyril Furtlehner, and Beatriz Seoane. Learning a restricted boltzmann machine using biased monte carlo sampling. SciPost Physics , 14(3):032, 2023.

A. Decelle, G. Fissore, and C. Furtlehner. Spectral dynamics of learning in restricted boltzmann machines. Europhysics Letters , 119(6):60001, nov 2017.

Aurélien Decelle, Giancarlo Fissore, and Cyril Furtlehner. Thermodynamics of restricted boltzmann machines and related learning dynamics. Journal of Statistical Physics , 172:15761608, 2018.

A. Anonymous. Cascade of phase transitions in the training of energy-based models. arXiv: , 2024.

Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and Fujie Huang. A tutorial on energybased learning. Predicting structured data , 1(0), 2006.

Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation , 14(8):1771-1800, 2002.

Ruslan Salakhutdinov and Iain Murray. On the quantitative analysis of deep belief networks. In Proceedings of the 25th international conference on Machine learning , pages 872-879, 2008.

Guillaume Desjardins, Aaron Courville, Yoshua Bengio, Pascal Vincent, and Olivier Delalleau. Tempered markov chain monte carlo for training of restricted boltzmann machines. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 145-152. JMLR Workshop and Conference Proceedings, 2010.

Tijmen Tieleman. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th international conference on Machine learning , pages 1064-1071, 2008.

Enzo Marinari and Giorgio Parisi. Simulated tempering: a new monte carlo scheme. Europhysics letters , 19(6):451, 1992.

Russ R Salakhutdinov. Learning in markov random fields using tempered transitions. Advances in neural information processing systems , 22, 2009.

Oswin Krause, Asja Fischer, and Christian Igel. Population-contrastive-divergence: Does 399 consistency help with rbm training? Pattern Recognition Letters , 102:1-7, 2018. 400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

Davide Carbone, Mengjian Hua, Simon Coste, and Eric Vanden-Eijnden. Efficient training of energy-based models using jarzynski equality. Advances in Neural Information Processing Systems , 36, 2024.

E Nijkamp, R Gao, P Sountsov, S Vasudevan, B Pang, S-C Zhu, and YN Wu. Mcmc should mix: learning energy-based model with neural transport latent space mcmc. In International Conference on Learning Representations (ICLR 2022). , 2022.

Louis Grenioux, Éric Moulines, and Marylou Gabrié. Balanced training of energy-based models with adaptive flow sampling. In ICML 2023 Workshop on Structured Probabilistic Inference and Generative Modeling , 2023.

Clément Roussel, Jorge Fernandez-de Cossio-Diaz, Simona Cocco, and Remi Monasson. Accelerated sampling with stacked restricted boltzmann machines. In The Twelfth International Conference on Learning Representations , 2023.

Aurélien Decelle and Cyril Furtlehner. Exact training of restricted boltzmann machines on intrinsically low dimensional data. Physical Review Letters , 127(15):158303, 2021.

Oswin Krause, Asja Fischer, and Christian Igel. Algorithms for estimating the partition function of restricted boltzmann machines. Artificial Intelligence , 278:103195, 2020.

Andrew Yale, Saloni Dash, Ritik Dutta, Isabelle Guyon, Adrien Pavao, and Kristin P Bennett. Generation and evaluation of privacy preserving synthetic health data. Neurocomputing , 416:244-255, 2020.

A Details of the pre-training of a low rank RBM

A.1 The low-rank RBM and its sampling procedure

Our goal is to pre-train an RBM to directly encode the first d principal modes of the dataset in the model's coupling matrix. This approach avoids the standard procedure of progressively encoding these modes through a series of second-order phase transitions, which negatively impact the quality of gradient estimates during standard training. It also helps prevent critical relaxation slowdown of MCMCdynamics in the presence of many separated clusters.

Given a dataset, we want to find a good set of model parameters ( w , θ and η ) for which the statistics of the generated samples exactly match the statistics of the data projected onto the first d directions of the PCA decomposition of the training set. Let us call each of these α = 1 , . . . , d projections m α = u α · v / √ N v the magnetizations along the mode α , where u α is the α -th mode of the PCA decomposition of the dataset. A simple way to encode these d -modes is to parameterize the w -matrix as:

where u and ˆ u are respectively the right-hand and left-hand singular vectors of w , the former being 433 directly given by the PCA, while w α are the singular values of w . Using this decomposition, the 434 marginal energy on the visible variables, H ( v ) = log ∑ h exp H ( v , h ) can be rewritten in terms of 435 these magnetizations m ≡ ( m 1 , . . . , m d ) 436

Now, the goal of our pre-training is not to match the entire statistics of the data set, but only the 437 marginal probability of these magnetizations. In other words, we want to model the marginal 438 distribution 439

where δ is the Dirac δ -distristribution. In this formulation, the distribution of the model over the 440 magnetization m can be easily characterized 441

where N ( m ) = ∑ v ∏ d α =1 δ ( m α -1 √ N v u T α v ) is the number of configurations with magnetiza442 tions m , and thus S ( m ) = log N ( m ) /N v is the associated entropy. Now, for large N v the entropic 443 term can be determined using large deviation theory, and in particular the Gärtner-Ellis theorem: 444

with the rate function 445

and 446

Then, given a magnetization m , we can compute the minimizer µ ∗ ( m ) of ϕ ( µ ) -m T µ which is 447 convex, using e.g. Newton method which converge really fast since we are in small dimension. Note 448 that in practice we will obviously use finite estimates of ϕ , assuming N v is large enough. As a result 449 we get µ ∗ ( m ) satisfying implicit equations given by the constraints given at given N v : 450

It is then straightforward to check that spins distributed as 451

fulfill well the requirement, as 〈 u T α v / √ N v 〉 p prior = m α . In other words, we can generate samples 452 having mean magnetization m α just by choosing v i as 453

461

A.2 The Restricted Coulomb Machine

As introduced in [41], it is possible to exactly train a surrogate model for the RBM, called the 462 Restricted Coulomb Machine (RCM), on a low dimensional dataset without explicitly sampling the 463 machine allowing to learn even heavily clustered datasets. We will briefly outline the main steps to 464 train the RCM. A more detailed explanation can be found in Appendix A.2. 465

The RCM is an approximation of the marginal distribution of the RBM with {-1 , 1 } binary variables: 466

We then project both the parameters and variables of the RBM on the first d principal components of 467 the dataset: 468

with α ∈ { 1 , . . . , d } and v the projection matrix of the PCA. The projected distribution of the model 469 is then given by 470

where we ignore the fluctuations related to the transverse directions and S [ m ] accounts for the 471 non-uniform prior on m due to the projection of the uniform prior on s for the way to compute it. 472

The RCM is then built by approximating 473

which is valid for x large enough. The probability of the RCM is thus given by: 474

where 475

This can be easily inverted as 476

in order to obtain the RBM from the RCM. The model is then trained through log-likelihood 477 maximization over its parameters. However, this objective is non-convex if all the parameters are 478 trained through gradient ascent. To relax the problem, since we're in low dimension, we can define a 479 family of hyperplanes ( n , z ) covering the space and let the model only learn the weights of each to 480 the hyperplane. We can then discard the ones with a weight low enough for the approximation (21) to 481 be bad. 482

The gradients are given by 483

484

The positive term is straightforward to compute. For the negative term, we rely on a discretization of 485 the longitudinal space to estimate the probability density of the model and compute the averages. 486

...

...

...

...

...

...

...

Initialization

Sampling

1 mcmc step

Swap

RCM sampling

(only pre-train+PCD)

1 mcmc step

(only JarRBM and PCD)

B Sampling via Parallel Tempering using the learning trajectory 487

488

489

490

491

492

493

494

495

496

497

498

499

500

Assuming we have successfully trained a robust equilibrium model, there remains the challenge of efficiently generating equilibrium configurations from this model. Although models trained at equilibrium exhibit faster and more ergodic dynamics compared to poorly trained models, the sampling time can still be excessively long when navigating a highly rugged data landscape. Consequently, we devised a novel method for sampling equilibrium configurations that draws inspiration from the well-established parallel tempering approach. In this traditional method, multiple simulations are conducted in parallel at various temperatures, and configurations are exchanged among them using the Metropolis rule. Unlike this conventional technique, our method involves simultaneously simulating different models that are selected from various points along the training trajectory. This approach is motivated by the perspective that learning represents an annealing process for the model, encountering second-order type phase transitions during training. In contrast, annealing related to temperature changes involves first-order phase transitions, making traditional parallel tempering less effective for sampling from clustered multimodal distributions.

501

502

503

504

505

A sketch of the Parallel Trajectory Tempering (PTT) is represented in fig. 6. Specifically, we save t f models at checkpoints t = 1 , . . . , t f along the training trajectory. We denote the Hamiltonian of the model at checkpoint t as H t , and refer to the Hamiltonian of the RCM model as H 0 . We define GibbsSampling( H , x , k ) as the operation of performing k Gibbs sampling updates using the model H starting from the state x . In all our sampling simulations we used k = 1 .

The first step is to initialize the models' configurations efficiently. This involves sampling N 506 chains from the RCM model, x (0) 0 ∼ RCMSampling( H 0 ) , and then passing the chains through 507 all the models from t = 1 to t = t f , performing k Gibbs steps at each stage: x (0) t ∼ 508 GibbsSampling( H t , x (0) t -1 , k ) . 509 The sampling process proceeds in steps where we update the configuration of each model except H 0 510 with k Gibbs steps, and sample a completely new configuration for the RCM model H 0 . Following 511 this update step, we propose swapping chains between adjacent models with an acceptance probability 512 given by: 513 p acc ( x t ↔ x t -1 ) = min (1 , exp(∆ H t ( x t ) -∆ H t ( x t -1 ))) , (26) where ∆ H t ( x ) = H t ( x ) -H t -1 ( x ) . 514 We continue alternating between the update step and the swap step until a total of N mcmc steps is 515 reached. The sampling procedure is illustrated in the following pseudo-code: 516 Input: Set of models {H t } , t = 0 , . . . , t f , Number of Gibbs steps k , Number of MCMC steps 517 N mcmc 518 Output: Configurations x t for t = 1 , . . . , t f 519 Initialize: Sample N chains from the RCM model x (0) 0 ∼ RCMSampling( H 0 ) 520 for t = 1 to t f do 521 x (0) t ∼ GibbsSampling( H t , x (0) t -1 , ˜ k ) 522 end for 523 for n = 1 to ⌊ N mcmc /k ⌋ do 524 for t = 1 to t f do 525 x ( n ) t ∼ GibbsSampling( H t , x ( n -1) t , k ) 526 end for 527 Resample x ( n ) 0 ∼ RCMSampling( H 0 ) 528 for t = 1 to t f do 529 Compute acceptance probability 530 p acc ( x ( n ) t ↔ x ( n ) t -1 ) = min ( 1 , exp ( ∆ H t ( x ( n ) t ) -∆ H t ( x ( n ) t -1 ) )) Swap x ( n ) t and x ( n ) t -1 with probability p acc ( x ( n ) t ↔ x ( n ) t -1 ) 531 end for 532 end for 533

534

A comparison of performances between PTT and standard Gibbs sampling is reported in Tab. 1.

535

536

537

538

539

C Training details

We describe in Tables 2 and 3 the datasets and hyperparameters used during training. The test set was used to evaluate the metrics. All experiments were run on a RTX 4090 with an AMD Ryzen 9

5950X.

D Training of the RBM using the Jarzynski equation

In this section, we describe a procedure similar to the one introduced in [37] for training the RBM 540 by leveraging the Jarzynki equation. In one of its formulations, the Jarzynski equation states that 541 we can relate the ensemble average of an observable O with the average obtained through many 542

A)

B)

C)

D)

PCD

JarRBM

PCD

A)

B)

repetitions of an out-of-equilibrium dynamical process. If we consider the training trajectory of an 543 RBM, p 0 → p 1 →··· → p t -1 → p t , we can write 544

where the average on the lhs is done over the last model p t , the averages on the rhs are taken 545 across many different trajectory realizations and W t is a trajectory-dependent importance factor. By 546 all practical means, under the assumption of having quasi-adiabatic parameters updates, namely 547 p (Θ t -1 → Θ t ) = p (Θ t → Θ t -1 ) , this means that we can assign to each Markov chain of the 548 simulation x ( r ) , r = 1 , . . . , R, an importance weight given by: 549

and then compute the gradient of the log-likelihood by means of a weighted average over the chains: 550

Notice that, since Eq. (27) is an exact result, the importance weights should, in principle, eliminate 551 the bias brought by the non-convergent chains used for approximating the log-likelihood gradient in 552 the classical PCD scheme. However, after many updates of the importance weights, one finds that 553 only a few chains carry almost all the importance mass. In other words, the vast majority of the chains 554 we are simulating are statistically irrelevant, and we expect to get large fluctuations in the estimate of 555 the gradient because of the small effective number of chains contributing to the statistical average. A 556 good observable for monitoring this effect is the Effective Sample Size (ESS), defined as [37] 557

which measures the relative dispersion of the weights distribution. A way of circumventing the weight 558 concentration on a few chains, then, is to resample the chain population according to the importance 559 weights every time the ESS drops below a certain threshold, for instance 0.5. After this resampling, 560 all the chain weights have to be set to 1 ( W ( r ) = 0 ∀ r = 1 , . . . , R ). 561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [No]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

870

871

872

873

874

875

876

877

878

879

880

881

882

883

884

885

886

887

888

889

890

891

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTraining RBMs is challenging and slow due to the multiple second-order phase transitions and associated slow mixing of MCMC sampling. The paper introduces a pre-training method, consisting in integrating the principal directions of the dataset into a low-rank RBM through a convex optimization procedure. The Gibbs-Boltzmann equilibrium distribution of the pre-trained model can be efficiently sampled via a static Monte Carlo process. Starting from the pre-trained model, the standard Persistent Contrastive Divergence (PCD) training procedure for RBMs partially overcomes the problem of second-order phase transitions. The pre-training method is tested on the MNIST 01 dataset, a synthesized “Mickey” dataset, and the Human Genome dataset (HGD). The method is shown to outperform the PCD algorithm and the Jarzynski reweighting method (JarRBM).\nThe paper also introduces a new method to sample from the trained model, called Parallel Trajectory Tempering (PTT), and compares it with the Annealed Importance Sampling (AIS) method.\n\nSTRENGTHS:\nTechnically sound.\n\nThe proposed pre-training method is shown to improve on other training schemes for RBMs (namely the standard PCD method, and the Jarzynski reweighting method).\n\nThe novel PTT method is shown to improve on standard Gibbs sampling.\n\nWEAKNESSES:\nMy main concern is about the usefulness of the RBM approach to generative modeling. (cf question Q1 below).\n\nThe PTT sampling method seems to require more memory than standard methods for RBMs.\n\nIt is unclear how novel the proposed pre-training method is. (cf question Q2 below)\n\nThe paper contains a link to a github repository that reveals the author’s identity (section 8 page 9):\nhttps://github.com/nbereux/fast-RBM\n\nMinor. Line 465: Appendix A.2 references itself.\n\nQUESTIONS:\nQ1. I am missing the big picture. Could you clarify the primary motivation for studying RBMs? How do you envision RBMs improving on generative models? In what scenarios could RBMs improve on other SOTA generative methods (e.g. transformer-based generative models) ? And according to what criterion/metric can these RBMs improve over these SOTA generative methods?\n\n\nQ2. Can you clarify what was done in the Reference below, and what is the contribution of the present work compared to this earlier work?\n\nAurélien Decelle and Cyril Furtlehner. Exact training of restricted boltzmann machines on 414 intrinsically low dimensional data. Physical Review Letters, 127(15):158303, 2021.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes to pretrain RBMS with a recently developed convex approximation, the restricted coulomb machine and then fine-tune the model using standard techniques like PCD. Further, a novel sampling technique, PTT is proposed that can sample from the final trained model by employing a sequence of model snaphsots during training, that are then connected via replica exchange in the style of parallel tempering.\n\nExperiments are conducted and results show that the sampled distributions, when projected to the first few principal components, match the true distribution better. Moreover, log-likelihood comparisons based on single training runs show that the proposed method starts at much higher likelihood values due to the initialisation and there is some evidence that the resulting model als reaches higher likelihood values. Further experiments for PTT show that it is more likely to jump between clusters of the distribution than PCD based on gibbs sampling\n\nDisclaimer: This is an emergency review. I have not had the time to do a detailled analysis, or implement/reproduce any of the results. While I am expert in the field, I will adapt my confidence accordingly. I will not fullfy abide to the review format.\n\nEdit: An edit has been performed that only included changes of the format, but not the content.\n\nSTRENGTHS:\n- The use of the approximation using the restricted coulomb machine as an initialisation is an interesting idea that is worth investigating and the results in Fig 3C suggest that this pretraining approach is very effective.\n\n- The idea of using the training models as sampling steps is an interesting approach as well.\n\nIn general, i can see that both techniques can become tools in a more general RBM toolbox, however both of them seem to be incremental changes, even though they could impact RBM training a lot.\n\nWEAKNESSES:\nThe main weaknesses of the present work are in two areas: experiments/comparisons and language, of which I only deem the former critical, while the latter will limit the potential impact of the article in the machine-learning community. As a result of the weaknesses, this paper has a number of misleading claims or claims that are not supported by the data presented in the work.\n\nExperiments/Comparisons:\n\n- The authors dismiss using PT for being \"expensive\" and do not compare against it. This is against evidence that even single steps of PT chains with a moderate number of parallel chains can be more efficient than PCD using similar resources. This is especially interesting, since the authors use PCD-100 for training, which allows a lot of resources for PT.  Note that the authors themselves reference [40] which shows an order of magnitudes improvement over Gibbs sampling already with 10 chains.\nAnother example is given by \n[*] https://www.sciencedirect.com/science/article/pii/S0004370219301948\nwhere the authors used PT for training with k=50 chains and only a single sampling step per iteration. Note that this paper uses a very simple heuristic to improve PT sampling: choose the reference (temperature ->infty) distribution as not the uniform, but the marginal data distribution. This is in contrast to [40] that used the uniform distirbution, leading to a significantly worse baseline.\n\n- There are almost no comparisons of log-likelihood values or values of normalisation constants for the proposed technique. While some are given in Figure 3C, they are only single run and only using approximated likelihoods. Due to this, the phrase \" significantly higher log-likelihoods\" in the conclusion is NOT supported by the data, given there is not enough data to test for significance or even measure the variability.\n\n- The training is also cut short, or the RBMs trained are not powerful enough, since 3B shows clear artefacts in all samples, indicating that none of the machines approximate the dataset well. Since RBMS with enough latent variables are universal function approximators, we can not get a definitive statement of whether the proposed pretraining does allow for better likelihoods. Since the experiments are not very expensive, this reviewer would propose to at least repeat experiments in order to obtain error bars on Fig 3C. \n\n- The learning rate of 0.01 used in the experiments seems to be on the high end. This is not only bad for PCD training, but also for obtaining high likelihood values, and could explain big parts of the leveling out of the graph in Fig 3C. While it is okay to use a high learning rate in the beginning, keeping it constant over the course of training seems like an oversight.\n\n- For PTT especially, there are no good comparisons that compare the quality of the samples in terms of representing the true distributions. While visual examples are shown that show visually good mixing, the baseline to compare to is again PCD, and not PT, nor stacked tempering. Since, again [40] showed that both alternatives clearly beat gibbs sampling given the same amount of resources, we do not know how good this sampling scheme really is, compared to strong baselines. This reviewer suggests to compare in at least one experiment the proposed approach to an approach with known normalisation constant and then measure how well estimators based on these samples work, similar to [*]. This would also partially verify 3F. \n- As an addendum to the previous point: [*] showed that the performance of AIS improves significantly with the choice of reference distribution. Using the same distribution as [*] or the pretrained distribution proposed in this work, might diminish any performance gains by PTT. This would still be a major improvement to the state of the art.\n\n\n\nLanguage:\n- While in general well written, this work reads like targeting a physics community, not the ML community. As a result, this paper includes  slang terms mostly encountered in statistical physics/thermodynamics, which do not have a clear meaning in the statistics community. This includes terms like \"phase transitions\" (first and second order), \"equilibrium models\" (not consequential for the article, nor the review, but this reviewer genuinely does not know what this term is supposed to mean), \"critical transitions\", \"relaxation times\", \"free energy\". Since most of these terms appear in sections where the authors try to explain the method and/or its consequences, a significant number of readers will not be able to understand those reasonings as they do not have a grasp of the physical analogies. This reviewer would propose to replace some of the terms by the statistical equivalent, or to introduce them. \n- Some of the explainations and reasonings are misleading. The initial paragraph highlights that RBMs are supposedly interpretable. While they are simple models, Binary RBMs are still universal function approximators and thus it is highly unlikely that the latent space has any meaning that aligns with any human interpretable semantics. If the authors disagree with this, This reviewer encourages them to add a citation to line 25.\n- Missing citations: the datasets used should be cited.\n- Figure references missing: the article does not always refer to the figure they are talking about, e..g, line 205. In general references should include the subfigure letter as the article does later, e.g., lines 263+\n\nQUESTIONS:\nSuggestions are added under weaknesses.\n\nLIMITATIONS:\nThe authors mark \"Yes\" for point 2 \"Limitations\" on the checklist. This reviewer has not found that the authors discussed the limitations of their work. This especially includes the guidelines that say that authors should \"reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs.\" However, in fairness, the authors answer \"[NO]\" in question 7, discussing the presence of error bars or significance tests to measure significance. However, since this is not part of the final publication, and the authors include misleading claims about significance of results, this must be discussed in the main text, or the phrasing weakened.\n\nI have not found any information on runtime or CPU/GPU hours, but the CPU/GPU were reported, so point 8 is partially fulfilled.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript suggests to apply \n\nAurélien Decelle and Cyril Furtlehner. Exact training of restricted Boltzmann machines on intrinsically low dimensional data. Physical Review Letters, 127(15):158303, 2021.\n\nto initialise persistent contrastive divergence (PCD) learning for RBM training and estimating the log likelihood / partition function of RBMs.\n\nSTRENGTHS:\nWhile one may argue that the novelty is limited because the interesting theoretical work was done in the abovementioned paper by Aurélien Decelle and Cyril Furtlehner, the idea is sound and the approach may be useful in practice.\n\nWEAKNESSES:\nThe novelty is limited because the interesting theoretical work was done in the abovementioned paper by Aurélien Decelle and Cyril Furtlehner.\n\nMy **main criticism** refers to the empirical evaluation, and I think these questions should be addressed:\n- Was enough effort out into the baseline methods (including hyperparameter choice)?\n- What would be an example where the PCA decomposition is misleading (i.e., not helping or even slowing down the process)?\n- What about MNIST with all 10 digits?\n\n**Details** (not ordered by importance):\n\n* „On the diametrically opposite side (on interpretability)  are generative ConvNets [9, 10], where the energy function is formulated as a deep neural network, which are capable of synthesizing photorealistic images but are almost impossible to interpret as a physical model.“: Not clear, perhaps add half a sentence to elaborate.\n\n* „second-order phase transition“: define what this is already in the beginning\n\n* Beginning of section 2: I suggest to add the analysis in \n\n  Fischer, Igel. Bounding the Bias of Contrastive Divergence Learning. Neural Computation, 2011\n  https://direct.mit.edu/neco/article-abstract/23/3/664/7646/Bounding-the-Bias-of-Contrastive-Divergence?redirectedFrom=fulltext\n\n  to the discussion of the limitations of CD.\n\n* „much better than those obtained with the standard Annealing Important Sampling (AIS) techniques“:\nIn [42], several methods are discussed, in particular one based on Bennett’s Acceptance Ratio method (BAR), which performed in general better than standard AIS. How does the proposed method perform in comparison to BAR?\n\n* What if linear PCA is not well suited to find a good representation of the data because of a highly non-linear latent structure?\n\n* I am not fully happy with the selected benchmark tasks. \nIn particular: How does the method perform on MNIST with all 10 digits?\n\n**Minor** comments: \n\nThe reference list should be revised. Inconsistent capitalisation, author first name abbreviations, etc.\n\nQUESTIONS:\nMy main criticism refers to the empirical evaluation, and I think these questions should be addressed:\n- Was enough effort put into the baseline methods (including hyperparameter choice)?\n- What would be an example where the PCA decomposition is misleading (i.e., not helping or even slowing down the process)?\n- What about MNIST with all 10 digits?\nSee my comments below for further questions?\n\nLIMITATIONS:\nI think the limitations should have been explored in more depth.\nWhat would be an example where the PCA decomposition is misleading (i.e., not helping or even slowing down the process)?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-24"}