Title: Follow Hamiltonian Leader: An Efficient Energy-Guided Sampling Method
Venue: NeurIPS 2024
Accepted: no
Keywords: Sampling

Hamiltonian Monte Carlo

Monte Carlo Markov Chain
Abstract: Our research underscores the value of leveraging zeroth-order information for addressing sampling challenges, particularly when first-order data is unreliable or unavailable. In light of this, we have developed a novel parallel sampling method that incorporates a leader-guiding mechanism. This mechanism forges connections between multiple sampling instances via a selected leader, enhancing both the efficiency and effectiveness of the entire sampling process. Our experimental results demonstrate that our method markedly expedites the exploration of the target distribution and produces superior quality outcomes compared to traditional sampling techniques. Furthermore, our method also shows greater resilience against the detrimental impacts of corrupted gradients as intended.

Abstract

Our research underscores the value of leveraging zeroth-order information for addressing sampling challenges, particularly when first-order data is unreliable or unavailable. In light of this, we have developed a novel parallel sampling method that incorporates a leader-guiding mechanism. This mechanism forges connections between multiple sampling instances via a selected leader, enhancing both the efficiency and effectiveness of the entire sampling process. Our experimental results demonstrate that our method markedly expedites the exploration of the target distribution and produces superior quality outcomes compared to traditional sampling techniques. Furthermore, our method also shows greater resilience against the detrimental impacts of corrupted gradients as intended.

1 Introduction

Score-based generative models [35, 26, 36, 20] introduce a novel approach to generative modeling that revolves around the estimation and sampling of the Stein score [26, 36]. The score represents the gradient of the log-density function ∇ x log π ( x ) evaluated at the input data point x . This type of approach usually relies on effectively training a deep neural network to accurately estimate the score. The estimated score is then utilized to navigate the sampling process, ultimately resulting in the production of high-quality data samples that closely match the areas of high density in the original distribution.

In our research, we investigate the sampling of a probability distribution given by π ( x ) ∝ e -U ( x ) , where U ( x ) is the energy function. In the context of energy-based score-matching generative models, the objective often involves sampling the modes in areas of high probability density. An approach as suggested in [36, 20], is to smooth the original distribution by convolving π ( x ) with an isotropic Gaussian distribution of variance σ 2 , yielding π σ ( x ) = ∫ π ( x ′ ) N ( x ; x ′ , σ 2 I ) dx ′ . By gradually decreasing the variance σ , π σ ( x ) recovers the original distribution π ( x ) .

Typically, the sampling of score-based approaches are integrated with numerical SDE solvers [38], for example, the Euler-Maruyama solver, as well as Monte Carlo Markov Chain (MCMC) techniques like Langevin Dynamics [30]. Furthermore, there is a notable similarity between score-based sampling algorithms and first-order optimization algorithms. Efforts have been made to merge these two methodologies, particularly from a perspective of sampling [42, 10, 28, 9, 44]. All these methods primarily concentrates on first-order information ∇ x U ( x ) to improve performance, while typically treating the zeroth-order information U ( x ) merely as a basis for rejecting samples [18, 32, 29].

We argue that incorporating zeroth-order information can significantly enhance the algorithm's overall effectiveness, particularly in instances where the first-order information is compromised. To address this, we draw inspiration from parallel tempering [39], a simulation method commonly used to identify the lowest energy state in systems of interacting particles. The fundamental principle of

parallel tempering involves operating multiple sampling replicas simultaneously, each at a different 36 temperature level. These temperatures typically range from low, where the system is prone to being 37 trapped in local minima, to high, which facilitates the system's ability to surmount energy barriers 38 and more thoroughly explore the energy landscape. 39

Drawing inspiration from this concept, we extend the Hamiltonian Monte Carlo (HMC) framework 40 [29] and introduce a novel algorithm that concurrently runs multiple replicas, sampling at both high 41 and low Hamiltonian energy levels. Moreover, this methodology combines both zeroth and first 42 order information from various chains, hence enhancing the effectiveness of sampling approaches. 43 The experimental findings demonstrate the efficacy of our approach in scenarios where relying 44 solely on first-order knowledge is insufficient. These findings illustrate the capacity of incorporating 45 zeroth-order information to greatly enhance the efficiency and accuracy of sampling operations in 46 energy-based score-matching algorithms. 47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

2 Background

2.1 Hamiltonian Monte Carlo

The primary purpose of MCMC is to construct a Markov chain that matches its equilibrium distribution to the target distribution. One of the most popular MCMC methods is Langevin Monte Carlo [17, 32], which proposes samples in a Metropolis-Hastings [18] framework for more efficient state space exploration. Another advanced method is HMC [29, 11, 2], which incorporates an auxiliary variable p and employs Hamiltonian dynamics to facilitate the sampling process. The Hamiltonian function is structured as a composite of potential energy U ( x ) and kinetic energy K ( p ) , defined as follows:

where x represents the position of a particle and p denotes its momentum. Kinetic energy K ( p ) is commonly formulated as K ( p ) = 1 2 p T M -1 p , where M corresponds to the mass matrix. For simplicity, we assume in this paper that the mass matrix M is equal to the identity matrix I . The joint distribution of position and momentum conforms to the canonical distribution:

where Z = ∫∫ e -H ( x,p ) dxdp . Samples from π ( x ) can then be obtained by marginalizing p from π ( x, p ) , which further requires ∫ p π ( x, p ) dp = constant. In the HMC algorithm, proposals are generated by simulating Hamiltonian dynamics and then subjected to a Metropolis criterion to determine their acceptance or rejection. A commonly employed numerical method for solving these equations is the Leapfrog integrator [3].

Recent progress in HMC techniques has focused on increasing their adaptability and applicability in a variety of contexts. Such developments include the NUTS sampler [21], which features an automatic mechanism for adjusting the number of simulation steps. The Riemann manifold HMC [15] leverages Riemannian geometry to modify the mass matrix M , making use of curvature information to improve sampling efficiency. Additionally, Stochastic Gradient Hamiltonian Monte Carlo [11, 27] investigates a stochastic gradient approach within the HMC framework. Our contribution is distinct from these methods and can be easily integrated with them.

2.2 Energy-based score-matching model

Probabilistic models often require normalization, which can become infeasible when dealing with high-dimensional data [25, 13]. Since the exact probabilities of less probable alternatives become less crucial as long as they remain relatively lower, rather than solely predicting the most probable outcome, models can be structured to interpret relationships between variables via an energy function. Within the context of generative models, these energy-based models (EBMs) are devised to assign higher energy values to regions of lower probability and lower energy values to regions of higher probability.

Score matching [22, 36] is a method used in statistical modeling and machine learning to estimate a 80 probability distribution or a probability density function from observed data. It is particularly useful 81 when direct estimation of the probability distribution is challenging, especially in high-dimensional 82 spaces. In score matching, the goal is to find an approximation to the probability density function 83 (PDF) of a dataset by estimating the score function, also known as the gradient of the log-density. 84

The score function represents the derivative of the log PDF with respect to the data. By matching the 85 estimated score function to the observed data, one can indirectly estimate the underlying probability 86 distribution. 87

A relationship between EBMs and score matching can be established by training EBMs through 88 denoising score matching [37]. The training objective is described below: 89

U θ is typically represented as a neural network, with θ denoting its parameters. Minimizing this objective ensures that ∇ x U θ ( x ) = -∇ x log π σ ( x ) and thus e -U θ ( x ) shall be proportional to π σ ( x ) .

3 Motivation

In our work, we assume to have access to both the gradient information ∇ x U ( x ) as well as the energy information U ( x ) . In certain scenarios, gradients may yield information that is either of limited or potentially detrimental. Our research examines situations where gradients are compromised, highlighting the importance of zeroth-order information, often associated with energy-based sampling.

We concentrate on showcasing the strengths of our method in three types of challenging but common scenarios, summarized as instability , metastability and pseudo-stability .

Y

xt

[Vf（x）!Vxf(x1)

-ni

nt

△

(x）)

X

Instability refers to a state in which a system lacks equilibrium or steadiness, often leading to unpredictable or erratic behavior. Metastability describes a condition where a system appears stable over a short period but is not in its most stable state, and it can transition to a more stable state under certain conditions. Pseudo-stability, on the other hand, denotes a situation where a system seems stable but is actually in an incorrect, suboptimal, or misleadingly stable state.

U(X)

High Energy

Low Energy

anchor point can enhance the stability of convergence, as demonstrated in Figure 1.

Metastability. Particles are prone to getting stuck in local minima when the gradients are not informative. For example, on the saddle point or a pleaute loss landscape. As a result, simulations frequently end up in a state of intermediate energy, which is different from the system's lowest energy state. This scenario is illustrated in Figure 2. Pseudo-stableminima

Pseudo-Stability. Certain situations may present a divergence between the gradient information and the ground truth. This divergence can hinder algorithms from accurately converging to the appropriate modes. In these instances, it becomes essential to incorporate energy information to rectify inaccuracies that arise from solely depending on gradients. An example of misleading gradients could be observed in Figure 3.

4 Algorithm

Many sampling methods typically rely on independent Markov chains, which can lead to the issues mentioned in Section 3. Taking inspiration from [39], our approach involves the utilization of multiple replicas. This approach

Pseudo-stableminima

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

enables us to implicitly encourage greater exploration among multiple particles while simultaneously preserving the optimal outcomes for exploitation purposes. We will elaborate on how our algorithm can be employed to tackle these challenges.

Firstly, we introduce a modified version of the leapfrog method, called the Elastic Leapfrog (eLeapfrog). In this approach, additional elastic forces are applied between each particle and a leader , incorporating an extra elastic energy term into the traditional Hamiltonian function. This modification aims to prevent particles from straying significantly from each other, thereby promoting local exploitation. We then divide the particles into groups and designate the particle with the lowest energy as the leader. Moreover, when combined with the eLeapfrog method, this approach encourages other particles to explore around the leader, efficiently addressing the problem of instability .

Due to the properties of HMC, introducing such an extra elastic energy term when pulling the particles towards the leader implicitly incorporates this energy into the momentum, thereby increasing the search ability of each particle. As a result, non-leading particles gain more energy for exploration, while the leading particle is more likely to concentrate on local exploitation. This approach helps mitigate the issue of metastability .

Finally, we integrate these techniques to present our compplete Follow Hamiltonian Leader (FHL) algorithm. The FHL algorithm capitalizes on both first-order and zeroth-order information while significantly improving the efficiency of space sampling compared to traditional sequential sampling methods. This enhanced approach fosters convergence towards the lowest energy states and increases the likelihood of escaping states with pseudo stability .

4.1 Elastic Leapfrog

To improve the efficiency of sampling, we integrate an elastic force component into the conventional leapfrog technique. This enhancement aims to dynamically guide particles towards a leading particle, facilitating their movement and improve their exploration ability. The method could be treated like temporarily storing potential energy within an elastic spring, which is then converted into kinetic energy. By adding extra elastic force, we could define the energy of elastic HMC as:

where E ( x ; ˜ x ) is the extra elastic energy imposed by Elastic Leapfrog and is defined as E ( x ; ˜ x ) = ρ 2 ∥ x -˜ x ∥ 2 2 . Our approach enables particles to efficiently navigate the sample space, guided by the leader. This local exploration strategy, though similar to concepts in [46, 7, 8, 40], is uniquely tailored for application in the realm of sampling.

4.2 Leader Pulling

Next, we introduce our leader pulling method. Initially, we represent the i th particle inside a group 167 as x i and select a leader based on a their energies U ( x i ) . The motivation is that we encourage each 168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

particle x i to be guided towards a chosen leader. The leader is chosen as the one of minimum energy and thus its index is l = arg min i U ( x i ) . The objective function for a group of n particles is:

where π i = exp ( -U ( x i ) ) / ∑ j exp ( -U ( x j ) ) and ρ i = λ · ( π l -π i ) / ( π l + π i ) . The specifics of the Elastic Leapfrog algorithm combined with leader pulling technique are detailed in Algorithm 1.

4.3 Follow Hamiltonian Leader

Incorporating zeroth-order information (i.e., function values rather than derivatives) serves two key purposes. Firstly, it provides a search direction that accelerates convergence and helps mitigate issues arising from corrupted first-order information (i.e., gradient inaccuracies), thereby speeding up the optimization process. Second, it helps ensure that we are sampling from the correct underlying distribution by properly accepting or rejecting the proposal.

To ensure that the sampling method maintains detailed balance-a requirement for most sampling algorithms-we evaluate the joint distribution of a group of particles. This evaluation determines whether to accept or reject a proposed move for the whole group, thereby preserving the integrity of the sampling process. This adaptation results in the creation of our algorithm FHL, extensively elucidated in Algorithm 2.

Algorithm 2 Follow Hamiltonian Leader

5 Experiment

In this section, we showcase the efficacy of incorporating zeroth-order information, specifically energy information, into our proposed method to improve the sampling process. We focus on demonstrating the advantages of our approach in addressing the benefits of our approach in handling three distinct adversarial gradient scenarios, as outlined in Section 3. To evaluate our method on the performance of the concerned questions, we conduct a comparative analysis against the following baseline algorithms:

LMC (Langevin Monte Carlo) : An MCMC method as described in [17] that uses Langevin dynamics to sample from probability distributions. It is also known as the Metropolisadjusted Langevin algorithm.

HMC (Hamiltonian Monte Carlo) : An MCMC algorithm that employs Hamiltonian dynamics for more efficient traversal of the state space, leading to better exploration and sampling from complex distributions [29, 11, 2].

U-LMC (Unadjusted Langevin Dynamics) : A variation of LMC without the Metropolis correction, referred to [32, 1, 42].

200

201

202

203

204

205

206

207

208

209

210

211

212

U-HMC (Unadjusted Hamiltonian Monte Carlo) : A form of HMC that excludes the Metropolis correction step, as in [34, 14].

5.1 Motivating Examples

We report on results addressing the challenges identified as instability , metastability , and pseudostability . Our findings lead us to conclude that the FTH method consistently outperforms other approaches in all scenarios examined. Detailed discussions and further analyses of these findings will be presented in the following subsections.

In our experiment, we simultaneously execute sampling with N particles, each completing a total of T sampling steps. For the FTH method, these particles are divided into N/n groups, with each group containing n particles. Throughout all experiments, we set n to 4 . For hyperparameter search, we select step sizes η = { 0 . 002 , 0 . 0002 , 0 . 005 , 0 . 0005 } for all methods and number of leapfrog steps L = { 4 , 8 , 16 } for HMC-type methods.

5.1.1 Instability

(a) Sampling from the

original distribution

in the form of

e

-

U

(

x

)

.

T

= 1000

in all experiments.

Benchmark

FHL

U-LMC

U-HMC

LMC

HMC

Benchmark

FHL

U-LMC

U-HMC

LMC

HMC

Figure 4: Sample from a Gaussian distribution N ( µ, Σ) where µ ∈ R d corresponds to the clean image. For each method, we plot the lowest-energy particle (in terms of U ( x ) among all particles in X T ). The upper-left image represents a direct sample from the distribution N ( µ, Σ) ; The lower-left image is generated by performing HMC sampling for T steps on the function U θ ( x ) , with an initial point set to x 0 = µ .

In our sampling process, we focus on efficiently directing particles to high probability density regions, 213 thereby avoiding unproductive exploration in regions with low probability. When sampling from a 214 single image, our goal becomes attaining the global optima, aligning this objective with those found 215 in optimization tasks. 216

217

218

219

220

221

For our experiment, we chose an image resembling the GitHub logo ( https://github.com/logos ), converted it into a vector format, and use this as the mean of a multivariate Gaussian distribution. The covariance matrix for this distribution, represented by Σ , is diagonal. The variance for each dimension of the distribution is randomly determined by a uniform distribution within the range of (0 . 25 , 1 . 25) . We carry out two similar but different types of experiments:

222

223

224

In the first experiment, we focus on sampling from the original distribution. This distribution is described mathematically as e -U ( x ) ∝ N ( µ, Σ) , with U ( x ) being the energy function that characterizes the system.

For the energy-based score-matching model, we employ a ResNet [19] architecture with 6 layers 225 of a hidden dimension of 256 . The results of the sampling process are detailed in Figure 4, where 226 the main objective is to assess the particles' capacity for effective convergence to the mode of the 227 distribution. In the first scenario, U ( x ) represents a convex function, whereas in the second scenario, 228 U θ ( x ) is presumed to be non-convex. The findings demonstrate that our approach, FHL, surpasses 229 other baseline methods in both situations. 230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

5.1.2 Metastability

Our research explores the concept of metastability, which arises in specific scenarios. Metastability refers to a state of intermediate energy in a dynamic system, differing from its lowest energy state. We examine an extreme scenario where gradients are entirely absent, and sampling methods only get access to the energy information about the distribution.

In Figure 5, it's evident that in this particular situation, we enforce the gradient to be near zero , resulting in all sampling methods, except FHL and LMC, behaving almost like random sampling. Nevertheless, owing to the leader pulling strategy, FHL retains its ability to locate the mode much faster.

5.1.3 Pseudo-stability

U-LMC

U-HMC

FHL

LMC

HMC

This section highlights the phenomenon showcased in Figure 3. Here, particles can become ensnared by gradient flows and be coerced into pseudo-stable regions. Despite the eventual recovery of the correct distribution by the sampling method, the convergence process can be exceptionally sluggish.

To elaborate, we examine a scenario where the samplers solely depend on gradients from ∇ log Q , while the energy function P remains deliberately undisclosed. The distributions P and Q are:

From Figure 6 we can see that FTH does not only capture the modes more quickly compared to the other methods but also successfully get out of the trap of the pseudo-stable regions.

Our study addresses the challenges of instability, metastability, and pseudo-stability, demonstrating that the FTH method consistently outperforms other approaches across various scenarios. Through illustrative experiments, we show that FTH rapidly captures modes and effectively escapes pseudo-stable regions, even when gradients are entirely absent. This superior performance is attributed to FTH's unique leader pulling strategy, which directs particles efficiently to high-probability density regions, thereby avoiding unproductive exploration in low-probability areas.

EnergyState

U-LMC

U-HMC

FHL

LMC

HMC

In the following section, we will illustrate the advantages of FTH in more general applications, particularly for energy-based (score-matching) models.

282

283

284

285

286

287

288

289

290

291

292

293

5.2 Energy-Based Generative Model

Energy-based models (EBMs) offer significant advantages for sampling because they naturally provide energy information that can be utilized to guide the sampling process. In an EBM, the energy function assigns lower energy values to more probable configurations, enabling the sampler to more effectively navigate the probability landscape and generate high-quality samples. This makes EBMs a powerful tool in scenarios where precise sampling is essential.

We investigate a scenario where energy functions guide the sampling process. We use the generative model outlined in [16] and adopt a conditional generation method that leverages classifier-derived gradients for sampling. The classifier's output is considered as the energy for guided sampling. The common classification tasks involving C classes are often solved by using a neural network f θ : R d → R C , which maps each input data point x ∈ R d to C -categorical outputs. The output are then used to define a categorical distribution of class y through a softmax function:

where f θ ( x )[ y ] represents the y -th component of f θ ( x ) , corresponding to the logit for class y . Once 294 the classifier is trained, p ′ θ ( y | x ) = exp( f θ ( x )[ y ]) could be used to sample for a specific class y . 295

We compare FTH with the standard HMC method using a limited number of sampling steps, con296 sistently accepting new proposals based on the potential energy during sampling. It is evident that 297 FTH produces higher-quality images than HMC. Additionally, our experiments reveal that FTH tends 298 to generate sharper images compared to the other method. This can be attributed to the assumption 299 that the classifier focuses on the object's features rather than the entire image. As a result, when the 300 prediction probability is high, the features that increase confidence become more prominent, while 301 unrelated background elements are filtered out. 302

303

304

305

306

307

308

5.3 Energy-Based Score-Matching Models

As indicated in [12], when two diffusion models are combined into a product model q prod ( x ) ∝ q 1 ( x ) q 2 ( x ) , problems can arise if the model reversing the diffusion uses a score estimate derived by simply adding the score estimates of the two independent models. We use energy-based scorematching models to illustrate this issue. It is important to note that such inconsistencies typically involve the composition of two or more diffusion models.

5.3.1 Synthetic Dataset 309

We first show an example of composing two distributions p 1 ( x ) and p 2 ( x ) , as illustrated in the left 310 column of Figure 8. The results show that FTH demonstrates a strong ability to converge to the 311 correct composition, with less particles fall out of the high-density region compared to others. 312

P(x)

GroundTruth

U-LMC

LMC

P2（x）

FTH

U-HMC

HMC

5.3.2 CLEVR Dataset

We use CLEVR dataset from [23] for our generation and sampling tasks. The energy model is adopted from [12], and we employ different samplers for generation. The dataset includes three classes: cube , sphere , and cylinder . We explore scenarios where we first sample from only one category and then from two categories.

(a) Baseline

(b) MALA

(c) HMC

(d) FTH

G

In the first experiment, there is no composition of models. As depicted in Figure 9, it is evident that FTH effectively generates the desired image without any extraneous shapes, whereas both MALA and HMC generate additional shapes.

In the second experiment, we combine two independent diffusion models, each trained separately to generate sphere and cylinder . As shown in Figure 10, it is clear that FTH excels at producing high-quality images with almost no over-

(a) Baseline

(b) MALA

(c) HMC

(d) FTH

177

lapping between objects, accurately rendering the intended shapes in a pristine manner. In contrast, the other methods generate the undesired shape cube . Additionally, FTH exhibits less noise, indicating greater stability for sampling.

6 Conclusion

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

In this study, we first recognize the significance of incorporating zeroth-order information into the 335 sampling process, highlighting the common limitations faced by conventional sampling methods. 336 These limitations include unstable sampling outcomes frequently associated with energy-based 337 score-matching models, the potential metastability arising from the multi-modal nature of the energy 338 function, and errors in gradient computation stemming from the complex structure of the composi339 tional distribution. Subsequently, we present an innovative approach that leverages parallel HMC 340 sampling to address the issues. Building upon HMC, we incorporate energy modulation techniques 341 to enhance the sampling process. Through this approach, our method is able to systematically reduce 342 the potential energy, leading to substantial advantages in practical implementations of sampling. 343

344

345

346

347

348

349

350

351

352

353

354

References

Christophe Andrieu, Éric Moulines, and Francis J Samson. Particle markov chain monte carlo for efficient numerical simulation. Statistical Science , 25(4):332-350, 2010.

Michael Betancourt. A Conceptual Introduction to Hamiltonian Monte Carlo, July 2018.

Charles K. Birdsall and A. Bruce Langdon. Plasma physics via computer simulation . Taylor and Francis, New York, 2005.

Jérôme Bolte and Edouard Pauwels. A mathematical model for automatic differentiation in machine learning. Advances in Neural Information Processing Systems , 33:10809-10819, 2020.

Steve P. Brooks, Andrew Gelman, Galin L. Jones, and Xiao-Li Meng. Handbook of markov chain monte carlo: Hardcover: 619 pages publisher: Chapman and hall/crc press (first edition, may 2011) language: English isbn-10: 1420079417. CHANCE , 25:53 - 55, 2012.

Augustin Louis Cauchy. Méthode générale pour la résolution des systémes d'équations simul355 tanées. Comptes Rendus de l'Académie des Sciences Paris , 25:536-538, 1847. 356

357

358

359

P. Chaudhari, A. Choromanska, S. Soatto, Y. LeCun, C. Baldassi, C. Borgs, J. T. Chayes, L. Sagun, and R. Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. In ICLR , 2017.

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

Pratik Chaudhari, Carlo Baldassi, Riccardo Zecchina, Stefano Soatto, Ameet Talwalkar, and Adam Oberman. Parle: parallelizing stochastic gradient descent. arXiv preprint arXiv:1707.00424 , 2017.

Changyou Chen, David Carlson, Zhe Gan, Chunyuan Li, and Lawrence Carin. Bridging the gap between stochastic gradient mcmc and stochastic optimization. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics , volume 51 of Proceedings of Machine Learning Research , pages 1051-1060, Cadiz, Spain, 09-11 May 2016. PMLR.

Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic Gradient Hamiltonian Monte Carlo. In Proceedings of the 31st International Conference on Machine Learning , pages 1683-1691. PMLR, June 2014.

Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 1683-1691, Bejing, China, 22-24 Jun 2014. PMLR.

Yilun Du, Conor Durkan, Robin Strudel, Joshua B. Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc. JMLR.org, 2023.

Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Tomas Geffner and Justin Domke. MCMC Variational Inference via Uncorrected Hamiltonian Annealing. In Advances in Neural Information Processing Systems , volume 34, pages 639-651. Curran Associates, Inc., 2021.

Mark Girolami, Ben Calderhead, and Siu A Chin. Riemann Manifold Langevin and Hamiltonian Monte Carlo.

Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference on Learning Representations , 2020.

Ulf Grenander and Michael I. Miller. Representations of Knowledge in Complex Systems. 389 Journal of the Royal Statistical Society. Series B (Methodological) , 56(4):549-603, 1994. 390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

W. K. Hastings. Monte carlo sampling methods using markov chains and their applications. Biometrika , 57(1):97-109, 1970.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020.

Matthew D. Hoffman and Andrew Gelman. The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo, November 2011.

Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(24):695-709, 2005.

Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1988-1997, 2016.

S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing. Science (New York, N.Y.) , 220(4598):671-680, 1983.

Yann LeCun, Sumit Chopra, Raia Hadsell, Marc Aurelio Ranzato, and Fu Jie Huang. A tutorial on energy-based learning. 2006.

Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-fit tests. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 276-284, New York, New York, USA, 20-22 Jun 2016. PMLR.

Yi-An Ma, Tianqi Chen, and Emily Fox. A Complete Recipe for Stochastic Gradient MCMC. In Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015.

Yi-An Ma, Yuansi Chen, Chi Jin, Nicolas Flammarion, and Michael I. Jordan. Sampling can be faster than optimization. Proceedings of the National Academy of Sciences , 116(42):2088120885, 2019.

Radford M. Neal. MCMC Using Hamiltonian Dynamics . May 2011.

G. Parisi. Correlation functions and computer simulations. Nuclear Physics B , 180(3):378-384, 1981.

B.T. Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964.

Gareth O. Roberts and Richard L. Tweedie. Exponential convergence of Langevin distributions and their discrete approximations. Bernoulli , 2(4):341 - 363, 1996.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.

Jascha Sohl-Dickstein, Mayur Mudigonda, and Michael DeWeese. Hamiltonian monte carlo without detailed balance. In Eric P. Xing and Tony Jebara, editors, Proceedings of the 31st International Conference on Machine Learning , volume 32 of Proceedings of Machine Learning Research , pages 719-726, Bejing, China, 22-24 Jun 2014. PMLR.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data 440 distribution. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and 441 R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran 442 Associates, Inc., 2019. 443

Yang Song and Diederik P. Kingma. How to Train Your Energy-Based Models, February 2021. 444

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and 445 Ben Poole. Score-based generative modeling through stochastic differential equations. In 446 International Conference on Learning Representations , 2021. 447

Robert Swendsen and Jian-Sheng Wang. Replica monte carlo simulation of spin-glasses. 448 Physical review letters , 57:2607-2609, 12 1986. 449

Y. Teng, W. Gao, F. Chalus, A. Choromanska, D. Goldfarb, and A. Weller. Leader stochastic 450 gradient descent for distributed training of deep learning models. In NeurIPS , 2019. 451

Yunfei Teng, Wenbo Gao, Francois Chalus, Anna Choromanska, Donald Goldfarb, and Adrian 452 Weller. Leader Stochastic Gradient Descent for Distributed Training of Deep Learning Models: 453 Extension, April 2022. 454

Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. 455 In Proceedings of the 28th International Conference on International Conference on Machine 456 Learning , ICML'11, page 681-688, Madison, WI, USA, 2011. Omnipress. 457

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for 458 benchmarking machine learning algorithms, 2017. 459

Qiwei Ye, Yuxuan Song, Chang Liu, Fangyun Wei, Tao Qin, and Tie-Yan Liu. Particle based 460 stochastic policy optimization. 2021. 461

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint 462 arXiv:1605.07146 , 2016. 463

S. Zhang, A. Choromanska, and Y. LeCun. Deep learning with elastic averaging SGD. In NIPS , 464 2015. 465

467

468

469

470

471

Follow Hamiltonian Leader: An Efficient Energy-Guided Sampling Method

(Supplementary Material)

A Additional Discussion for Section 3

A.1 Instability & Metastability 472

We now approach this problem from an optimization perspective. There is a strong connection 473 between optimization and sampling, particularly through the principle of simulated annealing [24], 474 which demonstrates how sampling methods can be transformed into optimization techniques. 475

With a slight abuse of notation, we consider the following objective function: 476

where x ∈ R 2 and x [ i ] denotes the i th dimension of x . This is a 2-dimensional optimization problem 477 with a condition number of 100, indicating it is somewhat ill-conditioned. 478

For n particles, the objective function is: 479

where we set ρ = 0 . 1 . We initialize x 1 = (2 , 2) and x 2 = ( -1 , -3) respectively, and optimize the 480 objective function using the gradient descent method. Note that when n = 1 , this method reduces to 481 vanilla gradient descent, while n = 2 incorporates our leader-pulling scheme. 482

Trajectory

Loss

2.00

gradient

gradient+leader

1.75

1.50

3

1.25

1.00

2

0.75

0.50

gradient

1

0.25

gradient+ leader

start

0.00

minimum

0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

0

10

20

30

40

50

Trajectory

Loss

2.0

4.0

gradient+leader

gradient

3.5

1.5

3.0

2.5

1.0

2.0

1.5

0.5

1.0

gradient

gradient +leader

0.5

0.0

minimum

0.0

1.0

0.5

0.0

0.5

1.0

1.5

2.0

0

20

40

60

80

100

From Figure 11, we can see that incorporating the leader-pulling scheme helps improve convergence. 483 This demonstrates that the leader-pulling scheme can address the issue of instability in optimization. 484 However, we also observe that a carefully chosen leader is usually required for our method, which 485 we will leave for future discussion. 486

487

488

489

Furthermore, as shown in Figure 12, the particle using the leader-pulling scheme explores much further compared to the vanilla heavy-ball method. This outcome is expected, as we want the method to enhance exploration and thereby resolve the metastability issue.

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

A.2 Pseudo Stability

These challenges are commonly encountered when sampling from compositional models, particularly when one of the distributions is a piecewise-constant distribution with its gradients are zero almost everywhere in its domain. To illustrate this, consider the example π ( x ) ∝ π 1 ( x ) · π 2 ( x ) . Here we consider ∂ log π 2 equals to zero everywhere.

̸

It's worth noting that while combining distributions in their logarithmic forms is straightforward, which leads to log π ( x ) = log π 1 ( x ) + log π 2 ( x ) + constant , omitting the constant log π ( x ) can be readily derived from the individual log π 1 ( x ) and log π 2 ( x ) . However, the composition of their gradients becomes problematic, as the computation of the sub-gradient ∂ x log π ( x ) = ∇ x log π 1 ( x )+ ∂ x log π 2 ( x ) in general due to the use of automatic differentiation in machine learning [4].

In this section, we focus on the disparity between gradient and energy in the context of combining two distributions as indicated in Section 3.

We analyze a composite probability distribution structured as π ( x ) ∝ π 1 ( x ) · π 2 ( x ) , leading to the construction of two specific distributions:

The first distribution, π 1 ( x ) , is given by:

where | X | represents the cardinality of the set X , indicating the total number of elements in X .

The second distribution π 2 ( x ) , is defined as

with Ω Y being the set where Ω Y = { x | d ( x, Y ) < ϵ } . In this context, the distance metric d is specified by d ( x, Y ) = arg min y ∈ Y ∥ x -y ∥ 2 , indicating the minimum Euclidean distance from x to any point in the set Y .

Observe that π 1 constitutes a smooth distribution, whereas π 2 is a piecewise-constant distribution. Consequently, for π 2 , the gradients are zero almost everywhere. When we consider the expression ∇ x log π 1 ( x ) + ∂ x log π 2 ( x ) , it could be simplified to ∇ x log π 1 ( x ) , which is not equivalent to ∂ x log π ( x ) in general.

517

518

519

520

521

523

524

525

526

527

528

529

530

531

532

In the subsequent subsections, we present two motivating examples: one in a low-dimensional setting and the other in a high-dimensional context. Throughout these experiments, we set σ 2 = 0 . 002 and ϵ = 0 . 2 . In this section, the outcomes of U-LMC and U-HMC are omitted because both techniques succumb to the issue of misleading gradients by nature, causing worse performance.

A.2.1 Low-dimensional Example

We propose an example inspired from [12] but in a different setting. In this revision, we begin by providing a more specific definition for two distributions. For the first distribution π 1 , we define:

and for the second distribution π 2 , we specify:

It's important to note that, by definition, Y is a subset of X . 522

Ground Truth

FHL

LMC

HMC

饿.

We perform a comparative study of our methods against established benchmarks, and the visual representations of this comparison can be found in Figure 13. Notably, among the compared methods, FTH distinguishes itself due to its outstanding performance, mainly attributed to its precise adjustment of particle positions. The comparative results highlight that the baseline methods often exhibit the tendency to erroneously converge towards incorrect modes due to the misleading gradients. Although rejection steps of HMC and LMC might mitigate incorrect sampling, particles initialized near high-energy modes struggle to escape this erroneous attraction by misleading gradients.

A.2.2 High-dimensional Example

We then present a case study in which we generate examples from a particular category within the Fashion MNIST dataset [43]. In this experiment, we select a total of 200 images, with 100 images

from the coat category and another 100 from trouser category. We denote the sets of data points from 533 the coat and trouser categories as X coat and X trouser respectively. Furthermore, we define X as the 534 union of X coat and X trouser, and Y is set to X coat in this case. 535

FHL

LMC

HMC

To increase the difficulty of the sampling task, we initially position each particle at the mean location of X trouser. The outcomes of the sampling are depicted in Figure 15.This setup showcases the FHL method's ability to accurately target and sample from the specified coat category, in contrast to baseline methods that undesirably draw samples from the trouser category.

B Supplementary Experiment

B.1 Experiment Setup

In Section 5.2 , we utilize a pre-trained classifier available on the public GitHub repository at https://github.com/wgrathwohl/JEM . This classifier is a WideResNet model [45] with a depth of 28 and a width of 2 .

We use a technique called one-step HMC [5] and thus the momentum gets refreshed for each step. More specifically, for both FTH and HMC we set the momentum damping factor to 0 . 9 and the mass matrix as 0 . 004 2 · I . We take step size as η = 0 . 2 . Since the mass matrix is set to a relatively small value which easily causes the instability of training, we always accept the proposed states based on the potential energy and ignore the kinetic energy.

In Section 5.3 , we mainly adapted the codes and models from https://github.com/yilundu/ reduce_reuse_recycle .

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

For Section 5.3.1, we initially train a 4-layer ResNet as the energy-based score-matching model on p 1 and p 2 independently. During the sampling process, we combine these models. We employ step sizes η = { 0 . 002 , 0 . 0002 , 0 . 005 , 0 . 0005 } for all methods and the number of leapfrog steps L = { 4 , 8 } for HMC-type methods.

For Section 5.3.2, we utilize a U-net architecture [33] as the energy-based score-matching model. 556 This architecture is directly obtained from a pre-trained model available at . For sampling, we use step 557 sizes η = { 0 . 01 , 0 . 035 , 0 . 05 , 0 . 1 , 0 . 2 } for all methods and set the number of leapfrog steps L = 4 558 for HMC-type methods. 559

B.2 Additional Results 560

B.2.1 Additional Images for Section 5.2 561

(a) Class of cats .

(b) Class of horses .

B.2.2 Additional Images for Section 5.3.2 562

(a) Baseline

(b) MALA

(c) HMC

(d) FTH

(b) MALA

(c) HMC

(d) FTH

(b) MALA

(c) HMC

(d) FTH

.1.

1.2

中

B.2.3 Zoomed Images for Section 5.3.2 563

(d) FTH

(b) MALA

(c) HMC

(d) FTH

C Supplementary Theorem

564

We now consider a scenario where the leader becomes corrupted, meaning the corrupted leader always 565 reports an unreasonably low energy but it is not actually in the lowest-energy position. In this situation, 566 the particles are optimizing a biased objective function. For simplicity, we consider a d -dimensional 567 Gaussian distribution p ∼ e -U ( x ) and its modification q ∼ e -ψ ( x ) with ψ ( x ) = U ( x ) + λ 2 ∥ x -z ∥ 2 . 568 We will analyze the Wasserstein distance between p and q for a fixed z ∈ R d as a function of λ > 0 . 569 We will demonstrate that even though we sample from the distribution q instead of p , the bias of the 570 sampler (i.e., the distance between p and q ) can be controlled by λ and vanishes as λ → 0 . 571

Assumption 1 . U : R d → R is M -Lipschitz-differentiable, i.e. ∀ x, y ∈ R d , 572

and U is m -Strongly-convex, i.e. ∀ x, y ∈ R d , 573

Theorem 1 . Let U be the negative logarithmic probability density function of a d -dimensional Gaussian distribution, which satisfies Assumption 1. Let us define the function ψ ( x ) as ψ ( x ) = U ( x ) + λ 2 ∥ x -z ∥ 2 . Given this setup, the Wasserstein-2 distance between the modified Boltzmann distribution q , characterized by q ∼ e -ψ ( x ) , and the original Gaussian distribution p , denoted as p ∼ e -U ( x ) , can be bounded as:

where ∥ · ∥ represents the matrix norm. Obviously, W 2 ( p, q ) → 0 when λ → 0 . 574

Proof. By definition, U is m -strongly convex since 575

The m corresponds to the smallest eigenvalue of Σ -1 which is therefore 1 / ∥ Σ ∥ . Then 576

The last equation was done by completing the square. Thus the new distribution is still a Gaussian 577 distribution, represented as 578

Consequently, the Wasserstein-2 distance can be determined as follows: 579

In our case µ p = x ∗ , µ q = (Σ -1 + λI ) -1 (Σ -1 x ∗ + λz ) , Σ q = Σ , Σ p = (Σ -1 + λI ) -1 . Since Σ q and Σ p can be jointly diagonalized by some orthonormal basis T ,

thus Σ q and Σ p commute. We can simplify the Wasserstein distance to

581

Then

Now we bound the first and second term independently. The first term is a direct conclusion from Theorem 15 in [41],

For the second term. We denote i th eigenvalue of matrix Σ as σ i , then ∥ Σ ∥ = max i σ i , such that 580

Thus, by combing the two terms together, the total Wassertein distance is bounded by

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly indicate the purpose and contribution of our paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [No]

Justification: Our work builds upon previous research, serving as a complementary addition to it. However, we shall add further discussion in the future.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We do provide assumption and proof for our theorem.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We show the details of how to reproduce the results and provide the codes for reproduction.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will do it once the paper gets accepted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes we do.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Due to time constraints, we were unable to include that. However, we plan to add such statistics in the future.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We primarily focus on proposing a new algorithm, and most of our experiments require only a moderate amount of resources.

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Yes we do.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no social impact of our work.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such a risk.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: No we don't have such a problem.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No we don't have such a problem.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No we don't have such a problem.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

669

Justification: No we don't have such a problem.

670

Peer reviews:
comments: SUMMARY:
This paper presents a novel parallel sampling method named "Follow Hamiltonian Leader" (FHL) designed to address sampling challenges by leveraging zeroth-order information, particularly when first-order data is unreliable or unavailable. The method incorporates a leader-guiding mechanism to enhance the efficiency and effectiveness of the sampling process. Experimental results indicate that FHL significantly improves the exploration of target distributions and outperforms traditional sampling techniques, especially in scenarios involving corrupted gradients.

STRENGTHS:
1. Innovative combination of zeroth and first-order information.
2. The effectiveness of the method is demonstrated in multiple task scenarios.
3. Theoretical analysis and prove are sufficient.

WEAKNESSES:
1. Is there any quantitative experiments like evaluating FID and IS on cifar10 datasets and I think it's more compelling whether a novel sampling methods combined with generative models can be used on image datasets with more complex distributions.
2. Lack of experiment of OOD in combination with EBMs or score-based models to valid the stability during sampling with proposed method.

QUESTIONS:
Why is E(x, \tilde x) defined in this way. And is there any relation between extra elastic energy and local entropy [1].


[1] P. Chaudhari, A. Choromanska, S. Soatto, Y. LeCun, C. Baldassi, C. Borgs, J. T. Chayes, L. Sagun, and R. Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. In ICLR, 2017.

LIMITATIONS:
1. Limited exploration of integration with other advanced MCMC methods.
2. Lack of quantitative experiments to demonstrate the advantage of proposed sampling method compared with other methods.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an interesting parallel sampling method that leverages zeroth-order information to address challenges in sampling from probability distributions, particularly when first-order data is unreliable or unavailable. The method incorporates a leader-guiding mechanism, enhancing efficiency and effectiveness by connecting multiple sampling instances through a selected leader. The proposed method, named Follow Hamiltonian Leader (FHL), extends the Hamiltonian Monte Carlo (HMC) framework by concurrently running multiple replicas at different energy levels and combining both zeroth and first-order information from various chains. Experimental results demonstrate that FHL significantly improves the exploration of target distributions and produces higher-quality outcomes compared to traditional sampling techniques, showing resilience against corrupted gradients and excelling in scenarios characterized by instability, metastability, and pseudo-stability.\n\nSTRENGTHS:\n- The proposed Follow Hamiltonian Leader (FHL) method markedly improves the efficiency and effectiveness of sampling processes, significantly expediting the exploration of target distributions and producing superior quality outcomes compared to traditional sampling techniques.\n- FHL demonstrates greater resilience against the detrimental impacts of corrupted gradients by incorporating zeroth-order information. This robustness makes the method particularly valuable in scenarios where first-order information is compromised, ensuring more reliable and accurate sampling.\n\nWEAKNESSES:\n- The proposed FHL method involves intricate modifications to the traditional Hamiltonian Monte Carlo framework, such as the leader-guiding mechanism and elastic leapfrog technique, which may increase the complexity of implementation and require significant computational resources.\n- The effectiveness of the FHL method heavily relies on the appropriate selection of the leader particle. If the leader is not accurately chosen, it could lead to suboptimal sampling performance, potentially compromising the overall efficiency and accuracy of the method.\n- While the paper presents experimental results to demonstrate the efficacy of the FHL method, there is a lack of in-depth theoretical analysis to rigorously establish the convergence properties and performance guarantees of the proposed approach.\n- The method’s scalability to high-dimensional problems or extremely large datasets is not thoroughly addressed. The parallel sampling approach may encounter challenges in maintaining efficiency and effectiveness as the dimensionality and size of the data increase.\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nThe authors have not adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

comments: SUMMARY:
This work proposes to incorporate the energy $U$ into the gradient-based sampling techniques. In particular, it proposes to choose the lowest energy particle as the leader and then add an extra elastic tension between the leader and followers in the Hamiltonian Monte Carlo method.

STRENGTHS:
The idea is simple and clear, the toy examples are easy to understand and demonstrate the benefit of the proposed method well. In addition, the authors conduct experiments for each of the three challenging sampling scenarios identified by the authors.

WEAKNESSES:
It might worth including the overhead of the proposed method, how much slower the algorithm is per iteration compared to HMC for instance.

The tension coefficient $\lambda$ is critical, setting it to 0 recovers the baseline. But I did not find an ablation over the $\lambda$, is it hard to choose? From my understanding, if you set $\lambda$ pretty large it might recover something like gradient descent and the sampling will collapse.

QUESTIONS:
1. How is $\lambda$ determined per experiment? Is it possible to include an ablation over it for the three scenarios at least?

2. In Figure 7, it is unclear to me which method is better. It will be better to provide some quantitative measurement like FID or IS scores?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an interesting new approach for improving sampling methods for energy-based generative models and score-matching models. The key idea is to incorporate zeroth-order information (energy values) in addition to the typical first-order gradient information used by most sampling algorithms like Hamiltonian Monte Carlo (HMC).\n\nThe authors identify several challenging scenarios where relying solely on gradients can be problematic - cases of instability, metastability, and pseudo-stability. They argue that incorporating energy values can help mitigate issues in these situations and improve sampling efficiency and quality.\n\nOverall, the core idea of leveraging zeroth-order information in addition to gradients is quite novel and the FHL algorithm is an elegant way to implement this for improving sampling efficiency and quality. The paper is well-motivated, the method is clearly explained, and the empirical results are compelling.\n\nSTRENGTHS:\n1. Novel idea of incorporating zeroth-order energy information into sampling algorithms like HMC, which typically only use gradients. This can help address issues like instability and metastability.\n\n2. The Follow Hamiltonian Leader (FHL) algorithm is an elegant way to exchange both energy and gradient information across parallel sampling chains in a principled manner.\n\n3. Thorough experimental evaluation across synthetic examples illustrating the identified challenging scenarios of instability, metastability, and pseudo-stability.\n\n4. Promising results showing improved sampling quality over baselines for energy-based generative models on real datasets like CLEVR.\n\n5. Clear motivation and well-explained methodology.\n\nWEAKNESSES:\n1. It would be better to show exploration of the sensitivity to key hyperparameters like the number of parallel sampling chains.\n\n2. Discussion of computational cost/overhead compared to baseline sampling methods are missing in the manuscript.\n\nQUESTIONS:\n1. While the empirical results are compelling, can you provide any theoretical analysis or guarantees about the convergence properties, sampling quality, or stationary distribution of the FHL algorithm? Even approximate bounds or insights would strengthen the theoretical grounding.\n\n2. What are the main computational and memory overheads introduced by running parallel chains in FHL compared to standard HMC? Is there a limit on scalability to high dimensions or is there a way to reduce overhead through approximations?\n\nLIMITATIONS:\nBased on the provided paper, the authors do not appear to have explicitly discussed the limitations or potential negative societal impacts of their work. The paper is primarily focused on presenting the technical details of the proposed Follow Hamiltonian Leader (FHL) sampling algorithm and its empirical evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-17"}