Microcanonical Langevin Ensembles: Advancing the Sampling of Bayesian Neural Networks
Authors: Emanuel Sommer
Jakob Robnik
Giorgi Nozadze
Uros Seljak
David Rügamer
Venue: ICLR 2025 Poster (2025)
Keywords: Sampling
Bayesian Neural Networks
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Despite recent advances, sampling-based inference for Bayesian Neural Networks (BNNs) remains a significant challenge in probabilistic deep learning. While sampling-based approaches do not require a variational distribution assumption, current state-of-the-art samplers still struggle to navigate the complex and highly multimodal posteriors of BNNs. As a consequence, sampling still requires considerably longer inference times than non-Bayesian methods even for small neural networks, despite recent advances in making software implementations more efficient. Besides the difficulty of finding high-probability regions, the time until samplers provide sufficient exploration of these areas remains unpredictable. To tackle these challenges, we introduce an ensembling approach that leverages strategies from optimization and a recently proposed sampler called Microcanonical Langevin Monte Carlo (MCLMC) for efficient, robust and predictable sampling performance. Compared to approaches based on the state-of-the-art No-U-Turn Sampler, our approach delivers substantial speedups up to an order of magnitude, while maintaining or improving predictive performance and uncertainty quantification across diverse tasks and data modalities. The suggested Microcanonical Langevin Ensembles and modifications to MCLMC additionally enhance the method's predictability in resource requirements, facilitating easier parallelization. All in all, the proposed method offers a promising direction for practical, scalable inference for BNNs.

Published as a conference paper at ICLR 2025
MICROCANONICAL LANGEVIN ENSEMBLES: ADVANCING THE SAMPLING OF BAYESIAN NEURAL NETWORKS
Emanuel Sommer
Jakob Robnik
Department of Statistics, LMU Munich
Physics Department
Munich Center for Machine Learning (MCML)
University California
Munich, Germany
Berkeley, USA
emanuel @etat. wwi.tuuuummmeence..co
jakob robni meonik@berkeley.com edu:
Giorgi Nozadze
Uroš Seljak
Department of Statisties, LMU Munich
Physics Department, University of California
Eraneos Analytics Germany GmbH Munich, Germany
Lawrence Berkeley National Laboratory Berkeley, USA
giorgi nozadzederraeoos... com
usel makkberkeeeey edu
David Rügamer
Department of Statisties, LMU Munich
Munich Center for Machine Learning (MCML) Munich, Germany
david@ater uni uni-muenchen.ce
ABSTRACT
Despite recent advances, sampling-basee inference for Bayesian Neural Networks (BNNs) remains significant challenge in probabilistic deep learning. While sampling- basen approaches do not require a variational distribution assumption, current sate-off-the-nn samplers still struggle navigate the complex and highly multimodal posteriors of BNNs. As a consequence, sampling still requires con- siderably longer inference times than non-Bayesian methods for small neu- ral networks, despite recent advances in making software implementations more efficient. Besides the difficulty of finding high-probability regions, the time un- til samplers provide sufficient exploration of these areas remains unpredictable. To tackle these challenges, we introduce an ensembling approach that leverages strategies from optimization and recently proposed sampler called Microcanon- ical Langevin Monte Carlo (MCLMC) for efficient, robust and predictable sam- pling performance. Compared to approaches based on the state-of-the-art No- U-Turn Sampler, our approach delivers substantial speedups up to an order magnitude, while maintaining or improving predictive performance and uncer- tainty quantification across diverse tasks and data modalities. The suggested Mi- crocanonical Langevin Ensembles and modifications to MCLMC additionally hance the method's predictability in resource requirements, facilitating easier par- allelization. All in all, the proposed method offers a promising direction for prac- tical, scalable inference for BNNs.
INTRODUCTION AND RELATED LITERATURE
Sampling-based inference for Bayesian Neural Networks (BNNs) has garnered significant interest as a principled approach to addressing the analytically intractable challenge of probabilistic deep learning (Izmailov et al.] |021 Wiese et al. 2023] Papamarkou et al. 2024). New methods, such as subspace inference (Izmailov et al.] 2020; Dold et 2024). are being explored alongside emerg- ing applications in diverse domains where effective uncertainty quantification is crucial, including

---

Published as a conference paper at ICLR 2025
healthcare (Peng et al. 2020) and physics (Cranmer et al. 2021 Papamarkou al. (2022): /Sommer et al. (2024) highlight several shortcomings in current sampling-bes based approaches, particularly the need for proper initialization of sampling procedures and the challenge of capturing multimodality.
Samplers and problem setup Hamiltonian Monte Carlo (HMC; Duane et al., 1987) and Under- damped Langevin Monte Carlo (Leimkuhler & Reich 2009) are the gold standard algorithms for
high-dimensional sampling problems. However, their performance is known to be sensitive to their hyperparameters, such as the step size, preconditioning. and the momentum decoherence rate (Neal
2011 They therefore combined with automatic hyperparameter adaptation algorithm. Sev- cral ensemble-based schemes have been developed in recent years (Sountsov & Hoffman [022] Hoffman & Sountsov 2022 Riou-Durand et al. 2023), but all of these schemes critically depend on the ensemble variance of the parameters to tune the critical momentum decoherence rate hyper-
parameter and in some cases also the (preconditioned) step size. As such, they are not applicable to the highly multimodal posteriors of BNNs.
In the context of gradient-free sampling, variety of ensembled algorithms have been developed, for example, Preconditioned Monte Carlo (pocoMC; Karamanis et al. 2022a b) Nested Sampling (Skilling 2004) and Elliptical Slice Sampling (Murray et al.) 2010). These algorithms address the multmodaliiyy problem; however, they scale poorly to high-dimensions
settings, For example, Elip- tical Slice Sampling is frequently used for BNNs (Izmailov et al. 2020, Dold al., 2024) for sam- pling in a small subspace (e.g., 2- or 3-dimensional) of the parameter space, but cannot be scaled to much larger dimensions. While alternative methods for multimodal sampling exist, they are also not well-suited for the high-dimensional and highly multimodal BNN setting. For instance, stochastic localization methods, like those proposed by Grenioux et al. (2024), can handle moderate multi- modality but are sensitive to hyperparameters (like the integration initialization to for stochastic localization) and lack scalability. Similarly, the Liouville Flow Importance Sampler (Tian et al.) 2024) offers unbiased sampling but requires the training of neural flow model, which needs to become increasingly complex as the dimensionality and complexity of the problem grows, limiting its scalability. The same applies the learned vector field in the path-guided particle-based method Fan et al. (2024), with an empirical comparison provided in Appendix A.1.3
HMC and NUTS It is therefore not surprising that a sequential HMC variant, the No U-turn Sampler (NUTS: Hoffman & Gelman 2014), is still viewed as "[t]he only MCMC algorithm that theoretically scales to high dimensions across a broad class of models" (Strumbelj et al. 2024). While the scalability of HMC allows the application to the full parameter space of moderately-sizer neural networks and its NUTS variant provides an (almost) tuning- free approach of HMC, it can- not sufficiently explore the many modes present in BNN posteriors( CrrerseessWiesee et al., 2023). The recent HMC variant Symmetric Split HMC (Cobb & Jalaian 2021) improves HMC's s memory scalability but comes with other drawbacks, including sensitive hyperparameters. We provide an empirical comparison and further discussion in Appendix AA.13 Standard priors, like isotropic Gaussians, can further lead to initialization issues, where samplers start in low-probability regions, resulting in slow convergence or samplers getting stuck. To address these issues, [Sommer et al. (2024) pro- pose Bayesian Deep Ensemble (BDE), an ensemble of many short warmstarted Markov chain Monte Carlo (MCMC) chains. With this, the authors increase the exploration capability and are able to achieve state-of-the-art predictive and uncertainty quantification (UQ) performance on com- mon benchmark tasks. Although much faster than previously employed methods by leveraging parallelization and efficient implementations, Sommer et al. (2024) rely on NUTS. This imposes significant computational burden and results in a method where the sampling phase still dominates the computational costs by far.
Our contributions In an effort to improve the scaling and efficiency of sampling-based inference for BNNs, we identified Microcanonical Langevin Monte Carlo (MCLMC: Robnik et al., 2023) as a possible alternative MCMC-based solution. In recent experiments, the authors were able to show that MCLMC can be computationally superior to NUTS while providing the same quality of samples in downstream metrics. While MCLMC demonstrates computational advantages over NUTS in uni- modal and sequential sampling tasks, it cannot address the multimodality, numerical instability, and scalability challenges of BNNs without significant modifications (e.g. Appendix A.2. We em- bed adapted MCLMC as the key ingredient in our approach, including deep ensemble initialization for enhanced exploration, adjustments to ensure numerical stability in high-dimensional settings,
2

---

Published as a conference paper at ICLR 2025
and optimizations targeting critical bottlenecks. The resulting method, Microcanonical Langevin Ensemble (MILE), comes with automated tuning and works reliably out of the box. Extensive ex- periments highlight that MILLE achieves state-of-thee of-the-art performance while being up to order of magnitude faster than previous sampling-basee approaches.
2 BACKGROUND
In this work, we denote neural networks with f X + V. C R", We parameterize the network with 0 E e C Rd denoting the vector of all flattened and concatenated weights and biases. To express the epistemic uncertainty about 9, we treat 0 as a random variable with prior p(D)))p(0))/)
density p(0) and denote its posterior density as p(0)D)
with observed data D E (X' x >))". We will assume a standard isotropic unit variance Gaussian prior NO, /) if not specified otherwise. A more detailed analysis of the prior influence is beyond the scope of this work. The posterior predictive density (PPD) quantifies the uncertainty of predicting unseen labels y e y given features x" E X by integrating over the posterior distribution of the parameters O:
" = 1 ply"l= <=" (ly)=",0)p((0 ()p(0(D)d
(1)
2.1 MONTE CARLO SAMPLING
In the setting of sampling- based inference, we estimate the analytically intractable integral in Eq. ! using samples from the posterior distribution with density p(O(D). These are obtained by Markov chain Monte Carlo (MCMC) methods, which construct Markov chain whose stationary distribution is the posterior distribution close to it. In practice, we gather these samples from K independent
chains, each with S samples, yielding the set (O(k.)) E [K], E [S] based on which we can perform prediction and uncertainty quantification (Andrieu et al. 2003 approximation of Eq. 4 then has the form
Gelman al. 2013). The
K
S y g(kw))
ply" x D) = K.S
ΣΣ
(2)
Error analysis and Netropolis-Hastinnee adjustment The error of this approximation is com- posed of three terms: initialization error, discretization error, and Monte Carlo error. The initializa- tion error is a transient effect caused by the chains" ensemble distribution having not yet reached the stationary distribution (burn-in phase). The Monte Carlo eтror is the variance caused by the finite number of samples S and chains K. The discretization error is caused by the finite step size used to numerically simulate the sampler's dynamics. This causes p(0 (D) to longer be the stationary dis- tribution. The Eatropolis-Hasting (MH) scheme is typically employed to completely eliminate the discretization error, but this comes rate depends exponentially on the squared energy error, which grows linearly with dimensionality. Therefore, the step size needs to be decreased as the number of parameters increases in order to maintain fixed acceptance rate, causing the sampler to move more slowly in higher dimensions. The MH algorithm is further prone to degeneration of the acceptance rate if the chains are initialized from a distribution, which is not already very close to the stationary distribution, causing slow con- vergence and large initialization error (Durmus & Eberle (023). Without the Metropolis adjustment on the other hand, the discretization error depends strongly on the step size. Hence, slightly reducing the step size causes the discretization error to become negligible compared the initialization and Monte Carlo error. Given these arguments, we choose to omit the MH adjustment in our method, relying instead on careful initialization and step size tuning to manage the error terms.
expense of shorter step size. This is because the acceptance
2.2 MICROCANONICAL LANGEVIN MONTE CARLO
A recently proposed sampling method outside the BNN research field is Microcanonical Langevin Monte Carlo (MCLMC). The time evolution of the MCLMC sampler (Robnik & Seljak 2024 is governed by the stochastic differential equation
d0 u dt, du (1 uu ((d- 1 logp(0 dt ddW),
(3)
3

---

Published as a conference paper at ICLR 2025
Optimization
MCLMC Warmup & Tuning
MCLMC Sampling
Varilla regularized Macimum Likelhood
Phase Eliriing desine energy acheduler.
Phase III
Thinning contrals memory inference requiements
Training
Refine ualimaile I using autocornlali lationss
L
Losa: Nag. Lagal Optinizor
Phase
Juting via paramoter
Thinnings ThimingSSuhampplng bound cosl.
k
g
early stopping
stops
staps
stops
10k stops
Deep Ensemble
Microcanonical Langevin Ensemble
Figure 1: Flowchart illustrating our proposed procedure for obtaining Microcanonical Langevin Ensemble (MILE) for BNNs. The process involves three main stages: optimization, MCLMC warmup and tuning. and MCLMC sampling. These steps are parallelized to generate an ensemble of K members. The number of MCLMC steps for each tuning phase and the final sampling phase are annotated, and carryovers between stages are highlighted in cireles.
where Win is the Wiener process and 7) is a free parameter equivalent to the typical length traveled in the parameter space before momentum decoherence, often denoted with L. |Robnik & Seljak (2024)
use the drift-diffusion discretization to numerically solve Eq. (3). The drift part (Eq. (3) without
the last, stochastic term) is solved by the minimal norm integrator (Takaishi & de Forcrand, 2006
Omelyan & Kovalenko 2013). In the limit of small step size e, the resulting Markov chain has p(0 as a stationary distribution (Robnik & Seljak 2024). Smaller step size results in smaller discretization error but increases the Monte Carlo and initialization errors because the sampler is moving more slowly. We therefore wish to reduce the step size to the point where the discretization error becomes smaller than the other two sources of error, but not much further. Robnik et al.]
(2023) propose the energy error variance per dimension (EEVPD) as a measure of the discretization error and show that one can control the discretization error by controlling EEVPD. Analogously to adapting the step size to match some targeted acceptance rate in detropolis-adjusted algorithms, one can adapt the step size to match some desired EEVPD in the unadjusted algorithms. This inexact but highly efficient approach with bias control is the one we will also take in our work.
Note that the stationary distribution is independent of the free parameter and in particular also holds for the deterministic dynamics, 77 = 0. This is not the case in HMC, where momentum resampling is essential to maintain the desired stationary distribution. Being more deterministic allows MCLMC to converge faster during the exploitation phase.
The parameter 7 is still important, as it controls the rate of the momentum decoherence and forces the dynamics to move to the unexplored parts of the parameter space. |Robnik et al. (2023) found that a good performance is achieved when L/e is on the order of the chain's autocorrelation time. As the autocorrelation time is quite expensive to evaluate, they also propose to set L to the size of the posterior modes, by computing the variance of the parameters.
Based on these considerations, they propose a three-stage tuning scheme, which we now refer to as the three phases:
1. Phase I: Adapt the step size match the desired EEVPD and complete the burn-in.
2. Phase II: Estimate parameter variance to obtain an initial estimate for L.
3. Phase III: Estimate autocorrelation time to refine the estimate of L.
3 MICROCANONICAL LANGEVIN ENSEMBLES
In order to embed MCLMC in a robust sampling pipeline (cf. Fig. that works effectively for BNNs and exploits the idea of ensembling, we will discuss the combination of optimization techniques and sampling as well as various aspects of required MCLMC modification in the following. Without these modifications, MCLMC struggles with exploration and exhibits high failure rates, particularly in high-dimensional settings (see Appendix A.1.2..
3.1 ENSEMBLING FOR REDUCED INITIALIZATION ERROR
As discussed in Section 2.I, the error in the approximation of the predictive posterior can be de- composed into three parts. While the initialization error is described in Section 2.1 as "transient
4

---

Published as a conference paper at ICLR 2025
effect", this error should not be mistakenly classified as unimportant, and treating it with special care is particularly important in the application of BNNs. Similar to most other samplers, it is not unlikely for MCLMC to get stuck in low-probayility regions of the high- -ishdimensional and highly complex posterior surfaces of BNNs. Our proposed solution is therefore to combine sampling with an optimization step (Fig. blue part). To prevent chains in the sampling step from being initialized in these unfavorable regions, for each of the K chains. As a byproduct, we obtain a deep ensemble (DE; Laakshmiaaraannan et al,, 2017) with K members | Similar to Sommer al. (2024), we found that using optimized neural networks as starting values can reduce this error notably and prevents the sampler from getting stuck.
K optimization steps to obtain starting values (=(A), A [K]
3.2 TUNING PHASE ADAPTIONS
In the previous subsection, we proposed to optimize starting values when using MCLMC by running an ensemble of MCLMC chains. Optimizing the starting values of each chain is, however, not sufficient to make MCLMC work numerically stable and efficiently for BNNs (see Appendix 1.1..)..
We therefore discuss three components of MCLMC's tuning phase requiring further adaptions in applications to BNNs, and propose scaling related modifications to phase III of MCLMC.
Step size Robnik et al. (023) suggest to initialize the step size e with vd. While this default works well for their applications, this would result in too large values even for rather small neural networks. Too large step sizes, in turn, will result in significant changes of the energy and therehy introduce serious numerical problems. As MILE initializes chains already in a region of high prob- ability, it is reasonable to reduce e notably. In practice, this means we can set the step size to the optimizer's 's learning rate used in the optimization step of MILE. Besides reducing the initial dis- cretization error, this also ensures more localized exploration early in the burn-in phase and close to the optimized solution while allowing for larger steps as tuning progresses (which can also be confirmed empirically).
Energy variance scheduler Another tuning parameter in phase I of MILE the energy variance scheduler allowing control of the trade-off between exploitation and exploration. Having already optimized K starting values, MILE does not require an excessive amount of exploitation but each MCLMC ideally focuses more on exploitation. In contrast to the tuning scheme proposed in Robnik jtt al. (003),, we do not employ a fixed desired energy variance level but use a linear scheduler starting with higher desired energy variance and gradually decreasing it. By doing this foster noisier exploration at the beginning of the warmup phase and increase the exploitation towards the end of it. Doing this in parallel across ensemble of chains, our idea resembles the popular strategy of using cyclical learning rates in SG-MCMC sampling (Zhang et al. 2020). But rather than going through various exploration and exploitation phases we do one exploration-exploitatoo cycle per chain and thus parallelize this idea
Effective sample size MCLMC further requires setting a desired effective sample size (ESS) level within the EEVPD estimation phase. While this level is generally choice of the practitioner and the available computational resources, one option is to set the ESS to 10% of the total number of posterior samples. As previous approaches working with NUTS also report an ESS of around 10% of the total number of samples, this value ensures a sample quality that is as good as the one of NUTS.
Phase III bottleneck When scaling the MCLMC tuning algorithm to higher dimensions, a com- putational bottleneck of phase III is to estimate the empirical ESS, which is done with Fast Fourier Transform (FFT. Bracewell & Bracewell 1986). The FFT is computed for each parameter across all samples. Although fast implementations scale with OS log S), this can severely impact the runtime for increased dimensions and amount of samples. We therefore propose to subsample parameters
Note that the optimization of these K models actually has negligible costs compared to the costs of current state-of-the-art sampling approaches.
We empirically found that the linear schedule from 0.5 to 0.1 worked well in a variety of settings. We also explored exponentially decaying schedulers starting with high desired energy variance, but these proved to be empirically inferior to the aforementioned linear schedule. This is in line with the intuition that excessive exploration is not necessary due the DE. initialization.
5

---

Published as a conference paper at ICLR 2025
if the dimensionality is d > 2000, which linearly decreases the runtime. We also suggest applying thinning to the samples such that the number of samples used for the FFT is bounded by 10c3.
3.3 EFFECTIVE COMPUTATIONAL BUDGET ALLOCATION
The third error caused by the finite number of samples and chains is the Monte Carlo error. Un- like NUTS, which varies the number of leapfrog steps for each proposal and requires one gradient evaluation per leapfrog step, MCLMC employs deterministic approach that requires two gradient evaluations per sample due to the use of minimal norm integrator. This structure makes MCLMC's computational requirements much more predictable, offering a significant practical advantage. In contrast, the number of steps taken by NUTS can exhibit substantial variance both within and across tasks, leading practitioners to set upper limit on leapfrog steps. With better foresceable computa- tional requirements, we can optimize the balance between the number of chains and the number of samples in MILE in order to minimize the Monte Carlo error for given computational budget. As recent literature suggests that MCLMC can be much more efficient than NUTS in obtaining effec- tive samples in simple problems, we propose collect relatively small number of samples from each chain and thereby limit computational resources spent on exploring individual local modes by MCLMC but relying DE for exploration.
Warmup stage It is not advisable to directly start chains from the DE optimized points, these modes do not coincide with the so-called typical sets (Retancourt (018)) we suggest allocating 40k steps, i.e., 80k gradient evaluations for MILE's warmup phase. Compared to NUTS, which often requires more than 90k leapfrog steps/gradient evaluations for warmup in BNN tasks, this is conservative lower bound. This is followed by the two shorter phases, II and III, where allocate 5k steps each to ensure robust estimations of the momentum decoherence scale
Sampling stage As MCLMC samples exhibit significant autocorrelation, we can further reduce memory costs by applying thinning without notable reduction in sample quality while also not in- creasing inference time as long as the costs of thinning remain negligible compared to the overall sampling time. In our setup, after 50k warmup steps, we propose using 10k steps for the sampling phase. The choice of thinning interval is then based on the memory and inference time constraints, ie., the posterior sample budget.
In this work, we explore two scenarios: one with budget of Ik samples per chain and another with 100 samples, corresponding to thinning intervals of 10 and 100, respectively. Overall, we propose fixed budget of 60k steps per chain, providing a predictable sampling process. By adjusting thinning appropriately, can further effectively manage memory and inference time requirements.
4 EXPERIMENTS
In this section, we evaluate the feasibility of applying MILE to sampling- inference for BNNs.
We replicate the benchmark by Sommer et al. (2024) but also extend it to other datasets (Ionosphere, Income, IMDB, MNIST, F-MNIST) and models (convolu- tional and attention-based neural networks).
Datasets and models:
Methods: As in Sommer et al. (2024), we investigate the improvement of our proposed approach over DE, but also compare against the current state-of-the-art BDE approach based on the NUTS sampler.
Following this, we conduct a series of ablation studies, carefully examining how MILEE scales compared BDE.
Runtime comparisons:
Tuning and hyperparameters: We finally validate the robustness of MILE's hyperparam- eters, supporting our claim that MILE is an auto-tuned off-the-shelf procedure like NUTS.
Details on the experimental setting and the implementation can be found in Appendix A.2 The
diagnostics and evaluation metrics employed are detailed in Appendices an3
and A.4

---

Published as a conference paper at ICLR 2025
Table 1: Average hold-out LPPD and RMSE performance well as wallclock runtime of the DE baseline, BDEs MILE, respectively,
six datasets different rows) over data splits. table including standard deviations can be found in Appendix[A.] The wallclock times the samplers represent the additional sampling time on top of the DE fit which is also reported.
LPPD (1)
RMSE ())
Time (min)
DE
BDE
MILE
DE
BDE
MILE
DE
BDE
MILE
A
0.024
0.558
0.612
0.309
0.214
0.206
0.62
2.25
0.84
B
0.390
0.625
0.645
0.251
0.242
0.236
5.67
48.29
5.40
C
-0.072
0.301
0.336
0.304
0.273
0.250
0.33
1.56
0.77
1.227
2.072
2.300
0.120
0.045
0.034
0.39
1.11
0.75
-1024
-0.760
-0.50
0.742
0.703
0.702
12.37
152.85
19.50
1.623
2.674
2.859
0.083
0.033
0.16
0.58
4.1
BENCHMARKS
The goal of this section is to demonstrate 1) the feasibility of MILE in BNN settings, 2) highlight its superior predictive performance measured using the root mean squared error (RMSE) or Aceuracy, its improved UQ measured using the LPPD, and 3) its improvements in runtime.
4.1.1 UCI BENCIMARKS
Using the six UCI datasets previously analyzed in the context of sampling- -baed inference in Som- mer et al. (2024) we replicate their benchmark using a ReLU network with two hidden layers and 16 neurons each (see Appendix A.2 for further details). We compare the predictive accuracy, UQ, and runtime of state-of-the-ar BDE and the DE baselines against MILE, configured as described in Section
Performance results Table[[show the results, indicating that MILE consistently matches or out- performs the other methods in predictive performance and UQ while significantly reducing the com- putational cost. This especially evident larger datasets like bi bikeshariin (B) and protein (P), where MILE achieves sampling speeds nearly ten times faster than BDE. In most cases, the additional MCLMC sampling after DE optimization is as fast as the DE fitting phase itself. It is noteworthy that this is big step for sampling- inference, yielding time complexity compa- rable to DE (ss xruntime), while providing better and more principled uncertainty measures.
Resource predictability As elabo- rated before, MILE's efficiency is driven by its much fewer gradient eval- uations and predictable runtime. Un- like BDE, where runtime is highly vari- able in the leapfrog steps taken by NUTS, MILE's fixed number of steps allows easy runtime forecasts once the cost of single gradient evaluation is known. This is especially crucial for expensive sampling task such as the posterior sampling of BNNs. Fig. 2 illustrates the significant variability BDE's computational cost for generat- ing 1000 posterior samples, both within and across tasks. This variability often exceeds the entire budget for sampling in MILE. Moreover, in parallel settings, the BDE's s performance is bottlenecked by the most expensive chain, while MILE's deterministic na- ture ensures perfect load balancing for concurrent sampling (see Section 3.3
airfoil
bikesharing
concrete
energy
protein
yacht
De+00
Gradlent Evaluationn per Chain for 1k Posterior Samples
BDE
MILE
Figure 2: Average gradient evaluations per chain for 1000 pos- terior samples for the experiments reported in Table

---

Published as a conference paper at ICLR 2025
Diagnostics For diagnostic purposes and to evaluate the quality of samples drawn by MILE and BDE, we evaluate the effective sample size (ESS, Vehtari et al. 2021) and the chainwise 4-split cR (Sommer et al. 2024). As can be seen in Figures 6a and 6b MILE also outperforms BDE in the sampling and local mixing quality, revealing higher average ESS and smaller cR values.
4.1.2 EXTENDED BENCHMARKS
In addition to existing sampling-based inference benchmarks, we extend our analysis to more com- plex and larger models, showcasing the successful application of MILE to convolutional neural networks (CNNs), classification tasks, and sequential models across different data modalities (see Appendix A.2 for further details). The main reason we are able advance existing benchmarks is the feasibility of sampling such tasks using MILE. As it would have taken weeks to obtain the inference using NUTS, we compare MILE mainly to the DE baseline but provide results for NUTS for smaller models in the Appendix. Additionally, we report the average performance of single DE member and individual MCMC chain, which highlights the additional benefits ensembling.
The results, presented in Table 2 not only confirm the accuracy and UQ improvements over the DE baseline but also demonstrate performance gains in much larger models and new prob- lem domains, confirming the positive effect of the additional exploration step taken by MILE.
Results
Table Hold out test performance of MILE baselines on various classification tasks using fully celly-connected (FCN), convolutional (CNN) and attention-hen haased (ATT) networks. ATT (v2, v3) employ pretrained embed- dings.
Dataset
Model
Params
Accuracy (f)
LPPD (f)
Avg. Single
Ensemble
Avg. Single
Ensemble
DNN
Chain
DE
MILE
DNN
Chain
DE
MILE
lonosphere
FCN (vl)
850
0.930
0.958
0.958
0.958
-0,404
-0.168
-0.309
-0.167
Income
FCN (v2)
2386
0.843
0.851
0.846
0.851
0.334
00.115
-0.318
-0.313
ATT (vl)
-0,550
-0.546
IMDB
ATT (v2)
0.779
0.781
0.782
-0.591
-0.583
-0,562
-0.481
IMDB
ATT (v3)
106190
0.786
0.790
0.786
0.788
-0.509
-0.493
-0.507
-0.491
MNIST
CNN (v1)
7452
0.916
0.939
0.956
0.970
-0,299
-0.209
-0.179
-0.129
F-MNIST
CNN (v1)
7452
0.742
0.767
0,863
0.885
-0,725
-0.684
-0.486
-0.430
F-MNIST
CNN (v2)
61706
0.890
0,919
0.918
0.925
-0,361
-0.225
-0.227
-0.216
4.2
ABLATION MODEL COMPLEXITY & RUNTIME
The UCI benchmarks have clearly showed the runtime advantage of MILE over BDE. To further in- vestigate the impact of model complexity (number of parameters) and dataset size on these runtimes, we conduct two ablation studies.
Scaling in model complexity F shows the evolution of the required sampling time and perfor- mance metrics for increased model complexity. For the 5,426-dimensional case, MILE completes sampling in under 30 minutes, while BDE takes several hours. More strikingly. MILE not only remains faster but its performance gap to BDE increases with higher model complexity. indicating both significant efficiency gains and superior performance in higher dimensions. To quantify the time complexity, we fit a linearized power-law model to obtain the expected sampling time with :o 31 log(d). Both fits explain almost all the variance in the data with 1?2 statistics of 0.98 for MILE and 0.96 for BDE, confirming the robustness of the speed ad- vantage. The model fits are illustrated as dashed lines in Fig. B
growing d: E[loe((sampter]]
Scaling in dataset size We analyze the impact of increasing dataset size. Fig. 3 (top right) illus- trates how sampling time evolves for a fixed task and neural network across different data subsets. Using the linear model [[[tampter] Bg + 3,n?. the given data points, with R²2 values of 0.99 for MILE and 0.96 for BDE. The ratio of the B, coeffi- cients, 6.6, again highlights MILE's superior scaling in 2 Notably, visual inspection suggests that BDE may scale worse than guadratically, but for consistency with MILE, we apply conservative quadratic fit. The scaling behavior again closely resembles a power-law and the observed quadratic trend in n likely stems from meloee-related hardware slowdowns dataset sizes increase.
obtain fits that again almost perfectly resemble
8

---

Published as a conference paper at ICLR 2025
Sampling Time Scaling Parameters
Sampling Time Scaling Observations
200
Sampling Time (man)
Samsil Time (9)
10
150
100
5
50
266
786
2594
5426
10000
20000
30000
40000
Number of Parameters
Number of O bservations
LPPD
RMSE
0.275
0.8
0.250
0.225
0.4
0.200
3x8 (266)
3x15 (786) 3x32 (2594) 3x48 (5426)
3x8 (266)
3x16 (786) 3x32 (2594) 3:48 (5426)
Hidden Layers Neurons (P (Parameters)
MILE
BDE
[
DE
Figure 3: Average sampling wallclock times (minutes, y-axis) of BDE (blue) and MILE (orange) for the bikesharing dataset across 4 NN architectures with increasing parameter count (x-axis) on the upper left. Average sampling wallclock times (hours. y-axis) for the protein dataset across varying training data sizes (x-axis) on the upper right. Dashed lines indicate power-law tively. In both cases the sampling time ratio between BDE and MILE is around 7-9, independently of the number of parameters and observations. This is result of NUTS always being close to its maximum number of iterations per sample, which we to the default value of 1024 gradient calls. It therefore uses around 1024 (1000 + 100) 11 10 gradient calls, displayed also in Figure 2 MILE on the other hand always uses 2 60000 = 12 x 10 calls, which gives a ratio of 9.2. The bottom row shows hold-out metric performances across network architectures. DE performance for the LPPD and RMSE metrics is indicated as grey reference. All charts come with standard errors over 3 data splits.
quadratic model fits respec-
4.3 ABLATION HYPERPARAMETER ROBUSTNESS
To substantiate the claim that our proposed MCLMC configuration in MILE is, in fact, robust and auto-tuned off- shels method, conducted a series of ablation studies by changing the default hyperparameters of MILE as suggested in Section and examine the impact of these changes on per- formance. For each ablation, we systematically evaluate the robustness of the approach by changing a single hyperparameter on an appropriate grid. We report both the impact of hyperparameter varia- tions on hold-out test performance metrics and the corresponding tuned values of the sampler's key parameters- the momentum decorrelation scale (L) and step size (e). While the effect on the latter ones is interesting the primary focus attain estimates of the two parameters that yield robust and good performance. For comparison, we also present the performance of BDE and the DE baseline.
The results are summarized in Fig. Across all cases, MILE consistently shows robust performance, with minimal sensitivity to changes in the hyperparameters. The changes in the major parameters J. and are generally small and with such marginal changes -especially to the important 6- do not significantly affect the overall performance. An exception the warmup budget, where slightly improved performance can be observed with a significantly extended warmup phase. How- ever, the improvements are marginal and come at a linear increase in runtime, making the additional computational cost unjustifiable.
Results
In conclusion, these results support the claim that MILE is effectively tuning-free, as in the con- sidered cases, the method's performance does not change drastically with different hyperparameters and is close to optimal under the default settings. This robustness ensures that practitioners can confidently apply the method without extensive tuning, further enhancing its practical utility.

---

Published as a conference paper at ICLR 2025
0.8
0.7
RMSE
0.23
0.7
RMSE
0.23
0.6
0.6
0.5
0.21
0.21
5
0.016
7.5
Stäp Size
0.008
5.0
0.007
0.008
2.5
0.006
0.004
10k
50k
100k
150k
10k
50k
100k
150k
200k
300
Warmup udgst/uudgnt
Energy Variance Schedule (From -To]
avg. DE (dottd)
avg. DE (dstte) avg. BDE (cashed)
avg. (dasheo)
(a) #Tuning steps/budget for the warmup.
(b) Energy variance scheduler ranges.
0.7
0.23
0.7
0.23
LPPD
RASE
LPPD
RMSE
0.22
0.5
0.5
0.009
0.008
3.0
Slap Size
0.008
2.75
Step Size
0.007
2.5
2:50
0.006
2.25
0.006
2.0
0.005
10
50
100
150
200
10
50
100
150
200
0.5
1.5
2.5
0.5
1.5
2
2.5
Effective Samples
Trust in Estimate
d0E DE (doned)
DE (datte) avg. BUE (dashed)
ovg. (cashed)
(c) #Effective samples to estimate EEVPD
(d) Trust in the estimate parameter.
Figure 4: Results the ablation studies conducted on the bi bikeshariin ng dataset for robustness of the MILE algorithm to its tuning parameters (x-axes, proposed defaults bold). Both the average hold-out RMSE and LPPD reported with their standard error for data splits. The samc holds for the major parameters of the sampling kernel L and the step size that estimate EEVPD" and "Trust in estimate" are minor parameters of the step size adaptation algorithm which determine the sample weighting during the EEVPD computation.
tuned by the proposed tuning. "#Effective samples
5 DISCUSSION
In this work, we proposed and evaluated Microcanonical Langevin Ensembles (MILE) for sampling- based inference Bayesian neural network posteriors. By adapting the recently proposed MCLMC sampler and combining it with starting values obtained via Deep Ensembles, our comparative analy- sis reveals substantial advancements in performance, sample quality, UQ, and runtime compared to KUTSS-bssel alternative. Furthermore, the method enhances the predictability of resource require- ments due to its deterministic number of gradient evaluations, which also simplifies parallelization. In conclusion, our proposed method can be considered reliable and efficient off-the-shelf method and thus a big step forward toward making sampling-base bbaeed inference feasible for generic BNNs.
Scope of this work and limitations While also yielding significant runtime savings for increased dataset dimensions, the goal of this work was overcome the most prevalent bottlenecks of NUTS- based ensembling- unfavorable scaling with respect to the number of parameters and unfore- secable resource allocation due its variable and high number gradient evaluations. As MCLMC and our modifications seem have solved these problems, next step for future work that did not compare this study is the transition to Stochastic Gradient sampler variants. This transition is typically not straightforward but would overcome another remaining limitation of sampling based inference, namely the scaling for large-scale datasets. Another possible enhancement MILE we did not investigate in this work is the use of alternative priors, e.g., discussed in Fortuin al. 2022
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Christophe Andrieu, Nando de Freitas, Arnaud Doucet, and Michael I. Jordan. An Introduction to MCMC for Machine Learning. Machine Learning. 50:5 -1:5-43,200 2003.
Michael Betancourt. A Conceptual Introduction to Hamiltonian Monte Carlo, 2018.
Ronald Newbold Bracewell and Ronald N Bracewell. The Fourier transform and its applications, volume 31999. McGraw-Hill New York, 1986.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milln and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github.co com gttp:///oolll//a//////// ///
Alberto Cabezas, Adrien Corenflos, Junpeng Lao, and Rémi Louf. Blackjax: Composable Bayesian inference JAX, 2024.
Adam D. Cobb and Brian Jalaian. Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting. In Proceedings of the Thirty-Sevetth Conference on Uncer- tainty in Artificial Intelligence, 2021.
Miles Cranmer, Daniel Tamayo, Hanno Rein, Peter Battaglia, Samuel Hadden, Philip Armitage, Shirley Ho, and David N. Spergel. A bayesian neural network predicts the dissolution of compact planetary systems. Proceedings of the National Academy of Sciences, 1 118(40)::222603311 2021.
Daniel Dold, David Rügamer, Beate Sick, and Oliver Dürr. Semi-structured subspace inference. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, Pro- ceedings of Machine Learning Research. PMLR, 2024.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive ics www.cci.eeee... edumml
Simon Duane, Anthony D) Kennedy, Brian J Pendleton, and Duncan Roweth. Hybrid monte carlo. Physics letters 195(2):216-22,, 1987.
Alain Oliviero Durmus and Andreas Hberle. Asymptotic bias of inexact Markov Chain Monte Carlo methods in high dimension, April 2023. arXiv:2108.0068) [es, math, stat].
Mingzhou Fan, Ruida Zhou, Chao Tian, and Xiaoning Qian. Path-guided particle- based sampling. In Proceedings of the 41st International Conference Machine Learning, volume 235 of Pro- ceedings of Machine Learning Research, pp. 12916-12934. PMLR, 21-27 Jul 2024.
Hadi Fanace-T. Bike Sharing Dataset. UCI Machine Learning Repository, 2013.
Vincent Fortuin, Adrià Garriga-Alonso, Sebastian W. Ober, Florian Wenzel, Gunnar Ratsch, Richard E Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network pri- ors revisited. In International Conference on Learning Representations, 2022.
Philip Gage. A new algorithm for data compression. C Users J., 12(2):23-38, feb 1994.
A. Gelman, Hwang, and A. Vehtari. Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6):977-0066, 2014.
Andrew Gelman, John B Carlin, Hal S Stem,, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian Data Analysis Third Edition (with Errors Fixed as of 15 February 2021). Published online, 2013.
Louis Grenioux, Maxence Noble, Marylou Gabrié, and Alain Oliviero Durmus. Stochastic local- ization via iterative posterior sampling. In Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 16337-16376. PMLR, 21-27 Jul 2024.
11

---

Published as a conference paper at ICLR 2025
Matthew D Hoffman and Andrew Gelman. The No-U-Tum Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research, 15:1351-1323 2014.
Matthew D Hoffman and Pavel Sountsov. Tuning-free free generalized hamiltonian monte carlo. In International conference artificial intelligence and statistics, pp. 7799-7813. PMLR, 2022.
Pavel Izmailov, Wesley Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and An- drew Gordon Wilson. Subspace Inference for Bayesian Deep Learning. In Proceedings of the Conference on Uncertainty in Artificial Intelligence, pp. 1169-1179, 2020.
Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, and Andrew Gordon Wilson. What Are Bayesian Neural Network Posteriors Really Like? In Proceedings of the 38th International Conference Machine Learning, PMIR 139,, 2021.
Minas Karamanis, Florian Beutler, John A. Peacock, David Nabergoj, and Uros Seljak. pocoMC: Preconditioned Monte Carlo method for accelerated Bayesian inference. Astrophysies Source Code Library, pp. ascl:2207.018, July 2022a. ADS Bibcode: 2022ascl.
Minas Karamanis, Florian Beutler, John Peacock, David Nabergoj, and Uroš Seljak. Accelcrating astronomical and cosmological inference with preconditioned Monte Carlo. Monthly Notices of the Royal Astronomical Society, 516(2):1644-1666 September 2022b.
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers; a decision-tree hybrid. In Proceed- ings of the Second International Conference on Knowledge Discovery and Data Mining, KDD'96, pp. 202-207. AAAI Press, 1996.
Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In ICML, volume 80 of Proceedings of Machine Learning Research, pp. 2801-2800 PMLR, 2018.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In Proceedings of the 31st Conference on Neural Information Processing Systems, 2017.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- based learning applied to document recog- nition. Proceedings of the IEEE, 86(1 86(11)2228-23244 1998. doi: 10.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. UtLhttp:///yn
lecun wwtp///wbb//mmntttc//////
Benedict Leimkuhler and Sebastian Reich. A Metropolis adjusted Nosé-Hoover thermostat. ESAIM: Mathematical Modelling and Numerical Analysis, 43(4):743-755, July 2009.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
Andrew L. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 201 1. Association for Computational Linguistics.
lain Murray, Ryan Adams, and David MacKay. Elliptical slice sampling. In Proceedings of the Thir- teenth International Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pp. 541-548, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
Radford M. Neal. MCMC Using Hamiltonian Dynamics. Chapman Hall / CRC Press,, 2011.
Igor Omelyan and Andriy Kovalenko. Generalised camonical-isokinete ensemble: speeding up multiscale molecular dynamics and coupling with 3D molecular theory of solvation. Molecular Simulation, 39(1):25-5))))) January 2013.
12

---

Published as a conference paper at ICLR 2025
I Ortigosa, R Lopez, and J Garcia. A neural networks approach to residuary resistance of sail- ing yachts prediction. In Proceedings of the International Conference on Marine Engineering (MARINE), volume 2007, pp. 250, 2007.
Theodore Papamarkou, Jacob Hinkle, M. Todd Young, and David Womble. Challenges in Markov Chain Monte Carlo for Bayesian Neural Networks. Statistical Science, 37(3), 2022.
Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, José Miguel Hernnndez- Lobato, Aliaksandr Hubin, Alexander Immer, Theofanis Karaletsos, Mohammad Emtiyaz. Khan, Agustinus Kristiadi, Yingzhen Li, Stephan Mandt, Christopher Nemeth, Michael A Osborne, Tim G. J. Rudner, David Rügamer, Yee Whye Teh, Max Welling, Andrew Gordon Wilson, and Ruqi Zhang. Position:
Bayesian deep learning is needed in the age of large-scale AI. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 19956-39556 PMLR, 21-27 Jul 2024.
Weiwen Peng, Zhi-Sheng Ye, and Nan Chen. Bayesian deep-learning len-eein-nn--eeedd based health prognostics toward prognostics uncertainty. IEEE Transactions on Industrial Electronics, 67(33)2223-222 2020. doi: 2T1.1109TTE.221992920
Lionel Riou-Durand, Pavel Sountsov, Jure Vogrine, and Charles C Margossian. Adaptive Tuning for Metropolis Adjusted Langevin Trajectories. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.
Jakob Robnik and Uros Seljak. Fluctuation without dissipation: Microcanonical langevin monte carlo. In Symposium on Advances in Approximate Bayesian Inference, pp. 111-126. PMLR, 2024.
Jakob Robnik, Bruno De Luca, Eva Silverstein, and Uroš Seljak. Microcanonical hamiltonian monte carlo. The Journal of Machine Learning Research, 24()):1469-11429,, 2023.
Jakob Robnik, G. Bruno De Luca, Eva Silverstein, and Uroš Seljak. Microcanonical Hamiltonian Monte Carlo. The Journal of Machine Leurning Research, 240):3111146909-111117 March 2024. ISSN 1532-4435.
Vincent Sigillito, Scott Wing, Lisa Hutton, and K. Baker. Ionosphere. Repository, 1989. URL htttps://dob /rtp:::://oi..o//1.... 2ttgg/1..243333 2ttg//00024432/500000
UCI Machine Learning
John Skilling. Nested sampling. Bayesian inference and maximum entropy methods in science and engineering, 735:395-400 2004.
Emanuel Sommer, Lisa Wimmer, Theodore Papamarkou, Ludwig Bothmann, Bernd Bischl, and David Rügamer. Connecting the dots: Is modeeccnneeteddees the key to feasible sample-based
inference in bayesian neural networks? In Proceedings of the 41st International Conference Machine Learning. PMLR, 2024.
Pavel Sountsov and Matt D. Hoffman. Focusing on Difficult Directions for Learning HMC Trajec- tory Lengths, May 2022. arXiv:2110.11576 [stat].
Tetsuya Takaishi and Philippe de Forcrand. Testing and tuning symplectic integrators for Hybrid Monte Carlo algorithm in lattice QCD. Physical Review E, 73(3):036706, March 2006.
Yifeng Tian, Nishant Panda, and Yen Ting Lin. Liouville flow importance sampler. In Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 48 48186-48210. PMLR, 21-27 Jul 2024.
Athanasios Tsanas and Angeliki Xifara. Accurate quantitative estimation of energy performance of residential buildings using statistical machine learning tools. Energy and Buildings, 49:560-567, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31 Inter- national Conference on Neural Information Processing Systems, NIPS' 17, pp. 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
13

---

Published as a conference paper at ICLR 2025
Aki Vehtari, Andrew Gelman, Daniel Simpson, Bob Carpenter, and Paul-Christian Bürkner. Rank- normalization, folding, and localization: an improved r for assessing convergence of mcme (with discussion). Bayesian Analysis, 16(2):667-718, 2021.
Jonas Gregor Wiese, Lisa Wimmer, Theodore Papamarkou, Bernd Bischl, Stephan Günnemann, and David Rügamer. Towards efficient meme sampling in bayesian neural networks by exploiting symmetry. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 459-474. Springer, 2023.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Bench- marking Machine Learning Algorithms, 2017.
I-C Yeh. Modeling of strength of high-performance concrete using artificial neural networks. Ce- ment and Concrete research, 28(12), 17:1777188888. 1998.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang. Changyou Chen, and Andrew Gordon Wilson. Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning. In Proceedings of the Eighth Interna- tional Conference on Learning Representations, 2020.
Erik Strumbelj, Alexandre Bouchard-Côté, Jukka Corander, Andrew Gelman, Håvard Rue, Lawrence Murray, Henri Pesonen, Martyn Plummer, and Aki Vehtari. Past, Present and Future of Software for Bayesian Inference. Statistical Science, 39(1):46 61, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
FURTHER RESULTS
A.1.1 BENCHMARKS
UCI benchmark Table
The results for the UCI benchmark including standard deviations are given in
Table 3: Average hold-out and standard error of LPPD and RMSF performance as well as wallclock of DE baseline, BDEs and MII E. for the six datasets used in Sommer et al. (2024) over 3 data splits. The wallclock times of the samplers represent the additional sampling time of the DE fit which is also reported.
LPPD (?)
RMSE
Time (min) BDE
DE
BDE
MILE
DE
EDE
MILE
DE (VMethods)
MILE
A
B.221 = 0021
0,558 0.034
0.812 + 0.011
0.509 +: 0.005
0.2114 =: 0.015
0.206 s: 0.015
0.02 = 0.15
2:25 +: 0.01
0.54 0.06
B
0.300 = 0.10
0,020 0.004
0.645 0.10
0,201 à: 0.004
0.244 =: 0.005
0.236
0.004
0,07 0.98
65.29 0.51
5.40 0.02
-0007 0.088
0,301
0.088
0.330 1: 0.054
0.301 0:307
0.2775 0.012
0.250 0.014
0.53 0.05
1.56 + 0.05
0.77 0.06
1:227
0037
0.036
0.066
0.30
=10034 0.020
-0.760 0.010
-0.750 0.018
0.742 0.011
0.006
12:37 1.00
152.85
0.01
1,933 0.101
2.874
0,316
2,850 1: 0.199
0.081
0:038
0.083 0.011
0.0533 0.011
0.10 0:01
0.58 :: 0.00
0.01 000
Chain variances Our analyses of between- and within-chain variances (Fig. 6 show distinctive pattern an increasing within-chain variance in layers further away from the input and output layer. Contrasting this with the work by SSmmmrr et al. (2024). this suggests that MILE also exhibits most disconnected modes in the first and last layers. This analysis can help to assess whether sampling the multimodal posterior surface of BNNs is an infeasible problem due to the combinatorial explosion of modes with an increased depth of the network (which is not the case).
Calibration We also compute calibration errors (see Definition 2) and analyze coverage for cred- ible intervals across various nominal coverage levels for the UCI benchmarks. Table [0) and Fig. 7 show that MILE achieves calibration quality comparable to the of BDE, confirming its effec- tiveness in uncertainty quantification.
Classification benchmark For a comparison with BDE, we ran the smaller tabular classification tasks both using BDE and MILE. For the larger experiments (both considerably larger in dataset and model complexity). BDE would require weeks to run and is thus omitted. The results of the comparative study are given in Table 4 and suggest on-par performance of MILE with BDE and clearly superior performance to the DE bascline.
Table Hold-out test performance of BDE, MILE and baselines on the two tabular classification tasks.
Dataset
Accuracy (!)
LPPD
Single
Eusemble
Single
Ensemble
Chain (RDE)
Chain (MIL.)
BDE
MILKK
Chain (RDE)
Chain (MILK))
MILEE
Jonosphere
Incomne
0.843
0,800
0,851
0.846
0.851
0.851
.0334
.0.315
-0.515
.0318
-0,311
40.13
A.1.2 ROBUSTNESS AND NUMERICAL STABILITY IN HIGH DIMENSIONS
Without our adjustments. MCLMC struggles with exploration and often fails to produce meaningful samples, especially in nigh-dimensionl settings. To highlight this, we conduct an ablation study, assessing failure rates when applying MCLMC to BNNs without our proposed adjustments. Specifi- cally, we use the same models as in Table and run 100 chains each of MILE and naïve MCLMC on various datasets with differing parameter dimensions and with the same DE initialization, recording the percentage of chains that resulted in numerical issues (e.g.. NaN values rendering all samples unusable). The results, given in Table 5 demonstrate the critical importance of our adjustments: MCLMC exhibits failure rates between 78% and 86%, while MILE consistently shows 0% failures across all datasets.
15

---

Published as a conference paper at ICLR 2025
Table 5: Failure rates (NaN chains) for naive MCLMC and MILE across different datasets.
Dataset
MCLMC (% NaN Chains)
MILE (% NaN Chains)
Airfoil
86%
0%
Concrete
80%
0%
Energy
78%
0%
Yacht
85%
0%
1.1.3 FURTHER COMPARISONS
Comparison with Path-Guided Particle-based Sampling We conduct an empirical comparison with the recently proposed ing the experimental setup of Section 5.2.1 Fa et al. (2024), we conduct BNN inference on 7 UCI classification datasets (Dua & Graff 2017) and report the average negative log- eiglieeli (NLL) and accuracy. contaiis the results that showcase clear pattern. MILE performs at least as good in terms accuracy as PGPS and is clearly superior in the NLL most cases. Notably, running MILE for all experiments and replications takes less than 5 minutes in under 1 minute. We compared PGPS and MILE in terms of runtime on the same hardware for the Sonar dataset across five independent runs as example. While MILE achieved a runtime of 0.94+ +0.06 minutes PGPS required 24.34 + 0.64 minutes. The major factor for the runtime gap is the nested computation detailed in PGPS (Algorithm 3). For example, the authors chose 100k overall steps each with 100 optimization steps and 300 Langevin adjustments for the UCI benchmark. This incurs high computational costs, even without considering the additional overhead of the tuning of the PGPS hyperparameters a and B.
Particle-based Sampling (PGPS, Fan et 2024). Follow-
consumer CPU, with many
Table 6: Hold-out test performance of Path-Guided Particle-based Sampling and MILE on the UCI classifica- tion tasks of Table and 4 of Fan et al. 2024) over five independent runs.
Dataset
# Classes
# Rows
NLL (L)
Accuracy (f)
PGPS
MILE
PGPS
MILE
SONAR
2
207
0.536 + 0.014
0.979 = 0.094
0.798 + 0.023
0.779 = 0.047
WINEWHITE
7
4898
1.979 0.009
1.110
0.010
0.565 = 0.008
WINERED
6
1599
1.964 0.012
0.007
0.594 + 0.018
0.604 0.019
AUSTRALIAN
2
689
0.5042 + 0.013
0.486:0.000
0.862 + 0.009
0.852 = 0.015
HEART
5
302
0.943 1 0.030
1.070
0.256 1 0.142
0.591 0.033
GLASS
6
213
1.685 1 0.030
1.160 = 0.083
0.585 1 0.080
0.643 0.063
COVERTYPE
7
8000
1.602 0.014
0.717 0.024
0.590 1 0.095
0.746 0.006
Comparison with Symmetric Split HMC We also conduct an empirical comparison with Sym- metric Split HMC (Sym-Spli-HHMC, Cobb & Jalaian| 2021). Symmetric Split HMC advances HMC but inherits the same hyperparameter sensitivity (e.g., depends on trajectory length and step size). These hyperparameters can limit the application in Bayesian neural network inference. In Cobb & Jalaian| (021). the authors use Bayesian Optimization (BO) to derive hyperparameters which intro- duces further complexity and significant computational burden. Unlike MILE, Symmetric Split HMC employs an MH correction step, which further increases the computational costs. Another downside Cobb & Jalaian (202I) that with an increased number of batches, the computational requirements increase notably. Both approaches have merit, but their main goal and contribution differ considerably. Symmetric Split HMC focuses on memory scalability, while MILE optimizes speed and performance. Nevertheless, an empirical comparison is interesting.
We replicate the multi-class classification task for the Fashion MNIIT dataset from Table 2 with the CNN (v2) model using SyM-Split-HMC. We use the optimized hyperparameters reported in Cobb & Jalaian (02T), Section 5.3,, for the same dataset and task. We conduct all experiments on the same hardware to ensure comparability of runtimes and report the results in Table 7 For a fixed amount of posterior samples, the performance of Symmetric Split HMC benefits from smaller batch sizes. However, as noted above and confirmed empirically, runtime increases notably for smaller batch sizes. We choose a batch size of 64 and run Symmetric Split HMC for 200 samples, requiring 15.5 hours. The intended goal of sampling 1000 samples (as with MILE) would take more than 3 days with this setting. For larger batches of 1024 images, we generate up to 3000 posterior samples for Symmetric Split HMC, but without a notable gain in performance. Regardless of the specification, it
16

---

Published as a conference paper at ICLR 2025
becomes clear that MILE achieves considerably better performance in a fraction of the required time for Symmetric Split HMC (without considering the cost running BO for hyperparameter tuning).
Table 7: Comparison of off-the-shelf MILE with Symmetric Split HMC on the Fashion-MND task using the CNNv2 model. Total Time does not consider the necessary BO step of Symmetric Split HMC.
Method
Accuracy (t)
LPPD (f)
Post. Samples
Total Time
MILB
0.925
-0.216
1000
Ih 21min
Sym-Split-HMC (Batch size: 64)
0.818
-0.548
200 (+50 burn-in)
15h 29min
Sym-Split-HNy HMC (Batch size: 1024)
0.820
-0.525
1000 (+200 burn-in)
6min
SyM-Spli-HHMC (Batch size: 1024)
0.813
-0.513
3000 (+300 burn-in)
18h 47min
A.2 EXPERIMENTAL SETUP AND FURTHER DETAILS
Software Our software is implemented in Python and mainly relies on the jax (Bradbury al. 2018) and BlackJAX (Cabezas et al. (024) libraries. We further use Docker for reproducible
experimental setup. Our code is available at https::// thu.. com/ Emanuel Sommer
MILE
Compute environment The experiments were run on two NVIDIA RTX A6000 GPUs and an AMD TM ThreadripperTM PRO 5000WX/3000WX CPU with 64 cores. Sampling 12 chains for most experiments allowed to parallelize the sampling on CPU such that multiple experiments can be run at the same time.
Benchmark data Table 8 gives overview of the data, and associated tasks and provides all references.
Table 8: Overview of the used datasets with task description and references.
ORS.
REFERENCE
A
AIRFOIL
REGRESSION
1503
5
DUA GRAFF
2017
B
BIKESHARING
REGRESSION
17379
13
FANALE-T
2013
8
E
ENERGY
REGRESSION
768
8
TSANAS & XIFARA
2012
P
PROTEIN
REGRESSION
45730
DVA GRAF
2017
Y
YACHI
REGRESSION
308
ORTIGOSA
2007
JONOSPHERE
BINARY-CLASE
351
34
SIGILLITO AL.
1989
BINARY CLASS.
48842
KOBAVI
IMDB
BINARY-CLANE
50000
TEXT
MAAS ET AL.
2011
MNIST
MULTI- CLASS.
60000
28x28
LECON & CORTES
2010
FISSIION)-MNIST
MULTI-CLASS. 60000
28x28
Optimization & sampling For all DE optimizations, we use ADAM with decoupled weight decay (Loshchilov & Hutter 2019) and use the negative log-likelihood loss as objective. We employ early stopping on validation set and use 70% train, 10% validation and 20% test split there is no predefined test set as for the MNIST and Fashion MNIST dataset. If not specified otherwise 12 DE members and 12 chains. For all NUTS based experiments, we use a burn-in of 100 samples and collect 1000 posterior samples with target acceptance rate of 0.8. Also, employ an isotropic standard Gaussian prior if not specified otherwise.
use
We do not adjust the effective number of samples in the MCLMC tuning even if we apply consid- erable amount of thinning, i.e., for 10000 samples with a thinning interval of 10, resulting in 1000 ESS of 100. However, for less than 1000 final samples, hold ESS fixed at 100 as lower bound. We validated the robustness of this choice by various experiments and ablation studies discussed in Section
final samples,
still
Regression tasks jet al. (202T); Sommer et al. (2024). That means, we parameterize the Gaussian likelihood with by the output neurons as location and log-scale. For the experiments aggregated in Table Table T0
We train distributional regression models for all regression tasks just as Izmailov
17

---

Published as a conference paper at ICLR 2025
and Fig. 2 configurations in Sommer et al. (2024), in particular, fully conneccon neural network that has two hidden layers with 16 neurons each.
Ablation studies For the ablation studies, we use the larger UCI benchmark datasets slightly larger networks than
bikesharing and protein. If not specified otherwise, before by considering three hidden layers of 16 neurons each. In order to analyze the behavior of the samplers, we further implement slim and deeper network with 6 hidden layers of just neurons each. The corresponding experimental results are reported in Fig. 6
Classification tasks For the classification tasks, we follow the classical way of directly parame- terizing the categorical distribution with as many output neurons datasets ionosphere and income, we use simple feed-forward neural networks with 2 (v1) and 4 (v2) hidden layers with 16 neurons each. We also consider new data modalities for the BDE, namely images and text. The corresponding architectures are described in Table [ Moreover, we use 10 chains for CNNs, 8 chains for the sequential models v1-2, and chains the sequential model v3 for these larger networks. We save only 100 samples per chain to be more memory ef- ficient and are thus able to showease improvement even for smaller overall amount of posterior samples. This is realized via thinning in both BDE and MILE.
have classes. For the tabular
Convolutional neural networks As convolutional neural network (CNN) architectures we choose a LeNet-5 (Lecun et al. 1998) architecture (CNNv2) and also consider a slightly smaller yet similar architecture (CNNv1). The architectures are described in detail in Table
Table 9: CNN architectures.
CNNv2
CNNv1 1 filter, 3x3 kernel, padding 2, ReLU
Conv
filters, 5x5 kernel, padding 2, ReLU
Pooling
2x2 Avg Pooling, stride 2 16 filters, 5x5 kernel, no padding, ReLU
Conv
Pooling
2x2 Avg Pooling, stride
FC
120 units, ReLU
8 units, ReLU 8 units. ReLU units, ReLU Output units
FC
84 units, ReLU
FC
Output units
FC
Sequential networks Fig. provides a schematic overview of the attention-based sequential model architecture. We explore two main configurations: one where all model parameters, including token and positional embeddings, ding (v2,v3). Both models use context length of 70 tokens, with padding or truncation for shorter longer sequences. We trained custom tokenizer with Byte-Pair Encoding (BPE, (aag 1994), targeting vocabulary sizes of 1K and 10k tokens for v1 and v2-3, respectively. To balance model complexity, token embeddings were set to 48 dimensions for the fully sampled model vl and 192 for the pretrained versions v2-3. Positional encodings are added before passing through an 8-head attention mechanism, with 64-dimensional query, key, and value vectors (Vaswani et al., 2017) for v1-2. For v3, we use 10-head attention mechanism with 100-dimensional query, key and value vectors. After average pooling, a feed-forward network with one hidden layer (64 neurons for the full model v1, 32 for the pretrained version v2) or two hidden layers for v3 with 128 and 32 neurons output the logits.
sampled (v1), and another using fixed, pretrained embed-
Prior indueed regularization As the prior acts as a regularizer during the sampling phase, we might exhibit performance degradation for larger classification models if the prior variance is cho- sen inappropriately small. For the larger CNN and ATT models, we therefore choose the standard isotropic Gaussians (0, 0.1I) AV 0.2I) (ATTv1,v2) and (0, 0.47) (ATTv3). While a dedicated study on the influence of priors within this framework is out of scope for this work, we think further tuning the prior variance changing the prior distribution could be promising.
18

---

Published as a conference paper at ICLR 2025
Text Input
Tokenization
Positional Encoding
Embedding
Multi-Head Attention
Fully Connected
Logits
Figure 5: Schematic overview the sequential attention-basee basod model architecture (ATT) that is applied to the IMDB Dataset.
A.3 DIAGNOSTICS
We report the BNN-specific diagnosties proposed in Sommer et al. (2024) Fig. The displayed chain variances are discussed in Appendix(A.II.
Effective sample size We observe very similar effective sample size (ESS) values for BDE as reported Izmailov et al. (2021): Sommer et al.] (2024). In most cases, MILE is on a same or slightly higher level of ESS. However, especially for weights that are close the input and output, we observe a much higher ESS than for BDE. For the airfoil dataset, this ESS increase also given for deeper layers.
Convergence of MILE and chainwise mixing Based on the results of Robnik et al. (2024) know that MILE will provide the same convergence guarantees as long as the initialization is done randomly or its effect becomes negligible as oo and the discretization As our work's 's focus is on empirical efficiency rather than guaranteed convergence, we a) do not use adjasjustmnnt but control discretization eтror using the EEVPD do not run chains for large number of steps, and c) start chains using deep ensemble initializations. This is a compromise that will induce bias the sampling distribution but ensures a more stable behavior during sampling, and, in turn, increases the ESS.
is MHHaajjssted.
To measure chainwise mixing, we proxy local chainwise convergence using the chainwise split metric eR with a split factor of 4. Our results demonstrate that MILE clearly improves chainwise mixing. However, all values remain notably higher than the conventional cutoff thresholds of 1.1 and 1.01 (Vehtari et al. 2021)
A.4 EVALUATION
Predictive performance Following Gelman al. (2014); Wiese et al. (2023) and Sommer et al. 2024). we choose the log posterior predictive density (LPPD) over test set D Prest defined as
LPPD 1
Σ
log
Σ k=
p p (0th))
(4)
Threst (v*,0")EDe..
K
in order to quantify the quality of the PPD approximation and UQ in general. Intuitively, the LPPD measures the average extent to which the predictive distribution accurately covers the observed labels.
Additionally, we use the root mean squared error (RMSE) for regression tasks and accuracy (ACC) for classification tasks assess point predictions. While LPPD evaluates the overall fit of the pre- dictive distribution, RMSE and ACC provide specific metrics for the accuracy of point predictions in their respective domains.
19

---

Published as a conference paper at ICLR 2025
airfoll
airfo
20
1.5
1.4
1.3
8
1.2
bikesharing
bikesharing
Eflective Sample
dhainwise
1.6
1.4
10
1.2
energy
energy
0
2
3
Hidden Layer
Hidden Layer
biases
weights
MILE
blases
weights
BDE
MILE
(a) Effective Sample Size.
(b) Chainwise mixing
measured by cR
airfoll
airfoll
1.0
Belveen Chain Variance
Within Chain Veriance
birinharing
bikesharing
2.0
0.20
0.15
1.5
0.10
0.05
0.00
energy
energy
1.5
0.4
1.0
0.2
0.5
0.1
0.0
2
7
Hidden Layer
Hidden Layer
blases
weights
BDE
MILE
blases
weights
BDE
MILE
Between Chain Variances.
(d) Within Chain Variances.
Figure 6: Different sampling diagnostics of a seven-layer BNN for three UCI benchmark datascts (in diffcrent rows) separated by layer (x-axis) over three data splits.
Calibration Following Kuleshov et al. (2018), we define calibration and the empirical (squared) calibration error in the regression setting. Intuitively expects samples from the true PPD to be contained in the Credibility Intervals (CIs) with the coverage probability of the CI. The following definition formalizes this.
Definition (Calibration) For some realized labeled dataset D = (ax, DD) E X x IR of random variables X, Y, we define a credible interval C1-a(xx, D) to be calibrated at level 1 a E (0, iff y. p(: x., D) it holds that
Ply E CI-a D)) a.
(5)
If y, CI-a(E,, we say that the CI C1-a(x.; covers Vs. Thus calibrated models have correct coverage probabilities.
This straightforwardly leads to the definition of the calibration error.
Definition (Calibration error) We define the empirical weighted calibration error (CalE) over
the hold-out validation data D. as the root mean squared difference of nominal 1 04 and
20

---

Published as a conference paper at ICLR 2025
Table 10: Mean Calibration Error for the DE baseline, BDE and MILE for six datasets. The nominal coverage levels used are 0.5, 0.75, and 0.95. The experimental setup identical to one Table !
Calibration Error (L)
Dataset
DE
BDE
MILE
Airfoil
0.077 = 0.012
0.083 + 0.009
0.086 = 0.018
Bikesharing
0.125 0.005
0.080 + 0.003
0.081 + 0.002
Concrete
0.066 0.002
0.050 + 0.003
0.068 0.017
Energy
0.215 0.013
0.032 + 0.003
0.061 0.014
Protein
0.054 0.011
0.056 + 0.002
0.057 + 0.003
Yacht
0.253 0.037
0.188 + 0.032
0.133 = 0.059
empirical âi CI coverages over a range of L relevant coverage levels a1,...,02
(ōu -oi)²) 1
CalE(D.) =
(6)
1=1
with
ôq = Del
Σ
Ify. E (2.,
(2..9))) JDD.
where wi are normalized weights of the coverage levels which are commonly considered be con- stant, Le., = 1 1 E [L].
We report this calibration error in Table 10 for DEs, BDE and MILE for multiple datasets in the distributional regression setting. We consider the coverage levels 0.5, 0.75, 0.9, and 0.95. For most cases, observe that the calibration error of MILE is on the same level as BDE and both methods often outperform the simple DE. For smaller datasets, however, estimating the empirical quantiles for small is less robust due limited test data size. Since the calibration error does not indicate whether the model is over- or underconfident, we also examine the coverage levels directly in Figure 7 The plots show a high variation in coverage quality of DE-based confidence intervals by exhibiting both strong structural under- and overconfidence, whereas the sampling-base methods generally better calibrated. The larger datasets, bi bikesharing and protein, which more likely provide enough data for reliable empirical coverage estimates. are good example of this: for bikesharing, DE is more underconfident, while for protein, is overconfident in contrast to the sampling-hased based alternatives. A visual inspection reveals that both BDE and MILE tend to be slightly underconfident, which is however often preferred by the practitioners over structural overconfidence, as seen for example with DE in the protein dataset. All in all, more careful analysis of calibration of MILE would be a great direction for future work.
21

---

Published as a conference paper at ICLR 2025
airfoil
bikesharing
concrete
0.95
0.90
0.75
Empirical Coverage
0.50
DE MILE
energy
protein
yacht
0.95
0.90
0.75
0.50
0.50
0.75
0.95
0.50
0.75
0.90
0.95
0.50
0.75
0.90
0.95
Nominal Coverage
Figure 7: Mean and standard error of empirical coverage (y-axes) for the DE baseline, BDE and MILE for six datasets (facets). The nominal coverage levels used are 0.5, 0.75, 0.9 and 0.95 (x-axes). The experimental setup identical to one Table
22


Reviews:
[{"review_id": "ZMipf7kIfv", "reviewer": "Reviewer_e8p6", "summary": "This interesting paper considers the application of microcanonical Langevin Monte Carlo, a variant of HMC, to sample from Bayesian neural network posteriors. The main idea of microcanonical LMC is that the velocity's norm is fixed and not changing unlike in HMC, which is supposedly helping with more stable exploration and allows for larger step sizes in the presence of steep landscapes. The original algorithm was proposed in the 2023 paper \"Fluctuation without dissipation: Microcanonical Langevin Monte Carlo\". The main contribution here is to consider an ensembled variant of that algorithm, and extensive numerical experiments on various Bayesian neural networks to show the practical performance of this method.", "questions": "State the implemented algorithm precisely, including how many gradient evaluations are used per step, and state whether it preserves the invariant distribution. State whether accept/reject step is needed or not.\n\nHow does the method compares with \"Scaling Hamiltonian Monte Carlo inference for Bayesian neural networks with symmetric splitting\"?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Microcanonical Langevin Monte Carlo is an interesting idea, and the numerical performance shows good improvements over NUTS, with impressive results in terms of predictive log posterior values.", "weaknesses": "The only novelty in terms of algorithmic development is the use of ensembling, Microcanical LMC was already proposed in the previous paper \"Fluctuation without dissipation: Microcanonical Langevin Monte Carlo\".\n\nThe method is not using minibatches, but instead, each step needs a full gradient, meaning that the cost of Bayesian neural networks is at least 2-3 orders of magnitude higher than using deterministic neural networks. Hence the practicality of the algorithm at this point is questionable.\n\nThere was no comparison done with the earlier method \"Scaling Hamiltonian Monte Carlo inference for Bayesian neural networks with symmetric splitting\" by Cobb et al., who managed to get HMC working with no bias using minibatches at each step, and an accept/reject step. It would be interesting to explore whether such a variant could be extended to Microcanonical LMC.\n\nA major problem is that the implemented algorithm is not clearly described in the paper, and it's not clear whether a Metropolis/Hastings step is used or not, but the authors did not claim they used one so I presume it's not used. The previous paper \"Fluctuation without dissipation: Microcanonical Langevin Monte Carlo\", (https://arxiv.org/pdf/2303.18221) claims that their Euler-Mayurama discretization (15) exactly preserves the invariant distribution, so there is no need for accept/reject step. The precise form of the implementation of (15) is not stated there either. I am very skeptical that an explicit discretization only using a single gradient evaluation per iteration can exactly preserve the invariant distribution, hence I am on the opinion that this is not a fully explicit scheme, unless the authors convince me otherwise in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oDdAVR6Dk6", "reviewer": "Reviewer_qCdd", "summary": "The authors propose an adaptation of the Microcanonical Langevin Monte Carlo method for Bayesian Neural Networks.", "questions": "The authors proposed a set of techniques to adapt the sampler for Bayesian Neural Network solvers. Is my understanding correct that to adapt Microcanonical Langevin Monte Carlo to BNNs, all we need to do is configure the parameters of the MCLMC and treat it as an ensemble method to reduce initialization error?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Clarity:\nThe paper is clearly written and well-organized.\n\nQuality:\nThe evaluation is conducted meticulously, with numerous ablations considered. The results effectively justify the proposed method.", "weaknesses": "Novelty:\nThe paper appears to be an incremental modification of the Microcanonical Langevin Monte Carlo method.\n\nSignificance:\nProposing efficient sampling methods for Bayesian Neural Networks is an important problem. However, the main results of the paper are achieved through careful parameter settings, particularly in establishing ensemble methods, tuning size, step size, energy variance scheduler, sample size, and so on. Each of these parameters is known to be crucial for obtaining better results. This makes the paper seem somewhat ad-hoc to me, lacking sufficient significance for developing improved BNN samplers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "olcgVuwqFx", "reviewer": "Reviewer_k13v", "summary": "This paper adapts the MCLMC algorithm to sample from BNN posteriors. The authors propose a series of changes to the 3-stage tuning scheme in the original paper of MCLMC algorithm and name the adapted algorithm MILE. The authors show that by doing these, MILE demonstrates superior predictive performance, improved uncertainty quantification and improved runtime.", "questions": "1. What is d in $(d-1)^{-1}$ in Eq. 3?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clear and detailed in terms of related work and the method.\n2. The empirical results look comprehensive and solid.", "weaknesses": "1. The authors claim that their method is “tuning-free”, but in 3.2, several parameters are still tuned. Maybe I misunderstood what the authors mean by “tuning-free”\n\n2. I feel that the author could elaborate more on which of the benefits of MILE mentioned in section 4 is inherited from the MCLMC algorithm, and which of them result from the authors’ adaptation. \n\nMinor:\n1. The acronym ESS first appear in 3.2.3 without any explanation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iAe0WKkiy4", "reviewer": "Reviewer_83Bk", "summary": "The paper proposes a Microcanonical Langevin Ensemble (MILE) approach which adapts the MCLMC to high-dimensional posteriors common in modern Bayesian deep neural networks (BNN). The paper integrates optimization strategies from deep ensembles to carefully adjust components of MCLMC to make it scalable. Through extensive experiments, the paper shows the superiority of MILE in prediction and UQ on popular benchmarks.", "questions": "In the “Performance results” paragraph the last sentence claims that “It is noteworthy that this is a big step for sampling-based inference, yielding a time complexity comparable to DE, while providing better and more principled uncertainty measures.” However, the time measured is on top of the DE fit as claimed in the caption of Table 1. Then the total time for MILE should be twice of DE. Please clarify.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I enjoyed reading the paper. Extensive experiments have been conducted where MILE has shown superior performance compared to its competitors. The main strengths of this paper are –\n1.\tThis paper is a significant step towards scalable full Bayesian inference with deep neural networks.\n2.\tThe paper explains the key contributions clearly.\n3.\tThe paper is well written. The authors presented their approach clearly with an intuitive explanation of the key components. \n4.\tThe authors presented extensive experiments to support their key claim which is the scalability of their approach for sampling high-dimensional Bayesian posteriors.", "weaknesses": "The paper can significantly improve if the authors can discuss/expand on the following points –\n(1)\tThe paper discusses careful tunings of the components of MCLMC.  An ablation study is needed to understand which adjustments are more important than others. My guess is the main speed-up is due to the warm starting using the DE. However, this demands a proper study.\n(2)\tThe work lacks theoretical justification. However, this is not a critical point for the paper as the authors have presented extensive experiments in support of their key claims. However, a discussion related to the convergence of the approach can improve the paper.   \n(3)\tThe numerical section seems to be lacking more competitive methods. “Path-Guided Particle-based Sampling” (ICML 24) can be another competitor approach that is proven to draw efficient samples from multi-modal Bayesian posteriors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)