Scaling Up Bayesian Neural Networks with Neural Networks
Authors: Zahra Moslemi
Yang Meng
Shiwei Lan
Babak Shahbaba
Venue: Accepted by TMLR ()

Abstract: Bayesian Neural Networks (BNNs) offer a principled and natural framework for proper uncertainty quantification in the context of deep learning. They address the typical challenges associated with conventional deep learning methods, such as data insatiability, ad-hoc nature, and susceptibility to overfitting. However, their implementation typically either relies on Markov chain Monte Carlo (MCMC) methods, which are characterized by their computational intensity and inefficiency in a high-dimensional space, or variational inference methods, which tend to underestimate uncertainty. To address this issue, we propose a novel Calibration-Emulation-Sampling (CES) strategy to significantly enhance the computational efficiency of BNN. In this framework, during the initial calibration stage, we collect a small set of samples from the parameter space. These samples serve as training data for the emulator, which approximates the map between parameters and posterior probability. The trained emulator is then used for sampling from the posterior distribution at substantially higher speed compared to the standard BNN. Using simulated and real data, we demonstrate that our proposed method improves computational efficiency of BNN, while maintaining similar performance in terms of prediction accuracy and uncertainty quantification.

Published in Transactions on Machine Learning Research (09/2024)
Scaling Up Bayesian Neural Networks with Neural Networks
Zahra Moslemi
zmostemi@uci.dd edu
Department of Statistics Universily of California Irvine, CA, USA
Yang Meng
mengy130ucci.o edu
Department of Statistics University of California Irvine, CA, USA
Shiwei Lan
slan 7 edu
School of Mathematical and Statistical Sciences Arizona State University
Tempe, AZ, USA
Babak Shahbaba
babaks Puei edu
Department of Statistics University of California Troine, CA, USA
Reviewed on OpenReview: https: // openreview. net/ forum? id- cD209Ug0X7
Abstract
Bayesian Neural Networks (BNNs) offer a principled and natural framework for proper un- certainty quantification the context of deep learning. They address the typical challenges associated with conventional deep learning methods, such as data insatiability, ad-hoc na- ture, and susceptibility to overfitting. However, their implementation typically either relies on Markov chain Monte Carlo (MCMC) methods, which are characterized by their com- putational intensity and inefficiency in a high-dimensional space, or variational inference methods, which tend to underestimate uncertainty. To address this issue, we propose novel Calibration- Caaibration-EmulaiionnSSoo Emulation-Sampllng (CES) strategy to significantly enhance the compu- lational efficiency BNN. In this framework, during the initial calibration stage, we collect a small set of samples from the parameter space. These samples serve as training data for the emulator, which approximates the map between parameters and posterior proba- bility. The trained emulator is then used for sampling from the posterior distribution at substantially higher speed compared to the standard BNN. Using simulated and real data, we demonstrate that our proposed method improves computational efficiency of BNN, while maintaining similar performance in terms of prediction accuracy and uncertainty quantifi- cation.
1 Introduction
In recent years, Deep Neural Networks (DNN) have emorged as the predominant driving force in the fiold of machine learning and are regarded as fundamental tools for many intelligent systems (Cheng et al. 2018 LeCun et al. 1998 Sze al. 2017). While DNN have demonstrated significant success prediction tasks, they often struggle with accurately quantifying uncertainty. Additionally, due their vulnerability to overfitting, they generate highly confident yet erroneous predictions (Su et al.] 2019 Kwon et al. 2022h. In recent years, there have been some attempts to address this issue. For example, the Ensemble

---

Published in Transactions on Machine Learning Research (09/2024)
Deep Learning method [Lakshminraayaaan et al. 2017) aggregates predictions from multiple models to improve reliability and uncertainty estimates. While these methods represent important progress in the right direction, developing a principled and computationally efficient framework for Uncertainty Quantification (UQ) within the context of deep learning remains a significant challenge and active area of research. This is especially important in domains where critical decisions, such as medical diagnostics, are involved. To address these issues, Bayesian Neural Networks (BNNss (MacKay) 1992 Neal 2012 Jospin et al., 2022) have emerged as an alternative to standard DNN, providing more reliable Iramework within the field of machine learning. Their intrinsic ability to capture and quantify uncertainties in predictions establishes a robust foundation for decision- under uncertainty. However, Bayesian inference in high-dimensional BNN poscs significant computational challenges due to the inefficiency of traditional Markov Chain Monte Carlo (MCMC) methods. In fact, not only BNN, but almost all traditional Bayesian inference met relying on MCMC techniques are known for their computational intensity and inefficiency when dealing with high-dimensional problems. Variational inference (Jordan al. 1999h methods have been proposed to speed up computation by approximating the posterior distribution, but they tend to underestimate uncertainty (Minka .00inka)0005...... Consequently, researchers have proposed various approaches to expedite the inference process (Welling & Teh 2011 Shahbaba ba et al. 2014 Ahn et al. 2014 Hoffman & Gelman 2014 Beskos et al. 2017;
Cui et al. 2016 Zhang et al.) 2017 2018
Li et al.] 2019))
Here, we focus on a state-of-the-ard approach, 2021 which has shown promising results in
called Calibratoon-Emulation- Sampling (CES) Cleary et al.
large-dimensional UQ problems such as inverse problems (Lan ct al. 2022). CES involves the following three
steps:
(i) Calibrate models to collect sample parameters and their corresponding expensive evaluation of posterior probability for the emulation step;
(ii) Emulate the paramctor-to-pontt to-osttrror map using the samples from Step (i); and
(iit) Generate posterior samples using MCMC based on the trained emulator at substantially lower cost.
This framework allows for reusing expensive forward evaluations from parameters to posterior probability and offers a computationally efficient alternative to existing MCMC procodures.
focuses on UQ in inverse problems and uses Gaussian GP models have well-established history of application conducting uncertainty analyses (Oakley & O O'Hagan, 2002 sensitivity assessments (Oakley & O'Hagan 2004), and calibrating computer codes (Kennedy & 0 O'Hagan 2002 Higdon et al. 2004). Despite their versatility, GP-based emulators are computationally intensive, with a complexity of O(N*) using the gquared-exponentia kernel, where N is the sample size. Lower computational complexity can be achieved using alternative kernels (Lan et al., 2015)) or various computational techniques (Liu et al., 2020: Bonilla et al., 2007; Gardner et al., [2018] Seeger et al., 2003). Nevertheless, scaling up GP emulators to high-dimensional problems remains limiting factor. Furthermore, the prediction accuracy of GP emulators highly depends on the quality of the training data, emphasizing the importance of rigorous experimental design. To address these issues, Lan et al. (2022) propased an alternative Emulative Autoencoder Monte Carlo (DREAMC) method, which
The standard CES method (Cleary et al. 2021
Process (GP) models for the emulation component.
in emulating computer models (Currin et al.] 1988).
CES scheme called Dimension-Reduen uses Convolutional Neural Networks (CNN) as emulator. DREAMC improves and scales up the application of the CES framework for Bayesian UQ in inverse problems from hundreds of dimensions (with GP emulation) to thousands of dimensions (with CNN emulation). Here, we adopt a similar approach and propose new method, called Fast BNN (FBNN), for Bayesian inference in neural networks. We use DNN for the emulation component of our CES scheme. DNN has proven to be a powerful tool in variely of applications and offers several advantages over GP emulation (Lan et al. 2022 Dargan et al.] (2020). Tt is computationally more efficient and suitable for high-dimensional problems. computational efficiency and flexibility.
The choice of DNN as an emulator enhances
Besides the computational challenges of building emulators, efficiently sampling from posterior distributions using these emulators also presents significant challenge due to the high dimensionality of the target distribution. Traditional Sattroools-Hasttng algorithms, typically defined finteeddimensional spaces, encounter diminishing mixing efficiency the dimensions increase (Gelman et al. 1997 Roberts & Rosenthal
2

---

Published in Transactions on Machine Learning Research (09/2024)
Beskos et al. 2009 To overcome this inherent drawback, a novel class of cimmension-independdee MCMC methods has emerged, operating within infiiitediimensional spaces (Beskos) 2014 Beskos et al. 2014 Boskos et al. 2017). More specifically, we use the Preconditioned Crank-Nicolson (pCN) algorithm. The most significant feature of pCN is its dimension robustness, which makes it well-suited for high- digh-dimensional sampling problems. The pCN algorithm is well- defined, with non-degenerate acceptance probability, even for target distributions
1998
2009 2011 Cotter et al., 2013 Law 2014 Beskos
infiniteddimensional spaces. As a result, when pCN is implemented on real-world computer in large but finite dimension N, the convergence properties of the algorithm are independent of N. This is in strong contrast to schemes such as Gaussian random-walt Metropolis- Metropolis-Hastingg Hastings and the Metropolis-adjustel Langevin algorithm, whose acceptance probability degenerates to zero as N goes to infinity.
In summary, this paper addresses the critical challenges of UQ in high-dimensional BNN. By incorporat- ing deep neural networks for emulation and leveraging the dimension-robust pCN algorithm for sampling, this research significantly enhances computational efficiency and scalability in Bayesian uncertainty quan- tification, offering robust counterpart to DNN, and scalable counterpart to BNN. Through extensive experiments, we demonstrate the fcasibility and effectiveness of utilizing FBNN to accelerate Bayesian UQ in high-dimensional neural networks.
2 Related Methods
Varions MCMC methods have been employed to explore complex probability distributions for Bayesian in- ference. In this section, we discuss some of the main MCMC algorithms related to our work. Additionally, we discuss a variety of state-of- theartttrrrttt methods utilized in our numerical experiments, which extend be- yond MCMC frameworks. These include Ensemble Deep Learning for Neural Networks (Perrone & Cooper BNNs with Variational Inference (Jaakkola & Jordan 2000), BNNs leveraging Lasso Approxima- MacKay 1992), Monte Carlo Dropout. (MC-Dropout) (Gal & Ghahramani] 2016), Stochastic Weight Averaging- Gaussian (SWAG) (Maddox et al.) 22019), and Accelerated Hamiltonian Monte Carlo (HMC) 2017 These techniques offer a comprehensive spectrum for evaluating our FBNN model.
1995
lion
Zhang ct al.
2.1 Hamiltonian Monte Carlo (HMC)
MCMC methods are designed for sampling from intractable probability distributions. The fundamental principle involves constructing a Markov chain whose equilibrium distribution coincides with the target distribution. Various algorithms exist for constructing such Markov chains, the simplest of which is the Metropolis-Hastingg (MH) algorithm (Metropolis et al., 1953 Hastings 1970), which relies on random walk to explore the parameter space. Hamiltonian Monte Carlo (HMC) is special case of the MH algorithm that incorporates Hamiltonian dynamics evolution and auxiliary momentum variables (Ncal 2011). Compared to using a Gaussian random walk proposal distribution in the MH algorithm, HMC reduces the correlation between successive sampled states by proposing moves to distant states that maintain a high probability of acceptance due to the approximate energy conserving properties of the simulated Hamiltonian dynamic. The reduced correlation means fewer Markov chain samples are needed to approximate integrals with respect the target probability distribution for given Monte Carlo error.
2.2 Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
As discussed carlier, HMC sampling methods provide a mechanism for defining distant proposals with high acceptance probabilities in Metsspolis-Hasttnns framework, enabling more ellicient exploration of the state space than standard random-wall walk proposals. However, a limitation of HMC methods is the required gradient computation for simulation of the Hamiltonian dynamical system; such computation is infcasible in problems involving a large sample size. Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) (Chen et al.) 2014] addresses computational inefficiency by using noisy but unbiased estimate of the gradient computed from mini-batch of the data. SGHMC is an effective method for Bayesian inference, particularly when dealing with large datasets, as it leverages stochastic gradients and hyperparameter adaptation to efficiently explore nigh-dimensional target distributions.
3

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm Precondilioned Crank-Nicolson (pCN) Algorithm
Notation:
k: iteration index.
w((); state of the algorithm at iteration k.
g(k) ~ (O, C): Gaussian noise with covariance mairix C at iteration k.
3: parameter controlling the proposal step size.
potential function related to the target probability distribution.
a(u(k), (( acceptance probability for the proposed state v(k),
Algorithm Steps:
Initialize iteration counter k t- 0.
Choose an initial state (0))
repeat
Generate noise g(h) with g(M) ~.(OC)).
Propose ((()) - v1 - PPulk) + Bg(k).
Calculate acceptance probability a(ull), t- min(L)exp(@(((()) - (vb(())))
if random number < a(ul*), v(() then
Set (k-1) ((())
clsc
Set u (k-1) + (*)) end if
Increment iteration counter + k
until stopping criterion (fixed number of iterations or convergence) is met.
2.3 Random Network Surrogate-HME (RNS-HMC)
Alternatively, we can reduce the computational cost of HMC by constructing surrogate Hamiltonians. As example, the random network surrogate-HMe non-linear bases to approximate posterior distributions. The goal is to explore and exploit the structure and regularity in parameter space for the underlying probabilistic model and construct an effective approximation of its geometric properties. To achieve this, RMC-HMC starts by identifying suitable bases that can capture the complex geometric properties of the parameter space. Through an optimization process, these bases are then used to form a surrogate function to approximate the expensive Hamiltonian dynamics. Unlike traditional HMC, which requires repeated evaluation of the model and its derivatives, RNS-HMC leverages the surrogate function to perform leapfrog integration steps, leading to substantially lower computational cost. Later, Li et al. (019) extended this idea by using neural network to directly approximate the gradient of the Hamiltonian.
method (Zhang et al. 2017 uses random
2.4 Preconditioned Crank-Nicolson (pCN)
Preconditioned Crank-Nicolson (Da Prato & Zabezyk, 2014] is variant of MCMC that incorporates a preconditioning matrix for adaptive scaling (Cotter et al. 2013). It involves re-scaling and random pertur- bation of the current state, incorporating prior information. Despite the Gaussian prior assumption, the approach adapts to cases where the posterior distribution may not be Gaussian but is absolutely continuous with respect to an appropriate Gaussian density. This adaptation is achicvod through the Radon-Nikodym derivative, connecting the posterior distribution with the dominating Gaussian measure, often chosen as the prior. The algorithmic foundation of pCN lies in using stochastic processes that preserve either the posterior or prior distribution. These processes serve as proposals for Cetropolis-Hastinee methods with specific dis- cretizations, ensuring preservation of the Gaussian reference measure. The key steps of the pCN algorithm are outlined in Algorithm

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm 2 Variational Inference in Bayesian Neural Networks (BNNs)
Initialization:
Choose an initial variational distribution go (W) for the weights W of BNN, parameterized by 0. Define the prior distribution p(W) over the weights.
while not converged do
E-Step: Estimate the Expectation of the log-likclihood over the variational distribution. Compute the gradient of the ELBO (Evidence Lower BOund) with respect to 0, where
gg (w) (w)) w)] (LLgq((W)ll(())
Here, X and Y are the inputs and outputs of the dataset, respectively, and KI. denotes the Kullback- Leibler divergence between the variational distribution and the prior.
M-Step: Maximize the ELBO with respect to 0 using gradient ascent:
0 - 0 VVLBBO))
where y is the learning rate. end while
Output: Variational distribution go (W) approximating the posterior distribution p(W|X, (il1,Y) Y).
2.5 Variational Inference
The concept of variational inference has been applied in various forms to probabilistic models. The technique offers way to approximate posterior distributions in Bayesian models (Jordan et al. 1999) The approximate distribution allows for more feasible inference, especially for complex models like neural networks. In the context of BNNs, variational inference was brought into focus by Hinton & Van Camp [1993] which, while not explicitly termed as such in the modern sense, laid the groundwork for later developments. A more direct application of variational inference to BNNs was detailed in later works (e.g.. Graves 2011). More
recently, Kingma & Welling (2014), Rezende et al. (2014) and Blundell al. (2015) significantly contributed to popularizing and advancing the use of variational inference in deep learning and Bayesian neural networks
through the introduction of efficient pradient-based optimization techniques.
A succinctly shows the iterative process of optimizing the parameters of variational distribution to approximate the posterior distribution of a BNN's weights. Through the alternation of expectation (E-Stop) and maximization (M-Step) phases, it aims to minimize the difference between the variational distribution and the true posterior, leveraging the Evidence Lower Bound (ELBO) (Jordan et al. 1999) as a tractable surrogate objective function. This approach enables the practical application of Bayesian inference to neural networks, facilitating the quantification of uncertainly in predictions and model parameters.
2.6 Laplace Approximation
Previous studies have shown that in the context of BNNs, the Laplace approximation serves as an efficient method for approximating the posterior distribution over the network's weights (Arbel et al. 2023 Blundell et al. 2015). At the core of the Laplace approximation is the assumption that, around the loss function 's minimum, the posterior distribution of the network's weights can be approximated by Gaussian distribution. This is achieved by finding the mode of the posterior, called the Maximum A Postcriori (MAP; equivalent to the minimum of the loss function in the Bayesian framework), and then approximating the curvature of the loss surface at this point using the Hessian matrix (Liang et al., 2018). The inverse of this Hessian used to define the covariance of the Gaussian posterior, thus simplifying the representation of uncertainty in the predictions. More specifically in BNNs, this approach can be used to approximate the posterior distribution of the weights given the observed data.
5

---

Published in Transactions on Machine Learning Research (09/2024)
2.7 Monte Carlo Dropout
Monte Carlo (MC) Dropout (Gal & Ghahramani 2016) was introduced as a Bayesian approximation method to quantify model uncertainty in deep learning. The core idea behind this method is to interpret dropout, a technique commonly used to prevent overfitting in neural networks, from a Bayesian perspective. Nor- mally, dropout randomly disables a fraction of neurons during the training phase to improve generalization. However, when viewed through the Bayesian lens, dropout can be seen as practical way to approximate Bayesian inference in deep networks. This approximation allows the network to estimate not just single set of weights, but distribution over them, enabling the model to express uncertainly in ils predictions. The MC Dropout technique involves running multiple forward passes through the network with dropout enabled. Each forward pass generates a different set of predictions due to the random omission of neurons, leading to a distribution outputs for a given input.
2.8 Stochastic Weight Averaging- Gaussian (SWAG)
weeragng-Gauusiian
Building on the idea of Stochastic Weight Averaging (SWA) (Ixmailov et al. 2018 Maddox et al. 2019 SWAG approximates the distribution of model weights by a Gaussian distribution, leveraging the empirical weight samples collected from training. This approach allows for a more muanced understanding of the model's uncertainty compared to SWA, which simply averages weights over the latter part of the training process. SWAG involves collecting a set of weights (Wi) over the last. N epochs of training, where W; represents the weight vector at epoch i. The mean j of the Gaussian distribution is then computed as the simple average of these weights: μ N E W, To capture the covariance of the weight distribution, SWAG
calculates the empirical covariance matrix: Σ = N-T I nWW =1)² This formulation assumes a diagonal, or low-rank plus diagonal, approximation of the covariance matrix to maintain computational efficiency. The resulting Gaussian distribution, characterized by μ and Σ, can then be used for uncertainty estimation and prediction by sampling weights from this distribution and averaging the predictions of the resulting models.
3 Bayesian UQ for Neural Networks: Calibration-EEmllatinn-SSaa
Standard neural networks (NN) typically consist of multiple layers, starting with an input layer, denoted as lo, followed by series of hidden layers 4 for / = I,..., 1, and ending with an output layer Im- In this architectural framework, comprising total of m + layers, each layer I is characterized by linear transformation, which is subsequently subjected to nonlinear operation g, commonly referred to activation function (Jospin et al. )022);
to X.
4 = (Wi4-++b) for all e (1, m- 1),
Im = Y.
Here, 0 = (W,b) b) are the parameters of the network, where W are the weights of the network connections and b are the biases. A given NN architecture represents set of functions isomorphic to the set of possible parameters 0. Deep learning is the process of estimating the parameters from the training set (x, == composed of series of input X and their corresponding labels Y. Based on the training set, a neural network is trained to optimize network parameters 0 in order to map X + Y with the objective of obtaining the maximal accuracy (under cortain loss function L(-)). Considering the error, we can write NN as a forward mapping, denoted as G, that maps each parameter vector to a function that further connects X Y with small errors En
( )n) N
0:0-YX,
0 (+)
(2)
More specifically,
(( + +Y, Yn In En, In g(onii)) (0,())
(3)

---

Published in Transactions on Machine Learning Research (09/2024)
where E represents random noise capturing disparity between the predicted and actual observed values in the training data. Here, Y is a continuous random variable in regression problems, or a continuous latent variable in classification problems.
To train NN, stochastic gradient algorithms could be used solve the following optimization problem:
0
arg min L(0: X, Y) AE
arg min L(Y G(x;0)) dEe
For example, the loss function L(0; X, Y) can be defined in terms of the negative log-likelihood function + follows:
1 _ - 9(x:0)li
200; =
(4)
The point estimate approach, which is the traditional approach in deep learning, is relatively straightfor- ward to implement with modern algorithms and software packages, but tends to lack proper uncertainty quantification (Guo et al. 2017 Nixon et al. 2019). To address this issue, stochastic neural networks, which incorporate stochastic components in the network, have emerged as a standard solution. This performed by giving the network cither stochastic activation functions or stochastic weights to simulate random samples for 0. The integration of stochastic components into neural networks allows for an extensive exploration of model uncertainty, which can be approached through Bayesian methods among others. Tt should be noted that not all neural networks that represent uncertainty are Bayesian or even stochastic; some employ deter- ministic methods to estimate uncertainty without relying on stochastic components or Bayesian inference Lakshminarayanan et al. 2017 Sensoy et. al., 2018). Bayesian neural networks (BNN) represent. a subset of stochastic neural networks where Bayesian inference is specifically used for training, offering a rigorous probabilistic interpretation of model parameters (MacKay 1992 Neal 2012). The primary objective is to gain deeper understanding of the uncertainty that underlies the specific process the network modeling.
To design a BNN, put prior distribution over the model paramcters, p(0). By applying Bayes' theorem, the posterior probability can be written as:
pY (()) P X, p)0))(()) d0
p X,Y)
(5)
& ( X, p())
(6)
BNN is usually trained using MCMC algorithms. Because we typically have big amount of data, the likelihood evaluation tends to be expensive. One common approach to address this issue is subsampling, which restricts
the computation to a subset of the data (see for example, Hoffman et al. 2010 Welling & Teh 2011 Chen et al. 2014). The assumption is that there is redundancy in the data and an appropriate subset of the data can provide good enough approximation the information provided by the full data set. In practice, it a challenge to find good criteria and strategics for an effective subsampling in many applications. Additionally, subsampling could lead to significant loss of accuracy (Betancourl, 2015
3.1 Fast Bayesian Neural Network (FBNN).
We propose an alternative approach that explores smoothness or regularity in parameter space, a charac- teristic common to most statistical models. Therefore, one would expect find good and compact forms of approximation of functions (e.g., likelihood function) in parameter space. Sampling algorithms can use these approximate functions, also known as "surrogate" functions, to reduce their computational cost. More specifically, we propose using the CES scheme for high-dimensional BNN problems in order to bypass the expensive evaluation original forward models and reduce the cost of sampling to a small computational overhead. Compared with MCMC methods, which require repeatedly evaluating the original (large) NN for the likelihood given the data, the proposed method builds smaller NN emulator that bypasses the data (i.e., cuts out the middleman) by mapping the parameters directly to the likelihood function, thus avoid- ing costly evaluations. That is, the emulator is trained based on the Karameter-likelhbor pairs, which are collected through few iterations of the original BNN. In contrast to subsampling methods, this approach

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm 3 Fast Bayesian Neural Network (FBNN)
Input: Training set {(X,, Yn) Prior p(0)
Output: Postcrior samples for model parameters
procedure FBNN(((X,, Y,)) p(0)
Calibration Step:
Initialize model parameters 0 using SGHMC
Save posterior samples { Jj=1 and the corresponding ( (X (Xx)) after a few iterations
Emulation Step:
Build an emulator of the forward mapping g based on (0)) 9 (Xn))²=1 using a DNN as the emulator
Sampling Step: Run approximate MCMC, particularly pCN, based on the emulator to propose 0' from 0.
end procedure
can handle computationally intensive likelihood functions, whether the computational cost is due to high- dimensional data or complex likelihood function (e.g., models based on differential equations). Additionally, the calibration process increases the efficiency of MCMC algorithms by providing a robust initial point in the high-density region. Algorithm 3 shows how our proposed method, called Fast. Bayesian Neural Net- wor (FBNN), combines the strengths of BNN in uncertainty quantification, SGHMO for efficient parameter calibration, and the pCN method for sampling. More details are provided in the following sections.
3.2 Calibration Early stopping in Bayesian Neural Network
By "calibration" mean collecting optimal sample of parameters to build emulator with reasonable level of accuracy. This is aligned with traditional calibration goals of balancing accuracy and reliability, but within a new context. Here, the calibration step involves an carly stopping strategy, aimed at collecting a largeted set of posterior samples without. fully converging to the target distribution. More specifically, we use the Stochastic Gradient Hamilton Monte Carlo (SGHMC) algorithm for a limited number iterations collect a small set of samples. These samples include both the model paramcters (@W1) and the outputs predicted by the model (G(X; (())) for each sample j out of a total of samples. The key focus of this training phase is not to obtain precise approximation of the target posterior distribution, but rather collecting small number of posterior samples the "training data" for the subsequent emulation step. The SGHMC algorithm plays crucial role in efficiently handling large datasets and collecting essential samples during the calibration step of the FBNN. Its ability to introduce controlled stochasticity in updates proves instrumental in preventing local minima entrapment, thereby providing a comprchensive set of postcrior samples that capture the variability in the parameter space.
3.3 Emulation Deep Neural Network (DNN)
The original forward mapping in BNN involves mapping imput dataset X to the response variable Y. For the likelihood evaluation using original forward mapping, it is necessary to calculate the likelihood L(O; X, Y) for each sample of model parameters. This means that with each iteration, when new set of model parameters is introduced, the original forward mapping needs to be applied to generate output predictions, followed by the calculation of the likelihood. In general, this process can be very time-consuming If, however, we have a small set of estimated model parameters along with their corresponding predicted outputs collected during the calibration step, an emulator can be trained to eliminate the intermediary step (passing through each data point), allowing us to map the parameters directly the likelihood function. This leads to a computationally efficient likelihood evaluation. Therefore, to address the computational challenges of evaluating the likelihood
with large datasets, build an emulator G using the recorded pairs (06), g(و -(X; { obtained during the calibration step. More specifically, these input-output pairs are used to train a DNN model as
8

---

Published in Transactions on Machine Learning Research (09/2024)
an emulator 9 of the forward mapping G:
();;0)) = DNN(O,G(X;O) = -K-- Fo(O), F&(.) = (w +b2) E C (RRd,,RRe.
(7)
(8)
Given a DNN model where represents the input and (Xx;) denotes the output, we set the dimensions as do = d and dg= D, where d represents the dimension of the input parameter vector O, and D represents the dimension of the output ()::)). Here, the matrices W₂ are defined in the space Rd xds and the vectors by in Rd++1. The functions 9k act as (continuous) activation mechanisms. In the context of our numerical examples, the activation functions for the DNN emulator are selected to ensure that both the function approximations and their derived gradients have minimized errors. This involves a grid scarch over a predefined set of activation functions lo ensure that the network efficiently approximates the larget functions and their gradients.
After the emulator is trained, the log- ligllikeihood can be approximated as follows:
=
(9)
By combining the approximate likelihood L (0; with the prior probability p(0). an approximate poste- rior distribution can be obtained. Similarly, could approximate the potential function using the predictions from DNN:
1 t5 = 2 Y - (X; O)I
(10)
Building upon the foundational concepts of using a DNN emulator Ge for approximating the forward mapping function G, we further elaborate on the implications and advantages of this approach for Bayesian inference, particularly in the context of handling large datasets and/or complex likelihood functions. The emulation step, which involves training the DNN emulator with input-oulput pairs (06), G(X; 0())), serves as a critical phase where the emulator learns to mimic the behavior of the original model with high accuracy. The utilization of DNN emulator to approximate the likelihood function in Bayesian inference presents a significant computational advantage over the direct use the original BNN likelihood. This advantage stems primarily from the inherent differences in computational complexity between evaluating the the likelihood with a DNN emulator which takes a set of model parameters as input and yields predicted responses and the original BNN model- which processes X input to produce the response variable.
In the sampling stage, the computational complexity could be significantly reduced if we use th instead of & in the accept/reject step of MCMC. If the emulator is good representation of the forward mapping, the difference bet ween Φe and tb would be small and negligible. Then, the samples by such emulative MCMC have the stationary distribution that closely follows the true posterior distribution. This approach not only ensures that the sampling process is computationally fcasible, but also maintains the integrity of the stationary distribution, closely approximating the true posterior distribution with minimal discrepancy. The integration of DNN emulators into the Bayesian inference workflow thus presents a compelling solution to the computational challenges associated with evaluating likelihood functions in complex models.
3.4 Sampling Preconditioned Crank-Nicolson (PCN)
In the context of the FBNN method, the sampling step is crucial for approximating the posterior distribution efficiently. The method employs MCMC algorithms based on a trained emulator to achieve full exploration and exploitation. However, challenges arise, especially in high-dimensional parameter spaces, where classical MCMC algorithms often exhibit increasing correlations among samples. To address this issue, the pCN method presented in Algorithm D has been used in our proposed framework as a potential solution. Un- like classical methods, pCN avoids dimensional dependence challenges, making it particularly suitable for scenarios like BNN models with high number of weights to be inferred (Hairer et al. 2009).
As explained in section 2.4 the pCN approach minimizes correlations between successive samples, critical feature for ensuring the representativenes of the samples collected. This characteristic is vital for FBNNs, as
9

---

Published in Transactions on Machine Learning Research (09/2024)
Target
PCN
0.6
00
924
04
012
030
0.08
63
02
0.00
01
64
-2
Figure 1: Sampling from a mixture of 25 Gaussians shown in (a) with 200k samples. SGHMC in (b) broadly explores the space, while pCN in (c) hones in the high-density regions for precise mode capture.
it directly impacts the network's 's ability to learn from data and make robust predictions. The pCN method excels in traversing the parameter space with controlled perturbations, enhancing the algorithm's 's ability to capture the most probable configurations of model parameters. This focus on effective exploration around the mode contributes a more accurate representation of the underlying neural network, ultimately improving model performance. Tn other words, the choice of pCN as the sampling method in FBNN is motivated by its tailored capacity to navigate and characterize the most probable regions of the parameter space. This choice reinforces the methodology's robustness and reliability, as pCN facilitates efficient sampling, lcading to more accurate and representative approximation of the posterior distribution.
To illustrate this, Figure I displays a simulation that contrasts the sampling mechanisms of SGHMC and PCN within a multimodal probability distribution. The task to sample from a mixture of 25 Gaussian distributions, represented in panel (a), using total of 200,000 samples. Here, the target distribution is multimodal with several distinct peaks (modes). Middle figure shows that SGHMC has explored the parameter space, although with a less concentrated sampling around the modes compared to the target distribution. This indicates that while SGHMC is effective at exploring the space, it may not capture the modes as tightly as the target distribution. In the right panel related to pCN sampler, the concentration of samples around the modes is much higher compared to SGHMC, which indicates that pCN more effective at exploring around the modes of the distribution. Thus, we believe the combination of SGHMC and pCN our proposed framework can complement each other for more effective exploration of the parameter space.
3.5 Theoretical Foundations
In this section, we aim to quantify the error between the true potential function and its emulation in the context of the FBNN method. Let Ω (0, 1)d and consider forward mappings in the Sobolev space Wm)((S) {f LP(S)) Daf E (P()) for all a E (N U (0))a with la| I n} with Il
Eosialgn [[B and |f maxosialsa D flboo Define the Sobolev-Slobodecki norm for (((x)-f(v) 1a- drdy) and || == max {IlSIboo ess ess supz,yE2
0<<<ll 11. (11 Sa Sa
f(x)-f()) }. 1 5/" S
Theorem 3.1. Let S spso xo and < 1. Assume 9,(X; WmP n (2) for 1,... D. For any 102), there is standord NN, ge, with ReLU activation functions such that
| Φ SC
(11)
and the depth K < clog(e =((as)) the number of weights and units N S ce ce- d/(n-x) leg2(e-n/(1xx)) -n((n-s)) with constant c(d, p, s) 0.
10

---

Published in Transactions on Machine Learning Research (09/2024)
Proof. Note that we have
((0) ((0)- -1((()x00 (X;0), (
there exists constant M > 0 such that maxigien [[;;(; )|,,, Ivvil M. For e/ (M > 0, by Theorem 4.1 of Gühring et al. (2020), there exists a standard NN with ReLU activation functions and the depth K and the number of weights and units as in the condition such that
Because (;(X;
( (
(x; - (x; Mbs S Se/(MD), j
Therefore we have
D
| $5 11.1.. S M Σ
19,(x: -95(X; 1ssp Se.
j=1
Denote the Hellinger distance between densities as du (=, emulation error propogates into the Hellinger error in the likelihood.
T²) ) = f(v --0--V77 VTT Then we describe how the
Theorem 3.2. Let x(:0) exp(-o(.:0) and n= (;;0) exp(-o0"( (.; 0)) denole the likelihood and ils emulation, respectively. Suppose the conditions of Theorem 3.1 holds for co. Then we have
( x²) Se.
(12)
where satisfies the constraints in Theorem 3.1
Proof. Compute the Hellinger distance
(T, = / -VT / exp (- (00(6:0) (0(v00))30000 -0000000000 (v:00) )} < Oy
(T.
5
$ 4 (():0) (y: 0) f)dy SC' / 1100 1 (y: dy S
where the last inequality is by Theorem 3.1 Taking square-root on both sides yields the conclusion.
Furthermore, given Gaussian prior for 0, To, we can characterize the discrepancy of posteriors with the original and emulated likelihoods in the following thcorem.
Theorem 3.3. Let (()) exp(-$(y: 0) (f) and (0) exp((-0'(1 ())= denote the posterior and the one with emulated likelihood, respectively. Suppose the conditions of Theorem 3.1 holds for p = co. Then we have
df a)) S...
(13)
where satisfies the constraints in Theorem 3.1
Proof. By Bayes' theorem, we have
(0) = ( exp( exp(-0(y;;)) (0), 0 0 7(y) / exp( exp(-0(y;: 0) To (0)d0) < +00
* #0)
Z exp( exp(-0" 10 (1: 0)) 0 Zz
exp( - 4 (y: 0 mm 0 d)) < to0.
8
11

---

Published in Transactions on Machine Learning Research (09/2024)
Therefore we compute the Hellinger distance
/
- (-2 (-2 -z) exp
1 (1: 01) To(d0)
(+(
2(v)
exp
(v:
12
$
2
oxp
2 ( 0)
-exp
(v: 0) )}
To(d)) + +12 - -Z)
(yy)
2
1
1
2
exp
p1; 0)
- -y: 01)
A(d0) + CZZ((y) (3))
C"
(v; ()] + / |exp(= 0)) xxp (yo))
(115 0)
2
C²
< ④ 101 C 1 < 2
where the last inequality is by Theorem 3.1 Taking square-root on both sides yields the conclusion.
4 Numerical Experiments
4.1 Setup
We demonstrate the effectiveness of our method on eleven synthetic and real-world datasets, comparing it against a comprehensive selection of baseline approaches.
Datasets. We experiment on series of regression and classification problems. Detailed information regard- ing these datasets, including the number of features and datapoints in each, and the number of parameters used in the main FBNN model for each dataset, is outlined in Table T We have also included the details of the DNN Emulator architecture for each dataset in Table 2
Baseline Methods. We present empirical evidence comparing our CES method against a broad array of baseline approaches including two baseline BNN methods equipped with the SGHMC and pCN samplers (shown as BNN-SGHMC and BNN-pCN), and BNN architectures incorporating Variational Inference, Lasso Approximation, MC-Dropout, SWAG, and RNS-HMC. Detailed information about these methods was pro- vided in Section 2 We also include the results from DNN, which does not provide uncertainty quantification, but serves as a reference point. Moreover, we provide the results of Deep Ensembles, which consist of multiple DNNs, each initialized with different random seeds. We refer to this method as Ensemble-DNN. Although the Ensemble- DNN approach allows for parallelization, falls short in providing a probabilistic framework for analysis, significant advatagg offered by our CES method.
As discussed earlier, one of the distinctive features of our main FBNN model, more specifically shown FENN (SGHMC-PCN), lies in its strategic integration of the SGHMC sampler during the calibration step and the pCN algorithm during the sampling step. This combination is carefully chosen to harness the complementary strengths of these two sampling methods. Further expanding our exploration, we introduce three additional FBNN models: FBNN (pCN-SGHMC), where pCN employed in the calibration step and SGHMC in the sampling step; FBNN (pCN-pCN), where pCN is used in both steps; and FBNN (SGHMC- SGHMC), where SGHMC is used in both calibration and sampling steps.
Throughout these experiments, we collect 2000 posterior samples for the BNN-SGHMC and BNN-PCN, with samples being collected at each iteration. In contrast, for the FBNN methods, we use a small number (200) of samples from either BNN-SGHMC or BNN-PCN (depending on the specific FENN model) along with the corresponding predicted outputs during the calibration step. These 200 samples serve as the "training data" " for the emulator. Moreover, we evaluate the efficacy of utilizing only the initial 200 samples from the BNN-SGHMC model across all the datasets. This was done demonstrate the necessity of collecting more samples, either using the original BNN employing the FBNN method, rather than relying our inference a limited number of initial samples.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Description of various datasets used evaluate the overall performance of our proposed approach against state-of-Che-arl baseline methods.
Task
Dataset
# Datapoints
# Features
# FBNN paramcters
Regression
Boston Housing
506
13
3,009
Wine Quality
1,599
11
241
Alzheimer
185,831
56
33,345
Year Prediction MDS
515,345
90
81,901
Simulation
5,000,000
1,000
52,832
Classification
Adult
40,434
14
2,761
Mnist
70,000
784
3,961
Alzheimer
185,831
56
33,345
celebA
202,599
39
1,521
SVHN
630,420
3072
171,313
Simulation
5,000,000
1,000
52,832
Table 2: Description of various datasets and their corresponding DNN Emulator architectures (Droupout layers have been used on input layer and first hidden layer)
Task
Dataset
Hidden Layers
# Neurons per Layer
Activation Functions
# Epochs
Dropout Rate
Regression
Boston Housing
2
3,32
ReLU
1000
0.7
Wine Quality
2
3,32
ReLU
1000
0.5
Alzheimer
2
4,64
ReLU
1000
0.5
Year Prediction
2
4,64
ReLU
1000
0.5
Simulation
3
8,64,32
ReLU
1000
0.5
Classification
Adult
2
4,32
ReLU
1000
0.5
Mnist
2
3,64
ReLU
1000
0.5
Alzheimer
2
4,64
ReLU
1000
0.5
celebA
2
3,32
ReLU
1000
0.5
SVHN
3
4,64,32
ReLU
1000
0.7
Simulation
3
8,64,32
ReLU
1000
0.5
It is also crucial to highlight that the BNN-SGHMC and BNN-pCN models are trained from a randomly chosen initial point for the MCMC sampling process. On the other hand, the FBNN methods, we employ the set of posterior samples collected during the last iteration the calibration step as the starting point for the subsoquent MCMC sampling.
Metrics. To thoroughly assess the performance and effectiveness of each method, we use range of key metrics. These include Mcan Squared Error (MSE) for regression tasks (Figure 2 and Accuracy for clas- sification tasks (Figure B). We also evaluate the models based on their computational cost, and various statistics related the Effeettive Sample Size (ESS) of model parameters. These statistics include the mini- mum, maximum, and median ESS, as well the minimum ESS per second. We also quantify the amount of speedup, denoted "spdup", metric that compares the minimum ESS per second of each model with that of BNN-SGHMC as the benchmark (Figure Analysing spdup is crucial as it provides comparative
13

---

Published in Transactions on Machine Learning Research (09/2024)
measure of efficiency, highlighting the model's capability to achieve high-quality parameter sampling with lower computational resource utilization relative to the benchmark BNN-SGHMC.
The effective sample size takes the autocorrelation among the consecutive samples into account. While we can reduce autocorrelation using the thinning strategy, this leads to a higher computational time for the same number of samples. Our spdup metric allows for a fair comparison of sampling algorithms (regardless of what thinning strategy used) by taking both autocorrelation and computational cost into account.
For UQ in regression cases, we evaluate the Coverage Probability (CP) set at 95%. In addition, we construct 95% Credible Intervals (CI) by the prediction results of Bayesian models, along with the average true output, to illustrate UQ in regression problems. For classification problems, we use Expocted Calibration Error (ECE) and Reliability Diagrams to evaluate UQ ECE addresses model calibration, aiming for accurate uncertainly estimates, while reliability diagrams offer a visual summary of probabilistic forecasts.
Figures 2 and summarize our results. More detailed results are provided in Tables Al and A2
4.2 Regression Tasks
We first evaluate our proposed method using a set of simulated and real regression problems.
Simulated Data. We begin our empirical evaluation by using simulated data. To this end, we utilize the rekeessngrsssonn function from the datasets package to generate a dataset consisting of 5,000,000 observations and 1,000 predictors.
Figure 5 compares the true and emulated log likelihood functions associated with posterior samples collected using BNN-SGHMC. The emulated values are based on the FBNN (SGHMC-PCN) model. As we can see, the two functions are similar, indicating that the emulator provides a reasonable approximation of the true target distribution.
Figure 2 compares the MSE among all models, showing that while the DNN method achieves the lowest MSE at 0.71, the FENN (SGHMC-PCN) model provides a similar performance. Notably, among all the FBNN variants, FBNN (SGHMC-PCN) provides the highest CP at 92.2%, demonstrating a level of calibration comparable to that of the BNN model. The Ensemble-DNN demonstrates comparable performance to FBNN (SGHMC-PCN) in terms of CP, yet it operates at a pace three times slower.
Examining the efficiency of sample generation, all FBNN variants have relatively higher ESS per second compared to all the other BNN models, except for BNN-RNS-HMCC. Among all the models, FBNN (SGHMC- PCN) has the highest min ESS per second at 0.043. Figure 4 indicates our model provides the highest speedup (16.33) compared to BNN-SGHMC as the baseline model, highlighting our method's computational efficiency. Considering these results, FBNN (SGHMC-PCN) emerges strong approach with a good balance between predictive accuracy, computational efficiency, and uncertainty quantification, making it the overall best option for Bayesian deep learning.
Figure Ch shows the estimated mcan and prediction uncertainty for both BNN-SGHMC and FBNN (SGHMC- pCN) models, alongside the smoothed average and 95% interval for the true output. For clarity and concise- ness within our figures, we have employed Principal Component. Analysis (PCA) and used the first principal component to transform the original data into a one-dimensional representative feature (x-axis in Figure 6 As can see, BNN and FBNN have very similar credible intervals. This consistency in credible interval bounds is significant. for UQ indicating that both models effectively and almost equally quantify uncertainly in their predictions.
Wine Quality Data. As the first real datasct for the regression task, we use the Wine Quality data Cortez et al. 2009). This dataset contains various physicochemical properties of different wines, while the target variable is the quality rating. The performance of FBNN (SGHMC-PCN) indicates a well-balanced approach, making it superior to the other models for several reasons. Firstly, it achieves competitively low MSE of 0.52, comparable to other high-performing models like BNN-SGHMC and SWAG, but it surpasses them in terms of speedup. Moreover, FBNN (SGHMC-PCN) exhibits robust predictive performance,
14

---

Published in Transactions on Machine Learning Research (09/2024)
Simulated Dataset
Wine Quality Dataset
Roston Housing Dataset
Attheimer Dataset
YesPradicionnto Dataset
50
24
1.
Figure 2: A comprchensive comparison of the MSE for various BNN methods across five regression datasets. Each subplot corresponds to a dataset and the color-coded bars represent the distinct hodoools evalu- ated. The main FBNN method, called FBNN (SGHMC-PCN), highlighted in bold, shows among the lowest values for MSE among all datasets.
Simalated Dataset
Adult Dataset
Altheinerr Dataset
Muist Dataset
colobA Dataset
Aocrmoy Chl
Figure 3: comprehensive comparison of the Accuracy for various BNN methods across five classification datasets. Each subplot corresponds to a dataset and the color-coded bars represent the distinct methodologies evaluated. The main FBNN method, FBNN (SGHMC-PCN), highlighted in bold, demonstrates superior performance by achieving the highest accuracy in four out of the five datasets examined and the second highest in one dataset.
with a competitively high CP. Figure Gp shows the prediction mean and 95% CI BNN-SGHMC and FBNN (SGHMC-PCN), as well as the smoothed average and 95% interval for the true output.
Boston Housing Data. The Boston housing dataset was collected in 1978 Harrison Jr & Rubinfeld 1978h. Each of the entries present aggregated data for homes from various suburbs in Boston. For this dataset, FBNN (SGHMC-PCN) stands out with a notable balance between MSE (3.82), CP (81.1%), and computational efficiency, completing the task in just 91 seconds. This model significantly outperforms all the other models in terms of speedup (11 (11.94), showeasing its effectiveness in sampling. Figure (p shows the 95% Cls and mean predictions of both BNN-SGHMC and FBNN (SGHMC-PCN). The FBNN (SGHMC-PCN)
15

---

Published in Transactions on Machine Learning Research (09/2024)
Adult dateoet
Atheiner dataset
Hrest delisset
Celebi delaset
Figure 4: Comparative analysis of speedup (spdup) for various Bayesian Neural Network (BNN) methods across tested datasets. Methods are ordered by efficiency within each dataset, highlighting the impact of model characteristics on sampling performance. Notably, 'FBNN (SGHMC-PCN) achieves the highest speedup among all datasets except for the Wine Quality dataset, where it ranks as the second highest, underscoring its exceptional efficiency in diverse analytical contexts.
model, in particular, displays well-calibrated uncertainty quantification, mirroring the performance of the BNN models, implying that its probabilistic predictions capture the model uncertainty.
Alzheimer Data. Next, we analyze the data from the National Alzheimer's Coordinating Center (NACC), which is responsible for developing and maintaining database of patient. information collected from the Alzheimer disease centers (ADCs) funded by the National Institute on Aging (NIA) (Beekly et al. (004). The NIA appointed the ADC Clinical Task Force to determine and define an expanded, standardized clinical data set, called the Uniform Data Set (UDS). The goal of the UDS is to provide ADC researchers a slandard set of assessment procedures to identify Alzheimer's disease (Beekly et al. 2007 p. We have used 56 key features for our analysis. These features were carefully selected to represent wide spectrum of variables relevant to Alzheimer's disease diagnosis, including functional abilities, brain morphometrics, living situations, and demographic information (Ren et al. 2022). For the regression case, the goal is to predict Left Hippocampus Volume, critical marker in the progression of the disease (van der Flier & Scheltens) 2009), as function of other variables. For this dataset, Figure 2 shows that the FBNN (SGHMC-PCN) model stands out for its balanced performance, recording the second lowest MSE at 0.48 and a relatively high CP at 91.6%. It shows a considerable improvement in computational efficioncy, evidenced by a specdup factor of 22 times compared to BNN-SGHMC as the baseline BNN model.
Year Prediction MSD Data. For this data, the goal is to predict the release year of song from audio features, Songs are mostly western, commercial tracks ranging from 1922 to 2011, with a peak in the year 2000s (Bertin-Mahieux 2011). In the context of the YcanProdictionMSD datasct, FENN (SGHMC-PCN) showcases its superiority over other models by achieving a good balance between accuracy, computational efficiency, and effective uncertainty quantification. With an MSE of 73.41, close to that of Ensemble-DNNN, and CP of 92.23% it outperforms most other models. Moreover, the computational efficiency of FBNN (SGHMC-PCN) is highlighted by its speedup factor of 11.98 (Figure 4 over the baseline model BNN- SGHMC.
16

---

Published in Transactions on Machine Learning Research (09/2024)
EnW True Libelheedd
BHH Preddatedd Libeitind
Concestion oflng Libehoot vouus using Thee Treslated Nastims
Pareopen Compuseett
Live
Princppll Composent
Princtpal Component
Princppal Compunent
Figure 5: Evaluating the performance of the emulator. The plot contrasts log likelihood values obtained using the true likelihood function against those derived using the emulated likelihood function. The x-axis and y-axis represent the first and second principal components of the model paramcters based on the MCMC samples obtained our simulation study.
Predicted and Outputs
328
Predicted Outputs
Predicted and true Outputs
45% inters val For Outbut
Tham OaTEaT
15
00
05
1.0
2.0
PC1
PC1
(a) Simulated data
(b) Wine Quality data
(c) Boston Housing data
Figure 6: Comparative analysis of predictive credible intervals and mean predictions for regression tasks. For each dataset, the 95% CI for BNN predictions and FBNN predictions are shown as shaded areas. The average predictions from BNN and FENN are represented with dashed lines. Additionally, the 95% CI for the true output as ground truth and the smoothed average true output are plotted as solid lines. The x-axis shows the first principal component of the predictors.
Fraction pasitives
BNN-pCN
FBNN(PCN- PCN)
FENN(SGHMC-SGHMG
F8NN(pCN-SGHMC)
FBNN(SGHMC.PCCM
20-
10-
2.2
64-
34
04
0:4
02
00:10
50
06
50-
061
000
655
375
1.00
0.00
055
0.5
4.5
1.00
200
635
650
025
1.00
000
625
ato
0.5
1.00
42
05
1.2
6.00
655
053
075
100
Mean predicted values
Figure 7: Reliability Diagrams for Simulated dataset in classification task. These diagrams incorporate equal frequency binning.
4.3 Classification Tasks
Next, we evaluate our method based on a set of simulated and real classification problems. The results are summarized in Figures] and More details results are provided Table A2
Simulated Data. As before, start with simulated dataset with binary outcome. For this, the make_ caate__llssffccatiooo function from the sklearn datasets package to generate dataset consisting of
17

---

Published in Transactions on Machine Learning Research (09/2024)
5,000,000 observations and 1000 predictors. Accuracy comparison in Figure shows FENN (SGHMC-PCN)) DNN, and Ensemble-DNN exhibit. comparable performance and outperform other models. While BNN- RNS-HMC achioves the highest speodup, it significantly underperforms in terms of accuracy. In contrast, FBNN (SGHMC-PCN) provides the second-highesl speedup at 32.00, showcasing its computational efficiency relative to BNN-SGHMC. Furthermore, it maintains the second highest accuracy rate of 96%, indicating an optimal balance between computational efficiency and accuracy.
For this example, BNN-MC-Dropout has the lowest ECE value compared to other methods (see Appendix), but it also have a lower accuracy rate. Among the FBNN variants, FBNN (SGHMC-PCN) presents a low ECE, closely aligning with the ECE value of BNN-MC-Dropout, while providing an accuracy rate similar to DNN. Moreover, as illustrated in Figure 7 the FENN variants, particularly FBNN (SGHMC-PCN), appear to be better calibrated across most probability ranges except. at the highest probabilities, suggesting a more reliable UQ Note that in these figures, a model with a reliability curve that closely follows the diagonal line is considered better calibrated, meaning its predicted probabilities are more aligned with actual outcomes.
Adult Data. Next, we use the Adult dataset (Becker & Kohavi 1996), where the classification task involves predicting whether an individual will earn more or less than $50,000 per year. Figure 3 demonstrates that all the methods achieve comparable accuracy rates, although DNN, Ensemble-DNN, and FBNN(SGHMC- PCN) have the best performance. FBNN (SGHMC-PCN) also stands out as the most computationally efficient method for uncertainty quantificattion with a speedup value of 54.91 relative to the baseline BNN approach. A low ECE value for the FBNN (SGHMC-PCN) model signifies its superior performance in terms of uncertainty quantification.
Alzheimer Data. We have used the same NACC dataset we previously discussed, but this time as a classification problem. Here, our objective is to predict cognitive status, classifying individuals as cither coguitively unimpaired (healthy controls, HC), labeled as class 0, or as having mild cognitive impairment (MCT) due to Alzheimer's 's disease (AD) or dementia due to AD, labeled as class 1.
Achieving the highest accuracy of 84% at a relatively low computational cost, FBNN (SCHMC-PCN) sur- passes all other models in correctly identifying Alzheimer's disease, making it the most reliable model among those tested. This model not only surpasses the accuracy of the standard DNN and Easemble-DNNe mod- els but also offers a balance between computational efficiency and high accuracy. FBNN (SGHMC-PCN) detonsttraes the highest sampling elficiency (speedup of 11), indicating it can achieve high accuracy with a lower computational cost compared to other Bayesian models. Moreover, for the FBNN model implemented on the Alzheimer dataset, the ECE value is low, and the reliability curve closcly tracks the diagonal line.
MNIST Dataset. The MNIST dataset is commonly used as a benchmark dataset for the hand-written digit classification task (Deng, 2012). Among the various models evaluated on the MNIST dataset, FBNN (SGHMC-PCN) stands out as the optimal choice, demonstrating exceptional performance across multiple metrics. It achieves the highest. accuracy of 94%, matching the top performance of Ensemble-DNN but with significantly improved efficiency and effectiveness in uncertainty quantification. It exhibits a substan- tial speedup of 16.77, and the lowest ECE at 0.241, suggesting that it not only provides highly accurate predictions but also reliably estimates the uncertainty associated with these predictions.
CelebA Dataset. CelebA (Luu et al. 2015) is an image dataset of celebrity faces annotated with 40 attributes including gender, hair color, age, smiling, etc. The task is to predict hair color, which is either blond Y = 1 or non-blond Y = 0. The FBNN (SGHMC-PCN) model stands out among the alternative methods, showcasing its superiority through several key performance metrics. It achieves the highest accuracy of 85%, and the highest speedup factor of 61.84, indicating an exceptional balance between computational efficiency and performance. The model achieves the lowest ECE of 0.493, indicating reliability in its predictive uncertainty.
SVHN dataset. real-world images of house numbers taken from Google Street View. The images are 32x32 pixels in size and have three color channels (RGB). The goal is to classify digit images into 10 classes. The results demonstrate
The Street View House Numbers (SVHN) dataset (Netzer al. 2011 includes labelled
18

---

Published in Transactions on Machine Learning Research (09/2024)
the superiority of FBNN (SGHMC-PCN) in terms of accuracy and computational efficiency. FBNN (SGHMC- PCN) achieved an accuracy of 96%, the second highest among all models. The speedup compared to the baseline BNN-SGHMC method was 14.99 times, the highest speedup value recorded. Additionally, the low ECE of 0.203 demonstrates better uncertainty quantification than most other methods, including BNN-VI, BNN-MCC-ropout and SWAG.
5
Conclusion
In this paper, we have proposed an innovative CES framework called FBNN, specifically designed to enhance the computational efficiency and scalability of BNN for high-dimennional data. Our primary goal is to provide a robust solution for uncortainty quantification in high-dimensional spaces, leveraging the power of Bayesian inference while mitigating the computational bottlenecks traditionally associated with BNN.
In numerical experiments, have successfully applied several variants of FBNN, including different configurations with BNN, to regression and classification lasks on both synthetic and real datasets. Remark- ably, the FBNN variant incorporating SGHMC for calibration and pCN for sampling, denoted as FBNN (SGHMC-PCN), not only matches the predictive accuracy traditional BNN but also offers substantial computational advantages. More specifically, our numerical experiments across various regression and classi- fication tasks consistently demonstrate the superiority of the FBNN (SGHMC-PCN) method over traditional BNNs and DNNs. In regression tasks, FBNN (SGHMC-PCN) demonstrates competitive MSE while signif- icantly enhancing computational efficiency, achicving notable speedups compared to baseline models. This efficiency does not come at the expense of accuracy, as evidenced by the competilive MSE values and robust uncertainty quantification metries. In classification tasks, FBNN (SGHMC-pCN) stands out by achieving high accuracy rates and low ECE values, which indicate reliable uncertainty quantification.
The superior performance of FBNN (SGHMC-PCN) can be attributed to the complementary strengths of SGHMC and pCN. SGHMC excels at broad exploration of the parameter space, providing an effective means for understanding the global structure during the calibration step. On the other hand, pCN is adept at efficient sampling around modes, offering a valuable tool for capturing local intricacies in the distribution during the final sampling step. By combining these samplers within the FBNN framework, we achieve a balanced approach between exploration (calibration with SGHMC) and exploitation (final sampling with PCN).
Future work could involve extending our method to more complex problems (c.g., spatiotemporal data) and complex network structures (e.g., graph neural networks). Additionally, future research could focus on improving the emulation step by optimizing the DNN architecture. Finally, our method could be further improved by embedding the sampling algorithm in adaptive framework similar to the method of Zhang ct al. 2018).
Acknowledgements
The authors thank the Editor, Action Editor, and anonymous reviewers for their insightful suggestions and constructive feedback, which significantly improved the article. This work supported by NSF grants NCS-FR-2319618 and DMSS2134256.
References
Sungjin Ahn, Babak Shahbaba, and Max Welling. Distributed stochastic gradient MCMC. In International conference on machine learning, pp. 1044 1052. PMLR, 2014.
Julyan Arbel, Konstantinos Pitas, Mariia Vladimirova, and Vincent Fortuin. A primer on Bayesian neural networks: review and debates. arXiv preprint arXiv:2309.16314 2023.
Barry Becker and Ronny Kohavi. Adult. UCI Machine Learning Repository, 1996. doi: 10.24432 2222//5522200
19

---

Published in Transactions on Machine Learning Research (09/2024)
Duane L Beekly, Erin M Ramos, Gerald van Belle, Woodrow Deitrich, Amber D Clark, Mary E Jacka, Walter A Kukull, et al. The national Alzheimer's coordinating center (NACC) database: an Alzheimer discase database. Alzheimer Discase & Associated Disorders, 18(4):270-277, 2004.
Duane L Beekly, Erin M Ramos, William W Lee, Woodrow D Deitrich, Mary E Jacka, Joylee Wu, Janene L Hubbard, Thomas D Koepsell, John C Morris, Walter Kukull, et al. The National Alzheimer's Coor- dinating Center (NACC) database: the uniform data set. Alzheimer Disease & Associated Disorders, 21 (3):249-258, 2007.
Thierry Bertin-Mahiiuxx Year Prediction MSD. http://dd...rr/1..4433////// C50K61.
UCI Machine Learning Repository, 2011.
DOI:
Alexandros Beskos. A stable manifold MCMC method for high dimensions. Statistics & Probability Letters, 90:46-52, 2014.
Alexandros Beskos, Gareth Roberts, and Andrew Stuart. Optimal scalings for local Metropolis Hastings chains on nonproduct targets in high dimensions. The Annals of Applied Probability, 19(3):863-88) 888, 2009. doi: 10. 10.1221/088 /2.51/100-APP63.
Alexandros Beskos, Frank J Pinski, Jesús Maria Sanz-Serna, and Andrew M Stuart. Hybrid Monte Carlo on Hilbert spaces. Stochastic Processes and their Applications, 121(10):2201 2230, 2011.
Alexandros Beskos, Mark Girolami, Shiwei Lan, Patrick E Farrell, and Andrew M Stuart. Geometric meme for infinite-dimensionall inverse problems. Journal of Computational Physics, 135:327-351, 2017.
Michael Betancourt. The fundamental incompatibility of scalable Hamiltonian Monte Carlo and naive data subsampling, In International Conference on Machine Learning, pp. 533 540. PMLR, 2015.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncortainty in neural network. In International conference on machine learning, pp. 1613 1622. PMLR, 2015.
Edwin V Bonilla, Kian Chai, and Christopher Williams. Multi-task gaussian process prediction. Advances in Neural Information Processing Systems, 20, 2007.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In International Conference on Machine Learning, pp. 1683-1691, 2014.
Jian Cheng, Pei-song Wang, Gang Li, Qing-hao Hu, and Han-qing Lu. Recent advances in efficient computa- tion of deep convolutional neural networks. Frontiers of Information Technology & Electronic Engineering, 19:64-77, 2018.
Emmet Cleary, Alfredo Garbuno-Inig Shiwei Lan, Tapio Schneider, and Andrew M. Stuart. Calibrate, emulate, sample. Journal of Computational Physies, 424:109716, 2021. doi: 10.1016//.jcp.20200 2002000.09777 109716.
Paulo Cortez, Antonio Cerdeira, Fernando Almeida, Telmo Matos, and Jose Reis. Wine quality. UCI Machine CC56ST.
Learning Repository, 2009. DOL:
Simon L Cotter, Gareth O Roberts, Andrew M Stuart, and David White. Meme methods for functions: modilying old algorithms to make them faster. 2013.
Tiangang Cui, Kody JH Law, and Youssef M Marzouk. Dimension-independers prelihood-informan MCMC. Journal of Computational Physics, 304:109-137, 2016.
Carla Currin, Toby Mitchell, Max Morris, and Don Ylvisaker. Baycsian approach to the design and analysis of computer experiments. Technical report, Oak Ridge National Lab., TN (USA), 1988.
Giuseppe Da Prato and Jerzy Zabezyk. Stochastic equations in infinite dimensions. Cambridge University Press, 2014.
20

---

Published in Transactions on Machine Learning Research (09/2024)
Shaveta Dargan, Munish Kumar, Maruthi Rohit Ayyagari, and Gulshan Kumar. A survey of deep learn- ing and its applications: a new paradigm to machine learning. Archives of Computational Methods in Engineering, 27:1071-1092, 2020.
Li Deng- The mnist database of handwritten digit images for machine learning rescarch. IFEE Signal Processing Magazine, 29(6):141-142, 2012.
Yarin Gal and Zoubin Chahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In International conference machine learning, pp. 1050 1059. PMLR, 2016.
Jacob Gardner, Gcoff Pleiss, Kilian Q Weinberger, David Bindel, and Andrew G Wilson. GPyTorch: Black- box matrix-matrix gaussian process inference with GPU acceleration. Advances in Neural Information Processing Systems, 31, 2018.
Andrew Gelman, Walter R. Gilks, and Gareth O. Roberts. Weak convergence and optimal scaling of random walk Metropolis algorithms. The Annals of Applied Probability, 7(1):110-120, 1997.
Alex Graves. Practical variational inference for neural networks. In Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS111, pp. 2348 2356, Red Hook, NY, USA, 2011. Curran Associates Inc. ISBN 9781618395993.
Ingo Gühring, Gitta Kutyniok, and Philipp Petersen. Error bounds for approximations with deep ReLU neural networks in wP norms. Analysis and Applications, 18(05):803 859, 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 2017.
PMLR, August
Martin Hairer, Andrew M Stuart, and Jochen Voss. Sampling conditioned diffusions. Trends in Stochastic Analysis, 353:159 186, 2009.
David Harrison Jr and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of environmental economics and management, 5(1):81 102, 1978.
Wilfred K. Hastings. Monte Carlo sampling methods using Markov chains and their applications. Biometrika, 57(1))97 109, 1970.
Dave Higdon, Mare Kennedy, James C. Cavendish, John A. Cafeo, and Robert D. Ryne. Combining field data and computer simulations for calibration and prediction. SIAM Journal on Scientific Computing, 26 11:448-666, 2004. doi: 10.1137/100645277332266888
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pp. 5-13, 1993.
Matthew D. Hoffman and Andrew Gelman. The no-u-turn sampler: Adaptively setting path lengths in hamiltonian monte carlo. Journal of Machine Learning Rescarch, 15(47):1503-1623 2014.
Matthew D. Hoffman, David M. Blei, and Francis R. Bach. Online learning for latent Dirichlet allocation. In John D. Lafferty, Christopher K. I. Williams, John Shawe-Taylor, Richard S. Zemel, and Aron Culotta (eds.), Advances in Neural Information Processing Systems (NurIPS), pp. 856 864. Curran Associates, Inc., 2010.
Pavel Iamailov, Dmitrii Podopríkhin, Timur Garipoy, Dmitry P. Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.), Proccedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 876-885. AUAI Press, 2018
Tommi S Jaakkola and Michael Jordan. Bayesian parameter estimation via variational methods. Statistics and Computing, 10:25-37. 2000.
21

---

Published in Transactions on Machine Learning Research (09/2024)
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37, 1999.
Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. Hands- on Bayesian neural networks tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29-48, 2022.
Marc C. Kennedy and Anthony O'Hagan. Bayesian calibration of computer models. Journal of the Royal Statistical Society Series B: Statistical Methodology, (3(3))425-4644 2002. doi: 1411111166-996880000
Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Yongchan Kwon, Joong-Ho Won, Beom Joon Kim, and Myunghee Cho Paik. Uncertainty quantification using Bayesian neural networks in classification: Application to ischemic stroke lesion segmentation. In Medical Imaging with Deep Learning, 2022.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Shiwei Lan, Julia A Palacios, Michael Karcher, Vladimir N Minin, and Babak Shahbaba. An efficient Bayesian inference framework for coalescent-based nouparametric phylodynamics. Bioinformatics, 31 2282-3288 2015.
Shiwei Lan, Shuyi Li, and Babak Shahbaba. Scaling up Bayesian uncertainty quantification for inverse problems using deep neural networks. SIAM/ASA Journal on Uncertainty Quantification, 10(4):1684- 1713, 2022. doi:
Kody JH Law. Proposals which speed up function-space MCMC. Journal of Computational and Applied Mathematics, 262:127-138 138, 2014.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to docu- ment recognition. Proccedings of the IEEE, 86(11):228-2324 1998.
Lingge Li, Andrew Holbrook, Babak Shahbaba, and Pierre Baldi. Neural network gradient Hamiltonian Monte Carlo. Computational Statistics, 34:281-299, 2019.
Faming Liang, Qizhai Li, and Lei Zhou. Bayesian neural networks for selection of drug sensitive genes. Journal of the American Statistical Association, (53 972, 2018.
Haitao Liu, Yew-Soon Ong, Xiaobo Shen, and Jianfei Cai. When gaussian process meets big data: A review of scalable gps. IEEE Transactions on Neural Networks and Learning Systems, 31(11):4405 4423, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference Computer Vision, pp. 3730 3738, 2015.
David JC MacKay. practical bayosian framework for backpropagation networks. Neural Computation, 4 (3):448-41)) 472, 1992.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for Bayesian uncertainty in deep learning. Advances in neural information processing systems, 32, 2019.
Nicholas Mctropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Tellor. Equation of state calculations by fast computing machines. The Journal of Chemical Physics, 21(6): 1087-1092, 1953.
Tom Minka. Divergence measures and message passing. Technical Report 1S9--RR-0005-17 January 2005.
22

---

Published in Transactions on Machine Learning Research (09/2024)
Radford M Neal. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2(11):2, 2011.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Roading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mcasuring calibration in deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.
Jeremy Oakley and Authony O'Hagan. Bayesian inference for the uncertainty distribution computer model outputs. Biometrika, (9(4):769-784) 2002. doi: N0.1093/biomet///
Jeremy E. Oakley and Authony O'Hagan. Probabilistic sensitivity analysis of complex models: A Bayesian approach. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 66(3):751-769, 2004.
Michael P Perrone and Leon N Cooper. When networks disagree: Ensemble methods for hybrid neural networks. In How We Learn; How We Remember: Toward An Understanding Of Brain And Neural Systems: Selected Papers of Leon N Cooper, pp. 342 358. World Scientific, 1995.
Yueqi Ren, Babak Shahbaba, and Craig E Stark. Hierarchical, multiclass prediction of Alzheimer's clinical diagnosis using imputed, multimodal NACC data. Alzheimer's & Dementia, 18:e066698, 2022.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pp. 1278-1281 1286. PMLR, 2014.
Gareth O. Roberts and Jeffrey S. Rosenthal. Optimal scaling of discrete approximations to Langevin diffu- sions. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60(1):255 11:2552668,, 1998.
Matthias W. Seeger, Christopher K. I. Williams, and Neil D. Lawrence. Fast forward selection to speed up sparse gaussian process regression. In International Workshop on Artificial Intelligence and Statistics, pp. 254 261. PMLR, 2003.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. uncertainty. Advances in neural information processing systems, 31, 2018.
Evidential deep learning to quantify classification
Babak Shahbaba, Shiwei Lan, Wesley O Johnson, and Radford M Neal. Split Hamiltonian Monte Carlo. Statisties and Computing, 24(3):339-344 2014.
Jiawci Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. One pixel attack for fooling deep neural networks. IEEE Transactions on Evolutionary Computation, 23(5):828 841, 2019.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer. Efficient processing of deep neural networks: A tutorial and survey. Procoedings of the IEEE, 105(12):2295-2322 2329, 2017.
Wiesje M. van der Flier and Philip Scheltens. Hippocampal volume loss and alzheimer disease progression. Nature Reviews Neurology, 5(7):361-362, Jul 2009.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning (ICMI.111), pp. 681,688, 2011.
Cheng Zhang, Babak Shahbaba, and Hongkai Zhao. Hamiltonian Monte Carlo acceleration using surrogate functions with random bases. Statistics and Computing, 27(6), 2017. ISSN 15731375 doi: 10.1007/ s11222-016-96994
Cheng Zhang, Babak Shahbaba, and Hongkai Zhao. Variational Hamiltonian Monte Carlo via score matching. Bayesian Analysis, 13(2), 2018. ISSN 19316690. doi: 11-114/1--B11600
23

---

Published in Transactions on Machine Learning Research (09/2024)
Appendix A: Comparing Various Deep Learning Techniques for Regression and Classification Problems
Table A1: Performance of various deep learning methods based on regression problems. For ESS, minimum, median, and maximum values are provided parenthesis.
Dataset
Method
MSE
CP
Time (s)
ESS
minESS/s
spdup
Simulated
DNN
0.71
3531
Dataset
Ensemble-DNN
0.74
91.3%
14633
BNN-VI
1.11
88.8%
8108
BNN-Lasso
INN-MC-Dropon
0.83
93.4%
2861
BNN-SGMMC
0.79
41281
(109, 829, 1642)
BNN-SGHMC( 200)
1.48
92.8%
3970
(21. 66, 162)
0.005
2.01
SWAG
0.73
90.0%
13488
(93, 1163, 1542)
0.006
2.61
BNN-RNS-HMC
1.62
87.5%
6016
1510)
BNN-pCN
0.81
89.3%
42523
(107, 814, 1533)
0.002
0.95
(pCN -GHHCC)
[132, 1241, 1615)
FBNN (pCN-pCN)
0.86
85.2%
4631
(94, 975, 1621)
0.020
7.69
FENN (SGHMC-SGHMC)
0.82
90.1%
4497
(137, 1236, 1638)
0.030
11.53
FENN (SGHMC-pCN)
0.71
92.2%
4312
(186, 912, 1606)
0.013
16.33
Wine Quality
DNN
0.43
26
Ensemble
BNN-VI
0.66
39.5%
28
BNN-Lasso
0.63
40.9%
42
ANN-MC-Dropod
0.67
32.3%
12
BNN-SGHMC
0.53
51.3%
505
(111, 837, 1538)
0.219
1.00
BNN-SGHMC((rss 200)
2.75
54.6%
(13, 111, 150)
0.213
0.91
SWAG
0.53
48.2%
97
(98, 1021, 1489)
1.010
4.39
BNN-RNS-HME
(107, 1520)
BNN-pCN
0.65
51.1%
620
(99. 1003, 1532)
0.159
0.69
FENN (PCN-SGHMC)
0.52
32.2%
68
(91. 912, 1533)
1.338
5.95
FBNN (pCN-pCN)
0.65
24.5%
(105, 1087, 1540)
1.567
6,86
FENN (SGHMC-SGHMC)
0.50
40.0%
70
(77, 806, 1536)
1.100
4.78
FENN (SGHMC-PON)
0.52
48.1%
57
(92, 897, 1636)
1.614
7.33
Boston Housing
DNN
3,21
14
Ensemble-De DNN
3.17
72.1%
74
BNN-VI VI
7.60
83.7%
85
BNN-Lasso
6.20
79.2%
68
BNN-MC-Dropout
10.12
81.2%
BNN-SGHMC
3.83
75.3%
888
(76, 649, 1536)
0.085
1.00
BNN-SGHMC(Hnn 200)
7.40
66.9%
(9, 87, 150)
SWAG
5.81
71.9%
104
(68, 724, 1532)
0.653
7.22
BNN-RNS-HME
9.42
73.4%
76
(73, 1032, 1504)
0.960
10.6
BNN-PCN
3.25
79.3%
(76, 649, 1536)
0.084
0.88
FENN (PCN-SGHMC)
4.16
41.7%
186
(71, 965, 1543)
0.381
4.22
FBNN (pCN-pCN)
3.81
47.1%
186
(80, 966, 1541)
FENN (SGHMC-SGHMC)
4.15
48.9%
94
(69, 979, 1542)
0.734
8.22
(SGHMC-PCN)
(93, 938, 1543)
Alzheimer
DNN
0.49
326
Dataset
Ensemble-DNN
0.42
89.3%
1597
BNN-YI
0.53
BNN-Lasso
0.52
83.5%
561
ANN-MC-DropomE
BNN-SGHMC
0.49
91.67%
6524
(102, 973, 1376)
0.015
1.00
BNN-SGHMC(fig 200)
3.17
72.7%
641
(7, 82, 150)
0.011
0.69
SWAG
0.74
89.3%
1214
(106. 1002, 1542)
0.087
5.58
BNN-RNS-HME
0.61
92.1%
7321
(96, 892, 1531)
0.013
0.83
BNN-PCN
(120,
1448)
FENN (pCN-SGIL (PCN-SGHMC) SGHMC)
0.50
90.2%
643
(116, 994, 1504)
0.180
11.53
(PCN-pCN)
0.56
(108, 998, 1498)
10.93
FENN (SCHMC-SGHMO)
0.55
88, 47
671
(118, 1012, 1541)
0.176
11.24
FBNN (SGHMC-PCN)
0.48
91.6%
632
(149, 984, 1497)
0.218
13.97
DNN
Dataset
Ensemble-DNN DNN
72.89
90.47%
7929
BNN-VI
78.21
88.43%
2146
BNN-Lasso
79.11
89.01%
3243
BNN-MC-Drood Dropout
83.08
87.89%
1287
BN SGHMCC
81.67
83.81%
25533
(122, 1005. 1540)
0.004
1.00
BNN-SGHMCI first 200)
105.92
94.13%
2613
[7, 84, 149)
0.002
0.56
SWAG
93.87
(113, 1537)
WIN-RNS-HMC
02.82
80.19%
17289
(109, 925, 1563)
0.006
1.57
BNN-PCN
82.45
85.74%
25635
(124, 873, 1631)
0.004
1.20
FENN (PCN-SGHMC)
74.92
88.73%
2815
(143, 1049, 1676)
0.050
10.63
PENN (pCN-pCN)
74.03
90.45%
3046
(138, 992, 1618)
0.045
9.48
PBNN (SCHMC-GHMM)
76.69
90.40%
2974
(146, 973, 1599)
0.049
10.27
FENN (SGHMC-PCN)
73.41
92.23%
2724
(156, 934, 1608)
0.057
11.98
24

---

Published in Transactions on Machine Learning Research (09/2024)
Table A2: Performance of various deep learning methods based on classification problems.
Dataset
Method
Acc
Time(s)
ESS(min, (eSSmin,med,maa med,
minESS/s
spdup
ECE
Simulated
DNN
96%
4257
Dataset
Ensemble- DNN
97%
17415
0.382
BNN-V VI
90%
4275
0.399
RAN--Lass
BNN-MC-Dropout
87%
2912
0.277
BNN-SGHMC
95%
43841
1459)
0.001
0.471
BNN-SGHMC(Bis 200)
83%
1218
(21, 59, 156)
0.004
1.61
0.498
SWAG
81%
4731
[81, 773, 1308)
0.017
16.97
0.482
BNN-RN
69%
(135, 1493)
0.513
BNN-PCN
91%
49774
(36. 207, 1417)
0.001
0.67
0.475
FENN GHHMC)
93%
5179
(134. 959. 1419)
0.051
24.13
0.409
FENN (pCN-pCN)
FENN (SCHMC-SGHMC)
95%
4617
(149, 891. 15401
0.032
30.76
0-406
PBNN (SGHMC-PCCV
96%
4489
(154, 1602)
0.070
32.00
0.396
Adult Dataset
DNN
85%
425
Ensemble-DNN
84%
2153
0.556
Lass-
83%
256
0.631
ENN-MC-DropHe
82%
187
0.540
BNN-SGHMC
83%
(16, 202, 1520)
BNN-SGHMCY 200)
78%
581
(1. 11, 148)
0.002
0.95
0.594
SWAG
79%
1641
(47. 912, 1532)
0.028
10.70
0.668
BNN-RNS-HLN HMCC
72%
6110
(89, 960, 1530)
0.014
5.44
0.658
BNN-pCN
83%
6227
(9, 117, 1518)
0.001
0.54
0.616
FENN (PON-SGHMC)
82%
642
1639)
0.135
50.63
0.680
FBNN (pCN-pCN)
82%
639
(88. 890, 1540)
0.137
51.46
0.592
FENN (SCHMC-SGHME
83%
612
(68, 941, 1541
0.111
41.52
0.583
FENN (SGHMC-pCN)
875,
Alsheimer
DNN
82%
51
Dataset
Ensemble-DNo
83%
262
0.542
BNN-VI
72%
0.546
BNN-Lasso
76%
256
0.524
BNN-MC-Dropout
0.129
BNN-SGHMC
81%
2736
(81, 588, 1526)
0.029
1.00
0.499
BNN-SGHMC 200)
69%
282
(8, 84, 149)
0.028
0.96
0.523
SWAG
913, 1562)
BNN-RNS-HHM HMC
58%
293
(84, 915, 1540)
0.286
9.55
0.521
BNN-PCN
73%
2660
(71. 424, 1534)
0.026
0.90
0.469
FBNN (PON-SGHMC)
70%
(76. 947, 1542)
0.274
9.26
0.508
FANN (pCN-pCN)
77%
274
(70. 931, 1542)
0.255
8.33
0.377
FBNN (SCHMC-SGHIM)
80%
973.
0.291
0-448
FBNN (SGHMC-pCN)
84%
280
192, 914, 1535)
0.328
11.09
0.376
Mnist Dataset
DNN
02%
231
Ensemble-
BNN-VI
86%
273
0.417
BNN-Lasso
87%
184
0-445
BNN-MoCDropout
89%
212
0.328
BNN-SGHMC
90%
8641
(15, 364, 1456)
0.001
1.00
0.280
BNN-SGHM 200)
78%
(1,
SWAG
83%
1294
(15. 431, 1376)
0.011
7.72
0.327
BNN-RNS-HMC
69%
4541
[14, 372, 1394)
0.003
2.05
0.349
BNN-pCN
(17,
FENN MOON-SGHMC
58%
927
(15. 412, 1383
0.016
10.78
0.352
FENN (PCN-CCN)
90%
931
166 393, 1421)
0.017
11.43
0.312
FENN (SCHMC-SGHMC)
91%
923
118 109, 1461)
0.019
13.01
0.283
FENN (SGHMC-pCN)
94%
914
233, 474, 1521
0.025
16.77
0.241
DNN
Dataset
Ensemble-DNN
82%
15445
0.569
BNN-VI
80%
1132
0.622
BNN-Lasso
79%
0.561
BNN-MC MAN-MC--DN
MC-Dropout
78%
1641
0.512
SGHC.
81%
17234
(19, 383, 1537)
0.001
1.00
0.567
BNN-SGHMCY(icc 200)
69%
1849
(1, 85, 149)
0.001
0.63
0.642
SWAG
70%
8913
(85, 1014, 1467)
0.009
8.65
0.534
BNN-RNS-HMMC
72%
1835
1494)
0.612
BNN-PCN
76%
19676
(19. 331, 1538)
0.001
0.95
0.529
FENN (CCN-SGHMC)
80%
1972
(99. 1155. 1542!
0.050
45.53
0.565
FENN DCN)
(116, 1155, 1542)
FENN (SGHMC-SGHHM
81%
1901
(93, 978, 1544)
0.048
44.30
0.568
FBNN (SGHMC-PCN)
85%
1892
(129, 785, 1517)
0.068
61.84
0.493
SVHN Dataset
DNN
96%
3748
Disemble-DNN
97%
17415
0.382
BNK-VI
ENN-Lasso
92%
3189
0.303
ONN-MC-Dropp
87%
2912
0.277
HNN-SGHMC
(31, 379, 1528)
BNN 200)
74%
2515
(5. 67, 262)
0.001
1.19
0.246
SWAG
63%
6204
499, 1383)
0.012
7.45
0.282
BNN-RNS-HMC
71%
410, 1384)
0.002
1.25
0.299
BNN-PCN
93%
17812
(27. 308 1317)
0.001
0.91
0.275
FENN (PCN-SGHMC)
0.018
0.263
FENN (CCN-pCN)
91%
1927
29. 372. 1280)
0.015
9.04
0.318
FENN ISCHMC-SGHMM
96%
1891
(31. 412. 1161)
0.017
10.81
0.218
FBNN (SGHMC-PCN)
(47, 486, 1533)
25


Reviews:
[{"review_id": "zAhgdI1Yth", "reviewer": "Reviewer_wiy7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper provides a method that practically improves the efficiency of an uncertainty quantification method, while preserving the performance.\n- The paper is self-contained and provides the necessary background for technique involved in the proposed method.\n\nWeaknesses: I am mainly concerned about the amount of contributions: the proposed FBNN (SGHMC-pCN) is modified based on the CES framework in prior work. The derivations for the theorems are standard, and the experiments are limited to small-scale datasets only, which the paper also acknowledges.\n\nSome questions please:\n- eq (3): where is $\\mathcal{G}_n$ defined?\n- \"It should be noted that not all neural networks that represent uncertainty are Bayesian or even stochastic; some employ deterministic methods to estimate uncertainty without relying on stochastic components or Bayesian inference.\" -- please add references.\n- Sec 3.2, calibration: could you justify why early stopping?\n- Sec 3.5: $\\alpha$ seems to be a $d$-dimensional vector; if that's the case, what is $|\\cdot|$ in $\\|\\alpha|$?\n- Sec 4.1: Why are BNN methods trained from a randomly chosen initial point for MCMC, whereas FBNN methods use warm starts? Is this unfair comparison, or is there some reason why warm starts can't be used for BNN?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**TL;DR**: this paper aims to improve the computational efficiency of Bayesian neural networks (BNNs), using an efficient implementation of calibration-emulation-sampling (CES). The proposed method improves efficiency while preserving the performance.\n\nSpecifically, each step of CES is modified as follows for improved efficiency:\n- Calibration: the paper uses the stochastic gradient Hamilton Monte Carlo (SGHMC) algorithm to collect posterior samples for emulator training. The SGHMC is run only for a limited number of iterations (i.e. early-stopped) for efficiency.\n- Emulation: The paper uses Fast Bayesian Neural Network (FBNN), a lightweight NN, as the emulator that maps from parameters to probabilities. The training data for FBNN is collected from the original Bayesian NN which is expensive to evaluate.\n- Sampling: the paper chooses to use preconditioned Crank-Nicolson (pCN), which minimizes the correlations between samples and efficiently traverses the parameter space.\n\nFor theory, the paper shows that neural network emulators with size $\\tilde{O}(\\epsilon^{-d/(n-1)})$ that can the likelihoods and posteriors up to $\\epsilon$ error in Hellinger distance.\n- Here $d$ is the dimension of the parameter vector, $n$ is an upper bound on the parameter norm.\n\nEmpirically, the paper verifies the effectiveness of the proposed method on 10 synthetic and real-world datasets with both regression and classification tasks.\n- Regarding efficiency, the original BNN requires 2000 samples, whereas FBNN can achieve comparable performance using only 200 samples. Overall, the proposed FBNN (SGHMC-pCN) has a 7 to 60 times speedup over the baseline BNN-pCN.\n- The paper also compares with different variants where SGHMC and pCN are used at different steps of CES, and finds that the proposed method (i.e. SGHMC for calibration and pCN or sampling) is the most efficient.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please consider experiments with larger scale and higher dimensional data, or have a more thorough discussions on the difficulty of scaling up, such as the expected scaling behaviors.\n\nFor writing, it would be better if the presentation could be more concise, such as summarizing the main findings from the experiments.", "broader_impact_concerns": "There are no direct ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2jRwdk8XiX", "reviewer": "Reviewer_Szsx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The paper provided extensive experimental results on both simulated and real datasets in this revision and included metrics for both calibration and accuracy. \n\n- The proposed method addressed an important problem of BNN by speeding up the computation.\n\nWeakness:\n\n-  The datasets used are still cross-sectional data with low-dimensional features. Instead of BNN, classic machine learning models should also perform well. \n\n- The definition of speed up is weird by comparing the effective sample size per second. How should it be interpreted as training / inference efficiency? Why deep ensemble is not compared in the speed up aspect. \n\n- The description of the method is sketchy. Especially, it is unclear how the emulation step was conducted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed a novel method to enhance the computational efficiency of Bayesian Neural Networks (BNNs) by Calibration-Emulation-Sampling (CES) strategy. The authors introduced a Fast Bayesian Neural Network that leverages Deep Neural Networks (DNN) for the emulation component. Their methodology includes a preconditioning Crank-Nicolson (pCN) algorithm for efficient sampling from posterior distributions. The paper demonstrates the effectiveness of FBNN through extensive experiments on both simulated and real datasets, showing enhanced computational efficiency while maintaining prediction accuracy and robust uncertainty quantification.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I suggest the author provide more details for the method section, and ideally make some illustrations to support the text.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wENxycHiB5", "reviewer": "Reviewer_Qk3U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n- The paper is clearly written in general. \n- The proposed idea seems technically sound. \n\nWeaknesses:\n- While the main selling point of the paper seems to be on scaling and speeding up BNNs, the experiments were conducted on very small scale datasets. \n- Some of the benchmark methods like MC dropout and SWAG can be applied to very large scale datasets with very deep neural networks. Based on my understanding, with the use of emulator, it can be hard to apply the proposed strategy to larger scale datasets. As such, I am not sure how much of a benefit the proposed speed up can bring in practice.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors of the paper proposes a method to accelerate Markov Chain Monte Carlo based Bayesian neural networks (BNNs). The authors introduce a three-step Calibration-Emulation-Sampling (CES) strategy. First, a calibration step is conducted, where the authors of the paper use a standard stochastic gradient MCMC method to gather posterior samples from the BNN's parameter space. These samples enable us to train an emulator, or a deterministic neural network that approximates the likelihood computation of the stochastic BNN, which can be computationally expensive. With the emulator in hand, we can then speed up the BNN. Experimental results demonstrate that the CES approach significantly speeds up the process compared to traditional stochastic gradient MCMC methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be interesting to look at how accurate the emulator prediction is empirically. \n- What is the architecture used for DNN/BNN and the emulator? How many parameters are there? How is the DNN trained? Such experimental details are important for reproducibility. \n- The accuracy achieved by DNNs for MNIST seems very low compared to other common implementations. Any explanation for this?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors addressed the reviewers' concerns, and all reviewers found that the paper satisfies both criteria for TMLR publication.
Decision: Accept (poster)