Sparsifying Bayesian neural networks with latent binary variables and normalizing flows
Authors: Lars Skaaret-Lund
Geir Storvik
Aliaksandr Hubin
Venue: Accepted by TMLR ()

Abstract: Artificial neural networks are powerful machine learning methods used in many modern applications. A common issue is that they have millions or billions of parameters, and therefore tend to overfit. Bayesian neural networks (BNN) can improve on this since they incorporate parameter uncertainty. Latent binary Bayesian neural networks (LBBNN) further take into account structural uncertainty by allowing the weights to be turned on or off, enabling inference in the joint space of weights and structures. Mean-field variational inference is typically used for computation within such models. In this paper, we will consider two extensions of variational inference for the LBBNN: Firstly, by using the local reparametrization trick (LCRT), we improve computational efficiency. Secondly, and more importantly, by using normalizing flows on the variational posterior distribution of the LBBNN parameters, we learn a more flexible variational posterior than the mean field Gaussian. Experimental results on real data show that this improves predictive power compared to using mean field variational inference on the LBBNN method, while also obtaining sparser networks. We also perform two simulation studies. In the first, we consider variable selection in a logistic regression setting, where the more flexible variational distribution improves results. In the second study, we compare predictive uncertainty based on data generated from two-dimensional Gaussian distributions. Here, we argue that our Bayesian methods lead to more realistic estimates of predictive uncertainty.

Published in Transactions on Machine Learning Research (10/2024)
Sparsifying Bayesian neural networks with latent binary vari- ables and normalizing flows
Lars Skaaret-Land
lareskaarre--undddmmb..
Bioinformatics and Applied Statistics Norwegian University of Life Sciences
Geir Storvik
geirs Smath. uio.
Department Mathematics University of Oslo
Aliaksandr Hubin
aliaksandr www.eeanndrrrbbbbreeeeeeeeeeeeee www.bbbiineemmm........o
Bioinformatics and Applied Statistics Norwegian University of Life Sciences
Reviewed on OpenReview: https: opw.reveeep.....e... net/ forum? id- d6kqUK=G3V
Abstract
Artificial neural networks are powerful machine learning methods used in many modern applications. A common issue is that they have millions or billions of parameters and tend to overfit. Bayesian neural networks (BNN) can improve on this since they incorporate paramoter uncertainty. Latent binary Bayesian noural networks (LBBNN) further take into account structural uncertainly by allowing the weights ence in the joint space of weights and structures. Mean-field variational inference typically used for computation within such models. In this paper, will consider two extensions of variational inference for the LBBNN: Firstly, by using the local reparametrization trick (T.CRT), we improve computations ional efficiency. Secondly, and more importantly by using normalizing flows on the variational posterior distribution of the LBBNN parameters, we learn a more flexible variational posterior than the mean field Gaussian. Experimental re- sults on real data show that this improves predictive power compared to using mean field variational inference on the LBBNN method, while also obtaining sparser networks. We also perform simulation study, where consider variable selection in a logistic regression setting with highly correlated data, where the more flexible variational distribution improves results.
turned on or off, enabling infer-
1 Introduction
Modern deep learning architectures can have billions of trainable parameters (Khan et al. 2020). Due to the large number of parameters in the model, the network can overfit, and therefore may not generalize well unseen data. Further, the large number of parameters gives computational challenges both concerning training the network and for prediction. Various regularization methods are used to try to deal with this, such early stopping (Prechelt 1998), dropout (Srivastava et al., 2014)) or data augmentation (Shorten & Khoshgoftaar 2019). These techniques are heuristic and therefore it is not always clear how to use them and how well they work in practice.
Another issue with deep learning models that they often make overconfident predictions. In Szegedy el al. (2013), it was shown that adding a small amount of noise to an image can trick a classifier into making a completely wrong prediction (with high confidence), even though the image looks the same to the human eyc. The opposite also possible, images that are white noise can be classified with almost complete certainty to belong to specific class (Nguyen al. 2015).

---

Published in Transactions on Machine Learning Research (10/2024)
256
24 F
244
21 22
222
Figure 1: A dense network the left, one possible sparse structure the right.
Bayesian neural networks (BNN, Neal 1992 MacKay 1995: Bishop 1997 use a rigorous Bayesian method- ology to handle parameter and prediction uncertainty. In principle, prior knowledge can be incorporated through appropriate prior distributions, but most approaches within BNN so far only apply very simple convenience priors (Fortuin 2022)) However, approaches where knowledge-based priors are incorporated are lead to
starting to appear (Tran et al.) 2022 Sam et al., 2024). Incorporating parameter uncertainty less overfitting and better-calibrated predictions, however, this comes at the expense of extremely high com- putational costs. Until recently, inference on Bayesian neural networks could not scale to large multivariate data due to limitations of standard Markov chain Monte Carlo (MCMC) approaches, the main quantitative procedure used for complex Bayesian inference. Recent developments of variational Bayesian approaches Gal 2016) allow us to approximate the postcrior of interest and lead more scalable mothods.
In the frequentist setting, the lottery ticket hypothesis (Frankle & Carbin 2018: Pensia et al. 2020) states that dense networks contain sparse subnetworks that can achieve similar performance of the full network. These sparse subnetworks are typically obtained by threshold pruning of the weights. Similarly, in the Bayesian framework, Deng et al. (2019) obtain spanse networks with magnitude-based weight, pruning. In a more recent work, Li et al. (2024) introduce a methodology where sparsity is inherent in the posterior from the very beginning of training, significantly saving on computational costs. In that work, the (non-Bayesian) inclusion parameters are also pruned, (and reintroduced back) using some pruning criteria. Another way obtain sparse networks in the Bayesian framework is through sparsity-indueing priors. Variational dropout KKigmma et al. 2015 Molchanov et al., 2017) uses the independent log uniform prior on the weights. However, this is an improper prior, which in combination with commonly used likelihood functions leads to an improper posterior, therefore (Hron et al. 2017)) argues that the obtained results can not be interpreted as a Bayesian methodology to sparsification, but rather falls into the same category as froquentist approaches. Another type of sparsity-inducing prior the independent scale mixture prior, where Blundell et al. (2015) proposed a mixture of two Gaussian densities, where using a small variance for the second mixture component leads to many of the weights having prior around zero.
In this work, consider the andsl- prior, used in latent binary Bayesian neural networks (LBBNN) introduced by Hubin & Storvik 20019 2024) and concurrently in Bai et al. (2020), resulting in formal Bayesian approach for obtaining sparse subnetworks. The spike-and-slab prior for a special case of LBBNN
with the ReLu activation function was studied from a thcoretical perspective in Polson & Ročková (0188) In Hubin & Storvik (2019) was empirically shown that using this prior will induce a very sparse network (around 90 % of the weights were removed) while maintaining good predictive power. Using this approach thus lakes into account uncertainty around whether each weight is included or not (structural uncertainly) and uncertainty in the included weights (parameter uncertainty) given structure, allowing for Bayesian approach to network sparsification (see Figure 1b, thus weights are removed based on (learned) inclusion probabilities rather than ad-hoc pruning.
In Hubin & Storvik 2019 2024;, a simple mean-field variational approximation was applied for performing computation. In this paper, we show that transforming the variational posterior distribution with normalizing flows can result in even sparser networks while improving prodictive power compared to the earlier work. Additionally, we demonstrate that the flow network handles predictive uncertainty well, and performs better
2

---

Published in Transactions on Machine Learning Research (10/2024)
than the mean-field methods at variable selection in logistic regression setting with highly correlated variables, thus demonstrating higher quality in structure learning. While the utility of LBBNN type models have been demonstrated in earlier papers (Polson & Ročková 2018 Bai et al. 2020: Hubin & Storvik 2019 2024h, the main contributions in this paper are:
extending the class mean-field variational distributions in the LBBNN context to include nor- malizing flows, allowing for modeling dependencies between weights while maintaining the network sparsification capabilities;
improvements in computational efficiency and bypassing the need for continuous relaxation of binary variables in LBBNNs through the use of the local reparametrization trick (Kingma al., 2015), by sampling the lower dimensional preactivations directly rather than the two much larger sets of weight and inclusion probability parameters;
demonstrating in most cases improvements in predictive power, sparsity, and variable selection of the normalizing flow-based approximate inference through experiments real and simulated data compared to the most relevant baseline method, LBBNN with mean-field inference;
demonstrating robust performance in predictive uncertainty quantification across multiple problems with LBBNN producing the best results for many datasets.
2
The model
Given the explanatory variable æ E R", for the response variable R= we assume a probability distribu- tion ((:n(x)) parameterised by the vector ()x),
y~ f((n())))
The vector ())) is obtained through a composition of semi-affine transformations:
(111
- Σ u
= 1
(1-1) 11)
wi + (1) , 1, 1,,
11) 100111114 1,...,,,
(1)
2=1
where ()( u, Additionally, 1ll-1) denotes the inputs from the previous layer (with a corre- sponding to the explanatory variables), the wh w0 are the weights, the oft's are the bias terms, and (1) (and (0) = n) the number of inputs at layer 1 of a total L layers. Further, we have the elementwise non-linear activation functions (1) The additional parameters Tif (0, denote binary inclusion variables for the corresponding weights.
(1)
Following Polson & Ročková 2018);; Hubin & Storvik 2019 ; Bai et al. (0200) we consider structure to be defined by the configuration of the binary vector 7, and the weights of cach structure conditional on this configuration. To consider uncertainty in both structures and weights, we use the spike-and-slab prior, where for each (independent) layer of the network, we also consider the weights to be independent:
(1) (1) p(w!) Mi5 ) = Pij N(w
(1)
@ 0, (a()))) + (1 (1-7%3 (00000))0))
p(7
(1) Bernoulli(7 (1) ()))
We will use the nomenclature from Hubin & Storvik (0019) and refer to this the LBBNN model. Here, 8(-) is the Dirac delta function, which is considered to be zero everywhere except for spike at zero. In addition, o² and a denote the prior variance and the prior inclusion probability of the weights, respectively. In practice, we use the same variance and inclusion probability across all the layers and weights, but this is not strictly necessary. In principle, one can incorporate knowledge about the importance individual covariates or their co-inclusion patterns by adjusting the prior inclusion probabilities for the input layer or specifying hyper-priors. This common in Bayesian model selection literature (Fletcher & Fletcher 2018 but not yet within BNNs.
3

---

Published in Transactions on Machine Learning Research (10/2024)
3 Bayesian inference
The main motivation behind using LBBNNs is that we are able to take into account both structural and parameter uncertainty, whereas standard BNNs are only concerned with parameter uncertainty. By doing inference through the posterior predictive distribution, we average over all possible structural configurations, and parameters. For new observation ÿ given training data, D, we have:
p D) = =E p(ulw, D)p(w, 7DD) du.
This expression intractable due the ultra-high dimensionality of w and Y and using Monte Carlo sampling approximation is also challenging due to the difficulty of obtaining samples from the posterior distribution, p(w, 7/D). Instead of trying to sample from the true posterior, problem, using variational inference (VT, Blei et al. 2017). The key idea is that we replace the true posterior distribution with an approximation, qo (., y), with O denoting some variational parameters. We learn the variational parameters that make the approximate posterior as close as possible to the true posterior. Closeness is measured through the Kullback-LLeiblrr (KT.) divergence,
turn it into optimization
KT (ur, ))lp, /|D) = E go 7) log go(w, 7) dur. p(w, 7/D)
(2)
Minimizing the KL-divergence (with respect to 0) is equivalent to maximizing the evidence lower bound (ELBO):
ELBO(ge) = (w)() [log w. 7) - KL [go >)llp(w, y]]-
(3)
The objective is thus to maximize the expected log-likelihood while penalizing with respect to the KL divergence between the prior and the variational posterior. How good the approximation becomes depends on the family of variational distributions (go. A E O} that is chosen.
3.1 Choices of variational families
A common choice (Blundell et al. 2015) for the approximate posterior in (dense) Bayesian neural networks is the mcan-ficld Gaussian distribution. For simplicity of notation, denote now by W the set of weights corresponding to a specific layer. Note that from here on, we drop the layer notation for readability, since the parameters at different layers will always be considered independent in both the variational distribution and the prior. Then
Nin Rost
le W)
II (wny ot2); ==
where Ain and Rout denote the number of neurons in the previous and current layer, respectively. Weights corresponding to different layers are assumed independent as well. The mean-field Gaussian distribution for Bayesian neural networks can be extended to include the binary inclusion variables following Carbonetto & Stephens 20122;
Riez Tout
To W D)
II
(MyyN(wis)) Prj, 02)) + -11--1)) +1-11(1)))((()) ((m,)) j=1
ęa, (7t) (arooull(na;;ū0
Here, T is the set of inclusion indicators corresponding to specific layer. However, the mean-field Gaussian distribution (Blundell et al. 2015) is typically too simple to be able to capture the complexity of the true posterior distribution. We follow Ranganath et al. (2016), and introduce a set of auxiliary latent variables z to model dependencies between the weights q, and use the following variational posterior distribution:
nin Rout
40 [W]T,))
IIII
(Ny.(wyg apg. 03T) + (1 7))))(w))))
(5)
1=1 j=1
gā Bernoulli(xy: axy)
4

---

Published in Transactions on Machine Learning Research (10/2024)
Zi
pa
12
143
141
A42
P43
o 2
2
0
o
i1
83
Wil
W12
W:3
Wil
W022
Wi3
C1+1
0442
C1:3
CG11
Ox
0x13
Figure 2: On the left, the mean-field variational posterior where the weights are assumed independent. On the right, the latent variational distribution allows for modeling dependencics between the weights.
where 2 = (21, follows a distribution (z)... For an illustration of the difference between the two vari- ational distributions in Equation (4 and Equation 16 see Figure 2 The novelty in our suggested variational distribution to combine both weight and structural uncertainty, addition to modeling dependencies be- tween the weights. As for W, also z is a of variables related specific layer and independence between layers is assumed also for z's. To increase the flexibility of the variational posterior, apply normalizing flows (Rezende & Mohamed, 2015) to (=(=). In general, a normalizing flow is composition of invertible transformations of some initial (simple) random variable
Z fx(#x-1); k L,.,,
The log density of the transformed variable z = zK is given as,
K
log det 8zk
log 9(2) = log (0(20)
Σ
(6)
Ozk-1
We typically interested in transformations that have Jacobian determinant that is tractable, and fast to compute, in addition to being highly flexible. Transforming the variational posterior distribution in a BNN with normalizing flows was first done in Louizos & Welling (2017). who coined the term multiplicative normalizing flows (BNN-FLOW), where the transformations were applied in the activation space instead of the weight space. As the weights are of much higher dimensions, the number flow parameters and thus the number of parameters of variational distribution would explode quickly. We will follow Louizos & Welling (017)) here. The main difference in our work is that by using the variational posterior in Equation ( we also get sparse networks.
For the normalizing flows, we will use the inverse autorogressive flow (IAF), with numerically stable updates, introduced by Kingma et al. 2016). It works by transforming the input in the following way:
2k-1 input
my, g(zx-1)
(7)
Kg sigmoid(8s)
Nt = Kb 2 2k-1 + K4) ② mk

---

Published in Transactions on Machine Learning Research (10/2024)
where 9 is neural network and @ denotes elementwise multiplication. Assuming the neural network in Equation 7 is autoregressive (i.e Zk,i can only depend on 11,1:i-),, we get a lower triangular Jacobian and nmn
8zk
log det
Σ
log Kg,i
3.2 Computing the variational bounds
Minimization of the KL in Equation ( is difficult due to the introduction of the auxiliary variable x in the variational distribution. In principle, z could be integrated out, but practice this is difficult. Following Ranganath et al. (2016), we instead introduce 2 as an auxiliary variable also in the posterior distribution by defining
p(w, 7, =|D) p(w, (D)r
where r(//w, 7) in principle can be any distribution. We then consider the KL divergence in the extended space for (w, 7, z):
KL 7, y. x|D)] - / q(w, y, 2) 2) log 7,2) dwdz
which, by utilizing the definitions of p(w, 7, x) and g(w, 7, 2) can be rewritten to
KL [g(w, y, 2)
7. (,,,
(w.r..)..) (og)(DDuu w 7) logg (ogr(2/ 7) + logp(D).
KT. [ >l=)[p((w, >)] + log ((2)
(9)
It follows directly from the chain rule for relative entropies (sce Cover 1999
Theorem 2.5.3) that
KL [g ",wp,, S KL [g(w,, 7, 2) |p( w. >D))
(10)
giving looser than the original upper bound, but the dependence structure in the variational postcrior distribution can compensate for this.
After doing some algebra, we get the following contribution the first term within the first expectation in Equation (9) from spocific layer:
KL [<], >=))]p((,,,))) = Σ ((ass/
log
+ log مss an
+ o 11,,(0,2100 +(1 aij) log 1-āg I-asy
2
20
Since we use autoregressive flows, the contribution to the second term in the first expectation simplifies to
nin
log (2) = (2
log Kk.
For the specific choice of r(2lw,, Y), follow Louizos al. 2017 in choosing
Rin
rB($B|w, w, y) = N(,, 77)
We define the dependence v = (V1, Vin) and 7== = (ff.... = 222 on w and y similar to Louizos & Welling
20177::
n (di 1
with <(e (w 3 7))
(11)
log 2 = nー (2

---

Published in Transactions on Machine Learning Research (10/2024)
Here, d1, d2 and e trainable parameters with the same shape For S, use hard-tant
opposed to tanh (used in Louizos & Welling (2017) as this works better empirically. For the second last term of Equation g we thus have:
dzB Oz
logr (z/w,7) logra (znlw,>) 7) + log det
This means that we must use two normalizing flows, one to gel from Zo to 2K, and another from z to BB. Here, we have used the inverse normalizing flow of the form 7 with only one layer, but this can, in general, be extended to an arbitrary number of them just like in Equation
For the biases of a given layer, we assume they are independent of the weights, and each other. We use the standard normal prior with the mean-field Gaussian approximate posterior. As do not use normalizing flows the biases, only need to compute the KL-divergence between two Gaussian distributions:
0b, + (Pby - 0)2
KL [d(b)]pp(b)]
Σ
log
Obe Ōbes
20
In practice, the ELBO is optimized through a (stochastic) gradient algorithm where the reparamctrization trick (Kingma & Welling 2013) combined with mini-batch is applied. This involves sampling the large P and W matrices.
4 Combining LBBNNs with the LCRT and MNF
We use approximation combining the local paramctrization trick (LCRT, Kingma et al. 2015 with the multiplicative normalizing flow method. This means sampling is based on the pre-aclivations hj
by Wiy (where for} denotes the outputs from the previous layer). The mean and the variance of the activation hy are:
E[hy] iuo, +
Σ ougūr
N
Varlh,l = 0h + Σ
(0 + + - oz
It should be noted that affects both the mean and the variance of our Gaussian approximation, whereas in Louizos & Welling (2017) it only influences the mean. Louizos & Welling (017 also sample one z for each obsorvation within the mini-batch. We found that empirically made no difference in performance only sample one vector and multiply the same with each input vector. We do this, as it is more computationally efficient.
A simplified version only applying the LCRT method assumos 2; for all i. We will denote this method by LBBNN-LCRT in section 5 The method combining both LCRT and MNF will be denoted LBBNN-FLOW.
In addition to a huge reduction in the number of variables to be sampled, we get a reduction in the variance of the gradient estimates, as shown Kingma al. (2015). Note also that the approximations induced by the sampling procedure for h can be considered an alternative variational approximation directly for (D))

---

Published in Transactions on Machine Learning Research (10/2024)
ANN
Weight regularization >
ANN L2
Parameter uncertainty
BNN
Structural uncertainty
LBBNN
More flexible variational distribution
Computational gains
BNN-FLOW
LBBNN-LCRT
Structural uncertainty
More ficxible variational distribution
LBBNN- FLOW
Figure 3: Illustration of the relations between the different methods considered in this paper. Exactly one design change is present between all direct neighbors disregarding the directions of the edges, thus allowing us to interpret the edges as treatments between specific pairs of methods.
5 Experiments
In this section, we demonstrate the robustness of our approach and show improvements with respect to the closest baseline methods of Hubin & Storvik (2024), (denoted LBBNN-SSP-MN in their paper), denoted LBBNN here, with the two approaches proposed in this paper. We consider both Bayesian model averag- ing (Hoeting et al. 1999) over the variational posterior distribution for the inclusion variables and weights, as well as the use of the median probability model (Barbieri et al. 2018), only sampling from weights with posterior probabilities for inclusion variables above 0.5. Median probability models have the potential to give significant sparsity gains. We also compare it other reasonable baseline methods that allow us to evaluate the direct treatment effects. Our aim is to demonstrate the potential for improvements by the use of the proposed method, focusing on comparison under similar settings. For all methods, further improvements are possible through extensive tuning of prper-paraeeters and different ad-hoc tricks commonly found in the Bayesian deep lcarning literature that are utilized to improve predictive performance (c.g. tempering the posterior (Wenzel et al.) 2020) or clipping the variance of the variational posterior distribution as done in Louizos & Welling (2017) Using these tricks (although tempting) would not allow us to evaluate the pure contribution of the methodology. For the classification experiments, we provide comparisons to a stan- dard frequentist neural network (ANN) without regularization, corresponding the maximum likelihood estimation of the network weights, in addition to one with T.2 regularization (ANN + T.2), corresponding to the maximum a posteriori estimator (MAP) with independent Gaussian priors from standard Bayesian neural network (BNN). We also have standard BNN, taking weight uncertainty into account. From there, we get to the LBBNN method by having an extra inclusion parameter per weight, allowing for a sparse BNN. For LBBNN, exactly the same parameter priors (slab components) as in BNN were used, allowing us to evaluate the effects adding the structural uncertainty. The multiplicative normalizing flow method (BNN-FLOW) is also closely related to a standard BNN, but here instead of sparsifying the network, we allow the variational posterior distribution to be more flexible than standard mean-field Gaussian, used in the BNN. Further, using the local reparametrization trick (LBBNN-LCRT) is mainly computational advantage compared to the LBBNN method. Finally, LBBNN-FLOW (proposed in this paper) is related to both BNN-FLOW and LBBNN-LCRT, in the sense that it can learn sparse BNN, and in addition, have a more flcxible posterior distribution than the mean-field Gaussian used in LBBNN-LCRT. For a graphical illustration of how these methods are related, we refer Figure 3
-1, x5-1
hard-tauhl(x) =
-15051
1>1
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table 1: Performance metrics on the logistic regression variable selection simulation study.
CS
LBBNN-LCRT
LBBNN-FLOW
mean TPR.
0.681
0.838
0.972
mean FPR
0.125
0.084
0.074
In addition to the classification experiments in the main text, we also provide experiments (both classification and regression) on various tabular datasets in Appendix [D Here, in addition to the methods detailed above, we also include Monte Carlo (MC) dropout (Gal & Ghahramani 2016), and (exact) Ganssian processes (more details about how this implemented in Appendix D] for the regression datasets. Finally, for the labular datasets, check the internal parsimony of all of the Bayesian baselines through the pwAiC, and PWAIC, penalties from Gelman et al. (0144), metrics that also have been used estimates of the effective number of parameters.
Additionally, we perform two simulation studies. In the first one, we consider variable selection in a logistic regression setting, with highly correlated explanatory variables. We further vary correlation levels and noise levels of lincar predictor. In additional experiments reported in Appendix we address different correlation structures and relations to the response, where also change the level of correlations bet ween the covariates and repeat that for several noise levels.
All the experiments with LBBNN were coded in Python, using the PyTorch deep learning library Paszke et al. 2019 but the approach of Carbonetto & Stephens (0022) was run in R.
5.1 Logistic regression simulation study
In this section, we do a variable selection experiment within logistic regression setting. As logistic regression is just a special case of neural nelwork with one neuron (and hence one layer), modifying the algorithuss is straightforward. We are limiting ourselves the logistic regression context to be able to compare the original baseline method from Carbonetto & Stephens (2012), who demonstrated that the mean-field variational approximation starts to fail the variable selection task when the covariates are correlated. As we are only interested in comparing the mean-field variational approach against the variational distribution with normalizing flows, we do not include comparisons with more traditional variable selection methods such as Lasso (Tibshirani 1996) or Elastic Net (Zou & Hastic 2005).
We use the same data as in Hubin & Storvik (2018), consisting of a mix of binary and continuous variables, with a binary outcome, and 2000 observations. The covariates, æ; are generated with strong and complicated correlation structure between many of the variables (see Figure 4). For more details on exactly
how the covariates generated, see appendix B of Hubin & Storvik (2018). The response variable, y, is generated according to the following lata-generating process:
N x,
2((3²²,0.5) 0.5)
exp(1)
(12)
~ Bernoulli
T+
with the regression parameters defined to be:
(-4, -1110111001111111111000001111110 0,000000000..,0000000000000,,,,0 0.00000,,
The goal is to train the different methods to select the non-zero elements of B. We consider the parameter B be included if the corresponding posterior inclusion probability a; > 0.5, i.c. the median probability model of Barbieri & Berger (2004). We fit the different methods 100 times (to the same data), each time computing the true positive rate (TPR), and the false positive rate (FPR).
9

---

Published in Transactions on Machine Learning Research (10/2024)
Correlations between all 20 variables
1.00
1.0 0.34-0.32
0.47
0.01 -0.32
-0.05
0.0 -0.02-0.03..0000 0.03-0.07-0.00
0.19
2 00.34
1.0
0.08
0.33
-0.32
0.00
0.03 -0.0100.00222-- -0.0 0.01 -0.01 -0.0 0.02
0.01
0.9000.96.0003
0.8
3 9-0.22-0.78
1.0
-0.3
-0.34
0.000
00.00.00.000011111
0.01 -0.01 0.0 0.03-0.02-0.03
0.0
0.0
0.75
0.01
0.68
0.75
4 4-047-0..01
-0.3
1.0
0..32-0.3
-0.0
0.01 -0100..0-00011-- -0.0 0.0
0.01 -0.00
0.02
6-0.32-0.32 0.53 0.34-0.78
1.0
0.02
0.01
0.02
0.04 0.02 0.02-0.05-00022..........0.0.0.. 0.03 0.31 0.02-0.28
-0.0 0.01 -..01-0.0110.0100.000
1.0
0.04
0.0
-0.01
0.0
0.01 -0.0 0.0 0.01 0.002-0001.0000--00000000
8 8+0.05 0.03 -0.01-0.05 0.02
0.01
0.04
1.0
0.01-0.02-0000 -0.0 -0.05 0.02 0.04 -0.01 0.01-00004-00020.00
0.25
Correlation
0.01
0.01
0.01 0.01
0..01-0.01-00
02-0.06
1.0
0.01
0.01
0.01 -..03 0.000..01 0.21 0.01-0.08
0.00
0.01
0.0
0.01
0.01
0.01
-0.0
0.0
0.01
0.85
0.35
1.0
0.01
0.0
0.01
0.01
0.0
0.0
0.07
0.01 0.17
12
0.02 0.0
0.01 0.04 0.01 0.0
0.01
0.01
0.01
1.0
0.01 0.77
0.18
0.02 0.01 0.01 0.06
0.01
13
-0.25
0.05 0.05
0.001 0.010.01
0.77.0.28
0.09
0.03
0.02
0.03
0.0 0.02 -.002..000000 0.01 0.04 0.002 0.0 -0.01 0.18 18 0.95
3.0
0.04.0.33.001
4.3
0.01
16
0.02 0.0
0.01
0.02 0.01 0.02 0 03 0.002 -..33 0.0 0.02 0.03
-0-00.0000
www.0.44.0000
3.0
0.1..0.0.-001
0.02
-0.50
17 0.04..03
0.0 0.03 0.01 0.02-0001 0.01 000.000-00000000
0.103303..1
0.03-0.01
0.96
0.02
0.00
0.01
0.02
0.03
0.02
01-0 -0.0 -0.0000000000000000000 -0.01-0 01-0..0000 0. -8 -03
0..0 0.02
0.02
0019. 0.8 0.66
0.04
0.26
0.280.0. 0.03
0.00 0.02
0.02
0.82-0.02
1.0
8
9
10
11
12
13
14
15
16
17
18
19
20
Figure 4: Plots showing the correlation between different variables in the logistic regression simulation study.
Carbonetto/Stephees
LBBNN-LCRT
1.0
1.0
TPR
TPR
0.8
0.8
FPR
FPR
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
2
4
6
8
9
1011113141516111811111
2
3
4
5
7
9
10
10111213141116171111929
8
LBBNN-FLOW
1.0
TPR
0.8
FPR
0.6
0.4
0.2
0.0
1
3
111112211111111111811
B
Figure 5: Bar-plots showing how often the weights included over 100 runs.
In this experiment we compare our approaches L.BENN-LCRT and LBBNN-FLOW against the algorithm proposed by Carbonetto & Stephens (2012). denoted as CS henceforth. That method is very similar LBBNN-LCRT, it the same variational distribution. But in CS, optimization done with coordinate
10

---

Published in Transactions on Machine Learning Research (10/2024)
ascent variational inference and without subsampling from the data. For the normalizing flows, use flows of length two with the neural networks having two hidden layers of 100 neurons each. We use a batch size of 400 and train for 500 epochs. We use standard normal priors for the weights and a prior inclusion probability of 0.25 on the inclusion indicators for all three approaches. Hence, we are in the setting of a Bayesian logistic regression, with variable selection.
The results are in Table We also show a bar-plot Figure 5) for each of the 20 weights over the 100 runs. We that LBBNN-FLOW performs best, with the highest TPR and the lowest FPR. It is especially good at picking out the correct variables where there is high correlation between many of them (for example 81 96)- We might attribute this to the more flexible variational posterior distribution, the mean-field Gaussian distribution used in the other three methods. CCarbonetto & Stephens (2012) also discuss how the mean-field approach can only be expected to be a good approximation when the variables are independent or at most weakly correlated.
opposed to
5.1.1 Extension to different levels of correlation
To extend the experiment to other correlation levels, we use the same data as in the simulation study from Hu- bin & Storvik (2018)) but with varying noise levels, using a E (1,0.1, (0110.00 0.01} that we use to generate 1) and with different correlation structures for Ti,j per signal level. For a correlation parameter 1, ..... 2000, 5,.., 20.
{1, 0.9........... 0.1),
the covariates were updated by the following rule: Σi, = -P-- - (1 (1-p) Then, the resulting predictors are standardized resulting in the final correlated covariates used to generate the responses just like described in equation 12 above. Finally, CS, LBBNN-LCRT, and LBBNN-FLOW were run 20 times per setting using exactly the same hyper and luning parameters as in the previous simulation study.
-i.j.
0
1
a= 0.1
a= 0.01
0=
a= 0.1
0 = 0.01
1.0
1.0
LCRT
LCRT
FLOW
FLOW
0.8
cs
0.8
CS
0.6
0.6
TPR
FPR
0.4
0.4
0.2
0.2
0.0
0.0
0.297
0.2
0.157
0.128
0.112
0.297
0.2
0.157
0.128
0.112
0.297
0.2
0.157
0.128
0.112
0.297
0.2
0.157
0.128
0.112
0.297
0,2
0.157
0.128
0.112
0.297
0.2
0.157
0.128
0.112
Mean correlation design matrix
Mean correlation the design matrix
Figure 6: On the left, true positive rates, with false positive rates to the right. The maximum correlation ranged between 0.954 and 0.997.
For this experiment, in Figure 6 we show plots of average true positive and false positive rates, where the dashed lines indicate a 90% empirical confidence interval obtained from 20 runs of the corresponding algorithms per settings on different seeds. Along the x-axis, we show (every other one to save space) the mcan absolute correlation between all covariates in the corresponding setting. As mentioned previously, for this experiment we consider 30 different settings, divided into 3 parts where we simulated three different noise levels in the linear predictor, and for each of these, we have 10 different correlation levels. The results in Figure i are consistent with those obtained on the original data from Hubin & Storvik (2018). Across all settings here, the LBBNN-FLOW is on average the most favorable method with respect to a trade-off between TPR and FPR, although in many cases the confidence intervals are overlapping.
We added three more studies with other complicated correlation structures, varying signal strength (3 dif- ferent levels in every study), and correlation levels (10 levels per signal level) between the covariates. These additional simulation studies provided in the Appendix B to this paper.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Other results with varying levels of correlations between one of the data generative covariates and one of the non-data generative covariates are reported in follow-up works (Sommerfelt & Hubin 2024 Høyheim that use our approaches. There, in all scenarios, both Sommerfelt & Hubin (2024) and Høyheim (2024) show that the flow-based inference is outperforming the LCRT onc.
2024)
5.2
Classification experiments
5.2.1 Experimental setup
We perform two classification experiments, one with the same fully connected architecture as in Hubin & Storvik (2019), and the other with a convolutional architecture (see appendix C for details on how this is implemented). On the fully connected architecture, we use two hidden layers with 400 and 600 neurons respectively, whereas, for the other experiment, we use the LeNet-5 (LeCun et al.] 1998) convolutional architecture, but with 32 and 48 filters for the convolutional layers. We emphasize that it is possible to use deeper and more complicated architectures (for example Resnet-18, He et al. 2016), which would likely improve the results reported in this paper. As the goal here is not to try to approach (or hack through tuning and engineering) state-of-the the-art results, we do not experiment any further with this. In both cases, we classify on MNIST (Deng 2012), FMNIST (Fashion MNIST) (iiao et al. 2017) and KMNIST (Kuzushiji MNIST) Clannwat et al.] 2018). All of these datasets contain 28x28 grayscale images, divided into a training and validation set with 60 000 and 10000 images respectively. MNTST and FMNIST well-known and often utilized datasets, so it is easy to compare performance when testing novel algorithms, while KMNIST is a more recent addition.
For both the fully connected and convolutional architectures, we use the Adam (Kingma & Ba 2014) optimizer, batch size of 100, and train for 250 epochs. All the experiments are run 10 times, and we report the minimum, median, and maximurn predictive accuracy over these 10 runs as well as expected calibration errors and negative test lieeliilioood. We measure predictive performance and predictive uncertainty handling within two approaches: First, the Bayesian model averaging approach, where we average over 100 samples from the variational posterior distribution, where for our proposed method uncertainty in both weights and structures following Hubin & Storvik (2019). For the standard BNN and BNN-FLOW, this model averaging is only over the weights, whereas for the frequentist approaches, we do not have any model averaging. Secondly, for the methods with both weight and structural uncertainty, we consider the median probability model (Barbieri & Berger) 2004), where we only do averaging over the weights that have posterior inclusion probability greater than 0.5, whilst others are excluded from the model. This allows for significant sparsification of the network. We emphasize that this is possible because we can go back to sampling the weights when doing inference, i.e. we sample only from the weights that have a corresponding inclusion probability greater than 0.5.
are taking into account
In addition (where applicable), we also report the density of the network, defined as the proportion of non- zero weights. The reported density (1-sparsity) is an average over these 10 runs. For the full variational model averaging approach, we consider the density to be equal to one since we do not explicitly exclude any weights when computing the predictions (even though a large proportion of the weights may have & small inclusion probability and in practice within any 10 samples over which we are marginalizing less than 100% of the weights will be used, yet ideally one wants to average over more than 10 samples).
5.2.2 Methods
For these experiments, are interested in comparing our method LBBNN against the closest baseline methods (see Figure 3h. For ANN + L2, we use weight decay of 0.5, inducing penalized likelihood, which corresponds to MAP (maximum apostcriori probability) solutions of BNN and BNN-FLOW under standard normal priors. For BNN and BNN-FLOW, we use standard normal priors for the weights. For the LBBNN- LCRT and LBBNN-FLOW the weights and biases in the network, and a prior inclusion probability of 0.10. For both q(z) and r(z) W, we use flows of length two, where the neural networks consist of two hidden layers with 250 neurons each. For the Bayesian model averaging approaches, we average over 100 samples from the variational posterior
methods, we also use the standard normal prior for the slab components of all
12

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: Performance metries (accuracy and density) on the KMNIST, MNIST, FMNIST validation data, for the fully connected architecture. For the accuracies (%), we report the minimum, maximum, and median over the len different runs. Density computed as an average over the ten runs. The best median results are bold.
KMNIST
Median probability model
Dense model
Method
min
median
max
density
min
median
max
density
LBBNN
89.34
89.69
90.19
0.113
89.62
89.87
90.29
1.000
LBBNN-LCRT
89.94
90.21
90.42
0.136
90.17
90.35
90.62
1.000
LBBNN-FLOW
90.89
91.04
91.37
0.096
91.23
91.43
91.65
1.000
BNN-FLOW
91.86
92.33
92.74
1.000
BNN
92.29
92.47
92.66
1.000
ANN
89.61
90.82
91.26
1.000
ANN + L2
87.20
87.84
88.22
1.000
MNIST
Median probability model
Dense model
Method
min
median
max
density
min
median
max
density
LBBNN
97.99
98.07
98.14
0.098
97.99
98.07
98.18
1.000
LBBNN-LCRI
97.83
97.98
98.05
0.103
98.00
98.10
98.20
1.000
LBBNN-FLOW
98.23
98.38
98.52
0.074
98.33
98.43
98.50
1.000
BNN-FLOW
98.39
98.54
98.67
1.000
BNN
98.40
98.48
98.58
1.000
ANN
97.98
98.10
98.18
1.000
ANN + L2
96.95
97.04
97.16
1.000
FMNIST
Median probability model
Dense model
Method
min
median
max
density
min
median
max
density
LBBNN
88.56
88.84
89.04
0.107
88.66
88.84
89.16
1.000
LBBNN-LCRT
87.44
87.76
88.05
0.141
87.76
87.98
88.05
1.000
LBBNN-FLOW
89.42
89.65
89.91
0.097
89.47
89.80
90.03
1.000
BNN-FLOW
89.05
89.44
89.62
1.000
BNN
90.02
90.17
90.37
1.000
ANN
89.31
89.58
89.81
1.000
ANN + L2
87.10
87.30
87.62
1.000
distribution. We emphasize that we do not use posterior tempering any other ad-hoc tricks for any of the Bayesian methods.
5.2.3 Results
The results with the fully connected architecture can be found Tables 2 3 and and for the convolutional architecture Tables comparable to the baseline LBBNN method, except for FMNIST where it performs a bit worse both with the fully connected and with the convolutional architecture. It is surprise that these results are similar, using the LCRT is mainly computational advantage. Secondly, we note that our LBBNN-FLOW method performs better than the two aforementioned methods in terms of prediction accuracy, on both convolutional and fully connected architectures, while having the most sparse networks. We also see that in terms accuracy BEBNN-FLOW performs mostly on par with the other two Bayesian architectures, BNN and BNN-FLOW, especially on the fully connected architecture where it gets comparable accuracy even with very sparse networks. In terms of predictive uncertainty handling, in almost all settings the best performance (across all addressed in our design mothods) is achieved by one of the LBBNN methods and in
and 7 Firstly, we see that using the LBBNN-ICRT gives results that are
13

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Expected calibration error on the fully connected architecture, where we report the minimum, maximum, and median over the ten different runs. The best median results are bold.
KMNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.031
0.034
0.039
0.039
0.041
0.046
LBBNN-LCRI'
0.072
0.076
0.079
0.118
0.120
0.123
LBBNN-FLOW
0.026
0.032
0.034
0.060
0.061
0.063
BNN-FLOW
0.098
0.102
0.106
BNN
0.069
0.072
0.075
ANN
0.077
0.081
0.092
ANN - L2
0.051
0.058
0.061
MNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.020
0.021
0.022
0.022
0.022
0.023
LBBNN-LCRT
0.029
0.030
0.032
0.047
0.049
0.051
LBBNN-FLOW
0.010
0.012
0.013
0.021
0.021
0.023
BNN-FLOW
0.033
0.035
0.036
BNN
0.027
0.028
0.029
ANN
0.016
0.017
0.018
ANN L2
0.038
0.039
0.040
FMNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.026
0.028
0.032
0.023
0.026
0.031
LBBNN-I.CR
0.047
0.051
0.057
0.072
0.074
0.075
LBBNN-FLOW
0.019
0.021
0.023
0.029
0.031
0.033
BNN-FLOW
0.054
0.056
0.057
BNN
0.032
0,035
0.039
ANN
0.083
0.085
0.087
ANN L2
0.030
0.033
0.035
the highest share of the cases by LBBNN-FLOW. Also, in most of the cases, the median probability model allows to improve the calibration of the predictions for image datascts. The higher density in general on the convolutional architectures is mainly a result of them being already sparse in the beginning. However, these networks could also be sparsified further by using more conservative priors on inclusions of the weights. We note that the frequentist networks perform slightly worse on these datasets with our chosen architectures. The results could likely be improved by adding more regularization, such as dropout or batch-normalizztion, but we do not do this here. The increased predictive power of using normalizing flows comes at a computational cost. With the fully connected architccture, we observed that it took around seconds train one epoch with LBBNN-LCRT, 13 seconds with LBBNN, and 17 seconds with LBBNN-FLOW On the convolutional architecture, it took 7 seconds per epoch with the LBBNN-LCRT, 18 seconds with LBBNN, and 28 with LBBNN-FLOW. Naturally, the frequentist networks are much more computationally efficient, as they only have half the parameters of a standard BNN. These results, however, were observed on a shared computational node and hence could be influenced by loads at different. times. Yet they are consistent with theoretical training costs reported in Table in Appendix A
an NVIDIA A10 GPU.
For the results on the tabular datasets, we refer Appendix D where we also report the expected calibration error for classification datasets and pinball loss for regression datasets.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: Negative test log-likelihood on the fully connected architecture where we report the minimum, maximum, and median over the ten different runs. The best median results are bold.
KMNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.527
0.537
0.553
0.544
0.555
0.571
LBBNN-LCRI
0.544
0.548
0.561
0.588
0.599
0.606
LBBNN-FLOW
0.569
0.589
0.626
0.605
0.621
0.670
BNN-FLOW
0.818
0.869
0.929
BNN
0.620
0.628
0.638
ANN
1.255
1.324
1.554
ANN L2
0.417
0.424
0.434
MNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.095
0.099
0.103
0.097
0.102
0.105
LBBNN-LCRT
0.111
0.114
0.117
0.132
0.134
0.137
LBBNN-FLOW
0.091
0.094
0.104
0.103
0,109
0.114
BNN-FLOW
0.160
0.173
0.199
BNN
0.117
0.120
0.127
ANN
0.167
0.184
0.193
ANN L2
0.119
0.120
0.122
FMNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.350
0.352
0.356
0.358
0.360
0.365
LBBNN-I.CR
0.422
0.425
0.435
0.439
0.441
0.443
LBBNN-FLOW
0.360
0.363
0.368
0.375
0.379
0.384
BNN-FLOW
0.416
0.421
0.423
BNN
0.515
0.519
0.530
ANN
0.794
0.850
0.864
ANN L2
0.362
0.365
0.370
6
Discussion
We have demonstrated that increasing the flexibility in the variational posterior distribution with normalizing flows improves the predictive power compared to the bascline method (with mean-field posterior) while obtaining more sparse networks, despite having a looser variational bound than the mean-field approach. Also, the flow method performed best on variable selection problem demonstrating better structure learning performance, while the mean-field approaches struggle with highly correlated variables. Also, the calibration of uncertainties predictive applications is similar with slight advantage of the proposed approach in this paper. Unlike dense BNNs, our methods have the additional advantage of being able to perform variable selection. The downside is that LBBNNs have an extra parameter per weight, making them less computationally efficient than dense BNNs. Using normalizing flows is further computational burden as could gain
must also optimize over all the extra flow parameters. If uncertainty handling not desirable, the minimal number of predictive parameters using the model trained with flows by relying on the posterior means of the median probability model's parameters. This approach is studied for simpler approximations in more detail in Hubin & Storvik (022)) but it omitted in this paper. Also, in the future, it would of interest develop reversible jump MCMC (Green Hastic, 2009) or other relevant MCMC methods for LBBNN and compare the variational approximations with the exact sample least the div-eiienii cases where the latter could be feasible.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Table 5: Performance metrics on the KMNIST, MNIST, FMNIST validation data, with the convolutional architecture. See the caption in Table 2 for more details.
KMNIST
Median probability model
Dense model
Method min
median
max
density
min
median
max
density
LBBNN
95.39
95.59
95.75
0.359
95.35
95.56
95.65
1.000
LBBNN-LCRI
94.85
95.07
95.21
0.428
94.98
95.42
95.54
1.000
LBBNN-FLOW
95.63
95.96
96.14
0.352
95.76
95.99
96.31
1.000
BNN-FLOW
95.89
96.38
96.52
1.000
BNN
95.06
95.37
95.63
1.000
ANN
94.61
94.96
95.26
1.000
ANN L2
92.08
92.61
92.76
1.000
MNIST
Median probability model
Dense model
Method
min
median
max
density
min
median
max
density
LBBNN
99.20
99.26
99.30
0.354
99.20
99.25
99.28
1.000
LBBNN-LCRT
99.05
99.25
99.29
0.406
99.23
99.30
99.36
1.000
LBBNN-FLOW
99.19
99.27
99.35
0.338
99.21
99.32
99.41
1.000
BNN-FLOW
99.27
99.36
99.40
1.000
BNN
99.22
99.30
99.38
1.000
ANN
98.99
99.13
99.18
1.000
ANN L2
97.95
98.29
98.42
1.000
FMNIST
Median probability model
Dense model
Method
min
median
max
density
min
median
max
density
LBBNN
90.85
91.48
91.71
0.354
90.95
91.33
91.69
1.000
LBBNN-I.CR
89.94
90.39
90.79
0.433
90.53
90.82
91.34
1.000
LBBNN-FLOW
90.70
91.60
92.08
0.366
91.27
91.60
92.30
1.000
BNN-FLOW
91.53
91.80
92.12
1.000
BNN
91.37
91.68
92.12
1.000
ANN
90.49
91.07
91.73
1.000
ANN L2
87.97
88.08
88.46
1.000
In this paper, we use the same prior for all the weights and inclusion indicators, although this is not necessary, A possible avenue of further rescarch could be to vary the prior inclusion probabilities, induce different sparsity structures or to incorporate the actual prior knowledge about prior inclusion probabilities of the covariates. Currently, we are taking into account uncertainty in weights and parameters, given some neural network architecture. In the future, it may be of interest to see if it is also possible to incorporate uncertainty in the activation functions. By having skip connections to the output, we could learn with uncertainties to skip all non-linear layers if a linear function is enough, or if a constant estimate of the parameters of the responses is enough (null model), or if one nceds some nonlincar layers. This could lead more transparent Bayesian deep learning models. But the success in that task relies on sufficiently good structure learning, where mean feeoooppro
known not work well (Carbonetto & Stephens 2012).
A possible application is to do a genome-wide association study (GWAS), using our method. Combining LBBNNs and GWAS has been proposed by Demetci el al. (2021). however, this only uses the mean-field posterior. With our normalizing flow approach, we can easily model dependencies within each SNP set, in addition to dependencies between the different SNP sets. Another set of promising applications is recovering structural equations in nonlinear dynamical systems as the robust and uncertainty-awet aware alternative [1 penalty used in the Sindy approach (Brunton et al. 2016).
16

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: Expected calibration error on the convolutional architecture, where we report the minimum, max- imum, and median over the ten different runs. The best median results are bold.
KMNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.041
0.045
0.048
0.050
0.051
0.053
LBBNN-LCRI
0.015
0.018
0.020
0.117
0.019
0.120
LBBNN-FLOW
0.034
0.039
0.043
0.028
0.030
0.034
BNN-FLOW
0.029
0.032
0.035
BNN
0.018
0.019
0.020
ANN
0.041
0.042
0.044
ANN - L2
0.018
0.022
0.025
MNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.014
0.016
0.017
0.016
0.017
0.018
LBBNN-LCRT
0.005
0.005
0.006
0.006
0.006
0.008
LBBNN-FLOW
0.010
0.012
0.009
0.010
0.011
0.023
BNN-FLOW
0.009
0.009
0.010
BNN
0.005
0.005
0.006
ANN
0.007
0.008
0.009
ANN L2
0.013
0.014
0.017
FMNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.022
0.025
0.026
0.028
0.030
0.034
LBBNN-I.CRY
0.013
0.018
0.021
0.018
0.020
0.021
LBBNN-FLOW
0.022
0.026
0.030
0.020
0.022
0.028
BNN-FLOW
0.023
0.027
0.031
BNN
0.021
0,022
0.025
ANN
0.059
0.064
0.068
ANN L2
0.020
0.023
0.029
In our experimental design, we check the effect of direct treatments from Figure 3 especially we were inter- ested in edges from LBBNN to LBBNN-LCRT and to LBBNN-FLÖW and from BNN-FLOW to LBNNN- FLOW as the closest neighbors to the proposed in this paper methodology. None of the results is completely uniform across all cases, however we demonstrated that as compared to BNN-FLOW, our proposed approach typically improves on sparsity without losing predictive performance and yields robustly well-calibrated un- certainty handling. Further, as compared to LBBNN, we improve the predictions and sparsity slightly. Improvements are observed in most addressed variable selection tasks when using the LBENN-FLOW ap- proach as compared to LBBNNNLRCT, while all variable selection tasks, LBBNN-FLOWW with respect to CS approach and in many cases outperforms it, which is especially the case for block diagonal correla- tion structures in Sections 5.1 and experiment 3 in Appendix B Methods beyond Figure inchuding other sparsity inducing Bayesian approaches to BNN (Molchanov that are less related to our approach and could not be incorporated with direct edges in Figure were omitted in this paper. Yel in the future, following the guidance from Herrmann et al. (2024). it would be interesting to perform a neutral comparison study (Boulesteix et al. 2013). This study should not be done by the authors of any of these papers avoid biases and tuning onc's own methods to beat (in some sense) all competitors and hack the state-of-tha the-rtt which recently became a common practice that often results in poor empirical evidence (Herrmann al.] 2024). This should also be ideally done new datasets not.
al. 2017 Deng al. 2019 Li et al. 2024
17

---

Published in Transactions on Machine Learning Research (10/2024)
Table 7: Negative test log- ligeeiill on the convolutional architecture where we report the minimum, max- imum, and median over the ten different runs. The best median results bold.
KMNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.272
0.283
0.294
0.281
0.289
0.298
LBBNN-LCRI'
0.260
0.267
0.276
0.250
0.262
0.274
LBBNN-FLOW
0.324
0.352
0.380
0.280
0.308
0.331
BNN-FLOW
0.289
0.314
0.354
BNN
0.306
0.339
0.360
ANN
0.467
0.525
0.576
ANN - L2
0.250
0.259
0.277
MNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.043
0.045
0.048
0.045
0.047
0.050
LBBNN-LCRT
0.029
0.030
0.034
0.029
0.030
0.034
LBBNN-FLOW
0.046
0.049
0.054
0.038
0.041
0.045
BNN-FLOW
0.040
0.042
0.043
BNN
0.033
0.036
0.039
ANN
0.046
0.050
0.060
ANN L2
0.057
0.060
0.067
FMNIST
Median probability model
Dense model
Method
min
median
max
min
median
max
LBBNN
0.276
0.282
0.291
0.284
0.293
0.300
LBBNN-I.CR
0.284
0.298
0.315
0.271
0.283
0.301
LBBNN-FLOW
0.345
0.360
0.374
0.310
0.324
0.342
BNN-FLOW
0.367
0.402
0.459
BNN
0.270
0.277
0.288
ANN
0.470
0.507
0.557
ANN L2
0.332
0.337
0.341
used in the papers describing the compared methods. For non-neutral comparison of LBBNN with other
sparse neural networks, curious reader can check the results in Hubin & Storvik 2019 2024
References
Jincheng Bai, Qifan Song, and Guang Cheng. Efficient variational inference for sparse deep learning with theoretical guarantee. Advances in Neural Information. Processing Systems, 33:466-477 2020.
Maria Maddalena Barbieri and James O Berger. Optimal predictive model selection. The annals of statistics, 32(3))870 897, 2004.
Marilena Barbieri, James O Berger, Edward George, and Veronika Rockova. The median probability model and correlated variables. arXiv preprint arXiw:1807.08330 2018.
Christopher M. Bishop. Bayesian neural networks. Journal of the Brazilian Computer Society, 4(1), 1997.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the Americon statistical Association, 112(518):859 877, 2017.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Anne-Laure Boulesteix, Sabine Lauer, and Manuel JA Eugster. A plea for neutral comparison studies in computational sciences. PloS one, 8(4)xo6156, 2013.
Karl W Broman, Hao Wu, Saunak Sen, and Gary A Churchill. R/qtl: Qti mapping in experimental crosses. bioinformatics, 19(7):889-890, 2003.
Steven Brunton, Joshua Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academyy of sciences, 113 (15):3932 3937, 2016.
Peter Carbonetto and Matthew Stephens. Scalable variational inference for Bayesian variable selection in regression, and its accuracy in genetic association studies. Bayesian analysis, 7(1):73 108, 2012.
Iikay Çinar, Murat Koklu, and Şakir Taşdemir. Classification of raisin grains using machinc vision and artificial intelligence methods. Gazi Mühendislik Bilimleri Dergisi, 6(3):200 209, 2020.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kaznaki Yamamoto, and David Ha. Deep learning for classical japanese literature. arXiv preprint arXiu:1812.01113 2018.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos, and José Reis. Wine Quality. UCI Machine Learning Repository, 2009. DOI:
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Pinar Demetci, Wei Cheng, Gregory Darnell, Xiang Zhou, Sohini Ramachandran, and Lorin Crawford. Multi- scale inference of genetic trait architecture using biologically annotated noural networks. PLoS genetics, 100$)=e1009744 2021.
Li Deng. The MNIST database of handwritten digit images for machine learning research [best of the web). IEEE signal processing magazine, 29(6):141-15 142, 2012.
Wei Deng, Xiao Zhang, Faming Liang, and Guang Lin. An adaptive empirical Bayesian method for sparse deep learning. Advances in neural information processing systems, 32, 2019.
David Fletcher and David Fletcher. Bayesian model averaging. Model averaging, pp. 31-55, 2018.
Vincent Fortuin. Priors Bayesian deep learning: A review. International Statistical Review, 90(3):563 591, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXio preprint ar Xrx: 03635, 2018.
Yarin Gal. Uncertainty in Deep Learning. PhD thesis, University Cambridge, 2016.
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pp. 1150-1059. PMLR, 2016.
Andrew Gelman, Jessica. Hwang, and Aki Vehtari. Understanding predictive information criteria for bayesian models. Statistics and computing, 24:997 1016, 2014.
Peter J Green and David I Hastic. Reversible jump MCMC. Genetics, 155(3):1391-1403, 2009.
David Harrison Jr and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of environmental economics and manogement, 5(1):81 102, 1978.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference computer vision and pattern recognition, pp. 770 770-778. 2016.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Moritz Herrmann, F Julian D Lange, Katharina Eggensperger, Giuseppe Casalicchio, Marcel Wever, Matthias Feurer, David Rügamer, Eyke Hüllermeier, Anne-Taure Boulesteix, and Bernd Bischl. Position: Why we must rethink empirical research in machine learning. In Forty-first International Conference on Machine Learning, pp. 18228-18247, 2024.
Jennifer A Hocting, David Madigan, Adrian E Raftery, and Chris T Volinsky. Bayesian model averaging: a tutorial (with comments by m. clyde, david draper and ei george, and rejoinder by the authors. Statistical science, 14(4):382-41) 1999.
Eirik Høyheim. Improving sparsity and interpretability of latent binary Bayesian neural nelworks by intro- ducing input-skip connections. Master's thesis, Norwegian University of Life Sciences, 2024.
Jiri Hron, Alexander G de G Matthews, and Zoubin Ghahramani. Variational Gaussian dropout is not Bayesian. arXiv preprint arXiv:1711.0299 2017.
Aliaksandr Hubin and Geir Storvik. Mode jumping MCMC for Bayesian variable selection in GLMM. Computational Statistics 8 Data Analysis, 127-281-297, Nov 2018. ISSN 0167-9473. doi: 10. 10.1016/j.csda. 2018.05.020. URL http://dx doi org/10. 1016/o.cccseeeeeeeee... 2018.00...... 05.00
Aliaksandr Hubin and Geir Storvik. Combining model and parameter uncertainty in Bayesian neural net- works. arXic:1903. 07594, 2019.
Aliaksandr Hubin and Geir Storvik. Sparse Bayesian neural networks: Bridging model and parameter uncertainty through scalable variational inference. Mathematics, 12(6))788, 2024.
Aliaksandr Hubin, Geir Storvik, and Florian Frommlet. A Novel Algorithmic Approach to Bayesian Logic Regression (with Discussion). Bayesian Analysis, 15(1):263 333, 2020.
Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saced Qureshi. A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review, 53(8):5455 5516, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arv preprinl a1 198809 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes, 2013.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. Advances in neural information processing systems, 28, 2015.
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskover, and Max Welling. Improved vaniational inference with inverse autoregressive flow. Advances neural information processing systems, 29, 2016.
Ron Kohavi. Census Income. /CGGPS.
UCI
Machine
Learning Repository, 1996.
DOL:
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to docu- ment recognition. Proceedings of the IEEE, 86(11):2278 22222222444 1998.
Junbo Li, Zichen Miao, Qiang Qiu, and Ruqi Zhang. Training Bayesian neural betworks with sparse subspace variational inference. arXiv preprint arXiv:2402. 1025, 2024.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In International Conference on Machine Learning, pp. 2218 2227. PMLR, 2017.
Christos Louizos, Karen Ulirich, and Max Welling, Bayesian compression for deep learning. Advances in neural information processing systems, 30:3288-3200 2017.
David JC MacKay. Bayesian neural networks and density networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 354(1): 73 80, 1995.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov. Variational dropout sparsifies deep neural net- PMT.R, 2017.
works. In International Conference on Machine Learning, pp.
Sergio Moro, Paulo Rita, and Paulo Cortez. Bank Marketing. UCI Machine Learning Repository, 2012.
DOT: https:///oi..or/11..222 2ttp:/000.24///5252/1
Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCT Machine Learning Repository, 1995. DOL: https://doi.orr/100 wtt.d//..rrrg...44332././
Radford M Neal. Bayesian training of backpropagation networks by the hybrid Monte Carlo method. Tech- nical report, Citesoer, 1992.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence prodictions for unrecognizable images. In Proceedings of the IEEE conference computer vision and pollern recognilion, pp. 427-436, 2015.
llker Ali Ozkan, Mural Koklu, and Rıdvan Saraçoglu. Classification of pistachio species using improved k-nn classifier. Health, 23:e2021044, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http://papers. seraapersnnuuris..... cc/pperr/ wwtp-yoorrra---mmrrttttiieeeeeee oerpre/eeeeeeeeeeeeeeeeerrreeeee weepeaanee-eee--eaaree
Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. Optimal lottery tickets via subset sum: Logarithmic over-parameetrrization is sufficient. Advances in neural infor- mation processing systems, 03:2599-2610 2020.
Nicholas G Polson and Veronika Ročková. Posterior concentration for sparse deep learning. Advances in Newral Information Processing Systems, 31, 2018.
Lutz Prechelt.
Early stopping-but when? In Neural Networks: Tricks of the trade, pp. 55-69. Springer,
1998.
J. R. Quinlan. Credit Approval. UCI Machine Learning Repository. DOI: https://ddo.orr/11022222 htt:://oi.oor/0.2444333/ 14.23322 /C5FS30.
Rajesh Ranganath, Dustin Tran, and David Blei. Hicrarchical variational modols. In International conference on machine learning, pp. 324 333. PMLR, 2016.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International con- ference machine learning, pp. 1530-1538. PMLR, 2015.
Solve Sæbo, Trygve Almoy, and Inge S Helland. simrel- a versatile tool for linear model data simulation based on the concept of relevant subspace and relevant predictors. Chemometrics and Intelligent Laboratory Systems, 146:128 135, 2015.
Dylan Sam, Rattana Pukdee, Daniel P Jeong, Yewon Byun, and J Zico Kolter. Bayesian neural networks with domain knowledge priors. arXiv preprint arXiv:2402.184 18410, 2024.
Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmeentttinn for deep learning. Journal of Big Data, 6(1):1-48, 2019.
Philip Sebastian Hauglie Sommerfelt and Aliaksandr Hubin. Evolutionary variational inference for Bayesian generalized nonlinear models. Neural Computing and Applications, pp. 1-18, 2024.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Lya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929- 1958, 2014.
Christian Szegedy, Wojeiech Zaremba, Tlya Sutskever, Joan Bruna, Dumitru Erhan, Tan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint orXi::1312.61990 2013.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-283. 1996.
Ba-Hien Tran, Simone Rossi, Dimitrios Milios, and Maurizio Filippone. All you need is a good functional prior for Bayesian deep learning. Journal of Machine Learning Research, 23(74):1- 56, 2022.
UCL Dry Bean Dataset. UCI Machine Learning Repository, 2020. DOL: https:/ 1ttp////1...42/22111 C50S4B.
Petri Varvia, Janne Räty, and Petteri Packalen. mgpr: An package for multivariate Gaussian process regression. SoftwareX, 24:101563, 2023.
Florian Wenzel, Kevin Roth, Bastiaan Vecling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the Bayes posterior in deep neural networks really? In International Conference on Machine Learning, pp. 10248 10259. PMLR, 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for benchmarking machine learning algorithms. arXiv preprint
2017.
Hui Zou and Trevor Hastie. Regularization and variable selection via the clastic net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67()):301-32D) 2005.
Supplementary material
The code used for the experiments can be found in the accompanying zip folder and on GitHub https:
7/g1 www......m.aa.........a hwt.:///aarr...aa.ppppsss/gaaaae
22

---

Published in Transactions on Machine Learning Research (10/2024)
A Computational and memory efficiency
Table 8: Training Cost (per iteration of first order optimizer) and Memory Cost (for saving the trained model's parameters) for Different Methods. Assume (only) for this table the following notation: n number of data points in the minibatch; p number of trainable parameters the architecture; samples from parameters for gradient estimation; f additional flow parameters; w speed-up factor due to sampling directly from neurons (number of weights per neuron); ratio of non-pruned weights. The training iteration cost reflects the computational complexity for each method. The memory cost includes considerations for additional parameters and storage efficiency, especially for pruned weights. A non-pruned weight is assumed to need a double-precision number to be stored (64 bits). In the pruned model, we further assume to need 1 bit per position of whether the weight there is pruned.
number of
Method
Training Cost.
Memory Cost (bits)
Sparse Memory Cost (bits)
ANN
O(n p)
64 p
Not applicable
BNN BNN-FLOW
On 2p-
64 2p
Not applicable
O (2p f) 8//))
64 (2p +
Not applicable
LBBNN
Onn 3p-s)
64 3p
2p + 65 (1 -a) -2p
LBBNN-LACT
On 3p s/w)
64 3p
a: 2p + 65 (1-1))0)))
15-aa -a) -a) 2p
LBBNN-FLOW
(n (3p + 8/w)
64 (2p + f)
a 2p + 65 ((1 - a) 2p) + 64
B Additional simulation studies
In this section, we describe three additional simulation studies, conducted to evaluate the performance of Bayesian variable selection using the same approaches as in Section 5.1 of the paper. Just like in the extension from Section 5.1.1., the studics involved generating simulated datasets and assessing the True Positive Rate (TPR) and False Positive Rate (FPR) of selected by different methods variables across 3 different signal- to-noise ratios and 10 correlation structures per signal level. 20 simulations on different seeds were run per setting per method, resulting in 600 simulations per method in each of these three studies. Bayesian variable selection performed using the median probability model. We do not tune any methods but rather use exactly the same hyperparameters for each experiment as we did in Section 5.1. However, we use different data-generative processes in each study. Below, we explain these processes.
B.1
Data generating process
B.1.1
Experiment 1:
For the first experiment, we use the Simrel package (Sæebo et al. 2015). which allows for simulating correlated data where the level of correlation can be controlled through the parameter, which is the declining (decay- ing) factor of cigenvalues of predictors. The parameter was ranging between 0.1 and 1. For this experiment, we have predictor variables, and sample 1000 data points, with the true effect vector defined as:
B=
1,1 1.I,..... 0,0, .
(13)
10 true signals noise covariates
We then generate the contimous outcome as: 7) x t G G~ G~N(000²)), E 110,1,0.1) Finally, we convert the continuous outcome to a binary outcomes as = 11 > median(n)).
B.1.2 Experiment 2:
Here, we simulate i.i.d. vij (00 1), then induce the following correlation structure: For a correlation parameter á E {1, 0.9......,, 0.1), update the covariates as: Fzjj = p æi.--10 (1-A) p) Zty.i= 1,.., 1000. 11, 11,...50. 50. The resulting predictors are then standardized, followed by binarizing all even indexed covariates
23

---

Published in Transactions on Machine Learning Research (10/2024)
We further define the true using the median threshold, ie. Fij I[ra, mxdian(xy)).; (2,,,..,,50 effect vector B as:
3=
,1,...,899
310 0,0 0,0,...,
(14)
10 true signals noise covariates
with Bi.... 810 (1), We then generate the binary outcome the same way as in Experiment 1, but with E (50, 10, 1}.
B.1.3
Experiment 3:
Finally, we take inspiration from the Rejoinder to Hubin al. (220 and simulate Quantitative trait locus (QTL) mapping using R/QTL package (Broman al., 2003). Here, just in Section 5.1, have block- randomly select each 3, to be a true parameter with probability of 0.25. The true 8 are then sampled as 8 ~ (1, Here, we generate the response as: = = o;), y1 ~ Bernoulli [Fexpp(m.] ), and we use a E (1.0.1,0011}
diagonal correlation structure between the covariates, Further, in this experiment,
B.2 Results
10
0=1 0=0.1
d=
0=1
a=
FLOW
FLOW
C5
FPR
0.4
0.4
0.2
0.2
0.0
0.0
0.471
0.442
0.377
0.277
0.138
0.496
0.49
0.384
0.266
0.137
0.504
0.432
0.345
0.286
0.142
0.471
0.442
0.377
0.777
0.138
0.496
0.43
0.384
0.266
0.137
0.504
0.422
0.345
0.286
0.142
Mean correlation in the design matri
Mean correlation in the design matrix
Figure 7: On the left, true positive rates, with false positive rates to the right. The maximum absolute
pairwise correlations between the covariates ranged between 0.102 and 0.994.
50
== 10
a=1
50
10
o=
1.0
1.0
LCRT
LCRT
FLOW
FLOW
0.8
0.8
0.6
0.6
TPR
FPR
0.4
0.4
0.2
0.2
0.0
0.0
0.1
0.072
0.049
0.035
0.027
0.101
0,074
0.048
0.035
0.027
0.1
0.073
0.049
0.035
0.026
0.1
0.072
0.049
0.035
0.027
0.102
0.074
0.048
0.035
0.027
0.1
0.073
0,049
0.035
0.026
Mean correlation in the design matrix
Mean correlation in the design matrix
Figure On the left, true positive rates, with false positive rates the right. The maximum absolute pairwise correlations between the covariates ranged between 0.102 and 0.995.
The results for these additional simulation experiments 1,2 and 3 can be found in Figures 7 and g respectively. In experiment 1, we see that the CS method generally has higher TPR, whereas the results
24

---

Published in Transactions on Machine Learning Research (10/2024)
1
0= 0.1
0 0.01
0=
0= 0.1
a= 0.01
1.0
1.0
LCRT
LCRT
FLOW
FLOW
0.8
0.8
C5
0.6
0.6
TPR
EPR
0.4
0.4
0.2
0.2
0.0
0.0
0.312
0.324
0.255
0.189
0.143
0,312
0.324
0.255
0.188
0.143
0. 0.312
0.324
0.255
0.18a
0.143
0.312
0.324
0.255
0.168
0.143
0.312
0.324
0.255
0.188
0.149
0. 0.312
0.324
0.255
0.188
0.143
Mean correlation in the design matri
Mean correltioo in the design matrix
Figure 9: On the left, true positive rates, with false positive rates to the right. The maximum absolute
pairwise correlations between the covariates ranged between 0.909 and 0.999.
are closer the FPRs, with the LCRT being slightly worse. In the second experiment, the results are quite close, with flows having slightly higher TPR for high levels, and slightly lower with smaller a. In the last experiment, we see that FLOW generally has the lowest FPR and highest TPR. As indicated in experiment and also in Sections 5.1 and 5.1.1, we can see the advantage of using flows when the covariance structure between the covariates is block-diagonal.
C Convolutional architectures
For convolutional layers, the variational distribution defined to be:
1hg.
4o W 2)
II
(igh Fudiikk ozyx) + (1 Yigk (gk))
(15)
(=1 1=1j=1ke
Gaye (1i5k) -eBenoull(Sygs' (eyj))
where nh, They and ny denote the height, width, and number of filters in the convolutional kernel. For the convolutional layers, use the following for the inverse normalizing flows:
V = ((Mat(W @ P)e) @ d1) (1 ④ (m)) -1)
(16)
log 7 = ((Mat(W + D)e) @ d2) (1 ④ (mA))e))) -1).
Here, Mat(-) denotes the matricisation operator (as defined in Louizos & Welling (017)), i.e. changing the shape of multidimensional tensor into a matrix.
Experiments on tabular datasets
In this experiment, we consider six classification datasets and three regression datasets. We compare our approach L.BNNN-F.OW against LBBNN-I.CRT. LBBNN, a dense BNN, BNN-FLOW, ANN+T22 ANN, and Monte Carlo (MC) dropout (with dropout ratcs corresponding to our prior inclusion probabilities) addition Gaussian processes for the regression datasets. Here, use an out-of-the-box version of the package from Varvia et al. (2023). using the Matérn 3/2 covariance function (see chapter 4 of Rasmuussen (2003) for how this is defined). For the mean function, we use the training data average. For the covariance function, the implementation defines three hyperparameters: the kernel variance and length scale, controlling the smoothness properties of the regression, and the error variance parameter, regulating how closely to the training data. We define these parameters to 1, 5, and 0.1 respectively.
For the neural networks, we use single hidden layer with 500 neurons and again train for 250 epochs with the Adam optimizer. We 10-fold cross- validation and report the minimum, mean and maximum accuracy
25

---

Published in Transactions on Machine Learning Research (10/2024)
over these 10 repetitions, in addition to the mean sparsity. We also report the expected calibration error (ECE), in addition to PWлíC, and PWAIC; for the classification datasets, while for the regression datasets we report RMSE, the pinball loss, PWAIC, and PWAIC2'
For the six classification datasets, most are taken from the UCI machine learning repository. The Credit Approval dataset (Quinlan) consists of 690 samples with 15 variables, with the response variable being whether someone gets approved for a credit card or not. The Bank Marketing dataset (Moro et al. 2012) consists of data (45211 samples and 17 variables) related to marketing campaign of Portuguese banking institution, where the goal is to classify whether the persons subscribed to the service or not. In addition to this, we use the Census Income datasct (Kohavi] 1996) with 48842 samples and 14 variables, where we try to classify whether someone's income exceeds 50000 dollars per year. Additionally, we have three datasets related to classifying food items. The first, the Raisins dataset (Cinar et al. 2020), consists of 900 samples and 7 variables, where the goal to classify into two different types of raisins grown in Turkey. Secondly, we use the Dry Beans datasct (UCT, 2020). consisting of 13611 samples, 17 variables, and 7 different types of beans. Lastly, the Pistachio datasel. (Ozkan el. al. 2021 consists of 2148 samples and 28 variables, with two different types of Pistachios.
For the regression datasets, we use the abalone shell data (Nash et al. 1995 J. the wine quality dataset (Cortez
et al. 2009), and the Boston housing data (Harrison Jr & Rubinfeld 1978). ) To avoid model misspecification, responses were standardized for the regression datasets. The variance the responses was then assumed
fixed and equal to 1 in all of the models, while the mean parameter was modeled.
The predictive accuracy and density for the classification datasets can be found in Table g and Table 10 and the expected calibration error in Table 11] Finally, have the PWAIC, and РWАIC₃ metrics in Table 12 and 13 respectively. For the regression datasets, we have the RMSE and density results in Table 14 in addition to pinball loss in Table 15 and PWAIC, and PWAIC, in Table 16
For the classification datasets, we see that our LBBNN-FLOW method typically performs well compared to the LBBNN baseline, mostly having higher predictive power, also with the median probability model. It also performs well compared our other baseline methods. On calibration, we see that our method again performs well compared to baselines. Finally, for the PWAIC, and PwAtCa: probability model typically reduces this metric. We note that our method often has lower values than BNN-FLOW and BNN, even though these methods have fewer parameters. As mentioned before, however, it is unclear how good of an estimate this is for the effective number of parameters in highly non-linear methods such as neural networks. For the regression examples, our method performs slightly worse than some baselines in terms of RMSE and pinball loss, however, LBBNN-ICRT performs best (on average) on one dataset.
that using the median
26

---

Published in Transactions on Machine Learning Research (10/2024)
Table 9: Performance results on the Credit Approval, Bank Marketing, and Census Income datasets, using 10-fold rros-validation.. The minimum, mean and maximum accuracies are reported, in addition to the density. The best results are bold.
Credit Approval
Median probability model
Dense model
Method
min
mcan
density
min
mean
max
density
LBBNN
81.16
85.22
91.30
0.431
81.16
85.80
91.30
1.000
LBBNN-LCRT
81.16
86.23
92.75
0.347
81.16
86.23
91.30
1.000
LBBNN-FLOW
84.10
88.55
94.20
0.348
82.61
87.68
91.30
1.000
BNN-FLOW
82.61
86.23
89.86
1.000
BNN
78.26
83.33
88.41
1.000
ANN
78.26
83.19
91.30
1.000
ANN + L2
73.91
83.19
89.86
1.000
ANN + MC dropout
81.16
86.52
92.75
1.000
Bank Marketing
Median probability model
Dense model
Method
min
mean
max
density
min
mean
max
density
LBBNN
89.75
90.66
91.74
0.430
89.75
90.61
91.43
1.000
LBBNN-LCRT
90.75
91.27
92.16
0.347
90.60
91.27
92.16
1.000
LBBNN-FLOW
90.58
91.38
92.08
0.347
90.75
91.36
92.03
1.000
BNN-FLOW
90.14
91.13
91.96
1.000
BNN
90.63
91.16
91.74
1.000
ANN
90.93
90.97
91.62
1.000
ANN + L2
90.75
91.15
91.77
1.000
ANN + MC dropout
90.70
91.17
92.06
1.000
Cencus Income
Median probability model
Dense model
Method
min
mean
max
density
min
mean
max
density
LBBNN
85.24
85.74
86.49
0.431
85.52
85.85
86.69
1.000
LBBNN-LCIT
85.36
85.90
86.77
0.349
85.63
85.92
86.57
1.000
LBBNN-FLOW
85.60
86.04
86.43
0.349
85.57
86.05
86.49
1.000
BNN-FLOW
85.05
85.32
85.79
1.000
BNN
84.77
85.27
86.45
1.000
ANN
84.56
85.09
85.54
1.000
ANN + L2
84.73
85.39
85.91
1.000
ANN + MC dropout
85.32
85.89
86.49
1.000
27

---

Published in Transactions on Machine Learning Research (10/2024)
Table 10: Performance results on the Dry Beans, Pistachio, and Raisin datasets, using 10-fold cross- validation. The minimum, mean and maximum accuracies are reported, in addition to the density. The best results are bold.
Dry Beans
Median probability model
Dense model
Method
min
mcan
max
density
min
mcan
max
density
LBBNN
90.88
92.65
93.90
0.442
91.25
92.80
93.82
1.000
LBBNN-LCRT
91.62
93.18
94.41
0.349
91.69
93.34
94.34
1.000
LBBNN-FLOW
89.26
92.38
93.90
0.279
89.85
92.57
94.19
1.000
BNN-FLOW
91.32
93.05
94.19
1.000
BNN
91.47
93.35
94.63
1.000
ANN
91.47
93.37
94.71
1.000
ANN + L2
91.54
93.38
94.71
1.000
ANN + MC dropout
91.40
92.96
94.04
1.000
Pistachio
Median probability model
Dense model
Method
min
mean
max
density
min
mean
max
density
LBBNN
91.12
93.46
96.26
0.433
91.12
93.36
95.33
1.000
LBBNN-LCIT
91.59
93.93
95.79
0.350
92.06
94.07
95.79
1.000
LBBNN-FT.OW
91.12
93.46
95.33
0.350
91.12
93.46
95.33
1.000
BNN-FLOW
90.65
93.60
96.26
1.000
BNN
92.52
94.07
96.26
1.000
ANN
92.06
94.11
96.73
1.000
ANN + L2
92.06
93.93
96.26
1.000
ANN + MC dropout
91.12
94.16
96.26
1.000
Raisins
Median probability model
Dense model
Method
min
mcan
max
density
min
mean
max
density
LBBNN
83.33
87.00
92.22
0.439
83.33
86.78
92.22
1.000
LBBNN-LCRT
81.11
86.11
91.11
0.349
81.11
86.78
92.22
1.000
LBBNN-FLOW
83.33
86.67
92.22
0.349
82.22
86.56
92.22
1.000
BNN-FLOW
82.22
87.22
91.11
1.000
BNN
81.11
87.89
92.22
1.000
ANN
81.11
86.44
90.00
1.000
ANN + L2
81.11
87.56
92.22
1.000
ANN + MC dropout
81.11
87.00
93.33
1.000
Table 11: Expected calibration error, with minimum, mean, and maximum values obtained using 10-fold cross-
validation. MPM denotes the medium probability model.
ECE (mii, muean, 20x))
Methas
Credit Approval [0.056, 0.079, 0.127)
Bank Mariseting (0.025, 0.031, 0.639;
Census Inscome
Dry Beans
Pistachio
Raisiae
LBBNN
(0.005, 0.010, 0.015)
[0.011, 0.016, 0.021)
(0.012, 0.028, 0.017)
(0.043. 0.073, 0.097)
LBBNN-MPM
[0.05, 0.880, 0.12
(0.033, 0.025, 0.033;
(0.006, 0.011, 0.016)
(0.002, 0.015, 0.031
(0.014, 0.028, 0.043)
(0.055, 0.069, 0.108)
ABRNN-LCRT
[0035, 0.071.0.0077 (0.97))
(0.01, 0.016, 0.020)
(0006, 0.010, 0.012)
(0.008, 0.014, 0.019)
(0.014, 0.022, 0.034)
(0.034 0.070, 0.094)
LOONN- MONNN-LCR-MMPN
(0.015, 0.073, 0.119)
(0.011, 0.015, 0.018;
(0.008, 0.012, (.016)
(0.007, 0.013. 0.025)
(0.012, 0.025, 0.005)
(0.057, 0.076. 0.115)
FLOW
[0.000, 0.069. 0.103)
0.007. 0.015)
(0.007,
0.016)
[0.010,
0.020)
(0.011, 0.024, 0.015)
(0.018.
0.100)
MUINN-FLOWAEE
(0.025, 0.071, 0.126)
(0.003, 0.008, 0.012:
(0.009, 0.012, 0.015
(0.008, 0.014, 0.026)
(0.012, 0.000. 0.047)
(0.038, 0.072, 0.096)
FLOW
[0.025,
0.012, 0.022)
(0.020.
0.017)
[0.007,
0.029)
(0.010, 0.029, 0.055)
(0.038. 0.089, 0.104)
BNN
(0.016, 0.097, 0.195)
(0.010, 0.015, 0.024)
(0.020, 0.025, 0.081)
(0.005, 0.013, 0.030)
(0.010, 0.089, 0.067]
(0.031, 0.070, 0.108)
ANN
(0072,
(D.022 0.017, (.028)
(0020, 0.031)
(0,007 0.013. D.221)
(0.027
(0040) 0.074, 0.109)
ANN 122
(0.020, 0.114, 0.178)
(0.013, 0.018, 0.024;
(0.055, 0.020, 0.026)
(0.007, 0.013. 0.020)
(0.014, 0.040, 0.053)
(0.025, 0.066, 0.111)
ANN MC dropous
[0,028,
(D.011, 0.014,
(04088 0.018)
(00833,
0.029, 0.061)
(0/27,
28

---

Published in Transactions on Machine Learning Research (10/2024)
Table 12: The PWAIC, metric, on our six classification datasets, where the minimum, mean, and maximum values are obtained with 10-fold vols-vaali
Phytan, (suii, mEmiL BUXXI
Medlod
CraLi Approval
Bank Markeiing
Crnaa Insxera:
Dry Bosit
Flsachio
Rataine
LBENN
115.62,
0.271,
LBDNN-MPM
(0.000, 0.000. 0.000)
(0.001, 0.002, 0.003)
(0.002, 0.002. 0.003)
(0.001. 0.002, 0.003)
(0.000, 0.000, 0.000)
(0.000, 0.000, 0.000)
LBBNN LORT
(0.013, 0,026, 0.041)
(0.575, 0.613, 0.725)
(0.907. 1.164, L.357)
(9.507. 13.32, 20.12)
(0.015, 0.054, 0,052;
(0.011, 0.023, 0.013)
LBENN BEINNN-CRT-MIEE LCRI-MPM
0,025,
(1.000 0.545. 0.718)
1.140,
(1,016, 0,054)
(0.011 0024, 0.045)
00.00 0.021, 0.033)
(0486; 0.542, 0.008
(1.845, 1.216, 1.488)
(12.83 17.66, 25.65)
(0.01,
0.058)
(0001] 0.05, 0.022)
FLONN-FLOW-MPM
(0.00) 0,022, (.03F)
(14476. 0.544, 0.021)
(1.065, 1.245, 1.501]
(0.311 0.420, 0.567)
(0.115, 1.113,, 0.169)
(0.011, 0026, 0.066)
BNN-FF.OW
(0.012, 0.029, 0.054)
(0.523, 0. 0.563, 0.635)
(1.255, 1.398, 1.745)
(0.462) 0.309. 1.089)
(0.017, 0.059, 0.062;
[0.012, 0.05, 0.055)
0.055,
1.442)
0.007,
0,073;
0022,
MC dropous
(0.016, 0.058, 0.248)
(0.566 0.600, 0.657)
(1.057, 1,201, 1.353)
(75 00, 105.3, 133.2)
(0.027, 0.040, 0.059:
(001), 0.077 0.022)
Table 13: The PWAIC, metric, on our six classification datasets, where the minimum, mean, and maximum values are obtained with 10-fold cross-valddatio.
PoiAto (mith, men.. max)
Method
Credit Approva 1.307,
Bank Marlorting
Cenous Tncome
Dry Boans
Pistachio
Raisine
LRBNN
65.53, 306.7
32.75)
5,058)
0.909, 0.647)
LBBNN-MPM
(0.000, 0.000, 0.000)
(0.001, 0.002, 0.003)
(0.002, 0.002, 0.003)
(0.001 0.002, 0.003)
(0.000, 0.000, 0.000)
(0.000, 0.000, 0.000)
LRBNN-I CRT
(0.021, 0.058, 0.111)
[10016 1.176, (.73)
(1.588, 2,081, 12:011
(10.55, 13.96, 20.86)
(0.028, 0.078, 0.158:
(0.05, 0.044, 0. 101)
LBENN LCRT MPM
00.017. 0.056, 0.114)
(0585, 1.188, 1.374)
(1,1639, 2,011. 2.217)
(0.002, 0.002, 0.005)
(0.026, 0.073, 0.151;
(0.06) 0.047, 0.112)
LHBNN-FLOW
0,23, (.072)
(0778), 0.926, ..154)
7.965, 21.91)
(13.30, 18.88, 20.37)
(0.125,
0.137)
(0o06, 0.68, (.197)
LBENN- FLOW-MPM
(0.044 0.047, (.083)
(0755, 0.504, 1.19d)
(1.781, 8.940. 21.97)
(0.111, 0.421, 0.509)
(0.018 0,070, 0.126)
(0.01,, 0o89, 0.188)
BNK-FLOW
(0.020, 0.074, 0.139)
[08566 1.049, 1,303)
(2,230, 4.745, 12.53)
(0.403, 0.601, 1.003)
(0.030, 0.097, 0.178)
(0.01, 0.46,, 0. 0.128)
(0.005, 0.009)
0.264)
dropont
(0.027, 1,112, 10.13)
(0.05, 1,068, 1.351)
(1.767, 4.148, 12.00)
(101.0, 148.4, 265.8)
(0.054, 0.102, 0.216;
(0.15, 0.64, 0.181)
29

---

Published in Transactions on Machine Learning Research (10/2024)
Table 14: Root mean squared error, using 10-fold cross-validation. The minimum, mean and maximum are reported, in addition to the density. The best results bold.
Abalone
Median probability model
Dense model
Method
min
mean
max
density
min
mean
max
density
LBBNN
0.597
0.677
0.799
0.435
0.577
0.665
0.795
1.000
LBBNN-LCRT
0.564
0.644
0.736
0.350
0.560
0.641
0.722
1.000
LBBNN-FL.OW
0.577
0.660
0.767
0.350
0.572
0.657
0.761
1.000
BNN-FLOW
0,585
0.654
0.733
1.000
BNN
0.579
0.651
0.759
1.000
ANN
0.575
0.657
0.801
1.000
ANN T,2
0.572
0.652
0.764
1.000
ANN MC dropout
0.579
0.655
0.759
1.000
Gaussian process
0.570
0.650
0.734
1.000
Wine Quality
Median probability model
Dense model
Method
min
mean
max
density
min
mean
max
density
LBBNN
0.768
0.805
0.854
0.435
0.757
0.799
0.845
1.000
LBBNN-LCRTT
0.742
0.782
0.820
0.350
0.741
0.780
0.820
1.000
LBBNN-FLOW
0.751
0.788
0.822
0.351
0.750
0.787
0.823
1.000
BNN-FLOW
0.747
0.778
0.815
1.000
BNN
0.749
0.767
0.790
1.000
ANN
0.740
0.761
0.792
1.000
ANN + L2
0.746
0.763
0.789
1.000
ANN + MC dropout
0.758
0.799
0.847
1.000
Gaussian process
0.700
0.739
0.777
1.000
Boston Housing
Median probability model
Dense model
Method
min
mean
max
density
min
mean
max
density
LBBNN
0.267
0.412
0.621
0.431
0.262
0.393
0.552
1.000
LBBNN-LCRT
0.245
0.374
0.560
0.350
0.233
0.366
0.534
1.000
LBBNN-FLOW
0.267
0.402
0.595
0.352
0.253
0.395
0.561
1.000
BNN-FLOW
0.241
0.363
0.524
1.000
BNN
0.240
0.351
0.481
1.000
ANN
0.247
0.364
0.597
1.000
ANN + L2
0.234
0.346
0.487
1.000
ANN + MC dropout
0.244
0.371
0.502
1.000
Gaussian process
0.217
0.349
0.444
1.000
30

---

Published in Transactions on Machine Learning Research (10/2024)
Table 15: Mean pinball loss grid between 0.05 and 0.95 in increments of 0.05. Min, mean, and max values were obtained using 10-fold cross- validatoii
Mean pinball (min, mean, max)
Method
Abalone
Wine Quality
Boston Housing
LBBNN
(0.210, 0.236, 0.260)
(0.291, 0.310, 0.325)
(0.100, 0.133, 0.164)
LBBNN-MPM
(0.220, 0.242, 0.260)
(0.296, 0.313, 0.328)
(0.097, 0.137, 0.167)
LBBNN-L.CRT
(0.203, 0.227, 0.251)
(0.289, 0.304, 0.316)
(0.087, 0.121, 0.150)
LBBNN-LCRTTMMI
(0.204, 0.228, 0.259)
(0.290, 0.304, 0.318)
(0.089, 0.123, 0.151)
LBBNN-FLOW
(0.208, 0.232, 0.253)
(0.289 0.306 0.321)
(0.091, 0.132, 0.158)
LBBNN-FLOW-MPM
(0.209, 0.231, 0.261)
(0.290, 0.306 0.319)
(0.092, 0.133, 0.154)
BNN-FLOW
(0.211, 0.232, 0.250)
(0.293, 0.303, 0.317)
(0.085, 0.120, 0.142)
BNN
(0.210, 0.230, 0.255)
(0.286, 0.295, 0.307)
(0.084, 0.117, 0.145)
ANN
(0.210, 0.231, 0.255)
(0.279, 0.292, 0.301)
(0.087, 0.119, 0.149)
ANN + +2
(0.208, 0.230, 0.255)
(0.283, 0.293, 0.301)
(0.083, 0.114, 0.142)
ANN + MC dropout
(0.211, 0.231, 0.252)
(0.295, 0.312, 0.328)
(0.089, 0.125, 0.156)
Gaussian process
(0.210, 0.231, 0.252)
(0.267, 0.283, 0.295)
(0.079, 0.117, 0.145)
Table 16: The PWAIC, and PwAIC, metric, on our three regression datasets, where the minimum, mean and
maximum values are obtained with fo-dold cross- validation.
Phane: (min, moon HEE..
PHAI's [min, TINN, Tise)
Method
Asalone (19977 4,480, (4,09)
Wine Quality
Boeson Housing
Abaloc
Wine Quality
Boston Lloucing
LBBNN
(6.304. 7.496, 8.017)
(0.057. 0.202, 0.500)
(1,87, 5.206, 2001)
(6.050, 7,852, 0.642)
(0.088, 0,210, 0.505)
LRRNNAPM
0.002)
0.003)
0.000, 0.000)
(0.000,
0.001, 0.005)
0.000,
(0.766 1468,, 2.236)
(4.975. 5,220, 6.318)
(0.013. 0.128, (A83)
(0.774, 1.628, 2.505)
5.888, 6.989)
(0.013, 0.133. 0.514)
LORT-MLCRT-M
(0.00), (1001, 0.00))
(ouiIi,, (0004, 0.100)
(0.000, 0.000. 0.000)
(1.001 0.001, 10000)
(1.001. 0.0014, cuo0l))
(0.000, 0.000, 0.000)
LEBNN FLOW
(0.563, 1.921 7.515)
(3,220, 4,202, 6,009)
(0.025. 0 0.229, 1.025)
(0,008 2,191, 9.525)
(3.277) 4.407. 7.352)
(0.025, 0,238, 1.157)
LBBNN-FLOW-APPM
(6.001
0.077)
(.A32)
(0.001, 0.012, 0.078)
(0074, 0.237.
01008 2.230)
(0.00)) (1062)
Il477)
0.622, 2.415)
(0.004, 0,021. Q.02)
DNN
(0.002, 8.007. 0.045)
[0.014. (.019, 0.030)
(0.000, 0.000, 0.000)
(0,002, 0.007, 8.045)
(0.014, 0.015, OU009)
(O 0..000. 0. 0.000, 0.000)
MC dropont
(7.205, 15 12, 44.78)
[18.70. 22,81, 27.57)
(0.34. 1.721, 6,531)
(1,972, 36.82, $30.4)
(20.35, 28.71, 71.86)
(0.397, 2 2.858 12.45)
31

---

Published in Transactions on Machine Learning Research (10/2024)
E Resnet-18 architecture for Cifar 10
Results on the Cifar 10 dataset with the Resnet-18 architocture, where the min, median, and maximum measurements are obtained from running each method ten times.
Table 17:
Cifar 10
Expected calibration error
Negative log likelihood
Density
Method
(min, median, maz)
(min, median, max)
mean
LBBNN
(0.027, 0.037, 0.067)
(1.350, 1.424, 1.584)
1.000
LBBNN-MPM
(0.689, 0.763, 0.855)
(50.78, 91.90 245.3)
0.819
LBBNN-LCRT
(0.053, 0.057, 0.064)
(1.004, 1.039, 1.173)
1.000
LBBNN-LCRIIMPPE
(0.221, 0.243, 0.268)
(1.733, 1.934, 2.117)
0.386
LBBNN-FLOW
(0.059, 0.062, 0.067)
(0.915, 0.950, 1.026)
1.000
LBBNN-FLOW-MPM
(0.347, 0.380, 0.412)
(2.003, 2.230, 2.299)
0.359
BNN
(0.132, 0.139, 0.146)
(2.799, 3.046, 3.429)
1.000
BNN-FLOW
(0.091, 0.094, 0.103)
(1.079, 1.097, 1.136)
1.000
ANN
(0.120, 0.207, 0.213)
(3.444, 4.173, 5.029)
1.000
ANN + BN
(0.195, 0.200, 0.208)
(1.822, 1.930, 2.133)
1.000
32


Reviews:
[{"review_id": "L8YZztEhTs", "reviewer": "Reviewer_9rKf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThere is general interest in the Bayesian community in sparse spike-and-slab models and how to perform tractable posterior inference in them. This paper combines several difference modern frameworks / approaches (i.e., hierarchical VI, normalizing flows, BNNs) to provide an approach for fitting deep sparse models. Combining these frameworks involves some tailored derivations, which are correct so far as I can tell, and which may be of broader interest. \n\n## Weaknesses\n\nThe paper does a poor job of motivating the proposed model, and fails to empirically substantiate its claimed advantages. The paper is vague throughout about why it is interested in developing LBBNNs. At various points, the paper suggests that LBBNNs might be motivated by the following aspects:\n- improved variable selection (particularly when covariates are correlated)\n- improved predictive performance\n- improved uncertainty quantification\n- improved computational efficiency\n- improved memory efficiency \n\nI think any one of these aspects would be interesting and motivating. However, the paper does not commit to any one of them being the main motivation, and the experiments do not convincingly show a major advantage along any one of them. I'll walk through each aspect below.\n\n**Variable selection**: The paper includes one experiment on this task which involves one simulated data set. The proposed approaches are compared to the logistic regression baseline of Carbonetto & Stephens (2012). Both proposed approaches have better TPR and FPR than the baseline on the one simulated data set. This is a good result, but it may be due entirely to properties of the single synthetic data set. The paper should simulate many synthetic data sets, and report error bars across them. Moreover, the claim of the paper is that the proposed approach will work better than the baseline as the covariates become more correlated. This claim would be substantiated by a plot showing that performance relative to the baseline improves as the covariates become more correlated in a series of simulated data sets.\n\n**Uncertainty quantification**: The paper shows that a baseline method based on a BNN with dropout \"has high [predictive] certainty most of the time\" while the baseline does not. This is measured using the entropy of the predictive distribution across a grid of covariate values. However, the paper does substantiate the implicit claim that higher certainty is bad in this context. Indeed, looking at the plots in Figures 6-8, I actually think Dropout looks the _best_ given how separated the training data are. It is trivial to create a new method that has higher entropy in its predictive distribution; but is that necessarily good? The paper isn't clear about that. \n\nIn the discussion, the paper also claims that the \"calibration of uncertainties [...] is similar with a slight advantage of the proposed approach\". However, I did not see any theoretical statement about the calibration of the proposed approach's predictive probabilities, nor any experiment which tests it.\n\n**Predictive performance**: The experiments (at least as I understand them) do not show a major improvement in predictive performance. The paper says that the performance of the sparse models are \"comparable\" to the dense ones. This is interesting, but only so far as there is another reason to prefer the sparse models.\n\n**Computational / memory efficiency**: The paper is not clear whether the proposed approach improves in terms of efficiency over a BNN baseline. If it does in fact improve, a plot showing performance as a function of wall-clock time should be included.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers a class of deep probabilistic models known as latent binary Bayesian neural networks (LBBNN) wherein a binary variable $\\gamma_{ij}^{(l)} \\in$ {0,1} is associated with each weight $w_{ij}^{(l)} \\in \\mathbb{R}$ in the neural network, such that $w_{ij}^{(l)} =0$ if $\\gamma_{ij}^{(l)} =0$. The exact posterior $P(\\boldsymbol{W}, \\boldsymbol{\\Gamma} \\mid -)$ in these models is highly intractable due in part to the combinatorics of the binary variables. \n\nThe paper adopts a variational approach to inference. Previous work by Carbonetto & Stephens (2012) showed that mean-field variational approximations to the posterior of spike-and-slab Bayesian logistic regression tend to perform poorly at variable selection when the covariates are correlated. The paper thus embraces the hierarchical variational inference approach of Ranganath et al. (2016), wherein the variational distribution is augmented with a latent variable $\\boldsymbol{z}$, becoming $q(\\boldsymbol{W}, \\boldsymbol{\\Gamma} \\mid \\boldsymbol{z}) q(\\boldsymbol{z})$, which induces dependence among the weights. The paper further combines this with normalizing flows (Rezende & Mohamed, 2015), modeling $q(\\boldsymbol{z})$ flexibly as coming from a composition of invertible transformations.\n\nTo tractably fit the proposed model with the proposed hierarchical + normalizing flows variational family, the paper further introduces an approximation based on the local parameterization trick (LCRT) of Kinga et al. (2015). The final fully specified proposed approach is referred to as LBBNN-FLOW. The paper then also proposes a simplified version of the approach that assumes the $\\boldsymbol{z}$ are all equal to 1, which it refers to as LBBN-LCRT. (Both approaches involve the LCRT approximation.)\n\nThe experiments show that:\n- The proposed approach performs better than the original logistic regression baseline by Carbonetto & Stephens (2012) on a variable selection task\n- The proposed approach has higher predictive uncertainty than a non-Bayesian NN\n- The proposed approach obtains comparable classification accuracy on *NIST data sets to non-sparse models", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper needs to be significantly revised to say clearly why LBBNNs are motivated, and in what aspects the proposed model improves over current practice. I am very open to any of the above-mentioned (or other) aspects, but the paper needs to commit to one (or more) and then provide experiments that clearly show improvement along those dimensions. \n\nI mentioned some of these possible experiments in the above section but here they are again:\n- If the paper wants to claims that LBBNNs improve in terms of variable selection, there needs to be a much more rigorous set of experiments that substantiate this claim, which include many simulated data sets, at different levels of covariate correlation. \n- If the paper wants to claim that LBBNNs improve in terms of predictive performance, than we need to clearly see evidence of that. \n- If the paper wants to claim that LBBNNs improve in terms of uncertainty quantification, then the paper needs to defend the implicit claim that higher predictive entropy is actually good, and correct in the given setting.  Moreover, any claims of calibration must be defended, theoretically or empirically. \n- If the paper wants to claim an improvement in efficiency, we should see performance as a function of wall-clock time.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gsiNa0Fu6J", "reviewer": "Reviewer_F54M", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- To make their ideas easy for readers to understand, they effectively explained their ideas through many figures and explanations.\n\n- They present various types of meaningful experimental results.\n\n# Weaknesses\n\n- I believe the proposed method lacks novelty since it combines existing methods. However, if the experimental evidence strongly supports the proposed method, this may not be a significant issue.\n\n- There is insufficient discussion on other sparse BNNs. Specifically, there are papers published in top conferences such as [1], [2], [3], and [4]. These should be mentioned, and the proposed method should explain how it differs from them.\n\n- I am concerned that the baseline is insufficient. Currently, the only sparse baseline method used for comparison is LBBNN. Given the lack of novelty in this paper, it is essential to compare its performance with other methodologies.\n\n- I am also concerned that classification experiments were conducted only on the MNIST dataset (and its variations) and the LeNet model, which are too simple. Also, there is no report on uncertainty measures in the manuscript.\n\n[1] Molchanov, D., Ashukha, A., & Vetrov, D. (2017, July). Variational dropout sparsifies deep neural networks. In International conference on machine learning (pp. 2498-2507). PMLR.\n\n[2] Deng, W., Zhang, X., Liang, F., & Lin, G. (2019). An adaptive empirical Bayesian method for sparse deep learning. Advances in neural information processing systems, 32.\n\n[3] Bai, J., Song, Q., & Cheng, G. (2020). Efficient variational inference for sparse deep learning with theoretical guarantee. Advances in Neural Information Processing Systems, 33, 466-476.\n\n[4] Li, J., Miao, Z., Qiu, Q., & Zhang, R. (2024). Training Bayesian Neural Networks with Sparse Subspace Variational Inference. arXiv preprint arXiv:2402.11025.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, they consider two extensions of Latent Binary Bayesian Neural Networks (LBBNN): the first is the use of the local reparametrization trick, and the second is the use of normalizing flows. They provide experimental results for their methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "- It would be beneficial to explain other sparse BNN papers (at least [1], [2], [3], [4]) and discuss how the proposed method differs from them.\n\n- It would be beneficial to experimentally compare with 2-3 of other sparse BNN methods (e.g., [1], [2], [3], [4]).\n\n- It would be beneficial to include experiments using more challenging datasets (e.g., CIFAR10, CIFAR100, SVHN) and more complex models (e.g., ResNet18). Additionally, it is recommended to report the Negative Log-Likelihood (NLL) or Expected Calibration Error (ECE) results for image classification.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZnVgFyhW28", "reviewer": "Reviewer_NqKg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe paper's strengths have stayed the same. The authors' model and inference procedure are a natural extension of previous methods. Their method description in Sections 2-4 is clear and easy to follow, and they have also further polished it since the last submission. Moreover, the authors have conducted a significant amount of ablation studies to verify the effectiveness of their proposed changes.\n\n## Weaknesses\nHaving carefully re-read the paper, I think the content of the paper is mostly acceptable. However, I have some issues with the claimed contributions. Concretely, the authors claim four things:\n1. \"extending the class of variational distributions to normalising flows allowing modeling dependencies;\" - This claim is too vague. It is unclear what class they extend, what dependencies they are modelling, and what the extension's purpose is.\n2. \"improvements of computational efficiency in LBBNNs through the use of the local reparametrisation trick (Kingma et al., 2015);\" - This claim is again too vague; please specify concrete metrics and results.\n3. \"demonstrating improvements in predictive power, sparsity, and variable selection through experiments on real and simulated data;\" - The relevance of these contributions is unclear.\n4. \"demonstrating robust performance in uncertainty quantification through the expected calibration error for classification and Pinball loss for regression.\" - All pinball loss regression-related content is in the appendix. If the authors want to claim this, then this content should appear in the main text.\n\nBesides this, I am concerned about one technical claim in Section 5.2. The authors claim, \"When we increase the amount of data, we can see that the Bayesian networks gradually get more certain about predictions, and the entropies (as desired) start to converge towards the data-generative one.\" At best, this statement is meaningless and at worst, it is misleading. If the authors wish to make such a claim, they must provide a formal statement and proof of the result.\n\nIf the authors address my concerns above, I believe it should conform to TMLR's acceptance criteria and could be published.\n\nHaving said that, I am sad to say that despite the significant experimental effort on the authors' part, the paper's presentation is severely lacking. \n\nFirst, the purpose of Section 1.1 is unclear to me. In my review of the previous submission, I complained that the introduction read like a literature review, so the authors wrote a better introduction. However, instead of removing their previous introduction, they moved it to Section 1.1 and renamed it \"Literature Background.\" I have two issues: 1) The introduction (which is reasonably well-written) and Section 1.1 are very similar in content. Furthermore, Section 1.1 has been repurposed as the literature background in a cursory way, as there are entire sentences (!) in it that match verbatim with sentences in the introduction, including the first sentence! 2) Section 1.1 still contains a lot of unnecessary content that does not help contextualise the authors' work. For example, it reviews early neural networks and gradient descent; these sections should be removed.\n\nI have similar issues with the description of the experiments in Section 5, which is a whopping nine and a half pages long. While most of the content is technically fine (apart from the one claim I take issue with above), the presentation leaves much to be desired. It contains long arguments, with the experiment setup, method description, and interpretation of results blended together, making it difficult to follow. I believe that with some effort to streamline the arguments, the authors could cut the length of Section 5 in half at least.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I served as a reviewer (K8JC) for this paper's first submission; see my first review at https://openreview.net/forum?id=t3OGrWRUve&noteId=0vpySh1cZ6. As the main focus and arguments of the paper haven't changed, I repeat my earlier summary below.\n\nThe authors propose modelling and optimisation improvements to latent binary Bayesian neural networks (LBBNN): Bayesian neural networks with a spike-and-slab prior and a variational spike-and-slab posterior on the weights. The authors propose using a conditionally factorised posterior approximation instead of a fully factorised one to improve the expressivity of the variational posterior and use a normalising flow to model the posterior distribution of the newly introduced conditioning variable. They perform some toy experiments on MNIST-type datasets and synthetic data.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Necessary:\n - As I outlined in the weaknesses section, please clarify the claimed contributions and point to precise advances/results to back them up.\n - Please clarify the claim about the convergence I note in the weaknesses section.\n - The bibliography style needs to be more consistent; for example, authors' names appear inconsistently; please fix it.\n - Figures 6-8: Some data points fall outside axes, so please clip these or enlarge the axes.\n - Figure 9: x-axis labels are missing; please fix this.\n\nNice to have:\n - Any improvement in the writing that would address the issues I outline regarding it in the weaknesses section above.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The conclusion of the reviewers of paper 2153 was the paper proposes an algorithm that is an almost straightforward but potentially useful combination of existing methods. However, they insisted that in order to prove this usefulness, a better discussion of the experimental results was necessary.

Reviewer K8JC of paper 2153 was able to check the new version, under the new pseudonym NqKg. However, Reviewers A9N5 and WWin of paper 2153 were unfortunately not able to review this new version. This is not ideal, but this gave me no choice but to invite two new reviewers, F54M and iDhV. Unfortunately, iDhV him/herself let us down. I thus had to find an emergency reviewer, 9rKf, in the middle of the summer, which led to a very long delay for this paper. I apologize to the authors for this delay, and take the opportunity to thank reviewer 9rKf.

The comments of NqKg are particularly important as this reviewer requested changes on paper 2153. He/she found this new version much better, but still believes that some of the flaws from the previous version remain: some unsupported claims, and lack of clarity in the exposition of the method.

Reviewer F54M was more positive on the readability and praised the authors' effors to include many figure to explain their method. However, he./she also raised point quite close to the ones that were raised for paper 2153. Some of them (lack of novelty, comination of existing ideas) are not part of the decision criteria, but some are quite important, for example, the lack of adequate baselines for comparison in the experiments.

Finally, while his/her opinion seems overall positive, reviewer 9rKf shares the opinion of NqKg (and of the reviewers of paper 2153) about the lack of clarity and the unsupported claims: many reasons are invoked for the use of Bayesian methods in deep learning, but the paper does not provide enough evidence that these promises are fullfilled.

The authors took some time to reply to these comments in depth, and submitted a revised version.

Reviewer 9rKf believes that his/her comments were taken into account seriously: "I appreciate the authors' revisions and response", " The authors' revision now contains a helpful list of the paper's contributions". This leads this reviewer to support publication: "this contributions are fairly subtle, but I now appreciate that the paper's main thrust is in adapting normalizing flow-based inference to LBBNNs." Reviewer NqKg also supports publication, after a minor revision: "I think the presentation (with the exception of two revisions I suggest below) is now adequate." Reviewer F54M, on the other hand, acknowledges that the authors response on the lack of baseline makes sense, but still believes that this makes the results of the paper slightly below the bar for publication. On the other hand, he/she does not oppose publication if it us supported by the other reviewers.

I must say that I share the impression of 9rKf and NqKg: the first version implemented some of the changes requested on paper 2153, but not completely, however the version submitted after the discussion with the authors is now satisfying. I will therefore recommend to accept the paper under minor revision, so that the authors can fix the figures as requested by NqKg.
Decision: Accept (poster)