Collapsed Inference for Bayesian Deep Learning
Authors: Zhe Zeng
Guy Van den Broeck
Venue: NeurIPS 2023 poster (2023)
Keywords: Bayesian Model Averaging
Weighted Model Integration
Bayesian Deep Learning
Collapsed Inference
TL;DR: We reveal a previously unseen connection between inference on Bayesian neural networks and volume computation problems, based on which we introduce a novel collapsed inference scheme that performs Bayesian model averaging using collapsed samples.

Abstract: Bayesian neural networks (BNNs) provide a formalism to quantify and calibrate uncertainty in deep learning. Current inference approaches for BNNs often resort to few-sample estimation for scalability, which can harm predictive performance, while its alternatives tend to be computationally prohibitively expensive. We tackle this challenge by revealing a previously unseen connection between inference on BNNs and volume computation problems. With this observation, we introduce a novel collapsed inference scheme that performs Bayesian model averaging using collapsed samples. It improves over a Monte-Carlo sample by limiting sampling to a subset of the network weights while pairing it with some closed-form conditional distribution over the rest. A collapsed sample represents uncountably many models drawn from the approximate posterior and thus yields higher sample efficiency. Further, we show that the marginalization of a collapsed sample can be solved analytically and efficiently despite the non-linearity of neural networks by leveraging existing volume computation solvers. Our proposed use of collapsed samples achieves a balance between scalability and accuracy. On various regression and classification tasks, our collapsed Bayesian deep learning approach demonstrates significant improvements over existing methods and sets a new state of the art in terms of uncertainty estimation as well as predictive performance.

Collapsed Inference for Bayesian Deep Learning
Zhe Zeng
Guy Van den Broeck
Computer Science Department University of California, Los Angeles zhezengdcs ucla. edu
Computer Science Department University of Californa,, Los Angeles guyvdbëcs ucla. edu
Abstract
Bayesian neural networks (BNNs) provide formalism to quantify and calibrate uncertainty in deep learning. Current inference approaches for BNNs often resort to few-sample estimation for scalability, which can harm predictive performance, while its alternatives tend to be computationally prohibitively expensive. We tackle this challenge by revealing previously unseen connection between inference BNNs and volume computation problems. With this observation, we introduce a novel collapsed inference scheme that performs Bayesian model averaging using collapsed samples. It improves over a Monte-Carlo sample by limiting sampling to subset of the network weights while pairing it with some closed-form conditional distribution over the rest. A collapsed sample represents uncountably many models drawn from the approximate posterior and thus yields higher sample efficiency. Further, we show that the marginalization of a collapsed sample can be solved analytically and efficiently despite the non-linearity of neural networks by leverag- ing existing volume computation solvers. Our proposed use of collapsed samples achieves a balance between scalability and accuracy.
various regression and classification tasks, our collapsed Bayesian deep learning approach demonstrates significant improvements over existing methods and sets a new state of the art terms of uncertainty estimation as well as predictive performance.
Introduction
Uncertainty estimation is crucial for decision making. Deep learning models, including those in safety- critical domains, tend to estimate uncertainty poorly. To overcome this issue, Bayesian deep learning obtains a posterior distribution over the model parameters hoping improve predictions and provide reliable uncertainty estimates. Among Bayesian inference procedures with neural networks, Bayesian model averaging (BMA) particularly compelling (Wasserman 2000 Fragoso al.) 2018 Maddox et al. 2019). However, computing BMAs is distinetly challenging since it involves marginalizing over posterior parameters, which possess some unusual topological properties such as mode-connectivity (Izmailov et al.) 2021). We show that even with simple oow-dimensional approximate parameter posteriors as uniform distributions, doing BMA requires integrating over highly non-convex and multi-modal distributions with discontinuities arising from non-linear activations (cf. Figure [a). Accurately approximating the BMA can achieve significant performance gains (Izmailov et al.) 2021). Existing methods mainly focus on general-purpose MCMC, which can fail to converge, provides inaccurate few-sample predictions (Kristiadi et al. 2022), because running longer sampling chains mean-field approximation
computationally expensive, and variational approaches that typically that ignores correlations induced by activations (Jospin et al. 2022).
In this work, we are interested in developing collapsed samplers, also known as cutset or Rao- Blackwellised samplers for BMA. collapsed sampler improves over classical particle-based methods by limiting sampling to subset of variables and further pairing each sample with a closed-form
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

12
60
66
(a) p(y æ, w) being Gaussian. See Example 3
p(y æ. w) being triangular. See Section 3
Figure The integral surface of (a) the expected prediction in BMA, and (b) our proposed ap- proximation. Both are highly non-convex and multi- The z-axis is the weighted prediction y ply æ, w) p(w D). Integration of (a) does not admit closed-fore form solution, yet integration of (b) is close approximation that can be solved exactly and efficiently by WMI solvers.
representation conditional distribution over the rest whose marginalization is often tractable. Collapsed samplers are effective at variance reduction in graphical models (Koller & Friedman 2009). however no collapsed samplers are known for Bayesian deep learning. We believe that this due to the lack of a closed-form marginalization technique congruous with the non-linearity in deep neural networks. Our aim to overcome this issue and improve BMA estimation by incorporating exact marginalization over (close approximate) conditional distributions into the inference scheme. Nevertheless, scalability and efficiency are guaranteed by the sampling part of our proposed algorithm.
Marginalization is made possible by our observation that BMA reduces to weighted volume com- putation. Certain classes of such problems can be solved exactly by so-called weighted model integration (WMI) solvers (Belle et al. 2015a). By closely approximating BMA with WMI, these solvers can provide accurate approximations marginalization
BMA (cf. Figure [b].. With this observation, we propose CIBER, a collapsed sampler that uses WMI for computing conditional distributions. In the fawpsa setting, CIBER delivers more accurate uncertainty estimates than the gold-standard Hamiltonian Monte Carlo (HMC) method (cf. Figure 2 We further evaluate the effec- tiveness of CIBER on regression and classification benchmarks and show significant improvements over other Bayesian deep learning approaches in terms of both uncertainty estimation and accuracy.
Bayesian Model Averaging as Weighted Volume Computation
In Bayesian Neural Networks (BNN), given a neural network fw parameterized by weights w, instead of doing inference with deterministic w that optimize objectives such as cross-entropy or mean squared error, Bayesian learning infers posterior (w (D) over parameters w after observing data D. During inference, this posterior distribution is then marginalized to produce final predictions. This process is called Bayesian Model Averaging (BMA). can be seen as learning an ensemble of an infinite number of neural nets and aggregating their results. Formally, given input æ, the posterior predictive distribution and the expected prediction for a regression task are
= f
x ply |
ply | x, w) p(w D) dw,
and
E Ep(g = ply x) dy.
(1)
For classification, the (most likely) prediction is the class arg max, p(y x). BMA is intuitively attractive because it can be risky to base inference on single neural network model. The marginal- ization in BMA gets around this issue by averaging over models according
Bayesian posterior.
BMA requires approximations to compute posterior predictive distributions and expected predictions, as the integrals in Equation I are intractable in general. Deriving efficient and accurate approximations remains an active research topic (Izmailov al. 2021). We approach this problem by observing that the marginalization in BMA with ReLU neural networks can be cast as weighted volume computation (WVC). Later we show that it can be generalized to any neural network when combined with sampling. In WVC, various tools exist for solving certain WVC problem classes (Baldoni et al.)

---

-10
-12
-10
predicioo
preditiion
predicionn
-1.
-25
2.0
0.5
2.2
-0.
0.2
2.5
1.0
-10 -23
0.5
1.5
(a) HMC with 10 samples
(b) CIBER with 10 collapsed samples
(c) HMC with 2h samples
Figure 2: Uncertainty estimates for regression. The red line the ground truth. The dark blue line shows the predictive mean. The shaded region is the 90% confidence interval of the predictive distribution. For the same number of samples, (b) CIBER closer than (a) small-sample HMC to (c) highly accurate but slow HMC with large number of samples. See the Appendix for details.
2014
Kolb et al. 2019: Zeng al. 2020c). This section reveals the connection between BMA and WVC. It opens up a new perspective for developing BMA approximations by leveraging WVC tools.
Definition 1 (WVC), weighted volume computation (WVC) problem defined by a pair (EP, Φ) where a region tip is conjunction of arithmetic constraints and weight Φ 60 + R is an integrable function assigning weights to elements in (10, The task of WVC compute the integral Jm o(z) da.
2.1 A Warm-Up Example
Consider a simple yet relevant setting where the predictive distribution ply x, w) is Dirac delta distribution with zero mass everywhere except at (w)) such that f p(y x, w) dy (z).
Example 2. Assume a model (w) = ReLU(w x) with a uniform posterior over the parameter: p(w D) & with E [-3,3]. Let the imput be 1. For parameter
[-3, 0). the model Jm always predicts O, and otherwise (i.e. w E (0, 3) it predicts w. Thus, the expected prediction dw. That a summation of two WVC problems 1, 0) and (CDT,w) w/6 with D (-3 S S 0) and TT = (0 S 3). The BMA integral decomposes into WVC problems with different weights due to the ReLU activation.
(Equation is Eptyie) (z) [y] = -b 0 dw 150
These WVC problems have easy closed-form solutions. This no longer the case in the following.
Example 3. Assume a model fw (z) and posterior distribution p(w | D) as in Example 2 Let the predictive distribution ply x, w) be Gaussian distribution pN (y: fw (z), ) 1) with mean fw (x) and variance 1. Given input x = 1, the expected prediction (Equation) 7 is
Eptsl==1]li = ( PPPY 0,1)
1
= dy dw +
U-PA(y W.
6 dy dw.
6
is a summation of two WVC problems with @ = (-3 w A (y E R) and DT (0
3) A (y E R), whose joint integral surface is shown in Figure la
These WVC problems do not admit closed-form solutions since they involve truncated Gaussian distributions. Moreover, Figure Ta shows that computing BMA, even in such a low-dimensional parameter space, requires integration over non-convex and multi-modal functions.
2.2 General Reduction of BMA to WVC
Let model fw be a ReLU neural net. Denote the set of inputs to its ReLU activations by R = {r:) where each r, is linear combination of weights. For a given input æ, the parameter space is partitioned by whether each ReLU activation outputs zero or not. This gives the WVC reduction
ply x) =
Σ (E R
p(y x, w) p(w! D) dw,
where B is binary vector. The region B is defined as A AR e where arithmetic constraint & is r, 2 if B, = and r; < 0 otherwise. The expected prediction Ep(y))) [y] is analogous but includes an additional factor and variable of integration y in each WVC problem.
This general reduction, however, is undesirable since amounts a brute-force enumeration that implics a complexity exponential in the number of ReLU activations. Moreover, not all WVC
3

---

problems resulting from this reduction are amenable to existing solvers. We will therefore appeal to framework called weighted model integration (WMI) that allows for compact representation of these WVC problems, and a characterization of their tractability for WMI solvers (Kolb et al. 2019). This inspires us to approximate BMA by first reducing it WVC problems and further closely approximating those with tractable WMI problems.
3 Approximating BMA by WMI
WMI modeling and inference framework that supports integration in the presence of logical and arithmetic constraints (Belle et al. 2015a Various WMI solvers have been proposed in recent years (Kolb al.) 2019). ranging from general-purposs ones to others that assume some problem structures to gain scalability. However, even with the reduction from BMA to WVC from the previous section, WMI solvers are not directly applicable. Existing solvers have two main limitations: (i) feasible regions need to be defined by Boolean combinations of linear arithmetic constraints, and (ii) weight functions need to be polynomials. In this section, we show that these issues can be bypassed using motivating example of how to form a close approximation to BMA using WMI.
In WMI, the feasible region is defined by satisfiability modulo theories (SMT) constraints (Barret et al. 2010): an SMT formula is a (typically quantifier-free) expression containing both propositional and theory literals connected with logical connectives; the theory literals are often restricted to linear real arithmetic, where literals are of the form (c" c"X= b) with variable X and constants c" and b.
Example 4. The ReLU model (x) of Example 2 can be encoded as an SMT formula (see box). The curly bracket denotes logical conjunction, the
{ W > 0 Z W. W x 0 Z
symbol = a logical implication, variable W is the
APOLU-
weight, and variable Z denotes the model output.
The encoding of ReLU neural networks into SMT formulas is explored in existing work to enable verification of the behavior of neural networks and provide formal guarantees (Katz et al. 2017] Huang et al. 2017 Siiaraman et al. 2020p. We propose to use this encoding define the feasible region of WMI problems. Let x F A denote the satisfaction of an SMT formula A by an assignment x, and [ t= A] be its corresponding indicator function. We formally introduce WMI next.
Definition S. (WMI) Let X be a set of continuous random variables. A weighted model integration problem is a pair M (A, 4),, where A is an SMT formula over X and 4 a set of per-literal weights defined as & = (&e hrec. where L is set of SMT literals and each de is a function defined over variables in literal The task of weighted model integration is to compute
WMI(A, 4)
II FEC
(e(x)) dx.
That is, the task is to integrate over the weighted assignments of X that satisfy the SMT formula A An approximation to the B.M of Example can be achieved with WMI using the following four steps:
Step 1. Encoding model fuo (1r). This has been shown as the SMT formula AROLU in Example 4
Step 2. Encoding posterior distribution p(w (D). The uniform distribution p(w D) 6 with
w E [-3,3] can be encoded as WMI problem pair (A pos: T pos) as follows:
4 pos {o((W)- with é= true
A pos 33 W <3
Step 3. Approximate encoding of predictive distribution ply w,,.), Recall that p(y w,,) x)
Pyry; fwr (x), 1) is Gaussian, which cannot be handled by existing WMI solvers. To approximate it with polynomial densities, we simply use a triangular distribution encoded as a WMI problem pair:
Y < Z + a Y 2 Z
6r, (Y, Z)
1-Y+2 with & = Y 2 Z
A pred
pred =
Φi (Y, = 1 with 6 = <
In the literature, WMI is defined on mixed problems continuous variables, ignore the discrete ones in the definition for succinctness.
domains. Since we only work with WMI

---

In this encoding, a is a constant that defines the shape of the tri- angular distribution. It is obtained by minimizing the L2 distance between standard normal distribution and the symmetric triangular distribution. We visualize this approximation in the right figure.
0.3
0.1
Oa on
Tranguter
Step 4. Approximating BMA by calling WMI solvers. With the
above encodings, the predictive posterior pyy x)) (Equation [ can be computed using two calls to WMI solver. For example, the uncertainty of prediction
for input I = 1 is
ply 1 = 1) WMI(A A (Y 1), Φ) /MI(A, Φ) 0.164 /
where A = RELL A A poe A Apred and Φ = 4 pos U L pred. Similarly, the expected prediction Ep(ul==1) [y] (Equation D
can be computed using two calls a WMI solver.
Ep(ul==1)M)
WMI(A, Φ*) / WMI(A, Φ) 0.752 /
where t- + U (dr(Y) =Y with é = true). The above formulations also work for unnormalized distributions since the WMI in the denominator serves to compute the partition function.
We visualize the integral surface of the resulting approximate BMA problem in Figure [b] It is very close to the integral surface the original BMA problem Figure Ia However, it can be exactly integrated using existing WMI solvers while the original one does not admit such solutions. Next, we show how this process can be generalized to a scalable and accurate approximation BMA.
CIBER: Collapsed Inference for Bayesian Deep Learning via WMI
Given BNN with large number of weights, naively approximating it by WMI problems can lead to computational issues, since involves doing integration over polytopes in arbitrarily high dimensions and this is known to be #P-hard (Valiant [979; De Loera et al. 2012: Zeng et al. (220cc.. Further, weights involved with non-ReLU activation might not be amenable to the WMI encoding. To tackle these issues, we propose to collapsed samples combine the strengths from two worlds: the scalability and flexibility from sampling and the accuracy from WMI solvers.
Definition 6. (Collapsed BMA) Let (Ws, W:) be a partition of parameters W. collapsed sample is a tuple (w,, 9). where w, is an assignment to the sampled parameters W, and is representation of the conditional posterior p( WW. e W,, D) over the collapsed parameter set W.. Given collapsed samples S, collapsed BMA estimates the predictive posterior and expected prediction as
[/ ply x. w) g(w,) dwa and
p(y x)
Σ (w.s))eS
[
E Ep(pla) [M]
Σ
pup w) g(wx) dw, dy
(W.SS)ES
The size of the collapsed set W. determines the trade-off between scalability and accuracy. The more parameters in the collapsed set, the more accurate the approximation BMA is. The fewer parameters in W., the more efficient the computations of the integrals are since the integration is performed in lower-dimensional space. Later in our experiments, we choose subset of weights at the last or second-to-last hidden layer of the neural networks to be the collapsed set. This choice is known to be effective in capturing uncertainty as shown in Kristiadi et al. (2020); [nook et al. (2015).
To develop an algorithm to compute collapsed BMA, we are faced with two main design choice questions: (Q1) how to sample w, from the posterior? (Q2) what should be the representation of the conditional posterior such that the integrals in Equation @ can be computed exactly? Next, we provide our answers these questions that together give our proposed algorithm CIBER.
4.1 Approximation to Posteriors
For (Q1), we follow Maddox et al. (2019) and sample from the stochastic gradient descent (SGD) trajectory after convergence and use the information contained in SGD trajectories to efficiently approximate the posterior distribution over the parameters of the neural network, leveraging the
interpretation of SGD approximate Bayesian inference (Mandt et al. 2017; Chen et al. 2020).
5

---

Given a set of parameter samples W from the SGD trajectory, the sample set is defined as W,, {w, w W]. For each assignment wx, an approximation (W.) to the conditional posterior p(We ws D) is necessary since the posteriors induced by SGD trajectories are implicit. Next, we discuss the choice of approximation to the conditional posterior that is amenable to WMI.
4.2 Encoding into WMI Problems
As shown in Section if BNN can be encoded as a WMI problem, the posterior predictive distribution and the expected prediction, which involve marginalization over the parameter space, can be computed exactly using WMI solvers. This inspires us to use the WMI framework as the closed-form representation for the conditional posteriors of parameters. The main challenge is how to approximate the integrand in Equation 2 using an SMT formula and polynomial weight function in order to obtain a WMI problem amenable to existing solvers.
For the conditional posterior approximation q( (We),
choose it to be uniform distribution that can be encoded into WMI problem as Mpos = (Apos Q poss with the SMT formula being true), where 4 and
pos NiEe (4: S W; u;) and weights being & pos (W.) = = 1 U; are domain lower and upper bounds for the uniform distribution respectively. While seemingly over-simplistic, this choice of approximation
the conditional posterior is sufficient robustly deliver surprisingly strong empirical performance as shown in Section The intuition is that uniform distributions are better than few samples. We further illustrate this point by comparing the predictive distributions of CIBER and HMC in a few-sample setting. Figure shows that even with the same 10 samples drawn from the posterior distribution, since CIBER further approximates the samples with uniform distribution, it yields a predictive distribution closer to the ground truth than HMC, indicating that using uniform distribution instead of a few samples forms a better approximation.
For the choice of predictive distribution p(y | Øt w), we propose to use piecewise polynomial densities. Common predictive distributions can be approximated by polynomials up to arbitrary precision in theory by the Stone-Weierstrass theorem (De Branges) [959). For regression, the de facto choice Gaussian and we propose to use triangular distribution as the approximation, i.c.. + - (x)()),, with domain /// fw (x)) S r, and r := a (x) where the constant a parameterizes the triangular distribution as described in Section 3 Here, (xx))
p(y x. w)
is the variance estimate, which can be function of input a depending on whether the BNN is homoscedastic or heteroscedastic. Then p(y x. w) can be encoded into WMI as:
{
Y (x) Y far (x)
de, (Y, W.) (1, W.) fu(( with Eg (z) Y)
(m)( with l (Y> (
predd=
s pred
Similar piecewise polynomial approximations are adopted for classification tasks when the predictive distributions are induced by softmax functions. Those details are presented in the Appendix.
4.3 Exact Integration in Collapsed BMA
By encoding the collapsed BMA into WMI problems, we are ready to answer (Q2), i.c., how to perform exact computation of the integrals shown Equation [
Proposition 7. Let the SMT formula A = ARELU A Apos A Apred, and the set of weights Tb =
pos U pred as defined in Section Let the of weights क- = )b( Y with { == true).
The integrals in collapsed BMA (Equation[) can be computed by WMI solvers as
I plu x, w)) q(w.) dw,- WMI(A A (Y =)), 4) WMI D), and f y
yyrl plu w,w)) g(w.) dw. dy- WMI(A, Φ*) //MIAA,
With both questions (Q1) and (Q2) answered, we summarize our proposed algorithm CIBER in Algorithm the Appendix. To quantitatively analyze how close the approximation delivered by CIBER is to the ground-true truth BMA, consider the following experiments with closed-form BMA.

---

6.04
Joly -00
0.03
log KL divergence
0.02
0.01
-13
-50
-9.01
-3.02
-5.
60
80
100
120 140
number of samples
Figure Posterior predictive distributions in Bayesian linear regression. The yy-xxss shows the absolute difference between an estimated predictive distribution ply x) and the ground-truth predictive distribution (3 x). Shaded regions are the 95% confidence interval. Best viewed in color.
Figure 4: KL divergence in Bayesian linear regression. The xr-aii shows the number of samples the MC method uses for estimations, ranging from 50 to 150. The blue curve shows the MC method, and the green dashed curve shows CIBER using 50 samples.
Regression. We consider Bayesian linear regression setting where exact sampling from the posterior distribution is available. Both the likelihood and the weight posterior are Gaussian such that the ground-truth posterior predictive distribution Gaussian as well. With samples drawn from the weight posterior, CIBER approximates the samples with uniform distribution as posterior p( p(w] (D) and further approximates the likelihood with a triangular distribution such that the integral p(y| x D) p(ylæ, w w)p(w(]))))))))) ( dw can be computed exactly by WMI.
We first evaluate the posterior predictive distribution estimated by CIBER and Monte Carlo (MC) method, using the same five samples drawn from the weight posterior. Results averaged over 10 trials are shown in Figure where the estimations by CIBER are much closer to the ground truth posterior predictive distribution than those by the MC method. Further, the averaged KL divergence between the ground truth and CIBER is 0.069 while the one for MC estimations is 0.130, again indicating that CIBER yields a better BMA approximation in the few-sample setting.
We further explore the question of how many samples the MC method needs match the performance of CIBER. The performances of both approaches are evaluated using KL divergence between the ground-truth posterior distribution and the estimated one, averaged over 10 trials. The result is shown in Figure where the dashed green line shows the performance of CIBER with 50 samples and the blue curve shows the performance of MC with an increasing number of samples. As expected, the MC method yields lower KL divergence as the number of samples increases; however, it takes more than 100 samples to match CIBER, indicating its low sample efficiency and that developing efficient and effective inference algorithms such as CIBER for estimating BMA is meaningful question.
For analyzing classification performance, Kristiadi et al. (2022) propose compute the integral = ((.))) df. with o being the sigmoid function and f. f(x": w) that amounts to the posterior predictive distribution. We consider a simple case with f(; w) such that the ground-tren -tuth integral can be obtained. With a randomly chosen æ, the ground-truth integral
Classification.
(riisiaa
x
0.823. The integral estimated by CIBER Ic 0.826 while the MC estimate is IMC = 0.732. That is, CIBER gives an estimate with much lower error than the MC estimation error, indicating that CIBER is able to deliver high-quality approximations classification tasks.
5 Related Work
Bayesian Deep Learning. Bayesian inference over deep neural networks (MacKay) 1992
proposed to fix the issue that deep learning models give poor uncertainty estimations and suffer from overconfidence (Nguyen et al., all, 2015; Hein et al,, 2019: Meronen et al. [023 202T}. Some methods samples from SGD trajectories to approximate the implicit true posteriors similar us: Izmailov ct al. (020)) (SI) proposes to perform Bayesian inference in a subspace of the parameter space spanned by few vectors derived from principal component analysis (PCA+E variational inference (PCA+VI(SD): SWAG (Maddox et al. 2019) proposes to approximate the full parameter space using approximate Gaussian posterior whose mean and covariance are from partial SGD trajectory with a modified learning rate scheduler.
Some other approaches using approximate posteriors include MC Dropout (MCD) Gaa & Ghahra mani] 2015 |015][206] (016) which one of the Bayesian dropout methods and recently, one of its modifications called Variational Structured Dropout (VSD) (Nguyen et al. 202I) using variational inference is pro-

---

Table 1: Average test log likelihood for the small UCI regression task.
BOSTON
CONCRETE
YACHT
NAVAL
ENERGY
CIBER (SECOND)
-2.471 0.140
-2.975 1 0.102
-0.678 = 0.301
7.276 = 0.532
-0.716 0.211
CIBER (LAST)
-2.471 + 0.140
-2,959 + 0.109
-0.687 = 0.301
7.482 + 0.188
-0.716 + 0.211
SWAG
-2.761 0.132
-3.013 + 0.086
-0.404 0.418
6.708 0.105
-1.679
1.488
PCA+ESS (SI)
1719910.133
-3,007 1 0.086
-0.225 1 0.400
6.541 1: 0.095
-1.563 + 1.243
PCA+VI
1711610.133.
-2.994 1 0.095
-0.396 0.419
6.708 0.10
-1.715 1.588
SGD
-2.752
-3.178 + 0.198
-0.418 = 0.426
6.567 = 0.155
-1.736 = 1.613
DVI
-2.410 0.020
-3.060 + 0.010
-0.470 + 0.030
6.290 + 0.040
-1.010 0.060
DGP
-2.330 + 0.060
-3.130 + 0.030
-1.390 + 0.140
3.600 + 0.330
-1.320 + 0.030
VI
-2.430 + 0.030
-3.040 + 0.020
-1.680 + 0.040
5.870 + 0.290
-2.380 0.020
MCD
-2.400 +: 0.040
-2.970 1 0.020
-1.380 = 0.010
4.760 0.010
-1.220 0.010
VSD
-2.350 1 0.050
-2.970 1 0.020
-.144 + 0.020
4.830 + 0.010
-1.060 = 0.010
posed. Other state-of-the wwe-of---h-rrrrr art approximate BNN inference methods including deterministic variational inference (DVT) (Wu al., 2019), deep Gaussian processes (DGP) (Bui et al. 2016) with Gaussian process layers and variational inference (VI) (Kingma & Welling 2013). Closely related to DGP is the deep kernel process (Aitchison et al. 2021 that writes DGPs as deep Wishart processes.
WMI Solvers. WMI generalizes weighted model counting (WMC) Sang et al. (2005), a state-of- the inference approach in many discrete probabilistic models, from discrete mixed discrete- continuous domains Belle et al. (2013a(b). Recent research on WMI includes its tractability (Zeng
2020c 2021 Abboud al. (0200) and the advancements in WMI solver development. Existing exact WMI solvers for arbitrarily structured problems include DPLL-based search with numeri- cal Belle al. 2013a); Morettin al. (2017 2019) symbolic integration de Salvo Braz al.)
al.
(2066) and compilation- based algorithms Kolb et al. (2018); Zuidberg Dos Martires et al. (2019); Derkinderen et al. (2020) that use extended algebraic decision diagrams (XADDs) (Sanner al.
2012) as a compilation target which a powerful tool for inference on mixed domains (Sanner & Abbasnejad 2012 Zamani al. 2012). Some exact WMI solvers aiming to improve efficiency for certain class of models are proposed such as SMI (Zeng & Van den Broeck 2019) and MP-WMI (Zeng ct al. (2000a) which are greatly scalable for WMI problems that satisfy certain structural constraints. Approximate solvers are also proposed including sampling-bast
ones (Zuidberg Dos Martires et al., 2020 and relaxation-based ones (Zeng et al., [020cc|). Recent WMI efforts converge in the pywmi library (Kolb et al. (019). The SMT formulas considered in this work can be seen as distributional constraints on continuous domains. There is also plenty of work in couro-symbolic Al exploring the integration of discrete constraints into neural networks models including the architectures (Ahmed et al. 2022b 2023) and the loss (Xu et al. 2018
Ahmed al. 2022c a).
6 Experiments
We conduct experimental evaluations of our proposed approach CIBER on regression and classi- fication benchmarks and compare its performance uncertainty estimation well as prediction accuracy with a wide range of baseline methods. More experimental details are presented in the Appendix.
6.1 Regression on Small and Large UCI Datasets
We experiment 5 small UCI datasets: boston, concrete, yacht, naval and energy. We follow the setup of Izmailov et al. (2020) and use a fully connected network with a single hidden layer and 50 units with ReLU activations. We further experiment 6 large UCI datasets: elevators, keggdirected, keggundirected, pol, protein and skillcraft. We use feedforward network with five hidden layers of sizes [1000, 1000, 500, 50. 2] and ReLU activations all datasets except skillerafi. For skillcraft, smaller architecture is adopted with four hidden layers of size [1000, 500, 50, 2). All models have two outputs for the prediction and the heteroscedastic variance respectively.
'Code and experiments available https: //ggtttttt..././///////e httpgghtt....//..--aaeaa..e

---

Table 2: Average test log likelihood for the large UCI regression task.
ELEVATORS
KEGGD
KEOGU
PROTEIN
SKIFLCRAFT
Pol.
CIBER (SECOND)
-0.378 1 0.026
1.245 0.090
1,125 1 0.269
-0.720 1 0.036
-1,003
0.035
2.555
0.115
CIBER (LAST)
-0.371 + 0.023
1.178 0.088
0.964 + 0.231
-0.720 + 0.036
.1,001
0.032
2.506
0.150
SWAG
-0,374 0.021
1080 0.035
0.749 = 0.029
-0.700 = 0.051
-1.180
0.033
1.533 1.084
PCA+ESS (SI)
-0,351 0.030
1.074
0.752 = 0.025
-0.734 = 0.063
-1.181
0.033
-0.185 2.779
(SD+
-0.325 0.019
1.085 0.031
0.757 0.028
-0.712 = 0.057
-1.179 0.033
1.764 0.271
SGD
-0.538 0.108
1.012 0.154
0.602 + 0.224
-0.854 = 0.085
-1.162 = 0.032
1.073 + 0.858
ORTUVGP
-0.448
1.022
0.701
-0.914
0.159
NL
-0.698 = 0.039
0.935 0.265 0.670 = 0.038
-0.884 = 0.025
-1.002 = 0.050
-2.840 0.226
Table 3: Average test performance for image classification tasks on CIFAR-10 and CIFAR-100.
METRIC
NLL
ACC
ECE
DATASET
CIFAR-10
CIFAR-100 100
CIFAR-10
CIFAR-100
CIFAR-10
CIFAR-100
CIBER
0.1927
0.0029
0.9193 + 0.0027
93.64
0.09
74.71 + 0.18
0.0130 + 0.0011
0.0168 + 0.0025
SWAG
0.2503 = 0.0081
1.2785 = 0.0031
93.59 0.14
73.85
0.25
0.0391 + 0.0020
0.1535 1 0.0015
SGD
0,3285 + 0.0139
1.7308 + 0.0137
93.17 0.14
73.15
0 0.11
0.0483 + 0.0022
0,1870 + 0.0014
SWA
0,2621 0.0104
1.2780 + 0.0051
93.61 0.11
74.30
0.22
0.0408 + 0.0019
0.1514 + 0.0032
SGLD
0.2001
0.0059
0.9699 0.0057
93.55
74.02
0.30
0.0082 + 0.0012
0.0424 1 0.0029
KFAC
0.2252 0.0032
1.1915 + 0.0199
92.65 0.20
72.38 +: 0.23
0.0094 0.0005
0.0778 + 0.0054
We run CIBER with two different ways of choosing the collapsed parameter set: CIBER (last) chooses all the weights at the last layer to be the collapsed set; CIBER (second) chooses three out of all the weights at the second-to-last layer to be the collapsed set. The heuristic we use for choosing the weights is to look into the sampled weights from SGD trajectories to see which ones have the greatest variance. The intuition that a greater variance indicates that the weight prone to have greater uncertainty and thus one might want to perform more accurate inference over it.
Baselines. We compare CIBER to the sf the-art approximate BNN inference methods. We separate these methods into two categories: those sampling from SGD trajectories as approximate posteriors, which includes SWAG (Maddox al. 2019), PCA+ESS (SI) and PCA+VI (SI) (Izmailov eet al. 2020), vs. those who do not, which includes the SGD baseline, deterministic variational inference (DVT) (Wu et al. 2019), Deep Gaussian Processes (DGP) (Bui et al. (016), variational inference (VI) (Kingma & Welling 2013). MC Dropout (MCD) (Gal & Ghahramani] 2015 2016),
and variational structured dropout (VSD) (Nguyen al. 2021). These methods achieved state- of-the-art performance on the small UCI datasets. We also compare to baselines Bayesian final layers (NL) (Riquelme et al. 2018), deep kernel learning (DKL) (Wilson et al. 2016), orthogonally decoupled variational GPs (OrthVGP) (Salimbeni et al. 2018) and Fastfood approximate kernels (FF) (Yang et al., 2015). which have achieved state-of-the-as art performance on the large UCI datasets.
Results. We present the test log likelihoods for small UCI datasets in Table and those for large UCI datasets in Table In both tables, the first block summarizes SGI--rajectory sampling-bar approaches and the second summarizes the rest. Underlined results are the best among all and bold results are the best among SGD-trajectory sampling-bar has substantially better performance than all others on three out of the five small UCI datasets four out of six large UCI datasets, with comparable performance on the rest, demonstrating that CIBER provides accurate uncertainty estimation. We also present the test seeodddman--quuuree error results in Appendix, where CIBER outperforms all other trajetroyy sampling based baselines on four out of five small UCI datasets and four out of six large UCI datasets; it outperforms all baselines on two small UCI datasets and one large UCI datasets and has comparable performance on the rest. This further illustrates that exact marginalization over conditional approximate posteriors enabled by WMI solvers achieves accurate estimation of the true BMA and boosts predictive performance.
approaches. From the results, our CIBER
6.2 Image Classification
CIFAR datasets. We experiment with two image datasets: CIFAR-10 and CIFAR-100 (Krizhevsky al. 2009) and evaluate the test performance using three metrics: 1) negative log likelihood (NLL)
9

---

Table 4: Average test performance for image transfer learning tasks.
METRIC
NLL
ACC
ECE
MODEL
VGG-16 16
PRERESNET-161
VGG- 16
PRERESNET- 164
VGG-16
PRERESNET- 164
CIBER
0,9869
0.0102
0.9684 0.0075
72.56
0.23
75.70 + 0.17
0.0925
0.0028
0.0704
0.0031
SWAG
1.3425 = 0.0015
1.3842 = 0.0122
0.11
76.30 = 0.06
0.1988 + 0.0028
0.1668 = 0.0006
SGD
1.6528 = 0.0390
1.4790 + 0.0000
0.07
75.56 + 0.00
0.2149 0.0027
0.1758 + 0.0000
SWA
1.3993 = 0.0502
1.3552 0.0000
71.92 0.01
76.02 = 0.00
0.2082 0.0056
0.1739 = 0.0000
that reflects the quality of both uncertainty estimation and prediction accuracy, 2) classification accuracy (ACC), and 3) expected calibration errors (ECE) (Nacini et al. 2015) that show the difference between predictive confidence and accuracy and should be close to zero for a well-calibrated approach.
We run CIBER by choosing the collapsed parameter set to be 10 weights and 100 weights at the last layer of the neural network models for CIFAR-10 and CIFAR-100 respectively. The weights are chosen using the same heuristic as the one for regression tasks, i.e., to choose the weights whose samples from the SGD trajectories have large variances. We compare CIBER with strong baselines including SWAG (Maddox et al. 2019) reproduced by their open-source implementation, standard SGD, SWA (Izmailov et al. 2018 8 SGLD (Welling & Teh, 201 and KFAC (Ritter et al. 2018).
Transfer from CIFAR-10 10 to STL-10. We further consider a transfer learning task using the model trained on CIFAR-10 10 be evaluated on dataset STL-10 (Coates al., 201 STL-10 shares nine out of ten classes with the CIFAR- 10 dataset but has a different image distribution. It is common benchmark in transfer learning to adapt models trained on CIFAR-IÓ STL-10.
Results. We present the test classification performance on dataset CIFAR-10 and CIFAR- 100 in Table and that of transfer learning in Table The neural network models used in the classification task are VGG-16 networks and the models used in the transfer learning task are VGG-16 and PreResNet-164. More results using different network architectures are presented in the Appendix. With the same number of samples as SWAG, CIBER outperforms SWAG and other baselines in most evaluations and delivers comparable performance otherwise, demonstrating the effectiveness of using collapsed samples in improving uncertainty estimation as well as classification performance.
7 Conclusions And Future Work
We reveal the connection between BMA, a way to perform Bayesian deep learning and WVC, which inspires us to approximate BMA using the framework of WMI. To further make this approximation scalable and flexible, we combine it with collapsed samples which gives our algorithm CIBER. CIBER compares favorably to Bayesian deep learning baselines on regression and classification tasks. future direction would be to explore what other layers can be expressed as SMT formulas and thus amenable to SMT encoding. Also, the current WMI solvers are limited to polynomial weights, and thus the reduction to WMI problems is applicable to piecewise polynomial weights. This limitation might be alleviated in the future by the development of new WMI solvers that allow various weight function families.
Acknowledgments
We would like to thank Yuhang Fan and Kareem Ahmed for helpful discussions. We would also like to thank Wesley Maddox for answering queries on the implementation of baseline SWAG. This work was funded in part by the DARPA PTG Program under award HR001 12220005, the DARPA ANSR program under award 2-0000-23-200004 NSF grants #11S-1943641, 1155-1956441, #CCF-18371 19, and gift from RelationalAI. GVdB discloses a financial interest in RelationalAI. 7Z. is supported by an Amazon Doctoral Student Fellowship.
References
Abboud, R., Ceylan, I. I., and Dimitrov, R. On the approximability of weighted model integration on DNF structures. arXiv preprint arXiv:2002.06726, 2020.
10

---

Ahmed, K., Li, T., Ton. T., Guo, Q. Chang, K.-W., Kordjamshidi, P., Srikumar, V., Van den Broeck, G. and Singh, S. Pylon: A pytorch framework for learning with constraints. Proceedings of the 36th AAAI Conference on Artificial Intelligence (Demo Track), feb 2022a.
Ahmed, K., Teso, S., Chang, K.-W., Van den Broeck, G., and Vergari, A. Semantic probabilistic layers for neuro-symbolic learning. In Advances in Neural Information Processing Systems 35 (NeurIPS), dec 2022b.
Ahmed, K., Wang, E., Chang, K.-W., and den Broeck, G. V. Neuro-symbolic entropy regularization. In The 38th Conference Uncertainty in Artificial Intelligence, 2022c.
Ahmed, K., Zeng, Z., Niepert, M., and Van den Broeck, G. Simple: A gradient estimator for k-subset sampling. In Proceedings of the International Conference on Learning Representations (ICLR), may 2023.
Aitchison, L., Yang, A., and Ober, S. W. Deep kernel processes. In International Conference on Machine Learning, pp. 140. PMLR, 2021.
Baldoni, V., Berline, N., De Loera, J. A., Dutra, B., Köppe, M., Moreinis, S., Pinto, G., Vergne, M., and Wu, J. A user's guide for latte integrale v1. 7.2. Optimization, 22(2), 2014.
Barrett, C., de Moura, L., Ranise, S., Stump, A., and Tinelli, C. The SMT-LIB initiative and the rise of SMT. In Proceedings of the 6th international conference on Hardware and software: verification and testing, pp. 3-3. Springer- Verlag, 2010.
Belle, V., Passerini, A., and Van den Broeck, G. Probabilistic inference in hybrid domains by weighted model integration. In Proceedings of IJCAI, pp. 2770-277777 2776, 2015a.
Belle, V., Van den Broeck, G., and Passerini, A. Hashing-based approximate probabilistic inference in hybrid domains. In Proceedings of the 31st Conference on Uncertainty in Artificial Intelligence (UAT), 2015b.
Bui, T., Hernández-Lobato D., Hermandez-Lobato, J., Li, Y., and Turner, R. Deep gaussian processes for regression using approximate expectation propagation. In International conference on machine learning, pp. 1472-144 1481. PMLR, 2016.
Chen, X., Lee, D., Tong, X. T., and Zhang, Y. Statistical inference for model parameters in stochastic gradient descent. The Annals of Statistics, 48(1):251-273, 2020.
Coates, Ng, A., and Lee, H. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215 223. JMLR Workshop and Conference Proceedings, 2011.
De Branges, L. The stone- weierstrass theorem. Proceedings of the American Mathematical Society, 10(5):822 824, 1959.
De Loera, J. A., Dutra, B., Koeppe, M., Morcinis, S., Pinto, G., and Wu, J. Software for exact integration of polynomials over polyhedra. ACM Communications in Computer Algebra, 45(3/4); 169-172, 2012.
de Salvo Braz, R., O'Reilly, C., Gogate, V., and Dechter, R. Probabilistic inference modulo theories. In Proceedings of the Twenty- International Joint Conference on Artificial Intelligence, pp- 3591-3599. AAAI Press, 2016.
Derkinderen, V., Heylen, E., Dos Martires, Z., Kolb, S., and Raedt, L. Ordering variables for weighted model integration. In Conference on Uncertainty in Artificial Intelligence, pp. 879-888. PMLR, 2020.
Fragoso, T. M., Bertoli, W., and Louzada, F. Bayesian model averaging: A systematic review and conceptual classification. International Statistical Review, 86(1):1-28, 2018.
Gal, and Ghahramani, Z.. Bayesian convolutional neural networks with bernoulli approximate variational inference. arXiv preprint 02158, 2015.

---

Gal, Y. and Ghahramani, Z. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Hein, M., Andriushchenko, M., and Bitterwolf, J. Why relu networks yield high-confidence pre- dictions far away from the training data and how to mitigate the problem. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 41-50, 2019.
Huang, X., Kwiatkowska, M., Wang, S., and Wu, M. Safery verification of deep neural networks, In International conference on computer aided verification, pp. 3-29, Springer, 2017.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint 1803.
2018.
Izmailov, P., Maddox, W. J., Kirichenko, P., Garipov, T,, Vetrov, D., and Wilson, A. G. Subspace inference for bayesian deep learning. In Uncertainty in Artificial Intelligence, pp. 1169-1179.
PMLR, 2020.
Izmailov, P., Vikram. S., Hoffman, M. D., and Wilson, A. G. G. What are bayesian neural network posteriors really like? In International Conference on Machine Learning, pp. (629-46640 PMLR, 2021.
Jospin, L. V., Laga, H., Boussaid, F., Buntine, W., and Bennamoun, M. Hands-on bayesian neural networks- tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2): 29-48, 2022.
Katz, G., Barrett, C.. Dill, D. L., Julian, K., and Kochenderfer, M. J. Reluplex: An efficient smt solver for verifying deep neural networks. International conference on computer aided verification, pp. 97-117. Springer, 2017.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. arXiv preprini arXiv:1312.6111 2013.
Kolb, S., Mladenov, M., Sanner, S., Belle, V., and Kersting, K. Efficient symbolic integration for probabilistic inference. In IJCAI, pp. 5031-5037, 2018.
Kolb, S., Morettin, P., Zuidberg Dos Martires, P., Sommavilla, F., Passerini, A., Sebastiani, R., and De Raedt, L. The pywmi framework and toolbox for probabilistic inference using weighted model integration. In Proceedings of the Eventy-Eightte International Joint Conference on Artificial Intelligence, IJCAI, pp. 6530-6532, 2019. doi: 10.24963/icai.20199999
Koller, D. and Friedman, N. Probabilistic graphical models. 2009.
Kristiadi, A., Hein, M., and Hennig. P. Being bayesian, even a bit. fixes overconfidence in relu networks. In International conference on machine learning, pp. 5436-5446. PMLR, 2020.
Kristiadi, A., Eschenhagen, R., and Hennig, P. Posterior refinement improves sample efficiency in bayesian neural networks. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. ( Advances in Neural Information Processing Systems, volume 35, pp. 30333-30346. Curran Associates, Inc., 2022.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
MacKay, D. J. The evidence framework applied to classification networks. Neural computation, 4(5): 720-736, 1992.
Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32: 13153-13164, 2019.
Mandt, S., Hoffmann M. D., and Blei, D. M. Stochastic gradient descent as approximate bayesian inference. arXiv preprint urXiv:1704 04289, 2017.
Meronen, I.,, Trapp, M., and Solin, Periodic activation functions induce stationarity. Advances in Neural Information Processing Systems, 34:1673-1666 2021.
12

---

Meronen, L.,, Trapp, M., Pilzer, A., Yang, L., and Solin, A. Fixing overconfidence in dynamic neural networks. arXiv preprint arXiv:2302.06:39, 2023.
Morettin, P., Passerini, A., and Sebastiani, R. Efficient weighted model integration via SMT- predicate abstraction. In Proceedings of IJCAI, pp. 720-728, 2017.
Morettin, P., Passerini, A., and Sebastiani, R. Advanced smt techniques for weighted model integra- tion. Artificial Intelligence, 275:1-27, 2019.
Nacini, M. P., Cooper, G., and Hauskrecht, M. Obtaining well calibrated probabilities using bayesian binning, In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015,
Nguyen, A., Yosinski, J., and Clune, Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Nguyen, S., Nguyen, D., Nguyen, K., Than, K., Bui, H., and Ho, N. Structured dropout variational inference for bayesian neural networks, Advances in Neural Information Processing Systems, 34, 2021.
Riquelme, C., Tucker, G., and Snoek, Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling, arXiv preprint arXiv:/802.090 2018.
Ritter, H., Botev, A., and Barber, D. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Coss D01-Cooferrecce Track Proceedings, volume 6. International Conference on Representation Learning, 2018.
Salimbeni, H., Cheng, C.-A., Boots, B., and Deisenroth, M. Orthogonally decoupled variational gaussian processes, Advances in neural information processing systems, 31, 2018.
Sang. T., Beame, P., and Kautz, H. Performing bayesian inference by weighted model counting. In AAAI, volume 5, pp- 475-481, 2005.
Sanner, S. and Abbasnejad, E. Symbolic variable elimination for discrete and continuous graphical models. In Twenty-Sixth AAAI Conference on Artificial Intelligence, 2012.
Sanner, S., Delgado, K. V., and De Barros, L. N. Symbolic dynamic programming for discrete and continuous state mdps. arXiv preprint arXiv: 2012.
Sivaraman, A., Farnadi, G., Millstein, T., and Van den Broeck, G. Counterexample-guiee learning of monotonic neural networks. In Advances in Neural Information Processing Systems 33 (NeurIPS), dec 2020.
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., and Adams, R. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pp. 2171-2180. PMLR, 2015.
Valiant, I.. G. The complexity of enumeration and reliability problems. SIAM Journal on Computing. 833):410-421, 1979.
Wasserman, L. Bayesian model selection and model averaging. Journal of mathematical psychology, 44(1):92-107, 2000.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-11), pp. 681-688, 2011.
Wilson, A. G., Hu, Z., Salakhutdinov, R., and Xing, E. P. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Wu, A., Nowozin, S., Meeds, E., Turner, R. E., Hernandez-LLoato, J. M., and Gaunt, A. L. Determin- istic variational inference for robust bayesian neural networks. In International Conference on Learning Representations, 2019.
13

---

Xu, J., Zhang. Z., Friedman. T., Liang, Y., and Van den Broeck, G. A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning, pp- 5502-5511. PMLR, 2018.
Yang. Z., Wilson, A., Smola, A., and Song, L. A la carte-learning fast kernels. In Artificial Intelligence and Statistics, pp. (098-1106. PMLR, 2015.
Zamani, Z.. Sanner, S., and Fang, C. Symbolic dynamic programming for continuous state and action mdps. In Proceedings of the AAAI Conference Artificial Intelligence, volume 26, pp. 1839-1845, 2012.
Zeng, Z. and Van den Broeck, G. Efficient search- based weighted model integration. Proceedings of UAI, 2019.
Zeng. Z., Morettin, P., Yan, F., Vergari, A., and Broeck, G. V. Scaling up hybrid probabilistic inference with logical and arithmetic constraints via message passing. In Proceedings of the International Conference of Machine Learning (ICML), 2020a.
Zeng. Z., Morettin, P., Yan, F., Vergari, A., and Van den Broeck, G. Relax, compensate and then integrate. In Proceedings the ECML-PKDD Workshop Deep Sestinoous-Discrer Machine Learning (DeCoDeML), 2020b.
Zeng, Z., Morettin, P., Yan, F., Vergari, A., and Van den Broeck, G. Probabilistic inference with algebraic constraints: Theoretical limits and practical approximations. Advances in Neural Information Processing Systems, 33: 33:11566 1564-11575. 2020c.
Zeng, Z., Morettin, P., Yan, F., Passerini, A., Van den Broeck, G., et al. Is parameter learning via weighted model integration tractable? In The 4th Workshop on Tractable Probabilistic Modeling. 2021.
Zuidberg Dos Martires, P. M., Dries, A., and De Raedt, L. Exact and approximate weighted model integration with probability density functions using knowledge compilation. Proceedings of the 30th Conference on Artificial Intelligence. AAAI Press, 2019.
Zuidberg Dos Martires, P. M. et al. Monte carlo intiedifferentiitionn for approximate weighted model integration. In Proceedings of the Ninth International Workshop on Statistical Relational AL International Workshop Statistical Relational A1, 2020.
14

---

A Proofs
Proposition Let the SMT formula A
ARLLU A pos Apred, and the set of weights &b
pos U pred as defined in Section Let the set of weights O = = =Y Y with & = true).
The integrals in collapsed BMA (Equation 2
can be computed by WMI solvers as
I ply æ. w) ((w,,) dw. = WMI A (Y = y). 2) WMI(A, Φ), and
I y ply a.w) ((w.) dw dy WMI(A, 21) WMI(A, Φ)
Proof. By construction, it holds that
ply x, w) x WMI(Apre pred A (Y = y), Fpred)
& II
dely, w. Lv.ww with (y, we) } Apred A AReLU
REC RECymed
g (we) WMI(A pos A (We = We), T iPpos)
II EE
(e(we) [w.F(1, with we F A pos
Thus, we have that the likelihood weighted by the approximate posterior would be
ply x, w) ((we)
II BE P
dely, w.) w.).v. with (y, we) } ARELU Aped A pos:
or equivalently,
mee Φely, we Iv.we
ply a, w) q(we)=
with (g. we) } A.
WMI(A, 4)
By integrating over the collapsed set W., it further holds that
/ ply x. w) q(w.) dw.
IICECpmarCCp. de(y, [..mel dw.
with (v, w.) T A
WMI(A, ()
WMI(A A (Y = y), ②)
WMI(A, 4)
which proves the first equation. Similarly, we have that
y pvu x, w) q(w.)
II
II de(w.)(w.Ft) with (v. wc) F A
By integrating over the collapsed W. and prediction y, holds that
f y ply x. w) q(w.) dw. dy
f y II dely, w. w.. #
II )e(w.)) [w. dw, dy
EE #E
with (y. w.) }
WMI(A, +)
WMI(A, Φ-)
(MI(A, b)
which finishes our proof.
15

---

Algorithm CIBER
Input: input H, sampled weights W, neural network model fw. prediction ground truth y
Ouput:
predictions and likelihoods
1: Choose a partition (Wer W: for network parameters
2: Derive approximate posterior g(we) from sampled weights {wo w E WW II cf. Section 4.2
3: Encode posterior ((w.) into WMI problem Mpos= = (4 pos & poe) // cf. Section 4.2
4: y + P 0 Initialization
5: for sample w, in {w,, | w E W do
6: Encode neural network model fReLU parameterized by (w,, W.) into an SMT formula fm
7: Encode predictive p(Y
wD, We) into WMI problem Mpred = (Apred, Q pred)
8:
SMT formula +- ARLLU A A Aos A prod
9:
Weights Φ <- ① pos U L pred
Weights $- + + U (Y)
Y with L == true]
11: Add prediction y WMI A, 4 /////MMI/AA// 4) prediction set y
I/ cf. Section 4.3
12: Add likelihood p = WMI A (Y = y²) 2) /W/MI 2) to set P
II cf. Section 4.3
13: return MEAN())), p(y" x)) = MEAN(P)
Table 5: Average test RMSE for the small UCI regression task.
BOSTON
CONCRETE
YACHT
NAVAL
ENERGY
CIBER (SECOND)
3.488 + 1.123
4.880 + 0.506
0.828 + 0.241
0.000 + 0.000
0.447 + 0.081
CIBER (LAST)
3,478 + 1.128
4.854 + 0.503
0.752 + 0.294
0.000 + 0.000
0.447 + 0.081
SWAG
3.517 + 0.981
5.233 + 0.417
0.973 + 0.375
0.001 + 0.000
1.594 + 0.273
PCA+ESS (SI)
5.194 + 0.448
0.972 + 0.375
0.001 + 0.000
1.598 + 0.274
PCA+VI (SI)
+ 0.951
5.142 + 0.418
0.973 + 0.375
0.001 + 0.000
1.587 + 0.272
SGD
3.504 0.975
5.194 0.446 0.446
0.973 + 0.374
0.001 + 0.000
1.602 + 0.275
MCD
2.830 + 0.170
4.930 + 0.040
0.720 + 0.050
0.000 1 0.000
1.080 + 0.030
VSD
2.640 + 0.170
4.720 + 0.110
0.690 + 0.060
0.000 + 0.000
0.470 + 0.010
B
Pseudo Code for CIBER
We summarize our proposed algorithm CIBER, Collapsed Inference Bayesian DEep LeaRning, for regression tasks, in Algorithm | For the classification task, the algorithm basically the same except the encoding of the predictive of the distribution. Specifically, for a given class y, the predictive distribution ply x, w) can be encoded into a WMI problem as shown below:
{
dr, (We) di, (W.) =1 with Lg = (fro > d)
with & = (f S d)
A pred (w) 2 -d L ppred
where Φe, is a cubic polynomial that approximates the sigmoid function such that the posterior predictive distribution p(u æ) can be solved by WMI solvers by p(v x) = WWMIA,, $). Further, the prediction of BMA for classification tasks is made by y" arg max,, y p(y xx).
C Additional Experiments
C.1 Toy Regression in Figure 2
We evaluate the predictive distributions obtained by our CIBER and HMC respectively, toy dataset generated by sampling 10 input uniformly distributed in the interval [-1, -0.5] and interval [0.5, 1]. For each input x, the corresponding target y computed from a cubic polynomial with Gaussian noises. We apply to these data a Bayesian neural network which is ReLU neural network with two hidden layers, where both parameter priors and likelihood are Gaussian distributions. We compare HMC and our CIBER in few-sample setting which is common in most Bayesian deep learning applications, with 10 samples from the posterior distribution.
estimation generated by HMC with sufficiently large number of samples of size 2, 000 is further presented as a ground truth.
16

---

Table 6: Average test RMSE for the large UCI regression task.
ELEVATORS
KEGGD
KEGGU
PROTEIN
SKILLCRAFT
POL
CIBER (SECOND)
0.088 + 0.002
0.142 + 0.074
0.115 + 0.007
0.438 +: 0.009
0.251 +: 0.010
2.212 0
CIBER (LAST)
0.088 + 0.002
0.142 + 0.072
0.118 + 0.012
0.438 + 0.009
0.251 +: 0.010
2.199 0.182
SWAG
0.088 = 0.001
0.129 = 0.029
0.160 + 0.043
0.415 + 0.018
0.293
3.110 = 0.070
PCA+ESS (SI)
0.089 + 0.002
0.129 + 0.028
0.160 + 0.043
0.425 + 0.017
0.293 + 0.015
3.755 + 6.107
PCA+VI (SI)
0.088 0.001
0.128 + 0.028
0.160 + 0.043
0.418 0.021
0.293 + 0.015
2.499
0.684
SGD
0.035
0.132 + 0.017
0.186
0.436 0.011
0.014
3,900
NL
0,101 = 0.002
0.134 0.036 0.036
0.120 0.003
0.447 0.012
0.011
4.380 = 0.853
DKL
0.084 + 0.020
1.100 + 0.010
0.110 + 0.000
0.460 + 0.010
0.250 + 0.000
6.617
OKTHVGP
0.095
0. 120
0.117
0.461
4,300 + 0.200
FF
0.089 + 0.002
0. 120 + 0.000
0.120 + 0.000
0.470 + 0.010
0.250 + 0.020
Table 7: Average test log likelihoods for image classification tasks on CIFAR-10 and CIFAR-100.
CIFAR-10
CIFAR-100
MODEL
VGG-16
PRERESNET- 164
WIDERESNET
VGG-16
PRERESNET-164
WIDERESNET
CIBER
0.1927 = 0.0029
0.1352 + 0.0014
0.1913 = 0 0.0029
0.9193 0.0027
0.8144
0.0065
0.7930 = 0.0065
SWAG
0,2503 = 0.0081
0,1459 = 0.0013
0.1076 = 0.0009
1.2785
0.0031
1.0703
0.4661
0.6719 = 0.0035
SGD
0,3285 + 0.0139
0.1814 # 0.0025
0.1294 : 0.0022
0.0137 1.7308
0.9465
0.0191
0.7958 0.0089
SWA
0,2621 1: 0.0104
0.1450 1 0.0042
0.1075 0.0004
1.2780
0.0051
0.7370 0.0265
0.6684 0.0034
SGLD
0.2001 1 0.0059
0.1418 1 0.0005
0.1289 0 0.0009
0.9699
0.0057
0.6981
0.0052
0.6780
0.0022
KFAC
0.2252 # 0.0032
0.1471 # 0.0012
0.1210 + 0.0020
1.1915 0.0199
0. 0,7881 + 0.0025
0.7692 0.0092
The results are shown in Figure 2 Even with the same 10 samples drawn from the posterior distribution, since CIBER further approximates the 10 samples with uniform distribution as q( ()) it yields predictive distribution p(y x) closer to the ground truth than HMC. The intuition behind is that using a uniform distribution instead of few samples forms a better approximation to the true posterior since the uniform distribution in collapsed sample represents uncountably many models.
C.2 Regression on Small and Large Datasets
Sampling from SGD Trajectories. During training, we use Gaussian log likelihood as the objective for obtaining smooth gradients and use early stopping to prevent over-fitting. At convergence, we start the sampling process by keeping running SGD and collecting the weights. At deployment time, we approximate the Gaussian predictive distribution with the triangular distributions.
Hyperparameters. The hyperparameters including learning rates and weight decay are tuned by performing grid search to maximize the Gaussian log likelihood using validation split.
Following the set-up of Izmailov al. (2020), the test performance small UCI datasets are averaged over 20 trials and the test performance of large UCI datasets are averaged over 10 trials. The test log likelihood results for small UCI datasets unnormalized while those for large UCI datasets are normalized. Besides the log likelihood results as shown in Section 6.1 dot-mean-ssqured-eroo (RMSE) results for small UCI datasets are further presented in Table 5 and RMSE results for large UCI datasets are presented in Table
Additional Results.
C,3 Image Classification
Sampling from SGD Trajectories. All the network models are trained for 300 epochs using SGD. We start the weight collection after epoch 160 with step size 5. We follow exactly the same hyperparameters as Maddox et al. (2019) including learning rates and weight decay parameters.
Additional Results. Following the set-up Maddox et al. (2019), we run experiments with VGG-16. PreResNet- 164 and WideResNet network models on both the image classification task and the transfer learning task. For the image classification task on CIFAR datasets, we present the log likelihood results in Table 7 the accuracy results in Table 8 and ECE results in Table 9 For the transfer learning task from dataset CIFAR-10 to dataset 111.-10, we present the results for model VGG-16 and PreResNet-16e 164 in Table A and the results for model WideResNet in Table
17

---

Table 8: Average test accuracy for image classification tasks on CIFAR-10 and CIFAR-100.
CIFAR-10
CIFAR- 100
MODEL
VGG-16
PRERESNET-I64
WIDERESNET
VGG-16
PRERESNET-I64
WIDERESNET
CIBER
93.64 = 0.09
95.95 = 0.06
95.63 = 0.16
74.71 = 0.18
79.23 = 0.25
81.25 = 0.35
SWAG
93.59 +: 0.14
96.09 t: 0.08
96.38 + 0.08
73.85 + 0.25
73.02 + 10.30
82.27 + 0.07
SGD
0.14
95.49 0.06
96.41 0.10
73.15 0.11
78.50 0.32
80.76 = 0.29
SWA
0.11
96.09 = 0.08
96.46 0.04
74.30=
80.19 = 0.52
82.40 = 0.16
SGLD
93.55 + 0.15
95.55 + 0.04
95.89 0.02
74.02 + 0.30
80.09 0.05
80.94 + 0.17
KFAC
92.65 + 0.20
95.49 + 0.06
96.17 0.00
72.38 + 0.23
78.51 + 0.05
80.94 + 0.41
Table 9: Average test ECE for image classification tasks CIFAR-I0 and CIFAR-100.
CIFAR-I0
CIFAR-100
VGG-16
PRERESNET-BE 164
WIDERESNET
VGG-16
PRERESNET- 164
WIDERESNET
CIBER
0.0130 = 0.0011
0.0250 + 0.0005
0.0760
0.0011
0.0168
0.0025
0.1423 0.0029
0.1650 = 0.0046
SWAG
0.0391 + 0.0020
0.0214 : 0.0005
0.0096
0.0006
0.1535
0.0015
0.1031 0.0471
0.0678 0.0006
SGD
0.0483 1: 0.0022
0.0255 1 0.0009
0.0166 0.0007
0.0014 0.1870
0.1012
0.0009
0.0479
0.0010
SWA
0.0408 1 0.0019
0.0203 1 0.0010
0.0087 1 0.0002
0.1514
0.0032
0.0700 0.0056
0.0684
0.0022
SGLD
0.0082 = 0.0012
0.0251 # 0.0012
0.0192 + 0.0007
0.0424 0.0029
0.0363 + 0.0008
0.0296
0.0008
KFAC
0.0094 + 0.0005
0.0092 = 0.0018
0.0060 = 0.0003
0.0778 0.0054
0.0158 + 0.0014
0.0379 0.0047
Table 10: Average test performance for image transfer learning tasks using Nete
METRIC
NLL
ACC
ECE
CIBER
0.8259
0.0148
75.02 : 0.31
0.0336 0.0009
SWAG
1.0142
0.0032
76.96 + 0.08
0.1303 + 0.0008
SGD
1.1308 = 0.0000
76.75 + 0.00
0.1561 1: 0.0000
SWA
1.0047
0.0000
77.50 + 0.00
0.1413 0.0000
18


Reviews:
[{"review_id": "VqO0lR00tF", "reviewer": "Reviewer_eHBs", "summary": "The authors propose to tackle the intractable problem of Bayesian model averaging (BMA) in Bayesian neural networks by re-formulating it as “collapsed BMA”, where a small number of “collapsed” samples from a subset of the parameter space (e.g., the last layer) is equipped with a posterior conditioned on the remainder. The key contribution is to cast the marginalization over this conditional posterior as a weighted volume computation (WVC) problem, computing integrals over feasible regions of arithmetic constraints that are to be inferred from binary ReLU activation patterns. As such enumeration of constraints is computationally infeasible, a weighted model integration (WMI) framework is used to solve BMA in a multi-step procedure of encoding the involved distributions and calling existing WMI solvers. The proposed algorithm competes favorably against a number of baselines.", "questions": "* RE [W2]\n  * [Q1] Can you elaborate on the appropriateness for the approximation with polynomial densities, especially for (a) the Gaussian case in high dimensions, and (b) the classification case?\n  * [Q2] Have you explored any alternatives to the uniform distribution for the conditional posterior? The empirical evidence provided in Fig. 2 is also not clear to me: (a) How can HMC and CIBER use the “same 10 samples” when HMC recovers the true posterior after convergence, and CIBER samples from the SGD trajectory? (b) I fail to see how Fig. 2b is closer to 2c than 2a.\n* RE [W3]\n  * [Q3] Fig. 3: Did you repeat the sampling process? If not, the superiority of CIBER established from 5 samples might be a lucky shot.\n  * [Q4] Classification example (l. 240): Kindly elaborate on this, I do not understand the origin of the presented numbers.\n* [Q5] l. 37: How does collapsed sampling limit to a subset of *variables*? From your descriptions, this should be a subset of *weights*.\n* [Q6] l. 99: Should one of the inequality signs be a strict “greater” or “lesser”?\n* [Q7] l. 161: CIBER claims to resolve the issue of WMI-encoding non-ReLU networks. How is this achieved? Proposition 7 still contains * $\\Delta_{ReLU}$.\n* [Q8] Eq. 2: The sum is over tuples $(w_s, q)$ but $w_s$ never appears in the summand and $q$ is the same for all $w_s$. Can you make more explicit how the running index enters the sum? On a related note, I do not understand the definition of $W_s$ (l. 183) because I don’t quite see how $w_s$ and $w$ are connected.\n* [Q9] l. 283: Could you elaborate on what you mean by “greater variance indicates that the weight is prone to have greater uncertainty and thus one might want to perform more accurate inference over it”?\n* [Q10] l. 314: Should the number of collapsed samples not depend on network architecture rather than number of classes in the final layer? \n\n—\n\nMinor remarks\n\n* l. 31: I would not agree that “existing methods mainly focus on MCMC” in BNN; approximate schemes like deep ensembles are quite popular.\n* l. 39: The claim that “collapsed samplers are effective at variance reduction in graphical models” seems irrelevant in the context of your proposal.\n* l. 62: “it can be risky to base inference on a single neural network model” sounds fairly hand-wavy.\n* l. 79 (example 2): The notation is mixing up bold and non-bold w.\n* l. 148: How is that the “uncertainty” of the prediction?\n* l. 183: The definition of $W_s$ seems overly complicated, why not just say “given a set of parameter samples $W_s$”?\n* l. 265: What is DPLL?\n* l. 272: I would recommend removing the “Boston” dataset due to its racism issues.\n* l. 300: There seems to be an “and” missing before “four”.\n* l. 302: “root mean-squared error”, not “rooted-mean-squared-errors”.\n* Some references could be added in\n  * l. 35/36 (cutset / Rao-Blackwellised samplers)\n  * l. 51 (HMC)\n  * l. 110/111 (“various” solvers; single reference)", "limitations": "I miss a discussion of the limitations, especially since there are numerous approximations and heuristics involved. Societal impacts (or rather, the absence of such) are indeed mentioned.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* [S1] **Significance**. The authors address an important problem with a focus on practical application.\n* [S2] **Originality**. The proposed solution seems novel and creative.\n* [S3] **Structure**. Structural elements like enumerated steps and questions are helpful to follow the argumentation.\n* [S4] **Intuition**. The authors make an effort to build intuition with several examples and illustrations.\n", "weaknesses": "* [W1] **Storyline**. While the authors clearly try to establish a coherent structure with well-motivated steps, the many components make it hard to follow and not lose sight of the key idea.\n   * The main topic, collapsed inference, is not addressed until page 5 – not being familiar with WVC/WMI, I felt somewhat lost in notation and descriptions of approximate procedures until the concept is even formally introduced.\n   * I would recommend to have Def. 6 early on, and to map the 4-step procedure in Section 3 directly to the collapsed BMA problem.\n   * I It would be helpful to provide more intuition around all the notation around WVC/WM – to readers mainly familiar with Bayesian inference and Bayesian deep learning, these concepts might appear quite foreign.\n   * I Also, if space constraints allow it, I would move the central CIBER algorithm into the main paper.\n * [W2] **Simplifications**. The concept seems to rely on a number of simplifying assumptions that I feel are not properly defended. \n   * I Encoding the posterior predictive distribution with polynomial densities\n   * I Choosing the conditional posterior, a key object in your proposal, to be uniform with an explanation that reads like “better than not doing it at all”\n* [W3] **Evidence**. The empirical results do not quite support the claims made.\n   * I Some of the figures/examples do not seem convincing to me, see Q2–Q4.\n   * I Your claim of “set[ting] a new state of the art” is not thoroughly funded in my opinion. \n     * I cannot find details on how the architecture was chosen for the large datasets.\n     * Baselines: In the transfer learning task, the comparison seems limited (SWA and SWAG are quite similar, and I would not consider SGD a “strong” baseline). Also, I miss Laplace approximation and especially deep ensembles as popular inference schemes that arguably operate in the few-sample setting as well.\n     * The methods are evaluated on small-scale UCI tasks, and the results are not entirely conclusive.\n   * I miss an ablation w.r.t. the number of collapsed samples, which should be an important hyperparameter of your method.\n* [W4] **No code**. With a novel method of computing BMAs, it would be helpful to have a look at the code. Also, it would allow to inspect implementation details of the experiments.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank the reviewer for appreciating our work for its significance, originality, novelty, and structures. We will address your concerns below, with references in the general response.\n\n[Questions on W2, W3]\n\n[Q1] The appropriateness of piecewise polynomial approximation is justified by the Stone-Weier theorem [1], stating that any continuous functions can be uniformly approximated by polynomials up to arbitrary precision, which holds for high dimensional space. That is, functions $f$ including multivariate Gaussian and softmax can all be effectively approximated by piecewise polynomials, with theoretical guarantees that for arbitrary $\\epsilon$, there exists a piecewise polynomial $p$ such that $\\parallel f - p \\parallel < \\epsilon$ holds, where $\\parallel \\cdot \\parallel$ denotes the supremum norm.\n\n[Q2] We agree that to explore what are the alternative approximations to the posterior is interesting for future work; in our work, it is chosen to be uniform, and we provide a further justification for this choice in the general response.  In Figure 2, 10 samples are drawn from a posterior distribution where HMC uses them for inference while CIBER first applies a uniform approximation to them before the closed-form inference. This figure demonstrates that for overconfident ReLU NNs, using HMC with 10 samples does not help the overconfidence issue much: its confidence intervals are irregular and often too narrow, and many predictions have the shaded region much narrower than those in Figure 2c. While in Figure 2b, the shaded region is wide and for the predictions that are more off from the ground truth, the shaded region is even wider meaning that those predictions come with less confidence, showing that using a uniform approximation to posterior helps estimate the uncertainty in a consistent way.\n\n[Q3] We repeat the sampling for the synthetic Bayesian linear regression experiment for 10 trials as suggested by the reviewer. The resulting comparison of the predictive distributions is presented in Figure A.1 in the rebuttal pdf, where the estimation by CIBER is much closer to the ground truth posterior predictive distribution than the Monte Carlo method. Further, the averaged KL divergence between the ground truth and CIBER is 0.069 while the one for MC estimation is 0.130, again indicating that CIBER yields a better BMA approximation in the few-sample setting.\n\n[Q4] Starting from Line 240, we aim to provide a comparison between CIBER and the Monte Carlo method in the classification setting. A previous work [2] proposes that such comparison on BMA approximation can be achieved by comparing how close an estimation is to the ground-truth integral as defined in Line 241. We follow the setting in [2] and the results show that the integral estimation $I_c$ by CIBER is closer to the ground-truth integral $I$ than the MC estimation $I_{\\mathit{MC}}$.\n\n[Q5] We mention “limiting sampling to a subset of variables” only in Line 37 in the context of the collapsed sampler in the probabilistic graphical model literature and in the remaining of the paper, it is mentioned as the sampling of parameters in the context of Bayesian deep learning.\n\n[Q6] We’ll fix this typo.\n\n[Q7] This issue is resolved by the sampling part in the collapsed samples. Specifically, we propose to use collapsed samples to combine the strengths from two worlds: the scalability and flexibility from sampling and the accuracy from WMI solvers, where flexibility refers to the fact that given a neural network that might contain layers not amenable to WMI encoding, we can sample a subset of network weights including those in such layers, and further condition the network on the sample, to result in a sub-network that is amenable to the WMI encoding.\n\n[Q8] As mentioned in Definition 6, $(\\mathbf{W}_s, \\mathbf{W}_c)$ is a partition of parameters $\\mathbf{W}$, meaning that $\\mathbf{W}_s \\cup \\mathbf{W}_c = \\mathbf{W}$ and $\\mathbf{W}_s \\cap \\mathbf{W}_c = \\emptyset$. With lower case denoting the assignment, it holds that $\\mathbf{w}_s \\cup \\mathbf{w}_c = \\mathbf{w}$ and $\\mathbf{w}_s \\cap \\mathbf{w}_c = \\emptyset$, that is, $\\mathbf{w}_s$ is implicit in Equation 2. We provide an expanded version of this equation in Section B in the rebuttal pdf to clear this confusion.\n\n[Q9] How to choose the collapsed parameter set is an open question in the design of collapsed samplers. One heuristic we provide is to choose the weight parameters whose samples from the SGD trajectory have high variance. This is because the lower the variance a weight has in its samples, the closer the weight is to a deterministic variable, and there is no need to maintain a distribution over deterministic weights. Thus, we choose the weights with high variance in their samples instead since they are further from being deterministic than the others.\n\n[Q10] The numbers in Line 314 are the size of the collapsed parameter sets; for the number of collapsed samples, it is independent of the network architecture and the number of classes, and we keep it the same as the number of samples that the baseline SWAG [3] for a fair comparison.\n\n[On W1]\n\nWe agree that the suggested change in the storyline can help readers have a full picture of the collapsed inference scheme before introducing the connection between BMA and WVC, which is easy to make. We would also like to point out that, which part is easier or more important, the collapsed inference scheme vs. the reduction from BMA to WVC, depends on the reader's background and that other readers might prefer the current presentation.\n\n[On W4]\n\nWe have provided our code using an anonymous link and have sent the link to AC.\n\n[fianl remark]\n\nWe hope that our responses address your concerns and we kindly hope that you would consider raising the score accordingly. We feel that the foundational contribution of this paper has arguably not been fully appreciated, at least not incorporated into the scores, and should be given more weight."}, {"review_id": "C9HmXCaFMJ", "reviewer": "Reviewer_e8p8", "summary": "The paper proposes to use techniques from weighted volume computation to deterministically marginalize over (a subset of) the weights in a BNN rather than sampling them when computing the predictive posterior. The experiments find the method to perform competitively with some standard baselines from the literature on UCI and CIFAR benchmarks\n\nI think this is a nice methodological contribution, although I wish the experiments reported the computational cost and explored the impact of only being able to select a subset of the weights for volume computation (i.e. more detailed analysis of the limitations). So overall I would lean towards acceptance.", "questions": "I’d mainly like to see some discussion on the points mentioned in the weaknesses.\n", "limitations": "The computational cost is mentioned but not further substantiated by runtime figures and comparisons.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Using volume computation for BNNs is new as far as I am aware, and I find work that connects different sub-fields in the literature generally quite valuable.\n* The paper is very well written, I found the low-dimensional running example used throughout the paper very effective.\n* The method seems to perform well in the experiments.", "weaknesses": "* The paper mentions that applying volume computation over all weights would not be feasible, but does not report how expensive it is on the subset that it select.\n* Similarly, I would have liked to see some analysis of the compute time-performance tradeoff as one varies the number of weights that are used in volume computation rather than sampled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply thank the reviewer for appreciating our work for its novelty, clear presentation, and thorough empirical evaluations. We truly appreciate that you find the connection between sub-fields proposed in our work to be quite valuable. In what follows, we will address your concerns.\n\n[runtime complexity]\n\nWe provide the runtime of CIBER on dataset CIFAR-100 dataset in the classification experiment with the three neural network models being VGG-16, PreResNet-164, and WideResNet28x10 respectively as below, to showcase the empirical computational complexity of CIBER. We compare our runtime with SWAG which is known to be a simple and scalable Bayesian deep learning approach summarized in the table below. The runtime of CIBER consists of two main processes: 1) training time, which is exactly the same as the runtime of SWAG, and 2) WMI solving time. The table shows that CIBER is almost as efficient as SWAG and that WMI solving brings little computational overhead due to the efficiency of the WMI solvers.\n\n| MODEL           |   VGG-16   | PreResNet-164 | WideResNet28x10 |\n|-----------------|:----------:|:-------------:|:---------------:|\n| Training / SWAG | 2h3m59s    | 6h3m18s       | 29h39m28s       |\n| WMI Solving     | 11m43.134s | 11m20.638s    | 19m1.076s       |\n\n[runtime-performance tradeoff]\n\nWe perform analysis on the runtime-performance tradeoff as suggested by the reviewer, with results presented as Figure A.4 in the rebuttal pdf. The analysis is carried out in the Bayesian linear regression setting where the ground-truth posterior predictive distribution is available. The performance of CIBER is evaluated using the KL divergence between the estimated posterior predictive distribution and the ground truth. As we increase the size of the collapsed set, the BMA integrals are defined over a higher dimension, leading to longer WMI solving time; in the meanwhile, the accuracy brought by the closed-form approximate inference over the collapsed set leads to lower KL divergence. We'll include this analysis in the camera-ready version.\n"}, {"review_id": "7QhWCJ1H33", "reviewer": "Reviewer_PnkR", "summary": "The paper provides a closed-form approximation for the posterior predictive distribution in Bayesian deep learning, in both regression and classification. The paper is overall clear and quite well written. The theory seems reasonable, however a theoretical analysis on the approximation error, depending either on the number of model parameters or on the number of predictive samples, is not provided. Currently, it is not particularly clear to me the sensitivity of the results to the number of samples. As Monte Carlo will eventually win for a large number of samples, I believe it is important to understand up to what point the proposed method is better. ", "questions": "\nSome more granular comments below.\n\n- I find Figure (2) not very convincing. The authors claim that the predictive uncertainty using 10 samples from CIBER is closer than the one using 10 sample from HMC to the benchmark approximation given by 2k samples from HMC. First of all, the blue prediction from CIBER seems further off than HMC with 10 samples. Consequently, the uncertainty interval is wider. While it is positive that the larger uncertainty is adjusted to to the wider prediction error, to me HMC-10 seems closer to HMC-2000 than CIBER-10. It would also be interesting to see what happens with a few more samples, say 20 or 30, which is still low but often possible. Also, I assume the 10 samples were taken in the asymptotic regime, that is after HMC converged?\n\n- Also in Figure 3, the authors use 5 samples only. As this is very low, it would be interesting to see at what point MC becomes better. E.g. is it 7 samples? 10? 100? 1000? This would give the reader a more clear perspective of when an algorithm should be used instead of the other.\n\n- Line 128: It would be good to expand on the meaning of this definition. The example below clarifies a bit, but, being the terminology non-standard, it is still not immediate to understand what \"satisfaction of an SMT formula\" means, and the idea underlying the definition of WMI.\n\n- I cannot find in the appendix the piecewise polynomial approximations for classification mentioned at line 213. \n\n- In proposition 7, the authors derive expressions for predictive density and predictive mean. What about other statistics, like variance and entropy? Can these be derived in closed form as well? If so, it would be good to provide an expression.\n\n- To me the definition of collapsed BMA remind a lot partially stochastic neural networks, that is when Bayesian inference is performed on part of the network only. In fact, give a posterior distribution p(w|D), where w=[w_c, w_s], you can always decompose p(w|D)=p(w_c|w_s, D)p(w_s|D). If you approximate p(w_s|D) with an empirical distribution at a bunch of samples, there you have collapsed BMA. In partially stochastic neural networks, usually p(w_s|D) is just a single Dirac delta rather than an ensemble. In this paper, instead, w_c are taken along the trajectory of SGD, similarly as in SWAG. Perhaps it would be worth to explain the connection.\n\n- In the conditional posterior approximation, how are the lower and upper limits of the uniform distribution defined?\n\n- The paper mentions [Kristiadi et al. 2022] a few times. In these paper, the authors also provide a closed-form approximation for the predictive distribution in classification, i.e. the multi-class probit approximation. Since their rationale is similar as for this paper (closed-form approximations in a low-sample regime), it would be good to see a comparison.", "limitations": "There does not seem to be much discussion about limitations. The authors could better discuss in which scenarios their approximation is better or worse than other methods.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is quite well written. It attempts to provide closed-form approximation for the posterior predictive distribution in Bayesian deep learning, which is a meaningful efforts considering that Monte Carlo samples can be expensive. The approximations proposed by the paper seem reasonable. The paper proposes a number of experiments and benchmarks against several methods.", "weaknesses": "- The paper lacks of a sensitivity analysis to the number of samples.\n\n- It would be great if the error for some of the approximations could also be expressed analytically. For example, can we say anything about the error of a triangular approximation?\n\n- If we strip the paper of its motivating terminologies (WMI and collapsed samples), seems similar in spirit to partially stochastic Bayesian inference, with a uniform posterior approximation and a triangular approximation of the likelihood to work out a closed-form approximation for the predictive. There does not seem to be a direct connection/comparison with this part of the literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for appreciating our work for its clear presentation, solving a meaningful problem, reasonable approximation, and thorough empirical evaluations. In what follows, we will address your concerns, with the references put in the general response due to the character limit.\n\n[Q1] Figure 2 demonstrates that for overconfident ReLU NNs, using HMC with 10 samples does not help the overconfidence issue much: its confidence intervals are irregular and often too narrow, and many predictions have the shaded region much narrower than those in Figure 2c. While in Figure 2b, the shaded region is wide and for the predictions that are more off from the ground truth, the shaded region is even wider meaning that those predictions come with less confidence, showing that using a uniform approximation to posterior helps estimate the uncertainty in a consistent way. Still, we provide a quantitative analysis of the number of samples. Please see our response to [Q2] below.\n\n[Q2] We perform a comparison between CIBER and the MC method for the Bayesian linear regression setting as suggested by the reviewer to see how many samples the MC method needs to match the performance of CIBER with the result presented in Figure A.2 in the rebuttal pdf. The performances of both approaches are evaluated using KL divergence between the estimated posterior distribution and the ground-truth one, averaged over 10 trials. In Figure A.2, the dashed green line shows the performance of CIBER with 50 samples and the blue curve shows the performance of MC with an increasing number of samples. As expected, the MC method yields lower KL divergence as the number of samples increases; however, it takes more than 100 samples to match CIBER, indicating its low sample efficiency and that developing efficient and effective inference algorithms such as CIBER for estimating BMA is a meaningful question.\n\n[Q3] Please see [satisfaction to SMT formulas] in the general response.\n\n[Q4] For classification, a piecewise polynomial approximation with the polynomial degree being three is applied to the sigmoid function which we visually present in Figure A.3 in the rebuttal pdf, where again is obtained by minimizing the L2 distance. We’ll include the explicit expression of the piecewise polynomial approximation in the next version.\n\n[Q5] For variance, we derive the closed-form expression in Proposition B.1 in the rebuttal pdf; this is a direct generalization of our results because variance can be derived from the second moment of the distribution which is an expectation over $y^2$, a polynomial. While for entropy, the existing WMI solver does not allow the log operation and thus there’s no direct result. We do think it is an interesting future research question on what statistics can be exactly/approximately solved by WMI solvers and how such results can motivate the derivation of new algorithms for accurate and reliable inference.\n\n[Q6] The idea of partially stochastic neural networks [4] is to partition the weights into two sets: a deterministic set where the weights are assigned to be MAP estimation and a stochastic set where the weights are stochastic and approximated using existing Bayesian deep learning methods such as Laplace approximation, variational inference or SWAG. This is different from CIBER where the neural network is fully stochastic. CIBER also partitions the weights into two sets: a sampling set where weights are approximated by SWAG and a collapsed set where the weights are approximated using our proposed closed-form inference. A future research direction can be to combine both by applying the closed-form inference to the stochastic set in the partially stochastic neural networks to study if the observations on stochasticity in [4] still hold. We’ll this discussion and the related work on the partially stochastic neural networks including [4] in the camera-ready version.\n \n[Q7] The lower and upper limits for each weight are defined by the minimum and the maximum of the weight samples drawn from the SGD trajectories respectively.\n\n[Q8] The closed-form approximation in [2] is motivated by a binary probit approximation when the logit conforms to a Gaussian distribution, that is, the posterior predictive is a Gaussian, which is exactly the same setting we have for the synthetic classification in order to provide a quantitative analysis of the approximation when the BMA ground truth is available. In this setting, the probit approximation is almost exact. Still, such approximation is specific to classification while our proposed approximation using a reduction to weighted volume computation is applicable to layers with various activations as shown in experiments. Also, it assumes mean-field approximation that ignores correlations induced by activations such as ReLU while the ReLU can be naturally encoded in the weighted model integration framework.\n\n[piecewise-polynomial approximation] \nThe piecewise polynomial approximation enjoys theoretical guarantees on approximation error by the Stone-Weier theorem [4], stating that any continuous functions can be uniformly approximated by polynomials up to arbitrary precision, which holds for high dimensional space. That is, functions $f$ including multivariate Gaussian can all be effectively approximated by piecewise polynomials, with theoretical guarantees that for arbitrary $\\epsilon$, there exists a piecewise polynomial $p$ such that $\\parallel f - p \\parallel < \\epsilon$ holds, where $\\parallel \\cdot \\parallel$ denotes the supremum norm. In our work, we choose to approximate Gaussian using the triangular distributions which is common in numerical analysis and has been adopted in various applications [5,6].\n"}, {"review_id": "NtAqBdLqLs", "reviewer": "Reviewer_cv4P", "summary": "The paper presents a new method for calculating Bayesian integrals such as the Bayesian model average (BMA) based on volume computation schemes. Specifically, the authors draw inspiration from a weighted volume computation (WVC) problem. Since the WVC is intractable for common neural networks they approximate it with weighted model integration (WMI) that can compute volumes of literals connected with logical connectivities. Some approximations are used to accommodate WMI solvers, such as approximating a Gaussian likelihood with a triangular distribution, and using the WMI solver on part of the network (usually a few dozens of parameters). Results show superior performance compared to sgd-based Bayesian methods and comparable results to Bayesian last-layer methods.", "questions": "* Perhaps I do not understand something, since some SMT formulas are built from logical conjunction with conflicting conditions, such as the Relu one, how can they be satisfied? There is no $W$ that satisfies both conditions for the same $x$, no?\n* Why did you approximate the Gaussian likelihood with a triangular distribution instead of a truncated Taylor series for the exponential function?\n* Have you tested your method's sensitivity to temperature scaling?", "limitations": "The authors did not address the limitations of their method. Please see the weaknesses/questions sections.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Overall I liked the idea of the paper and the novel view it provides for computing Bayesian integrals. Strengths:\n* Novel and interesting characterization of the BMA as a volume computation problem\n* Novel way of approximating the BMA and other integrals using WMI \n* The authors suggested practical and efficient ways to accommodate WMI to the Bayesian learning framework\n* The use of a motivating example throughout is a good idea\n* For the most part the paper is written clearly", "weaknesses": "There are several issues I would like the authors to address:\n\n**Method.**\n* The assumption of a uniform posterior is a bit odd and I didn't understand the intuition as presented in L199 in the main text and L510 in the appendix. It is very reasonable to assume that the posterior is not uniform, even when evaluated only on part of the network. A proper justification or intuition for this choice is missing. Also, the good empirical results can be a byproduct of being Bayesian on only part of the network as evident in [1].\n\n**Experiments.**\n* In my opinion, the authors exaggerate in how they present their method performance (e..g, \"significant improvements\", \"new state of the art\"). First, the method is not compared against all relevant Bayesian methods so it cannot be considered as SoTA. Second, the improvements are generally mild, and when factoring Bayesian last-layer methods (which are for some reason presented in the Appendix), the method is only comparable to them. Furthermore, with more modern NNs CIBER performance is usually inferior to several baseline methods.\n* It is not clear why the authors didn't present the performance of Bayesian last layer methods on the CIFAR datasets. In my opinion, it's as important as presenting SGD-based Bayesian methods.\n * A quantification of the method complexity (e.g., wall-clock time) is missing. To me, it is not clear how expensive WMI solvers are and how it grows with the number of parameters. I would have expected to see a comparison to baseline methods w.r.t that aspect as well.\n* Experimental details are severely lacking (e.g., was there a validation split? What were the hyper-parameters? Did you do a grid search over hyper-parameters? If so, on what values? ).\n* Information about how to choose some of the constants such as $\\alpha, l_i, u_i$ is missing. Have you searched for possible values? In general, how did you set their values? \n* Code wasn't added to the submission. When factoring this bullet and the last 2 bullets, in my opinion, the results in this paper are not reproducible and this harms this paper's score.\n\n**Clarity of presentation.**\n* I think a short background on the main concepts of SMT will make the paper self-contained and easier to read.\n* In Fig. 2 it would help to plot the training points as well.\n\n[1] Sharma, M., Farquhar, S., Nalisnick, E., & Rainforth, T. (2023, April). Do Bayesian Neural Networks Need To Be Fully Stochastic?. In International Conference on Artificial Intelligence and Statistics (pp. 7694-7722). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for appreciating our work for a novel view of computing Bayesian integrals, a practical and efficient framework, and a clear presentation. In what follows, we will address your concerns, with the references put in the general response due to the character limit.\n\n[uniform approximation to posterior]\nPlease see our answer in the global response.\n\n[fully/partially Bayesian]\nThe idea of partially stochastic neural networks in [4] is to partition the weights into two sets: a deterministic set where the weights are assigned to be MAP estimation and a stochastic set where the weights are stochastic and approximated using existing Bayesian deep learning methods such as Laplace approximation, variational inference or SWAG. This is different from CIBER where the neural network is fully stochastic instead of being partly stochastic. CIBER also partitions the weights into two sets: a sampling set where weights are approximated by SWAG and a collapsed set where the weights are approximated using our proposed closed-form inference. A future research direction can be to combine both by applying the closed-form inference to the stochastic set in the partially stochastic neural networks to study if the observations on stochasticity in [4] still hold.\n\n[baselines]\nIn the empirical evaluations, we compare with baselines ranging from the mostly related ones using samples from SGD trajectories for posterior approximations to variational-inference-based ones, deep ensemble, MC dropout, etc, with strictly more baselines than [8]. The Bayesian final layer method [7] is included in both the main paper in Table 2 for log-likelihood and in the appendix in Table 6 for RMSE as direct comparisons to the numbers reported in [8], where the Bayesian final layer method is only reported in regression tasks and not the classification tasks.\n\n\n[runtime complexity]\nWe provide the runtime of CIBER on dataset CIFAR-100 dataset in the classification experiment with the three neural network models being VGG-16, PreResNet-164, and WideResNet28x10 respectively as below, to showcase the empirical computational complexity of CIBER. We compare our runtime with SWAG which is known to be a simple and scalable Bayesian deep learning approach summarized in the table below. The runtime of CIBER consists of two main processes: 1) training time, which is exactly the same as the runtime of SWAG, and 2) WMI solving time. The table shows that CIBER is almost as efficient as SWAG and that WMI solving brings little computational overhead due to the efficiency of the WMI solvers.\n\n| MODEL           |   VGG-16   | PreResNet-164 | WideResNet28x10 |\n|-----------------|:----------:|:-------------:|:---------------:|\n| Training / SWAG | 2h3m59s    | 6h3m18s       | 29h39m28s       |\n| WMI Solving     | 11m43.134s | 11m20.638s    | 19m1.076s       |\n\n\n[experimental details/code]\nWe put the experimental details of both the regression and the classification experiments in Section C.2 and Section C.3 respectively. The way that the hyper-parameters including learning rates and weight decays are chosen exactly follows [3] as mentioned in Appendix. The hyper-parameter $\\alpha$ as described from Line 142 to Line 145 is 2.45. The parameters $\\ell, u$ are the bounds of the uniform posterior defined by the minimum and the maximum of the weight samples drawn from the SGD trajectories respectively. We’ll include these details in the next version to make it more self-contained. We’ve also provided our code using an anonymous link and sent the link to AC for your reference.\n\n[presentation]\nWe’ll include the suggested changes in the presentation in the next version.\n\n[Q1] An assignment $\\mathbf{x}$ satisfies an SMT formula $\\Delta$ defined over variables $\\mathbf{X}$ if the formula $\\Delta$ is evaluated to be true after substituting the variables by their assignments. For example, given an SMT formula encoding a ReLU, $( (W · 1) > 0 \\Rightarrow Z = W · 1) \\land (W · 1 ≤ 0 \\Rightarrow Z = 0)$, the assignment $(W = 3, Z = 1)$ satisfies the formula since both $(W · 1) > 0 \\Rightarrow Z = W · 1)$ and $(W · 1 ≤ 0 \\Rightarrow Z = 0)$ are evaluated to be true in this conjunction. Another satisfying assignment would be $(W = -1, Z = 0)$. While an assignment $(W = 3, Z = -1)$ does not satisfy $\\Delta$ since $(W · 1) > 0 \\Rightarrow Z = W · 1)$ is evaluated to be false, even though $(W · 1 ≤ 0 \\Rightarrow Z = 0)$ is evaluated to be true, due to the conjunction of both. We’ll include the definitions and intuitive explanations of the satisfaction of a logical formula in the camera-ready version for readability.\n\n[Q2] One message we would like to deliver in this work is that a closed-form inference, even formed by low-order polynomials, is able to deliver accurate approximation, as shown in Figure 1. To form an approximation to the Gaussian distribution, the Taylor series is not applicable since mathematically it is for approximating the function locally near a certain point instead of approximating the whole distribution. The approximation of Gaussian using a triangular distribution in our work is common in numerical analysis and has been adopted in various applications [3,4].\n\n[Q3] We agree that it will be interesting future work to study the combination of CIBER and temperature scaling."}]

Author rebuttal:
We would like to thank all the reviewers for their insightful comments and valuable suggestions, and for deeming our work to be well written (all reviewers) with helpful intuitions (eHBs, cv4P) while presenting a novel and creative view (eHBs,cv4P,e8p8) to an important problem (eHBs,PnkR) that builds a valuable connection between sub-fields (e8p8).

[rebuttal pdf / code]
We’ve attached a rebuttal pdf for the figures. Additionally, we provide our code for all experiments using an anonymous link and have sent the link to AC.

[uniform approximation to posterior]
We would like to provide a further justification for the uniform approximation to posterior using a comparison with the Monte Carlo method as follows. Given samples from a posterior $\mathbf{w_i} \sim p(\mathbf{w}\mid \mathcal{D})$ , a Monte Carlo sampling procedure estimates the posterior predictive distribution by aggregating over a finite number of samples, i.e., $p(y \mid x, \mathcal{D}) = \frac{1}{N} \sum_{i = 1}^n p(y \mid \mathbf{w_i}, x)$, which is equivalent to an expectation of the predictive $p(y \mid \mathbf{w}, x)$ under a discrete uniform distribution $\mathcal{U}(\mathbf{w_1}, \cdots, \mathbf{w_n})$, that is, $p(y \mid x, \mathcal{D}) = E_{\mathbf{w} \sim \mathcal{U}} [p(y \mid \mathbf{w}, x)]$. In CIBER, instead of aggregating over a finite number of weight samples using a discrete uniform distribution, we consider a continuous uniform distribution $\mathcal{U}^\prime[ \mathbf{w_{\mathit{min}}}, \mathbf{w_{\mathit{max}}}]$ where the weight limits are estimated by the samples, to estimate the predictive posterior using a continuous ensemble, i.e., $p(y \mid x, \mathcal{D}) = E_{\mathbf{w} \sim \mathcal{U}^\prime} [p(y \mid \mathbf{w}, x)]$, which can be solved exactly using our propose reduction to WMI problems.

[satisfaction to SMT formulas]
An assignment $\mathbf{x}$ satisfies an SMT formula $\Delta$ defined over variables $\mathbf{X}$ if the formula $\Delta$ is evaluated to be true after substituting the variables by their assignments. For example, given an SMT formula $X_1 + X_2 > 1$, the assignment $(x_1 = 3, x_2 = -1)$ satisfies the formula since $3 + (-1) > 1$ is evaluated to be true, while the the assignment $(x_1 = 0, x_2 = -1)$ does not satisfy the formula since $0 + (-1) > 1$ is evaluated to be false. The WMI problem intuitively defines a conjunction of weighted polytopes where the polytopes are defined by the SMT formulas and the weights are defined by the per-literal weights, and the task of WMI amounts to compute the weighted volume of these polytopes by integrating all the weighted assignments that satisfy the SMT formulas. We agree that these terminologies, which might be common in the logic community, are not familiar to many readers. We’ll include their definitions and intuitive explanations in the camera-ready version.


[limitations]
In this work, we draw the connection between BMA and WVC which inspires us to provide closed-form approximation to BMA integrals using WMI solvers for accurate inference. Still, the WMI encoding of neural network models is only applicable to layers that can be expressed as SMT formulas. Also, the current WMI solvers are limited to polynomial weights, and thus the reduction to WMI problems is applicable to piecewise polynomial weights. This limitation might be alleviated in the future by the development of new WMI solvers that are flexible in the weight function families. We’ll include this discussion of limitations in the camera-ready version.

[References]

[1] De Branges, Louis. "The Stone-Weierstrass theorem." Proceedings of the American Mathematical Society 10.5 (1959): 822-824.

[2] Kristiadi, Agustinus, Runa Eschenhagen, and Philipp Hennig. "Posterior Refinement Improves Sample Efficiency in Bayesian Neural Networks." Advances in Neural Information Processing Systems.

[3] Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and Wilson, A. G. A simple baseline for Bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32:13153–13164, 2019.

[4] Sharma, Mrinank, et al. "Do Bayesian Neural Networks Need To Be Fully Stochastic?." International Conference on Artificial Intelligence and Statistics. PMLR, 2023.

[5] Scherer, William T., Thomas A. Pomroy, and Douglas N. Fuller. "The triangular density to approximate the normal density: decision rules-of-thumb." Reliability Engineering & System Safety 82.3 (2003): 331-341.

[6] Karsh, P. K., T. Mukhopadhyay, and S. Dey. "Stochastic low-velocity impact on functionally graded plates: Probabilistic and non-probabilistic uncertainty quantification." Composites Part B: Engineering 159 (2019): 461-480.

[7] Riquelme, Carlos, George Tucker, and Jasper Snoek. "Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling." International Conference on Learning Representations. 2018.

[8] Izmailov, Pavel, et al. "Subspace inference for Bayesian deep learning." Uncertainty in Artificial Intelligence. PMLR, 2020.


Meta-review: In the rebuttal authors satisfactorily replied to open questions and concerns, leading all reviewers to increase there scores, three of which vote for acceptance. The authors should definitely apply the promised changes in the revised version of their paper.
Decision: Accept (poster)