Sparse Uncertainty Representation in Deep Learning with Inducing Weights
Authors: Hippolyt Ritter
Martin Kukla
Cheng Zhang
Yingzhen Li
Venue:  (2021)
Keywords: Bayesian neural networks
uncertainty estimation
memory efficiency

Abstract: Bayesian neural networks and deep ensembles represent two modern paradigms of uncertainty quantification in deep learning. Yet these approaches struggle to scale mainly due to memory inefficiency issues, since they require parameter storage several times higher than their deterministic counterparts. To address this, we augment the weight matrix of each layer  with a small number of inducing weights, thereby projecting the uncertainty quantification into such low dimensional spaces. We further extend Matheron's conditional Gaussian sampling rule to enable fast weight sampling, whichenable our inference method to maintain reasonable run-time as compared with ensembles. Importantly, our approach achieves competitive performance to the state-of-the-art in prediction and uncertainty estimation tasks with fully connected neural networks and ResNets, while reducing the parameter size to $\leq 47.9\%$ of that of a single neural network. 

Under review as a conference paper at ICLR 2021
SPARSE UNCERTAINTY REPRESENTATION IN DEEP LEARNING WITH INDUCING WEIGHTS
Anonymous authors Paper under double-blind review
ABSTRACT
Bayesian neural networks and deep ensembles represent two modern paradigms uncertainty quantification in deep learning. Yet these approaches struggle to scale mainly due memory inefficiency issues, since they require parameter stor- age several times higher than their deterministic counterparts. To address this, we augment the weight matrix of each layer with small number of inducing weights, thereby projecting the uncertainty quantification into such low dimen- sional spaces. We further extend Matheron's 's conditional Gaussian sampling rule to enable fast weight sampling, which enables our inference method to main- tain reasonable run-time compared with ensembles. Importantly, our approach achieves competitive performance to the tfate-of-the-arr in prediction and uncer- tainty estimation tasks with fully connected neural networks and ResNets, while reducing the parameter size to I 47.9% of that of single neural network.
INTRODUCTION
Deep learning models are becoming deeper and wider than ever before. From image recognition models such as ResNet-101 (He al., 201 and DenseNet (Huang al., 2017) to BERT (Xu et al., 2019) and GPT-3 (Brown et al., 2020) for language modelling, deep neural networks have found consistent success in fitting large-scale data. As these models are increasingly deployed in real-world applications, calibrated uncertainty estimates for their predictions become crucial, espe- cially in safety-critical areas such as healthcare. In this regard, Bayesian neural networks (BNNs) (MacKay, 1995; Blundell et al., 2015; Gal & Ghahramani, 2016; Zhang et al., 2020) and deep ensembles (Lakshminarayanan et al., 2017) represent two popular paradigms for estimating un- certainty, which have shown promising results in applications such as (medical) image processing (Kendall & Gal, 2017; Tanno et al., 2017) 7) and out-of -iitriiiiitiiitioon detection (Ovadia et al., 2019).
Though progress has been made, one major obstacle to scaling up BNNs and deep ensembles is the computation cost in both time and space complexities. Especially for the latter, both approaches re- quire the number of parameters to be several times higher than their deterministic counterparts. Recent efforts have been made to improve their memory efficiency (Louizos & Welling, 2017; Swiatkowski et al., 2020; Wen et al., 2020; Dusenberry al., 2020). Still, these approaches re- quire storage memory that is higher than storing a deterministic neural network.
Perhaps surprisingly, when taking the width of the network layers the infinite limit, the resulting neural network becomes "parameter efficient". Indeed, an infinitely wide BNN becomes a Gaussian process (GP) that is known for good uncertainty estimates (Neal, 1995; Matthews al., 2018; Lee et al., 2018). Effectively, the "parameters" of a GP are the datapoints, which have considerably smaller memory footprint. To further reduce the computational burden, sparse posterior approxima- tions with small number of inducing points are widely used (Snelson & Ghahramani, 2006; Titsias, 2009), rendering sparse GPs more memory efficient than their neural network counterparts.
Can we bring the advantages of sparse approximations in GPs which are infinitely-wide neural networks - to finite width deep learning models? We provide an affirmative answer regarding mem- ory efficiency, by proposing an uncertainty quantification framework based on sparse uncertainty representations. We present our approach in BNN context, but the proposed approach is applicable to deep ensembles well. In details, contributions are as follows:

---

Under review as a conference paper at ICLR 2021
We introduce inducing weights auxiliary variables for uncertainty estimation in deep neural networks with efficient (approximate) posterior sampling. Specifically:
We introduce inducing weights - lower dimensional counterparts to the actual weight matrices for variational inference in BNNs, well memory efficient parame- terisation and an extension to ensemble methods (Section 3.1).
We extend Matheron's 's rule to facilitate efficient posterior sampling (Section 3.2).
We show the connection to sparse (deep) GPs, in that inducing weights can be viewed as projected noisy inducing outputs in pre-activation output space (Section 3.3).
We provide an in-depth computation complexity analysis (Section 3.4), showing the significant advantage terms of parameter efficiency.
We apply the proposed approach both BNNs and deep ensembles. Experiments in clas- sification, model robustness and -ii-of--iitribution detection tasks show that our inducing weight approaches achieve competitive performance their counterparts in the original weight space on modern deep architectures for image classification, while reducing the parameter count to less than half of that of a single neural network.
VARIATIONAL INFERENCE WITH INDUCING VARIABLES
This section lays out the basics on variational inference and inducing variables for posterior approx- imations, which serve foundation and inspiration for this work. Given observations D (X. Y) with = [x1, .c., NN Y = [yI..... Yv we would like to a neural network p(y|æ, W1-L) with weights W..F. to the data. BNNs posit prior distribution p PWW.. over the weights, and construct an approximate posterior (...t.) to the intractable exact posterior p ((1....)) x ((D(W...))(W...))
where (D.)...)) p(Y|X, WI.L) 11 plyn WT..).
Variational inference Variational inference (Jordan et al., 1999; Zhang al., 2018a) constructs approximation (0) to the posterior p(0)D) x p(0)p(D(() by maximising a variational lower-bound:
logp(D) 2 L(q(0)) == (,) [log (D[0)] - KLL [g(0)
(1)
For BNNs, 0 = WWith and simple choice of is fully factorised Gaussian (FFG)) ((W..z) II 1 W
((m (6-5), (f (((i")) with mi m oz (6-5) the mean and variance of W:(6.)) and d'm, d' the respective number of inputs and outputs to layer 1. The variational parameters then 6b = [m, (Hoffman al., 2013) and with Monte Carlo sampling from the q distribution (Titsias & Lázaro- Gredilla, 2014; Kingma & Welling, 2014). By setting to an FFG, a variational BNN can be trained with similar computational requirements as deterministic network (Blundell al., 2015).
(i.5) o/ ) Gradients of (1) w.r.t. & can be estimated with mini-batches of data
Improved posterior approximation with inducing variables Auxiliary variable approaches (Agakov & Barber, 2004; Salimans et al., 2015; Ranganath et al., 2016) construct the (()) dis- tribution with an auxiliary variable a: q()) = ʃ q(0)a ((lfla/((a ( da, with the hope that a potentially richer mixture distribution q()) can achieve better approximations. As then (()) becomes intractable, an auxiliary variational fower-bound is used to optimise q(0,a) a):
logr(D) 2 L(g(0, a)) = (log (angpp(DD00) (D(0)) + Ry((,a) log p(0)r(a(0)
(2)
(((la){(a)
Here r(a(d) is an auxiliary distribution that needs to be specified, where existing approaches often use "reverse model" for r(al0). Instead, we define r(0]a) in a generative manner: r(a(0) (f) is the "posterior" of the following "generative model",, whose "evidence" is exactly the prior of 0:
r(al0)= = p(a 0) (()))) such that p(0) = p(a)p(0)o)da = ((B).
(3)
Plugging in (3) to (2) immediately leads to:
C(q(0, a)) = E, (log
Ey(a) [K ( [[){a)))]]] KL(g(a)l
(4)
This approach returns an efficient approximate inference algorithm, translating the complexity of inference in 0 to if dim(a) < dim(O) and q(0, a) = g(0)a)g(a) has the following properties:
2

---

Under review as a conference paper at ICLR 2021
1. A "pseudo prior" p(a))(()a) a) is defined such that p(a))p(laadda = p(0);
2. The conditional distributions g(Ola) and p(0]a) are in the same parametric family, so that they can share parameters;
3. Both sampling 0 ~ g0)) and computing (l ())(a( can be done efficiently;
4. The designs of q(a) and p(a) can potentially provide extra advantages (in time and space complexities and/or optimisation easiness).
We call a the inducing variable of f, which is inspired by varationally sparse GP (SVGP) with in- ducing points (Snelson & Ghahramani, 2006; Titsias, 2009). Indeed SVGP is special case: @ = f, the GP prior is p(f]X) = GP(O, Kxx), p(u) GP Kzz), p(f, u) p(u)p(f)X, u), ((f]u) - p( u), qf,, u) - p(]] u))(u), and Z are the optimisable inducing inputs. The vari- ational lound is [(q u)) Eg(r) (log p Y [ [g p(u)], and the variational param- eters are / = (Z, distribution parameters of g(u).. SVGP satisfies the marginalisation constraint (3) by defiiiton, and it has [gL [
= Also by using small M dim(u) and exploit- ing the q distribution design, SVGP reduces run-time from ((N5) to (NNM² where N is the num- ber of inputs in X, meanwhile it also makes storing a full Gaussian g(u) affordable. Lastly, u can be whitened, leading the "pseudo prior" p(f, v) p(f|X, KZZZ ((v) p(v) 0, which could bring potential benefits in optimisation.
In the rest of the paper we assume the "pseudo prior" p(0, a) satisfies the marginalisation constraint (3), allowing us to write p(0, a) == p(0, a). It might seem unclear how to design p(0, a) for an arbitrary probabilistic model, however, for a Gaussian prior on @ the rules for computing conditional Gaussian distributions can be used to construct p. In section 3 we exploit these rules to develop an efficient approximate inference method for Bayesian neural networks with inducing weights.
SPARSE UNCERTAINTY REPRESENTATION WITH INDUCING WEIGHTS
3.1 INDUCING WEIGHTS FOR NEURAL NETWORK PARAMETERS
Following the design principles of inducing variables, we introduce to each network layer a smaller inducing weight matrix Ul, and construct joint approximate posterior distributions for inference. In the rest of the paper we assume factorised prior across layers p(W1.)) = P(Wi), and for notation ease we drop the indices when the context is clear.
Augmenting network layers with inducing weights Suppose the weight W E R Xdin has A(0,0²1 where vec(W)) concatenates the columns of the weight matrix into a vector. A first attempt to augment p(vec(W)) with an inducing weight variable U E MgMMuxXMM may be to construct multivariate Gaussian p(vec(W), vec(U)), such that f p( vee(e))) vec(U))dU = (N, o²1). This means for the joint covariance matrix of (vee(W), vec()))) it requires the block corresponding prior covariance o²I. I. We are then free to parameterise the rest of the entries in the joint covari- ance matrix, as long as this full matrix remains positive definite. Now the conditional distribution UUW)U) is function of these parameters, and the conditional sampling from (WIU) is further discussed in Appendix 1. Unfortunately, as dim(vec( 10"), using a full covariance Gaussian for p( ((vevc()))) (W), vec(U)) becomes computationally intractable.
Gaussian prior p(W) p(vec(W))
the covariance of vec(W) to match the
is typically large (e.g. of the order of
This issue can be addressed using matrix normal distributions (Gupta & Nagar, 2018). Notice that the prior p(vec(W)) = (0,00 has an equivalent matrix normal distribution form as p(W)= (N 0.I,o".1 with o,., oe > the row and column standard deviations satisfying o = d,Te- Now introduce the inducing variables in matrix space, in addition to U we pad in two auxiliary variables U, E T Mout Ua E Rddoe x Min, so that the full augmented prior is:
(W.
W
U
U.. := MM E,, Ec),
E,)
(5)
U,
U
with
L,
0
Zr
D.)
D.
0
s.t. E, L.LLT--
or
0
Z,Z, + D2
ZT
[
0.11
o.Z Z.Z.
and
Le
s.t. Σ Lo
Zo
De)
o.Lo
D2
3

---

Under review as a conference paper at ICLR 2021
(b) Matheron's rule (for vectors)
Extended Matheron's rule (for matrices)
(ours)
inducing weight
augmentation
W
+
Figure I: Visualisation of (a) the inducing weight augmentation, and compare (b) the original Math- eron's rule to (c) our extended version. The white blocks represent random noises from the joint.
See Figure I(a) for a visualisation. Matrix normal distributions have similar marginalization and conditioning properties as multivariate Gaussian distributions. Therefore the marginalisation straint (3) is satisfied for any Ze, Zr, D. and D,.. The marginal distribution the inducing weight p(U)= (0,(0, Vr, V.) with V, Z, + D and V. Z. D In the experiments we use whitened inducing weights which transforms U so that p(U) MN(( 1, (Appendix E), but for clarity, we continue using the formulas presented above the main text.
The matrix normal parameterisation introduces two additional variables Ur, Ue without providing additional expressiveness. Hence it is desirable integrate them out, leading to a joint multivariate normal with Khatri-Rao product structure for the covariance:
p(vec(W), vec(U)) =N -x(0, (0(00t o2 1
0.L 0Z 00,4, V 8 Vr
O. o.
(6)
As the dominating memory complexity here is O(dou Mout + +in Min) which comes from storing Zr and Z.. we see that the matrix normal parameterisation of the augmented prior memory efficient.
Posterior approximation in the joint space We construct f'cctorieed posterior approximation across the layers: g(Wi.L. I.,, U..)) = 1I ((Wi)Ui))(Ui)).
The simplest option for a( (WIUU) is g(WJU) p(vec(W)) vec(U)) (ev(((U)))()(()) -N(wiu, Ewie). similar to sparse GPs. A slightly more flexible variant rescales the covariance matrix while keeping the mean tied, i.c. q(W [ = o( vec vec(U)) / (Wwlu, X Eww)), which still allows for the KL term to be calculated efficiently (see Appendix B):
R(A) = KL ([WW)[[))))( [U]] = din dowt (0.5)² log X - 0.5). W E Rdou xdin
Plugging = (.W.i.... {U into (4) results in the following variational lower-bound
C(g(Wi.L, U1.1)) = Eg.w.... [log (D] (((()))))) [UL(g(U)) (((U))),
with X; the associated scaling parameter for (U,). Therefore the variational parameters are now (Zc, Zr, De, Dr, X, dist. params. of {U)) for each network layer.
Two choices q(U) A simple choice is FFG (U)) = N(m,,, diag(v.)) which performs mean-field inference in U space (c.f. Blundell et al., 2015), and in this case [[))l has closed-form solution. Another choice "mixture of delta measures" q(U) = Ex_
(+))). In other words, keep K distinct sets of parameters (U1.L in inducing space that projected back into the original parameter space via the shared conditional distributions q (W;)U) to obtain the weights. This approach can be viewed constructing "deep ensembles" in U space, and we follow ensemble methods (e.g. Lakshminarayanal et al., 2017) to drop (IL[(()))PP(U)) in (8).
Often the inducing weight U is chosen have significantly lower dimensions compared W. Combining with the fact that q (W(UU) and (W) only differ in the covariance scaling constant, sparse representation of uncertainty for the network layer, as the
we see that U can be regarded major updates in (approximate) posterior belief is quantified by q(U).
3.2 EFFICIENT SAMPLING WITH EXTENDED MATHERON'S S RULE
Computing the variational lower-bound (8) requires samples from (W), which asks for an efficient sampling procedure for the conditional (W[U). Unfortunately, (WW(U) derived from (6) with
4

---

Under review as a conference paper at ICLR 2021
noisy proj.
f
h
ReLU
X
Z
X
f
h
X
ReI.U
noisy proj.
X Z
Figure 2: Showing the U variables in pre-activation spaces. To simplify we set de 1 w.l.o.g.
covariance rescaling is not matrix normal, so direct sampling remains prohibitively expensive. To address this challenge, we extend Matheron's rule (Journel & Huijbregts, 1978; Hoffman & Ribak, 1991; Doucet, 2010) efficiently sample from gW]U).. The idea that one can sample from conditional Gaussian by transforming sample from the joint distribution. In detail, we derive in Appendix C the extended Matheron's 's rule sample W (W(U))
W AW + oZ
- W.U p(W, -
(9)
Here W, U p(W, U., Ur, U) means we sample W, W,, Ur, U from the joint and drop Ue, U,. In fact U., U, are never computed: as shown Appendix C, the samples W, U can be obtained by:
= 01 Ü= ZZ + L,E,De + D.E.L + DDE Ei ~ MM(10, Idane Idrn); (10)
B2, Es, Es MN M (M,)) ) L, Cholesky (2,2,) L. Cholesky (oo2.)
Therefore the major extra cost to pay is (22M + 2M + Mon Min din) required by inverting V,,, Ve, computing Ly, L., and the matrix multiplications. The extended Matheron's rule is visualised in Figure I with a comparison to the original Matheron's 's rule for sampling from q( (vec(W)) vee(U"). This clearly shows that our recipe avoids computing big matrix inverses and multiplications, resulting in significant speed-up for conditional sampling.
3.3 UNDERSTANDING INDUCING WEIGHTS: A FUNCTION-SPACE PERSPECTIVE
We present the proposed approach again but from function- spacee inference perspective. Assume network layer computes the following transformation of the input X [.,,..., uN.. x, Rd.x1,
F WX, = (F), W Roue Xd,,, X E R XN .g.)) is the non-linearity.
(11)
As has a Gaussian prior p(v)c(W)) = O2), each of the rows in F = [f. Edouel IT IT, e NN11 has Gaussian process form with linear kernel: f||X GP(O, Kxx), Kxx (xx(m,n) TT a. Inference on F directly has O(N3 + dou N2) cost, so sparse approximation needed. Slightly different from the usual approach, introduce auxiliary variables Ue [uf,.., u Rdoue Min as follows, using shared "inducing inputs" z__ E Rd Min;
pff, ù X) GP (0, (0.K1x2211 p(uflú,) N (i.//.,0"D.))
(12)
By marginalising out the "noiseless inducing outputs" (ú.) (derivations in Appendix D), we can compute the marginal distributions p(U.) == )((u{)) MM(00, Vo) and
p(F U.) MN (U. 1, 02 Z -T Zz'Z.)
(13)
Note that PUW) MN (U.V
1, o2(1 - 2, v. -Z,) (see Appendix A.2). Since
W MM. Σ,, Σx) leads to WX A MN(MX, Σ,, TT T X), this means p(F X, U.) the
push-forward distribution of (U) for the operation F WX:
F~ (FF
W~ W~P(W().),, WX
As {uf} are the "noisy" versions of (ú,) in f space, U. can thus be viewed "scaled noisy inducing outputs" in function space (see the red bars in the 2nd column of Figure 2).
So far the inducing variables Ue is used columns of F only. In other words the output dimension d for each f, (and uf) remains large (e.g.> > 1000 in fully connected layer). Therefore dimensionality reduction be applied to the
compact representation of the correlation between
5

---

Under review as a conference paper at ICLR 2021
Table
Computational complexity for a single layer. We assume W E R xden, U R Mout x Min, and K forward passes are made for each of the N inputs. ("It uses a parallel com- puting friendly vectorisation technique (Wen et al., 2020) for further speed-up.)
Method
Time complexity
Memory complexity
Deterministic-W
O(Ndindoul
(ddindoue)
FFG-W Ensemble-V1 W
O K dend)ou))
((2din
O N K dii doout
O(Kdin Oddindout den + dout)
Matrix-normall k-tied FFG-W
O den dhout)
ON K dout
Oldindou + k(din + dout) ) Oldin 2(dnn + dout))
rank-1 BNN
O(N K Ki dout)
FFG-U
O ON 2M + + K Mow + Mindowddn)) same as above
(din Man dont Mont
2M Mont)
Ensemble-U
Oldin Min + dour M Mout + K Men Mous)
column vectors of Ue and F. In Appendix D we present a generative approach do so by extending
probabilistic PCA (Tipping & Bishop. 1999) to matrix normals: p(U,) = p(U.]U (U)))) where the projection's parameters are {Z,, ]. and p(U,, U) matches the marginals of (5). This means U can be viewed as the "noisy projected inducing output" of the GP whose corresponding "inducing inputs" are Z (see the red bar in the 1st column of Figure 2). Similarly the column vectors in U, X can be viewed as the noisy projections of the column vectors in F.
In Appendix we further show that the push-forward distribution g g(F U) of q(WIU) only differs from p(F]X, U) in the covariance matrices up to the same scaling constant A. Therefore the resulting dunction-space variational objective almost identical to (8), except for scaling coefficients that are added to the R(Ar) terms to account for the change in dimensionality from vec(W) to (Fc(F). This result nicely connects posterior inference in weight- and function-space
3.4 COMPUTATIONAL COMPLEXITIES
In Table we report the computational complexity figures for two types of inducing weight ap- proaches: FFG qU) (FFG-U) and Delta mixture q(U)) (Ensemble-U?). Baseline approaches in- clude: Deterministie-W variational inference with FFG ((W) (FFG-W, Blundell et al., 2015), deep ensemble in W (Ensemble-W. Lakshminarayanan proaches such as matrix-normal g(W (Matrix-normal-WW Louizos & Welling (2017)), variational inference with k-tied FFG q(W) (k-tied FFG-W, Swiatkowski al. (2020)). and rank-1 BNN (Dusenberry et al., 2020). The gain in memory is significant for the inducing weight approaches, in fact with Mf,. < din and Mout < dout the parameter storage requirement smaller than a single deterministic neural network. The major overhead in run-time comes from the extended Matheron's 's rule for sampling q( (U) Some the computations there are performed only once, and in our experiments we show that by using relatively low-dimensional U, the overhead is acceptable.
al., 2017), well parameter efficient ap-
EXPERIMENTS
We evaluate the inducing weight approaches on regression, classification and related uncertainty estimation tasks. The goal is to demonstrate competitive performance to popular W W-space uncer- tainty estimation methods while using significantly fewer parameters. The evaluation baselines are: (1) variational inference with FFG ((W) (FFG-W, Blundell et al., 2015) v.s. FFG q(U) (FFG-U, ours); (2) ensemble methods in W space (Ensemble-W Lakshminarayanan et al., 2017) v.s. ensem- ble in U space (Ensemble-U' ours). Another baseline is training deterministic neural network with maximum likelihood. Details and additional results can be found in Appendix and G.
4.1 SYNTHETIC 1-D REGRESSION
We follow Foong et al. (2019) to construct a synthetic regression task, by sampling two clusters of inputs ž1 u 11, 0.7], I2 U [0.5, 1). and targets y ~ (coos(4 + 0.8). 0.01). As ground truth we show the exact posterior results using the NUTS sampler (Hoffman & Gelman, 2014). The results visualised in Figure 3 with the noiseless function in black, predictive mean in blue, and

---

Under review as a conference paper at ICLR 2021
www.ww
W M
W
(a) FFG-W
FFG-U
(c) FCG-W
(d) FCG-U
(e) Ensemble-U
(1) NUTS
Figure 3: Toy regression results, with observations in red dots and the ground truth function in black.
speed
model size
Table 2: CIFAR in-distribution metrics (in %).
relaton run-time
FFC-W (N-100)
relative param. size
FFG-W
30
FFC-W (N-500)
CIFARIO
CIFAR100
FFG.U IM-128, N=100)
Method
Acc.
ECE
Acc.
ECE
FFG [M-128, N-500)
deterministic
IM=256, N=100)
Deterministic- -
93.02
5.23
72.68
19.41
FFC-U (M-256. N-500)
Ensemble-W
94.94
1.25
76,61
6.25
FFG-U
FRG-W
93.22
0.55
73.44
5.49
FFG-U
91.52
1.31
75.69
5,20
Ensemble-U
92.20
0.80
76.10
2.49
number samples (K)
Figure 4: Resnet- 18 run-times and model sizes.
up to three standard deviations shaded area. Similar to prior results in the literature, FFG-W fails to represent the increased uncertainty away from the data and in between clusters. While un- derestimating predictive uncertainty overall, FFG-U show small increase in predictive uncertainty away from the data. In contrast, per-layer full covariance Gaussian in both weight (FCG-W) and inducing space (FCG-U) as well as Ensemble-U better captures the increased predictive variance, although the mean function is more similar to that of FFG-W.
4.2 CLASSIFICATION AND NN-DITRIBUTION CALIBRATION
As the core empirical evaluation, we train Resnet- 18 models (He et al., 2016b) on CIFAR-10 and CIFAR- 100 (Krizhevsky et al., 2009). To avoid underfitting issues with FFG-W, useful trick is to set an upper limit maz on the variance of ((W) (Louizos Welling, 2017). This trick is similarly applied to the U-space methods, where cap À S Amaz for qWWIU), and for FFG-U we also set o max for the variance of q(U). .. In convolution layers, we treat the 4D weight tensor W of shape Cout Carbw matrix. We use U matrices of shape 128 128 for all layers (i.e. M Mnn = Mout = 128), except that for CIFAR-10 we set Mout = 10 for the last layer.
(Cout, Can, h, w)
In Table 2 report test accuracy and test expected calibration error (ECE) (Guo et al., 2017) first evaluation of the uncertainty estimates. Overall, Ensemble-W achieves the highest accuracy, but is not well-calibrated as variational methods. For the inducing weight approaches, Ensemble- U outperforms FFG-U on both datasets. It is overall the best performing approach on the more challenging CIFAR-100 dataset Glose-to-Enseemle--W accuracy and lowest ECE).
In Figure we show prediction run-times on trained models, relative to those of an ensemble of deterministic networks, well relative parameter sizes single ResNet-18. The extra run-time costs for the inducing methods come from computing the extended Matheron's 's rule. However, as they can be calculated once and then cached when drawing multiple samples, the overhead reduces to a small factor when using larger number of samples K and large batch-size N. More impor- tantly, when compared deterministic ResNet-18 network, the inducing weight models reduce the parameter count by over 50% (5, 352, 853 vs. 11, 173, 962, 47.9% even for a large M 128.
pprrm-e choices We visualise in Figure 5 the accuracy and ECE results for the inducing clear from the right-most panels that perfor- mances in both metrics improve as the U matrix size M is increased, and the results for M = 64 and M 128 are fairly similar. Also setting proper values for Amaas maaz is key to the improved per- formances. The left-most panels show that with fixed dman values (or with ensemble in U space), the preferred conditional variance cap values Amaz are fairly small (but still larger than which corresponds to a point estimate for W given U). For Mmax which controls variance in U space, see from the top middle panel that the accuracy metric is fairly robust to Fmaz as long as Xmnax is not too large. But for ECE, careful selection of a mar is required (bottom middle panel).
weight models with different syper-parameters.

---

Under review as a conference paper at ICLR 2021
FFG-U, 0mux00.10
FFG-U, A.0000.03
FFG-U, Omax-0.1, A
FFG-U, Omax D0.00
FFG-U, 0010
Ensemble-U, A0.x=0.03
Ensembie-U
FFG-U, 0.300
ECE(%) accuracy(9g) %6)
Ensemble-U, A0.
ECE(%) accuracy(
accuracy(
90-
50
a0
85
85
0.63
0.1
6.3
0.1
0.3
16
32
64
128
ECE
20-
10-
10
10
Amax
Omax
M
Figure 5: Averaged CIFAR-10 accuracy (1) and ECE (4) results for the inducing weight methods
with different hyper-parameters. Models reported in the first two-columns uses M 128 for U
dimensions. For Amar (and maaz = 0) point estimates for the corresponding variables.
Ensemble-W CIFAR-10
FFG-W
FFG-U
Ensemble-U CIFAR-10
Accuracy
ECE(%)
80
20
60
10
40
2
3
4
CIFAR- 100
CIFAR-100
80
Accuracy(95
ECE(%)
30
60
20
40
10
20
Skew intensity
Skew intensity
Figure 6: Accuracy (f) and ECE (4) on corrupted CIFAR. We show the mean and two standard errors for each metric on the 19 perturbations provided in (Hendrycks & Dietterich, 2019).
4.3 MODEL ROBUSTNESS AND
DETECTION
To investigate the inducing weight's robustness to dataset shift, we compute predictions on corrupted CIFAR datasets (Hendrycks & Dietterich, 2019) after training on clean data. Figure 6 shows accu- racy and ECE results. Ensemble-W W is the most accurate model across skew intensities, while FFG- W. though performing well on clean data, returns the worst accuracy under perturbation. The induc- ing weight methods perform competitively to Ensemble-W, although FFG-U surprisingly maintains slightly higher accuracy on CIFAR- 100 than Ensemble-U despite being less accurate on the clean data. In terms of ECE, the inducing weight methods again perform competitively to Ensemble-W. with Ensemble-U sometimes being the best among the three. Interestingly, while the accuracy of FFG-W decays quickly as the data is perturbed more strongly, its ECE remains roughly constant.
We further present in Table the utility of the maximum predicted probability for out-of-distribuue (OOD) detection when presented with both the in in-distribution data (CIFAR10 10 and CIFAR100 test sets) and an OOD dataset (CIFAR 100/S and CIFAR VHN). The metrics are the area under the receiver operator characteristic (AUROC) and the area under the precision-recall curve (AUPR). Again Ensemble-I W performs the best in most settings, but more importantly, the inducing weight methods achieve very close results despite using the smallest number of parameters.
5 RELATED WORK
Parameter-efficient uncertainty quantification methods Recent research has proposed Gaus- sian posterior approximations for BNNs with efficient covariance structure (Ritter et al., 2018; Zhang et al., 2018b; Mishkin et al., 2018). The inducing weight approach differs from these in introducing structure via a hierarchical posterior with low-dimensional auxiliary variables. Another line of work reduces the memory overhead via efficient parameter sharing (Louizos & Welling, 2017; Wen al.,
8

---

Under review as a conference paper at ICLR 2021
Table 3: OOD detection metrics for Resnet- trained on CIFARI0/100.
In-dist.
CIFARIO
CIFAR100
OOD
CIFAR100
SVHN
CIFARIO
SVHI
Method / Metric
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
Deterministic-W
.87..00
.86+.00
.92+.01
.82..02
.73=.00
.762.00
.80+.00
.724.01
Ensemble-W
.89
.91
.95
.44
.77
.80
.85
.77
FFG-W
.872.00
.89
.89+.01
.862.0
.75=.00
.78+.00
.79+.02
.674.04
FFG-U
862..0
.88+.00
.90+.00
.87..01
.77=.00
.79+.00
.84..01
.764.01
Ensemble-U
.86=.00
.88+.00
.89+.01
.84+.02
.77=-00
.80+.00
.83+.00
.744.01
2020; Swiatkowski et al., 2020; Dusenberry al., 2020). They all maintain a "mean parameter" for the weights, making the memory footprint at least that of storing a deterministic neural network. Instead, approach shares parameters via the augmented prior with efficient low-rank structure, reducing the memory use compared to a deterministic network. In a similar spirit to our approach, Iz- mailov al. (2019) perform inference in d-dimensional sub-space obtained from PCA weights collected from an SGD trajectory. However, this approach does not leverage the layer-structure of neural networks and requires dx memory of single network.
Sparse GP and function-space inference As BNNs and GPs are closely related (Neal, 1995; Matthews et al., 2018: Lee et al., 2018), recent efforts have introduced GP-inspired techniques to BNNs (Ma al., 2019; Sun al., 2019; Khan al., 2019; Ober & Aitchison, 2020). Compared to weight-space inference, function space inference is appealing since its uncertainty is more directly relevant for many predictive uncertainty estimation tasks. While the inducing weight approach per- forms computations in weight-space, Section 3.3 establishes the connection to function-space pos- teriors. Our approach is related to sparse deep GP methods with Ue having similar interpretations as inducing outputs in e.g. Salimbeni & Deisenroth (2017). The major difference is that U lies in low-dimensional space, projected from the pre-activation output space of a network layer.
Priors on neural network weights Hierarchical priors for weights has also been explored (Louizos et al., 2017; Krueger al., 2017; Atanov et al., 2019; Ghosh et al., 2019; Karaletsos & Bui, 2020). However, we emphasise that F(W,U) is a pseudo prior that is constructed to assist posterior inference rather than to improve model design. Indeed, parameters associated with the inducing weights are optimisable for improving posterior approximations. Our approach can be adapted to other priors, e.g. for a Horseshoe prior p(0, v) = p(Ol))p(v) = (0; C+) 1), the pseudo prior can be defined as 5(0, v, a) p a) p(())((())))))) such that f p(0pv, () p(0(v). In general, pseudo priors have found broader success in Bayesian computation (Carlin & Chib, 1995).
CONCLUSION
We have proposed parameter-efficient uncertainty quantification framework for neural networks. augments each of the network layer weights with a small matrix inducing weight, and by ex- tending Matheron's 's rule matrix-normal related distributions, maintains relatively small run-time overhead as compared with ensemble methods. Critically, experiments on prediction and uncertainty estimation tasks demonstrate the competence of the inducing weight methods to the state-of-the-an while reducing the parameter count to less than half of a deterministic ResNet-18.
Several directions are be explored the future. First, modelling correlations across layers might further improve the inference quality. We outline an initial approach leveraging inducing variables in Appendix E. Second, based on the function- sepccion-space interpretation of inducing weights, better ini- tialisation techniques can be inspired from the sparse GP and dimension reduction literature. Lastly, the small run-time overhead of our approach can mitigated by better design of the inducing weight structure well as vectorisation techniques amenable to parallelised computation.
REFERENCES
Felix V Agakov and David Barber. An auxiliary variational method. In International Conference on Neural Information Processing, pp. 561-566. Springer, 2004.

---

Under review as a conference paper at ICLR 2021
Andrei Atanov, Arsenii Ashukha, Kirill Struminsky, Dmitriy Vetrov, and Max Welling, The deep weight prior. In International Conference on Learning Representations, 2019.
Eli Bingham, Jonathan P Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D Goodman. Pyro: Deep universal probabiliistic programming. The Journal of Machine Learning Research, 20(1):973-978, 2019.
Charles Blundell, Julien Corebiiee, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning, pp. 1613-1622, 2015.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.144 14165, 2020.
Bradley P Carlin and Siddhartha Chib. Bayesian model choice via markov chain monte carlo meth- ods. Journal of the Royal Statistical Society: Series B (Methodological), 57(3):473-48, 1995.
Arnaud Doucet. A note efficient conditional simulation of gaussian distributions. Technical report, University of British Columbia, 2010.
Michael W Dusenberry, Ghassen Jerfel, Yeming Wen, Yi-an Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable Bayesian neural nets with rank-1 factors. In Proceedings of the 37th International Conference on International Conference on Machine Learning, pp. 9823-9833, 2020.
Andrew YK Foong, Yingzhen Li, José Miguel Hemándezz-ooatto and Richard E Turner. 'in- between' uncertainty in Bayesian neural networks. arXiv preprint arXiv: 1537, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. international conference on machine learning, pp. 1050-1059, 2016.
Soumya Ghosh, Jiayu Yao, and Finale Doshi- Model selection in Bayesian neural networks via horseshoe priors. Journal of Machine Learning Research, 20(182):1 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330, 2017.
Arjun K Gupta and Daya K Nagar. Matrix variate distributions, volume 104. CRC Press, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016b.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint
2019.
Matthew D Hoffman and Andrew Gelman. The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593-1623 2014.
Matthew D Hoffman, David M Blei. Chong Wang. and John Paisley. Stochastic variational infer- ence. The Journal of Machine Learning Research, 14(1):1303-1444 2013.
Yehuda Hoffman and Erez Ribak. Constrained realizations of gaussian fields-a simple algorithm. The Astrophysical Journal, 380:L5-L8, 1991.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
10

---

Under review as a conference paper at ICLR 2021
Pavel Izmailov, Wesley Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and An- drew Gordon Wilson. Subspace inference for Bayesian deep learning. Uncertainty in Artificial Intelligence, 2019.
Michael Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-233, 1999.
Andre Journe and Charles Huijbregts. Mining geostatistiss. Academic press London, 1978.
Theofanis Karaletsos and Thang D Bui. Hierarchical gaussian process priors for Bayesian neural network weights. arXiv preprint arXiv:2002.04033, 2020.
Alex Kendall and Yarin Gal. What uncertainties do need in Bayesian deep learning for computer vision? In Advances in neural information processing systems, pp. 5574-5584, 2017.
Mohammad Emtiyaz E Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa. Approxi- mate inference turns deep networks into gaussian processes. In Advances in neural information processing systems, pp. -004,,104, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.699 69800 2014.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 and cifar-100 datasets. https://www. cs. toronto. edu/kriz/cifar. html, 6:1, 2009.
URI:
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint 171.00479, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. In Advances neural information processing systems, pp. 6402 6413, 2017.
Jachoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep neural networks as gaussian processes. In International Conference on Learning Representations, 2018.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 201222112200 22277, 2017.
Christos Louizos, Karen Ullrich, and Max Welling. Bayesian compression for deep learning. In Advances in neural information processing systems, pp. 3288 -298, 2017.
Chao Ma, Yingzhen Li, and José Miguel Kernándezz-Loato Variational implicit processes. In International Conference on Machine Learning, pp. 4222 4233, 2019.
David JC MacKay. Bayesian neural networks and density networks. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 354(1):73-80) 1995.
Alexander G.
Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahra- mani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018.
Aaron Mishkin, Frederik Kunstner, Didrik Nielsen, Mark Schmidt, and Mohammad Emtiyaz Khan. Slang: Fast structured covariance approximations for Bayesian deep learning with natural gradi- ent. In Advances in Neural Information Processing Systems, pp.
2018.
Radford M Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1995.
Sebastian W. Ober and Laurence Aitchison. Global inducing point variational posteriors for Bayesian neural networks and deep gaussian processes. arXiv preprint arXiv:2005.00 148, 2020.

---

Under review as a conference paper at ICLR 2021
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dil- lon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's 's uncertainty? eval- uating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pp. 13991-14002, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Deiito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Rajesh Ranganath, Dustin Tran, and David Blei. Hicrarchical variational models. In Proceedings of the 33rd International Conference on Machine Learning, pp. 324-333, 2016.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In International Conference on Learning Representations, 2018.
Tim Salimans, Diederik Kingma, and Max Welling. Markov chain Monte Carlo and variational inference: Bridging the gap. In Proceedings of the 32nd International Conference Machine Learning, pp. 1218-1226, 2015.
Hugh Salimbeni and Mare Deisenroth. Doubly stochastic variational inference for deep gaussian processes. In Advances in Neural Information Processing Systems, pp. 4588-4599, 2017.
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In Advances in neural information processing systems, pp. 1257-1264, 2006.
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse Functional variational Bayesian
neural networks. In International Conference Learning Representations, 2019.
Jakub Swiatkowski, Kevin Roth, Bastiaan S Veeling, Linh Tran, Joshua V Dillon, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. The k-tied normal distribution: A compact parameterization of gaussian mean field posteriors in Bayesian neural networks. Proceedings of the 37th International Conference on International Conference on Machine Learning, pp. 6631-66411 2020.
Ryutaro Tanno, Daniel E Worrall, Aurobrata Ghosh, Enrico Kaden, Stamatios N Sotiropoulos, An- tonio Criminisi, and Daniel Alexander. Bayesian image quality transfer with cnns: exploring uncertainty in dmri super-resolution. In International Conference on Medical Image Computing and Computer-Assiste Assisted Intervention, pp. 611-619 Springer, 2017.
Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Mothodology), 61()):11-622) 1999.
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes, In Artificial Intelligence and Statistics, pp. 567-574, 2009.
Michalis Titsias and Miguel Lázaro-Gredilll. Doubly stochastic variational bayes for non-conjugate inference. In International conference on machine learning, pp. 1971-1979, 2014.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference Learning Representations, 2020.
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. BERT post-training for review reading comprehension and aspect-based sentiment analysis. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). Association for Computational Linguistics, 2019.
Cheng Zhang, Judith Bütepage, Hedvig Kjellström, and Stephan Mandt. Advances in variational inference. IEEE transactions on pattern analysis and machine intelligence, 41 41(8):2008-226 2018a.
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. International Conference on Machine Learning, pp. 1852-5861, 2018b.
Ruqi Zhang, Chumyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for Bayesian deep learning. In International Conference on Learning Representations, 2020.
12


Reviews:
[{"review_id": "tHp6Xlovbwi", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n# summary\n\nThis paper proposed a method on uncertainty estimation in deep neural\nnetworks. Compared with BNN and deep ensemble, the proposed approach in this\nwork has a storage advantage. Furthermore, this work provides a better\ntrade-off between accuracy and calibration.\n\n\n# pros\n\n1.  The approach in this work is quite interesting. The idea of augmenting\n    weights with auxiliary low-dimensional latent variables in a deep neural\n    network seems natural at first sight, but this approach is novel as far as\n    my knowledge is concerned. Although VI with local latent variables is an\n    old technique, this application in deep neural network is novel.\n2.  The authors also proposed an efficient approach that can sample from the\n    variational approximation conditioning on the latent variable. Since the\n    original weight is large, such a sampling is necessary.\n3.  This paper provides extensive empirical results and sufficient theoretical\n    results. Experimental results show the proposed approach achieve a good\n    balance between accuracy, calibration and memory requirements.\n\n\n# cons\n\n1.  My major concern is all experiments are conducted on ResNet18, and this is\n    not a typical choice for practical problems. I think experiments on other\n    larger net such as ResNet56 on CIFAR10 will make this paper more\n    convincing.\n2.  It is not clear to me how the authors choose the \"mixture of delta\n    measures\", i.e. how to choose U<sup>k</sup>? Can the authors comment on this? In\n    this case, q(U) is a categorical distribution. It seems better if  q(U)\n    also depends on input data, however, the authors choose to use a fixed\n    q(U). Can the authors comment on this choice?\n\nOverall speaking, the idea and the presentation of this work are great in my\nopinion.\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "n4DQA_680-M", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Bayesian deep learning attempts to incorporate uncertainty estimation in the modern neural network models. However, common approaches such as Bayesian neural networks and Deep Ensembles incur large memory overhead because of the increased parameter sizes. Motivated by the inducing points approach in the sparse variational Gaussian Processes area, this paper proposes the Gaussian variational posterior relying on the compact *inducing weights* $U$ and the conditional distribution $p(W|U)$. Because the variational parameter shifts from $W$ to the smaller $U$, the resulting model potentially uses even fewer parameters than a deterministic counterpart. In general, I think the paper proposes an interesting approach that could help addressing the storage issues of current Bayesian deep learning models. \n\n**Novelty**\nThe proposed approach is both novel and elegant. Given the high dimensionality of the parameters, many BNN approaches consider structured covariance for the variational posterior, which involves the balancing between computational costs and approximating capacity. And these approaches all maintain a \"mean parameter\", thus the memory costs are at least as large as the original network. In comparison, this paper turns to an augmented space using the joint Gaussian distribution, and instead model the variational posterior for the compact *inducing weights*. In consequence, the resulting model could potentially have even fewer parameters than a deterministic counterpart, while being able to conduct uncertainty quantification. Given that the inducing points approach has achieved big successes for scalable GPs, I think the proposed method could be impactful for uncertainty modelling in deep learning models. \n\n**Experiments**\nThis paper conducts experiments covering both image classification and out-of-distribution detection. Empirically, their model has increased calibration compared to the deterministic network while using $\\leq 47.9\\%$ of parameters. However, as shown in Figure 4, the proposed approach incurs large computational overheads especially when using small number of samples. \n\n**Questions**\n1. What are the inducing weights for the conv layer? Are they matrices of shape [128, 128, kern_size, kern_size]?\n2. The Ensemble-U approach uses a dirac measure $\\sum_k \\delta_{U^k_l}$ in each layer and drops the KL penalty. Are dirac measures of different layers independent, or like deep ensemble, the particles are tied into $K$ disjoint groups across layers? If it was the former, I would reckon that the particles of the same layer would converge to the same place; if it was the latter, it should be made clearer in the paper.\n3. How does the sample size $K$ influence the performance? In practice, I think only a small number of $K$ should be used.\n4. A hyperparameter $\\sigma_{max}^2$ is set for the variance of $q(W)$. And the similar hyperparameter $\\lambda_{\\max}$ is set for the variational posterior. Especially for $\\lambda_{max}$, increasing it from $0.1$ to $0.3$ observes large performance drop. Is this an issue due to the expressiveness of the proposed posterior ? \n\n**Clarity** \nIn spite of my questions regarding the model details, the paper well presents its main methods. Besides, I think Sec3.3 is worthy of more polishing, since it looks confusing when you start by studying $U_c$ instead of $U_r$ or $U$. And a few typos to be corrected, 1. Abstract: whichenable 2. Last paragraph of introduction: our approach achieve 3) Related works: function-space inference is appealing to ... ", "extra_scores": {}, "extra_text": {}}, {"review_id": "KUPFW1L8T_E", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis work proposes a specific parametrisation for the Gaussian prior and approximate posterior distribution in variational Bayesian neural networks in terms of inducing weights. The general idea is an instance of the sparse variational inference scheme for GPs proposed by Titsias back in 2009; for a given model with a prior p(W) perform variational inference on an extended model with a hierarchical prior p(U) p(W | U), that has the same marginal p(W) = \\int p(U)p(W | U)dU as the original model. The authors then consider “U” to be auxiliary weights that are jointly Gaussian with the actual weights “W” and then use the decomposition p(W|U)p(U), q(W|U)q(U) for the prior and approximate posterior (which can easily be computed via the conditional Gaussian rules). Furthermore, they “tie” (almost) all of the parameters between q(W|U) and p(W|U) (similarly to Titsias, 2009). The main benefit from these two things is that since the mean and covariance of the Gaussian distribution over W conditioned on U can be efficiently represented as functions of U, whenever dim(U) << dim(W) we get reductions in memory for storing the distributions over the parameters in the network. The authors furthermore, discuss how to efficiently parametrize the joint distribution over W, U, discuss different choices for q(U) (that can lead to either traditional VI or something like deep ensembles). In addition, they also discuss how more efficient sampling from q(W|U) can be realised via an extension of the Matheron’s rule to the case of matrix random variables. Finally, they evaluate their method against traditional mean field variational Bayesian neural networks and deep ensembles on several tasks that include regression, classification, calibration and OOD performance. \n\n### Pros\n- The method provides a novel way to induce parameter efficiency in variational bayesian neural networks\n- It connects to the sparse GP literature and the trick seems to be general enough, in that it can be applied to any distribution that admits some parametrisation in terms of a Gaussian random variable.\n- The extension of the Matheron’s rule can be of independent interest\n- Extensive set of experimental tasks, with a nice ablation study for lambda_max and sigma_max\n\n### Cons\n- Comparison against alternative approaches that induce parameter efficiency are missing\n- Results are mixed and sometimes not very convincing\n\n\n### Recommendation \nWhile I find the main idea very interesting and the presentation of it relatively clear, I unfortunately cannot recommend acceptance of this work as is. The main motivation behind improving parameter efficiency can also be performed with other, perhaps much simpler, ways and comparisons against such approaches is missing. Furthermore, the results, at least in their current state, are a bit mixed and thus not very convincing. \n\n### Detailed feedback\nOverall, this work is interesting and relatively easy to follow. Using inducing variables in neural networks in not a new concept, as it has been used before in, e.g., [1, 2], but the motivation of using them as a means towards reducing the number of parameters of each distribution is new. The authors explain the main idea behind them in a clear manner. Furthermore, they clearly explain the need for using matrix normal distributions, explaining how the Kronecker product factorisation of the covariance improves the parametrization efficiency, along with how efficient sampling from q(W|U) can be performed. In addition, performing parameter efficient deep ensembles in U space is a nice bonus of this formulation. Section 3.3 however is dense and can be a bit hard to follow (whereas Appendix D is clearer). I would therefore suggest that the authors briefly describe the main idea in a couple of sentences and refer the readers to Appendix D instead.\n\nMy main point for feedback is for the experimental section. The authors argue at the beginning of section 4 that “the goal is to demonstrate competitive performance …. While remaining computationally efficient.” Is that claim for the training or evaluation phases? From what I see, during the training phase, the inducing weights framework is not faster or more efficient than the baselines. The main advantage of the inducing weight is instead memory reduction in terms of parameters which translates to “memory efficiency” and not “computational “efficiency”. If then memory efficiency is the main target, I believe that some reasonable baselines are missing. For the case of a variational Bayesian neural network a simple baseline that performs pruning post-hoc (e.g., the one presented at the FFG-W paper or the one from [3]) in order to reduce the parameter count; remove weights by either setting them to exact zero or equal to the prior and then use the variational posterior for those that survive. It would be interesting to see how such an approach would fare not just on accuracy, but also on the ECE and OOD. Similarly for deep ensembles, a baseline where you perform pruning (e.g., simple magnitude based) would also serve as a better baseline for the Ensemble-U.  Both of these baselines would provide a more complete picture and would be a better signal in understanding whether the inducing weights framework is a better choice overall.\n\nAs for other points\n\n- At figure 3 you only show FCG-U and not the FFG-U which is used in all of the other experiments. How does the uncertainty look like with FFG-U? \n- Sometimes, both FFG-U and Ensemble-U underperform compared to FFG-W and Ensemble-W and it would be good to know if it is due to less free parameters to optimise or due to the model definition itself. How does FFG-U perform when increasing dim(U) such that the parameter count is the same as FFG-W? Similarly for Ensemble-U vs Ensemble-W.\n- The second point at the end of page 2 only makes sense in hindsight (i.e., after one reads section 3). Perhaps the authors could expand on them a bit better so that it is self-contained (e.g., explain what you mean with “q(theta|alpha) can be efficiently parametrised”).\n\n\n[1] Structured and efficient variational deep learning with matrix Gaussian posteriors, C. Louizos & M. Welling, 2016\n[2] Global inducing point variational posteriors for Bayesian neural networks and deep gaussian processes, S. Ober & L. Aitchison, 2020\n[3]\u0010 Practical Variational Inference for Neural Networks, A. Graves, 2011\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KQIhPB1DX3s", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to define the weights approximate posterior of a NN with inducing variables.\n\nThe main benefit of the approach lies in the compactness of the description (Fig4, right). However, if memory size is the main issue, entropy should be considered instead of just a count of parameters. Moreover, if complexity and memory footprint is the main concern, MC dropout sounds like a reasonable alternative to Bayesian NN. Why do the authors do not compare their proposal to a conventional MC dropout approach?\n\nOverall, the contribution of the paper appears to be valid but relatively limited in scope compared to what exists in the literature (inducing variables are known, Bayesian NN are known, ensemble methods are known). What is the gain in knowledge brought by the paper? I might have missed a piece of the contribution, but the way the paper is written does not help. It is not self-contained (many previous works have to be read to follow the story), and remains relatively opaque to the non-expert. It lacks a clear and eye-bird picture of the approach (including both training and inference steps), to position and compare it to existing works.  \n\nThe overhead in optimizing (1) or (2), compared to training a deterministic NN, is not discussed. \n\nVariables d_in^l is used in Section 2, but only defined in Section 3. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The scores for this paper have been borderline, however the decision has been greatly facilitated by the participation of the authors and reviewers to the discussion and, more importantly, by active private discussion among reviewers and AC. Specifically, from the private discussion it seems that the reviewers find interesting ideas in this paper, but are overall are not entirely convinced about its significance, at least in the way the paper is currently positioned and motivated. 

More specifically, the reviewers found the main idea of using inducing weights interesting, both technically (e.g. associated sampling scheme) but also in terms of application (sparsity). The results are insightful from a theoretical perspective. That is, the inducing weights and their treatment does seem to result in interesting and potentially useful statistical properties for the model. On the other hand, it is important to note that the high-level idea of variational inducing weights, with usage of matrix normals in this setting, as well as connection to deep GPs has been studied before, as pointed out by R2 (refs [1,2]). Furthermore, even after discussions the motivation is still not entirely convincing, especially in conjunction with the experiments. Although various interesting ideas exist in the paper, both R2 and R3 in particular remain unconvinced about what is the main benefit (e.g.  pruning or runtime efficiency) stemming out of the proposed idea. Another reviewer agreed with this point in the private discussions.  Apart from overall clearer positioning of the paper, the claimed benefit would need to be supported by experiments tailored to illustrate this main point. The authors argued against some of the suggested comparisons (e.g. past pruning methods), and further discuss that there is no established experimental benchmark for the parameter efficiency of BNNs. I indeed sympathize with both of these arguments; however, I believe that if the reviewers' suggestions for extra experiments are rejected, it remains the responsibility of the authors to find a slightly different way of motivating their work and demonstrating its efficiency in some convincing, meaningful and more well-defined setting with the appropriate benchmarks.  

Decision: Reject