Sparse Uncertainty Representation in Deep Learning with Inducing Weights
Authors: Hippolyt Ritter
Martin Kukla
Cheng Zhang
Yingzhen Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian neural networks
uncertainty estimation

Abstract: 
Bayesian Neural Networks and deep ensembles represent two modern paradigms of uncertainty quantification in deep learning. Yet these approaches struggle to scale mainly due to memory inefficiency, requiring parameter storage several times that of their deterministic counterparts. To address this, we augment each weight matrix with a small inducing weight matrix, projecting the uncertainty quantification into a lower dimensional space. We further extend Matheron’s conditional Gaussian sampling rule to enable fast weight sampling, which enables our inference method to maintain reasonable run-time as compared with ensembles. Importantly, our approach achieves competitive performance to the state-of-the-art in prediction and uncertainty estimation tasks with fully connected neural networks and ResNets, while reducing the parameter size to $\leq 24.3\%$ of that of a single neural network.

Sparse Uncertainty Representation in Deep Learning with Inducing Weights
Hippolyt Ritter' MaMtti Kukla², Cheng Zhang? & Yingzhen Li
University College London Microsoff Research Cambridge, UK $Imperial College London j. ritterrcss ucl. ac ackuk, (Martin. Kukla, Cheng. Zhang) @microsoft. com, yingzhen 1 1i@imperial.....mm ac.uk
Abstract
Bayesian Neural Networks and deep ensembles represent two modern paradigms of uncertainty quantification in deep learning. Yet these approaches struggle to scale mainly due to memory inefficiency, requiring parameter storage several times that their deterministic counterparts. To address this, we augment each weight matrix with small inducing weight matrix, projecting the uncertainty quantification into a lower dimensional space. We further extend Matheron's conditional Gaussian sampling rule enable fast weight sampling, which enables inference method to maintain reasonable run-time as compared with ensembles. Importantly, our approach achieves competitive performance the state-of-the-art in prediction and uncertainty estimation tasks with fully connected neural networks and ResNets, while reducing the parameter size < 24.3% of that of single neural network.
Introduction
Deep learning models are becoming deeper and wider than ever before. From image recognition models such as ResNet-101 (He et al., 201 and DenseNet (Huang et al., 2017) to BERT (Xu et al., 2019) and GPT-3 (Brown et al., 2020) for language modelling, deep neural networks have found consistent success in fitting large-scale data. As these models are increasingly deployed in real-world applications, calibrated uncertainty estimates for their predictions become crucial, especially in safety-critical areas such as healthcare. In this regard, Bayesian Neural Networks (BNNs) (MacKay, 1995; Blundell et al., 2015; Gal Ghahramani, 2016; Zhang et al., 2020) and deep ensembles (Lakshminarayanan et al., 2017) represent two popular paradigms for estimating uncertainty, which have shown promising results in applications such as (medical) image processing (Kendall & Gal, 2017: Tanno al., 2017) and distr--distribution detection (Ovadia al., 2019).
Though progress has been made, one major obstacle to scaling up BNNs and deep ensembles is their high storage cost. Both approaches require the parameter counts be several times higher than their deterministic counterparts. Although recent efforts have improved memory efficiency (Louizos Welling, 2017; Swigtkowski et al., 2020; Wen al., 2020; Dusenberry al., 2020), these still use more parameters than deterministic neural network. This is particularly problematic in hardware-constraaine edge devices, when on-device storage required due to privacy regulations.
Meanwhile, an infinitely wide BNN becomes Gaussian process (GP) that is known for good uncertainty estimates (Neal, 1995; Matthews al., 2018: Lee et al., 2018). But perhaps surprisingly, this infinitely wide BNN is "parameter efficient", as its "parameters" are effectively the datapoints, which have considerably smaller memory footprint than explicitly storing the network weights. In addition, sparse posterior approximations store smaller number of inducing points instead (Snelson Ghahramani, 2006; Titsias, 2009), making sparse GPs even more memory efficient.
Work done Microsoft Research Cambridge.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Can we bring the advantages of sparse approximations in GPs which are infinitely-wide neural networks to finite width deep learning models? We provide an affirmative answer regarding memory efficiency, by proposing an uncertainty quantification framework based on sparse uncertainty representations. We present our approach in BNN context, but the proposed approach is also applicable to deep ensembles. In detail, our contributions are as follows:
We introduce inducing weights - an auxiliary variable method with lower dimensional counterparts the actual weight matrices for variational inference in BNNs, well as memory efficient parameterisation and an extension to ensemble methods (Section 3.1).
We extend Matheron's 's rule to facilitate efficient posterior sampling (Section 3.2).
We provide an in-depth computation complexity analysis (Section 3.3), showing the signifi- cant advantage in terms of parameter efficiency.
We show the connection to sparse (deep) GPs, in that inducing weights can be viewed as projected noisy inducing outputs in pre- rreaactivation output space (Section 5.1).
We apply the proposed approach BNNs and deep ensembles. Experiments in classification, model robustness and out-of-distributicn detection tasks show that our inducing weight approaches achieve competitive performance their counterparts in the original weight space on modern deep architectures for image classification, while reducing the parameter count to S 24.3% of that of single network.
We open-source our proposed inducing weight approach, together with baseline methods reported in the experiments, as PyTorch (Paszke et al., 2019) wrapper named bayesianize https: //g thub. wttp://cooo/oooo////////o/cccott wwt.cooooaoeeossiiieeeeeeee /bwp//ayyeiiieee..eeeaa As demonstrated in Appendix I, our software makes the conversion of a deterministic neural network to Bayesian one with a few lines of code:
import bnn our pytorch wrapper package net = torchvision models.resneeee bnn. bayesianizee (net, inference peerrence=iinduiin "inducing") # convert into a Bayesian one
# construct deterministic ResNet18
2 Inducing variables for variational inference
Our work is built on variational inference and inducing variables for posterior approximations. Given observations D (X, YJ with X = [21,.... NN Y [y1- .... yN we would like to fit neural network p(y|ær, (x, WI-L) with weights W WIZL to the data. BNNs posit a prior distribution p( (W Wizz) over the weights, and construct an approximate posterior q( 1:L) the exact posterior p( 1:2 Wi-l/D) (D) x
p(D] W pp( W...), where (DD[WI.L) X WI.L) = I p(ym W.L.).
Variational inference (Hinton & Van Camp, 1993; Jordan et al., 1999; Zhang et al., 2018a) constructs an approximation (()) to the posterior D) x (D()) by maximising variational lower-bound
Variational inference
log 2 L(((O)) == Eg [logr(D(0)] KL (((O)l(()))
For BNNs, O- = [Wi.)) and simple choice of q is Fully-factorized Gaussian (FFG): W (WI.))==l II' II (V(, oi ofth)), with mi (t.f) 0z
(6.5) and d' d'out the respective number of inputs and outputs to layer l. The variational parameters are then & = (m, (Hoffman et al., 2013) and with Monte Carlo sampling from the q distribution (Titsias Lázaro- Gredilla, 2014; Kingma& & Welling, 2014). By setting to an BNN, variational BNN can be trained with similar computational requirements as a deterministic network (Blundell et al., 2015).
(i-1) the mean and variance of w/ (k.))
(4,)
(i.)) 1 3
ME- Gradients of L w.r.t & can be estimated with
of data
Improved posterior approximation with inducing variables Auxiliary variable approaches (Agakov & Barber, 2004; Salimans et al., 2015; Ranganath et al., 2016) construct the q()) dis- tribution with an auxiliary variable a: q()) = f q(0 ())((a))))))) da, with the hope that potentially richer mixture distribution (() can achieve better approximations. As then (0) becomes intractable, an auxiliary variational lower-bourd is used to optimise q(0, a) (see Appendix B):
log (a))r(a() (((a)aa))
log 2 L(q(O, a)) E [log (D + EE Eg)0,.)
(2)

---

Here r(a(0) is an auxiliary distribution that needs to be specified, where existing approaches often use "reverse model" for r(a)@). Instead, we define r(a ) in a generative manner: r(al0) is the "posterior" of the following "generative model", whose "evidence" is exactly the prior of A:
r(a ((a x ((a)p(Ola) such that p(0) == / p(a))(0)a)da p(0).
(3)
Plugging Eq. (3) into Eq. (2):
[(a(O, a)) = Ey( (log) By(a)) (a)] (LLga())iE))
(4)
This approach yields efficient approximate inference algorithm, translating the complexity of inference in 0 to a. If dim(a) < dim(O) and g(0, a) - q(0 ()) has the following properties:
1. A "pseudo prior" p(a)p(0) (0)a) ( is defined such that f (((a)) (la))da p(();
2. The conditionals (0)aa) a) and p(0 a) are in the same parametric family, so can share parameters;
3. Both sampling 0 ~ (0) and computing [gT (0 a))l
(0) a)] can done efficiently;
4. The designs of q(a) and p(a) can potentially provide extra advantages (in time and space complexities and/or optimisation easiness).
We call a the inducing variable of 0, which is inspired by variationally sparse GP (SVGP) with inducing points (Snelson & Ghahramani, 2006; Titsias, 2009). Indeed SVGP is special case (see Appendix C): = f, a u, the GP prior is X) = GP((, Kxx), p(u) = GP Kzz), p(f, u) = (f)), u), (ff]u) = p(f u), qf, u) pff ()) and Z are the optimisable inducing inputs. The variational lower-bound is L(g(f, u}) = (F) [log p( Y ({]])]] - [LL[g())]pp and the variational parameters are Φ = {Z, distribution parameters of (u)). SVGP satisfies the marginalisation constraint Eq. (3) by definition, and it has (f) small M = dim(u) and exploiting the distribution design, SVGP reduces run-time from (NN)) to O(NM2 + M where N is the number of inputs in X, meanwhile it also makes storing full Gaussian q(u) affordable. Lastly, u can be whitened, leading to the "pseudo prior" p((,, v)
0. Also by using
p(f]X, K 1/2 v))vv, p(v) (v; I) which could bring potential benefits in optimisation.
We emphasise that the introduction of "pseudo prior" does not change the probabilistic model as long as the marginalisation constraint Eq. (3) is satisfied. In the rest of the paper assume the constraint Eq. (3) holds and write p(0, a) := p(0, a). It might seem unclear how to design such p(0, a) for an arbitrary probabilistic model, however, for Gaussian prior 0 the rules for computing conditional Gaussian distributions can be used to construct p. In Section we exploit these rules to develop an efficient approximate inference method for Bayesian neural networks with inducing weights.
3 Sparse uncertainty representation with inducing weights
3.1 Inducing weights for neural network parameters
Following the above design principles, we introduce to each network layer smaller inducing weight matrix U; to assist approximate posterior inference in W.. Therefore in our context, 0 = W 1. and U1. In the rest of the paper, assume a factorised prior across layers p( p(W V.L) ((W)) and drop the / indices when the context is clear to ease notation.
Augmenting network layers with inducing weights Suppose the weight W E Rdou xdi.. has a Gaussian prior p( () = p(vec( (W)) = (O, I²1) where vec (W) concatenates the columns of the weight matrix into vector. A first attempt to augment p( vec(W)) with an inducing e Mout x XMn may be to construct a multivariate Gaussian p(vec(W) vec( such that ʃ p( vec W), vee(U)) ) (vec(W) vec U)), ), it requires the block corresponding to the covariance of vec (W) to match the prior covariance I²I. I. We are then free to parameterise the rest of the entries in the joint covariance matrix, as long as this full matrix remains positive definite. Now the conditional distribution p( (WUU) is function of these parameters, and the conditional sampling from p( PW)UU) is further discussed in Appendix D.I. Unfortunately. as dim dimevee([W) W)) is typically large (e.g. of the order of 10"). using a full covariance Gaussian for p(vec( (W), vec(U)) becomes computationally intractable.
weight variable
O²I This means for the joint covariance matrix of
We address this issue with matrix normal distributions (Gupta Nagar, 2018). The prior ((vec(W)) (N has equivalent matrix normal distribution form as p( p(W)
3

---

oN with or. > 0 the row and column standard deviations satisfying o o, Now we introduce the inducing variable in matrix space, as well as two auxiliary variables U, E Rouu Xdon, Uc E Rd XMi, so that the full augmented prior is:
W U
U U
p W. U, U) = MN Er, En),
(5)
0,I 0
0
with
L, =
s.t. E, = -,, =
Z
Dr
Z Z + D²
0ol
0²ol
0 7.
and Le Ze
D.) s.t. Σ. = LoL
o.Ze
Z. + +
See Fig. I(a) for a visualisation of the augmentation. Matrix normal distributions have similar marginalisation and conditioning rules as multivariaae Gaussian distributions, for which we provide further examples in Appendix D.2. Therefore the marginalisation constraint Eq. (3) is satisfied for any Ze E R Ma xdi., Z, E TR. Moue Xdout and diagonal matrices De, D,.. For the inducing weight U we have p(U) = (M L .. with V, Z, + D and V. Z. + D In the experiments we use whitened inducing weights which transforms U so that p(U) I. (Appendix H), but for clarity we continue with the above formulas in the main text.
The matrix normal parameterisation introduces two additional variables U., U. without providing additional expressiveness. Hence it is desirable to integrate them out, leading to a joint multivariate normal with Khatri-Ras Rao product structure for the covariance:
p(veo( W), UU) -N(") ve)( (..2..000 (0.00.2
1'18 0F
O. V
0.
(6)
As the dominating memory complexity here is O(dott + Min) which comes from storing Z, and Ze. we see that the matrix normal parameterisation of the augmented prior is memory efficient.
Posterior approximation in the joint space We construct a factorised posterior approximation across the layers: q( (WI.. Ul... = II a( ((W,)U,))((U)) Below we discuss options for q (W[U). The simplest option is g( (WJU)- = p( (W) (U) (wwiu Ewjo), similar to sparse GPs. slightly more flexible variant adds a rescaling term X to the covariance matrix, which allows efficient KL computation (Appendix E):
<( U)) = ((veo (W) ve)( = N (,
X EwE
R(A)= = KIL [g( W UJ P W U)] = din doue (0.5x - log À - 0.5).
(8)
Plugging W W.I,,C. a = U.. and Eq. (8) into Eq. (4) returns the following variational lower-bound (R(A) +
C( W I.L, )l..)) = ( ..)) Pog (l WI.)) - _
([L(((U))))>(U))) (9)
with X the associated scaling parameter for q( ((W() Again as the choices of Ze: Z,, Des D, do not change the marginal prior p( (W), are safe to optimise them as well. Therefore the variational parameters are now @ = (Ze, Z,., De, Dr, X, dist. params. of {U)) for each layer.
Two choices of (U) A simple choice is FFG g(veo(U) = Am, diag( vw)), which performs mean-field inference in space (c.f. Blundell et al., 2015), and here KL(q ()) (U) has a closed- form solution. Another choice is a "mixture of delta measures" ((U) = Σx J(U = U(1),,
ie. we keep K distinct sets of parameters (U((A) k- in inducing space that are projected back into the original parameter space via the shared conditionals g( [W,) U,) to obtain the weights. This approach can be viewed as constructing "deep ensembles" in U space, and we follow ensemble in Eq. (9).
methods (e.g. Lakshminarayanan et al., 2017) to drop (L()
Often U chosen to have significantly lower dimensions than W, i.e. Min << den and Mout << dout- As qi (U) and p( ( U) only differ in the covariance scaling constant, U can be regarded as a sparse representation of uncertainty for the network layer, as the major updates in (approximate) posterior belief is quantified by q(U).

---

Matheron's rule (for vectors)
Extended Matheron's rule (far matrices)
(ours)
(4) inducing weight
mugmentation
W
X(
Figure 1:
Visualisation of (a) the inducing weight augmentation. and compare (b) the original Matheron's s rule to (c) our extended version. White blocks represent samples from the joint Gaussian.
3.2 Efficient sampling with extended Matheron's rule
Computing the variational lower-bound Eq. (9) requires samples from o (W), which requires an efficient sampling procedure for o( UW) Unfortunately, q( (( derived from Eq. (6) & Eq. (7) is not a matrix normal, so direct sampling is prohibitive. To address this challenge, we extend Matheron's 's rule (Journel Huijbregts. 1978; Hoffman Ribak, 1991; Doucet, 2010) efficiently sample from q( (W))), with derivations provided in Appendix F.
The original Matheron's rule applies to multivariate Gaussian distributions. As running example, consider two vector-valued random variables w, u with joint distribution p( w. u) = N(O0 E). Then the conditional distribution p(w) u)) - (HHureee /ur Ewjjul (u)) is also Gaussian, and direct sampling from it requires decomposing the conditional covariance matrix Ewiu which can be costly. The main idea of Matheron's 's rule is that can transform sample from the joint Gaussian obtain sample from the conditional distribution p( (ww2)) as follows:
Ewew
Eur
w i Ewn Euu( i W. N(O, E), E
(10)
Σ Euw
Euu
One can check the validity of Matheron's rule by computing the mean and variance of w above:
E a [w] Σ Ewn Eu Mwluu V [w] Eww
Ewn Eui Enw Ewju
It might seem counter-intuitive at first sight in that this rule requires samples from a higher dimensional space. However, in the case where decomposition/nnveriioo of Σ and Enu can be done efficiently, sampling from the joint Gaussian p( w. u) can be significantly cheaper than directly sampling from the conditional Gaussian ( w.).. This happens e.g. when Σ directly parameterised by its Cholesky decomposition and dim ()) << dim so that sampling W. ū ~ (0, Σ) is straight- and computing Eii. is significantly cheaper than decomposing Emjnn
Unfortunately, the original Matheron's rule cannot be applied directly sample from g( (W) This because q U) = q(veo( W)] vec( ()) differs from p(vec(W) vee(U) only in the variance scaling A, and for p(vec( (W)l vec(U)), its joint distribution counter-part Eq. (6) does not have an efficient representation for the covariance matrix. Therefore a naive application of Matheron's 's rule requires decomposing the covariance matrix of ((vee(W), vec which is even than direct conditional sampling. However, notice that for the joint distribution p( (W, U... U,., U) in higher dimensional space, the and column covariance matrices Σ, and E, parameterised by their Cholesky decompositions, so that sampling from this joint distribution can be done efficiently. This inspire to extend the original Matheron's 's rule for efficient sampling from q( (W[UU) (details Appendix F, when À = it also applies to sampling from p( (W|U))
expensive
W = AW + o oZ + (U AU) -Z:: W. ~ p W. U. U,, Ū) = MN E,, E.). (11)
Here W. ~ p W. U., U,, U) means we first sample W, U... U, Ú from the joint then drop U., U.: in fact U., U, never computed, and the other samples W, U can be obtained by:
W EE,, U Z.E, Z. L,E.D. D,E D,E,D
E, ~ MN Inmm I: Ez, Es. E. MN IMsaes I M,.)); L, chol Z, L. chol Z.T.)
(12)
The run-time cost is O(2 + +2M3 + dout Mout oott-o Min + Min dou required by inverting V,,, V... computing L,, L.. and the matrix products. The extended Matheron's 's rule is visualised in Fig. 1
5

---

Table Computational complexity per layer. We assume W E RR xdin, U E RoMa x M,, and K forward passes for each of the N inputs. (" uses parallel computing friendly vectorisation technique (Wen et al., 2020) for further speed-up.)
Method
Time complexity
Storage complexity
Deterministic- W
O(Ndin
Oddindout)
FFG-W Ensemble- W
K O dond
((2dsndous) O(Kdindows (ddnndout + dhn douu)) O Oldin + -(ddi + dout) Oldin + 2((in + dout)
O (N K dont O (N K dont O (N K Kdinde dout)
Matrix-ormmal- W k-tied FFG-W W
rank-1 1 BNN
K (INNdnnOou
FFG-U
O(N + 2M + 2M out K MouttMout Min + Moudoud same as above
O Oden Min + dout Mout 22 Mons)
Ensemble-U
(ddin Min + dana Mout + K M Mone)
with comparison to the original Matheron's 's rule for sampling from g(vec (W) vec(U)). Note that the original rule requires joint sampling from Eq. (6) (i.e. sampling the white blocks in Fig. 1(b) which has ((doutdan + MoutMMn )) cost. Therefore our recipe avoids inverting and multiplying big matrices, resulting in a significant speed-up for conditional sampling.
3.3 Computational complexities
In Table we report the complexity figures for two types of inducing weight approaches: FFG (U) (FFG-U) and Delta mixture q (U) (Ensemble-U). Baseline approaches include: Deterministic- W, variational inference with FFG & (W) (FFG-W, Blundell et al., 2015), deep ensemble in W (Ensemble-W, Lakshminarayanan et al., 2017), well parameter efficient approaches such as matrix-normal g( (W) (Matrix-normall-V W, Louizos & Welling (2017)), variational inference with k-tied FFG q( (W) (k-tied FFG-W, Swiętkowski et al. (2020)), and rank-1 BNN (Dusenberry et al., 2020). The gain in memory is significant for the inducing weight approaches, in fact with Min < din and Mout < dout the parameter storage requirement is smaller than a single deterministic neural network. The major overhead in run-time comes from the extended Matheron's 's rule for sampling q( (W[U). Some of the computations there are performed only once, and in our experiments we show that by using a relatively -ow-dimeniional U and large batch-sizes, the overhead is acceptable.
4 Experiments
We evaluate the inducing weight approaches on regression, classification and related uncertainty estimation tasks. The goal is to demonstrate competitive performance to popular P W-space uncertainty estimation methods while using significantly fewer parameters. We acknowledge that existing parameter efficient approaches for uncertainty estimation (e.g. k-tied or rank-1 BNNs) have achieved close performance to deep ensembles. However, none of them reduces the parameter count to be smaller than that of a single network. Therefore we decide not to include these baselines and instead focus on comparing: (1) variational inference with FFG o( (W) (FFG-W, Blundell et al., 2015) v.s. FFG (U)) (FFG-U, ours); (2) deep ensemble in W space (Ensemble- W, Lakshminarayanan et al., 2017) v.s. ensemble in U space (Ensemble- U, ours). Another baseline is training deterministic neural network with maximum likelibood. Details and additional results are in Appendices and K.
4.1 Synthetic 1-D regression
The regression task follows Foong et al. (2019), which has two input clusters I1 ~ 24 [1,, T2 ~ 2 [0.5, 1]. and targets y y~ NCoss((4x 4x + 0.8), 0.01). For reference we show the exact posterior results using the NUTS sampler (Hoffman & Gelman, 2014). The results are visualised in Fig. 2 with predictive mean in blue, and up to three standard deviations as shaded area. Similar to historical results, FFG-W W fails to represent the increased uncertainty away from the data and in between clusters. While underestimating predictive uncertainty overall, FFG-U shows a small increase in predictive uncertainty away from the data. In contrast, a per-layer Full-covariance Gaussian (FCG) in both weight (FGG-W) and inducing space (FCG-U) as well as Ensemble- better capture the increased predictive variance, although the mean function is more similar to that of FFG-W.

---

W
M
W
W
W
M
(a) FFG-U
(b) FCG-U
(c) Ensemble-U
(d) FFG-W
(e) FCG-W
(1)
NUTS
Figure 2: Toy regression results, with observations red dots and the ground truth function in black.
Table 2: CIFAR in-distribution metrics (in %).
Speed
relative param count
Model size (R50)
relative run time
FFGUU R181
FF5
CIFARIO
CIFAR 100
FFG.U (M-128, R591
FFG (M-64, R18)
Method
Acc. 1
ECE
Ace.
ECE
FRG (M-64 R501
determiisttic
FFG-W
PFG-U
Deterministic
94.72
4.46
75.73
19.69
FFG (850)
94.7%
Ensemble-W W
95.90
1.08
79.33
6.51
FFG-W
94.13
0.50
74.44
4.24
32
32.01
150
250
FFG-U
94.40
0.61
75.37
2.29
Number of samples
M
Ensemble-U
94.94
0.45
75.97
1.12
Figure 3: Resnet run-times & model sizes.
4.2
Classification and in-distribution calibration
As the core empirical evaluation, we train Resnet-50 models (He et al., 2016b) on CIFAR- 10 and CIFAR-10 1-0 (Krizhevsky et al., 2009). To avoid underfitting issues with EFG-W, a useful trick is to set an upper limit mee on the variance of ()) (Louizos & Welling, 2017). This trick is similarly applied to the U-space methods, where
cap À < Amaz for g (W) and for FFG-U we also set o² maz for the variance of (UU). In convolution layers, we treat the 4D weight tensor W of shape (Cout: Cin: h, w) as a Cout X Cin hur matrix. We use U matrices of shape 64 x 64 for all layers (i.e. M = Min = Mout = 64), except that for CIFAR-10 we set Mout = 10 for the last layer.
In Table 2 we report test accuracy and test expected calibration error (ECE) (Guo et al., 2017) as first evaluation of the uncertainty estimates. Overall, Ensemble-Ve V achieves the highest accuracy, but is not well cell-calibrtted variational methods. For the inducing weight approaches, Ensemble- U outperforms FFG-U U on both datasets; overall it performs the best on the more challenging CIFAR- 100 dataset toneembbb- W accuracy and lowest ECE). Tables 5 and 6 in Appendix K. show that increasing the U dimensions to M = 128 improves accuracy but leads to slightly worse calibration.
In Fig.3 3 we show prediction run-times for batch-size = 500 on an NVIDIA Tesla V100 GPU, relative to those of an ensemble of deterministic nets, as well as relative parameter sizes to a single ResNet-50. The extra run-times for the inducing methods come from computing the extended Matheron's 's rule. However, as they can be calculated once and cached for drawing multiple samples, the overhead reduces to small factor when using larger number of samples K, especially for the bigger Resnet-50. More importantly, when compared to a deterministic 50,, the inducing weight models reduce the parameter count by over 75% (5, 710, 902 vs. 23, 520, 842) for M = 64.
paraamt choices We visualise in Fig. 4 the accuracy and ECE results for computation- ally lighter inducing weight ResNet-18 models with different hyper-paramedt (see Appendix J).
FFG-U, m0.00
FFG-U.Ar.=00.0
FFG-U, 00.30
FFG-U, Arca -0.10
Ensemble-U, A..0
Ensemble-U
FFG-U, Arw =0.30
Ensemble-U, U, A0.
ECE(%) (ccuracy(%))
accuracy
ECE(%) accuracy
80
30
80
85
85
0.03
0.1
0.3
01
16
64
125
ECE(%)
20-
Amax
Omax
M
Figure 4: Ablation study: average CIFAR-10 accuracy (t) and ECE ()) for the inducing weight
methods on ResNet-18. In the first two columns M 128 for U dimensions. For Xmaas Tmax we use point estimates for U, W respectively.
7

---

Ensemble-W
FFG-W
FFC-U
Ensemble-U
CIFARLO
CIFAR100
BCE(S) Accuracy
50
40
20
Skew intensity
Skew intersity
Figure 5: two emrs. for Acct and ECE,, on corrupted CIFAR (Hendrycks & Dietterich, 2019).
Table 3: OOD detection metrics for Resnet-50 trained on CIFARI0/100.
In-dist > OOD
C10
C100
C10 - SVHN
C100 + C10
C100
SVHN
Method / Metric
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
AUROC
AUPR
Deterministic
.842.00
.80
.93..1
.85..01
.742.00
.742.00
.81..1
.72+.02
Ensemble-W W
.89
.89
.95
.92
.78
.79
.86
.78
FFG-W
.884.00
.90+.00
.90..01
.64..11
.76..00
.79+.00
.80..01
.694.01
FFG-U
.892.00
.91..00
.44.01
.911.01
.772.00
.79=.00
.834.01
.74+.01
Ensemble-U
.90+.00
.914.00
.93+.00
.91
.774.00
.79#.00
.824.01
.72+t.02
Performance in both metrics improves as the U matrix size M is increased (right-most panels), and the results for M 64 and M 128 are fairly similar. Also setting proper values for Amaz: maa is key to the improved results. The left-most panels show that with fixed maa values (or Ensemble- U), the preferred conditional variance cap values Amas are fairly small (but still larger than which corresponds to point estimate for W given U). For man which controls variance in U space, see from the top middle panel that the accuracy metric is fairly robust to maa as long as Amaz not too large. But for ECE, careful selection of omar is required (bottom middle panel).
4.3 Robustness, out-of-distribution detection and pruning
To investigate the models' robustness to distribution shift, we compute predictions on corrupted CIFAR datasets (Hendrycks & Dietterich, 2019) after training clean data. Fig. shows accuracy and ECE results for the ResNet-50 models. Ensemble- is the most accurate model across skew intensities, while FFG-W, though performing well on clean data, returns the worst accuracy under perturbation. The inducing weight methods perform competitively to Ensemble- W with Ensemble-U being slightly more accurate than FFG-U as on the clean data. For ECE, FFG-U outperforms Ensemble- U and Ensemble- W, which are similarly calibrated. Interestingly, while the accuracy of FFG-W decays quickly as the data is perturbed more strongly, its ECE remains roughly constant.
Table further presents the utility of the maximum predicted probability for -iitt---iittiittionn (OOD) detection. The metries the under the receiver operator characteristic (AUROC) and the precision-recall curve (AUPR). The inducing- methods perform similarly to Ensemble-W W; all three outperform FFG-W and deterministic networks across the board.
Parameter pruning We further investigate pruning as pragmatic alternative for more parameter-efficient inference. For FFG-U, we prune entries of the Z matrices, which contribute the largest number of parameters to the inducing methods, with the smallest magnitude. For FFG-W of W to depending on their variational mean-to- -ova--n-varincee ratio and repeat the previous experiments after fine-tuning the distributions the remaining variables. We stress that, unlike FFG-U, UU the FFG- W pruning corresponds
FFG-U
FFG-W
75
ECE (%) Accuracy
follow Graves (2011) in setting different fractions
65
12
poct-hoc change of the probabilistic model and no longer performs inference the original weight-space.
10
200
Parameter size (% det.)
For FFG-W, pruning 90% of the parameters (leaving 20% of pa- rameters as compared to deterministic counterpart) worsens the ECE, in particular CIFARI00 Fig. 6. Further pruning to 1%
Figure 6: CIFAR100 pruning accuracy(1) & ECE()). Right- most points are w/out pruning.

---

worsens the accuracy and the OOD detection results as well, On the other hand, pruning 50% of the Z matrices for FFG-U reduces the parameter count to 13.2% of deterministic net, the cost of only slightly worse calibration. See Appendix K for the full results.
Discussions
5.1 A function-space perspective on inducing weights
Although the inducing weight approach performs approximate inference in weight space, we present in Appendix G a function-space inference perspective of the proposed method, showing its deep connections to sparse GPs. Our analysis considers the function-space behaviour of each network layer's 's output and discusses the corresponding interpretations of the U variables and parameters.
The interpretations are visualised in Fig. 7. Similar to sparse GPs, in each layer, the Ze parameters can be viewed as the (transposed) in- ducing input locations which lie in the same space as the layer's input. The U. variables can also be viewed as the corresponding (noisy) inducing outputs that lie in the pre-activation space. Given that the output dimension dout can still be high (e.g. 1000 in a fully con- nected layer), our approach performs further dimension reduction in a similar spirit as probabilistic PCA (Tipping & Bishop, 1999), nower-dimensiones
h
X
ReLU
X
6
U
2
which projects the column vectors of U. space. This returns the inducing weight variables U, and the projec- tion parameters are {Z,,. }. Combining the two steps, it means the column vectors of U can be viewed as collecting the "noisy projected inducing outputs" whose corresponding "inducing inputs" are row vectors of Z (see the red bars in Fig. 7).
naisy
noisy
proj.
proj
2
Figure 7: Visualising vari- ables in pre-activation spaces.
In Appendix G we further derive the resulting variational objective from the function-space view, which is almost identical to Eq. (9), except for scaling coefficients on the R(Ar) terms to account for the change in dimensionality from weight space to function space. This result nicely connects posterior inference in weight- and function-space
5.2 Related work
Parameter-efficieet uncertainty quantification methods Recent research has proposed Gaussian posterior approximations for BNNs with efficient covariance structure (Ritter et al., 2018; Zhang et al., 8b; Mishkin et al., 2018). The inducing weight approach differs from these introducing structure via hierarchical posterior with low-dimensional auxiliary variables. Another line of work reduces the memory overhead via efficient parameter sharing (Louizos & Welling, 2017; Wen et al., 2020; Swiętkowski et al., 2020; Dusenberry et al., 2020). The third category of work considers a hybrid approach, where only selective part of the neural network receives Bayesian treatments, and the other weights remain deterministic (Bradshaw et al., 2017; Daxberger et al., 2021). However, both types of approaches maintain "mean parameter" for the weights, making the memory footprint at least that of storing a deterministic neural network. Instead, our approach shares parameters via the augmented prior with efficient low-rank structure, reducing the memory use compared to a deterministic network. In a similar spirit, Izmailov et al. (2019) perform inference in a d-dimensional sub-space obtained from PCA on weights collected from an SGD trajectory. But this approach does not leverage the styer-structure of neural networks and requires d' memory of single network.
Network pruning in uncertainty estimation context There is a large amount of existing research advocating network pruning approaches for parameter-efficient deep learning, e.g. see Han al. (2016); Frankle & Carbin (2018); 8); Lee et al. (2019). In this regard, mnean-field VI approaches have also shown success in network pruning, but only in terms of maintaining a minimum accuracy level (Graves, 2011; Louizos et al., 2017; Havasi et al., 2019). To the best of our knowledge, our empirical study presents the first evaluation for VI-based pruning methods maintaining uncertainty estimation quality. Deng et al. (2019) considers pruning BNNs with stochastic gradient Langevin dynamics (Welling Teh, 2011) as the inference engine. The inducing weight approach is orthogonal to these BNN pruning approaches, it leaves the prior the network parameters intact, while the pruning

---

approaches correspond to post-hoc change of the probabilistic model to using sparse weight prior. Indeed our parameter pruning experiments showed that our approach can be combined with network pruning to achieve further parameter efficiency improvements.
Sparse GP and function-space inference As BNNs and GPs are closely related (Neal, 1995; Matthews et al., 2018; Lee et al., 2018), recent efforts have introduced GP-inspired techniques to BNNs (Ma al., 2019; Sun et al., 2019; Khan al., 2019; Ober & Aitchison, 2021). Compared to weight-space inference, function-space inference is appealing as its uncertainty is more directly relevant for predictive uncertainty estimation. While the inducing weight approach performs com- putations in weight-space, Section 5.1 establishes the connection to function- -pacc posteriors. Our approach is related to sparse deep GP methods with U. having similar interpretations as inducing out- puts in e.g. Salimbeni & Deisenroth (2017). The major difference is that U lies in a low-dimensional space, projected from the pre-activation output space of network layer.
The original Matheron's 's rule (Journel & Huijbregts, 1978; Hoffman & Ribak, 1991; Doucet, 2010) for sampling from conditional multivariate Gaussian distributions has recently been applied to speed-up sparse GP inference (Wilson et al.. 2020, 2021). As explained in Section 3.2, direct application of the original rule to sampling W conditioned on U still incurs prohibitive cost as p(vec( W), vec(U) does not have a convenient factorisation form. Our extended Matheron's 's rule addresses this issue by exploiting the efficient factorisation structure of the joint matrix normal distribution p(W, Ue. U,. U), reducing the dominating factor of computation cost from cubic (d di (.)) )) to linear (n)) We expect this new rule to be useful for a wide range of models/applications beyond BNNs, such as matrix-variate Gaussian processes (Stegle et al., 201 1).
Priors neural network weights Hierarchical priors for weights has also been explored (Louizos et al., 2017; Krueger et al., 2017: 7017;A Atanov et al., 2019; Ghosh et al., 2019; Karaletsos & Bui, 2020). However, we emphasise that p( W, ,U) is a pseudo prior that is constructed assist posterior inference rather than to improve model design. Indeed, parameters associated with the inducing weights are optimisable for improving posterior approximations. Our approach can be adapted to other priors, c.g. for a Horseshoe prior p(0, v) = p(0))))(v) = N((: 0, (²) (1:; 0, 1), the pseudo prior can be defined as p(0, 1/, a) p(0)v, p a ())(()(()) such that I (00|,, p(a)da p(0lz). In general, pseudo priors have found broader success in Bayesian computation (Carlin & Chib, 1995).
6 Conclusion
We have proposed parfmeter-efficient uncertainty quantification framework for neural networks. It augments each of the network layer weights with a small matrix of inducing weights, and by extending Matheron's 's rule matrix-normal related distributions, maintains relatively small run-time overhead compared with ensemble methods. Critically, experiments on prediction and uncertainty estimation tasks show the competence of the inducing weight methods to the state-of-the-art, while reducing the parameter count to under a quarter of a deterministic ResNet-50 before pruning. This represents a significant improvement over prior Bayesian and deep ensemble techniques, which so far have not managed to go below this threshold despite various attempts of matching it closely.
Several directions are to be explored in the future. First, modelling correlations across layers might further improve the inference quality. We outline an initial approach leveraging inducing variables in Appendix H. Second, based on the function-space interpretation of inducing weights, better initialisation techniques can be inspired from the sparse GP and dimension reduction literature. Similarly, this interpretation might suggest other innovative pruning approaches for the inducing weight method, thereby achieving further memory savings. Lastly, the run-time overhead of our approach can be mitigated by better design of the inducing weight structure as well as vectorisation techniques amenable to parallelised computation. Designing hardware-specine implementations of the inducing weight approach also a viable alternative for such purposes.
References
Agakov, F. V. and Barber, D. An auxiliary variational method. In ICONIP, 2004.
Atanov, A., Ashukha, A., Struminsky, K., Vetrov, D., and Welling, M. The deep weight prior. In ICLR, 2019.
10

---

Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P.. Horsfall, P., and Goodman, N. D. Pyro: Deep universal probabilistic programming. JMLR, 2019.
Blundell, C., Comeeiiee J., Kavukenoglu, K., and Wierstra, D. Weight uncertainty in neural networks. In ICML, 2015.
Bradshaw, J., Matthews, A. G. d. G., and Ghahramani, Z. Adversarial examples, uncertainty, and trans- fer testing robustness in Gaussian process hybrid deep networks. arXiv preprint arXiv: 024476 2017.
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., al. Language models are few-shot learners. NeurlPS, 2020.
Carlin, P. and Chib, Bayesian model choice via Markov chain Monte Carlo methods. JRSS B, 1995.
Daxberger, E., Nalisnick, E., Allingham, J. U., Antorán, J., and Hernández-Lobato, J. M. Expressive yet tractable Bayesian deep learning via subnetwork inference. In ICMI,, 2021.
Deng, W., Zhang, X., Liang, F., and Lin, G. An adaptive empirical bayesian method for sparse deep learning. NeurIPS, 2019.
Doucet, A. A note on efficient conditional simulation of Gaussian distributions. Technical report, University of British Columbia, 2010.
Dusenberry, M. W., Jerfel, G., Wen, Y., Ma, Y-a.. Snock, J., Heller, K., Lakshminarayanan, B., and Tran, D. Efficient and scalable Bayesian neural nets with rank-1 factors. In ICML, 2020.
Foong, A. Y., Li, Y., Hernández-Lobato, J. M., and Turner, R. E. 'In-between' uncertainty in Bayesian neural networks. arXiv preprint arXiv: 11377, 2019.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICLR, 2018.
Gal, Y. and Ghahramani, Z. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
Ghosh, S., Yao, J., and Doshi-Velez, F. Model selection Bayesian neural networks via horseshoe priors. JMLR, 2019.
Graves, Practical variational inference for neural networks. In NeurIPS, 2011.
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. calibration of modern neural networks. In ICML, 2017.
Gupta, A. K. and Nagar, D. K. Matrix variate distributions. CRC Press, 2018.
Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffmaan coding. In ICLR, 2016.
Havasi, M., Peharz, R., and Hemándezz-obatto J. M. Minimal random code learning: Getting bits back from compressed model parameters. In ICLRR 2019.
He, K., Zhang, X., Ren, S., and Sun, Deep residual learning for image recognition. CVPR, 2016a.
He, K., Zhang, X., Ren, S., and Sun, Identity mappings in deep residual networks. In ECCV, 2016b.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
Hinton, G. E. and Van Camp, D. Keeping the neural networks simple by minimizing the description length of the weights. In COLT, 1993.

---

Hoffman, M. D. and Gelman, A. The No-U-Turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. JMLR, 2014.
Hoffman, M. D., Blei, D. M., Wang, C., and Paisley, Stochastic variational inference. JMLR, 2013. Hoffman, Y. and Ribak, E. Constrained realizations of Gaussian fields-a simple algorithm. Apl, 1991.
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. Densely connected convolutional networks. In CVPR, 2017.
Izmailov, P., Maddox, W. J., Kirichenko, P., Garipov, T., Vetrov, D., and Wilson, A. G. Subspace inference for Bayesian deep learning. UAI, 2019.
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. An introduction to variational methods for graphical models. Machine Learning, 1999.
Journel, A. G. and Huijbregts, C. Mining geostatistics. Academic press London, 1978.
Karaletsos, T. and Bui, T. D. Hierarchical Gaussian process priors for Bayesian neural network weights. In NeurlPS, 2020.
Kendall, A. and Gal, What uncertainties do we need in Bayesian deep learning for computer vision? In NeurIPS, 2017.
Khan, M. R. E., Immer, A., Abedi, E.,, and Korzepa, M. Approximate inference turns deep networks into Gaussian processes. In NeurlPS, 2019.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR, 2015.
Kingma, D. P. and Welling, M. Auto-encoding variational Bayes. In ICLR, 2014.
Krizhevsky, A., Nair, V., and Hinton, G. CIFAR-10 and CIFAR-100 datasets. URL: https:///ww. cs. toronto. edu/kriz/cffar html, 2009.
Krueger, D., Huang, C.-W., Islam, R., Turner, R., Lacoste, A., and Courville, A. Bayesian hypernet- works. arXiv preprint arXiv: 100::1710..0 10.04759, 2017.
Iakshminarayannn, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.
Lee, J., Sohl-Dickstein, J., Pennington, J., Novak, R., Schoenholz, S., and Bahri, Y. Deep neural networks Gaussian processes. In ICLR, 2018.
Lee, N., Ajanthan, T., and Torr, SNIP: Single-shot network pruning based on connection sensitivity. In ICLR, 2019. 2019.
Leibfried, F., Dutordoir, V., John, S., and Durrande, N. A tutorial on sparse Gaussian processes and variational inference. arXiv preprint arXiv:2012. 13962, 2020.
Louizos, C. and Welling, M. Multiplicative normalizing flows for variational Bayesian neural networks. ICML, 2017.
Louizos, C., Ulirich, K., and Welling, M. Bayesian compression for deep learning. In NeurIPS, 2017. Ma, C., Li, Y., and Hermández-Lobato, J. M. Variational implicit processes. In ICMI., 2019.
MacKay, D. J. Bayesian neural networks and density networks. NIMPR A, 1995.
Matthews, A. G. d. G., Hron, J., Rowland, M., Turner, R. E., and Ghahramani, Gaussian process behaviour in wide deep neural networks. In ICLR, 2018.
Mishkin, A., Kunstner, F., Nielsen, D., Schmidt, M., and Khan, M. SLANG: Fast structured covariance approximations for Bayesian deep learning with natural gradient. In NeurIPS, 2018.
Neal, R. M. Bayesian Learning for Neural Newworks. PhD thesis, University of Toronto, 1995.
12

---

Ober, S. W. and Aitchison, L. Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes. In ICML, 2021.
Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., and Snoek, J. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In NeurlPS, 2019.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, Lin, Z., Gimelshein, N., Antiga, L.,, et al. PyTorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
Ranganath, R., Tran, D., and Blei, D. Hierarchical variational models. In ICML., 2016.
Ritter, H., Botev, A., and Barber, D. A scalable Laplace approximation for neural networks. In ICLR, 2018.
Salimans, T., Kingma, D., and Welling, M. Markov chain Monte Carlo and variational inference: Bridging the gap. In ICML, 2015.
Salimbeni, H. and Deisenroth, M. Doubly stochastic variational inference for deep Gaussian processes. In NeurIPS, 2017.
Snelson, and Ghahramani, Z. Sparse Gaussian processes using pseudo-inputs. In NeurIPS, 2006.
Stegle, O., Lippert, C., Mooij. J. M., Lawrence, N. D., and Borgwardt, K. Efficient inference in matrix-variate gaussian models with iid observation noise. In NeurlPS, 2011.
Sun, S., Zhang, G., Shi, J., and Grosse, R. Functional variational Bayesian neural networks. In ICLR, 2019.
Swiętkowski, J., Roth, K., Veeling, B. S., Tran, L., Dillon, J. V., Mandt, S., Snoek, J., Salimans, T., Jenatton, R., and Nowozin, S. The k-tied Normal distribution: compact parameterization of Gaussian mean field posteriors in Bayesian neural networks. In JCML, 2020.
Tanno, R., Worrall, D. E., Ghosh, A., Kaden, E., Sotiropoulos, S. N., Criminisi, A., and Alexander, D. C. Bayesian image quality transfer with CNNs: exploring uncertainty in dMRI super-resolution. In MICCAI, 2017.
Tipping, M. E. and Bishop, C. M. Probabilistic principal component analysis. JRSS B, 1999.
Titsias, M. Variational learning of inducing variables in sparse Gaussian processes. In AISTATS, 2009.
Titsias, M. and Lázaro-Gredilla, M. Doubly stochastic variational Bayes for non-conjugate inference. In ICML, 2014.
Welling, M. and Teh, Y. W. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
Wen, Y., Tran, D., and Ba, J. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In ICLR, 2020.
Wilson, J. T., Borovitskiy, V., Terenin, A., Mostowsky, P., and Deisenroth, M. Efficiently sampling functions from Gaussian process posteriors. In ICML, 2020.
Wilson, J. T., Borovitskiy, V., Terenin, A., Mostowsky, P., and Deisenroth, M. Pathwise conditioning of Gaussian processes. JMLR, 2021.
Xu, H., Liu, B., Shu, L.,, and Yu, P. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In ACL, 2019.
Zhang, C., Bütepage, J., Kjellström, H., and Mandt, S. Advances in variational inference. TPAMI, 2018a.
Zhang, G., Sun, S., Duvenaud, D., and Grosse, R. Noisy natural gradient as variational inference. In ICML., 2018b.
Zhang, R., Li, C., Zhang, J., Chen, C., and Wilson, A. G. Cyclical stochastic gradient MCMC for Bayesian deep learning. In ICLR, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] (Our work
considers reducing model parameter sizes for Bayesian neural networks.)
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] (this is for the corrupted CIFAR dataset (Hendrycks Dietterich, 2019).)
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "YKuFhosHgS", "reviewer": "Reviewer_6QzC", "summary": "This paper proposes a method for reducing the storage cost of BNNs. The idea is to apply the inducing point method and fast Matheron's sampling rule, commonly used in GPs, to standard variational BNNs. Experiments show that the resulting BNNs stay competitive to standard BNNs while enjoying up to 75% reduction in parameter size.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\nThis paper is generally well written, though not without blemishes (see below). The method is solid and can be seen as a \"natural\" counterpart of the standard inducing weight method in function space (GPs) for weight space (BNNs).\n\n\n**Cons:**\n\n* While the proposed method significantly reduces the storage cost of BNNs, it introduces a large overhead during prediction. E.g. the proposed BNNs are around twice as slow as the vanilla counterparts for the commonly-used number of MC samples (e.g. 10, see Tab. 4). This makes me wonder whether the trade-off is worth it at all, considering ResNet-50 (the architecture that the authors use) for ImageNet only requires just under 100MB of storage space. In contrast, prediction latency has a real-world, tangible effect in any real-time predictive system, e.g. in self-driving cars. I hope to see a discussion from the authors during the rebuttal period.\n\n* The paper can be hard to read at times. I think this is because of how dense the presentation of the paper is. Given how dense this paper is already, it is thus not good to see that the authors still tried to squeeze every little bit of space by using e.g. text style symbols (e.g. \\textstyle\\sum) in display equations. I think it would be much better presentation-wise if the authors move some least important sections to the appendix, and then be more judicious with whitespace.\n\n\n**Minor comments/suggestions:**\n\n* Please *do not* just copy-and-paste bibtex entries from Google Scholar. Many references are either inaccurate or inconsistent. For instance, Hendrycks & Dietterich, Kingma & Ba are in ICLR; \"A tutorial on sparse gaussian processes ...\" -> \"A tutorial on sparse **G**aussian processes\"; etc.\n* L.36, L.41: please be consistent with the em dashes.\n* Starting from L.57: The usage of the boldface in math should be consistent. The matrices $\\mathbf{X}, \\mathbf{Y}$ are in bold, but $W_l$ among other quantities are not.\n* L.60: Hinton and Van Camp, 1993 should be cited.\n* L.65: Missing iterates over $i, j$ in the definition of $\\phi$\n* L.78: \"This approach returns ...\" -> \"This approach yields ...\"\n* L.78-79: Break the sentence up\n* L.81: \"... so can share parameters\" what does it mean exactly?\n* L.87-88: Too dense with the inline math, please use display math\n* L.94-95: Inline math hard to read. Consider adding some words in between the two $\\widetilde{p}(\\mathbf{v})$'s\n* L.134-135: More efficient compared to what complexity?\n* Fig. 1: Use $\\left( ... \\right)$, esp. for the middle term consisting of green and white vertical bars\n* L.175: \"... the non-linearity ...\" -> \"... a non-linearity ...\"\n* L.184: What does \"d\" on top of \\sim mean?\n* Fig. 3: Need Ensemble-W result\n\n\n---\n\n**Post-rebuttal**\n\nI increased my score. Nevertheless, I still think that the presentation of the paper needs to be improved:\n\n* The application of parameter-efficient-but-slower BNNs should be discussed more explicitly in the introduction. This will help motivate the method more. Also, it is instructive to show the parameter efficiency of the method on larger networks (that take more than 100MB of storage).\n* The exposition of the method should be improved more---there should be a balance between information content and density.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I might have missed it, but I could not find any explicit discussion regarding the limitations of the method. This is especially crucial due to the first point in my \"Cons\" above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "XClxb8W6j1-", "reviewer": "Reviewer_8V4M", "summary": "The paper proposes a parameter-efficient method for uncertanity quantification in deep neural network using per layer inducing weights. The paper develops a network weight sampling scheme based on inducing weights. The proposed method employs Matheron's rule for efficient sampling. The method requires fewer parameters than even deterministic networks, while keeping good calibration and performance, at the cost of longer compute times, when few MC weight samples are used.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\nThe paper proposes a novel idea (best to my knowledge) of using inducing points known from sparse variational GPs in the context of Bayesian neural networks.\nThe paper adequately cites and describes differences with related work.\n\nQuality\n\nSubmission is technically sound and claims are well supported through detailed derivations and exhaustive experimentation. \nMethods used are appropriate and the work looks complete.\nThe paper is careful and honest about evaluating the strengths (smaller number of parameters) and weakness (larger computational cost) of the proposed method as well as comparison to the baseline alternatives (weight pruning).\n\nIt would be interesting to see the training times additionally to the predictions times. As far as I understand from appendix G.2, during training the method uses 1 MC sample, and this might be the regime where the method is much less effective (>10x slower). \n\nI am not sure whether the full weight matrix W gets materiallized during training of the FFG-U method. \nIf this is the case, does it diminish all the gains in terms of memory requirement during training / testing? \n\nThe paper is very math heavy and while most of the calculations looks good to me, I might have missed some errors. \n\nClarity\n\nThe submission is clearly written and well organised. It adequately informs the reader.\nThe paper refers to complex mathematical objects and procedures (e..g Matheron's sampling, Matrix Normal closed form derivations), but aims to inform the reader through references and even explaining the basics of these concepts.\n\nSignificance\n\nThe paper proposes an interesting method for reducing the number of network parameters by a large factor. \nHowever, it increases the computational cost during testing and very likely also during training. \nTherefore, I expect that this method might find applicability in areas which require low memory and calibrated predictions, but the prediction time is not as important.\nNevertheless, from the theoretical standpoint, the paper develops an interesting connection between variational inference BNNs and sparse variational GPs.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "11"}, "extra_text": {}}, {"review_id": "NWkBmz4N6sn", "reviewer": "Reviewer_tCFn", "summary": "The paper proposes to represent uncertainty in neural networks by augmenting neural network layers with inducing weights and then assuming that the resulting vector of weights is drawn from a matrix normal distribution which is structured so that it has fewer parameters than the original network. The parameters of the distribution are learned via variational inference and the authors present a procedure to sample efficiently from this distribution for optimizing the model parameters and for inference at test time. Experiments on classification and out of distribution detection are presented to validate the approach.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Quality and Clarity**\n\nThe overall idea is clear but quite heavy on notation and hence may require multiple readings to process entirely. I would recommend adding a table of symbols either in the body of the paper or in the appendix to enable readers to keep track of the notation. I would also suggest modifying Figure 1 to illustrate how the method is memory efficient since that is the key selling point of this work.\n\n**Originality and Significance**\n\nWhile this is one of many approaches to representing uncertainty in deep learning, the ability to effectively represent uncertainty without increasing, and indeed sometimes decreasing, storage cost is certainly significant. My only concern is that the method appears quite sensitive to hyperparameter choices, especially from the discusiion in Section 4.2 (Lines 260-268) and associated plots and without any guidance on how to choose the hyperparameters, it might be difficult to translate the success of the method to datasets other than the ones considered in the paper.\n\n**Strengths**\n\n1. A new memory efficent approach and associated sampling procedure is proposed for representing uncertainties in deep learning.\n2. Experiments on a range of task validate the efficacy of the approach for both prediction and out-of-distribution detection.\n\n**Weaknesses**\n\n1. The writing is a bit heavy on notation and the details may be hard to parse in one reading.\n2. The approach appears quite sensitive to hyperparameters and it is not clear how to set the hyperparameters for new datasets.\n\n**Queries and Suggestions**\n\n1. Is it possible to give some guidance on choosing the hyperparameters $\\sigma_{\\max}$, $\\lambda_{\\max}$, $M$ in general or is an ablation study the only way to figure it out? If it is the latter then I would recommend highlighting that fact a bit more clearly in the paper and providing any insights you may have on reasonable ranges over which to search, based on your experience with the experiments. Exploring the efficacy of AutoML in finding the optimal hyperparameters might also be an interesting option.\n\n2. Why does pruning FFG-W to 2% of the size of the deterministic network give the lowest ECE in Figure 7? That appears to be violating the observed trend quite significantly and is probably worth investigating or at least providing some, even speculative, thoughts on.\n\n3. I would suggest including some comparison with other space efficient approaches for uncertainty estimation like rank-1 BNNs. Even if they are not as space efficient as this approach it would be instructive to see if the extra space they require gives any performance improvement on the accuracy/ECE metrics and if not then those approaches can be convincingly ruled out in favor of this one.\n\n4. I also did not notice any wall-clock time numbers for the different methods. It would be beneficial to have them since it does appear that the space savings come at the cost of slightly higher time complexity. While that is not a big issue, it would be good to see how much longer the method takes on average to get a better sense of the tradeoff.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have provided suggestions on addressing the limitations in the review above. I believe there is no potential negative societal impact of this work.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "GW1osFEmh85", "reviewer": "Reviewer_pv3L", "summary": "This paper proposes a layer-wise variational inference scheme for Bayesian neural networks that uses \"inducing weights\" for each layer before up-sampling to the full (layer-wise) weight distribution. The scheme is equivalent to a hierarchical prior formulation with an analogous hierarchical variational posterior. Experiments are performed on the standard BDL benchmarks of synthetic one dimensional regression, ResNets on CIFAR-100, and out of distribution detection.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "tldr: This method has a lot of promise, but the methodological writing is close to impenetrable currently, so I tend to reject it. Another round of revision to clean up the experiments and focus the writing presentation would help this paper have impact.\n\n## Originality\n\n*Are the tasks or methods new?* Yes, they seem to be a new version of VI for BNNs.\n\n*Is the work a novel combination of well-known techniques?* Yes, I mostly followed along the Matheron's rule derivation, which is a nifty idea to reduce the computational complexity of prior sampling over the conditional distribution from the weights and the inducing weights.\n\n*Is it clear how this work differs from previous contributions?* Yes, I think so. It bears a bit more resemblance to the GP style hierarchical priors and posteriors of [Karaletsos & Bui, NeurIPS, '20](https://arxiv.org/abs/2002.04033) than is mentioned in the paper. Variational matrix normal posteriors are also pretty closely related and probably deserve a bit of discussion as well (even though you do compare to them, which is helpful) [Louizos & Welling, ICML, '16](https://arxiv.org/pdf/1603.04733.pdf).\n\n*Is related work adequately cited?* \n- See above comment plus also the probabilistic meta-hypernetwork approach of [Karaletsos et al, '18](https://arxiv.org/abs/1810.00555)\n- Hypernetworks and their Bayesian counterparts are also relevant as hypernetworks can reduce the parameter count of standard networks significantly, while Bayesian hyper-networks should contain some of the same advantages of this work (better uncertainty quantification, etc...) [Ha et al, '16](https://arxiv.org/abs/1609.09106) and [Krueger et al, '17](https://arxiv.org/abs/1710.04759).\n\n## Quality\n\n*Is the submission technically sound?* Mostly so, a few comments:\n- line 148: from my understanding the \"mixture of delta posterior\" is not theoretically justified due to support mismatch in KL space, which isn't an issue at all if you're not going for the bayesian interpretation (just all it ensembling), but could prove problematic if you're trying to train through that KL.\n    - Thus, is it okay to drop the KL term when training?\n\n- line 631 (but throughout): My understanding for sampling this joint would be to use Kronecker algebra to have the cholesky factors of the covariance (both defined under eq 5) and then to compute the MVM $(L_r \\otimes L_c)z$ where $z$ is a standard gaussian random vector (before reshaping into a matrix). But, what you're taking is instead a non-square matrix vector product and so it's really tough to parse if that's the correct set of algebra.\n    - Also, from the Kronecker approach, I think you can just write down the algebra and simplify to get out $\\bar W$ and $\\bar U$ (although I haven't had the chance to do so), which would simplify the presentation a good deal.\n    - Eq 31 (and following): I don't follow the shapes of these matrices any more because of how the matrices are being written down, as well as how these matrix matrix products are being written down.\n\n- line 638: \"we can reduce variance ...\" this statement concerns me, are you actually doing the full Matheron's rule here? or some sort of variance reduction method? How does this play into the variance based proofs in the following lines?\n\n- line 640: why do we need to take the choleskies of these matrices? my understanding is that given any $AA' = \\Sigma$ we can use $A'z$ to produce valid samples. \n\n- eq 36: it's a bit of a misnomer to call this a GP because its just a linear kernel (and this could be easily derived by marginalizing linear regression).\n\n- 3.2 and 3.3: which version is actually implemented?\n    - Do the variables $U_c$ play a role in the Matheron's rule based approach?\n    - If not, are they in the implementation anywhere or are they just an object of theoretical interest due to their connection to the deep GP literature?\n    - line 305: the scale of the KL term is probably important, and suggests that there's probably a missing constant or something there. After all, the scale determines if you're doing proper Bayesian inference or a generalized version (which isn't practically important, just you should be clear).\n\n*Are claims well supported (e.g., by theoretical analysis or experimental results)?* \n- The experimental work demonstrates that the hierarchical VI scheme does outperform SVI, but it definitely doesn't seem to outperform either ensembling approach or really a deterministic NN, so the claims of \"significant improvement\" are not really well supported.\n\n*Are the methods used appropriate?* \n- I'm not a fan of the comparisons and think that some other reduced space parameterization technique really ought to be compared to (even if the memory overheads are somewhat higher). \n    - From what I can tell, MIMO networks [Havasi et al, ICLR, '20](https://arxiv.org/pdf/2010.06610.pdf), rank-1 BNNs [Dusenberry et a l, ICML, '20](https://arxiv.org/abs/2005.07186), and subspace inference [Izmailov et al, '19](https://arxiv.org/abs/1907.07504) are probably also relevant comparisons. At least one of these would be very helpful (probably MIMO or rank-1 BNNs b/c I think that they are using the same architectures that you are).\n    - This is especially needed because FFG-W (aka SVI) is such a weak baseline and then your approach is often dominated by your own baseline of Ensemble-U.\n        - Even the claim of \"not including these baselines [because they increase the parameter count]\" (lines 223-225) is insincere because FFG-W doubles the parameter count as well.\n    - A comparison to the standard Bayesian approaches for BDL is also missing [VOGN, SGMCMC, SWAG, GG-MC, etc.], but may be less relevant if the authors wish to focus solely on the parameter counting aspects. But these should still probably be cited.\n\n- Similarly, a comparison to a post facto pruning method (down to 25% of the weights) would also be helpful. Potentially, something like [Savarese et al, '20](https://arxiv.org/pdf/1912.04427v4.pdf) would be a good comparison or simple magnitude based pruning.\n    - Pruning represents a similar reduction in model capacity that the proposed approach does as well.\n    - Fig. 7 is a nice attempt but a deterministic comparison should be performed.\n    - One could always just imagine doing post facto pruning, and then a cheap and cheerful bayesian approach on retraining in the sparse space, say SWAG ([Maddox et al, NeurIPS, '19](https://arxiv.org/abs/1902.02476)).\n\n- An alternative comparison (and probably more relevant) would probably be the sparsity based SGMCMC procedure of [Deng et al, NeurIPS, '19](https://proceedings.neurips.cc/paper/2019/file/2d71b2ae158c7c5912cc0bbde2bb9d95-Paper.pdf), which is both designed for producing sparsity in deep networks and for producing strong uncertainty calibration.\n\n*Please note that I'm not suggesting the authors compare to all of these, please just pick one or two in the rebuttal (but cite the possibility of the rest). Preferably the SGMCMC one and/or deterministic pruning.*\n\n*Is this a complete piece of work or work in progress?* Complete work.\n\n*Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?* No weaknesses are discussed from my reading. The obvious one is what happens if $U$ is too small of a vector / matrix, in which the model will have too little capacity from the approximation algorithm. This might be seen in Fig. 4 right but I'm not sure.\n\n## Significance\n\n*Are the results important?* Possibly.\n\n*Are others (researchers or practitioners) likely to use the ideas or build on them?* Probably, this is a popular field right now.\n\n*Does the submission address a difficult task in a better way than previous work?* While the approach beats SVI, that's a very low bar. Both ensemble based approaches (one of which is your own baseline) seem to outperform it.\n\n*Does it advance the state of the art in a demonstrable way?* Empirically maybe not, in terms of practical sampling mechanisms for matrix normals, probably so.\n\n*Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?* A unique theoretical approach, same experiments as many papers in this space.\n\n## Discussion\n\n- The uncertainty bands in Fig. 3 are pretty nice overall.\n\n- Fig 4 looks like a mixed bag, the accuracy of FFG-U (which I take to be your method) is reasonably worse in terms of accuracy than even a standard deterministic NN (although better calibrated).\n\n- My overarching takeaway from the experiments is that the performance across all experiments goes *deep ensembles > ensembles-U > FFG-U > FFG-W* which suggests that FFG-U only outperforms FFG-W, but the hierarchical prior structure has some sort of performance limitations (as standard deep ensembles tend outperform ensembles over $U$).\n    - This limits the practical usefulness of the results to be only that the approach reduces the parameter count, but then further comparisons to pruning would be helpful.\n    - This ends up being hidden, as a result, but looking at the uncertainty quantification results in Tables 10+, it seems like generically the deterministic NN outperforms FFG-U and ensemble-U in terms of accuracy, but loses in terms of ECE, thus continuing to weaken the empirical evidence.\n        - This type of result really shouldn't be hidden deep in the appendix.\n\n- At its core, one of the weaknesses of SVI for NNs is that it tends to significantly underfit the data (see for example [Blier & Ollivier, NeurIPS, '18](https://papers.nips.cc/paper/2018/file/3b712de48137572f3849aabd5666a4e3-Paper.pdf)), hence, why the variance is clipped practically (lines 240-246), yet, your approach still seems like it'll underfit (given empirically that you still have to clamp the variances and the improved ECE, which is often an indicator of the amount of posterior variance).\n    - Should your approach get around this under-fitting issue or is its success solely due to the increased amount of structure?\n    - What happens if you don't clamp the variance in your approach?\n\n\n### Questions\n\n- More details on the *Ensemble-U* baseline would be very helpful. Is this just an ensemble of five models with the extra regularizer of $p(\\text{vec}(W) | \\text{vec}(U))$ as the prior?\n    - If that's not quite what it is, then that should be the baseline.\n    - Also, see the comment on line 148 above.\n\n- What does the reduction in parameter count mean practically? Is there actually a memory reduction or is this negated by the requirement of forming the full layerwise matrices $W$ via sampling using $U$?\n    - Does it induce some sort of sparsity in weight space or something along those lines?\n\n- What is the say \"accuracy vs model size\" tradeoff required on CIFAR-10 / 100? It's hard to parse from the two small points in Fig. 4.\n\n- Why does ECE skyrocket in Fig 7 as more pruning is performed? \n    - This seems to be the case in Fig 5 bottom right for small $M$.\n\n- What is training time like? In Fig. 4 you mention prediction time? It seems like it'll be a few times slower due to the extended Matheron's rule comments.\n\n- Why should magnitude based pruning not change the probabilistic model in FFG-U?\n\n- Overall, what happens if you ignore the layer-wise approximation and do the reduction on the full space (e.g. $\\text{vec}(W)$ refers to the vector of all parameters of the NN)? Does the approach still work?\n\n### Writing Comments\n\n- Fig 4 left is too small and detailed, had to zoom into 400% to get it to be visible, as well as to be able to parse taht there's actually two separate architectures in there. I'd suggest displaying only one architecture and the two inference types to make this clearer.\n    - Can this be shown on a log scale as well? Is there ever a \"break-even point\" where with enough samples, the Matheron's rule approach is as fast, so to speak?\n\n- line 657: mean + noise isn't helpful.\n\n- Personally, I'd suggest replacing most of the tables with plots using the x axis as the skew intensity level. The tables are really tough to parse as nothing is bolded or anything.\n\n- throughout: define \"push-forward distribution\" is this just the transformed (matrix) normal distribution after some sort of linear transformation?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "maybe suggest why smaller neural networks are better or worse for society?", "review_text": "", "extra_scores": {"ethical_concerns": "n/a.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper comes up with a variational inference strategy for learning a factored weight matrix, that they can then sample over efficiently using Matheron’s rule.  They need <25% of the parameters of the original deep ensemble but are competitive across metrics.  Four reviews were given for this paper with scores 6, 7, 4, 7.  Thus there was a general consensus to accept with one reviewer arguing for a reject.  In general the reviewers found the paper technically strong, novel and important in terms of the problem it addresses.

One of the major criticisms of the paper is that it is too dense and the reader is overloaded with technical background.  All reviewers seem to agree with this point (i.e. in discussion: “I wholly agree .. that the presentation of this paper is its major weakness”), but differ quite significantly in how important they think this is.  Some of the reviewers applaud the authors for striving for completeness, but seem to have found it a little excessive.  In particular, the reviewers didn't think presenting both the function space and weight space perspective was necessary.  Thus the authors are strongly recommended to move one of these to the appendix to make space for other parts of the paper.

There’s some criticism of the priors used - the reviewer felt like the empirical comparison wasn’t apples to apples because of the priors were different between their method and the baselines.  This seems reasonable, but the choice of prior seems core to the proposed method (needs to be decomposable due to Matheron’s rule).  Thus it would seem acceptable to not require the authors to change the prior of their method.

The final criticism is that the method is slower than the original ensemble, so sparsity doesn’t help - though there is still (in theory) considerable memory savings.  Again, the reviewers differed in their assessment of how important this issue is.  

The majority felt that this is a strong paper. Considering that the major cited weakness is in the presentation of the technical background, the recommended decision is to accept the paper with the ask that the authors rework the technical background to make it more accessible.  In particular, the reviewers recommended that the function space perspective could be moved to the appendix in the camera ready.  This would free up space to make the paper less mathematically dense and allow more space for other parts of the paper.

Decision: Accept (Poster)