Structured Dropout Variational Inference for Bayesian Neural Networks
Authors: Son Nguyen
Duong Nguyen
Khai Nguyen
Khoat Than
Hung Bui
Nhat Ho
Venue: NeurIPS 2021 Poster (2021)
Keywords: Bayesian neural networks
Variational Dropout
Structured Approximate inference

Abstract: Approximate inference in Bayesian deep networks exhibits a dilemma of how to yield high fidelity posterior approximations while maintaining computational efficiency and scalability. We tackle this challenge by introducing a novel variational structured approximation inspired by the Bayesian interpretation of Dropout regularization. Concretely, we focus on the inflexibility of the factorized structure in Dropout posterior and then propose an improved method called Variational Structured Dropout (VSD). VSD employs an orthogonal transformation to learn a structured representation on the variational Gaussian noise with plausible complexity, and consequently induces statistical dependencies in the approximate posterior. Theoretically, VSD successfully addresses the pathologies of previous Variational Dropout methods and thus offers a standard Bayesian justification. We further show that VSD induces an adaptive regularization term with several desirable properties which contribute to better generalization. Finally, we conduct extensive experiments on standard benchmarks to demonstrate the effectiveness of VSD over state-of-the-art variational methods on predictive accuracy, uncertainty estimation, and out-of-distribution detection.

Structured Dropout Variational Inference for Bayesian Neural Networks
Son Nguyen f.l
Duong Nguyen
Khai Nguyen' ! Khoat Than 3.1
Hung Bui
Nhat Ho*?
VinAl Research, Viet Nam University of Texas, Austin
3 Hanoi University of Science and Technology
Abstract
Approximate inference in Bayesian deep networks exhibits a dilemma of how to yield high fidelity posterior approximations while maintaining computational efficiency and scalability. We tackle this challenge by introducing novel varia- tional structured approximation inspired by the Bayesian interpretation of Dropout regularization. Concretely, we focus on the inflexibility of the factorized structure in Dropout posterior and then propose an improved method called Variational Structured Dropout (VSD). VSD employs orthogonal transformation to learn structured representation on the variational Gaussian noise with plausible complex- ity, and consequently induces statistical dependencies in the approximate posterior. Theoretically, VSD successfully addresses the pathologies of previous Variational Dropout methods and thus offers standard Bayesian justification. We further show that VSD induces an adaptive regularization term with several desirable properties which contribute to better generalization. Finally, we conduct extensive experiments on standard benchmarks to demonstrate the effectiveness of VSD over state-of-the-es variational methods on predictive accuracy, uncertainty estimation, and out-of -nttof-distribution detection.
1 Introduction
offer a probabilistic interpretation for deep learning models by imposing a prior distribution the weight parameters and aiming to infer posterior distribution instead of only point estimates. By marginalizing over this posterior for prediction, BNNs perform procedure of ensemble learning. These principles improve the model generalization, robustness and allow for uncertainty quantification. However, exactly computing the posterior of non-linear BNNs is infeasible and approximate inference has been devised. The core challenge is how to construct an expressive approximation for the true posterior while maintaining computational efficiency and scalability, especially for modern deep learning architectures.
Bayesian Neural Networks (BNNs)
49 63
Variational inference is a popular deterministic approximation approach to deal with this challenge. The first practical methods were proposed in [22] g 39], in which the approximate posteriors are assumed to be fully factorized distributions, also called mean-field variational inference. In general, the mean-field approximation family encourages several advantages in inference including computational tractability and effective optimization with the stochastic gradient-based methods. However, it ignores the strong statistical dependencies among random weights of neural nets, leading to the inability to capture the complicated structure of the true posterior and to estimate the true model uncertainty.
'These two authors contributed equally. CCorrespondence to Son Nguyen: <v. cv.sonnv27evina..f 1o2
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

To overcome this limitation, many extensive studies proposed to provide posterior approximations with richer expressiveness. For instance, [47] treats the weight matrix as a whole via a matrix variate Gaussian [24] and approximates the posterior based on this parametrization. Several later works have exploited this distribution to investigate different structured representations for the variational Gaussian posterior, such as Kronecker-factored [89] 71 72,, k-tied distribution
[77], non- centered or rank-l parameterization [21 Another original idea to represent the true covariance matrix of Gaussian posterior is by employing the low-rank approximation [67
35 80). For robust
approximation with multimodality, [48] adopted hierarchical variational model framework [69] for inferring an implicit marginal distribution in high dimensional Bayesian setting. Despite significant improvements in both predictive accuracy and uncertainty calibration, some of these methods incur a large computational complexity and are difficult to integrate into deep convolutional networks.
Motivations. In this paper, we approach the structured posterior approximation Bayesian neural nets from a different perspective which has been inspired by the Bayesian interpretation of Dropout training 77 [5T. More specifically, the methods proposed in [9 20 reinterpret Dropout regular- ization as approximate inference in Bayesian deep models and base this connection to learn variational Dropout posterior over the weight parameters. From the literature, inference approaches based on Bayesian Dropout have shown competitive performances in terms of predictive accuracy on various tasks, even compared to the structured Bayesian methods aforementioned, but with much cheaper computational complexity. Moreover, with the solid and intriguing theories effective regularization 8I 26 831, generalization bound 52, 59]. convergence rate and robust optimiza- tion 54 71 Dropout principle offers several potentials to further improve approximate inference in Bayesian deep networks. However, since these Bayesian Dropout methods also employed simple structures of the mean-field family, their approximations often fail obtain satisfactory uncertainty estimates [7]. In addition, Variational Dropout methods based on multiplicative Gaussian noise also suffer from theoretical pathologies, including improper prior leading to ill-posed true posterior, and singularity of the approximate posterior making the variational objective undefined (31).
Contributions. With the above insights, we propose a novel structured variational inference frame- work, which rationally acquires complementary benefits of the flexible Bayesian inference and Dropout inductive bias. Our method adopts an orthogonal approximation called Householder transfor- mation to learn a structured representation for multiplicative Gaussian noise in Variational Dropout method (39 57 As consequence of the Bayesian interpretation, we go beyond the mean-field family and obtain a variational Dropout posterior with structured covariance. Furthermore, to make framework more expressive, deploy hierarchical Dropout procedure, which is equivalent to inferring joint posterior in a hierarchical Bayesian neural nets. We name the proposed method as Variational Structured Dropout (VSD) and summarize its advantages as follows:
Our structured approximation is implemented on low dimensional space of variational noise with considerable computational efficiency. VSD can be employed for deep CNNs in direct way while maintaining the backpropagation in parallel and optimizing efficiently with gradient- based methods.
2. Especially, VSD has a standard Bayesian justification, in which our method can overcome the non-Bayesin Bayesian perspective of previous Variational Dropout methods. Our inference framework uses a proper prior, non-singular approximate posterior and derives a tractable variational lower bound without further simplified approximation.
critiques from the non-
3. Compared with previous Bayesian Dropout methods which are relatively inflexible by some strict conditions, VSD is more efficient on both the criteria of expressive approximation and flexible hierarchical modeling. Therefore. VSD is promising to be inference and in particular for BNNs.
approach for Bayesian
4. To reinforce the complementary advantages unified in our proposal, we also investigate the inductive biases induced by the adaptive regularization of structured Dropout noise. We further provide an interpretation that VSD implicitly facilitates the networks to converge to a local minima with smaller spectral norms and stable rank. This properties suggests better generalization and we present empirical results to support this implication.
5. Finally, we carry out extensive experiments with standard datasets and different network architec- tures to validate the effectiveness of our method on many criteria, including scalability, predictive accuracy, uncertainty calibration, and out-of-distributie detection, in comparison to popular varia- tional inference methods.

---

Notation. For a matrix A, | and A denotes the Frobenius norm and the transpose matrix, A; and A, denote the i-th row and the j-th column. For an interger i, e, the i-th standard basis, 1, E R' is the vector of all ones. The diagonal matrix with diagonal entries the elements of a vector x is denoted by diag(x), The inner product between two matrices A and B is denoted by (A, B).
2 Background
Variational inference for Bayesian neural networks: Given dataset 'D consisting of input-output
pairs (X, Y) == {(xn; )n))== In BNNs, we impose a prior distribution over random weights p( (W)
whose form is in a tractable parametric family and aim to infer an intractable true posterior p( [W]] (D). Variational inference (VI) [30] 33 can do this by specifying variational distribution do ( (W) with free
parameter and then minimizing the Kullback-Leibler (KL) divergence D (gp (W Ipp( (W [D)).
This optimization is equivalent to maximizing the Evidence Lower Bound (ELBO) with respect to
variational parameters ø as follows:
( = (w) log (D)))
(W))l
By leveraging the reparameterization trick 40 combined with the Monte Carlo integration, we can derive an unbiased differentiable estimation for the variational objective above. Then, this estimation can be effectively optimized using stochastic gradient methods with the variance reduction technique such as the local reparameterization trick (39]
Variational Bayesian inference with Dropout regularization: Given deterministic neural net with the weight parameter θ of size K x Q. training this model with stochastic regularization tech- niques such as Dropout 22 74 can be interpreted as approximate inference in Bayesian probabilistic models. This is because injecting a stochastic noise into the input layer is equivalent to multiplying the rows of subsequent deterministic weight by the same random variable, namely with each datapoint (x..yn) and noise vector ç. we have: Yn (x,, @ {) = x,, diag This induces BNN with random weight matrix defined by W := diag(E) Applying VI this Bayesian model, with some specific choices for prior and approximate posterior, the variational lower bound (T) can resemble the form of Dropout objective in the original deterministic network. This principle is referred as the KI condition [I8]. Gal et al. (20). Kingma et al. (39 used this principle to propose Bayesian Dropout inference methods such as MC Dropout (MCD) and Variational Dropout (VD).
Dropout inference is practical approximate framework especially in high dimensional setting. How- ever, the scope of Bayesian inference in these methods is restricted in terms of flexibility of both prior and approximate posterior. Concretely, the Dropout posteriors go (W) in MCD and VD both have simple structures of mean-fiell approximation which often underestimate the variance of true poste- rior. possibly leading to a poor uncertainty representation [17]. Moreover, in theory. VD employed an improper log-uniform prior which can result in an ill-posed true posterior and generally push the parameters towards the penalized maximum likelihood solution [1]. In addition, VD) also suffers from the singularity issue of approximate posterior that makes the KI, divergence term undefined. Our work gains an efficient remedy to these pathologies.
3 Variational Structured Dropout
We focus on Bayesian Dropout methods using multiplicative Gaussian noise with correlated parameter- ization (39]. This procedure induces random weight W = diag(E) where the Dropout noise & is multivariate Gaussian with diagonal covariance qa(E) = A111 diag(a)), and the droprate vec- tor. The corresponding Dropout posterior then is given by ga (W) = Law (diag(()@). This distribu- tion on each column exhibits a factorized structure with the form of (W.,)) ) = (N diag(a )). whilst allows a correlation on each row because each W. i: is shared by the same scalar noise &, respectively. The parameters & == (a, Q) are optimized via maximizing a variational lower bound as follows: L(ő) := (y log logp(D))) W) P (qo( (W)llp(W)) (W)) (gy log p(P),, θ) DeL (go (go (W)lpp( (P()), where the later equation is derived from the change of variables formula.
3.1 The orthogonal approximation for variational structured noise
Intuitively, a richer representation for the noise distribution can enrich the expressiveness of Dropout posterior via the Bayesian interpretation. We implement this intuition with assumption that the
3

---

Dropout noise could be sampled from a Gaussian distribution with full covariance matrix instead of a diagonal structure. namely, (E) = (11x. .) with Σ is a positive definite matrix of size K x K. To make this covariance matrix learnable, we first represent E in the form of the spectral decomposition: Σ PAP where P is an orthogonal matrix with its eigenvectors in columns, A is a diagonal matrix where diagonal elements are the eigenvalues. By the basis-kernel representation theorem 66 [6]. we parameterize the orthogonal matrix P as a product of Householder matrices in the following form: P = Hr. -I.. I,,HH, where H. = I 2v.v?' /Ilva lī.. V, is the Houscholder vector of size K, and 7 is the degree of P. This parameterization relaxes the orthogonal constraint of matrix P, and can then directly optimize the covariance matrix Σ via gradient-based methods.
Notably, this transformation can be interpreted as sequence of invertible mappings. More explicitly, we extract zero-mean Gaussian noise r/()) from the original noise g()) (V k: diag(a)) in the form of {())_ reflections, we obtain the induced noise and the corresponding density at each step / as follows:
17 17("), and by successively transforming () through chain of T Householder
< = 1 111 = U, and qe(E) == ((1,, Udiag(a)UU))
By injecting the structured noise ç(1) into the deterministic weight O we obtain a random weight W(+) := ()) and an induced Dropout posterior (:(.) Law Law(diag (diag(5 ((()
(G())) )O) with fully correlated representation, in which the marginal column distribution is given by: q:( (W;)) .y)
(( diag(O.,) ) Udiag(a)UU (T) JTdiiag(ee)))) diag(G.))). Detailed discussion about the expressiveness of this cor- related structure is presented in Appendix [ With the above derivations, we use variational inference with approximate posterior q: (W) and optimize the variational lower bound as follows:
(())= == (w) log p (D) W) Dx (g:
ge) logp ((*) - DRz(a (W)l(())))l))))
Overcoming the singularity issue of approximate posterior. In Variational Dropout method with correlated parameterization, there is a mismatch in support between the approximate posterior and the prior, thus making the KL term DD K (g( (W)lP(((T)))) (W) undefined [31]. Specifically, the form W = d is equivalent to multiplying each row W;: by the same scalar noise &,, namely W,: j: = O.O; with ((S)) (11, oz). This means that the approximate distribution always assigns all its mass on subspaces defined by the directions aligned with the rows of O. These subspaces have Lebesgue measure zero causing the singularities in approximate posterior, and the KL term will be undefined whenever the prior (W) puts zero mass to these subspaces. However, in VSD the scalar noises not treated separately due to the structured correlation, namely VSD would injects each &a into the whole matrix O instead of some individual directions. Indeed, we have: W(VD) := di) θ + di
O + - (0) (diag(e:)() = = mi (
(0)
and WVSDD) := diag( = e + diag - Σ ni (diag( (diag(U.)))) where O(s) is
the matrix with only the i-th row retained. While VD causes singular components represented by
|= VSD maintains a trainable orthogonal matrix U which prevents the approximate posterior from having degenerate supports with measure zero, thereby avoiding the singularity issue. In the following section, we will present an appropriate choice of the prior distribution p( ()) such that the KI. term is well-defined, and then derive tractable objective function satisfying the KL. condition in Bayesian Dropout frameworks.
K (Gt))
3.2 Derivation of tractable variational objective
We consider employing an isotropic Gaussian as the prior distribution, namely p (W) = IT P (W.j) with p( .W,)) = (O, diag (87²)) and B a hyper-parameter matrix of the same size with W. With the previous analysis, our approximate posterior ge (W) would be absolutely continuous w.r. the prior p( W). and thus the KL term Dx (% (W)llp( W)) is defined. Furthermore, the Gaussian prior helps our proposal avoid the pathologies of improper true posterior and ill-posed inference in VD.
Note that, the prior p( () is fully factorized Gaussian which usually facilitates simple analysis and efficient computation. This factorized structure is chosen also because we have no reason for the correlation between non-identical weights at first. Moreover, several arguments indicate that simple prior over parameter p( (W), when interacts with neural nets architecture [(D; W), induces sophisticated prior over function p(f (D; W)), with desirable properties and useful inductive

---

0.736
25.45
0.028
erfor rale
0.732
ECE
0.024
0.728
24.74
0.020
1
3
8.70
0.026
effer rate
NLL
0.315
8.54
0.016
0.308
8.37
0.006
Figure 1: The performance of VSD when using the number of transformations T E (1,2,3}. Evaluation over 5 runs CIFARIO (above) and SVHN (below) with LeNet architecture.
biases [86]. However, when we are interested structured approximations parameter space, the factorized prior may raise some contradictions, By relative entropy decomposition, we have:
Q
Der (a:::)(:)))))))) (W)ll((W) (W))
DKI(w (lW + I(W.,, W W.)),
where I(.) is the mutual information measured by the distribution (e(-), and this term is validly defined in VSD. Maximizing the variational lower bound tends to encourage smaller KL term, and hence constrains the components in RHS of equation 3 Intuitively, relatively small mutual information can break the strong correlations between the columns of W. Several studies have focused on this limitation and suggested using richer priors such as matrix variate Gaussian (75] [9], doubly semi implicit distribution [58). Our solution for this scenario is derived naturally from equation ( in which we leverage the mutual information as an additional regularization term. Concretely, we maximize alternative variational objective follows:
(MI(Ø) (g) log ((D)O, (x (an((a(W)lleee W)llp(W)) I(W., W2,..., Wie) = Eg (2) log p ( - (x (W))l
(4)
where qf (W) == ( (W,)) is the product of marginal column distributions. From the information-thoreee perspective, augmenting the mutual information is a standard principle for structure learning in Bayesian networks [4I]. Particularly in our derivation, maximizing the alter- native objective LMI(O) (Ø) would help to sustain the dependence structure between columns of the network weights, and thus fixes appropriately the broken ELBO mentioned above. Interestingly, this technique is utilized reasonably in our method. This is because our dependence structure allows to specify explicitly the marginal distribution each column of W, leading tractable objective in equation (4) whose KL divergence between two multivariate Gaussian can be calculated analyti- cally in closed-form. We note that similar application to other structured approximations, such as low-rank, Kronecker-factore or matrix variate Gaussian, can be non-trivial. We also remark that our alternative variational objective might be not necessarily valid lower bound of the original model evidence, but would be the lower bound of new model evidence defined on an alternative prior ő( (W) which satisfies Dx (g: (W) (WW)) DrL (qī (W) (W)). Indeed, the correlated prior deter- mined by p( W) X p( p(W) 2: (W)))/g (W) meets this constraint, and thus we could reinterpret the use of mutual information adopting this prior at each iteration of the training procedure. To clarify, our idea was partly motivated by the similar technique that has been extensively adopted in deep latent variable models, in which a mutual information maximization is also added to the variational lower bound to mitigate the degenerate issue of amortized inference in these models 92
The KL condition in VSD. With the variational objective in equation (4). offer VSD complementary advantages of structured Dropout and Bayesian inference, we need to ensure DK (WW)) satisfies the KL condition. We solve this prerequisite by specifying the precision parameter 3 of the prior p( (W) via the Empirical Bayes (EB) approach. As a result, we obtain an optimal value for this hyperparameter an analytical form 3*. The optimal 3* is then substituted back into the prior, and thereby we get the optimal KL term with form independent of the deterministic weight as follows:
K LØ=1 1== ag U3U?
1
DEB DEB DRE
log
(5)
i=1
5

---

DAIN CEARLS
0.160
0.16
0.8
11.3
27.3
1.1
10.3
26.3
0.104
0.11
9.3
25.3
0.054
0.06
0.7
8.3
24.3
0.004
0.02
Video
Viddo
VSD
VSD*
NLL
error rate
ECE
Figure 2: The performance of ARD-VD and VSD when using the prior hierarchy (coressponding the labels VD and VSD"). The left and right y- axis are represented for SVHN and CIFARIO dataset, respectively.
A formal proof of equation 5 is in Appendix B
The number of transformations steps T in VSD. An appropriate choice of T' is essential in our method. For deep learning architectures, the degree 7 of the orthogonal matrix P might be relatively large. so is quite challenging to adjust the empirical value of T in principled way to meet the basis-kerne representation theorem. We can employ an efficient parameterization introduced in [90], in which we only need the Householder vectors {v., 2 1} with sizes much smaller than the order of the matrix P. This will facilitate tuning 7" with larger range in an applicable computation time. In our method, we instead only adopt a small T E {1, 2, as form of "low-degree" approxiamtion, and to make the reflections more expressive, we use a fully connected layer between successive Householder vectors, i.e. V, = FC((v-1). The low dimension of variational noise in VSD leads to a good adaptation, but with little trade-off in computational complexity when increasing the number of transformation steps T'. show the performance of VSD when using this neural parameterization with 7' E {1, 3) in Figure where larger 7 could potentially improve the results on predictive measures. However, to maintain computational efficiency. we recommend using T = or 2 in large-scale experiments. Indeed, this neural parameterization has been successfully implemented in the context of learning the latent space in deep latent variable models (79
3.3 Joint inference with hierarchical prior
We further promote our proposal by introducing prior hierarchy in VSD framework and then obtain a joint approximation for the Dropout posterior. This will facilitate the flexibility of Bayesian inference in our method in terms of the expressiveness of both prior distribution and approximate posterior. We design a two-level hierarchical prior given by: p(W, 2) = =p(Wlz, 3)p(z), where pl pW 3) = 11 P(W.;, 12.2. B.y) and p P(W,, W2, 3.)) = (O0 diag(z @ Brj")): : the hyperprior p(z) a distribution with positive support such as Gamma or half-Cauche distribution; the latent z has the size of the number of rows and is shared across columns of the weight matrix W; the hyper-parameter matrix 3 is treated as a scaling factor. This prior family has a centered parameterization and induces several compelling properties such as facilitating feature sparsity 46 12] model selection [2] or improving robustness, uncertainty calibration 15
We implement variational inference with joint approximate posterior, also referred to the joint Dropout posterior, which is parameterized as follows:
Lo [W, 2) gu (z)gs (W]l)))))) with 4s (W
N(zz o O, diag(z²
07))),
where Ts (Wlz) (zz)
is the conditional Dropout posterior, qu (z) chosen depending on the family of prior p(z) so that the reparametrization trick can be utilized. Sampling the random weight W from the joint variational posterior go (WW, z) includes two steps:
qu (z) and W qo( Wiz*), in which the second one can be reparameterized as: W = diag(z" (z" )diag(()))))))))
diag(z* c O)O, with k1, diag(o)). This new representation adapts to the vanilla Dropout procedure but allows our method to regularize each unit layer with different levels of stochasticity. We derive some insights about the role of hierarchical prior in our framework in Appendix A..2 We apply the Houscholder transformation to the variational noise and obtain a new joint approximate posterior:
the noise& & ~ [11,,,
qe W, z) = q. (z)qq( ([z)) with qe (W : 1z) = (2z o θ,,, V, (Vddia((a)V
where V, = diag(z o O,₁)) Similarly, we define of (W (Wiz) 11 qe( W (z) and then optimize an alternative variational objective given by:
L(o, v)= == (w)))) logp(D/W) E (z) (DKL (W x) p W |z B)) - (gx

---

Table 1: Computational complexity per layer of MAP and different variational methods.
Table 2: Computation time of variational methods compared to standard MAP (1x).
Method
Time
Memory
Time/epochs (s)
Methods
MAP
O(KL
OLL B))
LoNet5
AlexNet
ResNet18
BBB
O1sKL)8))
O(sKL 4[8)
BBB-LTR
1.53x
1.75x
3.28x
BBB-LTR
((2
((2L(8)
MNF
2.86x
3.40x
4.88x
VMG
O(m² 2KLBB)
((KL
VD
1.18x
1.15x
1.32x
SLANG
Olr2KL rBKL)8))
O(rKL sKL
VSD T =
1,25x
1.32x
1.86x
ELRG
O(r² (r 2)K
O((r + 2)L)Bl
VSD T =
1.35x
1.49x
2.90x
VSD
OIK² + OK)8) Oirk KL)
O(K² + K
VSD-low rank
O(K? K|B))
We have chosen the (inverse) Gamma and log-Normal distribution for p(z) and ((z) respectively. These distributions have positive supports, can be reparametrized and the KL-divergence between them also has a closed-form due to the conjugacy. A full derivation of L( v) including the KL condition is given in Appendix C The parameterization of the prior hierarchy our method flexible without any simplifying assumptions about hyperprior p(z). We can directly apply it for ARD- Variational Dropout framework (ARD-VD) [36] derivation is in Appendix (F). As the experimental results are shown in Figure 2 the hierarchical prior significantly improves the performance of both ARD-VD and VSD on predictive metrics. Therefore, we aim to introduce VSD with hierarchical prior as unified framework and a general-purpose approach for Bayesian inference, particularly for BNNs.
3.4 Scalability of Variational Structured Dropout
Approximating a structured posterior directly on the random weights of deep convolutional models is challenging. Besides expensive computation, it difficult to employ the local reparameterization trick (39]. leading to the high variance issue training. We apply VSD to convolutional layer by learning structured noise with the size of the number of kernels and imposing
convolutional weights: & ~ (11,, Udiag(a) (a: U²) )") and W iyt: = Osjk with j, k are the indexes representing height, width, and kernel respectively. This simple solution greatly reduces computational complexity while being able to captures the dependencies among kernels of the convolutional layer.
We present the complexity of MAP and VI methods in terms of computational cost and memory storage in Table [ with the detailed analysis given in Appendix D VSD adopts the advantage of Dropout training and maintains efficiency on both criteria. We also give more results in Table about the empirical computation time of VSD and some other methods. Based on these tables, VSD shows more effective running time even than the mean-field BBNs. Although there trade-off when using larger number of T, VSD does not incur much extra computation time compared to VD.
3.5
On explicit regularization of Variational Structured Dropout
There are several compelling theories to explain the tremendous success of Dropout technique. in which regularization-based is of the most active approaches [81] 26 55
10, We would follow this direction to investigate inductive biases induced by the structured Dropout in VSD, and from which consolidate our claim of complementary advantages unified in the proposed method. To characterize the regularization of VSD, we consider a deep linear neural net with L layers parameterized by (0(1 12=1- and define some notations as: x is an input data, B is the data batch; h, is the i-th hidden layer; Jx(x) denotes the Jacobian of network output w.r.t h;(x;; (x(x) and Ha (x) denotes the Hessian of the loss w.r.t hx(x) and the network output, respectively. Then we have J, (II ( & @li-L the transposition of linear multiplication of weight matrices from i-th layer to the last one. From a detailed derivation presented Appendix E VSD induces an explicit regularization given by:
RysD = E(x-B) Σ H; diag (h,) Udiag(a) U diddig(h..).
Note that, H, can be approximated by J Ho after ignoring the non-PSD term which less important empirically [73]. This regularizer offers some intriguing but popular interpretations related
7

---

Table 3: Results for VSD and baselines vectorized MNIST, CIFARIO and SVHN. Results are averaged over
5 random seeds. For all metries, lower better.
MNIST
CIFARIO
SVHN
Method
FC 400x2
FC 750x3
CNN 32x64x128
NLL
em: rate
ECR
NLL
er:: rale
ECR
NLL
eT.. rale
ECE
NLL
eTr. rale
ECE
MAP
0.098
1.32
0.011
0.109
1.27
0.011
2.847
34.04
0.272
0.855
12.26
0.086
BBB
0.109
1.59
0.011
0.140
1.50
0.013
1.202
30.11
0.098
0.545
10.57
0.017
MCD
0.049
1.26
0.007
0.057
1.22
0.007
0.794
26.91
0.024
0.365
9.23
0.013
VD
0.051
1.21
0.007
0.061
1.17
0.008
1.176
27.45
0.156
0.534
9.47
0.055
ELRG
0.053
1.54
0.871
29.43
VSD
0.042
1.08
0.006
0.048
1.09
0.006
0.730
24.92
0.020
0,299
8.39
0,008
D.F 0.057
1.29
0.009
0.063
1.21
0.009
1.815
26.44
0.042
0.783
9.31
0.070
SWAG
0.044
1.27
0.008
0.043
1.25
0.007
0.799
26.94
0.012
0.312
8.42
0.021
to the curvature of loss landscape 83 10 (see detailed explanation in Appendix E We now show novel properties of VSD induced by the orthogonal matrix U.
VSD imposes Tikhonov-like regularization and reshapes the gradient of network weights: We rewrite
our regularization corresponding to layer i-th by: R() = Ex(xB)) B3 [ 1/2 diag(h,)))))) (ndb)))ddag(a/// This form can be interpreted as the Tikhonov-like regularization imposed on the square root of Hessian matrix H, in which the Tikhonov matrix r == diag h.) Udiag( is automatically learned during training. This principle can improve the conditioning of the estimation problem. Furthermore, when considering the case of regression problem, we have:
RYsD E(x-8)
Udiag(a)UU U" diag(h,) (o
(6)
This is a data-dependent regularization with adaptive structure determined by the matrix IT diag(h,) (achh,,din (ddag(a)) U diag(h,). ddagghh) From the algorithmic perspective, this regularizer allows VSD to reshape the gradient of network weights according to the geometry of the data based on both scale and direction information [14 25). Meanwhile ITT (h, (iad)ddiaaa)iia (h,) only plays scaling factor in VD.
VSD penalizes implicitly the spectral norm of weight matrices: Let Ω2 == diag h, )J J then our regularizer can be rewritten as: R = TE (x= Σk=1 a U U. Since the trainable matrix U satisfies UT = 1 for any k, penalty on U, implies that VSD likely prefers a solution with smaller spectral norms of the matrix 2 J,diag(hih)) (h;) and thus of the network weights. This implication points us to the well-studied theories about generalization bound based on the spectral norm [4] 65). Concretely, Neyshabur et al. (65] suggests that smaller spectral norm and stable rank can lead to better generalization. This expectation can be observed empirically in VSD through Table 9 in Appendix E A more solid investigation about the generalization of VSD is of interest.
4 Experiments
In this section, we provide experimental evaluations to show the effectiveness of our proposed methods compared with the existing methods in terms of both predictability and scalability. We focus mainly on variational inference methods of the following two directions: the first one is direct approximations
of the posterior on the random weights of Bayesian nets, including Bayes by Backprop (BBB) (8),
Variational Matrix Gaussian ( (VMG) [47, low-rank approximations (SLANG, ELRG) [56 80;; and
the other one is the Bayesian Dropout methods with MC Dropout (MCD) 19 20 Variational
Dropout (VD) (39. and our method- method-Variational Structured Dropout (VSD). In addition, we evaluate
the performance of point estimate framework MAP and two standard con-variational baselines
Deep Ensemble (D.E) 44) and SWAG [50). Details about data descriptions, network architectures,
hyper-parameter tuning are presented in Appendix T
4.1 Image classification
We now compare the predictive performance of the aforementioned methods for classification tasks on three standard image datasets: MNIST [45), CIFARIO 43), and SVHN [64]. We evaluate the
8

---

Table 4: Image classification using AlexNet architecture. Results averaged over random seeds.
CTFAR1O ACC
CIFAR100
SVHN ACC
STL.10
AlexNet
NLL
ECE
NLL
ACC
ECE
NLL
ECE
NLL
ACC
ECE
MAP
1.038
69.58
0.121
4.705
40,23
0.393
0.418
87.56
0.033
2.532
65.70
0.267
BBB
0.994
65,38
0.062
2.659
32.41
0.049
0.476
87.30
0.094
1.707
65.46
0.222
MCD
0.717
75.22
0.023
2,503
42,91
0.151
0.401
88.03
0.023
1.059
63.65
0.052
VD
0.702
77.28
0.028
2.582
43.10
0.106
0.327
90.76
0.010
2.130
65.48
0.195
ELRG
0.723
76.87
0.065
2,368
42,90
0.099
0.312
90.66
0.006
1.088
59,99
0.018
VSD
0.656
78.21
0.046
2,241
46.85
0.112
0.290
91.62
0.008
1.019
67.98
0.079
D.E
0.872
77.56
0.115
3,402
46.42
0.314
0.319
90.30
0.008
2.229
68.51
0.241
SWAG
0.651
78.14
0.059
1.958
49.81
0.028
0.331
90.04
0.031
1.522
68.41
0.161
Table 5: Image classification using R architecture. Results are averaged over 5 random seeds.
ResNet18
CIFARIO ACC
CIFAR100 ACC
SVHN ACC
STL10 ACC
NLL
ECE
NLL
BCE
NLL
ECE
NLL
ECE
MAP
0.644
86.34
0.093
2,410
55.38
0.243
0.232
95.32
0.028
1.401
71.26
0.199
BBB
0.697
76.63
0.071
2.239
41.07
0.100
0.218
94.53
0.047
1.290
71.55
0.179
MCD
0.534
87.47
0.084
2.121
59.28
0.227
0.207
95.78
0.026
1.333
72.28
0.188
VD
0.451
87.68
0.024
2,888
56,80
0.284
0.164
96.11
0.017
1,084
73.29
0.084
ELRG
0,382
87.24
0.018
1.634
58.14
0.096
0.145
96.03
0.003
0.811
73.66
0.080
VSD
0.464
87.44
0.061
1,504
60.15
0.116
0.140
96.41
0.003
0.769
74.50
0.083
D.E
0.488
88.91
0.069
1.913
60.16
0,203
0.171
96.36
0.020
1.197
73.16
0.177
SWAG
0.330
88.77
0.026
1.417
62.45
0.028
0.130
96.72
0.016
0.843
73.15
0.069
predictive probabilities using negative log-likelihool (NLL), error rate, and expected calibration error (ECE) (3). Details on experimental settings available in Appendix [.4
The synthesis results of this experiment are in Table On MNIST, VMG achieves err. rates of 1.17% and 1.27% with FC 400x2 and FC 750x3 respectively, while SLANG reports 1.72% err. rate with FC 400x2. In general, VSD outperforms consistently other variational methods in most settings. For D.E and SWAG, VSD exhibits competitive results on all three metrics. Especially, the figures on NLL and ECE indicate well-calibrated probabilities in our model. This is also common but noteworthy behavior in structured approximations. On the other hand, for the remaining methods such as MAP and BBB, the error rates are worse by a large margin compared to VSD (respectively about 9% and 5% on CIFAR 10, 4% and 2% on SVHN). On CIFARIO and SVHN, these two methods and VD all show poor results on both NLL and ECE measures, implying that it will be difficult for them to reason properly about the model uncertainty especially in the
context. For MC Dropout, we observe a pretty good performance with the second-bes best result in variational methods that is similar to those reported of other works [56] 68, 80) These results of the Bayesian Dropout methods are competitive with structured methods such as VMG, SLANG, ELRG. This further reinforces our motivation about the potential of Dropout methods for improving the predictive performance.
4.2 Scaling up Bayesian deep convolutional networks
We conduct additional experiments to integrate VSD into large-scale convolutional networks. We reproduce the experiments proposed in 80 (ELRG), in which we trained AlexNet and ResNet18 on
4 datasets CIFARI0 10, SVHN, CIFAR 100 43 and STILIO [1] to evaluate the predictive performance of our proposal compared to other methods.
The final results are given in Table 4 and Table 5 in which the top two results will be highlighted in bold. For AlexNet, the performance of VSD is more consistent and higher than other variational methods. Modern deeper architectures facilitate deterministic estimates like MAP to better learn discriminative information extracted from training data but also makes its predictions more confident when picking excessively on unique optima. Therefore, although MAP has comparable predictive accuracy on some settings, it comes at a trade-off with the worst results on both NLL and ECE, Meanwhile, ELRG with a low-rank structure on the variational posterior gains desirable properties on uncertainty metrics. Its performance on NLL and ECE are competitive to that of VSD, however, our method obtains significant improvements on accuracy metric. For the remaining methods, BBB performs poorly on CIFARIO 10 and CIFAR 100 in predictive accuracy, but still has better performance than MAP in terms of NLL and ECE.

---

distribution SVHN
Out of distribution CIFAR10
Out distribution CIFAR100
BBB
MC Dropout
6
VD
3
Predictive Entropy
Predictive Entropy
Predictive Entropy
Figure 3: Histograms of predictive entropy for LeNet architecture trained on SVHN dataset.
For ResNet18 architecture, while MAP and BBB still exhibit the same behavior as mentioned above, VSD continues to achieve the convincing results, namely, it has the best performance on CIFARI00 and SVHN over all three metrics when compared to variational STLI0, VSD also gets competitive statistics on NLL and accuracy. Overall, compared with MCD, VD, and ELRG, our method maintains good performance with better stability.
methods. On CIFAR1O and
4.3 Predictive entropy performance
We now evaluate the predictive uncertainty of each model on out-of-distribution settings that have been implemented in previous works [44] 48 68 80]. We evaluate the entropy of predictive distribution
Ky"l=" D) and use the density of this entropy to assess the quality of uncertainty estimates. Basically, an accurate and well- well-calibrated model is expected to represent entropy values being concentrated mostly around 0 (i.e. high confidence) when the test data comes from the same underlying distribution as the training data, and in the opposite case, the predictive entropies should be evenly distributed (i.e. higher uncertainty). In fact, the deep learning models do not achieve simultaneously on both expectations at the most ideal, but instead, accurate and well-calibrated ones tend to exhibit moderate level of confidence on in-distribution data, and then provide a reasonable representation for uncertainty estimates on nut-of-distribution data.
For LeNet, we train the model on SVHN dataset and then consider out-of-distribune data from CIFAR10 and CIFAR100. The results are shown in Figure All methods work well on in-distribution data SVHN with the entropy value being distributed mostly around zero. However, the entropy densities of MAP and VD are concentrated excessively. This indicates that these methods would tend to make overconfident predictions on -fisti-diitiibuution data. This claim is consolidated by the qualitative results on CIFAR10 and CIFARI 100 datasets. In contrast, MCD, BBB, and VSD are well-calibrated with a moderate level of confidence for in-distribution data. On CIFAR 10 and CIFAR 100 datasets, VSD gains better results with entropy values being distributed over a larger support, meaning that the predictions of VSD are closer to uniform on unseen classes.
We run similar experiment, in which we train AlexNet on CIFAR10 and use SVHN CIFAR100 as of-disiriibttionn data. The results are shown in the top row of Figure in Appendix G.I While MAP and VD still exhibit the same overconfident phenomenon as on LeNet, we observe the underconfident predictions of BBB and MC Dropout even on in-distribution data, which possibly leads to high uncertainty on ditto--dittribution data. We hypothesize that this is because the models trained with these methods are likely underfit with a low accuracy on the in-distribution training data. In contrast, VSD estimates reasonably the predictive entropy in both settings. The remaining scenario with 8 trained on CIFAR 100 is shown in the bottom row of Figure 4 with the same behaviors.
Conclusions
We proposed a novel approximate inference framework for Bayesian deep nets, named Variational Structured Dropout (VSD). In VSD, we learn a structured approximate posterior via the Dropout principle. VSD is able yield flexible inference while maintaining computational efficiency and scalability for deep convolutional models. The extensive experiments have evidenced the advantages of VSD such as well-calibrated prediction, better generalization, good uncertainty estimation. Given a consistent performance of VSD as presented throughout the paper, an extension of that method to other problems, such Bayesian active learning or reinforcement learning, of interest.
10

---

Acknowledgements
We would like thank the anonymous reviewers for their insightful comments and valuable sug- gestions. We also thank Ngo Trung Nghia (VinAl Research) for helpful discussions throughout this work. Khoat Than was funded by Gia Lam Urban Development and Investment Company Limited, Vingroup, and supported by Vingroup Innovation Foundation (VINIF) under project code VINIF,2019.DALD
References
[1] A. A. Alemi, B. Poole, I. S. Fischer, J. V. Dillon, R. A. Saurous, and K. P. Murphy. Fixing broken elbo. In International Conference on Machine Learning, 2018.
[2] O. Arenz, M. Zhong, and G. Neumann. Trust-region variational inference with Gaussian mixture models. Journal of Machine Learning Research, 21(163): 2020.
[3] A. Asuncion and D. Newman. UCI machine learning repository, 2007.
[4] Bartlett, D. Foster, and M. Telgarsky. Opectrally-normalln margin bounds for neural networks. arXiv preprint (rXiv:/706.08990) 2017.
[5] R. v. d. Berg. L. Hasenclever, J. M. Tomczak, and M. Welling. Sylvester normalizing flows for variational inference. arXiv preprint 1rX3vv1803.0053 2018.
[6] C. H. Bischof and X. Sun. On orthogonal block climination. Preprint MCS-P450-0794, Mathematics and Computer Science Division, Argonne National Laboratory, 1994.
[7] J. Blanchet, Y. Kang, J. L. M. Olea, V. A. Nguyen, and X. Zhang. Machine learning's dropout training is distributionally robust optimal. arXiv preprint arXiv:2009.66/ 11, 2020.
[8] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pages 1613-1622. PMLR, 2015.
[9] M. Burger and A. Neubauer, Analysis of tikhonov regularization for function approximation by neural networks. Neural Networks, 16(1):79-90, 2003.
[10] A. Camuto, M. Willetts, U. Şimşekli, S. Roberts, and C. Holmes. Explicit regularisation in Gaussian noise injections. arXiv preprint arXiv:2007.07366. 2020.
[1]] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statisticy, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[12] T. Cui, A. Havulinna, Marttinen, and S. Kaski. Informative Gaussian scale mixture priors for Bayesian neural networks. arXiv preprint 2rXiv:2002.10244: 2020.
[13] E. Daxberger, E. Nalisnick, J. U. Allingham, J. Antorán, and J. M. Hemnández-Lobato. Expressive yet tractable bayesian deep learning via subnetwork inference. arXiv preprins arXiv:2010. 14689, 2020.
[14] Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
[15] M. Dusenberry, G. Jerfel, Y. Wen, Y. Ma, J. Snoek, K. Heller, B. Lakshninarayanan, and D. Tran. Efficient and scalable Bayesian neural nets with rank-1 factors. In International conference on machine learning, pages 2782-2792. PMLR, 2020.
[16] B. Efron. Large-scale inference: empirical Bayes methods for estimation, testing, and prediction volume 1. Cambridge University Press, 2012.
[17] A. Foong, R. Burt, Y. Li, and R. Turner. On the expressiveness of approximate inference in Bayesian neural networks. arXiv preprint arXiv:1909.0011 2019.
[18] Y. Gal. Uncertainty in deep learning. University of Cambridge, 1(3), 2016.

---

[19] Y. Gal and Z. Ghahramani, Bayesian convolutional neural networks with bernoulli approximate variational inference. arXiv preprint
2015.
[20] Y. Gal and Z. Ghahramani. Dropout as Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050- 1059, 2016.
[21] S. Ghosh, J. Yao, and F. Doshi- Velez. Structured variational learning of bayesian neural networks with horseshoe priors. In International Conference on Machine Learning, pages 1744-1753. PMLR, 2018.
[22] A. Graves. Practical variational inference for neural networks. In Advances in neural information processing systems, pages 2348-2355 2011.
[23] C. Guo, G. Pleiss, Y. Sun, and K. Q- Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[24] A. K. Gupta and D. K. Nagar. Matrix variate distributions, volume 104. CRC Press, 2018.
[25] V. Gupta, T. Koren, and Y. Singer. A unified approach to adaptive regularization in online and stochastic optimization. arXiv preprint arXiv: (706006670 2017.
[26] D. P. Helmbold and P. M. Long. On the inductive bias of dropout. The Journal of Machine Learning Research, 16(1):3403-345, 2015.
[27] D. P. Helmbold and P. M. Long, Surprising properties of dropout in deep networks. The Journal of Machine Learning Research, 18(1):72847331. 2017.
[28] J. M. Herúnndez-Lobato and R. Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In International Conference on Machine Learning, pages 1861-1869, 2015.
[29] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Im- proving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 2012.
[30] G. Hinton and D. Van Camp. Keeping the neural networks simple by minimizing the descrip- tion length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pages 5-13, 1993.
[31] J. Hron, A. Matthews, and Z. Ghahramani. Variational Bayesian dropout: pitfalls and fixes, In International Conference on Machine Learning, pages 2019-2028. PMLR, 2018.
[32] P. Izmailov, W. J. Maddox, P. Kirichenko, T. Garipov, D. Vetrov, and A. G. Wilson. Subspace inference for bayesian deep learning. Uncertainty in Artificial Intelligence, pages 1169-1179. PMLR, 2020.
[33] M. I. Jordan, Z. Ghahramani, S. Jaakkola, and L. K. Saul, An introduction to variational methods for graphical models. Machine learning, 37(2):183-233 1999.
[34] N. S. Keskar and R. Socher. Improving generalization performance by switching from adam to sgd. arXiv preprint urXiv: 2017.
[35] M. Khan, D. Niclsen, V. Tangkaratt, W. Lin, Y. Gal, and A. Srivastava. Fast and scalable Bayesian deep learning by weight-perturbation in adam. In International Conference on Machine Learning, pages 261 2611-2620. PMLR, 2018.
[36] V. Kharitonov, D. Molchanov, and Vetrov. Variational dropout via empirical Bayes, arXiv preprint arXiv: /8/ 1.00599 2018.
[37] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.698 2014.
12

---

[38] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling. Improved vari- ational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743-4747 2016.
[39] D. P. Kingma, T. Salimans, and M. Welling. Variational dropout and the local reparameterization trick. In Advances in neural information processing systems, pages 2575-2581 2015.
[40] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114, 2013.
[41] D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[42] R. Krishnan, M. Subedar, and O. Tickoo. Specifying weight priors in bayesian deep neural networks with empirical bayes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages
2020.
[43] A. Krizhevsky, G. Hinton, et Learning multiple layers of features from tiny images. 2009.
[44] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv: 2016.
[45] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278- -2244, 1998.
[46] C. Louizos, K. Ullrich, and M. Welling. Bayesian compression for deep learning, arXiv preprint 1rXiv:1705.08665, 2017.
[47] C. Louizos and M. Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. In International Conference on Machine Learning, pages 1708-1716, 2016.
[48] C. Louizos and M. Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 2218 2227. JMLR. org, 2017.
[49] D. J. MacKay, A practical Bayesian framework for backpropagation networks. Neural compu- tation, 4(3):448 472, 1992.
[50] W. J. Maddox, P. Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson. A simple baseline for Bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32: 12:1335313116, 2019.
[51] S.-i. Maeda. A Bayesian encourages dropout. arXiv preprint arXiv: 14Xiv:1412.70033 2014.
[52] D. McAllester. pac-Bayeee Bayesiay tutorial with dropout bound. arXiv preprint arXiv: 1en..1107.11 2013.
[53] Mianjy and Arora. On dropout and nuclear norm regularization. In International Confer- ence on Machine Learning, pages 4575-4584, PMLR, 2019.
[54] P. Mianjy and R. Arora. On convergence and generalization of dropout training. Advances in Neural Information Processing Systems, 33, 2020.
[55] P. Mianjy, R. Arora, and R. Vidal. On the implicit bias of dropout. In International Conference on Machine Learning, pages 3537-3545, 2018.
[56] A. Mishkin, F. Kunstner, D. Nielsen, M. Schmidt, and M. E. Khan. Slang: Fast structured covariance approximations for Bayesian deep learning with natural gradient. In Advances in Neural Information Processing Systems, pages 6245-6255. 2018.
[57] D. Molchanov, Ashukha, and D. Vetrov. Variational dropout sparsifies deep neural networks.
In International Conference Machine Learning, pages 2498-2500 PMLR, 2017.
13

---

[58] D. Molchanov, V. Kharitonov, A. Sobolev, and D. Vetrov. Doubly semi-implicit variational inference. The 22nd International Conference on Artificial Intelligence and Statistics, pages 25593-2602, 2019.
[59] W. Mou, Y. Zhou, J. Gao, and L. Wang. Dropout training, data-dependent regularization, and generalization bounds. International Conference on Machine Learning, pages 3645-3653, 2018.
[60] J. Mukhoti, P. Stenetorp, and Y. Gal. On the importance of strong baselines in Bayesian deep learning. arXiv preprint 1rXiv:1811.09385. 2018.
[61] M. P. Nacini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
[62] E. Nalisnick, J. M. Hernández-Lobato, and P. Smyth. Dropout as a structured shrinkage prior. In International Conference on Machine Learning, pages 4712-4722. PMLR, 2019.
[63] R. M. Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1995.
[64] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[65] B. Neyshabur, S. Bhojanapalli, and N. Srebro. A pac-Bayesian approach to spectrally- normalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564. 2017.
[66] C. Oh, Adamczewski, and M. Park. Radial and directional posteriors for Bayesian neural networks. arXiv preprint
2019.
[67] V. M. Ong. D. J. Nott, and M. S. Smith. Gaussian variational approximation with factor covariance structure. Journal of Computational and Graphical Statistics, 27(3):465 478, 2018.
[68] K. Osawa, Swaroop, A. Jain, R. Eschenhagen, R. E. Turner, R. Yokota, and M. E. Khan. Practical deep learning with Bayesian principles. arXiv preprint arXiv: 1906699060066 2019.
[69] R. Ranganath, D. Tran, and D. Blei. Hierarchical variational models. In International Conference on Machine Learning, pages 324-333, 2016.
[70] D. Rezende and S. Mohamed. Variational inference with normalizing flows. In International Conference Machine Learning, pages 1530-1538. PMLR, 2015.
[71] H. Ritter, A. Botev, and D. Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings, volume 6. International Conference on Representation Learning, 2018.
[72] S. Rossi, S. Marmin, and M. Filippone. Walsh-hadamard variational inference for Bayesian deep learning. arviv preprint arXiv: 1248, 2019.
[73] L. Sagun, U. Evei, V. U. Guney, Y. Dauphin, and L. Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.044
2017.
[74] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):199--1958 2014.
[75] S. Sun, C. Chen, and L. Carin. Learning structured weight uncertainty in Bayesian neural networks. In Artificial Intelligence and Statistics, pages 1283-1292, 2017.
[76] X. Sun and C. Bischof. A basis-kernel representation of orthogonal matrices. SIAM journal on matrix analysis and applications, 16(4): (6(4):1184411
1995.
[77] Swiatkowski, K. Roth, B. Veeling, L. Tran, Dillon, J. Snoek, S. Mandt, Salimans, R. Jenatton, and S. Nowozin. The k-tied normal distribution: A compact parameterization of Gaussian mean field posteriors in Bayesian neural networks. In International Conference on Machine Learning, pages 9289-9299. PMLR, 2020.
14

---

[78] M. Teye, H. Azizpour, and K. Smith. Bayesian uncertainty estimation for batch normalized deep networks. In International Conference on Machine Learning, pages 4907-4916. PMLR, 2018.
[79] J. M. Tomezak and M. Welling. Improving variational auto-encoders using convex combination linear inverse autoregressive flow. arXiv preprint arXiv:/706.02.222 2017.
[80] M. Tomezak, S. Swaroop, and R. Turner. Efficient low rank Gaussian variational inference for neural networks. Advances in Neural Information Processing Systems, 33, 2020.
[81] Wager, Wang, and P. S. Liang. Dropout training as adaptive regularization. In Advances in Neural Information Processing Systems, pages 351-359, 2013.
[82] S. Wang and C. Manning. Fast dropout training. In international conference machine learning, pages 118-126. PMLR, 2013.
[83] C. Wei, S. Kakade, and T. Ma. The implicit and explicit regularization effects dropout. In International Conference Machine Learning, pages 10181-10192. PMLR, 2020.
[84] Wenzel, K. Roth, B. S. Veeling, J. Šwiętkowski, L. Tran, S. Mandt, Snoek, T. Salimans,
R. Jenatton, and S. Nowozin. How good is the Bayes posterior in deep neural networks really? arXiv preprint arXiv:2002.02405. 2020.
[85] A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The marginal value of adaptive gradient methods in machine learning, arXiv preprint (5K.v::700500292, 2017.
[86] A. G. Wilson and P. Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. arXiv preprint arXiv:2002.0879)) 2020.
[87] A. Wu, S. Nowozin, E. Meeds, R. E. Turner, J. M. Hernandez-LLoate and A. L. Gaunt. Deterministic variational inference for robust bayesian neural networks. arXiv preprint arXiv: 181000388 2018.
[88] M. Yin and M. Zhou. Semi-implicit variational inference. In International Conference Machine Learning, pages 5660-5669. PMLR, 2018.
[89] G. Zhang, S. Sun, D. Duvenaud, and R. Grosse. Noisy natural gradient as variational inference.
In International Conference on Machine Learning, pages 5852-5861. PMLR, 2018.
[90] Zhang, Q. Lei, and I. Dhillon. Stabilizing gradients for deep neural networks via efficient svd parameterization. In International Conference on Machine Learning, pages 5806-5814 PMLR, 2018.
[91] H. Zhao, Y.-H. H. Tsai, R. Salakhutdinov, and G. J. Gordon. Learning neural networks with adaptive regularization. arXiv preprint arXiv:1907.06288, 2019.
[92] S. Zhao, J. Song, and S. Ermon. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 5885-5892, 2019.
[93] Zhou, J. Feng, C. Ma, C. Xiong, S. Hoi, et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. arXiv preprint 1rXiv:2010.05627, 2020.
[94] O. Zobay et al. Variational Bayesian inference with Caussian-mixtun approximations. Elec- tronic Journal of Statisties, 8(1):355-38, 2014.
15


Reviews:
[{"review_id": "x0bpyS1p2-", "reviewer": "Reviewer_6pWo", "summary": "This paper presents a novel variational structured approximation of the posterior distribution of Bayesian neural networks by the Bayesian interpretation of Dropout regularization.   An orthogonal transformation is used to learn a structured representation of the variational noise and consequently induces statistical dependencies in the approximate posterior. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The method is new and interesting. \n\nQuality: Technically sound with extensive experiments. \n\nClarity: Clearly written. \n\nSignificance: Compared with previous variational-based methods, the proposed VSD achieves good results by incurring structure into the covariance, which is expected and reasonable.  The main concern is its importance of practical use. For example, in comparison with the standard non-variational baseline such as SWAG, which serves as a simple baseline for Bayesian uncertainty in deep learning,  the results in Table 2 and Table 3 show that in most cases the proposed VSD performs worse than SWAG. It then raises the long-standing question that why the Bayesian methods, though with high complexity,  do not even yield competitive results as compared with simpler methods. It would be appreciated if a comprehensive comparison of VSD and the baseline SWAG, which is simple modification of non-bayesian methods to obtain uncertainty estimates,  is further clarified. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. There is a lack of comparison with non-variational baselines SWAG and SWAG in terms of predictive uncertainty in the out-of-distribution settings, as shown in Figure 3. \n\n2. The results in Table 1-3 lacks the variances for the random runs. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "uk5A_314q9g", "reviewer": "Reviewer_idcU", "summary": "This paper introduces Variational Structured Dropout (VSD), a scalable method building on previous Variational Dropout methods, and with a structured representation of the variational noise. The paper derives the method, introduces a hierarchical prior, and provides results on many benchmarks while comparing to many baselines. VSD performs reasonably well in all benchmarks on uncertainty metrics and accuracy. VSD also performs well on out-of-distribution uncertainty experiments.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is a good paper and a nice method. There is good motivation, good explanations and a thorough experiments section, where VSD is seen to perform well. Overall I like the paper, and so I focus the rest of the review on a few questions that, if answered, could increase my score:\n\n1. In Equation 4, the authors add a mutual information term to the variational lower bound. Is this new objective still a valid lower bound of the model evidence? It apears to me like it might not be due to the mutual information term. If indeed it is no longer a valid lower bound, I think the authors should make it clearer in the text.\n- Also, in Appendix I.1, the authors describe the KL annealing that they use, where they weigh the KL term by $\\lambda$. For $\\lambda \\neq 1$, the objective is no longer a valid lower bound. I think it is fine to do this (as it has been found to help in the past), but I think this additional hyperparameter should be explicitly mentioned in the main text somewhere.\n2. The results are all means over 5 random seeds. I would like to see standard deviations too, either in the main text or in additional tables in the Appendix.\n\nAdditional, more minor points:\n\n3. I think it would be nice to have some further intuition on what T=1 vs T=2 means.\n4. The authors could also compare to natural-gradient VI methods like VOGN (Osawa et al., 2019: \"Practical Deep Learning with Bayesian Principles\").\n\n\n---------- After author rebuttal -----------\nThanks to the authors for responding to my questions and concerns. They answered many of my questions. However, given very valid concerns from other reviewers (particularly the discussion with Reviewer VNbM), I have decided to keep my score and not increase it. I do this with the expectation that the authors will fix/clarify many of Reviewer VNbM's concerns, such as:\n- Being explicit in the text about the potential limitation that the new ELBO has a relatively poor regularisation, which may be the reason that the hyperprior is necessary and works well.\n- More importantly, being explicit in the text regarding *exactly* which implementation is used for which experiment, with key differences in the main paper, and smaller settings in an Appendix. Reviewer VNbM brought up a very important point regarding implementations and this should really have been in the submitted draft of the paper itself.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors could add a line or two on potential societal impacts, but this is not necessary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "lCA37mntphG", "reviewer": "Reviewer_VNbM", "summary": "The paper proposes a new extension to the well-known and widely used Monte-Carlo dropout scheme as variational approximation to the Bayesian posterior.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall difficult to read and understand, especially the discussion on the different parameterization choices.\nInspired by the MC-dropout, the Authors end up with a very involved parameterization of the posterior and to achieve this they propose a lot of incremental improvements and changes, which are not well motivated, in my opinion. \n\nBelow some detailed comments to justify my overall score:\n\n- From the presentation of the Householder parameterization in Sec. 3.1 (and around line 136), it seems that the $\\Sigma$ is full-rank.\nThis is possible only when the number of transformations via $H$ (T) is equal to the dimensions of $\\Sigma$ (K). \nIn other cases, this could be considered as low-rank parameterization of the covariance.\n- The parameterization of the vectors $v_t$ as a flow (i.e. $v_t = f(v_{t-1})$) is not discussed in the paper, but only after line 528 in the Appendix. \nThis is an important modeling choice that should have been present in the main paper. \n- The Authors propose to run an empirical Bayes procedure for choosing the prior parameters but the effect of this choice is not assessed. Minor: From the discussion around line 192 it might be confusing for a reader why $\\beta$ does not appear in Eq. 5; maybe it would be better to first show the full KL (Eq. 11 in Appendix) and then show the optimized one. As an alternative, you could use a different symbol for the optimized KL (like $\\tilde{\\mathbb{D}}_{KL}$).\n- Still regarding the empirical Bayes, given that the $\\beta$ is a matrix with same dimensions of $\\mathbf W$, would it be possible for the KL to collapse to 0? In the minimization of the variational objective, all it needs is for $\\alpha_j \\rightarrow 0$ and it's not clear how this could be prevented. Did you experienced this phenomenon in your experiments?\n- Connected with the point above, I can see why the Authors needed a hierarchical parameterization of the prior to enforce a stronger regularization. Regarding the hierarchical prior, the main paper doesn't discuss the parameterization of $p(z)$ and $q(z)$ (some details are present in the Appendix but should be reported also here).\nMinor: For Figure 2 with the double y-axis it's not possible to understand which axis is corresponding to which dataset.\n\nTwo important remarks/contradictions after checking the submitted code, which are not discussed anywhere:\n\n- From the presentation of Sec. 4.2 I seems that the Authors used this approach also for convolutional layers. This indeed is not what is happening, or at least this is not what is implemented in the submitted code, where the weights of the convolutional layers are simply optimized. The extension from fully-connected layers to convolutional ones should be fairly simple.\n- From the interpretation of line 233, I seems that the Authors will finally consider only the version with the hierarchical prior as their final proposal but, again, the code implements only the previous one (see for example classification/model.py:162, which implements only Eq. 5)\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and social impact are not discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "f2frPKktuiA", "reviewer": "Reviewer_uDiH", "summary": "The paper proposes Variational Structured Dropout, an extension to variational dropout that uses a full covariance matrix to model the posterior at the pre-activations. This covariance matrix, after spectral decomposition, is parameterized as the product of T Householder matrices.\n\nThe new parameterization of the posterior does not suffer from the singularity issue of Variational Dropout. When using a diagonal posterior approximation, the singularity issue is that the approximate posterior has zero measure, which results in the KL-term being undefined. VSD avoids this by using a full covariance matrix thus ensuring that the approximate posterior has non-zero measure.\n\nTo train the model using the new parameterization, the authors optimize a new variational lower bound. This lower bound is a lower bound to the ELBO.\n\nRegarding the prior, the paper advocates for using a hierarchical prior distribution where the prior is a zero mean Gaussian distribution over the weights, with the hyperprior determining the standard deviation.\n\nThe experiments first examine the impact of T, the number of matrices used to parameterize the approximate posterior. Next, large scale experiments on deep neural networks demonstrate the performance of VSD against a number of baseline Bayesian inference methods measuring both predictive performance and predictive entropy (for OOD detection).\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality:\n\nI find the idea of VSD quite interesting and it is, to my knowledge, novel. It is an excellent observation that one can address the singularity issues using a full covariance approximate posterior. Moreover, it can also be understood as a novel type of low rank parameterization of the approximate posterior.\n\nRegarding the hierarchical prior, I have not seen a hierarchical prior over the weights in this form, although it is quite possible that similar ideas have been proposed before.\n\n### Quality:\n\nThe work is high quality. It is well motivated, it addresses the concerns with variational dropout and it also improves performance.\n\nI have a slight concern regarding the new variational lower bound $L_{MI}$ . it is unclear to me what the effect of discarding the mutual information is. I would expect that with $L_{MI}$, we get highly correlated weights posteriors. Why doesn’t this training objective result in the weight posterior having high correlations, thus having very low probability mass?\n\nThe experimental section of the paper could be improved. It is good to see that numerous benchmarks and baseline methods are used, but all the methods seem to be underperforming. For instance, the ResNet-18 results are lagging significantly behind the reference implementations available at https://github.com/google/uncertainty-baselines/tree/main/baselines/cifar . Apart from this, the experiments look at both predictive performance and predictive entropy (for OOD samples), and they show very promising results.\n\nRegarding the derivations in the supplementary material: I read the results, but I did not carefully check them.\n\n### Clarity:\n\nThe paper does an excellent job at presenting the method, explaining the motivation and describing related works. The paper flows very well, each section leading into the next. \n\n### Significance:\n\nThe work has potential impact for both researchers and deep learning practitioners. For researchers, it is certainly interesting how the work addresses the theoretical concerns with VD. For practitioners, VSD could be an effective approach at capturing model uncertainty depending on the computational costs.\n\n\n### Further questions:\n\n* The paper mentions that using T > 2 has high computational cost. Is this because pre-activations are costly to compute, or that the method converges too slowly? What is the computational cost and complexity associated with the VSD parameterization?\n* How many samples (or forward passes) are used for prediction?\n* What hyperparameter values $\\alpha$ and $\\beta$ were used in the experiments?\n\n\n### Minor:\n\n* Line 235 convolutional instead of convolution?\n* Many of the references, such as [26] and many others cite Arxiv versions of papers instead of the versions that are published at ML conferences. Also names such as Bernoulli are sometimes not capitalized correctly.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limits and societal impacts are addressed adequately.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes structural variational dropout for BNNs. 

Reviewers think the proposed methodology is novel, and the paper's theoretical contribution is significant, in the sense that it addresses a well-known theoretical issue of variational dropout & MC-dropout. Given that MC-dropout is quite often used in practice, such theoretical contribution is useful.

Still the paper has the problem that, some reviewers are concerned with the experiments using implementations that are different from the version used in theoretical analysis. So in revision, either this difference needs to be explained (if it is a confusion), or, the practical approach needs to be justified better to so that they do not have the infinite KL issues of MC-dropout.
Decision: Accept (Poster)