Learnable Uncertainty under Laplace Approximations
Authors: Agustinus Kristiadi
Matthias Hein
Philipp Hennig
Venue:  (2021)
Keywords: Bayesian deep learning
Laplace approximations
uncertainty quantification

Abstract: Laplace approximations are classic, computationally lightweight means to construct Bayesian neural networks (BNNs). As in other approximate BNNs, one cannot necessarily expect the induced predictive uncertainty to be calibrated. Here we develop a formalism to explicitly "train" the uncertainty in a decoupled way to the prediction itself. To this end we introduce uncertainty units for Laplace-approximated networks: Hidden units with zero weights that can be added to any pre-trained, point-estimated network. Since these units are inactive, they do not affect the predictions. But their presence changes the geometry (in particular the Hessian) of the loss landscape around the point estimate, thereby affecting the network's uncertainty estimates under a Laplace approximation. We show that such units can be trained via an uncertainty-aware objective, making the Laplace approximation competitive with more expensive alternative uncertainty-quantification frameworks.

Under review as a conference paper at ICLR 2021
LEARNABLE UNCERTAINTY UNDER LAPLACE AP- PROXIMATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Laplace approximations are classic, computationally lightweight means for con- structing Bayesian neural networks (BNNs). As in other approximate BNNs, one cannot necessarily expect the induced predictive uncertainty to be calibrated. Here we develop formalism to explicitly "train" the uncertainty in decoupled way to the prediction itself. To this end we introduce uncertainty units for Laplace- approximated networks: Hidden units with zero weights that be added any pre-trained, point-estimated network. Since these units are inactive, they do not affect the predictions. But their presence changes the geometry (in par- ticular the Hessian) of the loss landscape around the point estimate, thereby af- fecting the network's 's uncertainty estimates under Laplace approximation. We show that such units can be trained via an uncertainty objective, making the Laplace approximation competitive with more expensive alternative uncertainty- quantification frameworks.
INTRODUCTION
The point estimates of neural networks (NNs) construcnt maximum a posteriori (MAP) es- timates via (regularized) empirical risk minimization empirriallly achieve high predictive perfor- mance. However, they tend underestimate the uncertainty of their predictions, leading to an overconfidence problem (Hein et al., 2019), which could be disastrous in safety-critical applications such as autonomous driving. Bayesian inference offers a principled path to overcome this issue. The goal is to turn a "vanilla" NN into a Bayesian neural network (BNN), where the posterior distribu- tion over the network's weights are inferred via Bayes' rule and subsequently taken into account when making predictions. Since the cost of exact posterior inference in a BNN is often prohibitive, approximate Bayesian methods are employed instead.
Laplace approximations (LAs) are classic methods for such purpose (MacKay 1992b). The key idea is to obtain an approximate posterior by "surrounding" a MAP estimate of a network with a Gaussian, based on the loss landscape's ' geometry around it. A standard practice in LAs is a single hyperparameter-tthe prior precision- which is inflexible (Ritter et al.] 2018h Kristiadi al. 2020). Here, we aim improving the flexibility of uncertainty tuning LAs. To this end, we introduce Learnable Uncertainty under Laplace Approximations (LUZA) units, which are hidden units associated with a zeroed weight. They can be added the hidden layers of any MAP-trained network. Because they are inactive, such units do not affect the prediction of the underlying network. However, they can still contribute to the Hessian of the loss with respect the parameters, and hence induce additional structures to the posterior covariance under a I.A. LULA units can be trained via an uncertainty-awarc objective (Hendrycks al. 2019 Hein et al., 2019, etc.), such that they improve the predictive
(UQ) performance of the Laplace-approximate BNN. Figure demonstrates trained LULA units in action: They improve the UQ performance of a standard LA, while keeping the MAP predictions in both regression and classification tasks.
In summary, we (i) introduce LULA units: inactive hidden units for uncertainty tuning of a LA, (ii) bring robust training technique from non- -ayessan literature for training these units, and (iii) show empirically that aUgA-augmented Laplace-apy approximapo BNNs can yield better UQ performance compared to both previous tuning techniques and contemporary, more expensive baselines.

---

Under review as a conference paper at ICLR 2021
1.0
0.7
0.5
(a) MAP
(A)
(c) LA+LULA
Figure Predictive uncertainty of a (a) MAP-trained, (b) Laplace-approximated (LA), and (c) AULL-augmented applace-pproximmaee (LA+LULA) neural network regression (top) and clas- sification (bottom) tasks. Black curve represent predictive mean and decision boundary, and shade represents =3 standard deviation and confidence in regression and classification, respectively. MAP overconfident and LA can mitigate this. However, LA can still be overconfident away from the data. LULA improve LA's uncertainty further without affecting its predictions.
BACKGROUND
2.1 BAYESIAN NEURAL NETWORKS
Let f R. x Rd R* defined by (z, 0) [ax; 0) be I--layer neural network. Here, O is the concatenation of all the parameters of f. Suppose that the size of each layer of f is given by the Z Then, for each I...., the /-th layer of f defined by
sequence of (ny p
a 1 = (0 6()
with 50 =
(p(a a0
< ifl= L,
(1)
where (1() X and ((0) E I are the weight matrix and bias vector of the layer, and yo is component-wee activation function. We call the vector h()) E Th the 1-th hidden units of Note that by convention, we consider no == and n2 := k; while (0) == and (L) = f(;
From the Bayesian perspective, the ubiquitous training formalism of neural networks amounts to MAP estimation: The empirical risk and the regularizer are interpretable as the negative log- likelihood under an i.i.d. dataset D = (xi, )m and the negative log-prior, respectively. That is, the loss function interpreted as
((0) = Σ
logp(un (x;;0) log:(()) log 2).
In this view, the de facto weight decay regularizer amounts to zero-mean isotropic Gaussian prior p(0) (ee I1) with a scalar precision hyperparameter X. Meanwhile, the usual softmax and quadratic output losses correspond to the Categorical and Gaussian distributions over in the
case of classification and regression, respectively.
MAP-trained neural networks have been shown to be overconfident (Hein et al. 2019) and BNNs can mitigate this issue (Kristiadi et al.) 2020). They quantify epistemic uncertainty by inferring the full posterior distribution of the parameters 0 (instead of just single point estimate in MAP training). Given that p(0 D) the posterior, then the prediction for any test point x R is obtained via marginalization
ply x D) = p(y) [(z;0))p(0 p(O D) do,
(3)
which captures the uncertainty encoded in the posterior.
2

---

Under review as a conference paper at ICLR 2021
2.2 LAPLACE APPROXIMATIONS
In deep learning, since the exact Bayesian posterior is intractable, approximate Bayesian inference methods are used. An important family of such methods is formed by LAs. Let OMAP be the mini- mizer of (2), which corresponds to a mode of the posterior distribution. A LA locally approximates the posterior using a Gaussian
p(0 D) = N( 0MAP, Σ) == : (() OMAP, ((exA)))))
Thus, LAs construct an approximate Gaussian posterior around MAP,, whose precision equals the Hessian of the loss at t the "curvature" of the loss landscape at θMAP- While the covariance of a LA is tied to the weight decay of the loss, a common practice in LAs is to tune the prior precision under some objective, inference are thought as separate, independent processes. For example, given a fixed MAP estimate, one can maximize the log-liielihood of LA w.r.. the prior precision obtain the covariance. This hyperparameter tuning can thus be thought as an uncertainty tuning.
post-hoc manner. In other words, the MAP estimation and the covariance
A recent example of LAs is the Kronecker-far factored Laplace KKFL,Ritter et al. 2018b). The key idea is to approximate the Hessian matrix with the layer-wise Kronecker factorization scheme proposed by [Heskes] 2000); Martens & Grosse 2015). That is, for each layer 1 = 1,..., T., KFL. assumes that the Hessian corresponding to the I-th weight matrix W E R can be written as the Kronecker product G(4) & (4) for some G E RR and A E R This assumption brings the inversion cost of the Hessian down to (n)) + -n ,. instead of the usual (n cost.
The approximate Hessian can easily be computed via tools such as BackPACK (Dangel al. 2020
Even with a closed-form Laplace-approximated posterior, the predictive distribution (3 in general does not have analytic solution since f is nonlinear. Instead, one can employ Monte-Carlo (MC) integration by sampling from the Gaussian:
8 S Σ ply (M:1((;00))
plu x.
D) D)
0, ~NO OMAP, Σ),
for S numbeer of samples. In the case of binary classification with f R" x Rd + R, one can use the following well-known approximation, due to MMcKay (1992a):
f(r; MAPP)
pyy z. =)
(4)
VI+ 7//0(zz)
where a is the sigmmod function and v(z) is the marginal variance of the network output f(x). which is often approximated via lincarization of the network around the MAP estimate:
((x) ~ (Vof(z; )) Σ of 0)
(5)
(This approximation has also been generalized to multi-class classifications by Gibbs (1997)) ).) In particular, as o(z) increases, the predictive probability of y = I goes to 0.5 and therefore the un- certainty increases. This relationship has also been shown empirically in multi-class classifications with MC-integration (Kristiadi et al. 2020)..
LULA UNITS
The problem with the standard uncertainty tuning in LAs is that the only degree-of- of-freedom available for performing the optimization is the scalar prior precision and therefore inflexible ! We shall address this by introducing "uncertainty units", which can be added on top of the hidden units of any MAP-trained network (Section 3.1) and can be trained via uncertainty-awant loss (Section 3.2)
3.1 CONSTRUCTION
Let f R x Rd + R be a MAP-trained L-layer neural network with parameters MAAP (W (1) 1L
The premise of our method is simple: At each hidden layer = 1,..., L 1, MAP: MAP
'While one can also use non-scalar prior precision, it appears to be uncommon in deep learning. In any case, such element-wise weight-cost would interact with the training procedure.
3

---

Under review as a conference paper at ICLR 2021
(24
f(z)
Figure 2: illustration of the proposed construction. Rectangles represent layers, solid lines represent connection between layers, given by the original weight matrices W((1) MAP..... w The additional units represented by the additional block the bottom of each layer. Dashed lines correspond to the free parameters W(1), ,r.,, W(L-1), while dotted lines to the zero weights.
suppose we add my E Zoo additional hidden units, under the original activation function, to h((). As consequence, we need to augment each of the weight matrices accommodate them.
Consider the following construction: for each layer = 1...., L of the network, we expand ( and (()) to obtain the block matrix and vector
(w
(1)
O = MAP
(l)e+mm))(n) 14m; 1); ((0)
MAP D
ER
(6)
WO
respectively, with mi since we not add additional units to the input. For I I,, we define
( = (wilh) 0) E 12xx(12-1tm
((L) = MMPP E R²
so that the output dimensionality is unchanged. For brevity, we denote W = (W W Refer to Figure 2 for an illustration and Algorithm 2 Appendix B for a step-by-step summary. Taken together, we denote the resulting augmented network as f and the resulting parameter vector as 8MPP Rd where d it the resulting number of parameters. Note that construction to convolutional nets by expanding the "channel" of a hidden layer
easily extend this
Let inspect the implication of this construction. Here for each 1,..., T 1, since they zero, the upper-right quadrant of ( deactivates the m,-1 additional hidden units the previous layer, thus they do not contribute to the original hidden units in the I-th layer. Meanwhile, the sub- matrix ( and the sub-vector 800) contain parameters for the additional my hidden units in the I-th layer. We are free to choose the the values of these parameters since the following proposition guarantees that they will not change the output of the network (the proof in Appendix A).
Proposition Let f: 12 x Rd + R2 a MAP-trained L-layer network parametrized by MAP. Suppose f I= x Idd + R and 0MAP E Bd are obtained via the previous construction. For any input E R", we have f(x; 0MAP) = fx; 0MAP).
So far, it looks like all our changes to the network are inconsequential. However, they do affect the curvature of the landscape of c and thus the uncertainty arising in LA. Let be a random variable in T and fMAP be instance of it. Suppose have p(0 D) R N/@ BMAP, Σ) over 0. where the covariance Σ is the inverse Hessian of the negative rog-posterior w.r.t. the augmented parameters at 6MPP Then, E contains additional dimensions (and thus in general, additional structured, non-zero uncertainty) absent in the original network, which depend on the values of the free parameters (W(()) BO0
posterior
'E.g. if the hidden units are a 3D array of (channel x height x width), then we expand the first dimension. Mooe formally: The principal curvatures of the graph of seen d-dimensional submanifold of Rd+1
4

---

Under review as a conference paper at ICLR 2021
The implication of the previous finding can be seen clearly in real-valued networks with diagonal LA posteriors. The following proposition shows that, under such a network and posterior, the construc- tion above will affect the output uncertainty of the original network f (the proof is in Appendix [A}.
Proposition 2. Suppose f R R= + I. is a real-valued network and f is as constructed above. Suppose further that diagonal caplace-approximater posteriors NOO 0MAP, diag(o)), N 0, diag(ō)) employed. Using the linearization (5) for any input E R", the variance over the output fz; 8) is at least that of (x;
In summary, the construction along with Propositions D and imply that the additional hidden units we have added to the original network are uncertainty units under LAs, i.e. hidden units that only contribute the approoxiappo uncertainty and not the predictions. This property gives rise to the name Learnable Uncertainty under Laplace Appraximations (LULA) units.
3.2 TRAINING
We have seen that by adding LULA units to network, we obtain additional free parameters that only affect uncertainty under LA. These parameters are thus useful for uncertainty calibration. Our goal is therefore to train them to induce low uncertainty over the data (inliers) and high uncertainty on outliers- the so-called out-o--distiibution (OOD) data. Specifically, this can be done by mini- mizing the output variance over inliers while maximizing it over outliers. Note that using variance makes sense in both the regression and classification cases: In the former, this objective directly maintains narrow error bars near the data while widen those far-away from them-cf. Figure (c, top). Meanwhile, in classifications, variances over function outputs directly impact predictive confidences, have noted in the discussion of (4 higher variance implies lower confidence.
Thus, following the contemporary technique from non-Bayesian robust learning literature Hendrrycks al. 2019 Hein et al. 2019 etc.), construct the following loss. Let IR Rd
R be an L-layer neural network with a MAP-trained parameters OMAP and let J R Rd + R
along with MAP be obtained by adding LULA units. Denoting the dataset sampled i.i.d. from the data distribution Din and that from some outlier distribution Dour, we define
LLALLA(OMAP) = [Dinl
Σ Enn
v(((Tin;; (MAP) Douul
Σ ZanEDan
((((aau); TMAP),
(7)
where v(f(x); TMAP) is the total variance over the k components of the network out- put (x; (); .,. fa (2;; 0) under the apploxc-app posterior p(0 D) Ar@i I MAP., E(GMAP)), which can be approximated via an S-samples MC-integral
v v(F(2):0MAP) = var (0OD) Oz)
(8)
2 - 1 F(xz )=::)))
[z: 0.)2
with 0, p(0 D).
S
S
Here, for clarity, we have shown explicitly the dependency of Σ on ŌAP.. Note that set Din to be the training set of dataset. Furthermore, throughout this paper, we use the simple OOD dataset proposed by Heein al., (2019) which is constructed via permutation, blurring, and contrast rescaling of the in-distribution dataset Din- As we shall show in Section this artificial, un- informative OOD dataset is sufficient for obtaining good results across benchmark problems. More complex dataset as Donr might improve LULA's performance further but is not strictly necessary.
simply
Since our aim is solely improve the uncertainty, and biases in MAP- in accordance to (6) This can simply be enforced via gradient masking: For all
maintain the structure of all weights
1 1= 1,..., L - 1, set the gradients of the blocks of () and 7(0) not corresponding to W) and 700) respectively, to zero. Furthermore, since the covariance matrix E(0) of the
posterior is a function of 0, it needs to be updated every iteration during the optimization of LLULA-
5

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Training LULA units.
Input:
MAP-trained network f. Dataset D, OOD dataset D Learning rate a. Number of epochs E.
1: Construct f from f by following Section 3.1
2: for i 1, 1,..., E do
3: p(0 D) Rs NN [MAPP E((MAA))
Obtain Laplace-approximatie posterior of
4: Compute LLULA (OUL)()MP)) via (7) using p(0
D). D, and Don
5: VLLULA (OMAP)
6:
mask gradient())
Zero out the derivatives not corresponding to 0
7: MAP AMAP - 0g
8: end for
9:
p0 D= R N() 0MAP, E((MAP)
> Obtain the final Laplace approximation
10: return f and p(0 D)
The cost scales in the network's depth and can thus be expensive. Inspired by recent findings that last-layer Bayesian methods are competitive to all-layer alternatives (Ober & Rasmussen 2019), we thus consider last-layer LA (Kristiadi al. 2020 hidden layer and assume that the first L-2 layers are learnable feature extractor. We use a diagonal last-layer Fisher matrix to approximate the last-layer Hessian. Note that backpropagation through this matrix does not pose a difficulty since modern deep learning libraries such as Py PyTorch and TensorFlow supports "double backprop" (backpropagation through a gradient) efficiently. Finally, the loss LLULA can be minimized using standard gradient-base optimizers. Refer to Algorithm for summary.
proxy: We apply LA only the last
Last but not least, the intuition of LULA training is as follows. By adding LULA units, we obtain an augmented version of the network's loss landscape. The goal of LULA training is then exploit the weight-space symmetry (i.e. different parameters but induce the same output) arisen from the construction as shown by Proposition | and pick one of these parameters that is symmetric to original parameter but has "better" curvatures. Here, we define "good curvature" in terms of the above objective. These curvatures, then, when used a Laplace approximation, could yield better uncertainty estimates compared to the standard non-LULA mon-LULA-augmmnnt auggmenttd Laplace approximations.
RELATED WORK
While traditionally hyperparameter optimization LA requires re- ne-training of the network- under
the evidence framework (MacKay 1992b) empirical Bayes (Bernardo & Smith (009). tuning it in post-hoc manner has increasingly becomes a common practice. Ritter et al. (2018a b) tune the prior precision of LA by maximizing the predictive log-likelihood. Kristiadi et al. (2020)
extend this procedure by also using OOD data to better calibrate the uncertainty. However, they are limited in terms of flexibility since the prior precision of the LAs constitutes to just a single parameter. LULA can be seen as an extension of these approaches with greater flexibility and is complementary them since LULA is independent to the prior precision.
Confidence calibration via OOD data has achieved state-of-ther -ie-thherrt art performance in Bayessia outlier detection. Henddrycks et al. (2019); Hein et al. (2019); Meinke & Hein (2020) use OOD data to regularize the standard maximum- likelioood training. Malinin & Gales 2018 2019) use OOD data to train probabilistic models based on the Dirichlet distribution. All these methods are non-Bayesian and non-post-hoc. Our work is thus orthogonal since aim at improving class of Bayesian models in a post-hoc manner. LULA can be seen as bringing the best of both worlds: Bayesian uncertainty that is tuned via the state-of-the- non- -ay-sia technique.
EXPERIMENTS
In this section, we focus on classification using the standard OOD benchmark problems. Supple- mentary results regression are in Section C.2

---

Under review as a conference paper at ICLR 2021
Table Average confidences (MMCs in percent) over ten prediction runs. Lower is better for OOD data while higher is better for in-distribution data. See Table for the AUR values.
Train Test
MAP
KFL
KFL+LL
KFL+OOD
KFL+LULA
MNIST
98.0+0.0
97.4+0.0
98.0+0.0
96.0=0.1
98.0=0.0
9..
96.6+0.1
E-MNIST
81.4:00.0
76.710.0
79.7:1.00
72.1::02
81.3100
73.7.0.2
71.4.10.4
F-MNIST
76.62:0.0
67.8+0.0
74.7=0.0
65.8+0.4
76.5-0.0
67.420.4
49,4=119
Unilorm Noise
99.51.0.0
97.0.10.0
98.8110.0
86.31114
99.510.0
8988.0.9
338812.5
Asymptotic
100.040.00
88.0+0.0
75.9++0.0
4..9++0.9
97.3=0.1
50.4+1..
46.0#2266
CIFAR-I0
95.2++0.0
79.0+0.0
92.9++..0
91.0=10.2
93.4=0.1
85.3+1066
88.8:10.3
SVHN
75.5+0.00
62.740.0
65.7=0..0
61.5+00.6
68.4=0.5
52.3+1.1
37.5:11.77
LSUN-CR
61.2.10.0
39.7.10.2
48.0.10.0
48.1
54.5100.6
40,911.0
23.1.1..5
CIFAR-100
79.3+0.00
56.1+-000
69.5
67.3
73.5=0.2
07.4=088
63.2++0.4
Uniform Noise
51.540.00
72.4:0.00
33,1:10.0
43.5
46.5-0.6
43.5+:1.1
34,4119
Asymptotic
100.010..0
49.340.1
41.2.10.0
78.912200
90.411.0
61.31200
41.41:000
SVHN
97.7+0.00
92.7++0.0
94.4
95.1=0011
96,4=0.0
96.6=0.0
91.5++0.5
CIFAR-10
64.3100.0
40.810000
42,2:1.00
48.6.10.3
54.7-0.3
56.210.22
40.2.10.6
LSUN-CR
60.3+0.0
334+0.1
36.4+0.0
49.0:10.5
54.0=0.4
55.0+1..4
1.00+
CIFAR-100
66.3+00.0
43.1+0.0
44.7#0.00
49.7+
56.1-0.3
57.7=0.2
42.0=0055
Uniform Noise
53.610..0
25.9.10.0
29.810.00
41.5
46.8
47,910033
22.9.10.6
Asymptotic
99.7++0.0
73.8+0.1
74,2++..0
35.8+00.7
49.4=0.6
53.7-20.6
22.4+0.3
CTFAR-I00 100
81.7:10.00
$7.1
65.010.0
57.3:1.22
70.3:0.11
66.410.11
44.3::0.2
SVHN
69.9+0.0
35.6+0000
43.6+0.00
40.2+0.8
54.4=0.4
49.7=00.7
23.4+0055
LSUN-CR
60.4.10.0
29.8.10.2
47.011..0
39.4.1..5
51.010.5
46.810.9
14.3
CIFAR-10
71.5++0.0
39.2+0000
47.010.0
42.6:10.2
56.8-0.1
52.2-0.2
318
Uniform Noise
86.311:0.0
27.8:10.0
46.8:10.0
56.4139
74.210.8
67.6124
34.311..0
Asymptotic
99.9++0.0
40.7+:0.1
25.5+0..0
21.8+0.9
34.0-1.2
29.340.8
12.6+0.2
Table 2: OOD detection performance measured by the AUR metric. Values reported are averages over ten prediction runs. Higher is better. Underline and bold faces indicate the highest values over the last four columns and all columns in given row, respectively.
Train Test
MAP
MCD
DE
KFL
KFL+LL
KFL+0OD
KFLALULA
MNIST
E-MNIST
84,8.10.0
86. 86.0000.0
85.8.000
86.310.11
84.8.10.0
86.3
88.3 0.2
F-MNIST
86.8:10.0
94.9+0.0
89.5+0.0
91.3+00.3
86.9+0000
91.0++..
9..0=0.4
Uniform Noise
62.8:10.0
80.0:1.11
76.710.0
90.0+11.0
62.911011
89.0400.7
09.
Asymptotic
0.2+00.0
75.02:0.3
67.9=000
99.0:10.1
12.5+0.3
98.7+t0.2
9920-10.2
CIFAR-10
SVHN
88.6:10.0
73.7+0.00
90.1100.0
89.7.10.3
89,3100.3
89,310044
96.3
LSUN-CR
94.6:10.0
93.0:10.1
96.4-:.00
95.3
94.9:10.2
95.2:10.5
99.7
CIFAR-100
84.1=20.0
79.0=2011
86.2+0.0
84.8+0.1
84.5++0.0
84.740..1
84.2+0.3
Uniform Noise
97.510.00
66.110011
97.7.0.11
97.710.11
95.2.10.5
98.40...4
Asymptotic
6.5+10.0
86.3+.0.1
98.7:10.0
80.21
64.7+33.2
8..5+t1.5
96.5+0066
SVHN
CIFAR-10
96.4.10.0
97.310.00
97.6
97.0.1.00
96.8.10.0
96.7.10.00
95.8
LSUN-CR
97.11:0.0
98.7+.0.0
98.6+0.00
96.9+0.1
96.9+.0.1
96.9++0.0
96.4+0.3
CIFAR-100
95.9-10.0
96.7-10.0
97.110.0
96.7+0.00
96.41:0.0
96.3-50.0
95.3+0.55
Uniform Noise
98.2-10.0
99.6.10.0
99.51000
98.310.1
98.2.10.0
98.1
99.1100.2
Asymptotic
15.6::0.0
79.6=#0.1
72.5+:0.0
99.1
97.8+0..1
97.1+0.11
99.1--0.3
CIFAR-100
SVHN
67.5110.0
73.310.11
72.94000
70.1
69.0:10.4
69.410.6
80.1-10.7
LSUN-CR
76.6.10.0
80.110.3
69.2.0000
70.911.9
72.4.00.6
72.4..00
94.6 1.3
CIFAR-10
65.6:10.0
69.4-0.1
69.4-..0
67.1
66.34-0.1
66.5+00.1
66. 66.3+0.2
Uniform Noise
52.0:10.0
82.710.1
68.710.0
49.1
48.2410.9
49,5-12.5
60.5+33,4
Asymptotic
1.6:20.0
66.8+0.1
92.3+000
93.5
89.9+1.1
91.6+0..7
97.310.3
5.1
IMAGE CLASSIHICATIONS
Here, aim show that LULA units and the proposed training procedure are (i) significantly better method for tuning the uncertainty of LA than previous methods and (ii) able to make vanilla LA better than non-post-hoc (thus more expensive) UQ methods. For the purpose of (i), compare

---

Under review as a conference paper at ICLR 2021
Table 3: Robustness to dataset shifts on the corrupted CIFAR-I0 dataset (Hendrycks Dietterich 2019). following OOvdia et al. (2019). All values are averages and standard deviations over all perturbation types and intensities (for total of 95 dataset shifts). For accuracy, higher better, while for ECE and NLL, lower is better.
Metries
MAP
MCD
DE
KFI.
KII.+II
KFL+0OD
KFIALLUIA
Acc.
29.4119.0
9.1
30.0
29.319.00
29.4:19.0
29.2:18.9
2..11.8.9
ECE
47.3.17.2
22.7_99.1
35.6.18.1
36.9
42.0.17.2
30,5
28.9.66.6
NLL
1.067:t1.09
2.347=0..33
3.321=0.97
3.410=0.91
3.721
3.061+0.80
3.028+00.72
LUgA-augmented KFL (KFL+LULA) against the vanilla KFL with exact prior precision (KFL),
KFL. where the prior precision is found by maximizing validation log-likelihood (KFL.LL:; Ritter
et al. 2018b), and KFI, with OOD training (KFL++OD; Kristiadi et al. 2020). Moreover, for the method with the following baselines, which have been found to
purpose of (ii), we also compare
yield strong results in UQ (Ovadia et al.) 2019); Monte-Carlo dropout (MCD: Gal & Ghahramani 2016) and deep ensemble (DE; Lakshminarayanna et al. 2017).
We use 5- and 8-layer CNNs for MNIST and CIFAR-10, SVHN, CIFAR-I00, respectively. These networks achieve around 99%, 90%, and 50% accuracies for MNIST, CIFAR-10. 10, and CIFAR- 100, respectively. For ICC-interration during the predictions, we use 10 posterior samples. We quantify the results using the standard metries: the mean-maximun-coniieene
(MMC) and area-under-ROC (AUR) metrics (Hendrycks & Gimpell (017). All results reported are averages over ten prediction runs. Finally, we use standard test OOD datasets along with the "asymptotic" dataset introduced by Kristiadi et al. (2020) where random uniform noise images are scaled with a large number (5000).
For simplicity, add LULA units and apply only at the last layer of the network. For each in- distribution dataset, the training of the free parameters of LULA units is performed for a single epoch using Adam over the respective validation dataset. The number of these units is obtained via grid-search over the set {64, 128, 256, 512}, balancing both the in- and out-distribution confidences (see Appendix B for the details).
OOD Detection The results for MMC and AUR are shown in Table and Table respectively. First, we would like to direct the reader's attention to the last four columns of the table. We can that in general KFL+LULA performs the best among all LA tuning methods. These results validate the effectiveness of the additional flexibility given by LULA units and the proposed training proce- dure. Indeed, without this additional flexibility, OOD training on just the prior precision becomes less effective, as shown by the results of KFL+OOD, Finally, as we can see in the results on the Asymptotic OOD dataset, LULA makes KFL significantly better at mitigating overconfidence far away from the training data. Now, compared to the contemporary baselines (MCD, DE), we can see that the vanilla KFL yields somewhat worse results. Augmenting the base KFL with LULA makes it competitive to MCD and DE in general. Keep in mind that both KFL and LULA are post-hoc methods.
Dataset Shift We use the corrupted CIFAR-10 dataset (Hendrycks & Dietterich 2019) for mea- suring the robustness of LULAA-augmented LA to dataset shifts, following Ovadia et al. 2019). Note that dataset shift is slightly different concept to OOD data: it concerns about small perturbations of the true data, while OOD data are data that do not come from the true data distribution. Intuitively. humans perceive that the data under a dataset shift lies near the true data while OOD data are farther away. We present the results in Table 3 Focusing first on the last four columns in the table, we see that LULA yields the best results compared other tuning methods for KFL. Furthermore, we see that KFL+LULA outperforms DE, which has been shown by Ovadia et al. (2019) to give the state-of-ther afrtthe-rrt results in terms of robustness to dataset shifts. Finally, while MCD achieve the best results in this experiment, considering its performance in the previous OOD experiment, we draw conclusion that KFL+LULA provides a more consistent performance over different tasks.
Comparison with DPN Finally we compare KFL+LULA with the (non- noon-Bayeiian) Dirichlet prior network (DPN, Malinin & Gales 2018) in the RotASed-MNIST benchmark (Ovadia al. 2019)
8

---

Under review as a conference paper at ICLR 2021
1
0.8
0.8
DPN
0.5
KFI.
0.6
LULA
100
150
50
100
150
0
50
100
150
(a) Brier score +
(b) MMC +
(c) AUR 1
Figure 3: LULA compared to DPN on the Rotated-MNIST benchmark.
We found that LULA makes the performance of KFL. competitive to DPN. We stress that KFL and LULA are post-hoc methods, while DPN requires training from scratch.
(Figure 3).
5.2 COST ANALYSIS
The cost of constructing a LULA network is negligible even for our deepest network: on both the 5- and 8-layer CNNs, the wall-clock time required (with standard consumer GPU) to add additional LULA units is average 0.01 seconds (over trials). For training, using the last-layer LA as a proxy of the true LA posterior, took on average 7.25 seconds and 35 seconds for MNIST and CIFAR-10, SVHN, CIFAR-I00, respectively. This tuning cost is cheap relative to the training time of the base network, which ranges between several minutes to more than an hour. We refer the reader Table (Apeendix C for the detail. All in all, LULA only effective, but also cost-efficient.
CONCLUSION
We have proposed LULA units: hidden units that can be added any pre-trained MAP network for the purpose of exclusively tuning the uncertainty of a Laplace approximation without affecting its predictive performance. They can be trained via an objective that depends on both inlier and outlier datasets to minimize (resp. maximize) the network's output variance, bringing the state-of- the-art technique from non-Bayesian robust learning literature the Bayesian world. Even with very simple outlier dataset for training, show in extensive experiments that LULA units provide more effective post-hoc uncertainty tuning for Laplace approximations and make their performance competitive more expensive baselines which require re re-training of the whole network.
REFERENCES
José Bernardo and Adrian FM Smith. Bayesian Theory. John Wiley & Sons, 2009.
Felix Dangel, Frederik Kunstner, and Philipp Hennig. BackPACK: Packing more into Backprop. In ICLR, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian Approximation: Representing Model Uncertainty in Deep Learning. ICML, 2016.
Mark N Gibbs. Bayesian gaussian processes for regression and classification. Ph. D. Thesis, De- partment of Physics, University of Cambridge, 1997.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why ReLU Networks Yield High- confidence Predictions Far Away from the Training Data and How to Mitigate the Problem. In CVPR, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. ICLR, 2019.
Dan Hendrycks and Kevin Gimpel. A Baseline for Detecting Misclassified and Examples in Neural Networks. In ICLR, 2017.

---

Under review as a conference paper at ICLR 2021
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep Anomaly Detection with Outlier Exposure. In ICLR, 2019.
Tom Heskes. On "Natural" Learning and Pruning in Multilayered Perceptrons. Neural Computation, 12(4), 2000.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, 2017.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being Bayesian, Even Just Bit, Fixes Overconfidence in ReLU Networks. ICML, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In NIPS, 2017.
David JC MacKay. The Evidence Framework Applied Classification Networks. Neural compu- tation, 1992a.
David JC MacKay. A Practical Bayesian Framework For Backpropagation Networks. Neural com- putation, 4(3):4484472, 1992b.
Andrey Malinin and Mark Gales. Predictive Uncertainty Estimation via Prior Networks. In NIPS, 2018.
Andrey Malinin and Mark Gales. Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustness. In NIPS, 2019.
James Martens and Roger Grosse. Optimizing Neural Networks With Kronecker-Factored Approx- imate Curvature. ICML, 2015.
Alexander Meinke and Matthias Hein. Towards Neural Networks that Provably Know when They don't Know. In ICLR, 2020.
Sebastian W Ober and Carl Edward Rasmussen. Benchmarking the Neural Linear Model for Re- gression. arXiv preprint 151i::1912..8444 2019.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift. In NeurIPS, 2019.
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online Structured Laplace Approximations for Overcoming Catastrophic Forgetting. In NIPS, 2018a
Hippolyt Ritter, Aleksandar Botev, and David Barber. A Scalable Laplace Approximation for Neural Networks. In ICLR, 2018b.
10

---

Under review as a conference paper at ICLR 2021
APPENDIX A PROOFS
Proposition 1. Let f IR x Rd + R2 be MAP-trained L-layer network parametrized by 0AAP- Suppose f IR." Rd and IMAP Id are obtained via the previous construction. For any input a E R", we have [x; 0MAPP = f(x; OMAP).
Proof. Let x Rn be arbitrary. For each layer I = 1,.... we denote the hidden units and pre- activations of f as h and a respectively. We need to show that the output of f. i.e. the last pre-activations 2(4), equal to the last pre-activations aL4) of
For the first layer, we have that
W(())
((1)
W ((0) ( + (())
(a (
a - b() =
W
x
+
((0)
=:
For every layer 1 = 1,..., L - 1, we denote the hidden units as the block vector
(p(a!9) h = (p((()))
(100)
=
7
Now, for the intermediate layer 1 = 2..... L 1, we observe that
200) ((107/111)) + g(0)
W
0
11-1))
(00
((())
T.0-1)
+
800
(( + +0+ b()
(1))
- + W9 7 (1-1) + (10)
a
Finally, for the last-layer, we get
(h(l-1)) 0) ((-11) +0())
T(4)_ W(( +(() (W(L)
+0+ - -4(
This ends the proof.
Proposition 2. Suppose f R x Rdd + R is a real-valued network and J is constructed above. Suppose further that diagonal sxplace-aprroxmmated posteriors N(@ 0MAP diag(or), N( 0, diag(ā)) are employed. Using the linearization (5, for any input E R", the variance over the output (x; 0) is at least that of f (x;;0)
Proof. W.L.ogg arrange the parameters 0 == (0T TTT) where Rd contains the weights corresponding to the the additional LULA units. If g(x) is the gradient of the output f(z; f) w.r.t. 0, then the gradient of f(z; 0) w.r.t. 0, say õ(x), can be written as the concatenation (g(z) .(()) TT where ((x) is the corresponding gradient of 0. Furthermore, diag(ō) has diagonal elements
Bai....,ddaa " T.
Hence we have
((z) = ā(æ) diag(T)g(z)
g(x) diag (()())))))))))))))))()))))) +ō(x) diag(?)g(z)
=v(x)
2 v(z),
since diag(ō) is positive-definite
11

---

Under review as a conference paper at ICLR 2021
Algorithm 2 Adding LULA units.
Input:
L-layer with MAP estimate OMAP (W MAP: = Sequence of non-negative integers
(mi)f=r
1: for
1 111,.... -,..., - do
2: vec W p(voc W
Draw from prior
3: BiU) p(bi))
Draw from prior
(w(1)
4: Wi(
MAP
MMP
D. The zero submatrix is size ny X mi-1
W
MAP
T(0)
5:
MAP
6: end for
7: Wi (W
MAP: 0) 8: MAP (L))
The zero submatrix of size m.-1
9: MAP (WAM (
L.
MAP: MAP
10: return OMAP
1.0
0.8
0.6
0.5
(a) Laplace
(b) LULA-Untraneed
(c) -UiA-Trained
Figure 4: Even when their weights are assigned randomly, LULA units improve the vanilla Laplace in terms of UQ. Fine-tuning the LULA weights improve it even further, in particular in terms of confidence far from the data-trained, LULA yields less confident prediction in this region.
APPENDIX B IMPLEMENTATION
We summarize the augmentation of network with LULA units in Algorithm 2 Note that the priors of the free parameters (1))) (()) (lines and 3) be chosen independent Gaussians this reflects the standard procedure for initializing NNs' parameters.
We train LULA units for single epoch (since for each dataset, have large amount of training points) with learning rate 0.01. For each dataset, the number of the additional last-layer units my is obtained via grid search over the set (64, 128, 256, 512) =: M,,, minimizing the absolute distance to the optimal MMC, i.e. and 1/1: for the in- and out-distribution validation set, respectively:
m2 = arg min - (M) + |1/k k MMC mEM,
where MMC,. and MMCout are the validation in- and out-distribution MMC of the Laplace- approximated, trained-LULA. respectively.
APPENDIX C ADDITIONAL EXPERIMENT RESULTS
C.I TOY DATASET
In practice one can simply set (1))), B011-1 randomly given prior, e.g. the standard Gaussian (see also Algorithm 2). To validate this practice, show the vanilla Laplace, untrained LULA,
12

---

Under review as a conference paper at ICLR 2021
Table Predictive performances on UCI regression datasets in term of average bigllikel.ood. The numbers reported are averages over ten runs along the corresponding standard deviations. The
performances of LULA are similar to KFL's. The differences between their exact values are likely due to MC-integration.
Dataset
MAP
MCD
DE
KFT.
KFL.+LUU.
Housing
-.79941:.012
2.731
3.045:10.009
-3.5061100055
3.495:10.047
Concrete
-3.40910.036
4.370_10.001
-0,05110.002
4.730.10.205
4,365 10.094
Energy
-2.270=t0.125
-2.120=t0. 104
-2.673++0.015
-2.707++0.030
-..698+0.014
Kin8nm
-0.923==0.000
-0.926=1.0000
1.086::0.022
-0.965-+0.003
-0.66910.003
Power
-3.15411..002
...85_10.003
-5.8004
-3.273.10.015
13.27710.002
Wine
-1.190=0.014
-1.185=0.0005
-1.038+000088
-1.624400.055
-1.630+0.092
Yacht
-1.8351000033
-1.72410.117
-3.272::0.079
-2.509:10.367
-2.663-10.776
Table 5: UQ performances on UCI datasets. Values are the average (over all data points and ten trials) predictive standard deviations. Lower is better for test data and vice-versa for outliers. By definition, MAP does not have (epistemic) uncertainty.
Test set
Outliers f
Datasel
MCD
DE
KFL
KFLALULA
MCD
DE
KFL
KFL+LULA
Housing
1.40
5.82
1.26
61.49
145.33
222.76
377.92
Concrete
1.00
8.11
10.44
16.89
964.63
30898.92
83241.42
Energy
1.32
4.40
1.05
1.08
45.69
126.11
1070.09
5163.53
Kin8nm
0.04
0.10
0.14
0.18
0.24
2.12
0.80
2.12
Power
14.73
19.85
2.85
3.20
273.08
12235.87
4148.98
221287.81
Wine
0.24
0.64
1.15
1.22
0.30
28.57
186.76
21383.17
Yacht
0.54
5.17
2.08
2.78
81.98
187.41
5105.69
13119.99
and fine-tuned LULA's confidence in Figure Even when set randomly from the standard Gaus- sian prior, LULA weights provide a significant improvement over the vanilla Laplace. Moreover, training them yields even better predictive confidence estimates. In particular, far from the data, the confidence becomes even lower, while still maintaining high confidence in the data regions.
C.2 UCI REGRESSION
To validate the performance of LULA in regressions, employ subset of the UCI regression benchmark datasets. Following previous works, the network architecture used here a single- hidden-layer ReLU network with 50 hidden units. The data are standardized to have unit variance. We use 50 LULA units and optimize them for 40 epochs using OOD data sampled uniformly from [-10, 10)". For MCD, KFL, and KFL+LULA. each prediction done MC- integration with 100 samples. For the evaluation of each dataset, we use 60-20-20 rain-validation- test split. We repeat each train-test process 10 times and take the average.
and
In Table we report the average predictive standard deviation for each dataset. Note that this metric the direct generalization of the ID error bar in Figure [ (top) multi dimension. The outliers are sampled uniformly from 100 10 Note that since the inlier data are centered around the ori- a Euclidean ball with radius Therefore,
gin and have unit variance, they lie approximately these outliers are very far away from them. Thus, naturally, high uncertainty values over these outliers are desirable. Uncertainties over the test sets are generally low for all methods, although KFL+LULA has slightly higher uncertainties compared to the base KFL. However, KFL+LULA yield much higher uncertainties over outliers across all datasets, significantly more than the base- lines. Moreover, in Table we show that KFL+LULA maintains the predictive performance of the base KFL. Altogether, they imply that KFL+LULA can detect outliers better than other methods without costing the predictive performance.
C.3 IMAGE CLASSIFICATION
In Table we present the sensitivity analysis of confidences under Laplace approximation w.t.. the number of additional LULA units. Generally, we found that small number of additional LULA units,
13

---

Under review as a conference paper at ICLR 2021
Table 6: In- and out-distribution validation MMCs for varying numbers of additional LULA units. "In" and "out" values are in percent. "Loss" is the value of the loss in Missing entries signify that errors occurred, see Section C.3 for details.
MNIST
CIFAR-10
CIFAR-I00
units
f
Out
Loss
In 1
Out +
Loss +
In
Out +
Loss
16
97.1
46.3
0.392
91.3
21.6
0.203
54.0
12.8
0.578
32
97.1
19.7
0.127
90.4
17.7
0.173
50.5
8.7
0.572
64
97.0
27.0
0.200
89.3
19.6
0.203
43.8
7.8
0.629
128
96.7
28.9
0.223
83.9
18.2
0.243
31.0
6.5
0.745
256
94.2
18.2
0.140
62.1
18.4
0.463
14.3
6.2
0.909
512
68.4
17.6
0.392
Table 7: Accuracies (in percent) image classification test sets. Values averages ten
trials.
Dataset
MAP
MCD
DE
KFL
KFL+LULA
MNIST
99.1=0.00
99.1=0.0
99.1=0.0
99.1=0000
99.1
CIFAR-10
90.1110.0
84,4-0.0
91.2-10.0
89.911011
89.510.11
CIFAR-100
53.2:10.0
60.5-0.1
59.6:0.0
52.2:10.2
51.340.4
e.g. 32 and 64, is optimal. It is clear that increasing the number of LULA units decreases both the in- and out-distribution confidences. In the case of larger networks, found that larger values (e.g. 512 CIFAR-10) make the Hessian badly conditioned, resulting numerical instability during its inversion. One might be able alleviate this issue by additionally tuning the prior precision hyperparameter of the Laplace approximation (as in (Ritter et al.) 2018b] Kristiadi et al. (2020). which corresponds to varying the strength the diagonal correction of the Hessian. However, emphasize that even with small amounts of additional LULA units, we can already improve vanilla Laplace approximations significantly, as shown the main text (Section 5.T).
We present the predictive performances of all methods in Table 7 LULA achieves similar accu- racies to the base MAP and KFL baselines. Differences in their exact values likely due to various approximations used (e.g. MC-integral). In the case of CIFAR-10M 100, we found that MAP underper- forms compared to MCD and DE. This might be because of overfitting, since only weight decay used for regularization, in contrast to MCD where dropout is used on top of weight decay. Due to MAP's underperformance. LULA also underperform. However, we stress that whenever the base MAP model performs well, by construction LULA will also perform well.
As a supplement we show the performance KFL+LULA against DPN in OOD detection on MNIST (Table 8). We found that KFL+LULA performance OOD detection are competitive or better than DPN.
C.4 COMPUTATIONAL COST
To supplement the cost analysis in the main text, we show the wall-clock times required for the construction and training of LULA units in Table 9
Table 8: LULA compared to DPN on OOD detection terms of MMC and AUR, both in percent.
MMC
AUR 1
Train Test
DPN
KFLALULA
DPN
KFLALULA
MNIST
F-MNIST
74.5++0.0
71.4+0.4
87.7++0.0
88.
F-MNIST
58.5++0.00
49.4=1.9
96.8++0.0
98.0=-0.4
Smooth Noise
10.5.10.0
22.2.10.6
100.0
100
Uniform Noise
61.0++0.0
33.8+2.55
98.2+00.0
99.8-10.1
Asymptotic
100.0:t0.00
46.0-12..6
0.2:10.0
99.0-10.2
14

---

Under review as a conference paper at ICLR 2021
Table 9: Wall-clock time of adding and training LULA units. All values are seconds.
Dataset
Construction
Training
MNIST
0.010110.000 0.01
7.252.10.033 10.
CIFAR-I0
CIFAR-100
0.009=0.000
34.913+002253
Table 10: Average confidences (MMCs in percent) 20-layer CNNs prediction runs.
Lower is better for OOD data. Values shown for each in-distribution dataset are Ece-lower is better. Underline and bold faces indicate the highest values over the last four columns and all columns a given row, respectively.
Train Test
MAP
MCD
DE
KFL
KFL+LL
KFL+0OD
KFLALULA
CIFAR-JO
15.0=0..0
6.810.3
4.2+00.0
9.940.4
5.8+0.8
6.1
5.2+00.5
SVHN
70.7=0.0
67.310.0
58.3+000
65.240.66
60.1#112
60.5+115
0,,5=1,9
LSUN-CR
$6.2.10.0
56.01002
51.31000
51.31088
45.5.11
47.31188
1.4
CIFAR-100
78.3=#0.0
69.9+0.0
66.5+0.0
72.9+0.3
67.6+0.88
68.340.8
6.,2-0.6
Uniform Noise
84.0:10.0
74.820.0
54.81000
74.24:1.5
63.3:12.66
65,2433.2
59.5+22.3
Asymptotic
99.9110.0
95.21000
7..61:0.0
87.1
80.412.44
82.1
7.0011.
SVHN
10.5100.0
2.7.
17.310.00
6.3.10.2
5.8.1..4
4.1.1..4
3.4.0.3
CIFAR-10
73.8=0.00
55.2+0.0
43.9==0.0
67.9+0.2
67.4+0.4
64.7+00.5
59.3+0099
LSUN-CR
64.8:10.0
50.610.11
42.0:1.00
61.8+0.3
61.5+0.44
59.74.0.3
45.5+ 1,5
CIFAR-100
74.8=10.0
56.1+0.0
45.0=0.0
69.4+0.3
68.9400.4
66,240.4
60.7+0.9
Uniform Noise
56.6.10.0
47.21000
36.5
$4.1.002
$3.8.10.3
$2.400.3
41.01177
Asymptotic
99.8100.0
65.5+00.
44.4
04.
83.5+00.7
79.2+
69.1
CIFAR-100
38.7:00.0
20.610.11
10.810.00
21.5
14.6
10.4:0.2
0.2
SVHN
69.3=0.0
53.5=0.0
47.1+0.0
53.5#0.4
46.8+00.4
42.8+0.4
26.7=0,55
LSUN-CR
67.1100.0
52.61003
46.010.0
$2.8.0..5
46.3.10.6
42.610.77
23.41111
CIFAR-10
73.5=10.0
57.9+0.1
52.2+0.0
59.3+0.22
53.1+0.11
49.0+0.3
40.7+0.3
Uniform Noise
79.9-0.00
52,420.1
47.3+0.0
61241.1
52.2113
48.741..
25,3+116
Asymptotic
97.4.00.0
48.110.1
41.7100
$2.8.1.3
45.0.1.77
39.41.22
37.8
C.5 DEEPER NETWORKS
We also asses the performance of LULA in larger network. We use 20-layer CNN on CIFAR- 10, SVHN, and CIFAR-106 Both the KFL and LULA are applied only at the last-layer of the network. The results, in terms of MMC, expected calibration error (ECE), and AUR, are presented To] Table We observe that LULA makes KFL better calibrated in both in- and out-distribution settings. Moreover, the LULA-imbued KFL competitive to DE, which has been shown by Ovadia et al. (2019) to be the best Bayesian method for uncertainty quantification. Note that, KFL+LULA is post-hoc method and thus be applied to any pre-trained network. In contrast, DE requires training multiple networks (usually 5) from scratch which can be very expensive.
best method for uncertainty tuning LAA:
We additionally show the performance of LULA when applied on top of KFL-aaproximmate DenseNet-121 121 (Huang al. 2017) in Tables 12 and 13 LULA generally outperforms previous uncertainty tuning methods for LA and is competitive to DE. However, we observe SVHN that LULA do not improve KFL significantly. This issue due to the usage the Smooth Noise dataset, which has already assigned low confidence in this case, for training LULA. Thus, we re-train LULA with the Uniform Noise dataset and present the result in Table [4 We show that using this dataset, we obtain better OOD calibration performance, outperforming DE.
15

---

Under review as a conference paper at ICLR 2021
Table 11: OOD detection performance measured by the AUR metric on 20-layer CNNs. Values reported are averages over ten prediction runs. Higher is better. Underline and bold faces indicate the highest values over the last four columns and all columns in a given row, respectively.
Train Test
MAP
MCD
DE
KFL
KFL+LL
KFL+0OD
KFLALULA
CIFAR-10
SVHN
91.4+10.0
89.0:10.0
92.8+0.0
90.9+0.2
90.6:t0.3
90.8+0.3
90.9++0.2
LSUN-CR
95.6.10.0
93.410.1
94.810.0
95.510.2
96.01..3
95.6110
9..01..3
CIFAR-100
85.1=0.0
85.5+000
87.2=0.0
85.1
85.1+0.1
85.1+0.11
85.240.1
Uniform Noise
86.0110.0
85.910.0
94.7.10.0
87.410.7
89.71009
89.211122
90.310.8
Asymplotic
12.7:10.0
35.0:10.3
77.9:10.0
56.91:444
70.5::599
67.246.9
1.
SVHN
CIFAR-10
93.2=#0.0
96.4+0.00
97.2++0.0
93.9+0.0
94.0+0000
94.240.1
93.9+0.2
LSUN-CR
96,410000
07.4
98.111000
96.1+10.0
96.1100.1
96.04:0.1
96.7400.1
CIFAR-100
92.9:10.0
96.1:0000
9.08:
93.5+0.0
93.5:10.0
93.7
93.510.2
Uniform Noisc
98.1=0.0
98 98.0+00.0
99.0==000
97.8+0.0
97,8+0.0
97.7+0.00
08.1+
Asymptotic
6.9+00.0
94.7=0000
95.4++0.0
1..0
60.6
69.3
83.2+2.0
CIFAR-100
SVHN
69.8:10.0
70.5+0.11
73.3+0.0
71.2+0.5
72.240.4
72.540.4
83,7-00.5
LSUN-CR
72.110.00
71.310.3
74.410.0
71.910..6
72.7100.6
72.7.0.88
1.3
CIFAR-I0
64.7+00.0
65.740.1
67.9=000
64.8+0.2
65.0++0.1
65.1
64.3+0.3
Uniform Noise
58.2:10.0
72.020.1
73.1=0.0
63.241.0
66.1=115
65.2+11.9
85,9--2.2
Asympiotic
13.6.10.0
76.110.1
78.6.1..0
72.111.4
74.2.12.0
76.61..55
6..6.11.8
Table 12: Average confidences (MMCs in percent) on DenseNet-12) over ten prediction runs. Lower better for OOD data. Values shown for each in-distribution dataset are ECE-lower is better. Underline and bold faces indicate the highest values over the last four columns and all columns in given row, respectively.
Train Test
MAP
MCD
DE
KFL
KFL+LL
KFL+OOD
KFLALULA
CIFAR-1O
23.3.10.0
10.210.5
8.1
21.2.044
16.310.5
15.5.0.4
10.3.10.7
SVHN
87.8+0.00
74.3+00.0
67.6+000
85.1
80.740.6
79.8
30.9:=1.5
LSUN-CR
80.0:1.00
70.2:1.22
63.42000
77.010.4
72.4122
71.810.7
34.0:12.33
CIFAR-100
86.711.0
79.6.10.0
74.2:10.0
84.810.1
81.610.3
81.010.2
7..7
Smooth Noise
76.0=0.1
80.3=0.0
54.2=0.1
70.6+
63.41.11
62.5+1.0
24.3==13 13
Uniform Noise
84.11:000
82.310.00
69.4110.0
77.810.6
70.411.55
68.711.5
7..8.10.8
Asymptotic
100.0:10.0
100.0:10.0
20.0:10.0
63.8111.5
44.5
43.4
58.4
SVHN
4.740.00
2.7-10.2
3.1-:0.0
4.3+0.2
3,940.2
3.64-0.2
3.0-10.2
CIFAR-10
69.911000
56.1110.0
50.5.10.
69.310.11
68.610..3
67.8.1..3
67.8.10.2
L.UN-CR
64.110.0
52.3+0.22
51.3-10.0
63.8
63.40.22
63.0+:0.2
61.4:10.4
CIFAR-100
69.410.0
57.3+0.0
51.1-0.00
68.910.1
68.240.3
67.5+0.22
67.5=0022
Smooth Noise
30.9.10.1
25.4
33.6.0..0
30.8.1..1
30.7.0..1
30.6.10.0
1
Uniform Noise
63.6+00.0
55.8+00.0
54.2-00.0
63.4+0.11
63.1+0.22
62.8+0011
62.8--022
Asymptotic
100.0=10.0
98.8=00.0
40.0+0.0
34.11:2.1
26.611.11
25.7+:1.2
22.7-0066
CIFAR-JO0
27.3+00.0
19.1-0.22
6,240..0
5.3+1..3
10.0+0.2
4.74-0.3
13.3:10.4
SVHN
76.2+#0.0
72.6=0.0
53.3+-0.0
18.1
54.9+1.3
40.0+11.4
15.6==0.8
LSUN-CR
71.110.00
67.210.2
50.5100.
46.91112
53.240.9
39.811144
1..4
CIFAR-10
77.8:10.0
70.2:10.0
59.31:0.0
57.7+:0.2
63.1100.2
$1.2+:011
40,3
Smooth Noise
93.1=0.1
90.4+0.1
76.9+0.1
61.9+2.2
70.7+1.5
50.3+2.6
14.1=1.4
Uniform Noise
79.8.10.0
56.110.1
53.6100.0
36,5.1..5
43.6.1...
28.51227
1.3
Asymptotic
100.0::0.0
96.5+10.0
20.0+10.0
48.6Gt3.33
61.542.44
35.8+3.55
21.15
16

---

Under review as a conference paper at ICLR 2021
Table 13: OOD detection performance measured by the AUR metric on DenseNet- 121. Values reported are averages over ten prediction runs. Higher is better. Underline and bold faces indicate the highest values over the last four columns and all columns in given row, respectively.
Train Test
MAP
MCD
DE
KFI.
KFL+LL
KFL=OOD
KFIALULA
CIFAR-1O
SVHN
88.2+40.0
92.5+0.0
94.1+0.0
89.5+00.2
90,9=0.3
91.2440.7
99.7+10.0
LSUN-CR
94.0.10.0
94.2.10.1
95.3.10.0
94.310.1
94.510.3
94.510.2
99.5
CIFAR-I00
88.9++0.0
88.5+0.00
90.5+0.0
89.0+0.0
89.1=0.0
89.2+0.0
87.9+0.T
Smooth Noise
96.2:+0.00
91.5
97.610.0
96.710.1
97.2-0.2
97.210.1
100.0:10.0
Uniform Noise
93.6:10..0
91.6110.0
94.5.10.0
94.8.10.2
95.7-003
95.9
92.0.10.3
Asymptotic
26.1+:0.0
18.0+0.1
100.0::0.0
97.9+0.2
99.5=0.1
99.5+0.1
97.2+0033
SVHN
CIFAR-10
93.6:+0.0
96.7:10.0
97.8:10.0
93.7110.
93.8-1..1
93.9:1.11
93.2:1.22
LSUN-CR
96.1+0.0
97.6=0.0
97.9=0.0
96.1=000
96.1=0.0
0.01=0.0
95.8+0.11
CIFAR-100
93.3.10.0
96.111000
97.410.0
93,41000
93,4.0.1
93.610.11
92.910.2
Smooth Noise
99.4++0.00
99.7++0.0
99.6+0.00
99.4+00.
99.4=0.0
99.4+000
99.6+t0.0
Uniform Noise
97.1+-0.0
97.5
98.0=0.0
97.1+000
97.0=0.0
97.0=00.0
96.6:00.1
Asymptutic
0.0.10.0
14,4.10.3
99.7.10.0
99.8110.1
99.910.0
100.0
100.0
CIFAR-100
SVHN
77.2+0.0
73.2+0.00
83.5+0.0
13.8
82.4=100
85.6.1.1
97.5++0.5
LSUN-CR
81.4+0.0
79.6#0.1
85.7+0.0
84.6=+0.8
83.7=0.6
85.7+1.1
96.2-+-..8
CIFAR-I0
74.8+-0.0
75.8+0.0
78.6+0.0
75.
75.2=002
75.5+002
03.9+00.3
Smooth Noise
59.4:10.11
54.610.1
62.010.1
71.41:9
68.121.2
76.212244
97.7+112
Uniform Noise
78.7:10.00
88.4+0.11
85.1
93.1
91.5=12.0
9..8
93.1
Asymptotic
4.5:t0.0
32,940.2
99.9+0.00
84.9+2.66
79.5=1.8
90.0+22.6
93.9+1.11
Table 14: LULA'S OOD detection performance on DenseNet-121 with uniform noises as the train- ing OOD data. Values are the ECE, MMC, and AUR metrics, averaged over ten prediction runs. Underline and bold faces indicate the highest values over the last four columns and all columns in given row, respectively.
Train Test
MAP
MCD
DE
KFL
KFL+LL
KFL+0OD
KFL*LULA
SVHN (RCF, MMC)
4.7-20.0
2.7:10.2
3.1+0.0
4.340.2
3,9:0..2
3.6:+0.2
3.6
CIFAR-10
69.9=0.0
56.1=0.0
50.540.0
69.340.1
68.6+1..3
67.8+0..3
39.1-: 1.8
LSUN-CR
64.1=0.0
52.3+0022
51.3+0.0
63.8+0.11
63.4+0.2
63.0++0.2
29.6+t2.3
CIFAR-100
69.4-0.0
57.310000
51.11000
68.910.11
68.2.10.3
67.5.0..2
42.5.11.7
Smooth Noise
30.9=0.11
25.4:10.0
33.6:00.0
30.8+0.11
30.7+t0.11
30.6+0.0
27.5++0.3
Uniform Noise
63.6-0.0
55.8+00.0
54.2+00.0
63.4+0.11
63.140..
62.8+0.11
22.9+10.6
Asymplotic
100.0.0..0
98.8.10.0
40.0.10.0
34.1.12.1
26.6.1..1
25.7.11.2
23.
SVHN (AUR)
CIFAR-10
93.6:1..0
96.7:1:000
97.810.00
93.7++0.0
93.8-1:..1
93.9:t0.1
98.7
LSUN-CR
96.1=0.0
97.6=+0.0
97.9+0.0
96.1+-0.0
96.1++0.0
96.1+0000
99.5++0.1
CIFAR-100
93.310.00
96.1.10.0
97.410.0
93.4.0.00
93.4.0.1
93.6.10.1
98.0.1..1
Smooth Noise
99.420.0
99,7-10.0
99.610.00
99.41000
99.410.0
99.4-10.0
99.6-10.0
Uniform Noise
97.1100.0
97.5:10.0
98.010..0
97.110.00
97.010.00
97.0.10.0
99.9.10.0
Asymptotic
0.0-00.0
14.4-0.3
99.740.0
99.8+-0.1
99,9+0.0
100.0-10.0
99.9+0000
17


Reviews:
[{"review_id": "i12Xhg-_GJG", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "***\nPOST DISCUSSION UPDATE\n***\nI am satisfied with the authors' response and will increase my score to an accept.\n***\nEND OF UPDATE\n***\n\nThe paper proposes an uncertainty estimation method for deep learning. The idea is, building on prior work on Laplace approximations for neural networks, to augment a pre-trained network with additional units such that predictions with point estimates remain unaffected, while the variance may change. The weights of those units are then trained on the validation set/out-of-distribution (OOD) data such as to minimise the variance on the in-distribution data and maximise it on OOD data.\n\nOverall, I think this is a novel and interesting method, which could be useful in practice considering that it builds on pre-trained networks. There are a couple of points which are unclear to me (see below for details) as well as some details regarding the experiments which could be improved, so that I would **not recommend acceptance** at this point, however I believe that these can be rectified over the discussion period.\n\nUnclear points:\n* How exactly are you training the LULA weights considering that the Hessian (and hence the variance of the corresponding weights and therefore the samples in the MC integral in eq 8) depends on them? I assume you are treating the Hessian as constant? Surely differentiating through its computation would be prohibitively expensive considering that it requires a pass over the entire training dataset? Would it be possible to construct an example (e.g. a reduced version of MNIST) where you can differentiate through the Hessian to see if this has much of an impact?\n* I'm somewhat confused by the discussion regarding the closed-form approximations for the predictive distribution around eq. 4. Isn't the point of those to avoid an MC approximation of the integral over the posterior? I understand that computing eq. 5 efficiently for a Laplace approximation over all weights is not generally supported in Pytorch and tensorflow, but wouldn't it be feasible for the last-layer approximation that you use? Or is the whole discussion mainly to motivate the LULA objective in eq. 7?\n* Are you using the multi-class equivalents of eq 4 and 5 for test-time prediction or do you do a MC integral over the approximate posterior? \n\nExperiments:\n* The setup seems slightly unfair to me in that proposed method is the only one (except KFL+OOD) to be tuned on out-of-distribution data. It would be good to see the DPNs included in Tab. 1 & 2.\n* In Tab. 1 it looks like the test predictions from the proposed method are somewhat underconfident. Could you also report test log likelihoods?\n* That being said, I find the evaluation a bit narrow in that it only looks at out-of-distribution detection. Could you add e.g. an evaluation of the robustness to data shift? For example, making the predictions using your trained networks for the corrupted CIFAR10 and CIFAR100 variants from (Hendrycks & Dietterich, 2019) shouldn't be too much trouble. See e.g. (Ovadia et al., 2019) for a recent use of those datasets in the literature. I understand that if you're making predictions using eq. 4 accuracies would be unaffected, but it would still be interesting to see log likelihoods and calibration errors.\n* Would it be possible to use a more recent architecture for CIFAR100? 50% test accuracy seem somewhat subpar e.g. compared to even small Resnets (which get over 70%).\n* Finally, as more of an optional request/suggestion, it would be nice to have some experiment in a domain other than image classification. Considering that you're only implementing inference for the last layer, I would imagine that your method would be compatible with any architecture (e.g. RNN, Transformer) that feeds into a fully-connected output layer? Having more variety in the empirical tasks would strengthen the paper.\n\nReferences:\nHendrycks & Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In ICLR 2019.\nOvadia et al. Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. In Neurips 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bOZR-qQoK2l", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a post-hoc uncertainty tuning pipeline for Bayesian neural networks. After getting the point estimate, it adds extra dimensions to the weight matrices and hidden layers, which has no effect on the network output, with the hope that it would influence the variance of the original network weights under the Laplacian approximation. More specifically, it tunes the extra weights by optimizing another objective borrowed from the non-Bayesian robust learning literature, which encourages low uncertainty over real (extra, validation) data, and high uncertainty over manually constructed, out-of-distribution data.\n\nUsing Eq (7) and (8) to quantify posterior uncertainty doesn't seem correct to me. Instead of manually building some fake OOD data and forcing their posterior variance to be large, plus forcing all real data points' posterior variance to be small, one should focus more on the different posterior uncertainty *within* the real observed data. For example, data in highly uncertain areas should naturally have higher posterior variance. But in the proposed method, they are all forced to have small variances as long as they are real data.\n\nAnother confusion I have is more related to the math behind. The extra network weights and hidden dimensions are designed not to have any connection to generate the output. If so, does it mean that no matter what values the extra network weights take, their curvature to the output prediction should be entirely flat? How would these values affect the uncertainty if their Hessian values are zero? It would be beneficial if the authors could explain more about why that's not the case.\n\n\n--------------------------\nPOST DISCUSSION UPDATE\n\nThe central part of this work about how the extra network weights only affect the curvature still confuses me. But I'm more motivated by the proposed objective function that borrows idea from non-Bayesian robust learning literature.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "L4N4qpUBoAe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "POST DISCUSSION UPDATE\n------------------------------------\nI like the proposed method and I will keep my score.\n\nEND OF UPDATE\n------------------------------------\nThe paper proposes a new method to learn uncertainty under Laplace approximations. The method relies on uncertainty units that do not change the predictions but increase the dimensionality of the parameters and help learn better uncertainty by the proposed loss function.\n\nStrong points:\n\n- The paper proposes an ad-hoc method that can be applied to any MAP trained network.\n- The proposed method is simple, powerful, and not expensive relative to the training time.\n- Good experimental analysis.\n\n\nClearly state your recommendation (accept or reject) with one or two key reasons for this choice.\n- I recommend to accept this paper.\n- This type of work is definitely needed to enable more powerful models that have accurate predictions and better uncertainty estimates with small additional cost.\n\n\nQuestions:\n- What are the hyperparameters that the method is sensitive for? For example, the size of OOD, number of epochs, learning rate,..,etc\n- How was the number of LULA units chosen? Why did you choose 64? Why not try 32 or less?\n- Were the hyperparametrs of LULA tuned on a separate validation set than the one used to tune the uncertainty?\n\nMinor comments:\n- Table 1 is very hard to read. It would be great if the best numbers are in bold as in Table 2.", "extra_scores": {}, "extra_text": {}}, {"review_id": "6WjZqHZQRnR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#################\nPost-Rebuttal Reviews: Thank the authors for the detailed responses. The proposed approach is an interesting add-up for the Laplacian approximations. However, I think the paper still deserves more works. As far as I am concerned, applying the approach to multi-layers instead of only the output-layer is important. I will keep my score for now.\n\n##################\n\nBayesian deep learning aims to bring uncertainty quantification to modern neural network models. Laplacian approximation, which transforms a trained deterministic neural network into stochastic by a second-order Taylor approximation, is especially appealing due to its pos-hoc nature. However, since the network parameters are trained and fixed, few parameters can be optimized in Laplacian approximation. To enhance the capacity of Laplacian approximation, this paper proposes the *learnable uncertainty units (LULA)*. The LULA units do not affect the network prediction, but the Hessians of the LULA units are nonzero. In consequence, training LULA units increases the capacity of Laplacian approximation by adapting the Hessians. \n\nIn general I think the paper proposes an interesting module for parameterizing the predictive variances. The proposed $v(x)$ in Eq(5) is similar to the expression of the neural tangent kernel (Jacot et. al., ‎2018), thus might be suitable for parameterizing the predictive variance. However, the current version of the paper leaves me several questions regarding the methods. \n\n1. It seems to me that the weight matrix does not affect the prediction as long as it is in the form of $\\tilde{W}^{(l)} = [ W_{map}^{(l)} \\; 0 \\;  \\backslash \\backslash  \\; \\hat{W}_1^{(l)} \\; \\hat{W}_2^{(l)} ]$. However, the proposed method sets $\\hat{W}_2^{(l)}$ as zero. Setting $\\hat{W}_2^{(l)}=0$ blocks the backpropagation through the augmented hidden units, which does not seem reasonable to me. \n2. The approximate Hessian $\\hat{\\Sigma}$ depends on the augmented weights, do you backprop though it when computing the gradients ? \n3. The predictive variance is $J^\\top \\Sigma J$. For multi-output networks like in classification, how do you compute it efficiently ?\n\n**Experiments**\nThis paper conducts experiments for out-of-distribution detection in order to test the performance. However, in my perspective, the experimental results are not strong enough to prove its superiority.\n1. More experiments on uncertainty quantification can be conducted to support the proposed methods. For example, the adversarial example experiment (Ritter et al, 2018) and the calibration experiment (Guo et. al., 2017). These experiments are only my suggestions. I think any experiment regarding to uncertainty quantification would be helpful.\n2. The paper uses Laplacian approximation only for the last layer, and all previous layers act as feature extractor. In this scenario, the proposed method is not that different with a parameterized variance neural network $f_{var}(x)$. Then the Laplacian approximation formulation in the paper is not very useful. Moreover, using only the last layer is not a fair comparison with KFL (Ritter et. al. 2018) either.\n3. Compared with KFL+LL and KFL+OOD, the vanilla KFL seems to be better based on Table 1 and Table 2. However, KFL+LL and KFL+OOD are fine-tuned versions of KFL, it is weird that tuning the prior variance hurts the performance. \n\n**A few typos**\nEq(2) misses the const $\\log p(D)$; \ntwo paragraphs before prop1, \"where d it the resulting number\"\n\n**References**\nJacot et. al., ‎2018, Neural Tangent Kernel: Convergence and Generalization in Neural Networks\nRitter et. al. 2018, A scalable laplace approximation for neural networks\nGuo et. al., 2017, On Calibration of Modern Neural Networks", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a simple modification to Laplace approximation  to improve the quality of uncertainty estimates in neural networks.

The key idea is to add “uncertainty units” which do not affect the predictions but change the Hessian of the loss landscape, thereby improving the quality of uncertainty estimates. The “uncertainty units” are themselves trained by minimizing a non-Bayesian objective that minimizes variance on in-distribution data and maximizes variance on known out-of-distribution data. Unlike previous work on outlier exposure and prior networks, the known out-of-distribution data is used only post-hoc.

While the idea is interesting and intriguing, the reviewers felt that the current version of the paper falls a bit short of the acceptance threshold (see detailed comments by R3 and R2’s concerns about Bayesian justification for this idea). I encourage the authors to revise and resubmit to a different venue.

Decision: Reject