Variational Inference on the Final-Layer Output of Neural Networks
Authors: Yadi Wei
Roni Khardon
Venue: Accepted by TMLR ()

Abstract: Traditional neural networks are simple to train but they typically produce overconfident predictions. In contrast, Bayesian neural networks provide good uncertainty quantification but optimizing them is time consuming due to the large parameter space. This paper proposes to combine the advantages of both approaches by performing Variational Inference in the Final layer Output space (VIFO), because the output space is much smaller than the parameter space. We use neural networks to learn the mean and the variance of the probabilistic output. Using the Bayesian formulation we incorporate collapsed variational inference with VIFO which significantly improves the performance in practice. On the other hand, like standard, non-Bayesian models, VIFO enjoys simple training and one can use Rademacher complexity to provide risk bounds for the model. Experiments show that VIFO provides a good tradeoff in terms of run time and uncertainty quantification, especially for out of distribution data.

Published in Transactions on Machine Learning Research (09/2024)
Variational Inference on the Final-Layer Output of Neural Networks
Yadi Wei
weigadi@ku.edv
Luddy School of Informatics, Computing, and Engineering Indiana Universily
Roni Khardon
rkbardon Oiu. edu
Luddy School of Informatics, Computing, and Engineering
Indiana University
Reviewed on OpenReview; https: // openreveew. net/ net/ forum? id= LLLLLL
Abstract
Traditional neural networks are simple to train but they typically produce overconfident predictions. In contrast, Bayesian neural networks provide good uncertainty quantification but optimizing them is time consuming due to the large parameter space. This paper proposes to combine the advantages of both approaches by performing Variational Inference in the Final layer Output space (VIFO), because the output space is much smaller than the parameter space. We use neural networks to learn the mean and the variance of the probabilistic output. Using the Bayesian formulation we incorporate collapsed variational inference with VIPO which significantly improves the performance in practice. On the other hand, like standard, non- Bayessya models, VIFO enjoys simple training and one can use Rademacher complexity to provide risk bounds for the model. Experiments show that VIFO provides a good tradeoff in terms of run time and uncertainty quantification, especially for out of distribution data.
1 Introduction
With the development of training and representation methods for deep learning, models using neural networks provide excellent predictions. However, such models fall behind in terms of uncertainty quantification and their predictions are often overconfident. (Guo el al.] 20117). Bayesian methods provide a methodology for
uncertainty quantification by placing a prior over parameters and computing a posterior given observed
data, but the computation required for such methods is often infcasible. Variational inference (VI) is one of the most popular approaches for approximating the Bayesian outcome, e.g., (Blundell et al. 2015 Graves 2011 Wu et al.) 2019). By minimizing the KT. divergence between the variational distribution and the true posterior and constructing an evidence lower bound (ELBO), one can find the best approximation to the intractable posterior. However, when applied to deep learning, VI requires sampling to compute the ELBO, and it suffers from both high computational cost and large variance gradient estimation. Wu et al. (2019 have proposed deterministic variational inference (DVT) approach to alleviate the latter problem. The idea relies the central limit theorem, which implies that with sulficiently many hidden neurons, the distribution of the output of each layer forms multivariate Gaussian distribution. Thus only need to compute the
mean and covariance of the output of each layer. However, DVI still suffers from high computational cost and complex optimization.
Inspired by DVI, we observe that the only aspect that affects the prediction is the distribution of the output of the final layer in the neural network. We therefore propose perform variational inference the final-layer output space (rather than parameter space), where the posterior mean and diagonal variance are learned by neural network. We call this method VTPO. Like all Bayesian methods, VIFO induces distribution over

---

Published in Transactions on Machine Learning Research (09/2024)
its probabilistic predictions and has the advantage of uncertainty quantification in predictions. At the same time, VTFO has single set of parameters and thus enjoys simple optimization as in non-Bayesian methods.
We can motivate VIFO from several theoretical perspectives. First, we derive improved priors (or regu- larizers) for VIFO motivated by collapsed variational inference (Tomczak et al, 2021 and empirical bayes (Wu et al. 2019). The new regularizers greatly improve the performance of VTFO. Second, we show that, for the linear case, with expressive priors VIFO can capture the same predictions as standard variational inference. On the other hand, with practical priors and deep networks VIFO exhibits limited expressiveness. We propose to overcome this limitation by using ensembles that enable fast training and further improve uncertainty quantification. Third, due to its simplicity, one can derive risk bounds for the model through Rademacher complexity. Thus, VIFO was motivated as an effective simplification of VI and DVI, but the ensembles of VIFO can be seen as a Bayesian extension of Deep Ensembles [Lakshminarayaaan et al, 2017) We discuss the connections other Bayesian predictors below.
An experimental evaluation compares VIFO with VI and other state of the art approximation methods and with non- non-Bayesian neural networks. The results show that (1) VIFO is much faster than VI and only slightly slower than base models, and (2) ensembles of VIFO achieve better uncertainty quantification on shifted and out-of-distribution data while preserving the quality of in-distribution predictions. Overall, VIFO provides a good tradeoff in terms of run time and uncertainty quantification especially for out-of-distribution data.
2 VIFO
In this section we describe our VIFO method in detail. We start with description of the non-Bayesian base model. Given a neural network parametrized by weights W and input æ, the output is fw fw(x) RK The base model provides probabilistic predictions by combining the output of the network with any prediction likelihood p(u]=). Traditional, non-Bayesian models, minimize (ogg(y)(2) or a regularized variant.
Remark 2.1. The base model and VIFO are applicable with any likelihood function and our development of VIFO below is general. To illustrate we discuss classification and regression. In classification, K is the number of classes. The probability of being class i is defined as
ply = 112) softmax(x)) = E, exp
exp
(1)
In regression, = (m,l) 1) is a 2-dimensional vector and == 2. We apply a function g on l that maps I to a positive real number. The probability of the output is:
=
exp (- (y m)²
p(ulz) N (,y,m, (()) /2mg(1)
(2)
(g(
By fixing the weights W, base models map x to z deterministically. Bayesian inference puts a distribution over W and marginalizes out to get a distribution over from which predictions can be calculated. Since exact marginalization is not tractable, variational inference provides an approximation which yields the well known ELBO objective for optimization:
p(W, D)
log 2 ()(W) log (WW)
Σ (ED))
[oww)logp KL(((W)lP(W)))
(3)
As shown by Wu et al. (2019), by the central limit theorem, with sufficiently wide neural network the marginal distribution of z is Gaussian. DVI explicitly caleulates an analytic approximation of the mean and variance of the output of each layer (valid for specific activation functions) and avoids the sampling typically used for optimization of the ELBO in other methods.
VIFO pursues this in a direct manner. It has two sets of weights, W, and W2 (with potentially shared compo- nents), to model the mean and variance of z. That is, (1(x) = fw, (z), (x) = ((fw, where g R + R+ maps the output to positive real numbers as the variance is positive. Thus, q<2[x)) = =N(()n,((>) diag((())) where py(x), of (z) are vectors of the corresponding dimension. We will call (2((2) the variational output distribution. As in the base model, given 2, y is generated from the likelihood p(y]z).
2

---

Published in Transactions on Machine Learning Research (09/2024)
taan gowd
pewc
(a) without auxilarry training
(b) with auxiliary training
Figure Predictive distribution of VIFO using MLP. Blue points are training data generated from a sinusoidal function, red points are the predicted mean, shaded area indicates the standard deviation. More details are in Appendix D.I.
Remark 2.2. VIFO in regression is different from the existing models known as the mean-variance estimator (Kabir et al. 2018 Khosravi et al. 2011 Kendall & Gal 2017). Instead, mean-variance estimators are the base models that VIFO can be applied on. Applying VIFO to these models results in four outputs: Am and Mt, which are the means of and I, and o and of, which are the variances of m and 1. These variances come from the variational output distribution. We sample m NN o and I ~ Ni,, of), then form (m,l). Like all Bayesian methods VIFO computes distribution over distributions which is lacking in non-Bayesian predictions.
Unlike VI which puts prior over W, VIFO models the distribution over z and therefore we put a prior directly over We consider two options, conditional prior p(2[ø) and simpler prior p(z). Both of these choices yield valid ELBO using the same steps:
logp(y/z) 2 Eg(=(s) log ply, 2|x) = (= [log p()]=) [L(g([[))ll[((()))
(4)
((=(x)
The approach has some similarity to Dirichlet-based models (Sensoy et al. 2018 Charpentier et al., 2020 Bengs et al.) 2202). However, we perform inference on the output whereas, as discussed by Bengs et al.) (2022), these models implicitly perform variational inference is interpreted as vector in the simplex and q(>(xr) and p(z) are Dirichlet distributions, whereas when using VIFO for classification has Gaussian distribution and p(y) =)) is the simplex. In other words, we model and regularize different distributions. We discuss related work in more details below.
the prediction. In particular, in that work
Eq. 4 is defined for every (x,y), y). For a datasot D = {(x, we optimize W1 and W2 such that
Σ (oocss))oggglull [[(((a)))
(X.))DD
is maximized. We regard the negation of the first term log]()]>)] the loss term and treat (L(((===))l<<(z)) as regularizer.
2.1
Auxiliary Training
As in prior work (Sun el al. Taux and include (L(g(z) (2) Taux)
2019 to improve uncertainty quantification introduce auxiliary input
p2z Faux)) ) as an additional regularization term. We include corresponding coefficients and Maxx on the regularizers, as often done in variational approximations (c.g., (Higgins et al.
2017 Sheth Khardon| 2017 Jankowiak et al. 2020 Wenzel et al. 2020 Wei et al.) 2021 Wei Khardon. 2022 )) Then, viewed as regularized loss minimization, the optimization objective for VIFO becomes:
min Σ W. (R.y)ED
[ log (KL Maux
(5)
Panx
3

---

Published in Transactions on Machine Learning Research (09/2024)
Generally the loss term is intractable, auxiliary data not available, we uniformly sample xaux Unif[= 2 Tmax + 2 where d = Imax I min
Monte Carlo samples to approximate it. In practice, since d (2)
(3) (8)
for each entry i. Figure [ shows an example where an MLP used to learn a complex function over
dimensional imput space, illustrating that such regularization can improve uncertainty quantification in the area where the data is missing.
2.2 Collapsed VIFO
Bayesian methods often sensitive to the choice of prior parameters. To overcome this, Wu et al. (2019) used empirical Bayes (EB) to select the value of the prior parameters, and Tomezak et al. (2021) proposed collapsed variational inference, which defined a hierarchical model and performed inference on the prior parameters as well. We show how these ideas are applicable in VIFO and derive empirical Bayes as a special case of collapsed variational inference. In addition to 2, we model the prior mean /4p and variance of as Bayesian parameters. Now the prior becomes p(z)ppp oz)p(ppp oB) and the variational distribution is ((2)=))((pp)))) oz) Then the objective becomes:
plu, 2, Up, 22)()
log pogl())>) 2 Bygale))o of) log q(2l=) o,)
= (logsl))
E ( [KL(q( of)1] KL(g(up (p)llp(p of)).
(6)
Similar Eq. (5, we treat the first term as a loss and the other two terms as a regularizer along with a coefficient 7) and aggregate over all data. Since the loss does not contain (4p and og. we can get the optimal q" (ppp by optimizing the regularizer and the choice of 17 will not affect q" (14, o)). Then the value of q² into Eq. (6) We next show how to compute q" (Mppaa) and the final collapsed variational inference objective. The derivations are similar to the ones by Tomczak et al. [2021) but they are applied on z not on W. Recall that K is the dimension of z.
plug in
Learn mean, fix variance Let p(=)up) = (2 71), p(u)) ((1,),,al). Then q" (uppx) is
arg min | (KTgglll)) (TGgo))))))))) <(7)
and the optimal q" (uplzz) can be computed as:
log q" (ppl>) x (my(Z) Mp) (p) (My(2)
Mpp)
My Mp
27
2xx
and (1pp=>) = (1pl - 1ira----g (x), an)) Notice that, unlike the prior, g" (pz) depends on x. If we put g"' back in the regularizer of Eq. 6 the regularizer becomes:
7 1 (T)()) + 7 ( (x(())
K 1 log (ogo)(x) + 2 log(> + +a) 2
K
27
As in Tomezak et al. regularization on (z) We refer to this method as "VIFO-mean!
2 < 1 in front of (x( po(x), which weakens the 2021 Eq. (7) puts factor 7T0
Figure 2 shows the learned prior for VIFO-mean and VI for the same example as in Figure I We observe
that VIFO-mean allows for diverse prior distributions and captures the data distribution well.
Other Regularizers The same approach be used for joint prior p(z)mp: of) W= of): p(pp) NiHH, 10, {af), p(oz) = [ 8). where IG inverse Gamma, yielding method we call "VIFO-mv". Similarly, the hicrarchical prior in empirical Bayes models the variance but not the mean p(o I p(zla)) = ==]00, of) and yields "VIFO-eb". Derivations are given in Appendix B
4

---

Published in Transactions on Machine Learning Research (09/2024)
-
(a) VIFO-mean
(b) V1-naive
(c) VI-mean
Figure 2: Induced predictions by learned prior distribution for different methods. Note that VI has prior over weights and VIFO has a prior over z. For each method we sample values from the prior and calculate predictions y based on the sampled values. We then plot the y values. As we can see, Vl-naive induces a uniform prior that does not capture the data distribution, VI-mean has an increasod variance in arcas where data is missing and VIFO-mean does larger extent. Details are given in Appendix D.1
3 Expressiveness of VIFO
VIFO is inspired by DVI and it highly reduces the computational cost. In this section we explore whether VIFO can produce exactly the same predictive distribution as VI. We show that this is the case for lincar models but that for deep models VIFO is less powerful. We first introduce the setting of linear models. Let the parameter be 0, then the model is:
y/2, aT)
(8)
For example, p(y)0Tz) N x, 1)) where 8 a constant for Bayesian linear regression; and p(y 110T x) 1Tx for Bayesian binary classification.
For simplicity, we assume f e Rd, where is the dimension of æ, and then the output dimension K 1. The standard approach specifies the prior of to be p(0) (/ ma So), and uses (0) = ( m. 5). Then the ELBO objective, with dataset X2= (I1, T2,..... ... INN e R and YN = (y1 V2. ........ yN) E RN
N Eo log x.)] KT(((()((p(()))
Σ Ey [og Dogr(us00 xz)] 1 [Ir(S log | 8|] (m -m 1 mo) + d
(9)
As the following theorem shows, if we use a conditional correlated prior and a variational posterior that correlates data points, then in the linear case VIFO can recover the ELBO and VI solution. We defer the proof and discussion of K 1 to Appendix A.I
Theorem 3.1. Let (<2|x) N(=lw x, x Vx) be the variational predictive distribution of VIFO, where w and V. are the parameters be optimized, and let p(2[XN) (2 N XN,XN and {22[XN) (22/w XN, X VXN) be correlated and data-specific prior and posterior (which means that for different data H, have different prior/pos priorpootterior over 2). Then the VIPO objectice is equivalent to the ELBO objective implying identical predictive distributions.
T
However, the next theorem shows, for the non-linear we cannot produce the variational output distribution q(2(2)) as if it is marginalized over the posterior on W.
Theorem 3.2. Given neural network fw parametrized by W and mean-field Gaussion distribution (W) over W, there may not exist set of porameters W such that for all inpul x we have Ff (2)) = fw (2).
5

---

Published in Transactions on Machine Learning Research (09/2024)
The proof is given in Appendix A.1 The significance of these results is twofold. On the one hand, we from Theorem 3.2 and the conditions of Theorem
3.1 that the representation is more limited, i.e., efficiency comes at some cost. On the other hand, Theorem 3.1 shows the connection of VIFO to VI, which gives better perspective on the approximation it provides. Morcover, this facilitates the use of cxisting improvements in VT for VIPO such as collaposed VT applied to VIFO.
In practice, a correlated and data-specific prior p(z|z) is complex, and tuning its hyperparameters would be challenging. Hence, for a practical algorithm propose to use simple prior p(z) independent of In addition, to reduce computational complexity, we do not learn full covariance matrix and focus the diagonal approximation. These aspects limit expressive power but enable fast training of VIFO and hence also ensembles of VIPO.
4 Rademacher Complexity of VIFO
In this section we provide generalization bounds for VIFO through Rademacher Complexity. We need to make the following assumptions. These assumptions hold for classification and with smoothed loss for regression as shown in Appendix A.2
Assumption 4.1. logp(yl=) is Lo-Lipschitz in ie., logp(ylz) log p(y))')) s Lollz 2'112.
Assumption 4.2. The link function g is Li-Lisschitz
Recall that the Rademacher complexity of set of vectors A S RN is defined as R(A) N [La[(-1,1 (suPOEA o.a.). The Rademacher complexity of the set of loss values induced by functions E over a datasct S has been used to derive gencralization bounds for learning of the class F. We need the following technical lemma, proved in Appendix A.2 that generalizes well known Lipschitz based bounds (Shalev-Sae Shalee---waazzza & Ben-Davice 2014) to multi-input functions.
Lemma 4.3. Consider an L-Lipschitz function Φ R + R, i.e δ(a1,b)) 6(a2,by) S L(la1 a2 + bb b2l)- For b IRN, let Φ(a.b) denote the vector (d(a),b));.. (ol,,, bN)). Let o(A B) denote {o(a, b) a A, B}, then
R(O(A x B)) S T(R(A) R(B)).
(10)
Applying the previous lemma sequentially over multiple dimensions we obtain: Corollary 4.4. Consider an L-Lipschitz function & : Rd + R, i.e., for any x, E Rd, M(x) - 6(x)) s L||> 2'll1. Let p(A*)= {Ø(01.ds) a1. a2...., ad A (ACRM, then R(ő(A")) S LdR(A).
With the assumptions and technical lemma, we derive the main result:
Theorem 4.5. Let H be the set of functions that can be represented with neural networks with porameter E W}. VIFO has two components, so the VIFO hypothesis class is H. x H {(fw; ()), () fw; (-) IW- (2), W1, W2 E W]. Let be the loss function for VIFO, 1(W, (x. |- (xlo) ()))())))]] Then the Rademacher complexity of VIFO is bounded R(I o (H H) 8) < 2(Lo (1, R(H S), where K is the dimension of and S is training dataset.
space W, H =
(F2),W
The proof is in Appendix A.2 and it shows how reparamertization can facilitate computation of Rademacher bounds for Bayesian predictors. The Rademacher complexity for VIFO is bounded through the Rademacher complexity of deterministic ncural networks. This shows one advantage of VIFO which is more amenable to analysis than standard VI due to its simplicity. Risk bounds for VI have been recently developed (e.g.. (Germain et al. 2016 Sheth & Khardon, 2017)) but they require different proof techniques. The Rademacher
(Golowich et al. 2018), where Bw bounds the norm of the weights and Bg bounds the input. The Rademacher complexity of VIFO of the same order.
complexity for neural networks is O Bw B vN
5
Related Work
VIFO is related to but distinct from number of variational and one-pass methods. Dirichlet-basse methods
Sensoy et al. 2018 Charpentier et al. 2020 Bengs et al. 2022 discussed above, implicitly perform vari-

---

Published in Transactions on Machine Learning Research (09/2024)
ational inference on the prediction and the network output provides parameters of Dirichlet distribution. Like VIFO they provide Bayesian predictions in a single pass over the network, but their relation to the standard variational inference in parameter space is non obvious. On the other hand, VIFO is a single pass method clearly related to VI in paramcter space which enables the benefits of collapsed variational inference. Thus VTFO can be seen to bridge between Dirichlet methods and VI. DUQ van Amersfoort et al. 2020) provides an alternative approach using one pass on the network. It first embeds examples into a latent space, similar to 2, but computes classification prediction and uncertainty quantification through RBF distances to centroids of classes in that space. Hence its predictions are very different. Another related line of work (Sun et a 2019: TTran et al. 2022) performs variational inference in function space. However, they focus on choosing& a better prior in weight space which is induced from Gaussian Process priors on function space, whereas VIFO directly induces a simple prior on function space. [harmaa el al. (2023) model the distribution of the last layer by adding random noise as imput and do not give an explicit form of the output distribution.
VIFO differs from other existing variational inference methods as well. The local reparamctrization trick (Tomczak et al., 2020; Oleksiienko et al. 2022) reduces the variance from sampling in VI. This is done by performing two forward passes with the mean and variance at each layer before sampling the output for the layer. Hence this modifies the sampling process of VI whereas VIFO only requires one pass on the network and samples only the output of the last layer for prediction.
Last-layer variational inference (Brosse et al. 2020 Kristiadi et al. 2020 Daxberger et al. 2021 Liu et al. 2020 Harrison et al. 2024) performs variational inference on the parameters of the last layer, while we perform variational inference on the output of the last layer. Note that the last layer usually contains more parameters than the output which has constant size. Last Layer Laplace Kristiadi et al. (2020); Daxberger et al. (2021 reduces training complexity by first estimating the MAP solution, and then estimating the covariance of the parameters in one pass. SNGP (Liu et al. 2020) is a variant of this method, that aims to mimic the sensitivity of Gaussian processes to distances among examples, by incorporating Fourier features at the one to last layer. Finally, VBLL (Harrison et al. 2024 still maintains a distribution on last layer parameters, but approximates the expectation over these parameters in closed form (for specific likelihoods)) to reduce the complexity of training and prediction. Thus, all these methods are much closer to VI because they maintain distributions over weights whercas VIFO produces distributions in output space.
VIFO shares some aspects with the model of Kendall & Gal (017), where both use neural networks to output the mean and covariance of the last layer. However Kendall & Gal (2017) use the cross entropy loss, - log By(=ts))(V1 22) instead of our loss in Eq. (5), they use dropout for epistemic uncertainty, and their objective has no explicit regularization. Hence unlike VIFO their formulation does not correspond to a standard ELBO. It is also interesting to compare VIFO to the Deep Variational information bottleneck (Alemi et al. 2017). The model is motivated from a different perspective but its final optimization objective, obtained after some approximations, is similar our Eq. 4.. In this sense the model is close to the VIFO- naive. However, in their formulation, z is the output of a bottleneck layer which is not the final layer (because it is meant to constrain the information that flows to the final layer), and p(z) which is the prior in our model is a posterior on the marginal posterior on z. Nonetheless, our development of rich priors through collapsed inference can help inform the choice of p(z) in that model, which is typically taken to be a standard Normal.
Various alternative Bayesian techniques have been proposod. One direction to get samples from the truc (ennzll posterior, as in Markov chain Monte Carlo methods Wenzel et al. (2020); Iamailov et al. (2021) Expectation propagation aims to minimize the reverse KT. divergence the true posterior (Teh et al. 2015 Li al. 2015). These Bayesian methods, including variational inference, often suffer from high computational cost and therefore hybrid methods were proposed. Stochastic weight averaging Gaussian (Maddox et al. a Gaussian distribution over parameters from the stochastic gradient descent trajectory in the base model. Dropout (Gal& & Ghahramani 2016) randomly sets weights 0 to capture uncertainty in the model. Deep ensembles (Lakshminarayaaaan et al, 2017) use ensembles of base models learned with random initialization and shuffling of data points and then average the predictions. These methods implicitly perform approximate inference. In addition these methods, there are also non-Bayesian methods to calibrate overconfident predictions, for example, temperature scaling (Guo et al., 2017) introduces temperature parameter to anneal the predictive distribution to avoid high confidence. VIFO strikes a balance between simplicity and modelling power to enable simple training and Bayesian uncertainty quantification. On the one hand, VIFO
2019 forms
7

---

Published in Transactions on Machine Learning Research (09/2024)
can be seen a simplification of VI. On the other hand, it can be seen an extension of the base model. From this perspective, the use of ensembles of VIFO, which extend the ensembles of Lakahmiaaaaaaannn et al. (2017), are highly motivated as a practical algorithm.
6 Experiments
In this section, we compare the empirical performance of VIFO with VI and hybrid methods that use the base model. In VIFO, W and W2 share all parameters except those in the last layer. VI candidates include the VI algorithm "VVI-naie"(Blundde et al. 2015)) with fixed prior parameters, and other variations from collapsed variational inference (Tomczak et al. 2021 and empirical Bayes (Wu et al. 2019). Non- Bayesian and hybrid methods include the base model ("SGD", because uses stochastic gradient descent optimizer), stochastic weight averaging ("SWA", which uses the average of the SGD trajectory the base model as the final weights) from Ixmailov et al. (2018) and SWA-Gaussian ("SWAG", which uses the SGD trajectory to form a Gaussian distribution over the neural network weight space) from Maddox et al. (2019). We use ensembles of the base models which are known as deep ensembles (Lakshminarayanan et al. 2017); ; and the ensembles of SWAG models, which are the multiSWAG model of Wilson & Izmailov (2020), ; both of which are considered strong baselines for uncertainty quantification (Ovadia et al. 2019). In addition to these methods, we include other approximate Bayesian algorithms for comparison. These include repulsive ensembles ("Repulsive", (Daiiiie""""""iuai""""""
(D'Angelo & Fortuin 2021)), the Dirichlet-based model ("Dir", (Sensoy et al. 2018)), dropout (Gal & Ghahramani 2016)), last layer Laplace with prior optimization ("Laplace", [Daxberger et al. 2021 and variational Bayes last layer ("VBLL", (Harrison et al. 2024h) Our main goal
is to show:
VIFO is much faster than VT and only slightly slower than base models;
Ensembles of VIFO preserve the quality of in-distribution predictions;
Ensembles of VIFO achieve better uncertainty quantification on shifted and out-of-distributio (OOD) data than all basclincs.
For our main experiments, we pick four large datasets, CIFAR10, CIFAR100, SVHN STL10, together with two types of neural networks, AlexNet (Krizhevsky et al. 2012) and PreResNet20 (He et al. 2016). The regularization paramcter 17 is fixed to 0.1 for both VIFO and VI, as this choice yields better performance compared with the standard choice y = Empirically we observe that using collapsed variational inference in VI does not improve the performance. This is because Tomezak et al (2021) used 17 to obtain their results whereas we use 1 0.1 which provides much stronger bascline. For auxiliary training, we experiment with Mlax E (0.0, 0.5, 1.0]. Larger values of Taux generally improve OOD data detection at the cost increased in-distribution loss, and there is no generic optimal value of Maux- In our main paper, we present only the case where 7/axx = 0.1 because it provides a balance between in-distribution and OOD performance, with performance other choices of Maux provided in the appendix. In addition, VIFO-mean and VIFO-mv perform better than other variants of VIFO. Thus, we only list these variants in our main paper and provide full results for other variants for VIFO and VT in the appendix. For each method 5 independent runs and report means and standard deviations in results. Complete details for the setup and hyperparameters are given in Appendix D.2 Our code is available on https: com com/weiyadi/VIFF
6.1 Run Time
Ignoring the data preproccssing time, we compare the run time of training epoch of VI, VIFO and the base model. In Table D we show the mean and standard deviation of 10 runs of these methods. Different regularizers do not affect run time, so we only show that of VI-naive for VI and VIFO-mean for VIFO. In addition, as shown Table I. VIFO is much faster than VI and slightly slower than the base model. As shown in Fig. E.7 VIFO converges faster than, or as fast as VI. Consequently, the training time until convergence for VIFO is shorter than for VI.
8

---

Published in Transactions on Machine Learning Research (09/2024)
Table 1: Running time (seconds) for training 1 epoch with batch size 512, AlexNet
dataset
CIFARIO
CIFARIO0
SVHN
STL10
size
50000
50000
73257
500
VI
8.51
0.41
8.27 0.40
11.56 0.39
1.75 + 0.41
VIFO
2.18 = 0.39
2.17 0.43
2.72 = 0.38
1.16 + 0.40
base
1.97 0.41
1.99 + 0.43
2.46 + 0.40
1.12 + 0.38
Table
Test log loss (4) of single VTFO and ensembles of VIFO.
VIFO-mean
VIFO-mv
single
ensemble
single
ensemble
CIFAR10
0.527 = 0.015
0.345 = 0.003
0.626 = 0.010
0.324 0.001
CIFAR100
2.253 + 0.032
1.688
0.006
2.688 + 0.029
1.725 0.003
STL10
1.333 = 0.065
1.055
0.008
1.531 = 0.019
1.123 0.008
SVHN
0.509 + 0.029
0.351 0.005
0.520 = 0.027
0.298 + 0.009
The differences in run time are dominated by sampling and forward passes in the network. Let P denote the number of parameters in the base model and thus each forward/ haardd/bbwwadddd pass takes O(P) time. The time complexity for computing the loss for each output of the base model is O(1). The base model only needs 1 forward pass without sampling and thus the time complexity is O(P). VIFO needs 1 forward pass and M samples to compute the loss so the time complexity is O(P + M). VI needs M samples the parameter space and M forward passes, thus the time complexity O(PM + M) = O(PM). The same facts apply for prodictions on test data, where the advantage can be important for real time applications.
6.2 Ensembles of VIFO
points out that the expressiveness of VIFO is limited. To overcome this, we use ensembles of VIFO, which independently train multiple VIFO models and average their predictions. Section 0.00661leaatt establishss fast training of VIFO, allowing us to train VIFO models simultancously while still maintaining the running time advantage of VIFO. We investigate the impact of ensemble size on performance in Appendix E.5 While increasing the ensemble size enhances performance, the improvement diminishes once the size exceeds 5. Therefore, we choose an ensemble size 5. 5. Table shows that with ensembles, VIFO with auxiliary training achieves much better log loss than when using a single model. The same holds without auxiliary training. This indicates that ensembles of VIFO are much more expressive than single VIFO. In the following experiments, use ensembles of VIFO. For fair comparison in the remainder of the paper, we use ensembles for all methods except for VI (which is time-consuming) and repulsive ensembles (which are themselves ensembles).
Theorem 3.2
6.3 In-distribution Performance
In this section we use log loss and accuracy to measure the performance for in-distribution data.
Fig. 3 and Fig. F.2 Appendix compare main methods in terms of log loss. First, we observe that repulsive ensembles and the Dirichlet method have much worse log loss than all other methods and they tend to give underconfident predictions. Second, observe that using auxiliary training slightly increases the log loss but the increase is negligible. Later for out-of-distributtin data. We observe that VIFO is competitive with all methods in terms of log loss, with relatively small differences between the top group of methods in each case. Fig. E. and Fig. E.3 show accuracy on test data in the same experiments, revealing that in many cases VIFO outperforms VI and it competitive with all methods. Finally, there no clear winner between VIFO-mean and VIFO-mv; VIFO-mv provides a small advantage overall but might be more sensitive as illustrated by the performance CIFAR100 with PreResNet20.
see that auxiliary training improves the uncertainty quantification
9

---

Published in Transactions on Machine Learning Research (09/2024)
1.0
0.9
2
0.5
20-
0.4
15-
0.3
Up0-mmest
WKD.oy
50D
sata
SWAG
Repiisiv
Oir
Dronsud
Laptoco
VOLL
VPO mesn
wD-my
spo
Sak
SWAG
Repritive
Oir
Drapost
Lapsocce
HELL
(a) CIFARIO
(b) CIFAR100
18
16-
12-
10.
WPO-nest
W50.ov
Inssive
SMd
Shoa
Remusive
Dr
Drspout
Lostare
VELL
WrO.a
Woniee
500
SMit
SWaa
Reprisive
Dr
Drapout
Lagtacc
HBLL
(c) STL1O
(d) SVHN
Figure 3: Test log loss (4) on PreResNet20. Dashed lines indicate the best version of VIFO. The error bar is three times of the standard deviation for better visualization and same for other figures.
6.4 Uncertainty Quantification
In this section we examine whether VIFO can capture the uncertainty in predictions for shifted and OOD data. We measure performance using ECF,, Entropy and AUC for detecting OOD data. These represent compreheusive set of measures from the literature. For datasets, for uncertainty under data shift, STL10 and CIFAR10 can be treated as a shifted dataset for each other, as the figure size of STL10 is different from CIFAR10, and STL10 shares some classes with CIFARIO so the labels are meaningful. For uncertainty under OOD data, we choose the SVHN dataset as an OOD datasct for CIFAR10 and STL10, as SVHN contains images of digits and the labels of SVHN are not meaningful the context of CIFAR1O.
Expected Calibration Error (ECE) ECE NNeinn al. 2015 Ovadia et al. 2019 is often used to measure the uncertainty quantification under data shift. We separate data into bins of the same size according to the confidence level, calculate the difference between the accuracy and the averaged confidence in cach bin and then average the absolute differences among all bins. Better calibrated models have lower ECE, ECE has its faults (for example the trivial classifier has zero ECE) but it is nonetheless informative. We selected the number bins to be 20.
Fig. shows the ECE of each method under data shift. As we can see, both VIFO-meat and VIFO-mv achieve the best performance compared to all other methods.
Entropy Entropy OOadia et al. 2019 of the categorical predictive distribution is used to measure the uncertainty quantification for out-of-distribattion (OOD) data as the labels for OOD data are meaningless. We want our model to be as uncertain as possible and this implies high entropy and low confidence (the maximum probability assigned to any class) the predictive distribution. We summarize the averagod entropy for the entire dataset in Fig. 5 and Fig. E.4 We can see that both VIFO-mean and VIFO-mv are better than all other methods except repulsive ensembles and the Dirichlet method. However, as observed
10

---

Published in Transactions on Machine Learning Research (09/2024)
0.0
0.25
0.35
0.00
0.10
0.05
WiO-maso
WFO my
Winenie
8SD
Sait
SHis
Hepustire
Dir
Dreptsst
Laplane
VOLL
WP0-mann mv
Woaine
$50
swi
sWit
Hecrdure Dir
Drapust
Laplane
VOLL
(a) CIFARIO-STLIO, AlexNet
(b) STLIO-CIFARIO, AlexNet
0.35
0.30
0.2
0.25
0.20
0.15
0.15
010-
0.10
0.0
0.00
mP0-masn
WFD.m mg
Monele
850
SWi
SWAG
Di
Drspost
Lapiaae
VELL
VeO-mesn
Wr0-ow
Mostie
960
SWit
SWno
Remosure
Di
Drspoost
HRLL
(c) CIFARIO-STL.1,, PreResNet20
(d) STT.10-CTFAA1O, PreResNet220
Figure 4: ECE (4) on AlexNet and PreResNet20 under data shift. Dashed line indicates the best performance of VIFO. Numerical results are listed the Appendix.
in Fig. 3 and Fig. E.2 repulsive ensembles and the Dirichlet method have poor performance in terms of log loss due to underconfient predictions. Hence they achieve high entropy by sacrificing in distribution performance whereas VIFO performs well. Further, we observe from Fig. 5 that auxiliary training greatly improve the performance of VIFO on PreResNet20. Auxiliary training only has a small impact on VIFO with AlexNet (see Fig. E.4 but VIFO already performs well without auxiliary training in this case.
We use maximum probability of the categorical predictive distribution as the criterion to separate in-distribution and OOD data and compute the area under the ROC curve (Malinin & Gales) 2018). AUROC overcomes the drawbacks of ECE and entropy bocause a trivial model cannot yicld the best performance. Detailed comparison plots are in given in Fig. E55 and Fig. E.6 in the Appendix. We first note that, as above, auxiliary training improves the performance on Net2 but not significantly on AlexNet. We found that there is no single method that consistently outperforms all other methods, Instead, for better visualization, show the comparison of VIFO-mean and VIFO-mv with other methods in Fig. G For each baseline, we count the number of experiments that VIFO performs better and get the corresponding proportion. We observe that overall, VIFO-mv is better than all other methods except the Dirichlet method and that it ranks better than VIFO-mean. Though the Dirichlet method performs better than VIFO on OOD data, its poor in-distribution performance makes less desirable. On the other hand, VIFO outperforms all other baselines for OOD and has strong in distribution performance and hence give better overall predictions.
AUROC
7 Conclusion
In Bayesian neural networks, the distribution of the last layer directly affects the predictive distribution. Motivated by this fact, we proposed variational inference on the final-layer output, VIFO, that uses neural network to directly learn the mean and variance of the last layer. We showed that VIFO can match the expressive power of VT in linear cases with strong prior but that in general it provides less expressive model.
11

---

Published in Transactions on Machine Learning Research (09/2024)
22-
22.
20-
10:
1.0
1.6
1.6
144
12-
10.
10
08
Upr0-mesn
WDD.oy
SGD
SWAa
Repiisie
Of
Laptone
VOLL
Weo meen
WD.my
5co
sai
Swas
Repritive
or
Drapost
Lapsacce
HBLL
(a) CIFARIO-SSHNN
(b) STLIO-SVHN
22
2.0
2.6
10-
WrO-nesh
W50.or
Insiive
Sadd
Remusivv
Dr
Drspout
Lostace
VALL
180 mann
Wosiee
s0D
Stit
SHaat
Raprisive
Drr
Drapost
HELL
(c) SVHN-CIFARIN
(d) SVHN STL10
Figure 5: Entropy (t) on PreResNet20.
1.0
vifo-meen
Proporton that WIF is beteer
0.4
0.2
0.0
Dropout Leplace
Figure 6: Comparison of VIFO with all other methods in terms of AUROC on OOD data. Y-axis is the proportion of experiments that VIFO is better than other methods. Exact AUROC values are provided the appendix.
On the other hand the simplicity of the model enables fast training of ensembles of VIFO and facilitates convergence analysis through Rademacher bounds. In addition, VIFO can be derived as non-standard variational lower bound, which provides an approximation for the last layer. This connection allowed us to derive better regularizations for VIFO by using collapsed variational inference over a hierarchical prior. Since VIFO treats cach input separately, we can incorporate auxiliary inputs to help the model distinguish in-distribution and out-of-distribution data. Empirical evaluation highlightod that ensembles of VIFO are competitive with outperform other methods in terms of in-distribution loss and out-of--istrrbuttion data
12

---

Published in Transactions on Machine Learning Research (09/2024)
detection. Hence VIFO gives new attractive approach for approximate inference in Bayesian models. The efficiency of VIFO also means faster test time predictions which can be important when deploying Bayesian models for real-time applications. Future work could explore more informative auxiliary input to improve the performance of VIFO, and investigate the connections to variational inference in functional space that induces more complex priors.
Acknowledgments
This work was partly supported by NSF under grants 1906694 and 2246261. Some of the experiments this paper were run on the Big Red computing system at Indiana University, supported in part by Lilly Endowment, Inc., through its support for the Indiana University Pervasive Technology Institute.
13

---

Published in Transactions on Machine Learning Research (09/2024)
References
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy. Deep variational information bottleneck. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. Pitfalls of epistemic uncertainty quantification through loss minimisation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Aduances in Neural Information Processing Systems, 2022. URL. https:// oot.pp/rrriiiee.ettoouurrrt net/forum?id=
epjxT ARZW5
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1613 1622, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceecding mml. /ttppp/ceeeiiii.......rrer////// http:///////bunne111...1
Nicolas Brosse, Carlos Riquelme, Alice Martin, Sylvain Gelly, and Éric Moulines. On last-layer algorithms for classification: Decoupling representation from uncertainty estimation, 2020. URL https: /tttss//arxiv. org/ attaaabb.//.044/0
Bertrand Charpentier, Daniel Zügner, and Stephan Günnemann. Posterior network: Uncertainty estimation without ood samples via density-based pscudo-counts. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1356- 1367. Curran Associates, Inc., 2020. URL 160 .dff
https////ppppeetlsspppppeeeee files/paper/
Francesco D' D'Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Sys- tems, 2021. URL https://opennrvvies wtt/foormmid--AAlpppppa
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux- effortless bayesian deep learning. In Advances in Neural Information Processing Systems 34: NeurIPS, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML'16, pp. 1050-1059. JMLR.org, 2016.
Pascal Germain, Francis Bach, Alexandre Lacoste, and Simon Lacoste-Julien. Pac-bayesian the- ory meets bayesian inference. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran As- sociates, Inc., 2016. URL https: http:///pooeeeee..........ee coourips.ccpappee /tt//pppe_fllleeppper///////// 442200112222933336999999999
.dff
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural net- works. Tn Sébastien Bubeck, Vianney Perchet, and Philippe Rigollet (eds.), Proceedings of the 31st Confer- ence On Learning Theory, volume 75 of Proceedings of Machine Learning Research, pp. 297 299. PMLR, 06-09 Jul 2018. URL https://proceeddngg mll http:///wwwoooo.o........ html
Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, vol- ume 24. Curran Associates, Inc., 2011. URL. https://prroceednng: weeurips.cc/ppper/20/11// Paper. 1ebccba33441eeebaa00000esss33 Paaeer--papppppp.fff.... pdf.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1321-13380 JMLR.org, 2017.
James Harrison, John Willes, and Jasper Snock. Variational bayesian last layers. In 5th International Conference on Learning Represendations, ICLR 2017, 2024.
14

---

Published in Transactions on Machine Learning Research (09/2024)
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision ECCV 2016, pp. 630 645, Cham, 2016. Springer International Publishing. ISBN 978-3-319-4699-00.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. bela-VAE: Learning basic visual conceрts with a constrained variational framework. In International Conference on Learning Representations, 2017. URL https: 7/openreview.o http://w..../errrmmm/eeeetttaaa
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Fhirth-Fourth Conference on Uncertainty in Artificial Intelligence, UAT 2018, Monterey, California, USA, August 6-10, 2018, pp. 876 885. AUAI Press, 2018. URL http: 7/auai //auai.org/uai2000 http://aaa200//porroedeagg/////p 31tp:////n../.ppp/33333../3 pdf
Pavel Izmailov, Sharad Vikram, Matthew D Holfman, and Andrew Gordon Gordon Wilson What are bayesian neural network posteriors really like? In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learn- ing Research, pp. 4629-4640. PMLR, 18-24 Jul 2021. URL https://proceeddii mlr, pross////s/
E.mailov21a.htmm
Martin Jankowiak, Geolf Pleiss, and Jacob Gardner. Parametric Gaussian process regressors. Tn Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 4702 4712. PMLR, 13 18 Jul 2020. URL https: :
//proceedings.o mlr. www.ppresssivviee apees///11//aaoooiaaabba..... jww.o/aaaaaa.aaaacccooa html
H. M. Dipu Kabir, Abbas Khosravi, Mohammad Anwar Hosen, and Saeid Nahavandi. Neural network-based uncertainty quantification: A survey of methodologies and applications. IEEE Access, 1:36218-3623 2018. doi: 10.1109/ACCESS.201 2tt.S................
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In T. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: //proceedings. wewpeediiinnnneuiii...e..
pdf
Abbas Khosravi, Saeid Nahavandi, Doug Creighton, and Amir F. Atiya. Compreheusive review of neural batwork-based prediction intervals and new advances. IEEE Transactions on Neural Networks, 22( 1341- 1316, 2011. doi: 10.1109/TNNS 2000/TTN.22011.22211000
Agustimus Kristiadi, Matthias Hein, and Philipp Hennig, Being bayesian, even just a bit, fixes overconfidence in relu networks. In Proceedings of the 37th International Conference on Machine Learning, ICML, volume 119 of Proceedings of Machine Learning Research, pp. 5436-544 5446, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Percira, C.J. Burges, L. Bottou, and K.Q. Weinberger (eds.), Advances in Newral Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL htttps:/pproceedingg .ddf.
neurips.cc/ http:ppc....ppe.ooc../cc/ceeeeee
Balaji Takshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles. In 1. Guyon, U. Von Luxburg, S. Bengio, Н. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL 18774445112225004771188aaaa22332 Papprrr.
Yingzhen Li, Jose Miguel Hornández-Lobato, and Richard E. Turner, Stochastic expectation propagation. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2. N1PS'15, pp. 2323 2331, Cambridge, MA, USA, 2015. MIT Press.
15

---

Published in Transactions on Machine Learning Research (09/2024)
Jeremiah Z. Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss: and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Tn Advances in Neural Information Processing Systems 33 NeurlPS, 2020.
Wesley J. Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A Simple Baseline for Bayesion Uncertainty in Deep Learning. Curran Associates Inc., Red Hook, NY, USA, 2019.
Andrey Malinin and Mark Gales. Prodictive uncertainty estimation via prior networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL htpp::tt////proeedinn .ddf
neurips.. htt.ppp.ccpppprrr.p
Tom Minka. Inferring Gaussian distribution, 2001.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian bimning. In Proceedings of the Twenty-Ninde AAAI Conference Artificial Intelligence, AAAI15, pp. 2901-2907. AAAI Press, 2015. ISBN 0262511290.
Illia Olekstienko, Dat Thanh Tran, and Alexandros Iosifidis. Variational neural networks, 2022. URL
https://arxiv. (trrg/abs/2200 015244
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can You Trust Your Model's Uncertointy? Evaluating Predictive Uncertainly under Dalaset Shift. Curran Associates Inc., Red Hook, NY, USA, 2019.
Kaare Brandt Petersen and Michael Syskind Pedersen. The matrix cookbook. 2012. URL http://wwy /www math. www.erlool..oooom. ht/////wlkooo//aattrrroooooooooo pdf
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classi- fication uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cosa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: https::/pprceed::: wwwesenerripp.. seuripp.cccpppprr a981 4391220004440ff44711166663000-00 PaperrpopPpppppppp
Shai Shalev-Shwarlz and Shai Ben-David. Understanding Machine Learning From Theory to Algoritimss. Cambridge University Press, 2014.
Mrinank Sharma, Sebastian Farquhar, Eric Nalisnick, and Tom Rainforth. Do bayesian neural networks need to be fully stochastic? In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent (eds.), Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pp. 7694-7722. PMLR, 25-27 Apr 2023. URL Sates//200//haaraa2222 html
mlr.
Rishit Sheth and Roni Khardon. Excess risk bounds for the bayes risk using variational inference in latent gaussian models. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL http:::/ppoooedigg.norri.....o http:////.pppp..eeee////////////
pdf
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. FUNCTIONAL VARIATIONAL BAYESIAN NEURAL NETWORKS. In International Conference on Learning Representations, 2019.
URL http:://pppaaeeeeeeeeeeeeeoonmms
Yee Teh, Leonard Hasenclever, Thibaut Licnart, Scbastian Vollmer, Stefan Webb, Balaji Lakshminarayanan, and Charles Blundell. Distributed bayesian learning with stochastic natural gradient expectation propa- gation and the posterior server. Journal of Machine Learning Research, 18, 12 2015.
16

---

Published in Transactions on Machine Learning Research (09/2024)
Marcin Tomezak, Siddharth Swaroop, and Richard Turner. Efficient low rank gaussian varia- tional inference for neural networks. In H. Larochelle, M. Ranzato, R. Hadselll M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 4610 4622. Curran Associates, Inc., 2020. URL httrps:/pprocednnss.e httpr/pps.c/paaar///////aeaa pdf.
Marcin B. Tomczak, Siddharth Swaroop, Andrew Y. K. Foong, and Richard E Turner. Collapsed variational bounds for bayesian neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan
(eds.), Advances in Newral Information Processing Systems, 2021. URLhttts::////// httpss://pparreeieeeeete/eeeeeee
id=yxh3tbJ0qmX
Ba-Hien Tran, Simone Rossi, Dimitrios Milios, and Maurizio Filippone. All You Need is Good Functional Prior for Bayesian Deep Learning. Journal of Machine Learning Research, 23:1-56, 2022.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using a single deep deterministic neural network. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, volume 119 of Proceedings of Machine Learning Research, pp. 9690 9700, 2020.
Yadi Wei and Roni Khardon. On the performance of direct loss minimization for bayesian neural networks. In I Can't Believe It's Not Better Workshop: Understonding Deep Learning Through Empirical Falsification, 2022. URL https://openrevier www./eeetoounnee -itermmmmddd_111ghrr 12g1tghrgttG
Yadi Wei, Rishit Sheth, and Roni Khardon. Direct loss minimization for sparse gaussian processes. In Arindam Banerjee and Kenji Fukumizu (cds.), Proceedings of The 24th International Conference on Arti- ficial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 2566-2574. PMLR, 13-15 Apr 2021. URL. https: httppss//prrccednnng.... mml.iiinnnsmm.......mmmm...mm. mlr. wtw.eeedddngg......ess/////1000 arress/v13/wwei2// hemm.b.mmml
Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Swigtkowski, Linh Tran, Stephan Mandt, Jasper Snock, Tim Salimans, Rodolphe Jenatton, and Scbastian Nowozin. How good the bayes posterior deep neural networks really? In Proceedings of the 37th International Conference on Machine Learning, ICML '00 JMLR.org, 2020.
Andrew G Wilson and Pavel Tamailov. Bayesian deep learning and a probabilistic perspec- tive of generalization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (ods.). Advances in Neural Information Processing Systems, volume 33, pp. 4697-4708. Curran Associates, Inc., 2020. URL https::/proceeddgg..eeee pdf
cc.paper filees/paper/2020//
Andrew Gordon Wilson, Pavel Tzmailov, Matthew D) Hoffman, Yarin Gal, Yingzhen Li, Melanie Pradier, Sharad Vikram, Andrew Foong, Sanae Lotfi, and Sebastian Farquhar. Evaluating approximate inference in bayesian deep learning. In Douwe Kiela, Marco Ciccone, and Barbara Capulo (eds.), Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track, volume 176 of Proceedings of Machine Learning Research, pp. 13-1124 PMLR, 06-14 Dec 2022. URL httph::tt///ste htttps:/pproceedings. mll. mlr press/v176/ m2lson22a.. html
Anqi Wu, Scbastian Nowozin, Edward Moeds, Richard E. Turner, Jose Miguel Hernandcz-Lobato, and Alexander L. Gaunt. Deterministic variational inference for robust bayesian neural networks. In In- ternational Conference on Learning Representations, 2019. URL http://pparrrieeeeeefefoooreeoee
B1108oAct7]
A Proofs
A.1 Proofs in Section 3
Proof of Theorem 3.1 Assume N d. Note that with the the correlated prior and posterior the covariance
function is rank deficient so have to interpret inverses and determinants appropriately. Here we use
17

---

Published in Transactions on Machine Learning Research (09/2024)
pseudo inverse and pscudo determinant. The VIFO objective is:
KL(((s(X))ll(((Xxx)))
(11)
1 Σ (ottla))logpuull))
N 2
==1
1 2 log (X
VX)) 1 [(( XN Tx m Xxx Xx Xx- my XN) (12)
1
First consider the loss term. Let L be the Cholesky decomposition of V, i.e. V LLT. By reparametrization, for e~ N~0, Ia), w I, + x. Le ~ Nww I,, x. LL" x4) and thus
R (otgl (0ogg(()la)) |=]] = 1E.-N w 5a + x. Le)]
= -EE-wro..jllogpuulc + Le) x;)l
-Eo-NN,,LLT log 0 x;)).
(13)
where the last equality uses reparametrization in reverse order. By aligning
and V = LL = S, we
recognize that Eq equation 13 is exactly the loss term in Eq equation Thus the low-dimensional posterior on 2 yields the same loss term as the high-dimensional posterior over W.
For the regularization, we use the pseudo inverse derivation from Eq (224) of Petersen & Pedersen 2012 where for CD we have A+ (DD to get
and the same for V. Thus,
(X = (
-X
ur (XX VXN: = -xIXXNX((xxX3)
=tr(s,'V),
and
(w XN m XN XX) -1 XN)
= (w (x Xx) X mo)
=(w (w +
mp)
=(w mo) TS, (w mp)-
For the The log-determinant term use the pseudb-deterrminnt Miinka 2001 non-zero eigenvalues. Let (Ai, u)) be the set of eigenvalues and eigenvectors of S V, ie,, So "VVV Awa, and let x (Xxx))=) denote the pseudo inverse of XN, then
which is the product of
S. Vu, 'v XX
(14)
thus (Ae, Xi is the eigenvalues and eigenvectors of (Xx VVN. XN. Since the rank of this matrix is at most d, other eigenvalues are 0 and the pseudo determinant is I d, which is exactly the determinant of S-1 Then the regularization term in equation 9 can be simplified to:
- KKL((2\XX))loda/x))) - - -
log \ | {(w (w ma) So (w mo)
N
(15)
2
By aligning m, V S, we can seet that equation 15 exactly the regularizer in equation ignoring
the constant.
18

---

Published in Transactions on Machine Learning Research (09/2024)
Note for the case K 1: Let 0 Ra*k For VI, we make a mean field assumption with q(Ox)
((00/mme, S1) and (0)) = I ((0x), where 02 the k-th column of 0. For VIFO, using mean field let g(za/z) = (z x, Vax) and ((z(zr) = [ (Zsl)z). By aligning = and V = Sk, we can find (otle))loogg (1,....k] = 10) (log p(ui Plog Ii. 0k2,)),, and
(L(G(0)(PP(O) = Σ (L(g(Ok), ((a)) = Σ
(16)
where the second = means equivalence ignoring a constant difference.
Consider a neural network with one single hidden layer, denote the weights of the first layer u, and the weights of the second layer Thus, the k-th output can be computed as:
Proof of Theorem 3.2
Σ (ulcoss)
(A)
Σ Wket
where the size of the hidden layer, D the input size and vi(a) = max(0, a) the ReLU activation function. We further simplify the setting by considering the special case where only xı is non-zero and Then the k-th output becomes:
(k) = WK, (2,1,
Consider a distribution g(wk,;) = NW d.), g(u,,a) = N (itiddd o.).
Then if x1 2 0,
E Be(w))(u)
[2(0]
Eu 1 (u1,,
#1.1
T U1,1 +
ou
Ou
(1 (1-
U1.1
r1;
(17)
1 +
01.1
dv
if x1 < 0, then
[2(0]]
E Eg(onyotu)
Ero
M1.1
Wh.1
11,
du
Oa
+
U1.1 du
(18)
+b
11.1
dn
where Φ and & are the pdl and edf of standard normal distribution and we directly use the expectation of the truncated normal distribution. Now consider i and ū that aim to recover (17 and (18). If ū1,1 2 0,
it cannot successfully recover (18) bocause the ReLU activation will have 0 when ø < 0 so that it cannot
recover 18 W1,1 < <0, for the same reason cannot recover 17
A.2 Proofs Section 4
Verifying Assumption 4.1
We next verify that Assumption 4.1 holds for classification and (with a
modified loss) for regression.
For K-classifieation, is K-dimensional and the negative log-likelihood
-log k//)) log exp(zx) - 26 + log exp(2:)
K
Ei exp(z:)
which is 1-Lipschitz in z. For regression, 2 (m, !) is 2-dimensional, and the negative log- likelihood is:
1
-logp(y/>) _/2 ( m) (y-mm)expp(11)
-x)
-
2
19

---

Published in Transactions on Machine Learning Research (09/2024)
Neither the quadratic function nor exponential function is Lipschitz. But replace the unbounded quadratic function (y-m)² with bounded version min ((n((()))) B and replace the exponential function exp(-() with min(exp(-(), B where B > 0, to guarantee the Lipschitzness. Now the negative log-likelihood is:
1
- -logp(y1z) = min (vv m)); B min(exp(-1), Bi} + 2¹
is Lippssit in m, (5 + B -Lipschhtt in L.
Verifying Assumption 4.2) For Assumption 4.2 we can use g(l) = log(1 + exp(l)) which 1-Lipschitz.
If (I) = exp(I) the exponential function, we can use a bounded variant that satisfies the requirement
g(l) max (exp(x))) Bg}.
Proof of Lemma 4- We prove the lemma for L= 1. If this is not the case, we can define ø' = L and use the fact that R(o ((x(AxB)) S LI(d'(Ax Let C, = {(a1 +br,... 02-1 +b-- of (a, b,), a++1 +b:41-. aN+ by)= A,b E BJ. It suffices to prove that for any set A, B and all i we have R(C;) S R(A) + R(B). Without loss of generality we prove the case for i = 1.
(R(C1) Es
sup oio(a),b)) + oaslar + bi)
rEc1
N o(a1 + Lo,(a, + bi)
Eg 2
NN...ON
sup aEA,BEB
1=2
N Lold', b1) + Σ orla + +b/) bb))
sup d'EA
2=2
N
Boa.....N
sup A.E
b1a b1(d
Σ
oj + +b;) +
Σ
o, b))
i=2
i=2
N
$
E.g....N
E
sup A,
|a| d'l + [b1 - 8/y + Σ
0, + +b,) + Σ
,, +b)
i=2
1=2
N
N
=
E
sup a.a'EA
01 - df
+
0:4
+ Σ a.d!
2
1=2
1=2
E Eoz,...o NN
sup DB
b1 - bj +
Σ
o,b
+
Σ o,bl
i=2
i=2
+ NR(B)
Proof of Theorem 4.5 We show that the loss is Lipschitz in fw, (x) and fw, (x). Fix any x, and W, W We denote the mean and standard deviation of (=(o)) by a and s and the same for qw (2/ar). We use for Hadamard product.
E (on p())2) E (en logpull))]
= (o..roonllgglllll + logp(vlu + +e e. s)]
<E <E.NroOn) [Lull (1- - 11) + - (s-s')
(Lipschitz)
STollu 1'112 + Bo V ||e (s s')
STollu d/1l + To Bellle (s 13)
(Jensen's 's Ineq)
=To(ll n'lla ||s -s'll2)
<Lolllu- - n1ll -
+ ||s s'll1).
20

---

Published in Transactions on Machine Learning Research (09/2024)
For the 6th line note that (s
(s =EIΣ,,
E,
(s
-)
||s - s'112- The loss function is Lipschitz in 14, which is exactly fw; (2). Further, is T--Iipschitz in the logit fw2 (I), thus, the loss function is (Lo max{1, [1, L1 L)-isscchizz in the concatenation of fw, (x) and fw, (2), each of which is of dimension K. The theorem now follows from Corollary 4.4
B Derivations of Collapsed Variational Inference
As is shown by TTomezak et al. (2021, for priors and approximate posteriors from the exponential family, we can derive the closed-form solution for the optimal q (1pp of)),
log4' (Pp, 22 x logglup: of)) + (e ob)],
(19)
for optimizing (hpp ob) for every single data. Our derivations follow the methodology of Tomczak et al. h021h h but they are applied on the output z instead of the weights W.
B.1 Learn mean, fix variance
Let p(z11pp) = Nx2lupp p(1pp) = (44p O, a.). Recall that ((2|x)) = N (t) Then
loga" (1ppz) x p(1p) + + ( (og
1
Up un
[Mp Mg(x)) (rp)))((p>>)))) M²(()) + [x)(
2a
27
α a + 7
a Hp a4n))(())
(ND a (1p (1p-at -t
207
(2), a1,)) 1). Pluging q" into the regularizer, the new regularizer becomes Then q² (1p) = N NN (244)
1 [=) + -
a Mp(x)) [ - 11" 1 [ + 2
K log(y + a)
K 2
27
2
B.2 Learn both mean and variance
Let p(z|up; of) (= of); p(uplo))= (0, = [ B), where IG indicates the inverse Gamma distribution. Let ((up) be diagonal Gaussian and g(0,)) be inverse Gamma. Use Mp. and dp. to denote the i-th entry of up and op respectively, then
log (Mp,s op.))
x looo loggp(mpp. 2..)) + Bg(((a)) [log p(2/1pp af)]
2/3 + 1/2
1
(Mpii (gg()))²
1 o (x)
x x- (a+ log of.
log 02
202 p.i
2 of
p.
1
t
Ma)( 2)
x - (a+ log 2.
B +
2(t Mg (x) (1) + 2
02)((1))
+ (1 + 1)
Mppt
20
t t+
follows the normall-nverse-gammmi distríbution. Thus q' (1mpp)) (( (x)), of and 9º (0flxx) Ia + 1),,3 + 2(441) (g(x)² + (z)())
Then the regularizer becomes
1 1T log (a (a+
B1 log
2(1 + t) Mg (x) + [0)(())
- T Toog logo,
(20)
2
-2-
21

---

Published in Transactions on Machine Learning Research (09/2024)
B.3 Empirical Bayes
Let p(oz)) Ig(o))a, p() - o2=),,
and let g(0)) be delta distribution. Then
[L(g(2)z) log
1 (z) K log 1 log oz)() K
(g(x)
T Mg(a)) + (a 1) of
B o
-
o3
of
By taking the derivatives of the above equation with respect to of and solving, we obtain the optimal pa(x) p,(x)+1 o (w)+42 we plug this back into the KL term, we get the regularizer:
p
K+20+2
K log (x( T (x) + +1 (z)) + 23
log (x)(
K + 2a + 2
K
1 (K + 2a + 2) (x) (g(2) + +z)o((x))
(21)
2 2
(x( My (2) (2(x)) 23
(a + 1)log Me(x)"n,, (,(x++11 (
However, if we include the negative log-prior
term
K42042
8 K+20+2
P²(x) ) adding them up will have
5 K 20 log (x(x)) M2(x) + 1 +T)) + 1 log of log + 2xx++ +x
(x) const,
which highly reduces the complexity of the regularizer. This performs less well in practice and therefore we follow Wu et al. (2019) and use equation 21
c Optimizing the Variational Distribution for All Data
In the previous section we show the derivation of collapsed variational inference where q" (pp. of) is optimized for every data point x. In this section we show how optimize q(pp: for all data and obtain different regularizers the ones mentioned in the above section. These perform less well in practice but we include them here for completeness. The closed-form solution for q" (up, of) for all data is
log logg" of)
Σ (x.y)E
logp(up, of) -
of]1
(22)
C.1 Learn mean, fix variance, optimize for all data
Let p(2)/pp) (2 7), p(up) ((up 0, o.). Given dataset D ((x, v)). get optimal q" (ppp
for all data. According equation 22
log q" (kp)
1
Σ
logp(up) (g(x))) (lgpp)
N
(x.))D
Σ
(lUp m,(xx)) (1tp - mg(x)) + 1T TTo2(
x
2a 1'
2
(x.
a a+7
Σ Mg(at)
M--NEO (R.))D
Σ maler)
11p
Mp
20y
N
N
(x.0)ED
22

---

Published in Transactions on Machine Learning Research (09/2024)
Then the optimal q- (1pp Mata 22 Ex My(x), 42, I). Let Byy 2/2 Mg(x). The regularizer now is:
Bolu.) (LL(((aa)))lll(ll//:: M1 KL 11(11)))
(r.9)
Σ
Be(Hv) K Klog7-11 1 log (ogg-11'ogo((a)) K + 11 T xz(2x)
(1n( (pp)
1 Up) (1)(x)))))))))))))))))))) -noi] mo)]
7
7
(a.0)
N
a a+7 -
a²
Klog Klog
K + K 7+a
+
(a 7)2 Fly Mq
2
7
Σ
1
(1 T og)()) + (x) (x(x))
1 1 log of (xx)
N 1 1
-
Bg Mg
NK
log(a + 7)
NK
2
2 7
a a+7/
2
2
(r,9)
(23)
We refer to this method as "ViFO-mean all".
C.2 Learn both mean and variance, optimize mean for single data, and variance for all data
Let p(z|pp: )f)) = (222ppp 07), p(uplo7) = =N(pp0 )0))), pl p(oz))
Ig( 03 la, 9). Consider that
logglum, ) + [g(+ (/ (otts of,i)
(24)
1ogg//pp. 10
2 n + logrlo (o,))) B2 (olgg(p((lppppt o sp.i)
(25)
t 1
B
x
2 2
2
log o0 - (a + log "i.
0
2
log on. 20 (1pps -mg(l))² + (x))
(26)
log of.i - (a + 1) log l
(1pps - pg)(x))² + (x),
(27)
2 2
20
++1
(x) - (a + 2) log op.
B
02 (2)
20
Up.
1+1 1 Max(o))
2(1 + 2)1 p.3
p.P
20 p.
(28)
Then by extracting the up part from equation 28
we have
++1 1
++ 1 Maxl())
2
logq 4" (Mp.ilo p.ir x) x 20
Upii
and thus q" (upli, of,) = -(ppl (x)),, +41 0z)) Then we try to marginalize out Up to compute (o))
log / /exp (logp(up,s) (2.) + (olllop((lpppp. ol.i dpp
logg"(o x N
Σ
(x.W)ED
1++
t 1 Mass(c)) dppps
1
x N
log
cxp
Upp
20 p.i
X,EED
tp² 9. (x)
3
(x)(2))
(a + 2) log 2
2(t 1)07,
20 p.
=-(a +
B
+
Σ
1
log
2TO² p.i
2 (I)
tp q. (x)
2
N
2
++1
20
2(t + 1)o p.
(x.))ED
3 + 2(++1)
N Σ En (x)
+
1
Σ (o)
= = + 5
logo p.
2N
0 2
23

---

Published in Transactions on Machine Learning Research (09/2024)
and (o) = 1 + 1)1, + 2(++1) N Ex Mg()) + 2N Ex o((x)) Let jy = Ex My(x)) and
En
0g V N E, og(x)), then the regularizer becomes:
(a + 1 2 T log 31
Σ
1
log o (x)
(29)
2(1 + t)
2
(a.9)
KNllo T(a)
NK
t+1
NK
log
(30)
T(a +
We refer to this method as "VIFO-mv all".
C.3 Empirical Bayes for all data
If we optimize of for all data, then we have
Σ
KL(((=)=))l((=o0)) - logp(o {
(2.))ED
[o)
px(x)] (1(x)) + (a + 1) oz +
= Σ (
K log oz - 1 1ooo [x() K K+
o3
of
o
R. ( K+20+2
where By / My and og = I. The
and the optimal variance being objective is:
NK
23 1
82
1 1 log logo,(x)² NK
log
Σ
2
K + 2a
2
2
1
K 2a +2
(1)((x) (0g()))((()))))))) + (x)))
228 + P4
2
0 x
This method is called "VIFO-eb_re all
D Experimental Details
D.1 Experiments Artificial Dataset
To generate Fig. and Fig. 2 we generate 100 training data points y 2in + 0.1e, NoO, 1), where Ptrain [- n U T π and Ptest E [[TT, s|. We use multilayer perceptron neural network with 5 layers, each layer containing 50 hidden units to fit the data. For VI, we pick the prior standard deviation 2 = 0.05. For both models, we select the regularization
3
to be 1.0 and for VIFO-mean, we select 7 = 0.3, atty parameter 7 = 0.1 and for VIFO we choose 7aux 1.0. For linear regression, we can explicitly compute the predicted variance if we use exponential function as the link function. Suppose p(ylz) NVy m, exp(()) and p(m x) (m pm, of )(p x) .V(1 (f) then ply x) Niy Mm, a + exp(m + /2)). See Appendix B.3 of Wu et al. (2019) for the derivation. To visualize the predictive distribution inducted by the prior, we draw multiple ss from prior, then draw multiple y's from likelihood p(y|z) and plot them in Fig.
D.2 Experiments on Large Image Datasets
In this section we elaborate the experimental details, including the choice of hyperparamelers, learning rates and the number of training epochs.
Number of training epochs: We train all methods in 500 epochs.
24

---

Published in Transactions on Machine Learning Research (09/2024)
Learning rate: For all methods other than SGD, SWA and SWAG, we use the Adam optimizer with learning rate 0.001.
We first list the choices of the variance for naive variational methods, The choice of prior variance significantly affects the performance. For image datasets with complex neural networks, the total prior variance of VI grows with the mumber of parameters 0.05 following the setting of Wilson et. al. (2022). Since VTFO samples in the output space which is small, using 0.05 regularizes too strongly and we therefore set larger value of 1 for the variance.
VI and VIFO:
have to pick a small variance and use
For collapsed variational inference, we pick 7 = 0.3, Greg = 2 = 0.05 for learn-mean regularizer (VI-mean,
VIFO-mcan, VIFO-mcan-all) and 0.5, 0.01,
0.1 for learn-mean-varian variance regularizer (VI- my, VIFO-nw,, VIFO-mv all), which exactly follows Tomezak et al. (2021) We pick 4.4798 and , = 10 for empirical Bayes (VI-eb, VIFO-eb, VIFO-eb _all). The choice of in empirical Bayes follows Wu et al.
(2019) but the choice of 3 is unclear in Wu et al. (2019) so we just perform a simple search from {1, 10, 100] and set 3 = 10 that yields the best result.
For both VT and VTFO, the regularization parameter v fixed 0.1.
The hybrid methods (SGD, SWA and SWAG) are not very stable so we have to tune learning rates carcfully for each dataset. We choose the momentum to be 0.9 for all cases and list all other information in Table D.1 Notice that it is hard to train the hybrid methods on SVHN using AlexNet, so we initialize with a pre-trained model that is trained with larger learning rate 0.1 to find a region with lower training loss, and then continue to optimize with the parameters listed in Table D.1
Hybrid Methods:
Dropout: For Dropout we add a Dropout layer following each activation layer in the base model and set the Dropout probability = 0.1.
Repulsive Ensembles: Repulsive ensembles multiple copies of the base model with kernel base penalty to make sure the models are diverse. We use RBF kernel with lengthscale being the median of the square of the norm.
Dirichlet-based models are deterministic and they interpret the output of the last layer as the parameters of dirichlet distributions, i.e., a(x) e(fw(x)), where maps the output to positive real numbers. Hence we run the Dirichlet models with the setting of the base model. We next explain the setting of hyperparameters. As discussed by Bengs et al. (2022), the models of Sensoy et al. (2018): Charpentier et al. 2020) implicitly perform variational inference:
Dirichlet:
p Dir(as), v/p Cat(p)
(31)
and the ELBO becomes
logp(y/ax) 2 Egip #) [lllogogppp) KL(((pl=)[Dirrp (Do)))
(32)
where q(plz) Dir(p/a(x)). In the experiments, following Sensoy et al. (2018 Bengs al. (022)) we
use uniform prior with ag [l...., 1]. As in VI and VIFO, pick the regularization parameter for KL divergence to be 0.1.
Last layer Laplace: We first train a neural network to obtain MAP solution with prior variance of 0.05. Then, we use the code from Daxberger et al. (2021) to optimize the prior precision hyperparameter through post-hoe marginal likelihood maximization.
VBLL: We adapt the code from Harrison et (2024) and utilize the default hyperparameters. We choose the discriminative classification setting, as yields the best OOD performance according to Harrison et al. 2024h.
25

---

Published in Transactions on Machine Learning Research (09/2024)
Table D.1: The parameters for running the hybrid algorithms. "lrr-narning rate, "wdh-weight decay,
"swag_ir the learning rate after we start collecting models SWA and SWAG algorithms, "swag start"- the epochs when we start to collect models, "epochs"- number of training epochs.
wd
swag
swag start
epochs
CIFARIO CIFARI00
0.05
0.0001
0.01
161
500
SVHN'
0.001
0.0001
0.005
161
500
STL10
0.05
0.001
0.01
161
500
E
Additional Plots
E.1 Accuracy PreResNet20
0.90
0.00
0.87
0.50
0.86
0.6
0.60
WFO-4400
180.m WK0.wv
Misane
SWAk
SHAG
Remoshive Dir
Drssiut
Lapiace
WFo-masn
WFO me
Vnnane
503
Siilo
Despuut
Lapiaaee
VELL
(a) CIFARIO
(b) CIFAR100
80X
DUX
0.70
0.5
0.0
0.65
000
0.5
0.2
0.4
WrO.mean WPO
Minaice
Rosucine DF
Dreauett
Lapiace
HELL
WP0 WDD.menn
IPO
Vanave
Rusutiire
Drspout
Lmplace
VELL
(c) STL1O
(d) SVHN
Figure E.1: Test accuracy (1) PreResNNe220. Dashed lines indicate the best version of VIFO.
26

---

Published in Transactions on Machine Learning Research (09/2024)
E.2 Results AlexNet
0.8
BIX
325
BUX
3.00
0.7
25
0.6
230
2.5
2.00
175
130
1070mmen
180
SWatt
Repuitivo
Drapost
Laptace
VeLL
WHO-mesn
WFO me
Vanaie
3008
SWAG
Ronstove
Drsapist
Lastiaee
VELL
(a) CIEARIO
(b) CIFAR100
20X
0.7
bux
2.4-
22.
16
14-
03
02-
10.
WFO W.O-mmon
1850 on
Monsove
sp0
Domuision
Drapout
Lostace
HELL
Wit0-maa
innsvve
stD
SHOOD
Repuideve
Drapout
Loptace
VELL
INO
(c) STL10
(d) SVHN
Figure E.2: Test log loss (4) of image datasets AlexNet. Dashed lines indicate the best version of VIFO. The error bar is three times of the standard deviation for better visualization and same for other figures.
27

---

Published in Transactions on Machine Learning Research (09/2024)
0.92
0.66
0.21
0.6-
0.0
0.88
VHO maas
WFO me
Wineiie
sco
SWi
SHAG
Hepistrre
Dir
Dreptos
Lapiaaee
VOLL
Wro-masn
Moaine
Swi
SWáto
Rkoddure Dir
Drappss
Laplane
HOLL
(a) CIFARIO
(b) CIFARIO0
0.922
0.60
0.9755
0.000
0.62
0.000
100 mason
WrO.m oo
Monaie
850
5Wi
SWiG
Dy
Dispout
Lapilace
HELL
09<30
WF mo
550
SWat
Dr
Dhnnoot
Lapitca
1984
(c) STL10
(d) SVHN
Figure E.3: Test accuracy (f) on AlexNet. Dashed lines indicate the best version of VIFO.
DUX
225
1.0
2.00
1.7
150
12
0.-
0.0
Vtro-maan
W00-mr
SWAG
Iepuincee Of
Drapsu
Lapiace
WFo-masn
WFO
Vnnane
533
SHilo
Hemostte
Despuut
Lopilaee
ViLL
(a) CIFAR10 SVHHN
(b) STL10
2.0
2.0
cremm 2WAN5T-10 1110
3.6.
16
1.4
WiO-meen
Minaiie
sweae
Repulsive Oii
Dhapost
Lapiace
HELL
VirO-mean
Wo
Swatt
Repusive
Croprist
Lapioca
VELL
(c) SIHR-O
(d) SVHN-STLIO
Figure E.4: Entropy (f) on AlexNet.
28

---

Published in Transactions on Machine Learning Research (09/2024)
E.3 AUROC Comparisons
3.0
0.00
9.5
030
042
010-
WD Wro.ssan
2004
1950
Bepdteso
Drspoct
Lapiice
19LL
Jist
Rupower
Dropoiot
(a) CIFARIO SVHNN
(b) STLI0-SVII
1.00
0.70
0.015
3.60
0.005
3450
0150
Reprrtove Of
Drspriot
toplace
1081
WP WDD-mo
Winore
Jala
slnd
Drspeed
Lastace
1881.
(c) STFARI
(d) SVHN-STTL10
Figure E.5: AUROC (f) on AlexNet.
0.5
0.66
F
0.4-
4
0.8
046-
0.55
264
062
400
Vieana
sc0
or
Despoid Lopiace
Wia W00- me
Mouce
5c0
osd
seau
Remacria Dr
Lociane
(a) CIFARIO-SVHN
(b) STLIO-SVHN
0.55
0.55
0.0
0.55
0.5
030
Liame
503
ald
Dessout
Louitie
Med monn
Masare
ald
Impussiit Or
Lositie
(c) SVHN-CIFARI
(d) SVHN-STLIO
Figure E.6: AUROC (1) on PreResNet20.
29

---

Published in Transactions on Machine Learning Research (09/2024)
E.4 Learning Curves
at evon
46 m0
10
4.0
3.3
1.0
3.0
2.5
06
2.-
15-
200
X00
300
302
(a) CIFARIO
(b) CIFAR100
10-
1.-
1.0
(20C7)
sp00ts
(c) STL10
(d) SVHN
Figure E.7: Learning curves for all datasets on AlexNet. We conducted 5 independent runs and report the mean and standard deviation (which very small). The results show that all cases, VIFO-mean converges fast as, or faster than, VI.
30

---

Published in Transactions on Machine Learning Research (09/2024)
E.5 Ensembles
67
2.75
2.50
3.20
2400
041
1.5-
150
Dreemite s/c
Eraendde Sit
(a) CIEARLO
(b) CIFAR100
0.00
1.4
127
10
0.5
(c) STL1O
(d)
SVHN
Figure E.8: Test losses size of ensemble, Results are shown for 5 independent runs on AlexNet.
31

---

Published in Transactions on Machine Learning Research (09/2024)
150
077
000
633
0.1
Craurrite soe
Craumbiie
(a) CIFAR10
(b) CIFAR100
2.20
DAVS
3.50
1.-
1.2
2.00
1
(c) STL10
(d) SVHN
Figure E.9: Test losses vs. size of ensemble. Results are shown for independent runs on ProResNet20.
32

---

Published in Transactions on Machine Learning Research (09/2024)
F
Numerical Results
Table F.1: CIFARIO, AlexNet, NLL
method
Maux 0
Taax 0.1
Taax 0.5
Maaxx 1.0
VIFO-naive
0.388 = 0.005
0.383 + 0.002
0.382 + 0.001
0.383 + 0.002
VIFO-mean
0.338 = 0.002
0.343 + 0.002
0.344 + 0.003
0.345 + 0.004
VIFO-mv
0.315 = 0.002
0.324 0.000
+ 0
0.311 0.002
VIFO-eb
0.347 = 0.003
0.345 0.001
0.345 + 0.003
0.347 0.002
VI-naive
0.329 = 0.006
VI-mean
0.350 = 0.010
VI-mv
0.315 = 0.013
VI-eb
0.343 0.004
SGD
0.375 0.004
SWA
0.363 = 0.006
SWAG
0.329 = 0.002
Repulsive
0.785 = 0.003
Dir
0.788 = 0.002
Dropout
0.413 = 0.013
Laplace
0.332 = 0.009
VBLL
0.373 = 0.003
Table F.2: CIFAR100, AlexNet, NLL
method
7aux = 0
Maux 0.1
7aux 0.5
Maux 1.0
VivO-naive
1.774 = 0.008
1.840 + 0.016
1.784 + 0.008
1.811 - 0.012
VIFO-mean
1.632 = 0.003
1.687 + 0.002
1.682 + 0.017
1.683 + 0.021
VIFO-mv
1.642 = 0.011
1.716 + 0.012
1.971 + 0.053
2.250 + 0.105
VIFO-eb
1.643 0.008
0.003 1.651
1.643 + 0.004
1.649 + 0.006
VI-naive
1.513 = 0.024
VI-mean
1.642 = 0.023
VI-mI
1.817 = 0.022
V1-eb
1.441 = 0.016
SGD
1.894 0.024
SWA
1.768 0.026
SWAG
1.768 = 0.022
Repulsive
2.540 0.016
Dir
3.218 = 0.008
Dropout
2.024 = 0.022
Laplace
1.738 0.006
VBLL
1.649 = 0.003
33

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.3: STL10, AlexNet, NLL
method
7aax 0
Taux 0.1
7aax = 0.5
Taux 1.0
ViFO-naive
1.113 = 0.006
1.117 0.005
1.112 + 0.005
1.135
0.008
VIFO-mean
1.030 = 0.004
1.056 0.005
1.066 + 0.005
1.067
0.011
VIFO-mv
1.078 = 0.005
1.121
0.002
1.112 + 0.010
1.101
0.008
VIFO-eb
1.141 = 0.007
1.134 0.008
1.127 + 0.007
1.135 0.002
VI-naive
0.975 = 0.010
VI-mean
1.021 0.013
VI-mv
1.095 0.018
VI-eb
1.560 = 0.054
SGD
1,419 = 0.025
SWA
1.139 = 0.011
SWAG
1.098 = 0.005
Repulsive
1.388 = 0.008
Dir
1.359 = 0.003
Dropout
2.359 = 0.045
Laplace
1.157 = 0.007
VBLL
1.102 = 0.008
Table F.4: SVHN, AlexNet, NLL
method
Maux 0
7aux 0.1
7aux = 0.5
7axx 1.0
VIFO-naive
0.253 = 0.001
0.256 + 0.001
0.257 + 0.001
0.251 + 0.002
VIFO-mean
0.211 = 0.002
0.214 0.002
0.209 + 0.001
0.208 0.001
VIFO-mv
0.165 = 0.002
0.169
0.001
0.166 + 0.001
0.170 = 0.002
VTFO-eb
0.211 = 0.001
0.214 0.002
0.212 + 0.002
0.214 0.001
VI-naive
0.175 = 0.002
VI-mean
0.219 = 0.034
VI-mv
0.173 = 0.003
VI-eb
0.182 = 0.004
SGD
0.351 = 0.002
SWA
0.251 = 0.012
SWAG
0.238 = 0.001
Repulsive
0.686 = 0.007
Dir
0.692 = 0.003
Dropout
0.211 = 0.009
Laplace
0.244 0.008
VBLL
0.177 = 0.001
34

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.5: CIFARIO, PreResNet20, NLL
method
Maux 0
Maux 0.1
7aux 0.5
Maux
ViFO-naive
0.473
0.005
0.487 0.011
0.486 + 0.010
0.479
0.008
VIFO-mean
0.393 = 0.010
0.433 0.018
0.411 + 0.013
0.430 0.020
VIFO-mv
0.361 = 0.010
0.371 + 0.010
0.367 0.006
0.363 0.016
VIFO-eb
0.419 = 0.014
0.429 + 0.016
0.413
0.425 0.008
Vl-naive
0.410 0.028
VI-mean
0.415 = 0.032
VI-mv
0.437 = 0.029
VI-eb
0.429 = 0.035
SGD
0.335 = 0.013
SWA
0.336 = 0.008
SWAG
0.307 = 0.010
Repulsive
0.875 = 0.007
Dir
0.961 = 0.022
Dropout
0.423 0.026
Laplace
0.324 0.009
VBLL
0.366 = 0.006
Table F.6: CIFAR100, PreResNet20, NLL
method
Maux 0
Manx 0.1
7aux 0.5
Maux 1.0
VIFO-naive
1.880 = 0.009
1.935 0.020
1.864 + 0.008
1.844
0.011
VIFO-mean
1.632 = 0.013
1.728 0.005
1.686 + 0.004
1.731 + 0.018
VIFO-mv
2.726 0.021
2.826 + 0.008
2.899 = 0.019
2.867 =
VTFO-eb
2.076 = 0.006
2.147 + 0.004
2.340 + 0.043
2.503 0.034
VI-naive
1.642 = 0.030
VI-mean
1.753 = 0.086
VI-mv
1.804 = 0.089
VI-eb
1.731 = 0.097
SGD
1.445 = 0.021
SWA
1.355 0.023
SWAG
1.354 0.019
Repulsive
2.948 0.020
Dir
3.580 = 0.009
Dropout
1.644 0.045
Laplace
1.550 0.014
VBLL
1.484 0.017
35

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.7: STL.0, PreResNet20, NLL
method
Maux 0
7aux 0.1
7aux 0.5
Maux
ViFO-naive
1.146 = 0.012
1.159 0.008
1.155 + 0.015
1.145
0.005
VIFO-mean
1.067 = 0.009
1.066
0.011
1.056 0.007
1.069 0.009
VIFO-mv
1.070 0.017
1.078 0.003
1.067 =
1.073 0.011
VIFO-eb
1.162 = 0.015
1.164 + 0.020
+.191
1.180
Vl-naive
0.920 0.032
VI-mean
1.000 = 0.029
VI-mv
1.002 = 0.036
VI-eb
1.018 = 0.028
SGD
1.203 = 0.008
SWA
1.100 = 0.006
SWAG
1.100 = 0.010
Repulsive
1.365 0.009
Dir
1.418 = 0.019
Dropout
1.665 0.059
Laplace
1.130 0.011
VBLL
1.072 = 0.009
Table F.8: SVHN, ProResNet20, NLL
method
Maux 0
Manx 0.1
7aux 0.5
Maxx 1.0
VIFO-naive
0.391 = 0.019
0.603 0.057
0.486 + 0.050
0.479
0.018
VIFO-mean
0.341 = 0.004
0.357
0.004
0.385 0.015
0.334 0.015
VIFO-mv
0.269 = 0.009
0.297 = 0.010
0.323 0.013
0.314 0.009
VTFO-eb
0.359 = 0.002
0.391 + 0.008
0.452+ + 0.018
0.402 + 0.028
VI-naive
0.314 0.024
VI-mean
0.342 = 0.040
VT-mv
0.359 = 0.033
VI-eb
0.379 = 0.057
SGD
0.337 = 0.011
SWA
0.320 = 0.009
SWAG
0.320 = 0.009
Repulsive
0.822 0.020
Dir
0.845 = 0.031
Dropout
0.421 0.068
Laplace
0.344 0.017
VBLL
0.324 0.008
36

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.9: CIFAR10, AlexNet, Accuracy
method
Maux 0
Maux 0.1
7aux = 0.5
Maux
ViFO-naive
0.914
0.001
0.916
0.001
0.915 + 0.001
0.916 0.001
VIFO-mean
0.916 = 0.002
0.914 0.001
1.04
0.001
VIFO-mv
0.914 = 0.001
0.910 0.001
0.912
0.002
VIFO-eb
0.914 = 0.002
0.914 0.001
0.003
0.002
Vl-naive
0.893 0.004
VI-mean
0.884 = 0.004
VI-mv
0.901 = 0.003
VI-eb
0.886 = 0.003
SGD
0.907 = 0.001
SWA
0.909 = 0.001
SWAG
0.909 = 0.001
Repulsive
0.915 0.001
Dir
0.915 = 0.001
Dropout
0.888 = 0.003
Laplace
0.911 0.001
VBLL
0.918 0.001
Table F.10: CIFAR100, AlexNet, Accuracy
method
Maux 0
Manx 0.1
7aux 0.5
Maxx 1.0
VIFO-naive
0.658 = 0.003
0.651 0.004
0.657 + 0.003
0.644 0.003
VIFO-mean
0.667 = 0.002
0.654
0.003
0.650 + 0.007
0.658 0.002
VIFO-mv
0.665 0.002
0.638 + 0.010
0.568 = 0.038
0.558 + 0.028
VTFO-eb
0.669 = 0.002
0.671 + 0.001
0.668 + 0.003
0.664 + 0.002
VI-naive
0.620 0.003
VI-mean
0.608 = 0.002
VI-mv
0.606 = 0.002
VI-eb
0.629 = 0.004
SGD
0.651 = 0.002
SWA
0.652 = 0.002
SWAG
0.652 = 0.002
Repulsive
0.653 0.002
Dir
0.654 = 0.001
Dropout
0.596 0.003
Laplace
0.622 0.002
VBLL
0.660 0.001
37

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.11: STL10, AlexNet, Accuracy
method
Maux 0
Jaux 0.1
7aux 0.5
Maux
ViFO-naive
0.670 = 0.003
0.672 0.002
0.674 + 0.002
0.665
0.001
VIFO-mean
0.680 = 0.002
0.666 0.003
0.669 + 0.003
0.663 0.003
VIFO-mv
0.678 = 0.002
0.665 + 0.002
0.665 - 0.002
0.668 0.002
VIFO-eb
0.662 = 0.002
0.668 + 0.002
0.670 +: 0.004
0.670 0.002
Vl-naive
0.661 0.003
VI-mean
0.649 = 0.009
VI-mv
0.644 = 0.006
VI-eb
0.414 = 0.025
SGD
0.614 = 0.004
SWA
0.598 = 0.006
SWAG
0.615 = 0.003
Repulsive
0.675 = 0.003
Dir
0.667 = 0.004
Dropout
0.622 = 0.004
Laplace
0.648 0.004
VBLL
0.625 = 0.003
Table F.12: SVHN, AlexNet, Accuracy
method
Maux 0
Manx 0.1
7aux 0.5
Maxx 1.0
VIFO-naive
0.962 = 0.001
0.962
0.001
0.962 + 0.000
0.962
0.001
VIFO-mean
0.961 = 0.001
0.962
0.000
0.962 + 0.001
0.963 + 0.001
VIFO-mv
0.963 0.001
0.962 + 0.001
0.963 = 0.000
0.962 0.000
VTFO-eb
0.962 = 0.000
0.961 + 0.001
0.961 + 0.000
0.961 + 0.000
VI-naive
0.953 0.002
VI-mean
0.945 = 0.008
VI-mv
0.955 = 0.001
VI-eb
0.950 = 0.001
SGD
0.955 = 0.001
SWA
0.954 = 0.002
SWAG
0.957 = 0.001
Repulsive
0.962 0.001
Dir
0.964 = 0.000
Dropout
0.949 0.001
Laplace
0.961 0.001
VBLL
0.960 0.001
38

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.13: CIFAR10, PreResNet20. Accuracy
method
Maux 0
Maux 0.1
7aux 0.5
Maux
ViFO-naive
0.897 = 0.003
0.896
0.002
0.893 + 0.003
0.899
0.003
VIFO-mean
0.906 = 0.002
0.900
0.004
0.907 + 0.002
0.904 0.005
VIFO-mv
0.903 = 0.001
0.902
0.002
0.902 + 0.001
0.903
VIFO-eb
0.900 = 0.003
0.900 + 0.005
0.902 + 0.002
0.901 0.003
Vl-naive
0.862 0.009
VI-mean
0.867 = 0.009
VI-mv
0.864 = 0.010
VI-eb
0.859 = 0.011
SGD
0.903 = 0.003
SWA
0.902 = 0.003
SWAG
0.907 = 0.002
Repulsive
0.903 = 0.002
Dir
0.901 = 0.002
Dropout
0.866 = 0.007
Laplace
0.899 0.002
VBLL
0.904 = 0.001
Table F.14: CIFAR100, PreResNet20, Accuracy
method
Maux 0
Manx 0.1
7aux 0.5
Maux 1.0
VIFO-naive
0.631 = 0.004
0.620 0.005
0.631 + 0.002
0.628 0.002
VIFO-mean
0.642 = 0.003
0.635 0.003
0.643 + 0.003
0.637 + 0.006
VIFO-mv
0.436 0.005
0.413 + 0.007
0.398 = 0.006
0.405 + 0.004
VTFO-eb
0.579 = 0.004
0.567 + 0.009
0.535 + 0.004
0.499 + 0.017
VI-naive
0.620 = 0.003
VI-mean
0.608 = 0.002
VI-mv
0.606 = 0.002
VI-eb
0.629 = 0.004
SGD
0.638 = 0.002
SWA
0.640 = 0.002
SWAG
0.642 = 0.002
Repulsive
0.610 0.005
Dir
0.603 = 0.003
Dropout
0.565 0.010
Laplace
0.633 0.004
VBLL
0.629 0.003
39

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.15: STL.10, PreResNet220, Accuracy
method
Maux 0
Maux 0.1
7aux 0.5
Maux
ViFO-naive
0.658 = 0.004
0.658
0.002
0.656 + 0.001
0.656
0.003
VIFO-mean
0.660 = 0.003
0.663
0.003
0.669 + 0.004
0.663 0.004
VIFO-mv
0.670 = 0.004
0.663
0.005
0.662 = 0.005
0.662 0.003
VIFO-eb
0.661 = 0.004
0.661 + 0.004
0.657 + 0.003
0.660 0.005
Vl-naive
0.690 0.009
VI-mean
0.674 = 0.010
VI-mv
0.688 = 0.009
VI-eb
0.638 = 0.011
SGD
0.672 = 0.002
SWA
0.673 = 0.003
SWAG
0.679 = 0.002
Repulsive
0.665 = 0.005
Dir
0.668 = 0.003
Dropout
0.621 = 0.011
Laplace
0.680 0.006
VBLL
0.665 = 0.002
Table F.16: SVHN, PreResNot20, Accuracy
method
Maux 0
Banx 0.1
7aux 0.5
Maux 1.0
VIFO-naive
0.930 = 0.004
0.911
0.014
0.912 + 0.004
0.915 0.008
VIFO-mean
0.933 = 0.004
0.929 0.001
0.930 + 0.001
0.936 0.009
VIFO-mv
0.934 0.006
0.935 + 0.005
0.928 = 0.008
0.930 + 0.002
VTFO-eb
0.929 = 0.004
0.917 + 0.005
0.905 + 0.003
0.924 + 0.007
VI-naive
0.914 = 0.009
VI-mean
0.901 = 0.015
VI-mv
0.902 = 0.012
VI-eb
0.890 = 0.019
SGD
0.929 = 0.007
SWA
0.936 = 0.006
SWAG
0.936 = 0.006
Repulsive
0.931 0.004
Dir
0.932 = 0.002
Dropout
0.877 0.023
Laplace
0.918 0.005
VBLL
0.930 0.003
40

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.17: BCE:CIFARIO-STT..0, AlexNet
method
Maux 0
Maux 0.1
7aux 0.5
Maux
ViFO-naive
0.042 = 0.003
0.042 0.002
0.047 + 0.004
0.045
0.003
VIFO-mean
0.056 = 0.002
0.053 0.003
0.055 + 0.001
0.059 0.002
VIFO-mv
0.067 = 0.002
0.069
0.002
0.068 + 0.002
0.068 0.003
VIFO-eb
0.039 = 0.003
0.042 + 0.002
0.038 + 0.002
0.038 0.002
Vl-naive
0.108 0.002
VI-mean
0.105 = 0.007
VI-mv
0.131 = 0.003
VI-eb
0.118 = 0.004
SGD
0.152 = 0.002
SWA
0.150 = 0.001
SWAG
0.144 = 0.001
Repulsive
0.246 = 0.003
Dir
0.248 = 0.002
Dropout
0.180 = 0.002
Laplace
0.169 0.007
VBLL
0.095 = 0.002
Table F.18: ECE:STL110-CIFAR10,, AlexNet
method
Maux 0
Manx 0.1
7aux 0.5
Maux 1.0
VIFO-naive
0.053 = 0.004
0.053 0.004
0.040 + 0.004
0.034 0.004
VIFO-mean
0.065 = 0.002
0.063 0.003
0.055 + 0.003
0.046 0.002
VIFO-mv
0.075 = 0.002
0.067 = 0.003
0.078 = 0.003
0.073 = 0.002
VTFO-eb
0.081 = 0.004
0.078 + 0.002
0.066 + 0.002
0.075 0.004
VI-naive
0.072 = 0.007
VI-mean
0.107 = 0.007
VT-mv
0.155 = 0.004
VI-eb
0.030 = 0.006
SGD
0.238 = 0.007
SWA
0.137 = 0.016
SWAG
0.117 = 0.007
Repulsive
0.197 0.006
Dir
0.146 = 0.004
Dropout
0.365 0.007
Laplace
0.085 0.004
VBLL
0.129 0.007
41

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.19: ECEECTFARRO-STTL10, PreResNet200
method
Maux 0
Maux 0.1
7aux 0.5
Maux
ViFO-naive
0.029 = 0.001
0.032 0.003
0.032 + 0.004
0.037
0.004
VIFO-mean
0.036 = 0.003
0.028 0.005
0.032 + 0.002
0.029 0.003
VIFO-mv
0.058 = 0.003
0.047
0.004
0.052 + 0.002
0.051 0.002
VIFO-eb
0.024 = 0.003
0.028 + 0.005
0.031 + 0.002
0.026 0.002
Vl-naive
0.126 0.004
VI-mean
0.147 = 0.002
VI-mv
0.159 = 0.003
VI-eb
0.137 = 0.005
SGD
0.092 = 0.004
SWA
0.089 = 0.001
SWAG
0.078 = 0.003
Repulsive
0.259 = 0.004
Dir
0.319 = 0.007
Dropout
0.146 = 0.002
Laplace
0.063 0.002
VBLL
0.095 = 0.001
Table F.20: ECE:STLIO-CIFARIO, PreResNet20
method
Maux 0
Manx 0.1
7aux 0.5
Maux 1.0
VIFO-naive
0.021 = 0.001
0.022 0.002
0.025 + 0.003
0.028 0.004
VIFO-mean
0.029 = 0.003
0.024 0.005
0.019 + 0.002
0.019 0.003
VIFO-mv
0.088 = 0.005
0.081 + 0.003
0.081 + 0.004
0.086 + 0.002
VTFO-eb
0.051 = 0.005
0.039 + 0.002
0.030 + 0.003
0.038 + 0.002
VI-naive
0.129 = 0.003
VI-mean
0.151 = 0.004
VI-mv
0.171 = 0.011
VI-eb
0.094 = 0.005
SGD
0.112 = 0.002
SWA
0.107 = 0.003
SWAG
0.093 = 0.004
Repulsive
0.185 0.006
Dir
0.200 = 0.005
Dropout
0.290 0.007
Laplace
0.107 0.003
VBLL
0.097 0.009
42

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.21: Eutroo::CIFARIOOSSVEN, AlexNet
method
Maux 0
7aaxx 0.1
7aux 0.5
Maux
ViFO-naive
1.357 = 0.009
1.399 0.023
1.286 + 0.049
1.355
0.033
VIFO-mean
1.344 = 0.020
1.336 0.021
1.313 0.042
1.342 0.039
VIFO-mv
1.344 = 0.020
1.342 0.018
+.354
1.332 + 0.011
VIFO-eb
1.330 0.045
1.330 + 0.026
1.296 0.027
Vl-naive
1.238 0.075
VI-mean
1.280 = 0.157
VI-mv
1.053 = 0.068
VI-eb
1.133 = 0.069
SGD
0.633 = 0.014
SWA
0.676 = 0.014
SWAG
0.705 = 0.011
Repulsive
1.990 = 0.021
Dir
1.970 = 0.002
Dropout
0.585 0.033
Laplace
1.245 0.025
VBLL
0.971 = 0.049
Table F.22: Entropy:STLL1O-SHHN, AlexNet
method
Maux 0
Maux 0.1
7aux 0.5
Maux 1.0
VIFO-naive
1.772 = 0.017
1.773 0.015
1.779 + 0.024
1.763
0.011
VIFO-mean
1.840 = 0.023
1.748 0.012
1.818 + 0.029
1.826 0.005
VIFO-mv
1.889 0.026
1.915 + 0.031
1.848 = 0.051
1.859 0.018
VTFO-eb
1.764 = 0.026
1.754 + 0.018
1.798 + 0.028
VI-naive
1.601 0.049
VI-mean
1.495 = 0.018
VI-mv
1.345 = 0.046
VI-eb
2.024 = 0.019
SGD
1.127 = 0.030
SWA
1.479 0.047
SWAG
1.525 0.010
Repulsive
2.208 0.006
Dir
2.157 = 0.008
Dropout
0.601 0.037
Laplace
1.349 0.007
VBLL
1.454 0.044
43

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.23: Fatropy:SVINN-IFFARRO, AlexNet
method
Maux 0
7aux 0.1
7aux 0.5
Maux
ViFO-naive
1.694
0.008
1.711
0.005
1.694 + 0.003
1.693
0.002
VIFO-mean
1.721 = 0.013
1.735 0.007
1.729 + 0.009
1.742 0.008
VIFO-mv
1.709 = 0.014
1.738 + 0.011
1.754
1.758 +
VIFO-eb
1.740 = 0.015
1.749 + 0.005
1.739 0.007
1.773 0.003
Vl-naive
1.624 0.048
VI-mean
1.711 = 0.052
VI-mv
1.488 = 0.036
VI-eb
1.577 = 0.058
SGD
1.237 = 0.009
SWA
1.323 0.035
SWAG
1.465 = 0.004
Repulsive
2.129 0.003
Dir
2.121 = 0.002
Dropout
1.206 0.047
Laplace
1.696 0.036
VBLL
1.466 = 0.013
Table F.24: Entropy:SVHN--TTL.0,, AlexNet
method
Maux 0
Taux 0.1
7aux 0.5
Maux 1.0
VIFO-naive
1.720 = 0.007
1.734 0.005
1.711 + 0.004
1.708
0.008
VIFO-mean
1.744 = 0.007
1.756
0.006
1.753 0.011
1.756 0.007
VIFO-mv
1.754 = 0.012
1.763 + 0.006
+.804
1.790 = 0.009
VTFO-eb
1.764 = 0.016
1.776 + 0.007
1.775
+ 0.017
1.765 + 0.030
VI-naive
1.685 0.056
VI-mean
1.772 = 0.042
VI-mv
1.512 = 0.041
VI-eb
1.631 = 0.047
SGD
1.277 = 0.009
SWA
1.371 = 0.013
SWAG
1.507 0.007
Repulsive
2.138 0.004
Dir
2.132 0.002
Dropout
1.272
0.054
Laplace
1.702
0.025
VBLL
1.508 0.009
44

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.25: Entropy:CIFARRIOSSVHH,, PreResNet20
method
Maux 0
7aux 0.1
7aux 0.5
Maux
ViFO-naive
1.465
0.007
1.516 0.009
1.553 + 0.006
1.540
0.004
VIFO-mean
1.469 = 0.009
1.553 0.018
1.599 + 0.005
1.580 0.010
VIFO-mv
1.559 = 0.013
1.611 + 0.016
+ + 0.017
1.543
VIFO-eb
1.532 0.011
1.540 + 0.013
1.582 + 0.018
1.005
Vl-naive
1.192 0.025
VI-mean
1.097 = 0.015
VI-mv
1.048 = 0.019
VI-eb
1.174 = 0.049
SGD
1.398 = 0.009
SWA
1.398 0.015
SWAG
1.586 = 0.016
Repulsive
1.971 0.006
Dir
2.246 = 0.002
Dropout
0.995 0.022
Laplace
1.523 0.010
VBLL
1.252 = 0.012
Table F.26: Entropy:STTLIO-SVVNN,, PreRosNet20
method
Maux 0
Maux 0.1
7aux 0.5
Maux 1.0
VIFO-naive
1.763 = 0.008
1.770 0.005
1.742 + 0.009
1.728
0.007
VIFO-mean
1.796 = 0.006
1.908 0.008
1.876 + 0.002
1.862 0.008
VIFO-mv
1.565 0.007
1.607 = 0.004
1.588 + 0.005
1.574 + 0.008
VTFO-eb
1.603 = 0.005
1.637 + 0.011
1.634 + 0.008
1.656 + 0.017
VI-naive
1.394 0.020
VI-mean
1.323 = 0.013
VI-mv
1.267 = 0.030
VI-eb
1.517 = 0.060
SGD
1.432 = 0.022
SWA
1.462 = 0.013
SWAG
1.579 0.013
Repulsive
2.160 0.005
Dir
2.194 0.004
Dropout
0.830 0.024
Laplace
1.313 0.006
VBLL
1.481 0.008
45

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.27: Entropy:SVHNN-CFAARR,, PreResNet20
method
Maux 0
7aux 0.1
7aux 0.5
Maux
ViFO-naive
1.531
0.008
2.019 0.026
1.942 + 0.051
1.959
0.039
VIFO-mean
1.515 = 0.012
1.908 0.080
1.850 + 0.065
1.780 0.028
VIFO-mv
1.568 = 0.014
1.758 + 0.051
1.848 + 0.067
1.706 0.057
VIFO-eb
1.539 0.004
1.775 + 0.038
1.898 + 0.022
1.923 0.015
Vl-naive
1.213 0.010
VI-mean
1.169 = 0.017
VI-mv
1.080 = 0.011
VI-eb
1.219 = 0.029
SGD
1.384 = 0.005
SWA
1.403 0.002
SWAG
1.476 = 0.005
Repulsive
2.006 = 0.013
Dir
2.250 = 0.001
Dropout
1.059 0.019
Laplace
1.502 0.008
VBLL
1.315 0.007
Table F.28: Entropy:SVHN-STLLI,, PreRosNet20
method
Maux 0
Maux 0.1
7aux 0.5
Maux 1.0
VIFO-naive
1.529 = 0.016
1.995 0.042
1.860 + 0.041
1.885 0.054
VIFO-mean
1.543 = 0.012
1.837 0.070
1.875 + 0.066
1.762 0.052
VIFO-mv
1.588 = 0.010
1.763 + 0.061
1.875 = 0.041
1.671 + 0.039
VTFO-eb
1.544 = 0.004
1.724 + 0.045
1.895 + 0.054
1.785 + 0.028
VI-naive
1.222 = 0.013
VI-mean
1.174 = 0.016
VI-mv
1.078 = 0.014
VI-eb
1.219 = 0.024
SGD
1.375 = 0.007
SWA
1.389 0.011
SWAG
1.464 0.005
Repulsive
2.014 0.011
Dir
2.252 0.002
Dropout
1.053 0.014
Laplace
1.506 0.008
VBLL
1.319 0.006
46

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.29: AUTROC:CFARROSSWHYY AlexNet
method
Maux 0
Maux 0.1
7aux = 0.5
Maux
ViFO-naive
0.860 = 0.005
0.857 0.008
0.833 + 0.018
0.853 0.018
VIFO-mean
0.873 = 0.005
0.855
0.008
0.858 + 0.007
0.856 0.008
VIFO-mv
0.893 = 0.007
0.871 + 0.004
0.873 + 0.004
0.861 0.006
VIFO-eb
0.860 = 0.007
0.859 + 0.007
0.864 +: 0.002
0.844
Vl-naive
0.898 0.016
VI-mean
0.886 = 0.029
VI-mv
0.893 = 0.009
VI-eb
0.885 = 0.010
SGD
0.851 = 0.004
SWA
0.862 = 0.009
SWAG
0.863 = 0.005
Repulsive
0.857 = 0.020
Dir
0.846 = 0.006
Dropout
0.822 = 0.011
Laplace
0.871 0.006
VBLL
0.889 = 0.003
Table F.30: AUROC:STLIO-SVHN, AlexNet
method
Maux 0
Manx 0.1
7aux 0.5
Maux 1.0
VIFO-naive
0.787 = 0.005
0.776 0.009
0.781 + 0.010
0.755 0.015
VIFO-mean
0.789 = 0.010
0.757
0.006
0.768 + 0.013
0.774 #
VIFO-mv
0.798 = 0.008
0.772 = 0.010
0.764
0.779 0.013
VTFO-eb
0.793 = 0.007
0.793 + 0.010
0.769 + 0.009
0.758 0.005
VI-naive
0.818 = 0.020
VI-mean
0.792 = 0.011
VI-mv
0.775 = 0.018
VI-eb
0.736 = 0.044
SGD
0.750 = 0.010
SWA
0.761 = 0.010
SWAG
0.769 = 0.005
Repulsive
0.799 0.013
Dir
0.779 = 0.004
Dropout
0.681 0.017
Laplace
0.783 0.006
VBLL
0.783 0.015
47

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.31: AUTROC:YYHN-CIFRRII, AlexNet
method
Maux 0
Maax 0.1
7aux 0.5
Maux
ViFO-naive
0.969 = 0.001
0.968
0.001
0.967 + 0.001
0.967
0.001
VIFO-mean
0.976 = 0.001
0.974 + 0.001
0.973 0.001
0.973 0.002
VIFO-mv
0.972 = 0.001
0.973 + 0.001
0
0.973 0.000
VIFO-eb
0.969 = 0.001
0.971 + 0.001
+.900+
0.973 0.003
Vl-naive
0.970 0.002
VI-mean
0.963 = 0.014
VI-mv
0.965 = 0.003
VI-eb
0.967 = 0.003
SGD
0.962 = 0.001
SWA
0.963 = 0.001
SWAG
0.968 = 0.001
Repulsive
0.972 = 0.002
Dir
0.976 = 0.001
Dropout
0.952 = 0.003
Laplace
0.971 0.002
VBLL
0.970 = 0.001
Table F.32: AUROC:SVHN-STL.O,, AlexNet
method
Maux 0
Manx 0.1
7aux 0.5
Maux 1.0
VIFO-naive
0.971 = 0.000
0.972 0.001
0.969 + 0.001
0.971
0.001
VIFO-mean
0.976 = 0.000
0.976 + 0.000
0.977 + 0.001
0.977
0.001
VIFO-mv
0.976 0.001
0.975 + 0.000
0.977 0.001
0.976 0.000
VTFO-eb
0.974 = 0.001
0.974 + 0.001
0.973 + 0.001
0.977 0.002
VI-naive
0.974 0.003
VI-mean
0.968 = 0.013
VT-mv
0.970 = 0.002
VI-eb
0.971 = 0.002
SGD
0.966 = 0.001
SWA
0.968 = 0.002
SWAG
0.972 = 0.000
Repulsive
0.975 0.002
Dir
0.979 = 0.001
Dropout
0.958 0.004
Laplace
0.975 0.002
VBLL
0.973 0.000
48

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.33: AUROC:CIFARIOOSVHN, PreResNet20
method
Maux 0
Maax 0.1
7aux 0.5
Maux
ViFO-naive
0.885 = 0.004
0.880 0.006
0.889 + 0.005
0.898
0.006
VIFO-mean
0.894 = 0.004
0.889 0.008
0.907 + 0.007
0.896 0.002
VIFO-mv
0.914 = 0.002
0.919 + 0.007
0.917 + 0.001
0.915 0.001
VIFO-eb
0.908 = 0.002
0.912 + 0.005
0.918 + 0.002
0.913
Vl-naive
0.863 0.008
VI-mean
0.868 = 0.011
VI-mv
0.868 = 0.008
VI-eb
0.858 = 0.013
SGD
0.950 = 0.001
SWA
0.950 = 0.001
SWAG
0.963 = 0.001
Repulsive
0.819 = 0.004
Dir
0.962 = 0.002
Dropout
0.837 = 0.010
Laplace
0.957 0.002
VBLL
0.902 = 0.003
Table F.34: AUROC:STLIO-SVIN, PreResNet200
method
Maux 0
Manx 0.1
7aux 0.5
Maxx 1.0
VIFO-naive
0.810 = 0.005
0.810 0.004
0.804 + 0.004
0.803 0.003
VIFO-mean
0.810 = 0.001
0.835 0.006
0.836 + 0.004
0.820 + 0.003
VIFO-mv
0.788 = 0.004
0.794 + 0.001
0.789 = 0.003
0.784 + 0.005
VTFO-eb
0.799 = 0.004
0.800 + 0.005
0.803 + 0.002
0.803 + 0.004
VI-naive
0.788 = 0.014
VI-mean
0.784 = 0.006
VI-mv
0.788 = 0.015
VI-eb
0.728 = 0.030
SGD
0.795 = 0.008
SWA
0.837 = 0.008
SWAG
0.877 = 0.005
Repulsive
0.804 0.005
Dir
0.812 = 0.004
Dropout
0.689 0.007
Laplace
0.810 0.004
VBLL
0.810 0.008
49

---

Published in Transactions on Machine Learning Research (09/2024)
Table F.35: AUROCSSVTIN-CIAARII, PreResNet20
method
Maux 0
Maax 0.1
7aux 0.5
Maux
ViFO-naive
0.890 = 0.005
0.958
0.003
0.934 + 0.010
0.948
0.004
VIFO-mean
0.893 = 0.002
0.931 0.010
0.925 0.006
0.928 0.014
VIFO-mv
0.912 0.004
0.936 + 0.005
0.927
0.916
VIFO-eb
0.916 = 0.007
0.922 + 0.002
0.923
0.941 0.009
Vl-naive
0.870 0.008
VI-mean
0.867 = 0.016
VI-mv
0.866 = 0.012
VI-eb
0.847 = 0.026
SGD
0.909 = 0.006
SWA
0.918 = 0.005
SWAG
0.919 = 0.006
Repulsive
0.843 = 0.006
Dir
0.970 = 0.003
Dropout
0.830 = 0.028
Laplace
0.890 0.006
VBLL
0.906 = 0.004
Table F.36: AUROC:SVHN-STLIO,, PreResNet200
method
Maxx 0
Janx 0.1
7aux 0.5
Maax 1.0
VIFO-naive
0.894 = 0.009
0.957 0.009
0.934 + 0.008
0.936
0.010
VIFO-mean
0.896 = 0.004
0.924 + 0.013
0.914 0.0100
0.921 0.009
VIFO-mv
0.917 0.006
0.933 + 0.008
0.928 - 0.009
0.922 + 0.001
VTFO-eb
0.911 = 0.006
0.916 + 0.005
0.923 + 0.004
0.936 + 0.006
VI-naive
0.870 0.010
VI-mean
0.867 = 0.016
VI-mv
0.865 = 0.011
VI-eb
0.846 = 0.027
SGD
0.907 = 0.007
SWA
0.913 = 0.008
SWAG
0.915 = 0.007
Repulsive
0.847 0.007
Dir
0.970 = 0.003
Dropout
0.829 0.029
Laplace
0.911 0.007
VBLL
0.909 0.003
50


Reviews:
[{"review_id": "Yf25WKHRVO", "reviewer": "Reviewer_c4nX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The method is reasonably simple (for a Bayesian method) and computationally efficient. \n- The discussion of representational capacity of the model is a good contribution, but should be expanded upon (discussed below).\n\nWeaknesses:\n- From figure 1, the method without auxiliary training seems very weak. The introduction of the auxiliary loss is reasonable, but the discussion of this term is lacking and seems essential for the method. \n  - The authors _must_ discuss how ablations of the sampling distribution for x_aux, how eta_aux should be tuned in practice, and any other details. \n- The baselines compared to are not sufficient. There are many comparable inexpensive single-pass uncertainty quantification methods that should be compared to \n  - for example: DUQ, SNGP, VBLL, last layer Laplace (these methods should also all be discussed in the related work)\n- VIFO is very similar to deep VIBs (Alemi et al, 2017), and that work should be discussed. \n- As far as I can tell, much of the presentation of the collapsed VI (including in appendix C) is very similar to Tomczak et al (2021). The authors should precisely lay out what is novel in their presentation. \n- While section 3 discusses the ability of VIFO to match the output of a linear variational model, it doesn't discuss the difference in regularization terms between these two models, which may be interested for understanding the output-VI approach. \n- The Rademacher complexity analysis is not particularly important to the paper and, while it is a reasonable technical result, does not add much to the paper overall. It could be moved to the appendix, ideally to make room for a more robust experimental characterization. \n- The results presented are for ensembles of VIFO. This is highly misleading, and the authors should:\n  - Include results for a single VIFO model \n  - Make sure to label VIFO ensembles as VIFO ensembles\n  - Include an ensemble of standard models as a baseline\n\nSmall comments:\n- Several typos: \"Beyesian\" in the abstract, VIFO not consistently capitalized\n- The discussion of the model in section 2 is a bit unclear. For example, the presentation of the regression model (having four outputs) is unclear and should be more formally presented. In general, I would recommend the authors make one clear subsection on classification and one on regression. \n- The authors repeatedly state that they perform variational inference on the _output layer_, which is misleading. The authors perform VI on the _outputs_, and the output layer (corresponding to the weights) are not regularized (by a term KL penalty).\n- The presentation of figures 3-6 probably do not need to be plots, and could be tables instead. \n- Inclusion of arrows for \"which direction is better\" in metrics is useful for casual readers; currently the authors include it for some figures but it would be helpful to include for all.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce the VIFO method, including parameterization of the model, introduction of auxiliary losses, and discussion of collapsed VI methods. The variational formulation is similar to variational policy parameterization in reinforcement learning and VAEs, in which the KL divergence of the function output is regularized, as opposed to model parameters (as in Bayesian NNs).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The following changes should be made:\n- Strengthen baselines, as described above (single-pass uncertainty methods, standard ensembles)\n- Add a discussion of similarities to VIFO\n- Discuss the novelty of the collapsed VI results. \n- Fix typos.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g16cPVQz82", "reviewer": "Reviewer_Q3xj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe model is reasonably simple and appears to achieve a good tradeoff between efficiency, in-distribution performance and out-of-distribution uncertainty quantification.\n\nThe paper provides both empirical evidence and theoretical justifications.\n\n**Weaknesses**\n\nThe abstract states that \"we incorporate collapsed variational inference with VIFO which significantly improves the performance in practice\". However, this is not clearly established by experiments in the main paper (although there are additional results in the appendix). Likewise, it isn't clear how performance scales with the number of models in the ensemble.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes variational inference applied to the model final-layer outputs (VIFO), directly predicting the mean and variance of each output unit (logits for classification). The authors optimize an evidence lower bound (ELBO), incorporating auxiliary training and collapsed variational inference. VIFO has the same expressiveness as standard variational inference for linear models, but is less powerful for deep networks. In practice, as VIFO may be trained fairly efficiently, this may be mitigated by using ensembles of VIFO models. On a few computer vision datasets, VIFO achieves similar (and sometimes superior) in-distribution performance as both deterministic and bayesian baselines, and also captures uncertainty in out-of-distribution data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. You mention that $W_1$ and $W_2$ have shared components. What exactly is shared, and what isn't?\n\n2. In Eq. 2, as g(l) as the normal distribution variance, should the notation be $\\mathcal{N}(y|m, g(l))$ instead of $\\mathcal{N}(y|m, l)$?\n\n3. Could you briefly describe similarities and differences between your proposed approach and [1], which was very recently published?\n\n4. Could you elaborate on the relationship between your work and the local reparametrization trick, which also moves the noise from the model weights to its activations (although not necessarily the last layer only)?\n\n5. Before introducing your approach, it could be helpful to briefly recap how standard (and deterministic) variational inference works.\n\n6. Table 1: mention time unit.\n\n7. How many models are there in each ensemble?\n\n[1] Harrison et al. Variational Bayesian Last Layers. ICLR 2024. https://arxiv.org/pdf/2404.11599", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IiFDqdD2dR", "reviewer": "Reviewer_wCD2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- This paper is generally well-written, and the presentation is clear.\n- The properties of the proposal have been carefully examined both theoretically and empirically, ensuring high overall quality. Notably, the representation power (expressiveness) of the proposed VIFO has been theoretically analyzed, presenting both positive (the linear case) and negative (the non-linear case) results.\n- The novelty of the proposal is sufficient for a TMLR paper, in my opinion.\n### Weaknesses\n- Since one of the advantages of VIFO is its lower computational cost, the time complexity of VIFO and its comparison to VI should be presented.\n- In the analysis of expressiveness in Section 3, the dimensionality of $\\theta$ is assumed to be the same as that of an input vector, $d$. However, this assumption does not hold in typical situations, as neural networks tend to have a much larger number of parameters. How does this assumption impact the theoretical results (Theorems 3.1, 3.2)?\n- In the experiments, the runtime for training 1 epoch has been measured. However, since the number of epochs can vary among compared methods, it would be valuable to show and compare the total runtime for training.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes applying variational inference (VI) to not the parameter space of neural networks but solely the final layer output space. This approach, called VIFO, aims at estimating the mean and variance of the output with a reduced computational cost compared to the standard VI. The theoretical analysis of VIFO's representational power demonstrates that it is equivalent to that of VI for linear models, while strictly less powerful for non-linear models. The efficiency and effectiveness of the proposed VIFO method have been empirically validated using real-world datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the concerns outlined in the Weaknesses section above.", "broader_impact_concerns": "I do not have any concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a method for marginalizing over the final layer of a deep neural network through using variational inference.  Such "last-layer" strategies are quite appealing, as they can be done far more efficiently than integrating over all the parameters of the model while still retaining some of the benefits of a Bayesian approach.  The reviewers all found that the paper was well written, interesting, sufficiently novel and relevant to the TMLR audience.  Initially multiple reviewers found the experiments lacked comparison to state-of-the-art baselines.  However, the authors added significant additional experiments, particularly to the highly related VBLL method, and the reviewers seemed satisfied with this new comparison.   Although the work is novel enough for TMLR, the reviewers pointed out that there is a significant amount of quite related work that should be discussed (and is in the latest version).  One reviewer suggested moving the Rademacher complexity section to the appendix (optional for the camera ready).

In the field, researchers have drawn out a kind of Pareto frontier of computational / memory efficiency of uncertainty quantification vs quality of the uncertainty estimates.  This paper provides a nice addition to the highly efficient part of that curve.  Overall a nice paper that provides a practically useful contribution.  The reviewers all voted to accept, so the recommendation is to accept the paper as is.  Please include the changes from the last revision in the camera ready version.
Decision: Accept (poster)