Rethinking Approximate Gaussian Inference in Classification
Authors: Bálint Mucsányi
Nathaël Da Costa
Philipp Hennig
Venue: NeurIPS 2025 poster ()
Keywords: Uncertainty quantification
Predictives
Predictive distribution
Sampling-free
Primary area: probabilistic_methods
TL;DR: This paper provides a framework for the sampling-free computation of predictives for approximate Gaussian models in classification.

Abstract: In classification tasks, softmax functions are ubiquitously used as output activations to produce predictive probabilities. Such outputs only capture aleatoric uncertainty. To capture epistemic uncertainty, approximate Gaussian inference methods have been proposed. We develop a common formalism to describe such methods, which we view as outputting Gaussian distributions over the logit space. Predictives are then obtained as the expectations of the Gaussian distributions pushed forward through the softmax. However, such softmax Gaussian integrals cannot be solved analytically, and Monte Carlo (MC) approximations can be costly and noisy. We propose to replace the softmax activation with element-wise normCDF or sigmoid, which allows for the accurate sampling-free approximation of predictives. This also enables the approximation of the Gaussian pushforwards by Dirichlet distributions with moment matching. This approach entirely eliminates the runtime and memory overhead associated with MC sampling. We evaluate it combined with several approximate Gaussian inference methods (Laplace, HET, SNGP) on large- and small-scale datasets (ImageNet, CIFAR-100, CIFAR-10), demonstrating improved uncertainty quantification capabilities compared to softmax MC sampling. Our code is available at https://github.com/bmucsanyi/probit.

Rethinking Approximate Gaussian Inference in Classification
Bálint Mucsányi*
Nathaël Da Costa"
Philipp Hennig
Tübingen AI Center University of Tübingen
Abstract
In classification tasks, softmax functions ubiquitously used output activations to produce predictive probabilities. Such outputs only capture aleatoric uncertainty. To capture epistemic uncertainty, approximate Gaussian inference methods have been proposed. We develop common formalism to descrihe such methods, which view outputting Gaussian distributions over the logit space. Predictives then obtained as the expectations the Gaussian distributions pushed forward through the softmax. However, such softmax Gaussian integrals cannot be solved analytically, and Monte Carlo (MC) approximations can be costly and noisy. We propose replace the softmax activation by element-wise normCDF sigmoid, which allows for the accurate sampling-free free approximation of predictives. This also enables the approximation of the Gaussian pushforwards by Dirichlet distributions with moment matching. This approach entirely eliminates the runtime and memory overhead associated with MC sampling. We evaluate it combined with several approximate Gaussian inference methods (Laplace, HET, SNGP) on large- and small-scale datasets (ImageNet, CIFAR- 100, CIFAR-10), demonstrating improved uncertainty quantification capabilities compared to softmax MC sampling. Our code is available at github. com/ httn//muueaayy//poottt
Introduction
Given an input space X. C' classes and training data (In, Go) C x fl..... C), the goal of probabilistic classification is to learn function h: X' - 1 C-1 where
A C-1 (pp...... Pc) E [0, 1 P1
+P
(1)
is the (C - 1)-dimmnsio probability simplex. The model h outputs the probability of inputs x E X belonging each class, he(x) p(c x). To learn such map to the simplex, h is typically defined as composition
h: X
R
softmax A C-1
where function f. learned from the training data, is mapped through the softmax activation function. In this case, C is called the logit space, and, for E X. f(x) is logit.
With the usual assumption of i..dd data, the likelihood under the model given by
N
C
((( (x (z
Il p(cn xx)) n=1
II
he
Sen.e Senie
(3)
==1 c
EEuual contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Figure 1: Mean KL divergence between the 'true' predictive (approximated with a 10000 sample MC approximation) and different pre- dictive approximations on a synthetic data set. This synthetic dataset consists of 100 logit means l and standard deviations a, whose components are i..d. uniformly dis- tributed. For a fair comparison we scale the logit dataset for the different activations to match the activations to first order, so that
x 10
Soltmux MC
1.0
NomCCF MC
Sigmoid MC
KL to true predictive
Softmax Mean Field
Exp (ours)
NommCD (ours)
0.6
Sigmoid (ours)
0.4
Pe [-1, 1], 0r E [0. 1] for sigmoid, He E [-1/2 - log 2, 1/2 1og2], de E 1/2] for softmax, Me [- / VT/8) de [0, / for normCDF. The MC approxima- tions capped at fixed computational bud- get of 10000 class samples, i.e. [10000/C] samples (see also Appendix C.I).
0.0
1020
40
60
80
100
120
140
160
180
200
Number of classes
where 5 is the Kronecker delta. This yields natural loss function for classification problems, the cross-entropy (CE) loss
[((En; Gn) = log(p((cw) (In))=z))
Σ Sen log(h)(zm))
(4)
For typical models h trained through such approximate maximum likelihood estimation, output probability vector hxx) E A C- should interpreted as an estimate of (gen(yy x)) the probability under the generative model. Notably, the output probability does not take into account the uncertainty of the model due to the finite nature of the data. the uncertainty quantification formalism, such models can only estimate aleatoric uncertainty and disregard epistemic uncertainty [21].
The probabilistic way capture epistemic uncertainty is to require the model to output second-order distribution (that is 'a distribution over probability distributions ) Namely, for each input æ E X' the model should output a probability measure over the simplex h(x)) E P(A C-1), as opposed to point estimate in the simplex (xx) A C-1 [46, 6].
number of methods for such distributional uncertainty quantification in classification rely on approximate Gaussian inference in logit space. That is, h is written as a composition
( softmax, P(AC-1)
h: X'
(5)
where is learned from the training data, ((RC) is the set of Gaussian measures on the logit space RC, and softmax, pushes forward the Gaussian probability measures through the softmax. For example, Heteroscedastic Classifiers (HET) [7] learn the output means and covariances of f explicitly with neural network. linearised Laplace approximations [11], only the means are learned explicitly with a neural network, while the covariances are obtained post-hoc from that neural network by approximating its parameter posterior distribution with a Gaussian, and then pushing it forward to logit space by linearising the network in the parameters. Last-layer Laplace approximations [25] are variant linearised Laplace approximations where only the posterior over last-layer parameters is
requiring no linearisation of the network. Spectral-Normalised Gaussian processes (SNGP) [31] are neural networks with last layers that are approximately Gaussian processes, through random feature expansions and Laplace approximations.
To obtain predictive probabilities from distributional classifier as in Eq. (5), we marginalise out by the measure-theee theoretic change of variables, using h(x) = softmax, f(x)):
Ep-h(a) P = //- -- h(x) = / softmax(y) (dy)
(6)
In addition to predictive probabilities, the distributional framework allows for the acquisition of other quantities of interest, such as variances, entropy, and information-theorrticc decompositions of the

---

predictive uncertainty into aleatoric and epistemic parts [13, 41, 55]. These are usually obtained by Monte Carlo (MC) sampling from the Gaussian measure f(z:). Indeed, while the probability density of h(x)) softmax, f((r) can be obtained in closed form using a change of variables formula [2], this does not seem of practical use-even predictives cannot be computed analytically. On the other hand, given a variance, the runtime and memory cost required for an MC approximation to achieve an estimator of said variance grows linearly with the number of classes. Crucially, this computational cost is not limited to training time, but also to inference time, when the computational budget may be much more limited. This limits the application such Gaussian inference methods classification with large number of classes, such as those common in computer vision and language modelling.
Summary of Contributions: Our work extends the classical probit approximation for the Gaussian sigmoid integral in binary classification [49, 34] in the following manner:
Generalising it to the multi-class setting (Sections 3.1 and 3.2).
(ii) Analysing theoretically (Section 3.3).
(iii) Analysing it empirically, and how it outperforms previous approximations (Fig. and Section 7),
(iv) Generalising it to second-order distributional Dirichlet approximations (Section 4).
(v) Combining it with advantageous training (Sections and 7).
Our proposed framework can be summarised as 3-step recipe:
ii) Choose approximate Gaussian inference method that outputs logit means and covariances.
(ii) Choose an output activation that yields a closed-form Gaussian integral (Section 3). (iii) At inference time, use our closed-form formulas (Section 3 for the predictives, Section 4 for other quantities of interest).
2 Related Work
In uncertainty quantification, methods that output logit Gaussians, which we refer to as approximate Gaussian inference methods, have become one of the dominant paradigms in uncertainty quantifica- tion [7, 11, 25, 31J. classification tasks, such methods have the downside of requiring sampling at inference to transform Gaussian distributions into predictive probability vectors.
In parallel, prior work has developed ways propagate probabilistie uncertainty through neural networks [16]. Several works develop analytic propagation of covariances up to the logits [56, 29]; our method complements these by addressing the propagation of the logit-space Gaussian distributions onto the probability simplex. Few works address this last step. The Laplace bridge [19] proposes map from Gaussians to Dirichlets on the probability simplex. However, this approach has no guarantees on the quality of its predictives, and it severely underperforms Monte Carlo sampling. Other approaches include the mean-field approximation to the Gaussian softmax integral [32] and the approximations of Shekhovtsov and Flach [48]. In this work, we propose approximate predictives that outperform existing closed-fome predictive techniques and provide second-order Dirichlet distributions for other closed-form quantities of interest. Furthermore, this is the first work to provide a theoretical analysis of such an approximation.
Other works have also proposed various activations for classification to replace the softmax. Particu- larly relevant is [15], which composes slement-wise sigmoids with normalisation and utilises auxiliary- variable augmentation to obtain closed-form ELBOO terms during training; however, its predictives require Monte Carlo estimation. Related manjugate/augmentatii based objectives exist for stick- breaking and tree-structured multi-class GPs [30, I], and for BNNs, a tempered-likeline likelihood/Dirrchllt perspective with Sactorise--Gaussia surrogate has been explored [24]. Yet, all these approaches still require sampling or numerical quadrature at inference time. [49, 34, 25] use the probit approximation in the binary case, which we generalise to the multi-class setting. [51] approximates the softmax by providing lower bounds for point estimate training, but replacing the softmax with these bounds during training does not yield a normalised multi-class likelihood. The binary cross-entropy loss with the element-wisg sigmoid activation has been reported to outperform cross-entropy in large-scale settings [54, 28]. Our work the first to use this advantageous loss while normalising the output to obtain proper multi-class probabilistic models with closed-form predictives from logit Gaussians.
3

---

Evidential deep learning (EDL) approaches [46] offer sample-free outputs through Dirichlet dis- tributions but have been shown be unreliable for uncertainty estimation and disentanglement [4, 23, 44, 40]. We focus on making approximate Gaussian inference methods sample-free in principled and performant manner; however, for completeness, we compare our approach with EDI. methods in Appendix K.8.
Finally, multiple authors have explored continuous approximations categorical distributions, which enable gradient-based optimization for discrete stochastic models through sampling, such as the Gumbel-max trick [35, 22, 36, 20]. These methods principally optimise single categorical distribution, while the present work is concerned with constructing probability densities over the (continuous) simplex of such categorical distributions.
3 Gaussian Inference with Closed-Form Predictives
3.1 Motivation from Softmax Models
A key difficulty in obtaining exact or closed-fome approximate predictive probabilities of h(v) softmax, f(x)) is that pushforwards of Gaussian distributions through the softmax are intractable. The softmax function is the composition softmax n exx, where is the normalisation function given by n (1)): :- q (qi + + (C) and exp applied element-wise. Thus, h can be written as composition (see Eq. (5))
h: X
) exp-, P((0,00))) Pi P(AC-1).
Now it is straightforward to see that (Appendix A.I)
| exp(y) Nu,, o (dy)) = exp 1 0²
2
(8)
and thus
/ exp(y) f(x)(dy) = exp (M(x) x2(x)
Exxexp, ((x) [Q]
(9)
JRc
where p(x) and (2) are, respectively, the mean and the diagonal of the covariance of f(x), and the vector division is element-wise. Note that approximations have been made far.
One can use this approximate the model's predictive in closed form:
Q
E Eq_exp, f(a))
Q
E Ep PP] = EQ_exp, ((x)
E Qc
Eq f(x)
ΣE Qc
(10)
exp (M(z) 0²(x))
2
EC exp (M((z) + (2(w))
3.2 The General Framework
The previous argumentation be applied general family of output activation functions: suppose the output activation is of the form n 40, where is the element-wiss application some activation function : R (0, oc). Then have (see Eq. (7))
h: X
((RC) L4 P((0, 9) -i P(AC-)).
(11)
Solving the one- me-dimensional integrals J& p(v) N(H, o (dy) (see Eq. (8)) allows to obtain the approximate predictive (see Eq. (10)
E EQ f(x)) Q
E Epsn(
P
(12)
Ec=l E Sq.mp.ttttt [Q.]
For instance, taking ip = ④ the standard Normal cumulative distribution function (normCDF)3, have by a classical result (Appendix A.2)
(vv) Ni (V, (dy)) ②
1
(13)
The resulting model distinct from multinomial probit model, Appendix B.

---

and thus
[(x)
(+)
Ep-h(x)
(14)
Ee=1 e==1
Me(m)
(1+
Taking instead ip = p. the logistic sigmoid, we approximate Eq. (13) using the probit approximation (Appendix A.3),
J p(u) N( ~
(15)
VI+ oo²
yielding in this case
m(x]
E Epwh(n)
(16)
Σ P
(e(w)
3.3 Theoretical Analysis of the Approximation's Quality
In Fig. 1, we see empirically that the quality of our predictive approximations outperforms exist- ing predictive approximations synthetic data. We explore the quality of our predictive approximations theoretically. We conduct the analysis for the case = p. Let
q = 4(1, o²) == Ee=aNinn.0))00
VI+ 2 =: 1(0, =
(17)
where á is the probit approximation (Appendix A.3). Moreover we write qe == g(Pt, o².), ge q(He, o?) where (Je: o²) the logit mean mean-variance pair for class c, p for the true predictive and ò for our approximate predictive, i.e.
40
= E (1 [P], D
(18)
Le qe
where has a as diagonal. To analyse the quality of the approximation, we thus want to bound DKL(p p). Clearly, when all the Qe are deterministic (i.e. constant random variables), where Q f(x) we have p = p. Alternatively, from the derivation Eq. (10), we see that the limiting case Eq. (12) is exact when Σe= Qe is deterministic. This a weaker assumption than requiring all Qe be deterministic individually. In fact,
derive rate of decay of DKL (p.(p.) p) in terms of
Var E Q.):
Theorem 3.1. Suppose the means and variances (Me, oZ) lie in some compuct set K. C R 00, oo) for each class e. Then
C Var
DKL. (p. S M(K) +0
Σ
Q.
(19)
c=l
as Var (Ie Q.) O and M(K) [0, oo] some constant which only depends K,, on C. such that M (K)- <0 when sup(p,0R)EX 2-9 and M = when q = q for all (p,o²) E K.
Appendix C.2 and Theorem C.2 provide a formula for M (KC) and proof. We also conduct an empirical investigation of whether the assumptions made in the theorem hold in practice, namely whether the logit means and variances lie a compact set, and whether M (AC) < co.
The term (XC in Eq. (19) results from the error due to the probit approximation, while the O term due to the splitting of the expectation into the numerator and denominator in the derivation Eq. (10). Importantly, Theorem 3.1 states that the former does not increase with the number of classes. For the
latter, requiring Var (EC, Qs) > 0 is weaker than requiring Var(Q.) - for all c (see Fq. (63)
in Appendix C.2); but we show in Appendix C.2 that assuming Var(Q.) - for all e gives almost twice strong decay rate in DKL. We include this additional result in Theorem C.2.
5

---

Distributional Approximations of the Gaussian Pushforwards
4.1 Dirichlet Matching
While in many applications, predictive probabilities are all that is needed from a model, we set out to build model that outputs tractable probability distribution the simplex h(x) E P(AC-1). As previously discussed, such a distribution allows, for instance, closed-form decompositions of aleatoric and epistemic uncertainties.
The formulation of Eq. (11) does not yet allow this. However, we will now see that one can obtain good Dirichlet approximations by constructing tractable approximation of Eq. (11) of the form
h:
G(RC)
a - + D(A
(20)
where D-A is the space of Dirichlet distributions on A--1 The family of Dirichlets presents an ideal choice of distributions over the simplex as, for such distributions, all quantities of interest are closed-form (see Appendix F). The map constructed by moment matching: we match the first
two moments of the Dirichlet distribution to the moments of the Gaussian pushforward sp ((x).
Conveniently, like the first moments described in Section 3, obtain second moments for exp
J exp(v)² N(u, o(,,o²) (dy) exp(2 2
(21)
and for d (Owen [43, Eq. 20,010.4))
J (1)) N(u, N(H, (dyy
2T
(22)
VV
VI+ VI+ 20
where 7' Owen's T function. For p. we have the following approximation (Daunizeau [10, Equation 23])
)
(1-P VI+zo=) (1-P(v=+)))
(
J plu)3 o(H, =
(23)
(1+
go
V1
This enables us to compute the second moment the pushforward distribution via
Q²
Eq f(z)) [Q2)
(24)
(Ee 2o)
Equ (e
De Qo
As in Eq. (12), the approximation in Eq. (24) is exact when Ee Qe is constant random variable.
If directly attempt to infer the parameters of the Dirichlet h(z) from Eqs. (12) and (24), we would obtain an overparametrised system of equations with 2C' equations and C' unknowns. Thus, instead use classical Dirichlet method of moments [38, Eqs. 19 & 23], which reduces these equations, as we will now describe.
The sum of the Dirichlet parameters Σg Ye may be estimated by
E P.P. E[P2)
(25)
E[P?] E[P.]²
for any 1 C. So, natural estimate of c= Ne that uses all P. is the geometric mean of Eq. (25). Since the mean the Dirichlet is 7/ 1=1 Yes we obtain using Eq. (25) an expression for the Dirichlet parameters
II
CC E
E P?]
1/C
E[P].
(26)
E[P3)
E.P.]
Hence, using Eqs. (12) and (24), we obtain the computationally feasible Dirichlet parameters
1/C
E [Q2. 5
E[
E
(27)
E[Q.]2
[

---

Figure 2: Approximating Gaussian pushfor- wards @. f(x) and p.f(ar through normCDF and respectively by Beta distributions p([(r)) with moment matching. With norm- CDF, can match exact moments, whereas with sigmoid, we match approximate mo- ments, Moreover, when f(xx) is the standard Normal (00, 1) (on the left in blue), the approximation for the pushforward through normCDF is exact (Appendix D.I).
2.((2)
p.(())
p(f(z))
p(f(a))
1.0
0.5
1.0
for h(x), where S max(E B[Qe),1). Taking the maximum with the expression for S an additional approximation to ensure 0. Note that by performing such moment matching to obtain Dirichlet parameters in Eq. (27), the mean of this Dirichlet matches precisely the approximate predictive Eq. (12). That is, the approximate predictivv of the exact distributional model (Eq. (11)) matches the exact predictive of the upproximate distributional model (Eq. (20)).
4.2 Intermediate Beta Matching
When = 4 or p. noting that their range is (0, 1), we can decompose the approximate distributional model (Eq. (20)) further:
h: X
G(R & B((O, 1)) 9 D(AC-1),
(28)
where (0, 1)) is the space of Beta distributions on (0, Like the output Dirichlets, obtain the intermediate Beta distributions by moment matching (Appendix D). Figure 2 indicates that such Beta approximations tend to be of high quality. In Appendix D.I, we give information- geometric motivation for this, which shows that the choice of activation is ideal for approximating the Gaussian pushforwards with Beta distributions. The quality of such Beta approximations encouraging for that of the output Dirichlet approximations, Analysing this theoretically presents an interesting direction for future research.
5
Training
With the choices o o p. could still train the underlying model with cross-entropy (CE) loss, replacing the softmax by the activation a = n or n o p. However, we can instead leverage the fact that & and p have values in (0, 1) to train using the binary cross-entropy (BCE) loss.
The BCE loss a valid choice of loss as it a (negative) strictly proper scoring rule (see Appendix I). Moreover, it has been observed to provide advantageous training dynamics compared to the CE loss [53, 28]. The choice of = or not only fits our framework for closed-form predictives, but allows one to leverage such advantagcous training dynamics (see experimental results in Section 7).
For example, in linearised Laplace and SNGP, neural network f: X' + RC is trained for the mean of f, while the covariance is obtained post-hoc. Thus, for these methods, we can train the mean function ff with the BCE loss
C((Ens Cn )N
Σ
1 log(v(f((())) + (1 5c - (ff((z)))
(29)
In contrast the previously mentioned methods, HET directly trains not only for the mean f but also for the covariance matrix Σ of f through the predictives. Thus, in this case, we use the BCE with the pushforward means Qe := E. (xxx)
C
C((In,C))
ΣΣ
(Be,, log(Q.) + (1 - 8c,,c) log(1 Qa))
(30)
n=
7

---

6
Computational Gains
The runtime cost of the MC predictive approximation is O(S C+ C9), and the memory cost is OSS . C+ C²), where S is the number of samples. The C2 term in memory comes from the logit covariance matrix; the C term in runtime comes from the need to Cholesky-decompon covariance matrix before sampling. For example, on a ResNet-50, sampling one thousand logit vectors from the logit-space Gaussian incurs approximately 7% overhead on the forward pass: see Appendix H for MC timing experiments. In embedded systems with much smaller models, this overhead is even larger. In contrast, the runtime and memory cost of calculating our predictives is OCC), which is negligible in comparison.
such
Indeed, the approximation Eq. (12) uses only the mean and the diagonal of the logit covariance to calculate the predictive. That is, the logit quantities required to compute a predictive scale as ((C) instead of (C²). This provides further opportunities for computational savings in the methods used to obtain the Gaussian distributions. E.g., in HET, the logit covariance is trained with both a diagonal term and low-rank term. Our framework thus allows dropping the low-rank term (Appendix J.2).
7 Experiments
We now investigate our two research questions:
Do we have to sacrifice performance for sample-free predictives? (Section 7.1)
What are the effects of changing the learning objective? (Section 7.2)
To answer the first question, we consider fixed (method, activation) pairs and verify that our methods perform on par with the Monte Carlo sampled predictives. In the main paper, we report results on Softmax (fo exp, trained with vanilla CE), NormCDF (p b and Sigmoid (p p) models. In Appendix K. we investigate the performance of Softmax models with closed-form predictives.
We consider Heteroscedastic Classifiers (HET) [7], Spectral- Normalised Gaussian Processes (SNGP) [31]. and last-layer Laplace-Approximated Networks [11] as backbones supplying logit-space Gaussian distributions. The resulting 21 (method, activation, predictive) triplets are evaluated on ImageNet- Ik [12], CIFAR-10, and CIFAR- 100 [26] on five metrics aligning with practical needs from uncertainty estimates [39]. Further, we test the Dustr--Distributtion (OOD) detection capabilities of the models on balanced mixtures of In-Distribution (ID) and OOD samples. For ImageNet, we treat ImageNet-CC [18] samples with 15 corruption types and 5 severity levels as OOD samples. For CIFAR-10 and C00, we use the CIFAR-10C corruptions.
Table I:
Comparison of ECE re- sults for different predictives us- ing a fixed Laplace backbone.
Method
Mean 2 stdl
Sigmoid Laplace
Closed Form
0.0101 1: 0.0006
MC 1000
0.0153 + 0.001
MC 100
0.0153 + 0.0012
10
1: 0.0014
NormCDF Laplace
For the second question, we evaluate our closed-form predictives and
MC 10
0.0112 1 0.0005
Closed-Form
0.0114 + 0.0007
Dirichlets against softmax models with the Laplace Bridge [19] and Mean Field [32] approximations, the linear-time approximation by Shekhovtsov and Flach [48], and
MC 100
0.0115 + 0.0006
MC 1000
0,0118 : 0.0007
MC sampling, see Section 7.2.
To provide a fair comparison, we reimplement each method as simple-to-use wrappers around deterministic backbones. For ImageNet evaluation, we use ResNet-50 backbone from the timm library [53] pretrained with the softmax activation function, and train each (method, activation) pair for 50 ImageNet- 1k epochs following Mucsányi et [40]. For Vision Transformer [14] experiments, refer to Appendix K.2. On CIFAR-10, 10, we train ResNet-28 models from scratch for 100 epochs. The CIFAR-100 experiments train 2ide models for 200 epochs. We search for ideal hyperparameters with a ten-step Bayesian Optimization scheme [47] in Weights & Biases [5].
The main paper focuses ImageNet results to highlight the scalability of our framework and only shows proper scoring results for CIFAR-10. For a complete overview CIFAR-10 and CIFAR-100 results, refer Appendix K.3 and Appendix K.4, respectively.
In all plots of the main paper and the Appendix, the error bars are calculated over five independent runs with different seeds and show the minimum and maximum metric value.
8

---

0.5
0.4
NLL
NLL
0.3
NoriCD
NeenC DF
NormCDF
Softma
Signokd
Signedi
Sufimak
Signed
Signeii
Noen
Sufima
Norm
Softmax
0.1
0.0
SNGP
Laplace
HET
Laplace
HET
SNGP
(a) Closed-form predictives are either on par with across all methods.
(b) Closed-form predictives consistently outper-
or outperform Softmax
form Softmax
across all methods.
Figure ImageNet (left) and CIFAR-IO (right) negative log likelihood results.
7.1 Quality of Sample-Free Predictives
In this section, we investigate whether there price to pay for sample-frre predictives and second- order distributions. To this end, we take the two best-performing methods on the Expected Calibra- tion Error (ECE) metric [42] -ormCDD and Sig- moid Laplace (see Fig. 4a) and evaluate their per- predictive performance, As Table shows, the closed-form predictive is always on par with the (unbiased) MC estimates on ImageNet while being cheaper. Appendix K.5 reports the KL divergence of different approximations to the 'true" predictive (estimated using 10,000 MC samples) on ImageNet. second-
Table Comparison OOD AUROC results ImageNet using severity level one for dif- ferent uncertainty estimators on the three best- performing (method, activation) pairs: Sig- moid Laplace, NormCDF HET, and NormCDF Laplace. The best estimator is derived from the motcnt-matched Dirichlet distributions in all three cases. MC estimates use 1000 samples.
Method
Mean = 2 std
Sigmoid Laplace
Table shows that the moment-matee order Dirichlet distributions consistently outperform both MC and sample-fre free predictives on the OOD detection task, showcasing the practical utility of second-order distributional knowledge.
Dirichlet Mutual Information
0.6388 0.0005
Closed-form Predictive Entropy
0.6353
0.0003
Monte Carlo Expected Entropy
0.6321
0.0001
NormCDF HET
Dirichlet Expected Entropy
0.6353
0.0021
Monte Carlo Expected Entropy
0.6281
0.0020
Closed form Predictive Entropy
0.6277
0.0021
7.2 Effects of Changing the Learning Objective
NormCDF Laplace
The previous section shows that for models trained with Sigmoid or NormCDF, our closed-form pre- dictives always perform par with MC sampling while being more efficient. However, our proposed objective (see Eq. (29)) changes the training dynamies and, subsequently, the performance of models. In this section, we showcase the perfor- mance of methods equipped with our closed-form predictives and learning objectives against Softmax models. For the latter, for an overview of predictive approximations and estimators). We employ our methods with the closed-form predictives and second-order Dirichlet distributions to demonstrate their competitive nature while being more efficient than MC sampling.
Dirichlet Expected Entropy
0.6321
0.0009
Monte Carlo Predictive Entropy
0.6296
0.0012
Closed Form Predictive Entropy
0.6296 0.0012
the best-performing predictive and estimator (see Appendices and F
We first evaluate how calibrated the models using the test NLL [17] and Expected Calibration Error (ECE) [42] metrics. Fig. 3b shows that on CIFAR-10, the NLL score of our closed- form predictives (Sigmoid, NormCDF) is consistently better than the corresponding Softmax results for all methods. On the large-scale ImageNet dataset, Sigmoid and NormCDF predictives either outperform par with Softmax (see Fig. 3a). The ECE metric requires the models' confidence (i.e., maximum class probability) to match their accuracy. Fig. 4a shows that on ImageNet, our closed-form predictives have clear advantage over Softmax across all methods.
Next, we turn to the correctness prediction task: whether the models can predict the correctness of their predictions. We consider correctness estimators tæx) maxigese Ep-h(a) [P.] E [0, 1] for inputs E X' derived from the predictive mean (Ep-h(=) [P]. Framed as binary prediction task, the goal of these estimators is to predict the probability the predicted class" correctness, defined v(x) [īx)) y] E {0, 1} with ((r) == arg maxiseso Epwa(z) [Pc] E {I....,C) C). We then

---

Namun CD
Softmax
Narui CDR
Softmax
Narni NarmChi
Log Prob 1
BCE
0.04
Sigmoid
Neverc
Soctmax
0.02
Sigmond
Soffmas
NormCD
Signoki
00.3
-0.4
0.00
-0.5
Laplace
HET
SNGP
SNGP
HET
(a) Closed-foree form predictives are more calibrated than Softmax across all methods. Note the re-
(b) Our closed- form predictives consistently out- perform Softmax on predicting their own correct- ness.
stricted y-limits for readability.
Figure 4: ImageNet ECE (left) and log probability proper scoring results for the binary correctness prediction task (right).
0.70
Accuracy
0.8
AUROC
0.65
0.6
0.60
Soltma
NoroCT3
Soltmas
NoroCTH
Signod
0.55
Stemad
NoroCDi
NoroCDPP
Sigmoid
Softmas
Sipmoid
Saftmax
0.0
0.50
SNGP
HET
Laplace
Laplace
HET
SNGP
(a) Closed-form predictives do sacrifice
(b) Closed-form predictives yield superior OOD
racy. Closed-form predictives either outperform or are on par with Softmax across all methods.
detection performance. Across all methods, the best- performing predictive is closed-form
Figure 5: ImageNet validation accuracy (left) and OOD detection AUROC results for ImageNet-C transforms of severity level one (right).
measure the binary log probability score of i(x),, given by v(x) (2) + (1 v(x)) log(1 f(x)). Fig. 4b shows that our dlsed-formm predictives outperform all Softmax predictives across all methods.
Closed-form predictives do not sacrifice accuracy. Fig. 5a evidences this claim on ImageNet: our closed-form predictives either outperform are on par with Softmax predictives. These results are in line with the findings of Wightman et al. [54], who also recommend training with per-class binary cross-entropy loss using the sigmoid activation function. Appendix K.6 shows that the BCE loss already yields performance gains in the model backbones (without the logit-space Gaussians).
Finally, we consider another binary prediction task, where general uncertainty estimator u(x) IR (derived from predictives or second-order Dirichlet distributions) is tasked to separate ID) and OOD samples from a balanced mixture thereof. As the uncertainty estimator can take on any real value, we measure the Area Under the Receiver Operating Characteristic curve (AUROC), which quantifies the separability of ID and OOD samples w.r.t. the uncertainty estimator. As OOD inputs, consider corrupted ImageNet-C [18] samples. Fig. 5b shows that our closed-form predictives outperform Softmax across all methods. Importantly, these closed-fome predictives and second-order Dirichlet distributions are considerably cheaper to calculate than Softmax MC predictions (see Section 6).
See Appendix K.7 for the other severity levels.
8 Conclusion and Limitations
We developed framework that allows for obtaining predictives and other quantities of interest from logit space Gaussian distributions closed form. Our experimental results suggest that the ubiquitous softmax activation should be replaced by normCDF sigmoid for uncertainty quantification tasks.
limitation is that our approximate predictives and approximate Dirichlet distributions not encode correlations between classes. While our theoretical analysis suggests strong bounds on the approximations, leveraging more expressive yet tractable second-order distributions on the simplex that can model correlations presents an interesting of future research.
10

---

Acknowledgments
The authors gratefully acknowledge co-funding by the European Union (ERC, ANUBIS, 101123955). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them. BM & PH are supported by the DFG through Project HE' 7114/6-1 in SPP2298/2. NDC is supported by the Fonds National de la Recherche, Luxembourg, Project 17917615. PH is member of the Machine Learning Cluster of Excellence, funded by the Deutsche Forschungsgemeinscaaff (DFG, German Research Foundation) under Germany's is Excellence Strategy- - EXC number 2064/1 Project number 390727645. The authors also gratefully acknowledge the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039A); and funds from the Ministry of Science, Research and Arts of the State of Baden- Maden-Württemberrg
References
[1] Achituve, A. Navon, Yemini, G. Chechik, and E. Fetaya. GP-Tree: A Gaussian Pro- cess Classifier for Few-Shot Incremental Learning. In Proceedings of the 38th Interna- tional Conference on Machine Learning, pages 54-65, PMLR, July 2021. URL https:
/ //proceedings. mlr gress/v139/cchiteiv
html. ISSN: 2640-3498.
[2] J. Atchison and S. Shen. Logistic-normal distributions: Some properties and uses. Biometrika, 67(2):261-272. Jan. 1980. ISSN 0006-3444. doi: 10.1093/biomec6672226... URL https: //doi. //doi.org/10. 1093/ htt////itmmm......
[3] N. Ay, J.J.Jost Jost, H. V. Lě, and L. Schwachhöfer. Information Geomelry, volume 64 of Ergebnisse der Mathematik und ihrer Grenzgebiete 34. Springer International Publishing, Cham, 2017. ISBN
978-3-319-564788 doi:
URL http:
//link.
sarinksrr.nn..... com/ 10. 09000009977300 3--------1-------
[4] V. Bengs, Hüllermeier, and W. Waegeman. Pitfalls of epistemic uncertainty quantification through loss minimisation. In S. Koyejo, S. Mohamed. A. Agar- wal. D. Belgrave, Cho. and A. Oh. editors, Advances in Neural Informa- tion Processing Systems, volume 35, pages 29205-29216. Curran Associates, Inc., /tile//paperr222///1ee
2022. URL. https: https///rroooeeediggg meweeeeeeddiinnnnnnne..ee..e. www..ps.cccppppp.e....ee be 0c16600fe11152669992222011111111 Par.aaaeee..e.aaaaaa-aooooeeeeee Conference,r. pdf.
[5] L. Biewald. Experiment tracking with weights and biases, 2020. URL httpps//wwwt wandb. com/. Software available from wandb.com.
[6] B. Charpentier, D. Zügner, and Günnemann. Posterior network: Uncertainty estimation with- out ood samples via density-based pseudo-counts. Advances in neural information processing systems, 111333366677 2020.
[7] M. Collier, Mustafa, E. Kokiopoulou, R. Jenatton, and Berent, Correlated input-dependent label noise in large-scale image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1551 1551-1560, 2021.
[8] N. Da Costa, B. Mucsányi, and P. Hennig. Geometric Gaussian Approximations of Probability Distributions. In F. Nielsen and F. Barbaresco, editors, Geometric Science of Information, pages 125-132, Cham, 2026. Springer Nature Switzerland. ISBN 978-3-032-03924-83 doi:
[9] C. Daganzo. Chapter 1 - An Introduction to Disaggregate Demand Modeling in the Trans- portation Field. Multinomial Probit, Economic Theory, Hconometrics, and Mathemati- cal Economics, pages 1-29. Academic Press, 1979. ISBN 978-0-12-2011555 150-4. doi: 10. 12-0001----1---11050----- arricle/pp1//9900200016666000000
URL httpss////ww..cceeeeereeettttt com/science/
[10] J. Daunizeau. Semi-analytical approximations to statistical moments of sigmoid and soft- max mappings normal variables, Mar. 2017. http:tt::/arxii htgg/abb/1733.. 00091. 10t::110000009 [stat].

---

[11] E. Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and P. Hennig. Laplace redux- -edux-effortless bayesian deep learning, Advances in Neural Information Processing Systems, 34:20089-2011 20103, 2021.
[12] J. Deng, W. Dong, R. Socher, L.-J. K. Li, and L. Fei-Fei. Imagenet: A large-scale scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 148-255, Icee, 2009.
[13] S. Depeweg, J.-M. Hernandez-Lobato, F. Doshi Velez, and S. Udluft. uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pages 1184- 1193. PMLR, 2018.
Decomposition of
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recogmition at scale. arXiv preprint arXiv:2010. 2020.
[15] T. Galy-Fajou, F. Wenzel, C. Donner, and M. Opper. Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, pages 755-765. PMLR. Aug. 2020. URL https://proceeddiggs mlr httpa///.......pooo/////////aa//
html. ISSN: 2640-3498.
[16] J. Gast and Roth. Lightweight Probabilistic Deep Networks, In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3369-3378, Salt Lake City, UT, June 2018. IEEE. ISBN 978-1-5386-6420-9) doi: (1009//VV..2018..00.00 //ieeexplore. ieee. www..e.eeeeeoo.e.eeeeeeeeee.eeee wwtp///oomet//////C /878453/
URL https:
[17] T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-378 2007.
[18] D. Hendryeks and T. Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. arXiv preprint arXiv: /2261, 2019.
[19] M. Hobbhahn, A. Kristiadi, and P. Hennig. Fast predictive uncertainty for classification with bayesian deep networks. In J. Cussens and K. Zhang, editors, Proceedings of the Thirty- -igght Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pages 822-832, PMLR, 01-05 Aug 2022. URL httpss::/prooceediiiiiee mlr. wwr.peees/ee http///8///ho////// htt//hoobhhhnn2aa. html.
[20] I. A. M. Huijben, W. Kool, M. B. Paulus, and R. J. G. van Sloun. A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2): 1371, Feb. 2023. ISSN 1939-3539, doi: 10.11///AAMM.2222..15557 9729603.
URL https://ieeexp/)
ieee org/ prg/detcment////
[21] E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning. 110(3):457-066 2021.
[22] E. Jang, S. Gu, and B. Poole. Categorical Reparameterizzation with Gumbel- Feb. 2017. URL https: /ttp:////p/eeeiieee//m nwt wwtp:/eeeeeettetteeeeeeeeeeeeeee fate//forrmm..err333855....
[23] M. Jürgens, N. Meinert, V. Bengs, E. Hüllermeier, and Waegeman. Is epistemic uncertainty faithfully represented by evidential deep learning methods?, 2024. URL https:t/s///at org/ abs/2402. 00122402..000....... 09056.
[24] S. Kapoor, W. J. Maddox, P. Izmailov, and A. G. Wilson. On uncertainty, tempering, and data augmentation in Bayesian classification. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, pages 18211-18225, Red Hook, NY, USA, Nov. 2022. Curran Associates Inc. ISBN 1---1-17387-18---1
[25] A. Kristiadi, M. Hein, and Hennig. Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks. In Proceedings of the 37th International Conference on Machine Learn- ing, pages 5436-5446. PMLR, Nov. 2020. URL httpst htttpp://prooeeeinggs mlr. a.t..press/v119 prisstadi220 html.
12

---

[26] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[27] A. Le Brigant, S. C. Preston, and S. Puechmorel. Fisher-Rao geometry of Dirichlet distributions. Differential Geometry and its Applications, 74:101702, Feb. 2021. ISSN 2926.2245. doi: 10. (pl6ii.difgeo.2020... 20100000000022 URL hwtps:tpp//// /tt////www// /www//wwaaoo scienceirreet.. wtt.s/ciiene/aariic.o.oooo pi1/$092622252200111x,
[28] Q. Li, Xiao, and L. Shen. BCE vs. CE in Deep Feature Learning, May 2025, URL http://arxiza wtw. aarrgg/bb2205555 05813. trXiv:2505.05813 [cs].
[29] R. Li, M. Klasson, A. Solin, and M. Trapp. Streamlining Prediction in Bayesian Deep Learning.
http///pperreeeeeeee/ Oct. 2024. URL htt.://pee.pppeeeeeeeee/e///////
[30] S. Linderman, M. J. Johnson, and R. P. Adams. Dependent Multinomial Models Made Easy: Stick-Breaking with the Polya-gamma Augmentation. In Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015. URL https: /tttps://pappr.... ctapps.cc/ppppee ppapprfflles////ppppppppppaeeeea ppaper__iles/pppprr/////////// /appr//011///aah/ Abstract. html.
[31] Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498-7512, 2020.
[32] Z. Lu, E. Ie, and F. Sha. Mean-Field Approximation to Gaussian-Softmax Integral with Application to Uncertainty Estimation, May 2021. URL http:://axxi..oggabbs////// 07584. arXiv:2006.07584
[33] D. J. MacKay. Choice of Basis for Laplace Approximation. Machine Learning, 33(1):77-86, Oct 1998. ISSN 1573-0565, doi: 10. 00.1023/A41005556115333 URL https:////ddo htttps://doi.orr//10.. 1023/A: A: 100155560175551111
[34] D. J. C. MacKay. The Evidence Framework Applied to Classification Networks. Neural Computation, 4(5):720-736, Sept. 1992. ISSN 0899-7667. doi: 10.1162/neco. 1992.4...99922 URL https: / /ttpe/eeeeeplreeeee/eeee ieee. org /ttp///cmmet//779999.... Conference Name: Neural Computation.
[35] C. J. Maddison, D. Tarlow, and T. Minka. Alsst Sampling. In Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
URL attps://papeett weraperssipss........ wwwsinccsspaaaaeeeee ht/ppaper_fiiie ceter_ffles/ppppeee http:/pppp//////////// 33776007910046677667711111333 Abstracttttt.....oeo html.
[36] C. J. Maddison, A. Mnih, and W. Teh. The Concrete Distribution: A Continuous Relaxation of Diserete Random Variables. Feb. 2017. URL http:///////eeeeeeeiieee. iet/fo $1 $155L1.1.
[37] J. Martens and R. Grosse. Optimizing neural networks with mooneckerffcctreee approximate curvature. In International conference on machine learning, pages 2408 2407- PMLR, 2015.
[38] T. P. Minka. Estimating a Dirichlet distribution, 2000. URL https: :ttppp::///ain //ttinkaa....//// httt///iia.giiiitiia github. io/ papprr//dariccllee/mmaaaa-d-ddid
pdf. Technical Report, MIT.
[39] B. Mucsányi, M. Kirchhof, F. Nguyen, A. Rubinstein, and S. Oh. Trustworthy machine learning. arXiv preprint 20.00:23100082211 2023.
[40] B. Mucsányi, M. Kirchhof, and S. J. Oh. Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. In The Thirty-eight Conference on Neu- ral Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: /
[41] J. Mukhoti, A. Kirsch, J. van Amersfoort, P. H. Torr, and Y. Gal. Deep deterministic uncertainty: new simple baseline. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 13384-24394, 2023.
13

---

[42] M. P. Naeini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
[43] D. B. Owen. A table of normal integrals. Communications in Statistics Simulation and Computation, Jan. 1980. URL https::/tsee www.f.tannnnnnnnn ccwm//oi//t/ /tt///b///// /0t//1001110//// 03610918008812168
[44] D. S. Pandey and Q. Yu. Learn to accumulate evidence from all training samples: theory and practice. In International Conference on Machine Learning, pages 26963-26989. PMLR, 2023.
[45] W. F. Reynolds. Hyperbolic Geometry on Hyperboloid. The American Mathematicaal Monthly, 100(5):442-455, May 1993. ISSN 0002-9890, doi: 10. 00110000000989900099111190000009 https://doi.
URL
1080 /00.000000020000... 1993. 11990430.
[46] M. Sensoy, L.. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. Advances in neural information processing systems, 31, 2018.
[47] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1):148-175, 2015.
[48] A. Shekhovtsov and B. Flach. Feed-forward Propagation in Probabilistic Neural Networks with Categorical and Max Layers. Sept. 2018. URL https: http//pppnneeeeeeeeeeeee net/forummidd SkMuPjRcKQ.
[49] D. J. Spiegelhalter and S. L. Lauritzen. Sequential updating of conditional probabilities on directed graphical structures. Networks, 20(5):579-605 1990. ISSN 1097-0037. doi: w..1022/net.3230200055 URL net.3230200500
witiorrrrrrlllyyl. com/ www.coimccc/cc/cccccccooocc..o.o 10. 1002/
[50] L. Tatzel, B. Mucsányi, O. Hackel, and P. Hennig. Debiasing mini-batch quadratics for applica- tions deep learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https: https://opents //tp:://ve//eveeee/eeeeom net netffoorrumId--DTTVVVV2
[51] M. Titsias RC AUEB. One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016. URL https:: wtwp://oooo.rriiiieeeeeerriepape neurips. httml.
[52] D. Tran, J. Liu, M. W. Dusenberry, D. Phan, M. Collier, J. Ren, K. Han, Z. Wang, Z. Mariet, H. Hu, et Plex: Towards reliability using pretrained large model extensions. arXiv preprint arXin:2207.07411, 2022.
[53] R. pytorch- image-modest 2019.
Wightman.
Pytorch image models.
htttp:://github..oommmm
[54] R. Wightman, H. Touvron, and H. Jégou. ResNet strikes back: An improved training procedure in timm, Oct. 2021. URL http: // arxiv. org/ abs/2110.00476. 2xXiv:2110.00470 [cs].
[55] L. Wimmer, Y. Sale, P. Hofman, B. Bischl, and E. Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty Artificial Intelligence, pages 2282-2292. PMLR, 2023.
[56] O. Wright, Y. Nakahira, and M. F. Moura. An Analytic Solution to Covariance Propagation in Neural Networks. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, pages 4087-4095. PMLR, Apr. 2024. URL https: http://///oceedeeeeeeeeee mlr.
press/ hatsss/v23///n/ 11338//right 2wright24a.... html. ISSN: 2640-3498.
[57] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. and C.-J. Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.000 00962, 2019.
Song, J. Demmel, K. Keutzer,
14

---

A Gaussian Integral Derivations
17
A.I Gaussian Exp Integral
17
A.2 Gaussian NormCDF Integral
17
A.3 Probit Approximation for Gaussian Sigmoid Integral
17
A.4 Mean Field Approximation for Gaussian Softmax Integral
18
Comparison with the Multinomial Probit Model
18
Theoretical Analyses
19
C.I Informal Analysis of Monte Carlo Approximations
19
C.2 Analysis of the Closed-Form Approximations (Theorem 3.1)
20
Moment Matching Beta Distributions
24
D.I Information Geometric Interpretation of the Pushforward through NormCDF
25
List of Predictive and Dirichlet Parameters Formulas
26
E.1
Predictive Formulas..
26
E.2
Dirichlet Parameters Formulas
27
List of Uncertainty Estimators
28
F.1
Predictive
28
F2 Monte Carlo
28
F3 Dirichlet
28
G Experimental Setup
29
Runtime Overhead of Monte Carlo Sampling
30
I
Benchmark Metrics
30
LI
Log Probability Proper Scoring Rule for the Predictive
1.2
Expected Calibration Error
31
L.3
Binary Log Probability Proper Scoring Rule for Correctness Prediction
31
1.4
Accuracy
32
1.5
Area Under the Receiver Operating Characteristic Curve for Out-of Distribution Detection
32
J Benchmarked Methods
32
J.1 Spectral Normalised Gaussian Process
33
J.2 Heteroscedastic Classifier..
33
J.3 Laplace Approximation
33
K Additional Results
34
K.1 Closed-Form Softmax Predictives
34
K.2 Vision Transformer Results ImageNet
34
15

---

K.3
CIFAR-10 Results
35
K.4
CIFAR-100 Results
................................
37
K.5
Alignment with the True Predictives
................................
37
K.6
BCE Loss Performance Gains
38
K.7
Further Out-of-Distribttion Detection Results
38
K.8
Comparison Evidential Deep Learning Methods
38
16

---

A Gaussian Integral Derivations
In this appendix, we derive the closed-form formula for the mean of Gaussian pushforwards through exp (Eq. (8)) and normCDF (Eq. (13)), as well as the approximations for pushforwards through sigmoid (Eq. (15)) and softmax.
A.1 Gaussian Exp Integral
By absorbing the exponential into the Gaussian probability density function, we get
/
(y
12)
exp(y) (H, (d) =
exp(y) exp
dy
20
R
o²
exp
11
+
exp
-202(00 - -p- -p o2)))
20 (y
(U-
dy
2
R
(31)
0²
N(p + 0², (d)
exp
H +
2
0²
exp
1+
2
A.2 Gaussian NormCDF Integral
Here, we derive the classical normCDF Gaussian integration formula [43, Eq. 10,010.8).
For > 0, Z ~ (0, 1) and ~ NN,, 0²) with Y and Z independent,
/ ((xy) N ((H, / p(z S w/A) NiH, (dy)
= Y
x
(32)
7//2 - Y +
p
VA-2
+ -
where we used 2/A-X+# ~ NO, 1) for the last equality. Taking À = 1, this gives the formula for -2==+o" the exact predictive with normCDF.
A.3 Probit Approximation for Gaussian Sigmoid Integral
Taylor expanding p to first order about 0,
p(y) = + 7 + o(y),
2
(33)
1 1
(v)) 2
V2n y + oly).
as y + O. Hence matching p and & to first order we get the approximation p()) $ (v So using Eq. (32) derive the probit approximation
[ p()) Ni, (dv))
(1)
{ Φ B
Th (
N (3) -b
P
(34)
Note that the approximation (2) is not strictly needed, as + is computationally tractable. However, adding (2) empirically improves the quality of the overall approximation. This may be due to the fact the thicker tails of p in the integrand of the left-hand side are better captured by p than on the right-hand side.
17

---

A.4 Mean Field Approximation for Gaussian Softmax Integral
For μl E RC, o² E RC and Σ diag(o²)) the mean field approximation to the Gaussian softmax
integral [32] is obtained follows
Ey [solmmxx (sflmmxx Y =
2 C+ + Σ
-
p(Y, - Y2)-1
d'Fc
2 C+ Σ
Elp(Yo - Yo)1
c'sc
2
2 C + Σ
Elp(Y. - peel
(35)
C'sc
2 2-C+ Σn P
He He
V1+ o
softmax,
VI+ T o
i...
Wy. YJ softmax
(36)
V1+ _
(1) is the mean field approximation, and (3) uses the probit approximation Eq. (34). [32] provides two other variants of this approximation with other choices of approximation (2).
B Comparison with the Multinomial Probit Model
In this appendix, we show that a model whose output activation a composition an element- normCDF activation & and normalisation is distinct from the classical multinomial probit model [9].
Given logit y E RC, the multinomial probit model sets
( arg max Y: 13e5C'
(37)
where Ye ye + c. and the e, are i..d. standard Normal. So
p(c v) = p(ye + > + car V ( +
(38)
which is generally not analytically tractable. On the other hand, model that uses normCDF and normalisation as output activation yields
plye + Eo > 0)
ple v)
(39)
Σe plye f > 0)
De_ v(v2)
In the case C= 2, the multinomial probit model Eq. (38) outputs closed-form probabilities. This allows us construct explicit counterexample the equivalence of the two models Eq. (38) and Eq. (39):
p(y) +e > 1/2 + 62) =p
-
El
E2
0)
911
922
+
( $ ( ) + (
2
2
(40)
18

---

C Theoretical Analyses
In this appendix, we provide (formal and informal) theoretical analyses of the quality of various predictive approximations. This complements the empirical analyses, for instance the synthetic experiment (Fig. 1) in [10].
Due to its information theoretic interpretation, natural divergence to equip the probability simplex AC C-11 with is the Kullback-Leible (KL) divergence
(pL q) Pc log
(41)
which is well defined if q lie in the interior of the simplex (pi, * 0, for all i). So, for predictive approximation, p. we would like to analyse (pI,(p. p), where p := Epsa.Niu. P] is the true predictive, and E are some logit space mean and covariance and is output activation (e.g. a n yp).
C.1 Informal Analysis of Monte Carlo Approximations
An N sample Monte Carlo estimate is defined as
s
ps = ((0)
(42)
where the p(s) are i.i.d. a,, Σ). The computational cost of MC integration is O(S C). This becomes prohibitive for large S and C. Thus, for fair assessment the quality of such an estimate terms of the number of classes, one should consider MC estimates P1S/01,
We now give an informal theoretical argument for the linear growth of the KL divergence between and P S/C] in terms of C, under the distributional conditions of the synthetic experiment (Fig. 1).
Taylor expanding Eq. (41) about p to second order obtain
DKL(P, q) =
Pe Ploog Pc log qe)
(
Pe
(Pe q0)²
Pe
1
+
qe
2p
C
(pc 20)²
Pe
qe
+
(43)
2pc
(Pe
9c)
2pc
C
where approximation (1) assumes |p a/|a is small, and (2) assumes pe. R 1/C.
In the synthetic experiment (Fig. 1), the logit class-wise means μa and variances a? sampled in an i.i.d. way. Let Q ~ Nuu, diag(a")) the unnormalised 'probabilities' and = Q/ D Qo the probabilities. We have
Q²
EQQ
E[Q
Var [Q]
Var[P] = E
(44)
[ 2o)
C
Qc
DC B(Qd)
2
C
19

---

where all operations are taken element-wise, and E[Qa] = B[Q1] follows from the fact that the Po and of are i.i.d. Thus
Var Q2c] C
Var[Q1]
E p
P
Var PPl 2
(45)
C
Now the MC samples p(s) are i..dd. copies of P. So have
BiIp- p 8/c| 1
Var[Q1] STE| [
[S/C|
E
p
E
(46)
where the outer expectation is over the (third-order) distributions over P. Plugging this into Eq. (43)
we get
Var 1
C
(p, )(
~ E
(47)
2 Q1) 5
which grows linearly with the number of classes, observed in Fig.
C.2 Analysis of the Closed-Form Approximations (Theorem 3.1)
In this section we extend and prove Theorem 3.1, and discuss whether its underlying assumptions are fulfilled in practice. We start with lemma.
Lemma C.1. Let y be family of (0, -valued random variables such that supy E -1 < for all k > Then for X, Y (0, -vvvuuedd random variables with Y E y and any e 0,
co
E
E
0 (Var(X)=- + Var(Y)²-c)
(48)
Y
Var(Y) or Var(X), Var(Y) - O.
Proof. Let X,Y be (0, -valuee random variables. We take bivariate Taylor expansion of yx for the random variables and around E[X] and [E[] respectively:
X
X
1 E
E[X]
(X- E[X])
(Y IE(T))
E[y]
E
(49)
{(Y)² (X E( - + n(X)
(Y - E(y))²
{(Y)3
where (X) E [E[X], .X] or [X E[X]] and ( e [E], Y] or [Y, E[Y]], using the Lagrange form of the remainder. So by the multivariate Hölder inequality we have for any 0 < e 1/2.
X
X
(X)
E
(X B[X])( ELY))
+
(Y B[Y1)
E(Y)2
[(Y)3
S E
FI(( B(X])
-
B
(50)
1/X
+ E
B[(Y -[YT)
E(Y)3/C
Note that
[X - E[X]) 1-200 S the BI(X [[X])* = M -e Var(X),
1/ < 2//e
1
1
sup E Yey
< sup E YEY
S sup E YEY
< < for 32/e,
EE(Y
min(( [Y
yk
E
(51)
So from Eq. (50) obtain
X
E
= 0 = + Var(Y))-o)
E
(52)
as Var(Y) or Var(X), Var(Y) + O.
20

---

Now recall our notation
1 VI (02 = H(,, 02) =
== Ee-pNN R
(53)
where à is the probit approximation (Appendix A.3). Moreover we write g. == g(pes o²., q.
Ü(He, o²), for the true predictive, P for our approximate predictive, and in addition p for the approximate predictive using the exact one dimensional integrals, i.e.
= Ep Ni. D) [P],
P =
9c
P
qe
(54)
Eo= 4e
Σ Ee=1
9d
where Σ has as diagonal. We now restate Theorem 3.1 with explicit expression for M (C) and an additional decay result for the O term:
Theorem C.2. Suppose the means and variances (Me, o?) lie in some compact set K. C TR 00 oo)
for each class c. Using the compactness of K, define
8(K) sup (u (9 9).
u(C)) := inf (n,0)) q >
((K) sup (E. 2-1
If (1C) > 1 then
1
DKL (p. s loo
+0
(55)
1-A
where
O 0 Var Σ Q.
Var
Σ Qc +0
(56)
and
0 = O
max Var Var(Qc)2-g Isesc
vE>0
as = C.
(57)
Proof. We have
DKI. (p.
Σ
Poog.e log Pr)
Pe Pollogpe log +
Pa Do. - log(e) (58)
(1)
(2)
Assuming A < 1, we can bound (2):
ge
qe
qe
(2)
log
log
Σ ae
E Ee= de
of
qd
0
Σ 90
- log
qe
+ log
-d'=1 ae
1==1 de c=1
Ee'=1 4e'
Je qe
D m1 90 - De qe'
-00=100))
))
c== le c=l
4n
log
1
de
+ log
1
Σd 9c
(59)
[ C8
DC 20=1 ge
40
log
A) + +log
Cu
1 1+8/u
ao log
E 9e o=1
1-A
1+ 0+1/u
log
7-A
21

---

Now for (I), first note that E (Ea 2o)
< oo where' Qe ~ P.N of) and &s > 0. By
compactness of K. we have fact sup((, E (E= (EE=)0)) 00 00 for any k > 0. Thus we
can apply Theorem C.I with y (<E On (1, ( to get that for any E 0.
|pc Del fel= 00 Var(QQ))
E9e
Qei
+ Var
Eee
(60)
c'=1
c'=1
as Var Ee' 2a) + or Var(Qe), Var (Z0=1 2c²)
- Thus Taylor expanding each term
of (1) around Pe: we get
(Pe - 70)2
(1) Σ Pe
Pc - De
Pe
2wo(Fo)²
C
(Po :2002
Pe
Pet
Pe 2w 2wo( (Dc)
c=1
1
c
c
Pe
Var(Qu)
Σ
Qe
+ Var
Σ
Qe
2wo (pe)²
r'=l
Σ
(ar(Q.)=-224v))
Σ Qel
(61)
2wo (pc)²
c'==1
_
2-2c
+ Var(Q.) -
E
0c
Var
Σ
Qe
e'=1
C
C
Var(Q.)1 -2r
Σ
Qe
d'=1
2-2c
Var(Q.)= Var
Σ
Qu
+ Var
Eee
Qe
where we (pe) E [Pe, Pe] Pel. where we used that Pes Per and hence we (Fe) bounded away
from 0 by compactness of K.. We now distinguish two cases. If we only assume Var then we see from Eq. (61) that
(DC 2o)
0o.))
(1) 0 Var Σ
(62)
If make the stronger assumption Var(Q.) + for all c, then using the inequality
c
Var Eec Σ Cov(Qa, 2c1)
Σ
Var(Qe,) 122
(63)
we see that Var
[EC Q.) =0 (maxissee Var(Q.)) and hence, from Eq. (61),
(1) O max Var(Q.)2-26 ISe5C
(64)
and the result follows since e is arbitrary.
IIn the case = + and Qe ~ ( oZ), due to the fast decay of the tail of t we may have
[E Q.) -k7 and the proof strategy fails in that case.
E
22

---

Sigmoid HET model CIFAR- 10
Sigmoid HET model CIFAR-IA 100
All predictions
All predictions
0.06
Correct predictions
0.09
Correct prodictions
0.05
0.08
0.04
0.07
0.03
0.06
0.02
0.05
0.01
0.04
-10
5
10.0
-7.5
-5.0
22.5
0.0
2.5
Sigmoid SNGP model on CIFAR-10 10
Sigmoid SNGP model on CIFAR- 100
predictions
0.20
predictions Correct predictions
0.08
Correct prodictions
0.15
0.06
0.10
0.04
0.05
0.02
10.0
-7.5
-5.0
2.5
0.0
2.5
5.0
Figure C.I: Scatter plots of logit mean and variance pairs (Je: o²) on real models. We see that these seem to remain constrained to compact set, with no much difference in the support as we increase the number of classes by an order of magnitude (from CIFAR- 10 to CIFAR-100),
weaker than requiring Var(Q.) - for all but see in Theorem C.2 that assuming the latter gives almost twice strong decay rate in DKL- This means that neither Eq. (56) nor Eq. (57) can proven from the other; both results are thus of interest.
Equation (63) shows that requiring Var (Ie- Q.)
key assumption in Theorem C.2 is the logit means and variances being restricted compact set. In Fig. C.1 we see empirically that this is approximately the case for HET and SNGP models, independently of the number of classes.
Theorem C.2 is of practical value ( == log
1+8/u T-
is well-behaved:
(MC) is independent of C₁
2. ((K) + 0 as 5 + 0.
In other words, given knowledge of the worst case error in the approximation Eq. (15) on the compact set K, we can bound the KI divergence terms of that error independently of the number classes. Due to the simplicity of our assumptions, the bound remains quite raw and could be strengthened with further distributional assumptions on the means and variances.
Finally, to obtain meaningful bound in Theorem C.2, we needed to assume A(K) = SUK( q-4 < 1. In Fig, C.2 we see that this can be assumed to hold on the compact sets on which logit means and variances tend to live (see also Fig. C.I).
23

---

True Predictive q
Probit Approximation
0.0
0.20
020
0.20
0.75
0.75
00
0.15
0.15
0.15
0.50
8
0.50
-02
0.10
0.10
0.10
0.25
003
0.05
0.05
0.05
0.00
-0.4
J
Figure C.2: Plot of the 'true' predictive (approximated with a 50000 sample MC approximation) versus the probit approximation q. well as i= We see that 9-2 tends to be negative, making the assumption (ee 2-4 < from Theorem C.2 easily fulfilled.
Note that, in the proof of Theorem C.2, we did not use the form of the probit approximation explicitly, and the theorem holds for any approximation scheme the dimensional integrals q.
D Moment Matching Beta Distributions
As noted in Section 4.2, when & 4 or p, construct mapping
p: G(RC) - B((),
(65)
by moment matching. Specifically, the parameters a, B E (0, o0) that match the moments of Q fp,f for some f E ((RC) must satisfy
E[Q] a + 3'
(66)
ala 1 (a + () + B 1)
B E
where all vector operations element-wise Multiplying the denominators the right-hand side of the equations of Eq. (66), we obtain a system of two linear equations with two unknowns (for each c), which can be solved uniquely, yielding
E[Q
E|Q²)
Q
E[Qa) - E[
B[Q],
(67)
E[Q [[
3
(1 E(Q).
E
E[Q 2
which give the parameters of the Beta distributions pff).
24

---

D.1 Information Geometric Interpretation of the Pushforward through NormCDF
NormCDF +
Sigmoid P
(.
$.9(R)
B((0,1) 1))
B))),)))
Figure D.I: Illustration of the statistical manifolds of the pushforward Gaussian distributions (R) through the normCDF and the sigmoid respectively compared to the statistical manifold of Beta distributions (0, 1)). NormCDF, unlike the sigmoid, makes the manifolds intersect at the point $,N((,, 1) = 1),
Here, we argue that the normCDF activation is an ideal choice for approximating Gaussian push- forwards with Beta distributions by interpreting Fig. D.I. This extends the work from [33], as shows how one can make sense of the 'right' basis for performing Laplace approximations in the classification setting, Also see the discussion in [8].
(.)( (R) the space of pushforwards of Gaussian distributions by normCDF and sigmoid respectively, and (((0, 1)). the space of Beta distributions, statistical manifolds naturally equipped with Riemannian metrics, that is their respective Fisher information metrics. We would like to visualise these manifolds. However, two difficulties arise.
The first difficulty is that, while these manifolds all lie in the infinite-dimensione vector space of signed mcasures on the open unit interval M((0, 1)), there is subspace V C (((0,1) 1)) which is 3-dimensional (V R3) and contains any two of these statistical manifolds (R.G(R) B(O 1)) C V or (R) B (0. 1)) C V). We will work around this by building distinct isometric embeddings (. - V. (R)(R) + V and (((0, 1)) + V for some 3-dimensional vector space V. This means that while the shape of the manifold illustrations is meaningful, the positioning of manifold with respect to another is not, apart from some design choices that we describe below.
The second difficulty is that some-if not all-of these manifolds cannot be embedded isometrically into Fuclidean space. As a workaround, we instead embed them into the 3-dimensional Minkowski space 22211 that is R-3 equipped with the pseudo- -ieman-nieannnian metric dz? + dx3 dxž.
The key observation is that (RR) and (I) are isometric to (I3). This because & and p are diffeomorphisms, so in particular sufficient statistics, and the Fisher information metric is invariant under sufficient statistics [3, Section 5.1.4]. Visually, this means that (I has the same shape irrespectively of the diffeomorphism activation function jp. One can thus observe that, given that ([(0, 1)) is not isometric to (IR), there exists no activation op such that (.)((R = (((0,)) To design an activation tp that maps Gaussians to Betas, the best one can hope to do is to map one specific Gaussian distribution Nu,, o²) to specific Beta distribution B(a, 33). This done with the map F T fa.0= where pu.02 and Fa,9 are the cumulative distribution functions of N( o²) and B(a, 33) respectively. Taking μ = 0, o² 1, = 3 = we get T = + and Fi, = id
yielding F=. 4b n. +.
Now (TR), and hence $. (R) and p. (R), ), is isometric to the hyperbolic plane Ay et al. [3, Example 3. We can embed this isometrically into Minkowski space with the classical hyperboloid model of the hyperbolic plane [45].
G(R)
2.1
(68)
For B((0, 1)), we use the isometric embedding from Le Brigant et al. [27, Proposition 2]:
B((0, 1)) 2
(69)
(a,, B) + (r)(), + 8))
25

---

where 7(a) : S (r) dr and vb is the digamma function.
Finally, we choose our embedding Eq. (68) for 2. (IR) such that intersects the embedding Eq. (69) at point, to highlight that the statistical manifolds & P.gg ((R) and [ 1)) intersect point in the infinitedimmensionall ambient space M( 1)), while (.) and (((0,1) do not.
Moreover, since @o,N(0, 1) B(1, 1), our moment matching approximation Eq. (66) is exact when f(æ) the standard Normal distribution.
E List of Predictive and Dirichlet Parameters Formulas
E.1 Predictive Formulas
We gather formulas for all predictive estimators p the true predictive E. F E diag(o"), used in our experiments (Section 7).
E.1.1 Softmax
Monte Carlo
One can Monte Carlo estimate the true predictive as follows:
((*)
(70)
where the p(*) are sampled i.i.d. from ((, E).
Mean Field (Appendix A.4) The Mean Field predictive [32] uses the following approximation for the true predictive:
P
p softmax
(71)
-I+
Laplace Bridge
The Laplace Bridge predictive [19] approximates the true predictive follows:
2
-he Re Ee=1 e-A Bd.)
=c=
(72)
c==
de
1
et +
C
where
VC12
VC12
:
VC VI
C
(73)
Shekhovtsov & Flach. integral yields:
The Shekhovtsov and Flach [48] approximation of the Gaussian-softmax
D
(74)
n + (A- - b) 8/#
where
A Σ exp(Ao/s) == exp((ss)) b: exp((((/s))
(75)
:= 14 max, for numerical stability,
2 == 2 G, 2 min 72 V
2 2
2
(76)
and === G L/2.
26

---

E.1.2 Exp
Our closed-fome predictive for the exp activation function (Section 3.1) is given by
exp
11 + 2
p
(77)
a 2 2e=1 exp Mo +
E.1.3
NormCDF
The closed-form predictive for the normCDF activation function (Section 3.2) computed
1 vI+a"
(78)
Ees
Me VI+o?
E.1.4
Sigmoid
For the sigmoid activation function (Section 3.2), the closed-form predictive can be computed as
(79)
DC
Dc V1+
E.2
Dirichlet Parameters Formulas
We now gather the formulas of the parameters y for the Dirichlet approximations to the Gaussian pushforwards.
E.2.1
Softmax
The Laplace Bridge method [19] uses the following Dirichlet parameters:
C
2
Y =
Ac
(80)
72
C
where
VC12
VC/2
Φ²
(81)
C
Ec=1
- c=1
o
E.2.2
Exp
Exp uses the closed-form parameters derived from Moment Matching the Gaussian pushforwards (Section 4.1):
C
1/C
7 =
II
0g
max
2c=1 ae' 1) b
(82)
b a3
Ee an
where
σ²
exp
b exp(22x+ 20²2).
(83)
2
E.2.3 NormCDF
NormCDF uses the closed-form parameters derived from Moment Matching the Gaussian pushfor- wards (Section 4.1):
1/C
C 0. max C
ac, 1) be
7 ==
I1
b,
Σ
C
ag
(84)
27

---

where
1
b
- 2T
(85)
VI+++* 0²
(II 0z
VI+ o²
vI+
20
E,2.4 Sigmoid
Sigmoid uses the closed-form parameters derived from Moment Matching the Gaussian pushforwards (Section 4.1):
1/C
C
0g max Ee= ac, 1)
be
7 =
(86)
be
ag
Ec ar
where
a
2²
(87)
1
(1-P)
b = P
1 0
1
o
p
VI+ 30
(1=P'v
[1+ o
F List of Uncertainty Estimators
In this section, we list the uncertainty estimators used in our experiments (Section 7),
F1 Predictive
Given a predictive p, we consider two uncertainty estimators.
Maximum Probability
arg max Pe- oE (I....))
(88)
Entropy
Pe log Pe-
(89)
F.2
Monte Carlo
Given S Monte Carlo samples
(())... ...... (S) with mean p. one can calculate predictive as their average and derive the estimators in Appendix F.I. However, Monte Carlo samples allow one to calculate two additional estimators detailed below.
Expected Entropy
Pe log lo
(90)
S
Mutual Infornattin/Jensen-Shhnnoo Divergence
8
C
De log P. +
3
p(a) log
(91)
S
==1
F.3 Dirichlet
Given second-order Dirichlet distribution with parameters 7, obtain the expected entropy and mutual information estimators without the need for Monte Carlo samples.
Expected Entropy
C [ Yot (ro+11) c'=1
+1))
V C
+ (T00+1)=
(92)
%
28

---

where & is the digamma function.
Mutual Information
%
(1lg yo + log
C Σ c'=1
10 (5
Ye
+
1)
-
(93)
c'== Te
c'==1
G Experimental Setup
This section describes our experimental setup in detail.
We have two main research questions:
What the effects of changing the learning objective?
Do we have sacrifice performance for sample-free predictives?
To answer the first question, we evaluate our closed-form predictives (Sigmoid, NormCDE, Exp) and moment-matched Dirichlet distributions against softmax models equipped with approximate inference tools (Laplace Bridge [19]. Mcan Field [32], Monte Carlo sampling). We consider Heteroscedastic
Classifiers (HET) [7]. Spectral-Normaliser Gaussian Processes (SNGP) [31], and last-layer Laplace approximation methods [11] as backbones (see Appendix J for details).
The resulting 21 (method, activation, predictive) triplets are evaluated ImageNet-1k [12] and CIFAR-10 [26] on five metrics aligning with practical needs from uncertainty estimates [39]]
1. Log probability proper scoring rule for the predictive,
2. Expected calibration error of the predictive's maximum- maximum-probability probability confidence,
3. Binary log probability proper scoring rule for the correctness prediction task,
4. Accuracy of the predictive's argmax.
5. AUROC for the out-of-distribution (OOD) detection task.
See Appendix for details.
For ImageNet, we treat ImageNet-C [18] samples with 15 corruption types and 5 severity levels as OOD samples. For CIFAR-10, 10, we use the CIFAR-10C corruptions.
For the second question, we consider fixed (method, activation) pairs and test whether our methods perform on par with the Monte Carlo sampled predictives.
To provide a fair comparison, we reimplement each method as simple-to-use wrappers around deterministic backbones.
For ImageNet evaluation, we use a ResNet-50 backbone pretrained with the softmax activation function, and train each (method, activation) pair for 50 ImageNet-Ik epochs following Mucsányi al. [40]. We train with the LAMB optimiser [57] using batch size of 128 and gradient accumulation across 16 batches, resulting in an effective batch size of 2048, following Tran et al. [52]. We further use cosine learning rate schedule with a single warmup epoch using a warmup learning rate of 0.0001. The learning rate is treated as a hyperparameter and selected from the interval [0.0005, 0.05) based on the validation performance. The weight decay selected from the set {0.01, 0.02). During training, we keep track of the best-performing checkpoint on the validation set and load it before testing. We search for ideal hyperparameters with a ten-step Bayesian Optimization scheme [47] in Weights & Biases [5] based on the negative log-likelihood.
On CIFAR-10, we train ResNet-28 models from scratch for 100 epochs. The only exceptions are the SNGP models that are trained for 125 epochs [31]. We train with Momentum SGD using batch size of 128 and no gradient accumulation. Similarly to ImageNet, we use a cosine learning rate schedule but with five warmup epochs and warmup learning rate 1c-5. The learning rate also treated as hyperparameter on CIFAR-I0. We use the interval [0.05, 1] for Sigmoid and NormCDF, and [0.01, 0.15) for Softmax and Exp. The optimal learning rates for Sigmoid and NormCDF are generally larger, as the class-wise binary cross-entropies are averaged instead of summed. The weight decay selected from the interval [le-6, 6, 1e-4]. Similarly to ImageNet,
29

---

Table H.1: Runtime overhead Monte Carlo sampling on ImageNet a percentage of total inference time across models and batch sizes.
Batch Size
Mean Field
sample
samples
samples
1000 samples
ResNet-22 (C=10)
2.75%
9.82%
10.63%
10.70%
10.70%
16
2.04%
10.61% %
11.51%
11.51%
11.92%
32
2.11%
10.98%
11.9%%
11.9%
12.42%
64
2.11%
10.88% %
11.90%
11.95%
12.34%
128
2.04%
8.12%
8.84%
8.96%
9.61%
256
1.17%
4.31%
4.73%
4.93%
7.14%
Wide-ResNet-26-- (C=100)
1.29%
8.54%
9.36%
9.40%
9.28%
16
1.16%
5.94%
6.58%
6.84%
8.56%
32
0.86%
3.36%
3.76%
3.98%
6.95%
64
0.47%
1,83%
2.03%
2.12%
5.82%
128
0.23%
0.90%
1.04%
1.04%
4.88%
256
0.12%
0.46%
0.53%
0.84%
4.47%
ResNet-50 (C=1000)
1.155
5.43%
5.88%
6.09%
6.30%
16
0.32%
1.25%
1.43%
1.70%
8.04%
32
0.16%
0.64%
0.72%
1.34%
7.51%
64
0.09%
0.36%
0.41%
1.13%
7.75%
128
0.05%
0.20%
0.22%
1.00%
7.82%
256
0.03%
0.10%
0.18%
0.95%
8.05%
use the best- pestpeerfrring checkpoint in the tests and use ten-step Bayesian Optimization scheme to select performant hyperparameters.
The hyperparameter optimization, training, and evaluation of the methods used in this paper took 0.8 GPU years on NVIDIA RTX 2080Ti GPUs in a university compute cluster. The individual runs required more than 50 GB of RAM and 3 days of runtime.
H Runtime Overhead of Monte Carlo Sampling
We measure the time obtain the predictives from the logit-space Gaussians compared to the time of a forward pass on an NVIDIA RTX 2080 Ti GPU (with similar results on Tesla A100 GPUs). As shown Table H.I, ResNet-50 with 1000 samples shows 6-8% overhead depending the batch size, confirming our claim in the main text. In contrast, the runtime cost of computing our closed-form forward pass, especially for larger batch sizes. These results demonstrate that even with diagonal covariance structure, MC sampling yields nontrivial overhead, especially for large numbers of classes and samples. comparison, our closed-form approach has negligible computational cost.
predictives is negligible compared
I Benchmark Metrics
Our experiments use five tasks/metrics:
1. Log probability proper scoring rule for the predictive,
2. Expected calibration error of the predictive's maximum- ciciiuum-rrobbbiltyy confidence,
3. Binary log probability proper scoring rule for the correctness prediction task,
4. Accuracy of the predictive's 's argmax.
5. AUROC for the -iitt--distribution detection task.
Below, we describe these metrics and their respective tasks.
30

---

1.1 Log Probability Proper Scoring Rule for the Predictive
First, we briefly discuss proper and strictly proper scoring rules over general probability measures based on [39].
Consider function S: Q y R where Q is family of probability distributions over the space ), called the label space.
S is called proper scoring rule if and only if
max By Sp = =p,,)(
(94)
ie.. p one of the maximisers of in q in expectation. S is further strictly proper arg m Bg Y) is the unique maximiser of S in q in expectation.
The log probability scoring rule for categorical distributions defined as
c
S(q. c) = Σ
80 log qe (x) = log (x(x))
(95)
c=11
where e [l..... C} is the true class and 8 is the Kronecker delta. S defined this way is a strictly proper scoring rule, i.e Sy, Y) is maximal if and only if
(V x) p(Y x) Ve (I,
(96)
The score above is equivalent to the negative cross-entropy loss.
1.2 Expected Calibration Error
To up the required quantities for the Expected Calibration Error (ECE) metric [42], we follow the steps below, based on [39].
1. Train neural network on the training dataset.
2. Create predictions and confidence estimates on the data.
3. Group the predictions into M bins based on the confidences estimates. Define bin B, be the set of all indices of predictions (in, C,,) for which
m m
Cn E
(97)
M M
The Expected Calibration Error (ECE) metric [42] is then defined as
M
Bml
ECE
Σ
|aco( Bm) - conf(Bm))
(98)
n
m=11
where
acc Bm)
Σ
10n = cn),
(99)
Bm
nE B.n
(B.) =
max fe(xn).
(100)
Bm
ISeSC
RE B.n.
Intuitively, the ECE is high when the model's per-bin confidences match its accuracy on the bin. We use M 15 bins in this paper.
I.3 Binary Log Probability Proper Scoring Rule for Correctness Prediction
The correctness prediction task measures the models' ability to predict the correctness of their own predictions. We consider correctness estimators ē(ι) E [0, 1] for inputs x X' derived from the predictives. Framed binary prediction task, the goal of these estimators is to predict the
31

---

probability of the predicted class' correctness. In particular, for an (input, target) pair (æ, y) with X E we the correctness target
= ((z, () =
max
hc(a) =y)
(101)
1SeSC
Dropping the dependency X' for brevity, the log probability score for binary targets é e {0, and estimators E [0, 1] defined as
log [og - 2)
if if
S E)
f log ē + (1 - E) log(1 - 2).
(102)
One can show that this is indeed strictly proper scoring rule [39].
1.4 Accuracy
For completeness, the accuracy of predictive h on dataset (Ins Cn) is
N
acc(h; (In, mm)
arg max he(xn)) = En cE(I....CC
(103)
N
==
I,5 Area Under the Receiver Operating Characteristic Curve for Out-of-Distribution Detection
Out-of-distribution detection is another binary prediction task where general uncertainty estimator u(x) R (derived from predictives second-order Dirichlet distributions) is tasked separate ID and OOD samples from a balanced mixture. The target OOD indicator variable o(x) is, therefore, binary. As the uncertainty estimator take on any real value, we measure the Area Under the Receiver Operating Characteristic curve (AUROC), which quantifies the separability of ID and OOD samples w.r.t. the uncertainty estimator.
Given uncertainty estimates u, u(x,,) and target binary labels o, on balanced dataset (Ii, On >= well as threshold E IR, we predict -futto--distribution) when un 2 and 0 (in-distribution) when un < t. This lets us define the following index sets:
True positives: TP(() = {n On = 1 Au 2 1}
False positives: FP(() {n On 0 Un 1}
(104)
False negatives: FN(() = {n
A < t}
True negatives: TN(() =(n : On OA Un <t)
The Receiver Operating Characteristic (ROC) curve compares the following quantities:
TP()))
TP(
TPR(t) (P) + [FN())
PP
(105)
[P(())
FP(()]
FPR(t) FP((t) + [TN()))
[N
Here, FPR tells us how many of the actual negative samples in the dataset are recalled (predicted positive) threshold t.
One can draw a curve (FPR(t), TPR(()) for from oo to +oo. This is the ROC curve. The area under this curve quantifies how well the uncertainty estimator u(x) can separate in-distribution and out-of-distribution inputs.
J Benchmarked Methods
This section describes our benchmarked methods and provides further implementation details.
32

---

J.1 Spectral Normalised Gaussian Process
Spectral normalised Gaussian processes (SNGP) [31] use spectral normalization of the parameter tensors for distance-awareness and a last-layer Gaussian process approximated by Fourier features to capture uncertainty. For an input a E X' and number of classes C, they predict a C-variate Gaussian
distribution
N (()())) T
v +I) ((z)Ic)
(106)
logit space.
B E RCDD a learned parameter matrix that maps pre-logits logits.
((x) cos (W (W + b) R2 is random pre-logit embedding of the input X. fL-1(x) denotes the pre-logit embedding. W is fixed semi-orthogonal random matrix, and b is also a fixed random vector but sampled from Uniform(0, 2Tr
is the (unnormalised) empirical covariance matrix of the pre-logits of the training set. This is calculated by accumulating the mini-batch estimates during the last epoch." 4
The method applies spectral normalization to the hidden weights in each layer using a power iteration scheme with a single iteration per batch to obtain the largest singular value. Liu et al. [31] claim this helps with input distance awareness.
J.2 Heteroscedastic Classifier
Heteroscedastic classifiers (HET) [7] construct Gaussian distribution in the logit space to model per-input uncertainties:
((f(x),(()))
(107)
where (x) E RRD is the logit mean for input æ E X' and
E(x) (a:) V(w) diag(d(x))
(108)
(positive definite) covariance matrix. Both the low-rank term V(x) and the diagonal term d(x)
are calculated as linear function of the pre-logit layer's output.
To learn the per-input covariance matrices from the training one has to construct predictive estimate from (x),, [xx) using any of the methods in Appendix E. This predictive estimate then trained using standard cross-entropy (NLL) loss.
HET uses a temperature parameter scale the logits before calculating the BMA. This is chosen using a validation set.
The off-diagonal terms of the covariance matrix do not affect the approximate predictive (Eq. (12)). This means that, our framework, one can discard the low-rank term V(x) and only model the diagonal term d(x) without a decrease in expressivity. To keep comparisons fair and use the same backbone with the same number of parameters, we also only model d(x) for softmax-based predictives.
JJ3 Laplace Approximation
The Laplace approximation [11] approximates the posterior p(0 'D) over the network parameters for a Gaussian prior p(0) and likelihood defined by the network architecture by a Gaussian. In its simplest form, it uses the maximum posteriori (MAP) weights OMAP E Rº the mean and the inverse Hessian of the regularised loss over the training set L(0; D) L(O; D) + 2|101| evaluated at the MAP as the covariance matrix:
OSAN ) -V OMAP,
82 00,00;
)
-NVOMAP
@LL(D:))
N OMAP,
+ XIp
(109)
00,00;
1
As we use cosine learning rate decay in all experiments, the model makes negligible changes in its pre-logit feature space the last epoch. Thus, the empirical covariance matrix is approximately consistent.
33

---

This is locally optimal post-hoc Gaussian approximation of the true posterior p(O D) based on second-order Taylor approximation. For details, see [50].
For deep neural networks, the Hessian matrix is often replaced with the Generalised Gauss-Newton (GGN) matrix. The GGN is guaranteed to be positive semidefinite for suboptimal weights and has efficient approximation schemes, such as Kronecker-Factored Approximate Curvature [37] or low-rank approximations.
Denoting our curvature estimate of choice by G. the logit-space Gaussian obtained by pushing forward the seight-space Gaussian measure through the linearised model around @MAP. For imput I E X, this results
N (f(z, (fI OMAP)) (Ja (Jouss (()) (G (G+ (Jons (t))
(110)
where fa) E RCxP is the model Jacobian matrix.
We use a last-layer KFAC Laplace variant in our experiments and use the full training set for calculating the GGN instead of a mini-batch based on recent works on the bias in mini-batch estimates [50].
K
Additional Results
K.1
Closed-Form Softmax Predictives
One can directly apply Eq. (10) to a neural network trained with the softmax cross-entropy loss in Eg. (4) to obtain closed-form predictives. However, we empirically found this approach to decrease performance, the denominator of Eq. (10) increased the order of billions during training. Considering multivariate normal random variable æ ~ (H, Σ) representing logit distribution,
C Σ
Var
Var(e">) = Eny (e Ess -
(111)
iee. when e#a is large for some j E {I,..., C). the variance of the denominator necessarily explodes, violating our assumption.
To mitigate this issue, one may optimise the regularised cross-entropy loss
C((an, Cn ( L((Zn; Cn) Cn)
Σ
Σ exp((c(xr))) -1
(112)
or the more numerically stable version
[((2n (n) L((xn, (n) Σ log Σ exp(fc(x.))
(113)
for an appropriately tuned À hyperparameter. The latter formulation can be stably trained even at an ImageNet scale. However, there are fundamental limitations to the predictive given by Eq. (10). Namely, for isotropic logit-space Gaussian distributions, o0(2)/2 introduces a constant shift in the log- its, under which the softmax activation function is invariant. Therefore, the predictive approximation collapses into the MAP prediction. The SNGP method predicts such isotropic logit-space Gaussians. Further, we empirically found that the Laplace method's 's predictives for models trained with the regularised cross- cross-entropy loss were also approximately isotropic; thus, this integral approximation yielded diminishing returns.
For completeness, we share the ImageNet and CIFAR-10 results of = exp in Tables K.1 and K.2.
K.2 Vision Transformer Results on ImageNet
Table K.3 shows results on VIT Little [14] backbones from the timm [53] library on the ImageNet validation set. the main paper, the hyperparameters are optimised using ten-step Bayesian hyperparameter optimization scheme of Weights and Biases.
34

---

Table K.1: Performance metrics of p exp on the CIFAR-10 validation dataset. We report the mean and two standard deviations.
Metric
SNGP
Exp HET
Exp Laplace
NLL
0.376 = 0.007
0.353 = 0.059
0.370 + 0.024
ECE
0.041 = 0.006
0.029 = 0.007
0.045 = 0.007
Log Prob.
-0.255 = 0.005
-0.238 = 0.038
-0.255 = 0.016
Accuracy
0.884 0.013
0.888 + 0.023
0.893 + 0.007
AUROC
0.592 = 0.006
0.603 + 0.009
0.596 + 0.011
Table K.2:
Performance metrics of yp exp on the ImageNet validation dataset. We report the mean
and two standard deviations.
Metric
Exp SNGP
Exp HET
Exp Laplace
NLL
0.866 = 0.031
0.929 + 0.042
0.986 + 0.149
BCE
0.058 = 0.004
0.022 + 0.007
0.058 + 0.013
Log Prob.
-0.380 = 0.009
-0.403 + 0.011
-0.402 + 0.014
Accuracy
0.784 0.001
0.779 + 0.001
0.785 + 0.002
AUROC
0.606 = 0.001
0.611 + 0.001
0.615 + 0.003
K.3 CIFAR-10 Results
This appendix section repeats the experiments presented the main paper on the CIFAR-10 dataset. For detailed description of the experimental setup, refer to Appendix G. Appendix describes the used tasks and metrics.
As stated in the main paper, our two research questions are:
What are the effects of changing the learning objective? (Appendix K.3.2)
Do we have sacrifice performance for sample-free predictives? (Appendix K.3.1)
K.3.1 Quality of Sample-Free Predictives
Similarly to the main paper, this section, we investigate our first research question: whether there is price to pay for sample-free free predictives. Table K.4 showcases the two best-performing (activation, method) pairs on the ECE metric and the CIFAR-10 dataset: Softmax and NormCDF Laplace. Mean Field (MF) is strong alternative for sample-free predictives, but it has no guarantees and can fall
Table K.3: NLL and accuracy metrics of ViT Little models on the ImageNet validation set. Error bars represent two standard deviations.
Method
NLL
Accuracy (%)
Laplace
Softmax
0.81988 + 0.0020
79.598 + 0.145
NormCDF
0.79603 + 0.0087
80.678 + 0.042
Sigmoid
0.79479 + 0.0002
80.104 + 0.206
HET
Softmax
0.87096 + 0.0242
78.604 + 0.378
NormCDF
0.79106 = 0.0075
80.514 + 0.425
Sigmoid
0.88175 = 0.0356
78.738 + 0.324
GP
Softmax
0.86724 - 0.0229
78.898 + 0.187
NormCDF
0.76974 = 0.0088
80.69 694 + 0.375 79.980 + 0.412
Sigmoid
0.81178 + 0.0115
35

---

Table K.4: Comparison of ECE results for different predictives using a fixed Laplace backbone.
Method
Mean + 2 std
Softmax Laplace
MC 100
0.0096 + 0.0013
MC 1000
0.0102 + 0.0016
MC 10
0,0120 + 0.0013
Mean Field
0.0121 0.0029
Laplace Bridge Predictive
0.5933 + 0.0105
NormCDF Laplace
Closed-form
0.0074 0.0012
MC 1000
0.0092 1 0.0022
MC 100
0,0095 + 0.0015
MC 10
0.0100 + 0.0020
0.04
NomrCD
ECB
Soltmax
0.02
Sgmoo
Sigmod
Seftmax
Sigmodd
Softmax
Laplaze
Figure K.1: CIFAR- 10 ECE results. Our closed-form predictives (=) outperform Softmax HET and SNGP. Laplace tunes its hyperparameters based on the ECE metric NormCDF Laplace is the overall best method, Note the restricted y-limits for readability.
behind MC sampling (see also Fig. 1). Empirically, our closed-fone predictives always perform par with MC sampling.
K.3.2 Effects of Changing the Learning Objective
As in the main paper, in this section, we use the best-performing predictive and estimator (see Ap- pendix E) for softmax models and employ methods with the closed-form predictives.
Calibration and Proper Scoring We first evaluate calibration using the log probability scoring rule [17] and the Expected Calibration Error (ECE) metric [42]. Fig. 3b shows that CIFAR- 10. the score of our closed-form form predictives (Sigmoid, NormCDF) are consistently better than the corresponding softmax results for all methods.
Fig. K. shows that on CIFAR- 10, our closed-fome predictives have a clear advantage on HET and SNGP. Laplace is post-hoc method that tunes its hyperparameters on the ECE metric, hence its enhanced performance. Our NormCDF predictive on par with Softmax.
Correctness Prediction Fig. K.2 shows that on the correctness prediction task, our closed-form predictives outperform all Softmax predictives across all methods, as measured by the log probability proper scoring rule.
Accuracy Closed-form predictives do not sacrifice accuracy. Fig. K.3 evidences this claim CIFAR-10: our closed-form predictives either outperform or are on par with Softmax predictives. The most accurate method is Sigmoid HET. These results support the findings of Wightman al. [54] that showcase desirable training dynamics of the class-wise cross-entropy loss.
Detection Finally, we consider the OOD detection task on a balanced mixture of ID (CIFAR-10) and OOD inputs. As OOD inputs, we consider corrupted CIFAR- -TFAR-10C samples. We use the AUROC metric to evaluate the methods' performance. As shown in Fig. K.4, the best- performing method is Sigmoid SNGP, closed-form method. Generally, Softmax performs par
36

---

NornCD
Softmas
Sigmodd
Norol oDD
Suftmox
Signoid
NormCDi
Saltmax
Log Prob. f
-0.4
Laplace
HET
SNGP
Figure K.2: CIFAR-10 log probability proper scoring results for the binary correctness prediction task. Our closed-form predictives consistently outperform Softmax on all methods.
1.0
0.8
Accuracy
0.6
0.4
Signol
Softmax
Signoid
Sollma
Sigmokd
NomChi
Softma
0.2
0.0
HEST
SNGP
Laplace
Figure K.3: Closed-form predictives do not sacrifice accuracy. CIFAR-10 accuracies. Our closed-
form predictives either outperform or on par with Softmax across all methods.
with closed-fome predictives. Intuitively, separating ID and OOD samples does not require fine-grained representation of uncertainty, unlike the ECE or proper scoring rules. Nevertheless, the closed-form predictives and second-order Dirichlet distributions considerably cheaper to calculate than Softmax MC predictions (see Section 6).
K.4 CIFAR-100 Results
We repeat the NLL and accuracy evaluation on CIFAR-100 using WideResset-28-5 models following a ten-step Bayesian hyperparameter sweep on Weights and Biases. Results are shown in Table K.5.
Table K.5: NLL and accuracy metrics of WideResNNet-2855 models the CIFAR-100 test Error bars represent two standard deviations.
Method
NLL
Accuracy (%)
Laplace
Softmax
0.92176 + 0.0128
78.225 + 0.412
NormCDF
0.88823 0.0094
78.900 0.325
Sigmoid
0.94209 + 0.0136
78.050 + 0.436
HET
Softmax HET
0.98513 # 0.0254
77.050 0.228
NormCDF HET
0.94643 + 0.0022
78.117 + 0.286
Sigmoid HET
0.95251 0.0142
77.583 + 0.175
SNGP
Softmax SNGP
0.97156 + 0.0348
77. 77.750- + 0.462
NormCDF SNGP
0.90011 0.0106
79.010 + 0.298
Sigmoid SNGP
0.99996 + 0.0265
78.350 + 0.215
K.5 Alignment with the True Predictives
Table K.6 shows the divergence of predictives using various approximation techniques from the true predictive estimated using 10, 000 Monte Carlo samples. Our closed-form predictives, NormCDF and Sigmoid, are favorable to both the mean field and Laplace bridge approximations.
37

---

AUROCT
Norm DFF
Stamooi
Normn
Sigmeid
Softmax
0.50
SNGP
Figure K.4: CIFAR-1OC OOD detection AUROC results for severity level one. Across all methods, the rest-performing predictive is closed-form
Table K.6: Kullback-Leibler (KL) divergence to the true predictive distributions (estimated using 10, 000 Monte Carlo samples) for different approximation methods on the ImageNet validation set using VIT Little backbones. The mean KL divergence is calculated over the validation set. Error bars represent two standard deviations over independently trained models using different seeds.
Method
KL Divergence True Predictive
NormCDF
0.0057 = 0.0008
Sigmoid
0.0064 = 0.0009
Softmax mean field
0.0330 = 0.0042
Softmax Laplace bridge
1.7900 = 0.1254
K.6 BCE Loss Performance Gains
Table K.7 shows that the Softmax and Sigmoid activation functions (and corresponding losses) already show improved performance on the vanilla models compared to softmax, highlighting that the performance gains observe cannot only he attributed but also the favorable training dynamics of the class-wise BCE loss.
closed-form predictive approximations
Table K.7: NLL results on ImageNet using ResNet-50 backbones and different activation functions. Error bars represent two standard deviations.
Activation Function
NLL
NornCDF
0.9345 + 0.0072
Softmax
0.9369 + 0.0025
Sigmoid
0.9146 + 0.0046
K.7
Further Out-of-Distributiin Detection Results
Fig. K.5 shows OOD detection results across all ImageNet-C severity levels. Our closed-form predictives consistently outperform Softmax.
K.8
Comparison to Evidential Deep Learning Methods
Table K.8 provides results for two evidential deep learning methods, EDL [46] and PostNet [6]. (We adopt the naming convention of [40].) We make the comparison because these methods also yield closed-form predictives and second-order quantities of interest, they build Dirichlet distributions whose advantages we discuss in Section 4.1. Following our experimental setup (Section 6), we optimise the hyperparameters of both methods using ten-step Bayesian Optimization scheme in Weights & Biases. Compared with the results presented in the main text, our approach outperforms the two EDL methods across all metrics, significantly on the latter metric indicate that our moment-matchen Dirichlet distributions capture second-order epistemic information more effectively than the loss-based EDL methods, which aligns with the findings of Bengs et al. [4].
NLL and OOD AUROC. The results
38

---

0.70
AUROC
0.65
0.60
NomCDD
Nomnt DF
0.55
Stamal
Sormas
Selmmx
Softmax
0.50
Laplace
(a) OOD detection AUROC with severity level one.
0.9
0.9
AUROC1
0.8
AUROC
0.8
0.7
0.7
NomCDF
Nomm DF
0.6
Nomi
Solmax
Sclmas
Sigmol
NomC
0.6
Signol
Nomc
Sigmal
Softmax
Softmat
0.5
0.5
Taplace
SNGP
Laplace
(b) OOD detection AUROC with severity level two.
(c) OOD detection AUROC with severity level three.
0.9
AUROCT
AUROC
0.8
0.7
0.7
NomCDE
Sollmax
Signos
Sofhna
Signnal
NomC
0.6
Signtal
NonCCDF
Stgnedd
Sofhmat
NomCCDF
NomCDr
Softmat
0.5
Taplace
SNGP
Taplace
(d) OOD detection AUROC with severity level four.
(e) OOD detection AUROC with severity level five.
Figure K.5: The OOD detection performance all methods increases steadily as we increase severity of the perturbed half of the mixed dataset on the ImageNet validation dataset. Our closed- form predictives consistently outperform Softmax.
Table K.8: Evaluation results of PostNet and EDL across CIFAR-I0, CIFAR-100, and ImageNet. Values mean two standard deviations.
Dataset
Metric
PostNet
EDL
ImageNet
NLL
1.10202 + 0.052
1.25541 0.076
Accuracy
75.602 + 0.627
75.632 + 0.326
ECE
0.058168 + 0.002
0.033219 + 0.001
OOD AUROC
0.60675 + 0.013
0.60167 0.018
C1F
NLL
1.17973 + 0.081
1.3645 1 0.107
Accuracy
75. + 0.192
75.383 0.273
ECE
0.12295 + 0.012
0.088213 0.005
OOD AUROC
0.59152 0.017
0.59471 0.02
CIFAR-10
NLL
0.37406 + 0.013
0.40403 +: 0.023
Accuracy
91.757 0.021
90.274 + 0.043
ECE
0.046192 + 0.002
0.041158 0.002
OOD AUROC
0.5931 + 0.014
0,60808 + 0.006
39

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims of improved uncertainty quantification capabilities are thoroughly verified in Section 7 and Appendices K.3, K.4 and K.7.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper's limitations are discussed in Section 8.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
40

---

All theoretical claims include the full set of assumptions and are proved in the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper accurately describes the experimental setup in Section and Ap- pendix G and the code is provided as supplementary material.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
41

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
See above.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) ca/
sttiii//giidess/oeeubmmiii sinoiiicsyyyyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//ipps.ccppullii httpcccccpppccceeeeeeeee wwwcc/ggiies.... /Caeee///omdemmmiimi ssi wooooiiyllllllloooooooooooiiiiil for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The main training, implementation, and hyperparameter optimization details are discussed in Section and Appendices G and J.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We run each experiment on five different seeds and report the min, mean, and metric values.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
42

---

The method for calculating the error bars should be explained (closed-fone formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources and time of execution are discussed at the end of Ap- pendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics bttpp://nneure cc htcp///liic/tthccciieeeeeeeeeele
Answer: [Yes]
Justification:
We fulfill all requirements.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper conducts no societally harmful research.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
43

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is risk in abusing ImageNet, CIFAR-100, CIFAR-10 classifiers.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the ImageNet, ImageNet-Real, CIFAR-10, CIFAR-100, and CIFAR- 1OH datasets and the timm library for the code and the pretrained ResNet-50s.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
44

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode www.coo//aaooo/o datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets; we only record metrics.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do not conduct experiments with humans.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
45

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper involves no use of LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp:r///..eu.eeeeeeee cc/Conffeeecces/2255 /LLM) for what should or should be described.
46


Reviews:
[{"review_id": "aZQrBRVftW", "reviewer": "Reviewer_bh5k", "summary": "The paper \"re-studies\" supervised classification problems from the perspective of softmax functions and other transformations from the *logit* space. Based on concerns around the way of modelling epistemic/aleatoric uncertainty, and the memory cost required for using Monte Carlo approximations to obtain predictive probabilities in the standard way --- the authors introduce closed-form predictives and other distributional approximations. Sadly, the work ignores a central line of research in ML, which is the Gumbel-Max trick, the Gumbel-Softmax trick, and all other contributions that in the period 2014-2020 (approx) did great advances in problems related to sampling from discrete distributions or dealing with their log-partition functions.", "questions": "Could the authors explain in some detail why the Gumbel-softmax trick or similar methods are not of interest here? Or why should they not be considered? Any connection or linking with them would be really helpful for this reviewer, who is actually a bit lost on \"why this work does not align with the references mentioned before\"\n\nIn this direction, I would also like to point out the first 3 paragraphs in Section 2 of Maddison et al. (NIPS 2014).\n\nThis TPAMI review can also help to find some central references:\n\n- Huijben, I. A., Kool, W., Paulus, M. B., & Van Sloun, R. J. (2022). A review of the gumbel-max trick and its extensions for discrete stochasticity in machine learning. _IEEE transactions on pattern analysis and machine intelligence_, _45_(2), 1353-1371.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "First of all, I have to say that I am surprised that the work seems to ignore and does not review classic methods to deal with classification. I was for instance wondering about the lack of reference to the following works:\n\n- Maddison, C. J., Tarlow, D., & Minka, T. (2014). A* sampling. NIPS.\n- Jang, E., Gu, S., & Poole, B. (2016). Categorical reparameterization with Gumbel-softmax. _arXiv preprint arXiv:1611.01144_\n- C. J. Maddison, A. Mnih, and Y. W. Teh, “The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables,” 5th Int. Conf. Learn. Represent. ICLR 2017\n\nFrom my point of view, the authors identify certain problems in the standard use of the softmax function and how it may affect certain predictive quantities, which are obviously related to the problem of sampling from a distribution (this is somehow described in the first paragraphs of pp. 2). In addition to the reasoning about aleatoric/epistemic uncertainty, which I think does not completely fit well here (for instance, the sentence starting in L30), they observe the general property of the logits before being used in the one-hot encoding.\n\nThe situation, therefore, is the following: Eq. 7 and Eq. 8 are presented as the main discovery or method to use, where it is a Gaussian expectation of the function $\\exp(y)$. I see the advantage of the analytical result, but I would be very surprised if this has not been considered before or studied.\nThis result is then pushed forward and used to make the main approximation in Eq. 10, whose contribution I don't understand very well, or at least I question myself about the convenience of the expectation on top of the one-hot-encoding fraction. Are we really allowed to do this approximation? Seems extreme in my opinion, some discussion could be perhaps helpful.\n\nIn that direction, the absence of reference to log-partition functions, the log-sum-exp operator, and other (very) well-known methods makes me feel lost, or at least very disconnected from concepts and principles that (as a reader and practitioner) I know and have used quite often. So I cannot judge the importance of the method in a good way without such links.\n\nAfter this central result on pp. 3 is included, the same sort of technique is reused two to three times, and the Gaussian integral of the exp() function and other simple functions is similarly considered in the Section 3. I reviewed experiments as well, but I cannot really value them well without first understanding the connections and solving the previous concerns and issues first.", "was_revised": true, "final_justification": "The rating scored was updated from 3: borderline reject to > 4: borderline accept. I added some comments in my first response message to the authors' rebuttal.", "rebuttal": "We thank the reviewer for their careful review of our manuscript.\n\n**Contextualisation.** We agree that better contextualization is needed. To this end, we have added a new Related Work section and an explicit list of contributions. To avoid repetition, these appear in full in our response to Reviewer `LBev`.\n\n**The Gumbel-max trick.** Thank you for highlighting this important related work. We have added all four of your suggested references of the Gumbel-max trick and related continuous relaxations to our Related Work section.\nWhile both our work and the Gumbel-max trick involve alternatives to standard softmax, they address different challenges. The Gumbel-max trick enables gradient-based optimization through discrete random variables by providing continuous relaxations of categorical sampling. Our approach instead provides fully _sample-free_ inference by deriving closed-form expressions for both predictives and second-order distributions over the probability simplex through moment matching. Among other things, our analytic approximate push-forward is radically faster than sampling (see also the response to Reviewer `a52w` below). The key difference is the _source_ of stochasticity. In Gumbel-max, the randomness originates from sampling discrete categories to enable gradient-based optimisation. In our setting, the randomness comes from Gaussian distributions over logits for _uncertainty quantification_, which we refer to as 'approximate Gaussian inference'. The prototypical example of an approximate Gaussian inference method is the Laplace approximation (e.g., [1]). This is an approximate Bayesian method that approximates the posterior over the neural network parameters with a Gaussian distribution. This Gaussian is then pushed forward to logit space, and the present work addresses how to further push this Gaussian forward onto the probability simplex *without* sampling.\n\n**Used approximations.** Thank you for examining our approximations carefully. To the best of our knowledge, Eqs. (7) and (8) have never been used for analytically approximating predictives. Importantly, Eqs. (7) and (8) are presented as a bridge between traditional softmax-based approximate Gaussian inference and our proposed predictive approximation for the sigmoid and normCDF activations, which we present as the main method to use. While the probit approximation has been used in _binary_ classification, we extend it to the multi-class case. We make this contribution clearer in our new list of contributions, which we display in full in our response to Reviewer `LBev`.\n\nIn Eq. (12), the expectation of the fraction is approximated by the fraction of the expectation. Such a type of approximation is well-known in the Bayesian literature and sometimes referred to as a mean-field approximation [2], though in this work, we refer to the mean-field approximation as a specific one that has been proposed to approximate the Gaussian softmax integral following Lu et al. [3]. We discuss the quality of our approximation later in the paper, both theoretically (Theorem 2.1) and empirically (in the synthetic experiment, Figure 1). Crucially, this approximation is highly accurate with our proposed normCDF and sigmoid activations, as shown in our experiments. As the variance across the different classes decreases, the approximation becomes more accurate (in the limiting deterministic case, the approximation is exact). The aim of Theorem 2.1 is to quantify this limiting behaviour, which we believe is the first error analysis for such a mean-field approximation in probabilistic deep learning.\n\nWe hope that our new Related Work section, the list of contributions, and clarifications above address the reviewer's concerns. We are happy to clarify any remaining points.\n\n[1]: Daxberger et al., (2021). Laplace Redux -- Effortless Bayesian Deep Learning\n\n[2]: Friedman & Koller, (2009). Probabilistic Graphical Models -- Principles and Techniques\n\n[3]: Lu et al., (2021). Mean-field approximation to Gaussian-Softmax integral with application to uncertainty estimation"}, {"review_id": "oe5dnyb5Ro", "reviewer": "Reviewer_LBev", "summary": "The authors use the fact that expectations over one-dimensional Gaussian distributed random variables\nare analytically tractable (or can be tightly approximated) for a number of transformations,\nsuch as the exponential function, or normal cdf, which often appear in classification objectives.\nThey use this to improve classification performance by predicting Gaussians in the logit space \nand transforming them using this result for a richer training signal. \n\nThe method is evaluated with several architectural variations on a range of image classification data sets\nand performance metrics, that demonstrate not only improved predictive performance, but also\nimproved calibration.", "questions": "Given the points mentioned above:\n- Q1: Can the authors provide a deeper discussion that highlights the strengths and innovations of their approach with respect to prior probit-based knowledge in the literature?\n- Q2: Can the authors discuss their approach with relation to the greater context of approaches mentioned above, and, if possible time-wise, a comparison against evidential deep learning or one of the Taylor approximation-based methods?", "limitations": "Societal implications are not discussed. I don't consider there to be any that go beyond the general implications due to deep learning. As such this is not a weakness of the paper.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- The paper is well-written and can be followed quite easily\n- The method is properly motivated with an extensive set of experiments (however, see the weakness below)\n- The proposal is generic enough to be added to a wide variety of prior approaches\n- The authors provide a detailed implementation\n\n## Weaknesses\n### Significance\nThe closed-form probit solution to the expectation of a normally distributed random variable is well-known in the literature (for recent work see, e.g., Kristiadi et al., 2020; Haussmann et al., 2020; Li et al., 2025 among others and the references therein).\nAs such, I would consider this common knowledge in the field with the remaining primary contribution being the application in a multi-class setting, which is rarer. \n(At least, I can't point to a specific reference, although it might have been explored.)\nAs such, the paper's main contribution is its detailed discussion and evaluation of such closed-form approximations. \n\n### Prior work\nThe second main weakness of the paper is its lack of discussion of prior work and experimental comparison against such work. Given the small theoretical contribution, such a discussion and evaluation\nwould be the main strength of this work.\n\nThe two main areas of research that are (partially) omitted are\n\n- **deterministic/sampling-free BNN methods:** \nAs there is a large variety of approaches in that area of research, I will only refer to a few representative ones. These are, Gast et al. (2020); Wu et al., (2019); Haussmann et al. (2020) for older approaches and Wright et al. (2024); Li et al. (2025) and the references therein for current work.\nAll of them aim to infer BNNs in a sampling-free manner and apart from Wright et al. also discuss sampling-free classification approaches. \nGast et al. parameterize a Dirichlet distribution (see below). Wu et al. derive a Taylor approximation-based approach (but do not evaluate it and stick to regression). Haussmann et al. discuss the Probit approximation, but stick to evaluating an adaptation of Wu et al.'s approximation. Finally, Li et al. use the Probit approximation for their classification experiments, refering to Kristiadi et al. (2020) as their motivation. \nGiven their relevance such related work should be discussed and compared against.\n\n- **Dirichlet-based approaches:**\nApart from Gast et al. (2020) mentioned above, a major research direction is known as _Evidential Deep Learning_. See Sensoy et al., (2018) who introduced it and the diverse literature that builds on it.\nWhile Sensoy et al. (2018) are referred to in passing, their relation should be discussed and compared against in greater detail.\n\n\n\n_____\nGast et al. (2018), _Lightweight Probabilistic Deep Networks_  \nHaussmann et al. (2020), _Sampling-Free Variational Inference of Bayesian Neural Networks by Variance Backpropagation_   \nKristiadi et al. (2020), _Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks_  \nLi et al. (2025), _Streamlining Prediction in Bayesian Deep Learning_  \nSensoy et al. (2018), _Evidential Deep Learning to Quantify Classification Uncertainty_  \nWright et al. (2024), _An Analytic Solution to Covariance Propagation in Neural Networks_", "was_revised": true, "final_justification": "I went through the other reviews as well as the rebuttals. I still think that the theoretical contribution and embedding into prior work are limited, but that the strengths are just enough overall outweigh them.", "rebuttal": "We are grateful to the reviewer for their thorough evaluation of our manuscript and for including several important references. We thank them for praising the clarity of our work, the extensiveness of our experimental analysis, the versatility of our framework, and the detailed implementation provided. We address their two central questions/concerns below.\n\n### Significance\n\nWe agree that the probit approximation for the Gaussian sigmoid integral is well-known, and certainly not novel to our work. It dates back to [1, 2], and as the reviewer points out, is used in recent works for binary classification, such as [3]. As the reviewer writes, we generalise it to multi-class settings. Furthermore, this is the first work that provides a theoretical analysis of such an approximation, to the best of our knowledge. We also show empirically that it outperforms other approximations from the literature. We extend it further to allow for not just predictive approximations but second-order Dirichlet approximations. Finally, we leverage the advantageous structure of the activation to train with the binary cross-entropy loss. Combined with our closed-form predictives, we demonstrate the improved uncertainty quantification capabilities of this pipeline.\n\nTo clarify these points, we now include a List of Contributions section at the end of the introduction:\n\n**List of Contributions**\n\nOur work extends the classical probit approximation for the Gaussian sigmoid integral in binary classification [1, 2] in the following manner:\n- Generalising it to the multi-class setting (Sections 2.1 and 2.2),\n- Analysing it theoretically (Section 2.3),\n- Analysing it empirically, and how it outperforms previous approximations (Figure 1 and Section 6),\n- Generalising it to second-order distributional Dirichlet approximations (Section 3),\n- Combining it with advantageous training (Sections 4 and 6).\n\nThese contributions are detailed in our revised manuscript, which also includes expanded Related Work and experimental comparisons.\n\n### Contextualisation\n\nWe agree that contextualising our work and comparing it to the existing literature is crucial, and we have spent a significant amount of effort in this rebuttal to do so.\n\nFirstly, we compare our approach to evidential deep learning methods [4, 5] and present the results below.\n\n**ImageNet**\n\n| Metric | PostNet | Sensoy et al. |\n|--------|---------|-----|\n| NLL | 1.10202 ± 0.052 | 1.25544 ± 0.076 |\n| Accuracy | 75.602 ± 0.627 | 75.632 ± 0.326 |\n| ECE | 0.058168 ± 0.002 | 0.033219 ± 0.001 |\n| OOD AUROC | 0.60675 ± 0.013 | 0.60167 ± 0.018 |\n\n**CIFAR-100**\n\n| Metric | PostNet | Sensoy et al. |\n|--------|---------|-----|\n| NLL | 1.17973 ± 0.081 | 1.3645 ± 0.107 |\n| Accuracy | 75.183 ± 0.192 | 75.383 ± 0.273 |\n| ECE | 0.12295 ± 0.012 | 0.088213 ± 0.005 |\n| OOD AUROC | 0.59152 ± 0.017 | 0.59471 ± 0.02 |\n\n**CIFAR-10**\n\n| Metric | PostNet | Sensoy et al. |\n|--------|---------|-----|\n| NLL | 0.37406 ± 0.013 | 0.40403 ± 0.023 |\n| Accuracy | 91.757 ± 0.021 | 90.274 ± 0.043 |\n| ECE | 0.046192 ± 0.002 | 0.041158 ± 0.002 |\n| OOD AUROC | 0.5931 ± 0.014 | 0.60808 ± 0.006 |\n\nFollowing our experimental setup (Section 6), we optimise the hyperparameters of both methods using a ten-step Bayesian Optimization scheme in Weights & Biases. Compared to the results presented in our paper, our approach outperforms the two EDL methods across all metrics, significantly so on NLL and OOD AUROC. The results on the latter metric indicate that our moment-matched Dirichlet distributions capture second-order epistemic information more effectively than the loss-based EDL methods, which aligns with the findings of Bengs et al. [6].\n\nWe also now compare to an additional softmax Gaussian integral approximation from [7].\n\nFinally, we have written a new Related Work section that contextualises our work w.r.t. all references shared by the reviewer. We paste this verbatim below.\n\n**Related Work**\n\nIn uncertainty quantification, methods that output logit Gaussians, which we refer to as *approximate Gaussian inference* methods, have become one of the dominant paradigms in uncertainty quantification [3, 8, 9, 10]. In classification tasks, such methods have the downside of requiring sampling at inference to transform Gaussian distributions into predictive probability vectors.\n\nIn parallel, prior work has developed ways to propagate probabilistic uncertainty through neural networks [11]. Several works develop analytic propagation of covariances up to the logits [12, 13]; our method complements these by addressing the propagation of the logit-space Gaussian distributions onto the probability simplex. Few works address this last step. The Laplace bridge [14] proposes a map from Gaussians to Dirichlets on the probability simplex. However, this approach offers no guarantees on the quality of its predictives, and it severely underperforms Monte Carlo sampling. Other approaches include the mean-field approximation to the Gaussian softmax integral [15] and the approximation of [7]. In this work, we propose approximate predictives that outperform existing closed-form predictive techniques and provide second-order Dirichlet distributions for other closed-form quantities of interest. Furthermore, this is the first work to provide a theoretical analysis of such an approximation.\n\nOther works have also proposed various activations for classification to replace the softmax. Particularly relevant is [16], which composes element-wise sigmoids with normalisation and utilises auxiliary-variable augmentation to obtain closed-form ELBO terms during training; however, its predictives require Monte Carlo estimation. Related conjugate/augmentation-based objectives exist for stick-breaking and tree-structured multi-class GPs [17, 18], and for BNNs, a tempered-likelihood/Dirichlet perspective with a factorised-Gaussian surrogate has been explored [19]. Yet, all these approaches still require sampling or numerical quadrature at inference time. [1, 2, 3] use the probit approximation for the Gaussian sigmoid integral in the binary case, which we generalise to the multiclass setting. [20] approximates the softmax by providing lower bounds for point estimate training, but replacing the softmax with these bounds during training does not yield a normalised multi-class likelihood. The binary cross-entropy loss with the element-wise sigmoid activation has been reported to outperform cross-entropy in some large-scale settings [21, 22]. Our work is the first to use this advantageous loss while normalising the output to obtain proper multi-class probabilistic models.\n\nEvidential deep learning (EDL) approaches [4, 5] offer sample-free outputs through Dirichlet distributions but have been shown to be unreliable for uncertainty estimation and disentanglement [6, 23, 24, 25]. We focus on making approximate Gaussian inference methods sample-free in a principled and performant manner; however, for completeness, we compare our approach with EDL methods in Appendix L.\n\nFinally, multiple authors have explored continuous approximations to categorical distributions, which enable gradient-based optimization for discrete stochastic models through sampling, such as the Gumbel-max trick [26, 27, 28, 29]. These methods principally optimise a single categorical distribution, while the present work is concerned with constructing probability densities over the (continuous) simplex of such categorical distributions.\n\n[1]: Spiegelhalter & Lauritzen, (1990). Sequential updating of conditional probabilities on directed graphical structures\n\n[2]: MacKay, (1992). The Evidence Framework Applied to Classification Networks\n\n[3]: Kristiadi et al., (2020). Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks\n\n[4]: Sensoy et al., (2018). Evidential Deep Learning to Quantify Classification Uncertainty\n\n[5]: Charpentier et al., (2020). Posterior Network\n\n[6]: Bengs et al., (2022). Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation\n\n[7]: Shekhovtsov & Flach, (2018). Feed-forward Propagation in Probabilistic Neural Networks\n\n[8]: Collier et al., (2021). Correlated Input-Dependent Label Noise in Large-Scale Image Classification\n\n[9]: Daxberger et al., (2021). Laplace Redux\n\n[10]: Liu et al., (2020). Simple and Principled Uncertainty Estimation with Deterministic Deep Learning\n\n[11]: Gast & Roth, (2018). Lightweight Probabilistic Deep Networks\n\n[12]: Wright et al., (2024). An Analytic Solution to Covariance Propagation in Neural Networks\n\n[13]: Li et al., (2024). Streamlining Prediction in Bayesian Deep Learning\n\n[14]: Hobbhahn et al., (2022). Fast Predictive Uncertainty for Classification with Bayesian Deep Networks\n\n[15]: Lu et al., (2021). Mean-field approximation to Gaussian-Softmax integral\n\n[16]: Galy-Fajou et al., (2020). Multi-Class Gaussian Process Classification Made Conjugate\n\n[17]: Linderman et al., (2015). Dependent Multinomial Models Made Easy\n\n[18]: Achituve et al., (2021). GP-Tree\n\n[19]: Kapoor et al., (2022). On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification\n\n[20]: Titsias, (2016). One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities\n\n[21]: Wightman et al., (2021). ResNet strikes back\n\n[22]: Li et al., (2025). Binary Cross-Entropy Loss Leads to Better Calibration than Focal Loss on Imbalanced Multi-Class Datasets\n\n[23]: Jürgens et al., (2024). The Epistemic Uncertainty of Evidential Deep Learning is not Faithfully Represented\n\n[24]: Pandey & Yu, (2023). Learn to Accumulate Evidence from All Training Samples: Theory and Practice\n\n[25]: Mucsányi et al., (2024). Benchmarking Uncertainty Disentanglement\n\n[26]: Maddison et al., (2014). A* Sampling\n\n[27]: Jang et al., (2017). Categorical Reparameterization with Gumbel-Softmax\n\n[28]: Maddison et al., (2017). The Concrete Distribution\n\n[29]: Huijben et al., (2023). A Review of the Gumbel-max Trick"}, {"review_id": "o72vV0Hd30", "reviewer": "Reviewer_a52w", "summary": "The paper proposes an alternative to Monte Carlo sampling when computing the predictive distribution in Bayesian NNs which model the logits using a Gaussian distribution per class in classification tasks.  The key idea is to decouple the nominator and denominator when normalizing the logits and compute each term using closed-form formulas involving Gaussian densities. The authors present a bound on the error invoked by the approximations made to derive the predictive distribution. The authors further fit a Dirichlet distribution over the simplex by means of moment matching. Empirically, the authors show comparable results in terms of accuracy to standard MC-based predictive distribution calculation while benefiting from increased uncertainty estimation and OOD detection.", "questions": "See Strengths And Weaknesses section.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper deals with a problem that is given less attention in the literature as far as I am aware. Commonly MC samples are taken which can be suboptimal. \n- Most steps in the method are supported either empirically or theoretically, for instance estimating the expectation of a ratio with the ratio between the expectation of the nominator and the expectation of the denominator. Or, using Beta distribution for relevant activation functions.\n- The paper is written clearly and the method has novelties in it.\n- The empirical results are reasonable, the method is comparable to the Softmax baseline in terms of accuracy and tends to improve in other metrics.\n\nWeaknesses/Questions:\n- The scope of the method has limitations as it assumes a Gaussian distribution over the logits per class which can be overly restrictive.\n- The authors present the computational benefit as one of the advantages of their approach. Yet, besides section 5 which presents it in terms of the number of operations I didn't notice any discussion. Perhaps the authors can show a comparison in terms of wall-clock time and memory to understand better when their method is preferred (e.g., as a function of the number of MC samples and classes).  \n- Perhaps I missed it, but do you have results for using your approximation for pre-trained networks with common cross-entropy loss and Softmax activation? If indeed models need to be trained or fine-tuned to accommodate the objective in Eq. 29 it can be a harsh restriction, especially in this era of large models. \n- In light of the Gaussian assumption over the logits, I believe that authors should have discussed, and compared where appropriate, to other alternatives for the Softmax likelihood. Several examples can be found in the Gaussian process literature. Non exhaustive list: one-vs-each [1], robust-max [2], Dirichlet observation model [3], logistic-softmax [4],  stick-breaking [5], and GP-Tree [6].\n- Technical questions:\n  - In Eq. 43, why do you assume that the expectation of $Q_c$ is equal for all classes?\n  - In Eq. 58, second line, shouldn't the last term be $\\sum_{c=1}^C p_c \\frac{(p_c - \\tilde{p}_c)^2}{2 \\omega (\\tilde{p}_c)^2)}$?\n  - Is there a justification to assume that $Var(\\sum_{c=1}^C Q_c) \\rightarrow 0$?\n- Minor:\n  - In Eq. 6 should it be $p(f(x))$ in the last term?\n  - Broken link in line 876 at the Appendix.\n\n[1] AUEB, T. R. (2016). One-vs-each approximation to softmax for scalable estimation of probabilities. Advances in Neural Information Processing Systems, 29.  \n[2]  Hernández-Lobato, D., Hernández-Lobato, J., & Dupont, P. (2011). Robust multi-class Gaussian process classification. Advances in neural information processing systems, 24.  \n[3] Kapoor, S., Maddox, W. J., Izmailov, P., & Wilson, A. G. (2022). On uncertainty, tempering, and data augmentation in Bayesian classification. Advances in neural information processing systems, 35, 18211-18225.  \n[4] Galy-Fajou, T., Wenzel, F., Donner, C., & Opper, M. (2020, August). Multi-class Gaussian process classification made conjugate: Efficient inference via data augmentation. In Uncertainty in artificial intelligence (pp. 755-765). PMLR.  \n[5] Linderman, S., Johnson, M. J., & Adams, R. P. (2015). Dependent multinomial models made easy: Stick-breaking with the Pólya-Gamma augmentation. Advances in neural information processing systems, 28.  \n[6] Achituve, I., Navon, A., Yemini, Y., Chechik, G., & Fetaya, E. (2021, July). GP-Tree: A Gaussian process classifier for few-shot incremental learning. In International conference on machine learning (pp. 54-65). PMLR.", "was_revised": true, "final_justification": "Please see my comment to the review.", "rebuttal": "We are very grateful to the reviewer for their thorough read of our paper, including the technical appendices, and for acknowledging the novelty of our work, the clarity of the presentation, the empirical and theoretical arguments that support the claims made in the paper, and the extensive set of experiments.\n\nBelow, we address the points raised by the reviewer one by one.\n\n**Logit-space Gaussian assumption.** As the reviewer points out, our work concerns methods that output Gaussian distributions over the logits. While this setting might seem restrictive at first, it encompasses many state-of-the-art methods for uncertainty quantification, making it arguably one of the dominant paradigms in uncertainty quantification for deep learning. Our empirical comparison of approximate Gaussian inference methods against evidential deep learning methods also supports this claim (see our response to Reviewer `LBev`). In the revised manuscript, we include a Related Work section which, among other things, describes the significance of such methods. To avoid repetition, we have pasted this new related work section in full in our response to Reviewer `LBev`.\n\n**Computational benefits.** We agree that providing more timing reports in the experimental section emphasizes the benefits of sample-free inference. We compare the time to construct Gaussians with diagonal covariance compared to the time of a forward pass. Below we show results on an NVIDIA RTX 2080 Ti. NVIDIA Tesla A100 GPU gave similar results. In reality, the overhead of MC sampling may be even greater, as each of these logit Gaussian samples needs to be mapped through the softmax function. In contrast, the runtime cost of computing our closed-form predictives is negligible compared to a forward pass.\n\n**Resnet-26 (C=10)**\n\n| Batch Size | 1 samples | 10 samples | 100 samples | 1000 samples |\n|------------|-----------|------------|-------------|--------------|\n| BS=1 | 10.47% | 11.33% | 11.32% | 11.09% |\n| BS=16 | 11.26% | 12.09% | 12.09% | 12.35% |\n| BS=32 | 11.42% | 12.47% | 12.47% | 12.75% |\n| BS=64 | 11.52% | 12.51% | 12.51% | 12.77% |\n| BS=128 | 8.91% | 9.65% | 9.71% | 9.94% |\n| BS=256 | 4.88% | 5.29% | 5.38% | 7.02% |\n\n**Wide-Resnet-26-5 (C=100)**\n\n| Batch Size | 1 samples | 10 samples | 100 samples | 1000 samples |\n|------------|-----------|------------|-------------|--------------|\n| BS=1 | 10.45% | 11.39% | 11.28% | 11.41% |\n| BS=16 | 7.65% | 8.32% | 8.50% | 9.21% |\n| BS=32 | 4.55% | 4.90% | 5.07% | 7.19% |\n| BS=64 | 2.68% | 2.88% | 2.99% | 5.72% |\n| BS=128 | 1.62% | 1.72% | 1.78% | 4.83% |\n| BS=256 | 1.05% | 1.12% | 1.36% | 4.32% |\n\n**Resnet-50 (C=1000)**\n\n| Batch Size | 1 samples | 10 samples | 100 samples | 1000 samples |\n|------------|-----------|------------|-------------|--------------|\n| BS=1 | 6.37% | 6.87%      | 6.97%       | 5.94%        |\n| BS=16 | 2.32% | 2.51%      | 2.69%       | 7.24%       |\n| BS=32 | 1.58% | 1.69%      | 2.22%       | 6.86%        |\n| BS=64 | 1.21% | 1.26%      | 1.72%       | 6.79%        |\n| BS=128 | 0.88% | 0.92%      | 1.40%       | 6.26%        |\n| BS=256 | 0.69%  | 0.73%      | 1.12%       | 5.88%        |\n\n**Pre-trained models.** Thank you for raising the important point about pre-trained models. While Appendix K.1 discusses the limitations of applying our approximation directly to off-the-shelf softmax models, we have an encouraging observation from our experiments: In our ImageNet setup, we started with pre-trained ResNet-50 and ViT-Little backbones and fine-tuned them with sigmoid/normCDF activations. We observed that these models adapted remarkably quickly, achieving near-optimal loss and accuracy after just 4-5 ImageNet epochs. This suggests that our approach remains practical in the era of large models, requiring only brief fine-tuning rather than training from scratch. Given the strong uncertainty quantification benefits demonstrated throughout the paper, we believe this short adaptation period is a worthwhile trade-off. This finding also opens interesting directions for future work on efficient uncertainty transfer between pre-trained models and our framework.\n\n**Softmax likelihood alternatives.** We appreciate the pointer to alternatives to the softmax likelihood and have clarified this in the paper. Of the cited works, robust-max [1], logistic-softmax [2], stick-breaking [3], and GP-Tree [4] do replace the softmax likelihood, whereas one-vs-each [5] is a training surrogate for softmax (not a normalised likelihood) and the Dirichlet/tempered perspective for BNNs [6] retains softmax while re-specifying aleatoric confidence. All of these approaches require sampling or one-dimensional quadrature at inference. By contrast, given Gaussian logits from any back-end, our method provides closed-form, sample-free predictives and second-order Dirichlet quantities, with a theoretical error analysis. To better contextualise our work in this significant body of literature, we now include a Related Work section in the paper. To avoid repetition, we have pasted it in full in our response to Reviewer `LBev`.\n\n### Technical Questions\n\n**T1:** Appendix C1 performs an informal theoretical analysis specifically under the setting of the synthetic experiment from Figure 1. In Eq. (43), we use that the logit means and variances are sampled i.i.d. for each class; hence, $\\mathbb{E}[Q_c] \\\\approx \\\\mathbb{E}[Q_1]$ for all $c$, and they are of the same order in the number of classes. The reviewer is correct in saying that this is not exact, but an approximation. Thank you for spotting this. We have replaced the equality sign with an approximation sign and explained the source of this approximation. Note that this approximation is valid in this informal analysis because we only care about the order of the quantities as a function of $C$, and in this case, $\\\\mathbb{E}[Q_c]$ will be the same order as $\\\\mathbb{E}[Q_1]$, i.e., constant in $C$.\n\n**T2:** Thank you for catching the missing summation in Eq. (58). We have now corrected this, and the rest of the analysis carries through as before. For completeness, we demonstrate this here:\n\n$$= \\sum_{c=1}^C p_c\\left(\\frac{p_c - \\tilde p_c}{p_c} + \\frac{(p_c-\\tilde p_c)^2}{2\\omega_c(\\tilde p_c)^2}\\right)$$\n\n$$= \\\\sum_{c=1}^C p_c - \\\\sum_{c=1}^C \\\\tilde p_c + \\\\sum_{c=1}^Cp_c \\\\frac{(p_c-\\\\tilde p_c)^2}{2\\\\omega_c(\\\\tilde p_c)^2}$$\n\n$$= \\sum_{c=1}^C\\frac{p_c}{2\\omega_c(\\tilde p_c)^2}\\cdot O\\left(\\operatorname{Var}\\left(\\sum_{c'=1}^C Q_{c'}\\right)^{1/2}\\right)^2$$ \n\n$$= O\\left(\\operatorname{Var}\\left(\\sum_{c=1}^CQ_c\\right)\\right)$$\n\nas $\\operatorname{Var}\\left(\\sum_{c=1}^C Q_c\\right)\\to 0$, where $\\omega_c(\\tilde p_c)\\in [p_c,\\tilde p_c] \\text{ or }[\\tilde p_c,p_c]$, where we used that $p_c$, $\\tilde p_c$, and hence $\\omega_c(\\tilde p_c)$ is bounded away from $0$ by compactness of $\\mathcal K$.\n\n**T3:** In practice we observe that $\\operatorname{Var}\\left(\\sum_{c=1}^C Q_c\\right)$ is very small (between $0.001$ and $0.01$). Meanwhile, the individual variances $\\operatorname{Var}(Q_c)$ remain meaningful for uncertainty quantification, as we see in the various experiments that the approximate Gaussian inference methods significantly outperform the neural networks' point estimates (Appendix K.5 for NLL). Moreover, there is a bound of the form $\\operatorname{Var}\\left(\\sum_{c=1}^C Q_c\\right) \\leq C \\sum_{c=1}^C \\operatorname{Var}(Q_c)$. On the other hand, $\\sum_{c=1}^C \\operatorname{Var}(Q_c)$ cannot be bounded by $\\operatorname{Var}\\left(\\sum_{c=1}^C Q_c\\right)$ in general. This shows that, for our approximation to be good, we do not necessarily require the individual uncertainties in the classes to be small, a small $\\operatorname{Var}\\left(\\sum_{c=1}^C Q_c\\right)$ is sufficient.\n\nTo demonstrate this, we run a synthetic experiment where we sample $200$ logit means and covariance matrices such that $\\sum_{c=1}^C \\operatorname{Var}(Q_c) > 0.1 \\cdot C$, take the minimum $\\operatorname{Var}\\left(\\sum_{c=1}^C Q_c\\right)$, and show that it is much smaller than $\\sum_{c=1}^C \\operatorname{Var}(Q_c)$, see the table below. This provides an existence proof that our approximation can work well even with significant individual class uncertainties, as long as  $\\operatorname{Var}\\left(\\sum_{c=1}^C Q_c\\right)$ is small.\n\n| C | 2 | 3 | 4 | 5 | 6 | 8 | 10 | 15 | 20 |\n| ---------------------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| Min Var[Sum sigmoid(Q_c)]    | 0.003678 | 0.005642 | 0.006437 | 0.020943 | 0.013800 | 0.029320 | 0.062187 | 0.034957 | 0.058531 |\n| Sum Var[sigmoid(Q_c)] at min | 0.430854 | 0.307303 | 0.604298 | 0.622324 | 0.986399 | 1.154063 | 1.582350 | 1.998524 | 2.906889 |\n\n### Minor\n\n**M1:** Our $\\mathrm f(x)$ is a Gaussian distribution, not a point estimate (see Eq. 5). So we believe our Eq. (6) is correct.\n\n**M2:** We have now fixed the link in the appendix. Thank you for pointing it out.\n\nThank you again for your detailed review. We believe we have addressed your main concerns regarding the scope of our method, computational benefits, and technical questions. We are happy to clarify any remaining points.\n\n[1]: Hernández-Lobato et al., (2011). Robust Multi-Class Gaussian Process Classification\n\n[2]: Galy-Fajou et al., (2020). Multi-Class Gaussian Process Classification Made Conjugate: Efficient Inference via Data Augmentation\n\n[3]: Linderman et al., (2015). Dependent Multinomial Models Made Easy: Stick-Breaking with the Pólya-gamma Augmentation\n\n[4]: Achituve et al., (2021). GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning\n\n[5]: Titsias, (2016). One-vs-Each Approximation to Softmax for Scalable Estimation of Probabilities\n\n[6]: Kapoor et al., (2022). On Uncertainty, Tempering, and Data Augmentation in Bayesian Classification"}, {"review_id": "xVRn9fnYKa", "reviewer": "Reviewer_uK2i", "summary": "The paper addresses the problem of uncertainty quantification in classification tasks. Building upon previous work on Gaussian inference in logit space, the Authors propose multiple enhancements to the standard procedure, aiming at\n- Sidestepping MC sampling, via closed-form evaluations of the Gaussian integral and the use of tractable approximations of the pushforwards.\n- Generalizing the final Softmax activation to sigmoid and normCDF, enabling accurate and MC-free approximations.\n\nThese methods are tested on standard image classification tasks.", "questions": "In line 173 they write \"For example, on a ResNet-50, sampling one thousand logit vectors from the logit-space Gaussian means an approximately 7% overhead on the forward pass\". Where is this number coming from?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The proposed methods are principled, and they address the concrete problem of MC sampling for approximate Gaussian inference.\n- There are some theoretical analyses supporting the proposed methodology  \n- Experimental results tested a good variety of activations and methods on standard image classification benchmarks, showcasing reasonable performance.\n\n## Weaknesses\n\n- From a readability perspective, I think that the paper could be greatly improved. While there are some extensive appendices and many details are reported, while reading the paper, I had a hard time piecing together the information and getting an overall picture. Some concrete examples:\n    - One of the main ingredients of the recipe is the approximation in equations 10 and 12, which is introduced unceremoniously by a simple $\\approx$ sign, without any further explanation highlighting its importance.\n    - A \"related works\" or similar section is missing, and I find the broader literature around the problem not adequately introduced. The works that are discussed are instrumental in the description of the methodology, but a broader context would be appreciated.\n    - Sections 3 (and its subsections) and 4 describe an undifferentiated sequence of additional methods for which it is hard (at least to me) to define a hierarchy of importance, and no indication is given about their possible use in concrete scenarios.\n    - As reflected in my confidence score, I do not consider myself an expert in uncertainty quantification. That being said, I had some hard times understanding what key metrics were evaluated in the experiments. For most of them, the inexperienced reader is diverted to external references to get some intuition of the tested metric.\n- I think that the experimental section critically lacks a report on the timings. One of the motivations behind the proposed methods is computational efficiency, which is theoretically discussed in Section 5. One of the first things I was looking for while reading the experimental section is a report of the actual timings to compute the predictive for the various methods, but I couldn't find one.", "was_revised": true, "final_justification": "My final thoughts after the rebuttal phase are positive. I hope that the explanations provided, which offer much better context for the methodology, will be included in the revised manuscript by the Authors. Also, the reported timings now make this a very compelling method.\n\nI maintain my endorsement for publication, but due to a lack of expertise in this specific area, I am not comfortable increasing further my already positive score.", "rebuttal": "We are grateful to the reviewer for their careful evaluation of our paper. We thank them for acknowledging the principled nature of our method, the theoretical analyses that support our claims, and the extensive set of experiments.\n\n**Eqs. (10) and (12).** We agree that the approximation in Eqs. (10) and (12) is the key approximation in our work. Therefore, it should be better introduced. It consists of bringing the expectation inside the fraction: the expectation of the fraction is approximated by the fraction of the expectation. Such an approximation is sometimes referred to as a mean-field approximation [1], although in this work, we specifically use the term to describe a method proposed for approximating the Gaussian softmax integral [2]. We discuss the quality of this approximation later in the paper, both theoretically (Theorem 2.1) and empirically (in the synthetic experiment, Figure 1). As the variance across the different classes decreases, the approximation becomes more accurate (in the limiting deterministic case, the approximation is exact). The aim of Theorem 2.1 is to quantify this limiting behaviour.\n\nTo improve the readability of the paper, we propose to explain in words after Eqs. (10) and (12) what the approximation consists of, and point the reader toward Theorem 2.1 to give the reader a glimpse of the analysis that is to come.\n\n**Related work.** A Related Work section indeed greatly helps contextualise the present work in the literature. To avoid repetition, we have included the full text of the new Related Work section in our response to Reviewer `LBev`.\n\n**Sections 3 and 4.** Section 3 addresses how to obtain Dirichlet distributions, which offer more expressivity than a single predictive value. These can be used, for instance, for uncertainty disentanglement or out-of-distribution (OOD) detection, as we showcase in the experiments.\n\nSection 4 discusses the advantageous training opportunities that the element-wise sigmoid and normCDF activations provide. The goal is to demonstrate that these activations are not only beneficial for approximate Gaussian inference but also for training the underlying neural network.\n\nWe appreciate the reviewer's point about connecting these sections to the broader framework. To address this, we have expanded our introduction with a detailed list of contributions that explicitly shows how each section builds toward our overall framework. We have pasted it in full in our response to Reviewer `LBev`.\n\n**Used metrics.** While Appendix H provides a formal overview of all the metrics we use, we agree with the reviewer that a more intuitive description would aid readability. We provide such an overview below and also give more context in the updated paper.\n  - The **negative log-likelihood (NLL)** metric measures how well the trained model fits the test datasets (i.e., whether it gives a high probability to the correct label). Given the model's predicted probability $p(y \\mid x)$ of label $y$ for input $x$, we compute $- \\log p(y \\mid x)$. As the negative of the log probability scoring rule, NLL penalises both over- and underconfidence, reaching its minimum only when the model outputs the ground-truth class probabilities. Lower values indicate better performance (indicated by the arrows in each plot's $y$-axis).\n  - The **expected calibration error (ECE)** measures whether a model's confidence aligns with its actual accuracy. We define confidence as the maximum predicted class probability (e.g., for predicted probabilities $(0.3, 0.2, 0.5)$, the confidence is $0.5$). Test samples are binned into fifteen equal intervals over $[0, 1]$ based on their confidence scores. Within each bin, we compute the L1 distance between the average confidence and the actual accuracy, then average this distance across all bins. A well-calibrated model has similar confidence and accuracy in each bin.\n  - The **correctness prediction task** is a binary problem to determine whether the model can predict its own correctness. For each input $x$, the model's predicted class $\\hat{y}(x)$ is the argmax of the predicted probability vector. We define the correctness indicator as $\\ell(x) := 1[\\hat{y}(x) = y]$, which is one if and only if the model correctly predicts the label for input $x$. These serve as the binary labels for the correctness prediction task. Given the model's uncertainty estimate $u(x) \\in [0, 1]$ for each input $x$ (e.g., its confidence, defined in the previous point), we evaluate how well $u(x)$ predicts $\\ell(x)$ using the binary cross-entropy loss over a test dataset. The negative of this value is often referred to as the log probability strictly proper scoring rule.\n  - Like correctness prediction, the **OOD detection problem** is also a binary prediction task, where a mixture of in-distribution and out-of-distribution samples is provided. The model's uncertainty estimate $u(x)$ (see above) is tasked to separate in-distribution and out-of-distribution samples. To measure the model's performance, we use the Area Under the Receiver Operating Curve (AUROC) metric, which measures how separable the in-distribution and out-of-distribution samples are based on the model's uncertainty estimate (where 1.0 indicates perfect separation and 0.5 indicates random performance).\n\n**Timings.** We agree that including more reports of timings in the experimental section helps to underscore the efficiency of our method. Regarding your specific question about the 7% overhead, this measurement was from preliminary experiments on different hardware (MacBook Pro M4). For more systematic comparisons, we measured the time to obtain the predictives from the logit-space Gaussians compared to the time of a forward pass on an NVIDIA RTX 2080 Ti GPU (with similar results on Tesla A100 GPUs). As shown below, ResNet-50 with 1000 samples shows 6-8% overhead depending on the batch size, confirming our claim.  In contrast, the runtime cost of computing our closed-form predictives is negligible compared to a forward pass, especially for larger batch sizes.\n\n**ResNet-26 (C=10) Runtime Overhead (sampling time as % of total inference time)**\n\n| Batch Size | Mean-Field | 1 sample | 10 samples | 100 samples | 1000 samples |\n|------------|------------|-----------|------------|-------------|--------------|\n| BS=1       | 2.75%      | 9.82%     | 10.63%     | 10.70%      | 10.70%       |\n| BS=16      | 2.04%      | 10.61%    | 11.51%     | 11.51%      | 11.92%       |\n| BS=32      | 2.11%      | 10.98%    | 11.94%     | 11.96%      | 12.42%       |\n| BS=64      | 2.11%      | 10.88%    | 11.90%     | 11.95%      | 12.34%       |\n| BS=128     | 2.04%      | 8.12%     | 8.84%      | 8.96%       | 9.61%        |\n| BS=256     | 1.17%      | 4.31%     | 4.73%      | 4.93%       | 7.14%        |\n\n**Wide-ResNet-26-5 (C=100) Runtime Overhead (sampling time as % of total inference time)**\n\n| Batch Size | Mean-Field | 1 sample | 10 samples | 100 samples | 1000 samples |\n|------------|------------|-----------|------------|-------------|--------------|\n| BS=1       | 1.29%      | 8.54%     | 9.36%      | 9.40%       | 9.28%        |\n| BS=16      | 1.16%      | 5.94%     | 6.58%      | 6.84%       | 8.56%        |\n| BS=32      | 0.86%      | 3.36%     | 3.76%      | 3.98%       | 6.95%        |\n| BS=64      | 0.47%      | 1.83%     | 2.03%      | 2.12%       | 5.82%        |\n| BS=128     | 0.23%      | 0.90%     | 1.04%      | 1.04%       | 4.88%        |\n| BS=256     | 0.12%      | 0.46%     | 0.53%      | 0.84%       | 4.47%        |\n\n**ResNet-50 (C=1000) Runtime Overhead (sampling time as % of total inference time)**\n\n| Batch Size | Mean-Field | 1 sample | 10 samples | 100 samples | 1000 samples |\n|------------|------------|-----------|------------|-------------|--------------|\n| BS=1       | 1.15%      | 5.43%     | 5.88%      | 6.09%       | 6.30%        |\n| BS=16      | 0.32%      | 1.25%     | 1.43%      | 1.70%       | 8.04%        |\n| BS=32      | 0.16%      | 0.64%     | 0.72%      | 1.34%       | 7.51%        |\n| BS=64      | 0.09%      | 0.36%     | 0.41%      | 1.13%       | 7.75%        |\n| BS=128     | 0.05%      | 0.20%     | 0.22%      | 1.00%       | 7.82%        |\n| BS=256     | 0.03%      | 0.10%     | 0.18%      | 0.95%       | 8.05%        |\n\nThese results demonstrate that even with a diagonal covariance structure, MC sampling yields a nontrivial overhead, especially for large numbers of classes and samples. In comparison, our closed-form approach has negligible computational cost.\n\nThank you again for your constructive feedback. We have incorporated all your suggestions to improve the paper's readability: adding a Related Work section, a list of contributions to explain the organisation of the paper, clear explanations of the key approximations, timing results, and intuitive descriptions of all metrics. We believe these changes significantly strengthen the presentation of our work.\n\n[1]: Friedman & Koller, (2009). Probabilistic Graphical Models -- Principles and Techniques\n\n[2]: Lu et al., (2021). Mean-field approximation to Gaussian-Softmax integral with application to uncertainty estimation"}]

Author rebuttal:

Meta-review: The reviewers of this submission all recommend borderline acceptance. Some of the reasons for these positive evaluations include writing quality (Reviewer LBev), studying a problem which is common-place but rarely explicitly considered specifically analytic integration of softmax-type expressions as opposed to sampling (Reviewer a52w, Reviewer uK2i), and good performance improvements on a set of reasonable baselines (Reviewer uK2i). Weaknesses include a lack of discussion of relevant prior work. I would be very surprised if the authors did not know of this work because some of the mentioned papers on Gumbel-Max extensions are very widely known, and think a better explanation is that the authors are aware of the work but simply didn't quite draw the connection due to their different goals. I think this weakness is minor enough to be addressable at a camera-ready stage, and would encourage the authors to indeed add the requested references mentioned by Reviewer bh5k. I did not detect any other major weaknesses, but on the other hand also think the paper's merits - especially the potentially very broad potential applicability of the ideas given ubiquity of softmax scores both in the approximate inference world and in deep learning more broadly - are sufficient for NeurIPS.

On basis of reviewer consensus, I recommend the work is accepted.
Decision: Accept (poster)