From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimation
Authors: Nikita Kotelevskii
Vladimir Kondratyev
Martin Takáč
Eric Moulines
Maxim Panov
Venue: ICLR 2025 Poster (2025)
Keywords: Uncertainty quantification
Bayesian methods
Statistics
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We show, how one can get predictive uncertainty measures using Bayesian approximations of statistical risk.

Abstract: There are various measures of predictive uncertainty in the literature, but their relationships to each other remain unclear. This paper uses a decomposition of statistical pointwise risk into components associated with different sources of predictive uncertainty: namely, aleatoric uncertainty (inherent data variability) and epistemic uncertainty (model-related uncertainty). Together with Bayesian methods applied as approximations, we build a framework that allows one to generate different predictive uncertainty measures.

We validate measures, derived from our framework on image datasets by evaluating its performance in detecting out-of-distribution and misclassified instances using the AUROC metric. The experimental results confirm that the measures derived from our framework are useful for the considered downstream tasks.

Published as a conference paper at ICLR 2025
FROM RISK TO UNCERTAINTY: GENERATING PREDICTIVE UNCERTAINTY MEASURES
VIA BAYESIAN ESTIMATION
Nikita Kotelevskijl..
Vladimir Kondratyev
Martin Takáč
Éric Moulines
Maxim Panov!
Depaatment of Machine Learning, MBZUAL, UAE
"CAIT, Skoltech, Russia
CCMAP, École polytechnique, France
ABSTRACT
There are various measures of predictive uncertainty in the literature, but their relationships to each other remain unclear. This paper uses decomposition of statistical pointwise risk into components, associated with different sources predictive uncertainty, namely alcatoric uncertainty (inherent data variability) and epistemic uncertainty (model-related uncertainty). Together with Bayesian methods, applied as an approximation, we build a framework that allows one to generate different predictive uncertainty measures. We validate method on image datasets by evaluating its performance in detecting out-of-distribution and misclassified instances using the AUROC metric. The experimental results confirm that the measures derived from our framework are useful for the considered downstream tasks.
INTRODUCTION
Nowadays, predictive models are applied in variety of fields requiring high-risk decisions such as
medical diagnosis (Shen et al. 2017 Litjens et | (017)) finance (Ozbayoglu et al., (2020] Heaton et al. 2017), autonomous driving (Grigorescu et al. 2020. Mozaffari et al. 2020) and others. A careful analysis of model predictions is required mitigate the risks. Hence, it of high importance to evaluate the predictive uncertainty of the models. In recent years, variety of approaches to quantify predictive uncertainty have been proposed (Kotclevskii al. 2022a Lahlou al. 2023
Kendall & Gal 2017 Van Amersfoort et al. 2020 Liu et al. 2000a|] Lakshminarayanan et al. 2017 Malinin & Gales 2021; Schweighofer al. 2023a b) and others. Specific attention has been paid to the distinction between different sources of uncertainty. It is commonly agreed consider two sources of uncertainty (Hüllermeier & Waegeman| 2021]: alcatoric uncertainty, that effectively reduces to the inherent stochastic relationship between the inputs (objects) and the outputs (labels), and epistemic uncertainty, which referred to as the uncertainty due to the "lack of knowledge" about the true data distribution. Distinguishing between aleatoric and epistemic uncertainties is crucial in practice because it helps identifying whether uncertainty can be reduced by gathering more data (epistemic) or if it is inherent to the problem (aleatoric), thus guiding better decision-making and model improvement.
Despite the practical importance and widespread usage of uncertainty quantification, there still no common strict formal definition of both types of uncertainly. This leads to a number of different
measures to quantify either type of uncertainty (see for example (Laksmminarayanan et al. 2017
Gal et al. [017]] Malinin & Gales [021 Hüllermmiier & Waegeman 2021 Kotelevskii et al. 2022a Schweighofer et al.]| 2023a)). Recently, lor iiformation-theorttic... measures, the step towards unified detinition was made in (Schweighofer et al. 2023a). However, it is not clear how all these measures
"Corresponding author. iikiiit.a... kotel evaki @mbzuai ac..ae

---

Published as a conference paper at ICLR 2025
of uncertainty related to each other in general. Do they complement contradict each other? Are they special cases of some general class of measures? In this paper, we introduce a statistical approach for predictive uncertainty quantification, reasoning in terms of pointwise risk estimation. This allows us to reconstruct a lot of known and popular measures of predictive uncertainty, as well as show how to build new ones. Our contributions
follows:
1. Following (Kotelevskii et al., 2022a Lahlou et al. 2023; Liu et al. 2019), we consider pointwise risk decomposition into distinct parts, that are responsible for capturing different sources of uncertainty. We show, that this decomposition, applied specifically to strictly proper scoring rules (Gneiting & Raftery 2007] Gruber & Buettner] (023) leads to general framework, that is amenable for generation of uncertainty measures.
2. We show how one can make this framework practical with the help of Bayesian estima- tion (see Section We show that commonly used measures of epistemic uncertainty, such as Mutual Information (Houlsby et al. 2011T] Gal et al., 2017). Expected Pairwise Kullback-Leibler divergence (EPKL) (Schweighofer et al. 2023a| Malinin & Gales) 2018). predictive variance and maximum probability score are special cases within our general approach. We also highlight the limitations of our framework, elaborating on discussions from (Schweighofer et al., 2023a Wimmer al., 2023).
3. We experimentally evaluate different predictive uncertainty quantification measures from the proposed framework in various tasks. Specifically, we consider out-tf--iitributtin detection and misclassification detection; see Section 6 Our results highlight the conditions under which each measure is most effective, providing practical insights for selecting appropriate uncertainty measures.
2 PREDICTIVE UNCERTAINTY QUANTIFICATION VIA RISKS
Assume we have a dataset Drr = [X,,, Yo) where pairs X, E Rd Y; E y are i.i.d. random variables sampled from a joint training distribution Px,(X, We consider classification task over K classes, i.e. y = {1...., (I....,K, K}. We can express this joint distribution as product: (x,
(Y,(( (X).
In practice, we typically consider a parametric model P(Y) XX0) with parameters 0 to approximate Pt-(Y X). We denote the true class probabilities for an input x as m/(x) Ptr X x). and the predicted probabilities as (x)) = P(Y X = x, 0). We will often omit the index 0 and denote the predicted probability vector by f).
2.1 POINTWISE RISK AS A MEASURE OF UNCERTAINTY
The goal of uncertainty quantification is to measure the degree of confidence of predictive mod- els, distinguishing between aleatoric and epistemic sources of uncertainty. In the paper, follow- ing (Kotelevskii al. 2022a Lahlou et 2023), we introduce uncertainty via the statistical concept of risk.
In machine learning, the main concern is the model's "error" at particular input point One way express this error is through the expected risk. Let é: R y - IR be loss function that measures how well fi)) matches the true label y. The pointwise risk R(7, æ) for model is defined as:
Rn n x) ((((()), dy X x).
Thus, pointwise risk is an expected loss received by specific predictor ij at particular input point J. Importantly, pointwise risk, while being natural measure of expected model error, can not be used directly as measure of uncertainty it is not possible to compute it due to unknown true data distribution. We will discuss possible ways to transform pointwise risk into the practical uncertainty measure in Section[]
Note, that we use distribution P, which might differ from Ptr. P(X) + Pxr(X) but P(Y | X r) =) (Y X = x) for any x. this situation is called "covariate shift". We consider this setup and assume 7イx) = P(Y X = x)) is valid for any x, meaning it is a vector of length K regardless of the input. Limitations of this assumption discussed in Appendix B
2

---

Published as a conference paper at ICLR 2025
2.2 ALEATORIC AND EPISTEMIC UNCERTAINTIES VIA RISKS
Predictive uncertainty can be divided into two parts. Alcatoric uncertainty expresses the degree of ambiguity in data and does not depend on the model, being an inherent property of the data given a particular choice of feature representation. Epistemic uncertainty is vaguely defined but typically is associated with the "lack of knowledge" of choosing the right model parameters 0 and model misspecification. When the source of uncertainty is not important, practitioners consider total uncertainty that aggregates all the sources.
Pointwise risk allows for the following decomposition:
R ( i x))
R Bayes (1 x) + ( x)) Rayess (1) x)
risk
Bayes risk
Excess risk
where RBaes is the pointwise Bayes risk, defined as:
Bayes (1) xr) = ((n(o), X X=
The Bayes risk represents the expected error from the true data-generative process 77x)). does not depend on the parameters of the model nor the choice of model architecture, and hence can be seen as a measure of aleatoric uncertainty. Note that in our analysis, we treat 2 as a noiseless observed variable. Alternatively, one could assume that only noisy observations H are available and include the uncertainty in observations into the definition of aleatoric uncertainty. However, we do not explore this scenario in this work. The second term equation (2) is "Excess risk" and represents the difference between the risks computed for the approximation and for the true model at a given input point xr Thus, it naturally represents a lack of knowledge about the true data distribution, i.e. epistemic uncertainty. We note that decomposition (2) was previously considered in the context of uncertainty quantification in (Kotelevskii et al., 2022a, Lahlou al. (023).
Although the decomposition these risk functions in general cases. Therefore, we consider specific class of loss functions, strictly proper scoring rules, which allows us to do theoretical analysis.
is useful, it doesn't 't provide much information about the properties of
3 RISKS FOR STRICTLY PROPER SCORING RULES
represent a class of loss functions that ensure that the minimizing predictive distributions coincide with the data-generating distribution P(Y X). Let's say a forecaster can produce a vector of predicted probabilities P E Pk. where Px space of discrete probability distributions over K classes. Then, {(P.y): Px x y - R the penalty the forecaster would have, given that event y is materialized. Its expected value with respect
Strictly proper scoring rules (Gneiting & Raftery 2007)
to some distribution Q we will denote as (P, Q) = f [(P( y))Q(y)
A scoring rule is called strictly proper if, for any P. E Px. it satisfies the condition E(P, Q) > E(Q, Q). with equality holding only when P = Q. Under mild assumptions (see Theorem 3.2 in GGeiiing & Raftery 2007)), ) any strictly proper scoring rule can be represented as:
((n.v) (G) n) (n)() G(n).
where (., is a sealar product, PK + is a strictly convex function, and Gn(()) = {G' (n)) Gxx (n)) is vector of element-wise subgradients.
Risk decompositions for strictly proper scoring rules. Here we derive the explicit expressions for different types of risks for strictly proper scoring rules (detailed derivations are given in Appendix
Total Risk (Total Uncertainty):
Ryo(1) A æ)) (G'(f) n - - 7)
(3)
Note, that Total risk depends linearly the true data generative distribution 7
* Bayes Risk (Aleatoric Uncertainty):
Reayes (1) x)) -Gn)-
(4)
Note, that Bayes risk is concave function of 1), since function G is convex.
3

---

Published as a conference paper at ICLR 2025
Log score
Brier score
Zero-one score
Spherical score
C(n) ?
Ex=1 A-1 me log leke
Σ
ma(1
m7)
maxe 1
Inlla
eap Rlng
En
Not defined
||a0 la
n +
m
E. sop E log Hn
v
Aleatoric (Bayes risk)
maxe T/k
1r/l2
Epistemic (Excess risk)
Dx.fulbil CE(nlid]
11
max; Uj
Targ maxa fis
1 (1
17 Tills
Total (Risk)
Phang money fi2
mll2 T Tolla
Table I: Expressions for risks and central predictions, computed for different strictly proper scoring rules. We omitted æ for clarity. DRL stands for Kullback-Leibler divergence and CE for Cross-Entron Easso-Enttopy.. See Appendix D for full derivations of risk instantiations and Appendix L for central predictions.
Excess risk (Epistemic Uncertainty):
RExe (17 x)) Do(1 f))
(5)
where Da 6) G(n) G(f)) (G'(()) 17 f)) is Bregman divergence (Bregman 1967)
In what follows, we will assume the dependency of risks on 77 n and for clarity will omit it, writing as function of x instead.
Specific Instances of Proper Scoring Rules. Different choices of the convex function G lead to different proper scoring rules. Table shows the results for some popular cases often used in machine learning algorithms (see detailed derivations in Appendix D
From Table see, that some of the risks correspond to well-known alcatoric uncertainty measures. For example, the Bayes risk for the Logg score is given by the entropy of the predictive distribution. For the Zero-one score, this component is given by the so-called MaxProb, also widely applied (Geifman
& EI- Yaniv 2017 Kotelevskii et al. 2022a Lakshminarayanan et al., 2017). For the Excess risk, we obtain different examples of Bregman divergence which lead some well-known uncertainty measures when coupled with the Bayesian approach to risk estimation (see Section
The derived equations are useful but require access to the true data-generative distribution which is typically unknown. One approach to deal with this problem was introduced in (Koteleskkii et al. 2022a). where authors considered a specific model no. namely Nadaraya- Wattoo kernel regression, as has useful asymptotic properties to approximate Excess risk. Another approach, the DEUP (Lahlou et al.) (023) proposed a method for estimating Excess risk by directly training a model to predict errors. However, in general cases, it is hardly possible to derive these results. In this paper, we consider a Bayesian approach to approximate 77 that allows us to derive both well-known from the literature and new uncertainty measures based on one unified framework.
Estimating risks.
4 BAYESIAN RISK ESTIMATION
The derived equations for risks depend on the true data generative distribution 1) and some approximation of it rj. particular, 1) appears in all the risks and is unknown. One needs to deal with that to obtain computable uncertainty measure. In the Bayesian paradigm, one considers a posterior distribution over model parameters p(0 (zr)) that immediately leads to a distribution over predictive distributions 770 p... The goal of this section is to give complete recipe for risk estimation under the Bayesian approach.
One can think of the risks functions of 1 and n. namely ( i),, where g is shortcut for any risk function (Bayes risk is a function of only one argument). We can approximate the risks with the help of posterior distribution using one of three ideas:
1. Bayesian averaging of risk. One can consider Eeg(ne n) to approximate an impact of the true model 17. In a fully Bayesian paradigm, the same can be done with s leading to the fully Bayesian risk estimate E, ((18, n)).
2. Central label (Posterior predictive distribution). One may use posterior predictive distri- bution îp,,, arg min, Egg Des (no || =) Egyp and plug it in risk equations instead of n
4

---

Published as a conference paper at ICLR 2025
and/or îj. This estimate naturally appears in decompositions of Bregman divergences (Pfau 2013 Adlam et al. 2022), and this context it is called "central label".
3. Central prediction. Interestingly, there is another natural estimate, that appears in the literature on Bregman divergences (Płau 2013 Adlam et al. 2022] Gruber & Buettner 2023
and referred as "central prediction It defined as v/ arg min, (= 110)
In general, one can consider nine approximations of Excess and Total risks (three approximation options for each of the two arguments) and three of Bayes risk. We denote the specific type of approximation listed above by positional indices 1, 2 and 3. For example, g(l.1) EgIee g( no).
g(a.1) Eeg(f, (f. ma)) and
In this section, we present some of the formulas for the resulting aleatoric, and epistemic uncer- tainty measures with brief remarks different approximations. For detailed derivations, other approximation options (not listed in the main part) and discussions, refer to Appendix
For Bayes risk obtain three cases it doesn't 't depend on fj:
R
Bayes (I) E
(1)
Raaess (x) -((p...) and
((2)
Bayes (z) 1 G(T).
For Excess risk we obtain the whole family of approximations. We list some of them here (see others in Appendix E
Expected Pairwise Bregman Divergence (EPBD):
(1.1)
Rixe (x) = Da n0).
In special case of Log score, it is Expected Pairwise KI. (EPKL.; Malinin & Gales
(202T): ; Schweighofer et al. (2023a)).
Bregman Information (BI):
R Exc
(1,2)
(a) Eg (np (...))
In special case of Log score, it is Mutual Information (Gal et al. 2017 Houlsby et al.
2011; Malinin & Gales) 2018). In case of Brier score, sum predictive variances (class-wise).
Reverse Bregman Information (RBI):
R(2,1) (a) = Eg DDo.in, n0)-
Its special case for Log score is known Reverse Mutual Information (RMI: Malinin & Gales (2021)). Modified Bregman Information (MBI):
R Ryxc (x) Do ()
(1,3)
It is similar to Bregman Information, but the deviation is computed from the "central prediction". not the central label (BMA).
Modified Reverse Bregman Information (MRBI):
R (3.1)
(x) = Ee (n 17).
Exse
This has similar structure to Reverse Bregman Information (RBI). Again, the deviation here is computed from another "central prediction".
One advantage of Bayesian models over point predictors that they can give different predictions for the same input. This means we don't have to average out uncertainty: instead, we can look at the disagreements between different versions of the model. From this perspective, EPDB especially promising because it doesn't 't rely averaging but considers all possible pairwise disagreements between models. Therefore does not average out uncertainty.
We observe that the general approach presented in this work allows us to obtain many existing uncertainty measures in the case of the Log score loss function while leading to the whole family of new measures (see Table We refer to Appendix Eo additional discussion and to Appendix [ for the discussion of connections these approximations between each other. The limitations of the approach are discussed in detail in Appendix B
5

---

Published as a conference paper at ICLR 2025
4.1 BEST RISK APPROXIMATION CHOICE
Given the plethora of measures, one may fairly ask if there is a best risk approximation of each type. For Excess and Total risks, elaborate it in Appendix E Here, address the choice for Bayes risk.
Recall, that for the estimation Bayes risk we have three options. However, it not clear, which of these approximations is better. To investigate it, we assume that there exists vector of true parameters 0* that for any input point the following holds: n7(x) = p(y x) = ply x, 8*). Note, that according to equation (4), Bayes risk is concave. Using r(.) = G.), the following follows from Jensen's 's inequality:
Egr(ne) S r(Beno).
At the same time, we know that for Bayes risk the following must hold for any 0:
r(7) rp( æ, 0)) s r(ne).
Hence, the following holds: r(n) v Eør(na) Therefore, we have the following relation:
r(n) z Egr(ne) s r(Bene),
(6)
or in other words that Rayes (x) is tighter approximation (on average), than Bayes (I). However, there seemingly no general answer for r(i).
(2)
4.2 CONNECTION TO ENERGY- BASED MODELS
Interestingly, we can establish a connection of our framework and energy-based models (Liu et al. 2020b). Let consider specific instantiation of MRBI for I Logscore. Also, let us recap the free energy function Exx; fa) from Liu et al. 2020b Grathwohl et al. 2020
K
fo(Ø)
Ex: fo) -ooog
exp
T
It can be shown (see Appendix G that
R (3.1) Exc
1 (x) - ( Bofo) Eor BBo
(3.1) From this decomposition we see, that Rxx is the difference between two different Bayesian estimates of the energy, where the first term is the energy, induced by expected logit, while second is expected
energy, induced by each individual set parameters. Therefore, it is interesting to check, whether (3,1)
Rexe predicted by framework, will be better, than either of its components.
RELATED WORK
The field uncertainty quantification for predictive models, especially neural networks, has seen rapid advancements in recent years. Among these, methods allowing explicit uncertainty disentanglement are particularly interesting due to the ability to use estimates of different sources of uncertainty in various downstream tasks. For instance, epistemic uncertainty is effective in oft-of-distribution
data detection (Hüllermeier & Waegeman 2021] Mukhoti al. 2023 Kotelevskii et al.) 2022b
Vazhentsev et al., 2023 Kotelevskii et al. 2024) and active learning (Gal et al. 2017 Beluch et al. 2018).
Another direction in uncertainty quantification involves credal set-based approaches, which represent uncertainty using sets of probability distributions. Caprio et al. (2023) introduced Credal Bayesian Deep Learning, leveraging credal sets for more conservative uncertainty estimates, while Caprio et al. (2024) proposed a novel Bayes' theorem for upper probabilities. These methods offer robust uncertainty quantification but can be computationally intensive, limiting their scalability.
Bayesian methods have become popular because they naturally handle distributions of model pa- rameters, leading to prediction uncertainty. Exact Bayesian inference is very computationally

---

Published as a conference paper at ICLR 2025
expensive Izmailov et al. 2021). many lightweight versions used in practice (Gal & Ghahra
mani 2016 Blei et al. [01 77 Lakshminarayaaann et al. 2017 Tsymbalov et al. 2018 Shelmanov
et al. 2021 Fedyamin al. 2021:] Thiin et al. 2020; 2021). Early approaches inspired by Bayesian
ideas GGal et al. 2017 Kendall & Gal 2017; Lakssmmnaraaaaann et al. (017) used information-based
measures like Mutual Information (Houlsby et al.) al.] [011] Malinin & Gales 2021) to quantify epis-
temic uncertainty and measures like entropy or maximum probability for alcatoric uncertainty. These methods, despite different computational costs, are widely used in the field.
However, the practical expense of Bayesian inference, even in its approximate forms, has led to the introduction of more simplified upproaches. Some of these methods leverage hidden neural network representations, considering distances in their hidden space as a proxy for epistemic uncertainty estimation (Van Amersfoort et al., 2020, Liu et al.) 2020a Kotelevskii et al., 2022a; Vazhentsev et al. 2022 Mukhoti et al. 2023). While they offer the advantage of requiring only single pass over the network, their notion of epistemic uncertainty, often linked to the distance of an object's representation to training data, captures only a part of the full epistemic uncertainty. Despite this limitation, their efficiency and effectiveness in oul-of-distribution detection have made them widely used. Another class of models for uncertainty quantification that utilizes generative models (e.g.. diffusion models or normalizing flows) has been recently explored (Berry & Meger 2023 Chan et al.. 2024). While these methods offer promising direction, they require training a generative model, which can be computationally intensive and may limit their applicability in certain settings.
Works by Gruber & Buettner 2023); Adlam et al. (2022) are close to ours. In both papers, decompo- sitions of Bregman divergence loss functions are discussed. While the main focus of these papers is generalized bias-variance decomposition, Gruber & Buettner (2023) also considers an application to uncertainty quantification. In terms of our framework, they arrive to the following decomposition (see derivation in Appendix Fb:
(1,1)
(3.1)
(2,3) + Exx
R Toe
-R Bayes
(8)
Fxx
(3.1) which is only partial case of our framework. There, they called R Rxx as generalized "variance", and use it as a measure of uncertainty for out-of-distribution detection. However, it is not clear, why exactly this decomposition and approximations were chosen. For example, as we showed in
(1) Bayes leads to tighter approximation of Bayes risk on average, than Rayes
(2)
Section 4.1
Works by Kotelevskii al. 2022aa Lahlou al. (2023) and Liu et al. (219) also consider the same decomposition of risks. However, in all these papers, only specific approximations were explored, and no general connection among existing measures was established.
Another recent work by Hofman et al. (2024) independently explored several of the decompositions presented in our study. However, the analysis of central predictions and the connections to Gruber & Buettner (2023) and the energy-based model were not addressed in their work.
Despite the diversity of these approaches, the arbitrary nature of choosing uncertainty measures has led to ambiguity in understanding uncertainty. This paper aims to address this gap by proposing unified framework that not only categorizes these diverse methods but also offers a more comprehensive understanding of uncertainty quantification.
6 EXPERIMENTS
In this section, we test different uncertainty measures derived from our general framework. Any Bayesian method that produces multiple samples of model weights (or parameters of the first-order distribution) can be used to compute our proposed measures. However, deep ensembles are considered the "gold standard" in uncertainty quantification (Lakshminarayanan et al.. (017). Therefore, for experiments, use deep ensembles trained with various strictly proper scoring rules as loss functions, As training (in-distribution) datasets, we consider CIFARIO, CIFARION (Krizhevsky 2009). and ImaylmageNet (Le & Yang 2015).
We evaluate the proposed measures of uncertainty by focusing on two specific problems: out-of- distribution detection and misclassification detection. For both problems, we trained five deep ensembles independently, each consisting of four members-resulting in a total of 20 models trained completely independently. All ensemble members shared the same architecture but differed due
7

---

Published as a conference paper at ICLR 2025
Table AUROC for different choices of function G for out-of-distribution detection with CIFARIO as in-distribution. We used 5 groups of ensembles of size 4. We see, that in most cases, uncertainty measures, based on Log Score, appear to be better. See more detailed results in Appendix J
CIFARIO0
TinylmageNet
CHEARIOC-1
CITARIOC-2
CHEARIOC-
CITARIOC
CITARIOC-5
R1))
91.36 = 0.05
96.01 = 0.39
90.84 1 0.04
60.96 = 0.09
67.84 2 0.09
72.59 = 0.08
71.31 = 0.08
83.48 = 0.13
Rgg
90.17 0.06
94.08 1.02
89.32 0.07
61.07
67.82 + 0.09
72.49 0.08
77.15 + 0.13
83.0
Rine
90.38
94.35
0.07
0.09
72.55
0.09
77.23
Rise
90.38
0.06
94.31 + 0.91
89.54 0.06
61.09 0.11
67.86 + 0.09
72.34
0.09
71.22 + 0.13
83.11 0.19
K
90.44 + 0.17
96.09 + 0.52
89.89 0.13
61,03 0.12
68.04 0.16
72.46
0.18
76.83 # 0.18
82,47 0.1
Brier
Rin
89.24
0.19
94.19 +: 0.49
88.37 0.12
11
66.59
0.22
71.19 0.3
75.7 15.7=022
81.52
Riw.
89.24 1: 0.19
94.19 4: 0.49
88.37 0.12
60.11 0.15
66.59 0.24
71.19 1 0.3
75.7 0.22
81.52 0.16
Ryw
89.24 1: 0.19
94.19 d: 0.49
88.37 1 0.12
60.11 = 0.15
0.24 66.59
71.19 - 0.3
75.7 1: 0.22
81,52 =: 0.16
Spherical
R
90.46
61.47
68.54
72.95 0.28
71.39 2: 0.34
82.99
Rew
89.11 +: 0.15
93.77 +: 0.66
88.28 + 0.2
60.54 0.18
66.88
0.26
71.36 + 0.29
75.96 + 0.29
81.89 0.33
89.17 + 0.14
93.86 + 0.66
88.35 + 0.2
60.54 0.18
66.88
0.26
71.37 + 0.3
75.97 + 0.29
81,91 0.34
(8.0)
89.1 0-1.5
93.78 + 0.66
88.27 + 0.2
60.54
66.88
0.26
71.35 0.29
75.95 + 0.29
81.88 0.33
randomness in initialization and training. We used ResNet18 (He et al., 2016) the architecture (additional details can be found in Appendix H). Note that in all the tables below, we report the average performance across different groups of ensembles (AUROC), along with the corresponding standard deviation.
Our experimental evaluation aims to answer the following questions:
1. Is there a best function that yields better results in the considered downstream tasks?
2. Is Excess risk always better than Bayes risk for out-of-distribution detection?
3. Is Total risk always better than Excess risk for misclassification detection?
4. Is the measure of uncertainty proposed by our framework better than other Bayesian esti- mates of the energy?
We emphasize that the goal of our experimental evaluation is not provide new state-of-the-ee measures or compete with other known approaches for uncertainty quantification. Instead, we aim to verify whether different uncertainty estimates are indeed related specific types of uncertainty. We refer readers to Appendix A where we discuss why one might expect either type of uncertainty to be good for a specific task. Additionally, we refer the reader to Appendix M for specific toy example where our measures of epistemic uncertainty can be computed in closed form. In this example, we also discuss cases of prior misspecification and model misspecification.
6.1 IS THERE A BEST FUNCTION PLUG-IN CHOICE?
In this section, we address the first question. For this, we consider some specific risks and different plug-in choices for the function G. As a loss function, we use strictly proper scoring rules generated by the same function G (hence, we do not include the Zero-One score, it is not differentiable). We present results in Table 2 (see additional results in Tables 5 in Appendix). We see that, in most cases, the Log Score performs better than the others. This result is reassuring because Log Score- based measures are the popular choice in the literature on uncertainty quantification. As for misclassification detection (see Table 4 and Tables 13 in Appendix), the results more comparable, but Log Score is still good choice.
Hence, the answer to the first question is the following: there is no all-time-best plug-in choice of G. However, Log Score-based measures (that are already popular in practice) are typically a good choice in the considered downstream tasks.
'The source code is publicly available at http:///g/.o.ooooo.////t///////t proper_ sccrr_sscrriii rul.rullee.
8

---

Published as a conference paper at ICLR 2025
Table 3: AUROC (Log Score) for OOD de- tection with ImnylmageNet ni-distribution.
Table 4: AUROC (Log Score) for misclassification detection. For loss function and uncertainty plug-in,
we use the same G, that corresponds Log Score.
ImageNet-A
ImngeNet-RR
ImaggeNet-O
CIFARIO-N and CIFAR 100-N stand for noisy versions of these datasets (see Appendix !
R
K3.22 0.24
82.25 + 0.39
72.21 0.22
83.76
82.78 0.37
73.18
83.61 +:
82.67 +: 0.37
72.86
CIFARIO
CIPARIOU
CIFARIO-N
CIFARIOLN
ThyylaggeNe
Ress
77.11 + 0.34
76.52 + 0.19
74.46 + 0.18
R
94.30 0.08
84.61 = 0.35
78.26 2: 4,11
819 0.29
$4.99 0.24
79.09 0.33
78.38 + 0.17
74.79 0.25
94.7
0.05
85.13 0.35
78.36 016:::477 4.77
81.9 1:
85.5 0:223
Riss
75.94 0.36
75.4 0.23
74.07 0.18
94.76 + 0.00
85.95 0.31
80,44 3.69
83.4 + 0.23
56.35 + 0.26
76.21 +
75.55 0.21
73.84 0.17
94.1
U.14
81.58 = 0.34
08.39 =: 6,17
74.52 à: 0.83
$1.33 : 0.29
Ras
77.56
0.17
94.23 + 0.14
82.9 = 0.31
09,64 + 6.1
75.74 0.78
83.22 0.28
R
R0-11
0.24 83.76
84.26 0.23
82.78 + 0.37
83.32 1: 0.31
74.93 0.17
73.18 0.2
Reps
Rec
93.72 = 0.16
94.01
1 0.14
$2.62
$0.89 0.35
90.3
0.34
69,98
66.85
61.7 = 6.09
73.37 4: 0.57
73.7 0.86
75.51
79.25
50.18 1 0.31
Rig
83.93 0.24
82.95 1: 0.36
0:116
Ri
94.5 0.06
85,43 = 0.35
75.73 4.68
81:67 :: 0.41
85.58 0.22
83.88 d: 0.19
82,99 4: 0.32
74.09 0.27
94.7 0:05
85.13 0.35
78.36 4.77
81:92 +: 0.36
85.5 0:25
Fiz: Rofo)
83.96 0.23
83.28
0A1
72.72 0.33
R1
94.4 0.06
85.01 0.37
76.26 5.04
81:67 + 0.39
85.18 0.25
82.99.1:003
82.31
1: 0.15
0.31
Ry
94.74 + 0.07
86.22 0.31
79.38
83.18 +
0.26
6.2 Is EXCESS RISK BETTER THAN BAYES RISK FOR OUT-OF-DISTRIBUUS DETECTION?
In this section, we evaluate different instances of our framework to identify out-of-distribution samples. Since the uncertainty associated with OOD detection of epistemic nature (see Appendix A for discussion), we expect that Excess and Total risks will perform well for this task, while Bayes risk will likely fail. In the main part, we consider two datasets as ii-distribution, namely CIFAR10 in Tad TinylmageNet in Table 3 For more experiments, we refer to Appendix
We distinguish between two types of out-of-distrbuttion data: "soft-OOD" and "hard-OOD". Both are special cases of covariate shift. "Soft-OOD" samples, such slightly changed versions of CIFARIO or TinylmageNer, have predicted probability vectors that are still meaningful (see discussion in Appendix B). In our experiments, CIFAR can be considered "soft-OOD", as it is corrupted version of CIFARIO, 10, while ImageNet-O (see description in Appendix D is "soft-OOD" for Tiny- ImageNet. WHard-OOD" samples, however, have completely non- -on-informative predicted probability vectors. For example, when a set of classes is considered during training, but an incoming image does not belong to those classes, the resulting probability distribution over training classes is meaningless.
we see that Excess risk is a good choice for "soft-OOD" (especially in case of Log score). However, as data become more "hard-OOD", the results unexpected Bayes risk typically outperforms Excess risk. We provide one possible explanation of this effect in Appendix A
From the Tables 2 and 3
This highlights crucial limitation of Excess risk (which includes ubiquitous BI, RBI, and EPBD) as a measure of epistemic uncertainty. These measures naturally appear when approximating Excess risk in a Bayesian way, which assumes a specific form of ground-truth distribution approxi- mation. However, this approximation becomes inaccurate for "hard-OOD" samples, making these measures poor choice in these cases. This criticism aligns with findings from (Wimmer al.) 2023 Schweighofer et al. 2023a Bengs et al. 2023). which indicate that Bregman Information (in particular case of Log score) is not an intuitive measure of epistemic uncertainty and does not follow their proposed axioms.
Therefore, the answer to the second question not absolute. For "soft-OOD" samples, where predicted probability vectors remain meaningful, Excess risk is good choice. For "hard-OOD" samples, Bayes risk is typically better. Total risk consistently shows decent results, making a safe choice when the nature of incoming data is unknown.
6.3
IS TOTAL RISK ALWAYS BETTER THAN EXCESS RISK FOR MISCLASSIFICATION DETECTION?
We now consider misclassification detection. Misclassification detection is intuitively connected to aleatoric and total uncertainties (see Appendix A). Therefore, expect Bayes and Total risks to

---

Published as a conference paper at ICLR 2025
perform well in this task, while all instances of Excess risk should perform typically worse (given the scenario that there is enough training (in (i-distribution) data).
It is known that standard versions of CIFARIO and CIFARIO0 lack significant alcatoric uncer- tainty (Kapoor et al. 2022). making it challenging to demonstrate the usefulness of the appropriate uncertainty measures immediately. To address this, create special versions of these datasets with introduced label noise (see details in Appendix[]] [.
From the Table we that Bayes and Total risks indeed outperform Excess risk for misclassification detection (see additional results in Appendix Kj. Moreover, the difference in performance becomes more significant more aleatoric noise is introduced into the training dataset. Hence, the answer this question is mostly positive. Bayes risk and Total risk are better for misclassification detection.
6.4 WHICH ENERGY ESTIMATE IS BETTER?
We discovered a connection between our approach and energy-based models that are used for out- of-distribution detection. However, in approach, energy appears as the difference between two different Bayesian energy estimates for a particular choice of G. Which one of these three quantities, based on energy, is better?
Based on the results in Table (see also Tables 5
10 and 13 in Appendix) we can conclude, both considered problems. Specifically,
that (x; fo) is consistently better, than Eg Ex;;
(3.1)
RExe performs better than both energy estimates for misclassification detection on all the datasets (3.1)
in Table 10 while for TinylmageNet the results are close (see Table 13 in Appendix). R Exc is also
preferred over energy estimates for Soft-OOD detection for ImageNer-O (see Tables and and CIFARIOC[1.2] (see Table(5).
When the outttf-distiibuution detection problem considered and "hard-OOD" encountered, (3.1).
is typically worse than both of the energy estimates in terms of AUROC. However, when "soft- OOD" is encountered, their difference (3.1) as a particular instance of Excess risk, becomes more effective than both of them (see results on ImageNet-O in Tables 3 anddd)) When the misclassification
(3.1), better than both of the estimates.
detection problem considered, Rysc
(3,1),
Hence, the answer is in line with what we discussed above: Excess risk (and in particular R (3,1) is good choice for "soft- Soott-OO." Therefore, when this is the case, R performs well. When "hard-OOD" is the case, Exx; Eofo) is typically better.
7 CONCLUSION
In this paper, we developed a general framework for predictive uncertainty quantification using pointwise risk estimates and strictly proper scoring rules as loss functions. We proposed pointwise risk as a natural measure of predictive uncertainty and derived general results for total, epistemic, and aleatoric uncertainties, demonstrating that epistemic uncertainties can be represented as Bregman divergence within this framework.
We incorporated Bayesian reasoning into our framework, showing that commonly used uncertainty measures, such as Mutual Information and Expected Pairwise KL divergence. are special cases within our general approach. We also discussed the limitations of our framework, elaborating on recent critiques in the literature (Wimmer et al., 2023 Schweighooer et al. 2023a). Moreover, we showed that even energy- based models fall into the framework as particular approximation of Excess risk.
Finally, in our experiments on image datasets, we evaluated these measures for out-of didsrribbutionn and misclassification detection tasks and discussed which measures are most suitable for each scenario.
ACKNOWLEDGEMENTS
This research was partially supported by RSF grant 20-71-10135. Nikita Kotelevskii would like to thank Sergey Kostrov for deriving the central prediction for the spherical score.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Ben Adlam, Neha Gupta, Zelda Mariet, and Jamie Smith. Understanding the bias-variance tradeoff' of bregman divergences. arXiv preprint arXiv:2202044166 2022.
William Beluch, Tim Genewein, Andreas Nürnberger, and Jan Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 9368-9377, 2018.
Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. On second-order scoring rules for epistemic uncertainty quantification. In International Conference on Machine Learning, pp. PMLR, 2023.
Lucas Berry and David Meger. Escaping the sample trap: Fast and accurate epistemic uncertainty estimation with pairwise-distance estimators. arXiv preprint 13X4: 2023.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 1126518):59-877, 2017.
Lev M Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming. USSR computational mathematics and mathematical physics, 7(3):200-217, 1967.
Michele Caprio, Souradeep Dutta, Kuk Jin Jang, Vivian Lin, Radoslav Ivanov, Oleg Sokolsky, and Insup Lee. Imprecise bayesian neural networks. arXiv preprint arXiv:2302.09655. 2023.
Michele Caprio, Yusuf Sale, Eyke Hüllermeier, and Insup Lee. A novel bayes' theorem for upper probabilities. In Fabio Cuzzolin and Maryam Sultana (eds.), Epistemic Uncertainty in Artificial Intelligence, pp- 1-12, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-55963-9
Matthew Chan, Maria Molina, and Christopher Metzler. Hyper-diffusion: Estimating epistemic and aleatoric uncertainty with single model. arXiv preprint arXiv:2402.03477 2024.
Kirill Fedyanin, Evgenii Tsymbalov, and Maxim Panov. Dropout strikes back: Improved uncertainty estimation via diversity sampling. In International Conference on Analysis of Images, Social Networks and Texts, pp. 125-137, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International conference on machine learning, pp. 1183-1192. PMLR, 2017.
Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-37, 2007.
Will Grathwohl, Kuan-Chieh Wang, Joern-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. In International Conference Learning Representations, 2020.
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3)3322-366 2020.
Sebastian Gruber and Florian Buettner. Uncertainty estimates of predictions via a general bias- variance decomposition. In International Conference on Artificial Intelligence and Statistics, pp- 11331-11354. PMLR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
11

---

Published as a conference paper at ICLR 2025
James B Heaton, Nick G Polson, and Jan Hendrik Witte. Deep learning for finance: deep portfolios. Applied Stochastic Models in Business and Industry, 33(1):3-12 12, 2017.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song. Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: critical analysis of out-of-distribution generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
Paul Hofman, Yusuf Sale, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty with proper scoring rules. arXiv preprint arXiv:2404. 12215, 2024.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:11 2011.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 110:457-506, 2021.
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are
bayesian neural network posteriors really like? In International conference on machine learning, pp. 4629-4640. PMLR, 2021.
Sanyam Kapoor, Wesley Maddox, Pavel Izmailov, and Andrew G Wilson. On uncertainty. tempering, and data augmentation in bayesian classification. Advances in Neural Information Processing Systems, 35:18211-188
2022.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Nikita Kotelevskii, Aleksandr Artemenkov, Kirill Fedyanin, Fedor Noskov, Alexander Fishkov, Artem Shelmanov, Artem Vazhentsev, Aleksandr Petiushko, and Maxim Panov. Nonparametric uncertainty quantification for single deterministic neural network. Advances in Neural Information Processing Systems, 01:36388-36322 2022a.
Nikita Kotelevskii, Maxime Vono, Alain Durmus, and Eric Moulines. Fedpop: A bayesian approach for personalised federated learning. Advances in Neural Information Processing Systems, 35: 8687-8701, 2022b.
Nikita Kotelevskii, Samuel Horváth, Karthik Nandakumar, Martin Takac, and Maxim Panov. Dirichlet- based uncertainty quantification for personalized federated learning with improved posterior net- works. Proceedings of the Thirty-Thile Third International Joint Conference on Artificial Intelligence, pp. 7127-7135, 2024.
A Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor I Butoi, Paul Bertin, Jarrid Rector-Brooks Maksym Korablyov, and Yoshua Bengio. Deup: Direct epistemic uncertainty prediction. Transactions on Machine Learning Research, 2023.
Balaji Lakshminarayanann, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I Sánchez. A survey deep learning in medical image analysis. Medical image analysis, 42:60-88, 2017.
Jeremiah Liu, John Paisley, Marianthi-Al Anna Kioumourtzoglou, and Brent Coull. Accurate uncertainty estimation and decomposition in ensemble learning. Advances in neural information processing systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance aware- ness. Advances in Neural Information Processing Systems, $3:7498-7511 2020a.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in neural information processing systems, 33:21464-21475 2020b.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations. 2021.
Sajjad Mozaffari, Omar Y Al-Jarrah, Mehrdad Dianati, Paul Jennings, and Alexandros Mouzakitis. Deep learning-basse vehicle behavior prediction for autonomous driving applications: A review. IEEE Transactions on Intelligent Transportation Systems, 23(1 1):33-47, 2020.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deep deterministic uncertainty: A new simple baseline. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 24384-24394, 2023.
Ahmet Murat Ozbayoglu, Mehmet Ugur Gudelek, and Omer Berat Sezer. Deep learning for financial applications: A survey. Applied Soft Computing, 93: 93:106384, 2020.
David Pfau. generalized bias-variance decomposition for bregman divergences. 2013. URL
http: htttp://www... http://w.avvvddd www.ddviidd.p. ccw http://sssttsssseeeeeeeeeee http////eettese//eeeeaaaaaaa wwwpeeerrriiddddooeo wwwrd__rrooo.r .d.f
Kajetan Schweighofer, Lukas Aichberger, Mykyta lelanskyi, and Sepp Hochreiter. Introducing improved information-theoretie measure of predictive uncertainty. In NewrlPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023a.
Kajetan Schweighofer, Lukas Aichberger, Mykyta lelanskyi, Günter Klambauer, and Sepp Hochre- iter. Quantification of uncertainty with adversarial models. In Advances in Neural Information Processing Systems, volume 36. pp. 19446-194844 2023b.
Artem Shelmanov, Evgenii Tsymbalov, Dmitri Puzyrev, Kirill Fedyanin, Alexander Panchenko, and Maxim Panov. How certain is your transformer? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1833-1840, 2021.
Dinggang Shen, Guorong Wu, and Heung-Il Suk. Deep learning in medical image analysis. Annual review of biomedical engineering, 19:221-244 2017.
Achille Thin, Nikita Kotelevskii, Stanislaass Denain, Leo Grinsztajn. Alain Durmus, Maxim Panov, and Eric Moulines. Metflow: new efficient method for bridging the gap between markov chain monte carlo and variational inference. arXiv preprint arXiv:2002. 12253, 2020.
Achille Thin, Nikita Kotelevskii, Amaud Doucet, Alain Durmus, Eric Moulines, and Maxim Panov. Monte carlo variational auto-encoders. In International Conference on Machine Learning, pp. 10247-10257. PMLR, 2021.
Evgenii Tsymbalov, Maxim Panov, and Alexander Shapeev. Dropout-based active learning for regression. In International Conference on Analysis of Images, Social Networks and Texts, pp. 247-258, 2018
Joost Van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. International conference on machine learning, pp. 9690-9700. PMLR, 2020.
Artem Vazhentsev, Gleb Kuzmin, Artem Shelmanov, Akim Tsvigun, Evgenii Tsymbalov, Kirill Fedyanin, Maxim Panov, Alexander Panchenko, Gleb Gusev, Mikhail Burtsev, et al. Uncertainty estimation of transformer predictions for misclassification detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 8237-8252, 2022.
13

---

Published as a conference paper at ICLR 2025
Artem Vazhentsev, Akim Tsvigun, Roman Vashurin, Sergey Petrakov, Daniil Vasilev, Maxim Panoy, Alexander Panchenko. and Artem Shelmanov. Efficient out-of-domalte detection for sequence to sequence models. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1430-1454, 2023.
Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty in Artificial Intelligence, pp. 2282-228229 PMLR, 2023.
14

---

Published as a conference paper at ICLR 2025
A WHY IS EPISTEMIC UNCERTAINTY GOOD FOR OUT-OF- DETECTION, AND TOTAL FOR MISCLASSIFICATION DETECTION?
Generally speaking, the out-of-distribution detection problem is not directly related to the conditional distribution p(y z). which is at the core of our paper and generally of supervised learning. The formally correct approach would be to consider some approximation of the covariate distribution p(x) and perform out-of-disrriutioon detection based on
However, in the literature, both aleatoric uncertainty (related solely to p(u x)) and epistemic uncertainty (associated with the quality of estimation for ply x)) are applied to solve the out-of- distribution problem and demonstrate certain performance in the task.
The success of aleatoric uncertainty that we observe in our experiments for "hard-OOD", might seem mysterious as, conceptually, it is not related to the covariate distribution at all. The empirical success of aleatoric uncertainty for detecting out-of-distribution related to the fact that the models are usually overconfident for in in-distribution objects, leading to very low uncertainty. At the same time, the model confidence for nut-of-distributt samples more arbitrary as nothing pushes the model to be confident in them. Thus, out-of-distrroo distribbiiii samples may have higher aleatoric uncertainty than in-distribution objects.
At the same time, the application of epistemic uncertainty to out- -itt----dittiittionn detection is more grounded, as one can expect that we have less knowledge about the actual dependence for out-of- distribution objects than we do for in-distribution objects. This relation can be formalized for some models, such as kernel methods. For example, the work by Kotelevskii et al. (2022a) shows that p(æ) epistemic uncertainty is proportional to the inverse of covariate density for Nadaraya- Watson kernel regression. Moreover, this method falls into the same risk decomposition, but built on another approximation.
For the Bayesian approximations we considered (except constant approximations), Excess risk can be seen "measure of disagreement" between predictions of the members of the ensemble. One may expect ensemble members to have arbitrarily different and diverse predictions for out-of-distribttion data, resulting higher values of Excess risk. Lots of literature on the topic has the same flavor of reasoning. For example, uncertainty papers about Mutual Information, EPKL (Malinin & Gales] 2021] Laakshinnaaaaanaai ynnan et al. 2017] Gal al., 2017 Schweighofer al.. 2023a), etc., are the particular case of our framework.
Similarly; total and aleatoric uncertainty measures are effective for misclassification detection because they capture the inherent ambiguity noise in the data associated with specific inputs. Aleatoric uncertainty reflects the variability in the model's predictions due to the inherent randomness or non-deterministic dependency between covariates and labels, which is common source of misclassi- fication. Therefore, when a model encounters an input that difficult to map to a certain class, the aleatoric uncertainty increases.
Measures of epistemic uncertainty effeetively capture the disagreement between different members or samples of Bayesian model. Hence, if there is a non-deterministic dependency between covariates and labels, epistemic uncertainty might increase as well, as different models may produce varying predictions for the same input. Therefore, the sum of both uncertainties namely, total uncertainty should be the most effective measure for detecting misclassification events.
B LIMITATIONS
We see two limitations to our approach.
Valid conditional 77x)) = p(y x)) for all x. This assumption implies, that regardless of the input I, the form of the probability distribution 77()) will not change. This means, that even for inputs, that do not belong to (X), the conditional should produce some categorical vector over the same number of classes. Let us consider an example of a binary classification problem, where we want model distinguish between cats and dogs (see Figure []. In this case, the distribution of covariates P., (X) is the distribution over images of all possible cats and dogs. An image a pigeon under this distribution should have a negligible probability. Now imagine, that somehow it happened
15

---

Published as a conference paper at ICLR 2025
dog, sufficient probability under Par. Conditional 17(z)) is meaningful.
cat, has sufficient probability under Pt.. Conditional 17(2) is meaningful.
A pigeon, has almost no probabil- ity mass under Par. Conditional my()) is vague.
Figure 1: The figure shows different examples of input objects in binary classification problem (cats vs dogs). The limitation of our approach is that n(x) (Y X =x) should be defined even for objects with tiny mass under Per (see discussion in Appendix
that x is actually an image of pigeon. Under our assumption, n7(x) should valid, so it should produce vector of probabilities over two classes: Cats and Dogs, despite input being an apparent out-of-distribution object. There no good way to define 77(x) for such input objects, hence we say it is vague. However, for unusual (but still in-distribution) inputs, like rare dog breeds, n)(x)) is meaningful.
Incorporation of Bayesian reasoning for estimation of In practice do not have access to (x)). Hence, we suggested approximating using the Bayesian approach and proposed two ideas to do it (inner and outer expectations). For Bayes risk the best Bayesian estimate is given by outer expectation (see Appendix D However, this is not the case for Excess risk. It appears (see discussion in Appendix D that Excess risk depends the estimate of the Total risk. But we never know in advance for a particular input x, in which regime (overestimated or underestimated Total risk) we are. Thus, we do not know what is the best choice for an approximation to epistemic uncertainty.
C DERIVATIONS OF DIFFERENT RISKS WITH PROPER SCORING RULES
We will start with the derivation of Total risk. In what follows, we will omit dependency for <(x:) and no (z)
K
R xx) = 1 ((no. dP (
(x) = Σ (G'((1))) no) - (%)) (G(o))))G)))))))))))())( m (G'( me) G(me) (G'()))) m)
Let us now consider Bayes risk:
RBayes(x) = ((n,))d(() x) Σ
(G((1), m) -2(()) - G(n)
A=1
(G'(1) n) - (G) 11) - G(n) = -G(m).
Finally, let consider Excess risk:
f(lne. [(ne, y)dP(y 2x) E(n. v)dP(y xx)=
R (10 (x) =
Total risk
Bayes risk
(G' (no), , no) - G(no) - (G'(no) 1) + +G(1)
G(v) G(naa -GG' (Ge) 11 - = Da Da(vilna)
16

---

Published as a conference paper at ICLR 2025
D DERIVATION OF RISKS FOR SPECIFIC CHOICES OF SCORING RULES
In this section, we will derive specific equations for Total, Bayes, and Excess pointwise risks to get the estimates of total, aleatoric, and epistemic uncertainties correspondingly. We will omit subscript 0 in this section, indicating an estimate by using a hat.
Recall equations for proper scoring rule and different risks:
[(n. i) = (G) n) G( - -G())
Tox (G) i) G(f) (G), 1))
R Bayes = -G(n),
RExe = G(n) - G(f) + (G)( i 9-1))
D.1 LOG SCORE (CROSS-ENTROPY)
G(n) Σ
Mhe log Ul
k=1
(n) = + log 7k
K
8(1. i) = (1 + log 1)
log 1
lk log 7/
k=1
K
K
log Th + 1 -
log
log m7 = - -log 17
Am
RTa
(1+ log Mi inc log îm (1 + log f1x))
1=1
K Σ
hú log nn
nE log nr
m log fnx) CE []]ii]]
K Σ
Bayes
77 log Th = Hly.
k=1
Hy = KL [7]11]]
RExe
Rio
Rayes
CE
D.2 QUADRATIC SCORE (BRIER SCORE)
K G(m) = Σ
mi (1 m7))
k=1
G'(1)) 27/ 1,
K
((n, i) = (2n) 1, 1)) -2m: +
+
mh mx)
k=1
2
1
21
1
1/
2
2m
1,
since constant does not affect optimization, we will use the following:
K
((1,
Σ
2 27p
17

---

Published as a conference paper at ICLR 2025
Tar
Σ 2nn
1)
Σ
iike
f(x) Σ
(2n 1)mm
2 + n mh) + 1 - 1 |1 + 1,
R Bayes
17 176) - 111m=112:
REsC Rio
Bayys
110 n ||n122 1 - 1---- 11 1 -
D.3 ZERO-ONE SCORE
G(r) max 1} 1,
G¹ G'(n) 1lk arg max 1751
8(1, i) = (T[k: = arg max Vi), 11) T[i = arg max 773} max 7/s +
j
max
II[i arg max mj] max 7 1 - Ili arg max nj} = Iīi * arg max nj},
j
Rio Σ
mallh = arg max mo] - msll[k = arg max mjll)
max AN +
max Íik
Yarg max,
max Íis: + Jarg max, fix
R Bayes
max Tek:
k
Rise
RIa
Bayes
Barg max, 7u 1 + max Te 7arg max, nh 'arg max, fl.
D.4 SPHERICAL SCORE
G(n)
||l|2 1,
G'(r)) = ||
T}
((
((,,))=(10)))
'Inll2" -1
n)
7
||l||a 1 ll
|17/12 +
=1 1171
|lml12
||7l12
K M.
RTx
Th IMll
11
+ +1 Σ
11
n
Irll
Mil
k=
R Bayes
1111122
RExc = Ryor Bayes
1 - |17/116
17
+ |lnl| - 1 -1
(1- 17
))
117/12
||11
18

---

Published as a conference paper at ICLR 2025
D.5 NEGATIVE LOG SCORE
K
G(1)
log Tk
k=1
1
("()))
K
K
(n. 1) = ( 1
1)
log 7) =
+
+
log 1]k
1)
1=1
since constant does not affect optimization, we will have:
()
log 7k:
T7
k=
R
ijs
log 1Tk
Uek
log Ui 1)
Ul
k=1
k=1
Bayes
log Tkk
RESC
RTo
Rayes
Thk VW
+
log în-
log 97
T3k
log
-1) T/
7] Dis [n]]
E IS THERE THE BEST APPROXIMATION?
We discussed the choice of the approximation for Bayes risk in the main part of the paper. Here, discuss whether there a best choice of Excess risk. If we know it, we can choose the best Total risk.
Recall that for Excess risk there are 9 possible options:
Expected Pairwise Bregman Divergence (EPBD):
(1.1) Exe
(x)) = E p(jD.... Bp.e) Do 11 vo))
Note, that since KL divergence is a special case of Bregman divergence, Expected Pairwise KL (EPKL (Malinin & Gales 2021 Schweighofer al. (023a)) is one of the special cases
of this Excess risk estimate.
Bregman Information (BI):
(1,2) Exc (x) p D..) (..) De na || nDD..)
R
DG
which special case is BALD (aal et al. 2017 Houlsby al. 2011 Reverse Bregman Information (RBI):
R (2,1)
(z) Ep. Do (p... |l (10
Exc
Its special case for Log score is known Reverse Mutual Information (Malinin & Gales 2021 Modified Bregman Information (MBI):
(1.3) Fxc
(x) Expe D...) De (10 (1 ll)
R
It is similar to Bregman Information, but the deviation is computed from the "central prediction", not central label (BMA).
19

---

Published as a conference paper at ICLR 2025
Modified Reverse Bregman Information (MRBI):
R
(3.1) Exc
(a:
pp D..) De n1)
This has similar structure to Reverse Bregman Information (RBI). Again, the deviation here is computed from another "central" prediction.
Forward Bias term:
(2.3) Exc
(a) Do (nD.. 11
It represents Bregman divergence between two different Bayesian estimates and can be viewed as a bias.
Reverse Bias term:
R (3.2)
Rxxe (z)= Do ADD.))
This applies similar reasoning the Forward Bias Term.
Finally, obtain similar constant measures:
REsc (x) Do (1 |1 O,
(3,3)
R
and
(2,2) R (x) = Da (nD. viD..) O,
which is coherent with the result obtained for the Total risk, when Excess risk (epistemic uncertainty) equal
However, it is not clear, what estimate of the Excess risk should use. Indeed, neither of these estimates are upper nor lower bounds for the true Excess risk. This is because contrary to Bayes risk, we don't 't have any idea if Excess risk with ground truth rj reaches any extreme. For another explanation, see Figure 2 and discussion in Appendix B For simplicity, we will consider only approximations with strategies (1) and (2), as for them we know, at least, which estimate of Bayes risk is better.
Rayee
RExc
Riot
(2) Bayes
R
Bayes
R
R Bayes
RExO
RTa
0
R
Bayos
R
Bayes
R
Figure 2: Different situations for risk estimates. Risks typed in black and above the axis are the true
ones. Risks, typed in color, and below are estimates. Two--poitted arrows show Excess risks.
(1)
Top. RoM underestimates Rio, Rayes better estimates RBayea and R R better estimates R
Bottom. Rio overestimates REa R better estimates Rayess, and R Rege better estimates Rexe- We
(1)
(2)
see, that for different estimates of RTa, have different best approximations for R See discussion
in Appendix B
In Figure 2 for simplicity, we consider only the Bayesian approximation of the first argument (ground- -ground-truth probability). In black, we have actual (real risks), while in color we have different estimates of risks. Also, tio-sided arrows, we show the Excess risk.
If we underestimate the Total risk (see top plot in Figure 2 the best choice for Excess risk will be (2) R()) as despite being a lower bound on Excess risk, is the best we can do (R in this case will
be even worse). However, if we overestimate Total risk, then there no single best choice. In the
20

---

Published as a conference paper at ICLR 2025
bottom plot, when Rio significantly overestimates Rio, the second idea for estimating Excess risk gives a better estimate, despite the first idea for Bayes risk still better.
Hence, the best estimate of Excess risk depends how well estimate Total risk. But never know in advance for a particular input æ, in which regime (overestimated or underestimated Total risk) we are. Thus, there is no best choice among these risks to approximate epistemic uncertainty.
F RELATIONS BETWEEN THE ESTIMATES
In this section, we discuss how the measures of uncertainty are connected. In the main text, we discussed several ways how one can estimate risk given an ensemble of models, posterior, or samples from it. In what follows, we show how one can further decompose these estimates of Excess risk.
(1.1) Let us start with Ryxe (I)) Using results of (Pfau 2013), we have:
Rixe (x) = Bp(o)D..) D.)) Da m7 [170)
ppt D..) Da (DD (o)) + p Da (naliDD..))
(2.1) R Exc (x) + R
Exc (x:).
(1,2)
Since all of these estimates are non-negative, the following holds true:
(1.1)
(2.1)
(2.2)
R
Exc
(x)
Rexc
(x) >
Exc (x) 0,
and
(1,1)
(1,2)
(2.2)
R Exc
(x) 2 R Exc
(x:) 2 R Exc (x) 0,
for any 2.
Moreover, one can show that the following holds:
(1.1) Tor (x) = Tor (x) = -R Bayes (x) + R Exc (x) = -Rpayes (x) + R (x))
(1)
(1.1)
=R
(2.1)
Rat (x) = R2(22) (x) = Rayes (x) + R (I) = -Raayes (I:) + RR (x),
(1.2)
(2)
(2,2)
(1)
R
(3.3)
(x) R Bayes (x) + R Exc
(3.3)
(I) = R Rayes (x) Rym (x)
(3.1)
RRxce (x),
(3.1)
Tar
R
Tox
(1,3)
(x))
R For
(2,3)
(x) R (x) R
(2.1)
Rexe (x).
(3,1)
From Pfau 2013) the following holds:
REsc (221)())) (z) = R (x) + Rexe
(2,3)
(3,1)
(9)
Additionally, Bregman information can be received as follows: (1,1)
BI(x) = Ryxc (x)
(22.1) - (x) = REse (x) (2.2)
(1,2)
(x) = RRnyes (x) RRyyes (x).
(10)
Reverse Bregman Information:
RBI(x) (22.1) (x) Exc
(2.2)
(x) (1.1.)
(x) RExo (x) = (1.1) R(1.1)(x)) (x) - (1.2)
(1.2)
(x).
Interestingly, the EPBD can be written in two equivalent forms:
(1.2) R (1,1) (x) = R (x: (2.1))
(x) (1,3) (x) + Rese (z).
(3.
(11)
An interesting observation from the (TT) is that in general there are two central points: central label and central prediction-wher the sum of expected deviations in terms of Bregman divergence lead the same result, known EPBD. To the best of our knowledge, this is a novel finding.
Now we are fully equipped to recover the decompasition, used in Gruber & Buettner (2023) (Equa-
tion (8), which is naturally appears, when using R (1.1) from the above, as well as Equation 9 and Equation (10
21

---

Published as a conference paper at ICLR 2025
G CONNECTION TO ENERGY-BASED MODELS
Here, we will consider specific case of the framework, instantiated for Logscore. In what follows, we will sometimes omit explicit dependency on æ for better presentation. However, this dependency is always assumed.
Recap, that for the special case of Logscore, Bregman divergence Kullback Leibler divergence. (3,1) (n no). Using results from Table Hence, Rxx
derive that:
log 7 = Eg log 1o log Σ exp Eg log no
Moreover fo(z) = Softmax -1 (2)), where T is temperature, that scales logits fa (x).
Hence, the logarithm of probability vector can further expanded:
exp fe
fo [
f0
log mo], log
log
exp
Eg
fo exp
T
T
From these equations, we can derive:
(3.1) Exe
E&KL(11
(10) = Eg
Th
log
=
Σ The log 77s
Σ 5h Eg log
1]8
Σ 1/8 Eo log
Σ
i7c log Σ
exp
Eo log no
Σ Tie Eo log 170
3
log Σ exp
Ro log 110
Furthermore, one may show that:
R
(3,1) - log Σ exp
Eg log no
-log Σ
-
exp
Bafo
T
1;
Eg log
Σ
exp
fo T
,
E; exp
Eafe 7
Egfe
f0 T
- -log
log
Σ
exp
+Ea log
Σ exp
exp Ep log
E, exp
4
fg
T
Eag
So Σ cxp
1)
Tlog Σ
cxp
TED log
T
T
T
Hence:
TR (x) = lolog Σ exp
(3,1)
Eefe
(-TTE log
fo Σ exp
T
T
B(x:Eafo)
ExE(s.f))
Therefore,
R RExc (x) = T
(3.1)
Eog
22

---

Published as a conference paper at ICLR 2025
Next, we can consider another approximation, namely R
Rgxe
(1.3)
EoKL( [1) =
Σ
Eo
110 log no
Eono log log mT]
So
So
Σ E no
EE log
Σ
exp
Σ
Eg log nol
+ log Σ exp
Eo log no
T
T
S0
Σ
Eo
n1
- Σ exp
Ep log
T
T
Σ Eo Bo
fo
log E exp
fo
[
log Σ exp
Eg log no
T
T
Σ Eo 110 for T
EEg log E exp
f0
EsnoEs
fo
Ey log
E exp
fo
+
T
T
T j
+ log Σ
exp Eo log 110)
Ecov(lr cov
[1o]x:
[ + log Σ exp Ei log 110
j
Σ cov
[[no]
[0
(3,1) RExso
Therefore,
(1.3) R
(3,1)
(1,1) Exc
Σ cov
[T10]
So 7 [1
Exc
Exc
Therefore, Expected Pairwise Kullback-Leibler divergence (EPBD in the partial case of Logscore) is equal to the sum of covariances, between predicted probability of a class, and the corresponding scaled (tempered) logit. To our best knowledge, it is the first interpretation of the result.
H TRAINING DETAILS
Training procedures for each dataset were similar. We used ResNet18 18 architecture. All the networks in the ensemble were trained entirely independently, each starting from a different random initialization of weights. They did not share any parameters.
For CIFARIO-based datasets, we used code from this repository: httpss:/gitthub com/ The training procedure consisted of 200 epochs with a cosine annealing learning rate. For an optimizer, we use SGD with momentum and weight decay. For more details see the code.
kuangliu/ httpll/tpppttttchiiiieeee
In Figure (left) we present performance summary statistics of the ensembles. Specifically, we show accuracy, macro averaged precision, recall, and F1-score.
For CIFAR100-based datasets, we used code from this repository: htttps://githb. com/
mar.ccunzaiipptorch...o ciffriillr.ooo The training procedure consisted of 200 epochs with learning rate decay at particular milestones: [60, 120, 160]. For an optimizer, we use SGD with momentum and weight decay. For more details see the code.
Similarly to CIFARIO, in Figure (middle) we present performance summary statistics of the ensembles of ResNet18 architecture.
For TinylmageNet. we used pre-trained models, provided by www..////AS-AIISS-AAatta..o
team They are provided with single training loss function. Training statistics for this dataset are in Figure
Please note, that for all these datasets, used different loss functions during training and various instantiations of G for the uncertainty quantification measures. The specific configurations are specified in the captions of the result tables.
23

---

Published as a conference paper at ICLR 2025
Chadbuiie Enderiaon
Endtrrio
Figure 3: Violin plots for different training loss functions and different metrics for ResNet18 Left: CIFARIO; Middle: CIFARI00; Right: TinylmageNet.
I DESCRIPTION OF DATASETS
1.1 NOISY DATASETS
In this section, we describe the noisy versions of CIFARIO and CIFAR100 datasets created for our experiments, namely CIFARIO-N and CIFARI00-N (N stands for "noisy").
In the datasets, the images the same as in the original dataset (covariates not changed), However, some of the labels are randomly swapped. Hence, only labels were changed, while covariates were kept in the original dataset. The motivation for the creation of this dataset due to the fact that conventional image classification datasets essentially contain no aleatoric uncertainty (Kapoor al. (2022). To mitigate the limitation, which is critical for our evaluation, introduce the label noise manually. By nature, this noise is aleatoric.
CIFARIO-N. We decide to do the following pairs of labels that are randomly swapped: to 7, 7 to to to 8, 8 to 3, 2 to 5, and 2.
CIFARI0O-N. We decided randomly swap the following pairs of labels: 1 to 7, 1, 3 8, to 3, 5, 5, to 2, 10 to 20, 20 to 10, 40 to 50, 50 to 40, 90 to 99, 99 to 90, 25 to 75, 75 to 25, 17 to 71, 17, 13 31, 31 to 13, and 24 42, 42 24.
Both there noisy datasets were used in training. When evaluating misclassification detection, we use original versions of these datasets.
1.2 IMAGENET DATASETS
Here we present descriptions of variations of ImageNet datasets we have used.
ImageNet-A. ImageNet-A (Hendrycks et al. 2021b) is dataset very similar to ImageNet test set, proven to be more challenging for existing classification models. It contains real-world, unmodified, and naturally occurring examples that are misclassified by ResNet models.
Closely related to ImageNet-A, ImageNet-O (Hendrycks et al. 2021b) is dataset that contains real-world examples, with classes that are not present in standard ImageNet dataset. It can be used to test out of distribution detection.
ImageNet-O.
This dataset contains renditions of 200 classes from ImageNet (Hendryeks et al., 202 Artistic rendition introduces significant distribution shift and present challenging task for classification models. It can be used to test robustness of model and out of distribution detection.
ImageNet-R.
J ADDITIONAL EXPERIMENTS ON OUT-OF- DETECTION
In this section, provide additional experiments diitt---iittibttinnn detection. Here. different instantiations of G for loss function and for uncertainty quantification, as well as try different in-distribution datasets. Results for CIFAR10 are in Tables and 7 For CIFAR 100, additional
24

---

Published as a conference paper at ICLR 2025
Table AUROC for out-of-distribttion detection for CIFARIO (in-distribution). As loss function for training and for uncertainty quantification we used Log Score.
CIFAR100
SVHN
TinylnageNer
CIFARIOC-1
CIPARIOC-2
CIPAR1OC-3
CIPAR1IC-4
CIFARIOC-S
LogSore R
90,93 +: 0.03
95.76 2 0.41
90.35 :: 0.07
60.66 0.11
67.44 0.11
72.14 :: 0.09
76,82 :: 0.09
W2,93 = 0.14
LogScore [1]
91.3 0.05
96.06 0.49
90.67 1: 0.04
60,8 1:
67.87 1: 0.09
72.11 1: 0.08
77.33 2: 0.09
83.48 ::
LogScore R
91.36 0:05
96.01 0.39
90.84 0.04
60.96 0.09
67.84 0.09
72.59 0.08
77.31 0.88
83.48 0.13
LogScore Reye
50.0
50.0
LogScore Rege
92.57
0.13
60.87
72.04
76.58
LogScore Rege
0.06
+ 0.09
72.54
71.22
LogScore Reve
91.95
+ 0.13
0.09
76.38 0.12
LogSoore Rive
+
50.0
LogSoore
R
90.06
93.92
89.19 + 0.07
61.06+ 0.11
0.09
72,45 + 0.08
77.1
82.93 0.2
LogSooee Riw.
89.99 + 0.08
89.12 0.07
61 0.11
67.79 + 0.00
72,43 0.08
77,07
82.9 0.2
LogScore Riw
90.38 0.08
94.35 0.94
89.57 1: 0.07
61.1 + 0.11
67.87 1: 0.09
72.55 :: 0.09
77.23 0.13
83.14
LogScoro Res
90.17 1: 0.06
94.08 1.02
89,32 1: 0.07
61.07 1: 0.11
67.82 1 0.09
72,49 1: 0.08
77.15 0.13
83.0 0.2
LogScore Rig
90.93 1: 0.03
95.76 21 0.41
90.35 1: 0.07
60.66 0.11
67.44 1: 0.11
72,14 4: 0.09
76.82 0.09
82.93 1: 0.14
LogScore Ry
90.95 =: 0.04
95.8 = 0.44
90.33 4: 0.06
60.7 +: 0.11
67.49 0.1
72,19 +: 0.09
76.88 0.09
83.0= 0.15
LogScore Ry
90.72 = 0.04
95.44 4:.44:2:00 0.53
90.03 +: 0.06
60.71 0.11
67.49 0.1
72.17 = 0.09
76.83 0.1
82.88 0.17
LogScore R
91.15 + 0.08
95.91 + 0.56
90.49 + 0.04
60.8 0.09
67.85 + 0.09
72.58 + 0.08
71.29 + 0.09
83.41 = 0.17
LogScore 1l0
91.3 + 0.05
96.06 0.49
90.67 + 0.04
60.88 + 0.1
67.87 + 0.09
72.61 + 0.08
71.33 + 0.09
83.48 + 0.15
1 Lggooore [1lx
90.81 + 0.04
95 39 0.71
90.06 + 0.05
60.96 + D.1
67.8 + 0.1
72,49 + 0.00
0..16+0111
81.18 0.19
LogSoore
91.15 + 0.05
95.91 + 0.56
90.49 +: 0.04
60.8 + 0.09
67.85 + 0.09
72.58 + 0.08
77.29 :: 0.09
83.41 + 0.17
LogSoore R
91.3 =: 0.05
96.06 : 0.49
90.67 1: 0.04
60.98 1: 0.1
67.87 1: 0.09
72,61 +: 0.08
77.33 0.09
83.48 0.15
LogSoore Rys 10.0
90,81 1: 0.04
95.39 0.71
90.06 1: 0.05
60.96 1: 0.1
67,8 1: 0.1
72,49 1: 0.09
77.16 1 0.11
83.18 0.19
LogSeore Bigr Bafo)
91,12 1. 0.02
96.69 0.34
90.68 1: 0.05
60.61 0.13
67.46 1 0.14
72.25 1: 0.12
76.97 1: 0.12
83,22 1 0.18
LogScore B, [e)
0.04
1: 0.13
0.13
0.11
76.96
83.21
0.16
results are in Table 8 For TinylmageNet, results are in Table In all these experiments, we computed energy scores only for Log score, as these terms naturally appear only for this instantiation.
Note, that for CIFAR1O and CIFAR 100, we used matching function G for the loss function and for uncertainty quantification. For
we had only one loss function, and we apply different instantiations of G only for the evaluation of uncertainty measures.
From all these tables one can observe, that Log Score-based measures typically outperform others, that justifies them popular practical choice in uncertainty quantification problems. Also, we see that for "hard-OOD" datasets, Bayes (and Total) risks usually perform better, than Excess risk. In "soft-OOD" cases, such as CIFAR1OC for CIFAR1O and ImageNet-O for TinylmageNet, their performance close, and Excess risk a good choice.
From the results for TinylmageNet() better, than other instantiations. Note, that since central prediction for Zero One Score is not well defined, excluded it from the table.
see, that matching combination of Log Score is usually
25

---

Published as a conference paper at ICLR 2025
Table 6: AUROC for out-of-distribution detection for CIFARIO (in-distribution). As loss function for training and for uncertainty quantification we used Brier Score. Note, that due to the symmetrical nature of Brier score. many instances results the same values.
CIBARTOU
SVHN
TingtnngeNet
CIFARTOC-1
CIFRROC-22
CIFARIOC3
CIHARTOC-4
CIEARIOC-5
BrierSoore Rgge
96.26
89.71 0.12
0.13
68.04:0.17
0.19
76822
$2.45
BrierScore Rges
90.38
0.18
96.26 + 0.34
89.71 0.12
61.02 + 0.13
68.04 0.17
72.46 = 0.19
76882 + 0.19
82.45 0.11
BrierSoore Rgg.
90.44 + 0.17
96.09 + 0.12
89.89 0.13
61.03 + 0.12
68.04 + 0.16
72:46 + 0.18
76883 + 0.18
82.47 + 0.1
BrierScore Res
0.0
50.0
50.0
BrierScore Ree
50.0
BrierScore Rys
89.24 2 0.19
94,19 0.49
88.37 0.12
60.11 0.15
66.59 0.24
71.19 0.3
15.7::0022 0.22
81.52 0.16
BrierScore Rys
50.0 + 0.0
50.0 0.0
50.0 0.0
50.0 +:
50.0
50.0 = 0.0
50.0 +: 0.0
50.0 1 0.0
BrierScore Rew
50.0 =: 0.0
50.0 0.0
50.0 0.0
30.0 4: 0.0
0.0
50.0 0.0
50.0 0.0
50.0 1: 0.0
BrierScore Rye
89.24 =: 0.19
94.19 =: 0.19
88.37 0.12
60.11 4: 0.15
66.59 :: 0.24
71.19 0.3
75.7 0.22
81,52 0.16
BrierScore R
89.24 0.19
94.19 + 0.49
88.37 0.12
60.11 + 0.35
66.19 + 0.24
71.19 + 0,3
7:.7++0.22
81,52 + 0.16
BrierScore Re
89.24 0.19
94.19 0.49
88.37 0.12
60.11 0.15
66.59 0.24
21 0.3
75.7 0.22
81.52 0.16
BrierScore
94.19 0.49
86.37 0.12
60.11
0.24
71.19
75.7
81,52
BrierScore (/53
96.26 0.34
89.71 0.12
+ 0.13
0.17
72.46
0.19
+ 0.19
82.45
BrierScore Ry
90.38 0.18
96.26 1: 0.34
89.71 = 0.12
61.02 1: 0.13
08.01:001
72.46
0.19
76.82 1: 0.19
82/15 0.11
BrierScore R
90.04 0.18
95.83 a: 0.34
89.29 0.12
60.99 1: 0.13
67.98 0.17
72.37 0.19
76.7
0.2
82.25 4 0.13
BrierScore R
90.38 + 18
96.26 + 0:34
89.71 0.12
61.02 0.13
68.44 0.17
72.46 0.19
76.82 0.19
82,45 0.11
BrierScore R (2:0)
90.38 a: 0.18
96.26 =: 0.34
89.71 = 0.12
61.02 à: 0.13
68.04
0.19
76.82 0.19
82.45
0.11
(3.1)
BrierScore Ryg
90.04 = 0.18
95.83 0.34
89.29 0.12
60,99 :: 0.13
67.8 0.17
0.19
76.7 0.27.00.2
82.25 2: 0.13
BrierScore R
90.38 0.18
96.26 = 0.34
89.71 0.12
61.02 2 0.13
72.66 0.17
76.82 0.19
$2.45
BrierScore R
90.38 0.18
96.26 0.34
89.71 0.12
61.02 0.13
68.04
72.46 0.19
76.82 0
82.45 0.11
BrierScore Ry
95.83 0.34
89.29 0.12
60.99 0.13
0.19
8.125
Table AUROC for out-of-distribution detection for CIFARIO (in-distribution). As loss function for training and for uncertainty quantification we used Spherical Score. Note, that due to the symmetrical nature of Brier score, many instances results the same values.
CIFAR100
SVHN
TinylmageNet
CIFARIOC.1
CIFARIOC-2
CIFARIOC-3
CIFARIOC-4
CIPARIOCSS
SphericatSeone R
90.09 0.04
95.78 0.64
8944 0.18
61.42
68.55 + 0.22
72886 0.26
77.3 0.31
82,89
SphericalSoone Rayes
90.42
96.22
89.11 0.18
61446
0.23
72.4
77.41
83.03 0.42
SphericalSeore Reess
90.46 1: 0.04
96.15 1 0.4
89.95 a: 0.19
6L,47 à 0.19
68.54 à 0.23
72.95 4 0.28
77.39 0.34
82.99 1: 0.43
SphericalSeere
1938
50.0 1: 0.0
50.0 :: 0.0
50.0 1: 0.0
50.0 0.0
50.0 1: 0.0
50.0 1: 0.0
50.0 0.00
50.0 0.0
SphericalSeore
R (5:0
$7.97 1 0.09
93.18 :: 0.55
87.2 1 0.18
59.2 1: 0.19
65.18 (3.18.100 0.2
69.61 1: 0.23
74,21 1: 0.22
80.13 0.25
SphericalScore
R
[5:11
$9.1 1 0.15
93.78 1: 0.66
88.27 1 0.2
60.54 1: 0.18
66.88 1: 0.26
71.35 1: 0.29
75.95 0.29
81.88 0.33
SphericalScore
(25)
$8.0 à: 0.17
93.06 =: 0.52
87.1 1: 0.13
59.29 0.11
65.18 à: 0.09
69.61 à: 0.1
74,24 1.244:0..11 0.17
80.39 0.21
SphericalScore
[2:2]
50.0 à: 0.0
50.0 = 0.0
50.0 4: 0.0
50.0 :
50.0 +: 0.0
50.0 d: 0.0
50.0 :: 0.0
50.0 0.0
SphesicalScore R [21]
89.03 +: 0.15
93.66 0.64
88.2 + 0.19
60.53 0.18
66.87 + 0.25
71.34 3:344:::00 0.29
75.93 d: 0.29
81.85 0.33
SphesicalScore
R [La:
89.12 + 0.15
93.77 0.65
88.29 + 0.2
60.54 + 0.18
66.8 + 0.26
71.36
75.96 + 0.29
81.89 0.34
SphericalScore
[1a:
89.17 0.14
93.86 0.66
88335 +
60:54 + 0.18
66.8 + 0.26
71.37 +
75.97 0.29
81.91 0.34
[1:
SphericalScore
89.11 0.15
93.77 0.66
88.28 0.2
60:54 0.18
66.8 + 0.26
71.36 + 0.29
75.96 0.29
81.89 0.33
SphericalSeure
R [530
90.09
0.01
95,78 0.64
89.4 1 0.18
61.42 0.19
68.45 1: 0.22
72.86 1: 0.26
77.3 1:
82.89
SphericalScore
8 [82)
90.03
0.05
95.7 0,65
89.33
0.18
61.42 0.19
68.44 0.22
72.84 0.26
77.28
82.86 0.38
SphericalScore
89.82
0.06
95.36 0.68
89.08 0.17
39
68.4
72.78 0.26
77.21 0:31
82.75
SphericalScore
[2:)
0.55
0.18
0.23
77.41
83.03
SphericalSeure
[2::
0.03
0,18
0.23
77.41
83.03 0.42
SphericalSeure Rar
[2:1
90.24
0.18
0.19
0.23
77.36
SphericalSeure Rgc
0.03
0.18
61.16
0.23
77.41
83.03
SphericalScure (1:2)
0.18
61.16103
0.23
77.41 0.33
83.03 0.12
SphericalScore Ris
0.03
0.18
6L.44.0..19
0.23
77.36
26

---

Published as a conference paper at ICLR 2025
Table AUROC for out-of-distribttion detection on CIFAR100 (in-distribution). For training and uncertainty quantification, we used: (Left) Log Score; (Middle) Brier Score; (Right) Spherical Score. Note that due to the symmetric nature of the Brier Score, many instances yield identical values.
Log Score
Brier Score
Spherical Score
CIFAR10
SVHN
CIFARIO
SVHN
CIFARIO
SVHN
Rayes
77.53 + 0.24
86.72 + 0.53
77.18 + 0.36
83.64 + 0.82
77.35 + 0.17
84.55 + 1.36
(24
Riaes
77.44 + 0.23
86.99 1 0.59
77.18 0.36
83.64 + 0.82
77.57 1 0.16
84.26 +: 1.56
(1)
Bayes
77.30 + 0.23
86.96 0.60
76.93 0.35
83.65 0.97
77.39 = 0.15
83.74 + 1.66
(3:3)
Rex
50.0 + 0.0
50.0 + 0.0
50.0 0.0
50.0 + 0.0
50.0 0.0
50.0 + 0.0
(3.2)
Rixe
65.33 + 0.17
66.83 1.92
50.0 0.0
50.0
57.90 0.44
57.16 + 2.88
(3.1)
Rig
72.68 1 0.13
75.98 + 0.72
63.26 1:51
61.34 - 1.70
66.32 0.46
69.07 +: 2.06
(2.3)
REw
62.97 0.20
63.81 + 2.00
0.0
50.0 0.0
57.11 0.45
56.03 + 2.86
(2:22
5.0
50.0
5.00
50.0 0.0
50.0 0.0
50.0 + 0.0
(2,1)
Rixe
0.13
74.17
6.52
61.34 1.70
63.94
65.66 + 2.25
(1..))
71.42 0.10
74.61 1.06
63.26
61.34 1.70
64.77 0.48
66.78 1: 2.23
Rgw
0.13
77.06 + 0.79
63.26
61.34 = 1.70
66.97 = 0.43
69.85 + 2.02
Rix
72.08 + 0.12
75.30 + 0.88
63.26 0.51
61,34 + 1.70
65.30 + 0.48
67.55 + 2.18
Rips
77.53 + 0.24
86.72 + 0.53
77.18 0.36
83.64 + 0.82
77.35 + 0.17
84.55 +: 1.36
(3,2)
77.55 + 0.24
86.74 + 0.52
77.18 + 0.36
83.64 + 0.82
77.08 + 0.18
84.31 +: 1.29
(3.1)
77.50 + 0.24
86.49 + 0.50
76.46 0.38
81.88 = 0.48
76.78 + 0.20
84.20 + 1.16
(4.3)
77.41 + 0.23
87.00 + 0.58
77.18 0.36
83.64 0.82
77.62 0.15
84.40 + 1.53
(2.2)
Ris
77.44 1: 0.23
86.99 0.59
77.18 0.36
83.64 + 0.82
77.57 0.16
84.26
Rip
77.39 + 0.24
86.77 + 0.55
76.46 0.38
81.88 + 0.48
77.65 + 0.16
84.75 + 1.44
RTw
77.41 = 0.23
87.00 + 0.58
77.18 + 0.36
83.64 = 0.82
77.62 + 0.15
84.40 + 1.53
(1,2)
77.44 + 0.23
86.99 + 0.59
77.18 + 0.36
83.64 + 0.82
77.57 0.16
84.26 + 1.56
R
77.39 1: 0.24
86.77 0.55
76.46 0.38
81.88 0.48
77.65 + 0.16
84.75 1: 1.44
E(x; [ol)
77.05 + 0.32
87.98 + 0.65
Es [B fo)
76.71 + 0.32
87.71 0.69
27

---

Published as a conference paper at ICLR 2025
Table 9: AUROC for out-of-distribution detection for TinylmageNer (in-distribution). As loss function for training, we used Proos-Entropyycorrepooo to Log Score). For uncertainty estimates, used different options. From left right: Log Score, Brier Score, Spherical Score, Zero One Score. Note, that ImageNet-O can be considered as "soft-OOD" for Tiny LinylmaeNNet. Note, that due to the symmetrical nature of Brier score, many instances results to the same values.
Lag Soor Noe
Beke Soors Neu
Syberical Scare
Zeno One Scace ImageNe
IraayNMe.A
O
ImageNNesA
OrageNee
IragsNeeck
TrageNee
ImageNen
IrugsNee
ImageNee
600
0.2
233
B007
89.76 0.2
52.77 0:17
75.18
$1116 2014
$1.10 1174
713 0.18
$1.16 103
$2.14 11.14
715 015
13.11 1125
B1A1 1031
75 316
30.8 0:11
10:11
90.00000
500 1.0
0.35
73.25-13
0:225
77.02 0.17
74.46 0.22
63.46 = 3.30
65.99
68.81 0.35
76.55
76.16:0.12
13.15:00.22 0:22
6230
0.37
0.366
00
5:0:000
02:
0:055
6330
50.0
500
0.0
5000:00
0:36
0:3
5407 0..8
03.46 0.30
6559 0.37
0:25
00:4::03
70.78 0.23
0:.44:0.:4
57.15 0.31
67.28 0.19
03.96 0.3
76.11 2:4.5
75.35 8:21
13394 11.3
63.46 3.30
65:9
0:25
72
01336:0021
0.34
26.52 8/19
34.0::10 (138
0/0
1:35
7131 1139
7136-1-021
1015 01:2
06.1
32.67
1:06
83.16 0:24
7.3.1.0.5
8000 013
70.33
83.77 0.2
82.84
0.36
73.3+-0.22
33.16 0.24
02.14::0 0.34
73.5+0018
9000 0/44
0:221:0025
53.88
0:13
75 3:35
01:00
03
53.76 0.04
12.78=0007
75 0.02
83 0:24
03:4::11
733
1 0M
03:4:004
033
17.41 0:3
0:31
56.25 2:25
83.32 0.31
14:0
51.35 = 325
$108
0:63
33.25 02)
033
0.11
12.52 023
01
2.1
033
83.75
23.15
233
0314::004
013
12
23
0.31
1400 0.11
81.35 025
$138 922
1463.2021
8322 0.23
13.27
75:85:0.00
12.52 0.3
$1
23.35 0:19
0-036
6:045
1.115:034
28

---

Published as a conference paper at ICLR 2025
Table 10: AUROC for misclassification detection. As a loss function for training and for uncertainty quantification we used Log Score.
CIFARIO
CIFARI00
CIFARIO-N
CIFARI00-N
TinylnageNet
LogScore R
94.76 4: 0.09
85.95 =: 0.31
80.44 3.69
83.41 =: 0.23
86.55 a: 0.26
LogScore R
94.7 0.05
85.13 0.35
78.36 4.77
81,92 A: 0.36
85.5 0.23
LegSoore Ree
0.24
LogScore Rpx
50.0 +
50.0 = 0.0
50.0 +: 0.0
50.0 + 0.0
50.0 + 0.0
LegSoore Rgc
92. + 0.19
70.71 0.38
61,28 6.53
65.51 + 0.99
71.77 + 0.13
LogScure R (0.)
94.4 1: 0.13
82.62 =: 0.34
69.98 6.08
75.51 d: 0.8
82.74 =: 0.34
LogScore R (4.0)
91.54 4: 0.19
67.42 à: 0.4
59.24 2:: 6.11
62.79 d: 0.97
66.83 =: 0.11
LogSoore (2.0)
50.0 +: 0.0
50.0
50.0 0.0
50.0
50.0 ::
LogScore R
(2.0
94.01 1: 0.14
80.69 0.35
67.7 6.09
73.7 0.86
80,18 :: 0.31
LogScore R
93.72 1:
80.3
66.85 5,91
73.37 :: 0.87
79.25 0.25
LogScore Rpx
94.23 0.14
+
75.74 0.78
89.22 0.28
LogScore Ax
9411
0.83
LogScore Rgs
80.44
LogSeore R
94.77 1 0.05
85.96 = 0.32
80.21 3,96
83.33 :: 0.26
86.5 0.25
LogScure R
94.74 1: 0.07
86.22 0.31
79.38 3.85
83.18 1: 0.29
86.65 0.26
LogScore Rg
94.4 0.06
85.01 0.37
76.26 5.04
B1.67 1: 0.39
85.18 d: 0.23
LogScore Rig
94.7 + 0.05
85.13 = 0.35
7W.36 +: 4.77
K1,92 +: 0.36
K5.5 0.23
LogScore R (00
94.5 +: 0.06
85.43 = 0.35
75.73 1: 4.68
81.67 = 0.41
K5.58 :: 0.22
LogScore R (1.b)
94.4 + 0.06
85.01 0.37
76.26 d: 5.04
81.67 = 0.39
85.18 =: 0.23
LogScore R (1.4)
94.7 + 0.05
85.13 = 0.38
78.36 4.77
81,92 + 0.36
85.5 0.23
LogScore R. (Na
91.5 0.06
85:43 0.35
75.73 4.68
81.67 :: 0.41
85,58 :: 0.22
LogScore Eo.
0.23
LogScore EgEis; [a]
93.38 4: 0.15
82.07 0.34
74.61 5.23
76.12 =: 0.63
82.4 0.24
K ADDITIONAL EXPERIMENTS ON MISCLASSIFICATION DETECTION
In this section, we present additional results for misclassification detection.
As training datasets, we consider CIFARIO, CIFAR100, their noisy versions as well as TinylmageNett (only with Cross-Entropy loss function).
In Table 10 we present results for all these datasets, when loss function and instantiation are generated by Log Score. In Tables 11 and 12 we present results for CIFAR-like datasets, as for TinylmageNet we have only Cross-Entropy loss function. From all these tables we see that, indeed, Bayes risk and Total risk are the best choices when encounter misclassification detection problem. The gap is even more significant, when more label noise is introduced. Moreover, instantiations for Log Score are typically provide better AUROC, than others. Similarly to the out-of-distribution detection, it justifies typical practical choice for Log Score-based measures.
For TinylmageNet, results presented in Table 13 Similarly to Table 9 one loss function used for training, while different instantiations of G were used for the measures of uncertainty. Interestingly, the best results reached by Zero One score, that is also popular choice of practitioners.
29

---

Published as a conference paper at ICLR 2025
Table 11: AUROC for misclassification detection. As loss function for training and for uncertainty quantification we used Brier Score. Note, that due to the symmetrical nature of Brier score, many instances results the same values.
CIFARIO
CIFARI00
CIFARIO-N
CIFARI00-N
BricrScore Riyee
94.68 = 0.3
86.02 = 0.26
80.64 3.17
84.1 = 0.13
BrierScore Rayes
94.68 + 0.3
86.02 + 0.26
80.64 + 3.17
84.1 + 0.13
BrierScore Bayes
94.22 1: 0.27
85.09 1 0.27
79.2 2.35
83,95 1. 0,14
(3.3)
BrierScore Rex:
5.0
50.0 0.0
50.0 0.0
50.0
BrierScore (3.2)
50.0 0.0
50.0 + 0.0
50.0 + 0.0
50.0 0.0
BrierScore Rese
94.2 1: 0.34
72.71 1: 0.34
76.31 1 3.39
65,34 1.441100 0.43
BrierScore R Exs
(2.3)
50.0 0.0
5..0
50.0 = 0.0
50.0 0.0
BrierScore E45
50.0++00 0.0
50.0 + 0.0
50.0 0.0
50.0 0.0
BrierScore Ripe
94.2 1
72.71 0.34
76.31 + 3.39
0.43 0.43
(1:30
BrierScore Rixe
94.2
72.71 0.34
76.31 3.39
65.34
(1.2)
BrierScore Rea:
94.2 0.34
72.71 = 0.34
76.31
65.34 0.43
BrierScore (1.1)
94.2 0.34
72.71 + 0.34
76.31 + 3.39
65.34 0.43
BrierScore R (3,3)
94.68 + 0.3
86.02 0.26
80.64 3.17
84.1 1 0.13
(3.2)
BrierScore R Tbc
94.68 1 0.3
86.02 0.26
80.64 3.17
84.1 1 0.13
BrierScore R
94.61 +: 0.31
79.78 3.23
0.19 83.24 1: 0.19
BrierScore R toc
94.68 + 0.3
86.02 + 0.26
80.64 + 3.17
84.1 = 0.13
BrierScore Ryg
94.68 1. 0.3
86.02 1 0.26
80.64 1 3.17
84.1 1 0.13
BrierScore Ric.
94.61 0.31
86.11
79.78 3.23
83.24
(1.3)
BrierScore Rio
+ 0.3
86.02 + 0.26
80.64 3.17
84.1 + 0.13
BrierScore R (1.2)
94.68 1 0.3
86.02 1 0.26
80.64 3.17
84.1 1 0.13
BrierScore R Toc
(1.1)
94.61 0.31
86.11 + 0.23
79.78 3.23
83.24 + 0.19
Table 12: AUROC for misclassification detection. As loss function for training and for uncertainty quantification we used Spherical Score.
CIFARIO
CIFAR100
CIPARIO-N
CIFARI00-N
SphericalScore R
91.01 :: 0,31
85.37 2: 0.25
79.36 : 2.85
80.13 1 0.6
SphericalSeore
R
Riss
94.15
0.29
84.73 0.24
80.8 :: 3.0
80.53 0.54
SphericalScore
(1)
93.64
SphericalSeore
R (9.)
50.0 0.0
50.0 0.0
50.0 = 0.0
50.0 =: 0.0
SpheriealScore
R (42)
92.81 0.49
65.29 0,48
69.95 : 5.62
61,62 62 0.33
SphericalScore
R (3.0)
93.46 0.47
77.3 4: 0.35
75.18 3.6
71.81 0.46
SphericalScore
92.76
60.8
SphericalScore
+
SphericalScore
93.39 =: 0.47
73.76 0.36
75.06 :: 3.53
68.84 0.42
SphericalScore
93.51 + 0.47
75.08 0.36
75.22 + 3.63
69.97 0.42
SphericalScorre R
93.57 1: 0.46
78.25
0.34
75.32 1: 3.69
72.69 1: 0.44
SphericalScore
R (1.33
93.49 0.47
75.83 0.35
75.2 3.61
70.59 0.43
SphericalScore 8
94.01 0.31
85.37 0.25
79.36 2.85
80.13 0.6
SphericalScore
93.98
SphericalScore
93.92 : 0.31
85.42 0.24
79.21 :: 2.79
80.13 11 0.61
SphericalScore
94.15 + 0.29
84.88 0.24
80.66 + 3.03
80.6 0.54
SphericalScorre
94.15 4: 0.29
84.73 4: 0.24
$0.8 1: 3.0
80.53 1: 0.54
SpheriealScore
R 2.43
94.14 0.3
85.23 0: 0.23
80.15 :: 2.99
80.82 0.55
SphericalScore
R (1a)
94.15 0.29
84.88 0.24
80.66 4: 3.03
80.6 0.54
SphericalScore
R (1:3)
94.15 0.29
0.54
SphericalScore
R
30

---

Published as a conference paper at ICLR 2025
Table 13: AUROC for misclassification detection (TinyImageNet). As a loss function for training used Cross- Eetss-E. Here, we try different instantiations of for uncertainty measures. Note, that due to the symmetrical nature of Brier score, many instances results to the same values.
Log Score
Brier Score
Spherical Score
Zero One Score
(3)
86.55 + 0.26
86.8 + 0.28
85.83 + 0.36
Bayes
85.5 + 0.23
86.8 + 0.28
86.8 + 0.28
87.23 + 0.35
Bayes
(1)
84.99 + 0.24
85.89 0.27
85.76 + 0.26
85.88 0.27
Bayes
(3,3)
Exc
50.0 = 0.0
50.0 0.0
50.0 + 0.0
(3.2)
71.77 = 0.13
50.0 0.0
81.82 + 0.47
(3.1)
82.74 1: 0.34
75.91 0.59
83.6 0.47
(2,3)
R Rxxe
66.83 1 0.11
50.0 0.0
78.32 1 0.58
(2,2)
R Rxxe
50.0 + 0.0
50.0 0.0
50.0 1 0.0
50.0 0.0
(2.1)
Rexe
80.18 + 0.31
75.91 0.59
79.62 1 0.54
73.59 0.44
(1,3)
Rexe
79.25 + 0.25
75.91 0.59
80.73 1 0.54
(1,2)
Rexe
83.22 1 0.28
75.91 0.59
81.95 1 0.55
79.52 0.68
R (1,1)
Rexe
81.33 + 0.29
75.91 0.59
81.02 1 0.54
76.78 0.56
(3,3)
R Tot
86.55 = 0.26
86.8 + 0.28
85.83 + 0.36
(3,2)
Tut
86.5 = 0.25
86.8 + 0.28
85.39 + 0.37
(3.1)
Tot
86.65 = 0.26
86.4 + 0.32
85.43 + 0.39
(2.3)
R Tot
85.18 = 0.23
86.8 + 0.28
86.9 0.29
(2.2)
Tot
85.5 = 0.23
86.8 = 0.28
86.8 0.28
87.23 0.35
(2,1)
R Tot
85.58 = 0.22
86.4 + 0.32
86.92 + 0.3
86.95 + 0.31
(1,3)
85.18 = 0.23
86.8 = 0.28
86.9 0.29
(1.2)
R Rot
85.5 = 0.23
86.8 + 0.28
86.8 0.28
87.23 + 0.35
(1.1)
Roo
85.58 1 0.22
86.4 1 0.32
86.92 1 0.3
86.95 0.31
E(a; Eofo)
83.34 + 0.23
Ex; fo)
82.4 + 0.24
31

---

Published as a conference paper at ICLR 2025
L DERIVATION OF CENTRAL PREDICTIONS
Central prediction is defined as 7j = arg min, E. (= For different instantiations of G, there will be different central predictions. In what follows, we will abuse the subscription 0, and simply write E and 77, assuming expectation with respect to 0.
Recall, that Bregman divergence is:
Do((lln) = G(z) G(n) (G'(m). 1))
L.1 LOGSCORE
K
G(n) = Σ m log fkk k=1
(n(r))= 1 log A|,
1 arg min
Σ 24 log 2k
E Σ
Mh log 7k
(1 + 1: 2) + E(1 + log 77, 178
arg min
Σ
2g log 2
E Σ
mh log m7s
Σ 2k log (exp log n), + Σ Th log 17
k
arg min
Σ
2 log 2/
Σ 24 log( E log Flg 1)
arg min
24 log 2k
2 logg exp E loggn) n)
k
Σ Fk
log(exp (eeeepEllog n)
log (exp E (oxpexEElogg) n) m²]
2
hr'
(exp E l)g
(exp IE log n)
arg min
Σ
k log 2
2x log
arg
KL
Σv' (exp E log 11)
Σ Ex (exp E. log 7)
h
=
(exp E log v)
hence, T7
Ex 1xxp E. log 1) 2/
I.2 BRIER SCORE
K
G(1) = Σ
Ti (1 1T),,
G'(1)) k=
217 - 1,
n arg min Σ Zu - -xx) + E
Σ
mi - -ma) - (21En - 1, 2) + [(21 - 1, 1,1
k
arg min
Σ (1 - 2x) + E
(1x(1 (1 - 71)) 2 Σ EyEE + 2F Σ via
A
b
K Σ
arg min
2 2k
E
Σ
mh
2
2
Σ
Z4 ETk + 2E
Σ
mk
21
k
k
arg min
2 2k
ZRET Σ
11
2
arg min
Σ
#
2
Fg. Eyjs + E
11
(EMJs) )2
Σ
(ETJa) )2
k
b
arg min
|| Enl1
+ E Σ 2 >
EmTHy )2]
arg min ||0
Enl
k
hence, πj En.
32

---

Published as a conference paper at ICLR 2025
L.3 ZERO-ONE SCORE
G(T)) max 7)
('(n)) [[ arg max n3x
max 77 + (ara max 1751 2- 11)
n = arg min max 2k
arg min E
max 2p max T/e
17) larg max, max, mi
arg min IE max z1 Zary max,
arg min max (2, Ell (arg max my])]
aragggm min max = - (z, p}}
arg arg/i(z, I|arg max 2n] P) O,
where p Ellg max, nyy are empirical frequencies of class labels, predicted by models, parametrized by samples from p(0 D.r).
We see, that minimum is always reached with r Uniform(K), where Uniform(K) is uniform categorical distribution over K classes. Please note, that there might be infinitely many solutions, when at least one of the components of p O.
Let us assume that there are no zero components, hence uniform distribution will be the only solution.
However, uniform distribution plug-in violates general decomposition in equation
This might be due to the requirement, that G must be a strictly convex, while G = max is only convex, and effectively operates with the single (maximal) component of the categorical distribution 7/. while undetermine others.
L.4 SPHERICAL SCORE
G(n) ||l|22 1,
G'(n))
||l
ij = arg min | || +
77
2)
|
arg minE |21|| < 111l2
z))
arg min |21|2 (z, E 17
||7/122
Let us introduce following notation:
f(z m) = arg min |2||2 - (z,
[|nll
?E |
E E
K
= Il| E gK
s.t. x
||
Ta = Ta R s.t. 1 Fo. |x1l|| = 1,
= E R 1 Fo,
F1, ||a]|a
33

---

Published as a conference paper at ICLR 2025
Then say, that there exists knp k. ki such that:
D = 21 + Ri 2 To + kr + kig,
[(kn Ki ka) 11:00 + ko + hiaall2 - (20 Kr + kaa, 211 + ki
(xo)? + +ha > (xLL) + k2 Σ af - (kip + (1v,, x1)
Σ (zo) + +12 + k.a k (ro m3)
Let us takes derivatives of f()) w.r.t ki kig to find values, that minimize it:
(k(g. k kia)
ka
0,
dkia
v (x0) ) + 22
22 kn =0= 0,
E(Oo) (20)3 + k2 +
= Ke = 0,
(Kin: k kia
dk
Σ (TO)² + k22
2== k
(x0)² + +k²).
kp 10012
1² - k = | ro 22 h: =
31 = (MI zo)
||00| 12
k. = 1.l1111l1 = Inn 221122 = =|17= (ME, mo)
112-
|
2
Finally we have:
k (NE
mr)) = xo + TE I
2 == 3o + K kaa = x0p + kin
|
k2
ME - (ME 20)) Troll
Fo +
|
|| (NE moo
T
Setting
|| and m = ME {ME, ma) ا
we conclude:
= 110ol12 n +
m
m
M TOY EXAMPLE
In this section, we consider a specific toy example, in which uncertainty measures can be computed in closed form.
Specifically, we consider the problem of binary classification, where the likelihood (predictive model) is simply Bernoull distribution, and the prior distribution over model parameter (the probability of success) is Beta distribution. Therefore, prior and likelihood are conjugate, and the posterior is Beta distribution well.
34

---

Published as a conference paper at ICLR 2025
Expected Pairwise Kullback-Leibler (EPKL). Here, want to estimate
Eg- (( = Eg- (0_ log 0*] - Eg- [00] ED [log 0] + R.g- [(1 - log) - -8*)] - Eg- [(1 [o 0)]
assume that p(0" D) Beta(a" ( and p(0 D) = Beta(a;, 3). Each of the components can be computed analytically:
1. Egg [0" log 0-] a0 [vbla" + - 18(0" + + -0++1 +1];
2. Eg. 0 0'
3. Ey log 0 = ((a) vb + B);
4. Eg. [(1- ((1-8)logg( ]*] 8" a-+3 (0 +1) v0a" + + 1)];
5. Eg- (1 - 8r)) a'
6. Eg [log(1 - 0)] (B3 1b 3).
If assume, that a and 3 B (the setup considered in the main part), then one can show, using these six equations that
1 EPKL
a + 3
Note, that this assumption (on the equal parameters) done for simplicity. In general case, one indeed consider different Bayesian models for the approximation of the ground truth distribution and for the approximation of the prediction. But since it is done at the cost of training separate Bayesian model, we believe might be not of big practical value.
Mutual Information (MI). as:
Following the derivations from the main part, MI be represented
((0)())))
Eg. [0' log 0*] - Eg- [0]] log Eg0 + Eg. [(1
log(1
(1]0) -Eg. [(1 - 0r)] log(1 Eg0))
Again, we use the same assumption on the equal parameters of the posterior distributions. With the equations from the above, obtain:
1 a 3 3 a ((a)) log [V((3)
MI
- (000) +og log(a +,).
Reverse Mutual Information (RMI). As showed in the main part, RMI be presented as follows:
Eg L 0'1
Eg- [+]] [oo Eg-0*] Eg- [00] E. [log +) (1 1-------0)0 E') [log(1 0)]
-T-
Following the same methodology as for previous cases, we obtain:
((B)+ - log(a B). RMI + (igga-((a++dodn + a + (
is easy see, that EPKL MI RMI (which consistent with the general result that R
(1.1)
R (1,2)
(2.1)
Exe + R
35

---

Published as a conference paper at ICLR 2025
Non informative Prior
Underestmating Prior
Correct Prior
Parener jostt Paemene joon
Farete joot) Frome poikig
Danis
23
00
015
00
(a) Uniform prior. a =
Underestimating prior.
(c) Correct prior. a 9.
3, 3 17.
Figure 4: Different shapes of the posterior distributions.
Expected Pairwise Brier Score (EPBS). In this case, would like estimate the following quantity:
E(0" - 0)2 = Eg- [0-]² 2gg [00) Re [0] E. [o]
If we assume, that the parameters of the posterior distributions are equal, then the whole equation above reduces to the double variance of In case of Beta distribution, it equals to:
20
EPBS 2var0
(a + () 1)
Note, that the posterior update rule for this simple Beta- Beta-Bernoulli model is:
a Opprior + C,
and
B = 3prion + 71
where by subscript "prior" highlight the parameters of the prior distribution, is the number of successes in observed training dataset, and n is the total number of observations.
Therefore see, that given larger training dataset, the estimates of epistemic uncertainty will shrink. This is the property we typically want to have from the epistemic uncertainty estimates - the bigger the dataset, the less uncertainty.
However, case of the prior misspecification (will be discussed below). leads to certain problems. the prior parameters Oprior and Sprior are already large (indicating highly concentrated prior), the posterior will also be highly concentrated, regardless of the data. This can lead to an underestimation of epistemic uncertainty because the model appears more certain than it should be.
M. VARIOUS DISTRIBUTION SHAPES
In this section, we explore different choices of the parameters and demonstrate, how the resulting posterior distribution will look like, given different prior and different sizes of training datasets. We show the resulting plots at the Figure Note, that as a ground -oound-truth we used 6 = 0.9. We see, that given enough data, the mass of the posterior is concentrated around the correct value of the parameter, regardless of the prior misspecification.
M.2 PRIOR MISSPECIFICATION
In this section, we demonstrate, how the metrics of epistemic uncertainty behave, given different (possibly misspecified) priors. We present the results in Figure 5 We see, that as we discussed above, these metrics depend on the values if and 3. Therefore, the more these values are, the less the estimate of epistemic uncertainty. Since uniform prior contains minimal evidence, demonstrates maximal uncertainty, and therefor less vulnerable for prior misspecification. Note, that in our experiments on deep ensembles, we considered uniform prior over parameters, therefore the effect of prior misspecification should be negligible.
36

---

Published as a conference paper at ICLR 2025
Expected Pairwise KL Sample Size
Expected Pairwise Brier Score
Sample Size
Expected KL Divergonce
0.125
Correct Prior
Prior
0.025
Prior
Plicr
Undovatinatin
0.020
Uscerraimating
Non interotivv Pricr
Varlance
Nor-informaton Pror
0.01
0,073
0.010
0.050
0.005
0.025
0.000
60
80
20
100
Sample Size
Sample Size
(a) EPKI. different prior choices.
(b) EPBS for different prior choices.
Mutual Information vs. Sample Size
Reverse Mutual Information vs. Sample Size
0.06
Prior
Prior
Undorssinnaing Prior informativee
0.05
Ueverstinating Piñe
Nor-infomative
0.04
RMI
0.04
90
40
60
40
160
20
40
(
80
100
Sample Size
Sample Size
(c) MI for different prior choices.
(d) RMI for different prior choices.
Figure Epistemic uncertainty metrics, given prior misspecification and different samples sizes.
M.3 ERROR POSTERIOR APPROXIMATION
In this section, we consider the case of choosing wrong posterior distribution for inference. We know, that our toy example, the correct family consider for inference is Beta. However, we will misspecify the choice and consider Normal distribution instead. We use two approaches for inference. Namely, Laplace Approximation (approximates the posterior around the mode), and Moment Matching (matches the mean and variance of the Beta distribution with Gaussian).
We present results in Figure 6 We considered uniform distribution as a prior, and we used training datasets of sizes 20 and 100. As expected, we see, that the given less data, errors in approximations are more severe. However, if bigger dataset provided, the approximation becomes more accurate.
Exact Beta Posterior Gaussian Approximations
Exact Posterior Gaussian Approximation ations
Exact Beta Postorio Moment Manching Gaumin
Exst Beta Poserie
Momeet Matching Countan
Laploce Aporodmation
10
Laplace Apprommaticn
Dessity
0.3
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.88
1.0
9
6
(a) Approximations for n=20.
(b) Approximations for n= 1=100.
Figure Resulting approximations given different sizes of training datasets.
37


Reviews:
[{"review_id": "PksRNNJd97", "reviewer": "Reviewer_fVqT", "summary": "The authors introduce a framework for quantifying uncertainty by decomposing it into aleatoric and epistemic. The authors use Bayesian methods to approximate and derive a unified framework for generating uncertainty measures. The framework is validated through experiments on classification tasks for image datasets, specifically on out-of-distribution detection.", "questions": "Does the loss function to quantify uncertainty and the training loss function need to be the same?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and clear.\n2. The paper presents a solid theoretical contribution by unifying different predictive uncertainty measures under a Bayesian risk decomposition framework. It connects well-known uncertainty quantification methods (e.g., Mutual Information, Expected Pairwise KL Divergence) with a common theoretical foundation, which adds clarity and depth to the topic.", "weaknesses": "### 1. Limited Scope of Experimental Section\n\nThe paper's experimental section only considers image classification tasks, which have been extensively researched in the uncertainty community. It could be strengthened by:\n\n   - **Broadening the task domain**: The paper could provide results on regression tasks to bolster the experimental validation.\n   - **Exploring state-of-the-art methods**: Consideration of more cutting-edge methods, such as generative models for uncertainty quantification, could enhance the experimental insights.\n\n### 2. Redundant Conclusions\n\nGiven the lack of novelty in experimental design (see 1), the conclusions drawn in the experimental section are similar to those in previous works [1, 2], which limits the novelty of the paper’s contributions to the literature.\n\n### 3. Assumption of Well-Approximated Data Distributions\n\nThe framework relies on the decomposition of total risk into Bayes risk and excess risk, which assumes that the underlying true data distribution can be well approximated. In practice, this assumption may not hold in many real-world scenarios, especially with complex, high-dimensional data.\n\n### 4. Lack of Baseline Comparisons\n\nThe paper appears to lack comparison to strong baselines. While the authors state:\n\n> \"We emphasize that the goal of our experimental evaluation is not to provide new state-of-the-art measures or compete with other known approaches for uncertainty quantification.\"\n\nThey also mention:\n\n> \"Instead, we aim to verify whether different uncertainty estimates are indeed related to specific types of uncertainty.\"\n\nHowever, the paper fails to provide comparisons to modern estimates of uncertainty such as [3, 4], which could help contextualize their findings and strengthen the evaluation.\n\n### 5. Lack of training details\n\nThe authors appear to have omitted important training details in Appendix H. Specifically, it is unclear:\n\n   - What loss function was used to train the models?\n   - How were the ensembles trained? Were they trained as completely independent networks, or did they share components?\n   \nProviding these details is crucial for reproducibility and understanding how the proposed framework was implemented. \n\n[1] Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Uncertainty in Artificial Intelligence, pp. 2282–2292. PMLR, 2023.\n\n[2] Kajetan Schweighofer, Lukas Aichberger, Mykyta Ielanskyi, and Sepp Hochreiter. Introducing an improved information-theoretic measure of predictive uncertainty. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023a.\n\n[3] Berry, Lucas, and David Meger. \"Escaping the sample trap: Fast and accurate epistemic uncertainty estimation with pairwise-distance estimators.\" arXiv preprint arXiv:2308.13498 (2023).\n\n[4] Chan, Matthew A., Maria J. Molina, and Christopher A. Metzler. \"Hyper-Diffusion: Estimating Epistemic and Aleatoric Uncertainty with a Single Model.\" arXiv preprint arXiv:2402.03478 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ODIMO4lORZ", "reviewer": "Reviewer_Ew1e", "summary": "This work proposes a generalized framework from which common measures of aleatoric and epistemic uncertainty can be obtained. This is done through Proper Scoring rules and the notion of risk. The framework assumes Bayes (aleatoric) and Excess (epistemic) risk, and explains that the risk may be estimated differently with different ways to approximate the true function and the learned function. Following this they observe which proper scoring rule tends to perform well, and when aleatoric/epistemic uncertainty performs well for a certain task. Lastly, they briefly describe energy based measures.", "questions": "- Previous work by Hofman et al [1] seems to be related and at least partially overlapping, but not cited. For parts of Table 1 it seems Hofman et al. give a similar result also based on proper scoring rules. From this it seems that a general framework for uncertainty measures already exists. Can the authors clarify how their proposed framework extends beyond this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper clearly outlines its contribution. By offering a better theoretical understanding of the commonly used uncertainty measures we may progress the field of disentanglement. \n- The paper covers a lot of ground in a limited span, discussing risk, proper scoring rules, different ways for risk estimation, and a substantial amount of experimental results. Overall I think the paper provides a nice overview of different uncertainty measures and how they relate.", "weaknesses": "- While the paper unifies various results into a shared formulation, parts of this are not entirely novel and the paper is not clear about this. For example, Schweighofer et al. (2023a)[2] already describe EPKL as a deviation from Mutual Information that does not assume the BMA to be the true model. They also discuss the relation to the reverse Mutual Information. This disagrees with the problem statement on Line 45 which suggests that it is not clear how these measures relate to each other. Similarly, results similar to Table 1 are already given by Hofman et al. [1] (see questions). Consider explicitly showing which parts of the generalization are novel, and which relationships are established (for example by [1] and [2]). \n- The current work fails to assert the usefulness of the proposed generalization. I would expect a strong evaluation to create hypotheses that follow from the generalization, and validate those experimentally. Only experiments 6.2 and 6.3 come with a hypothesis, but those hypotheses generally apply to uncertainty quantification and not the proposed generalization. This severely limits the impact of the paper. \n- At various locations the authors discuss that aleatoric and/or epistemic uncertainty are vaguely defined, but they do not try to maintain a precise definition following the literature. For example, they argue that epistemic uncertainty is the lack of knowledge of the right model parameters, but this would ignore model misspecification, which is accepted as a source of epistemic uncertainty. Similarly, on Line 36 authors say that aleatoric uncertainty is ambiguity in the label distribution, which ignores uncertainty in the inputs or an otherwise stochastic relationship between the features and the labels. I encourage the authors to attempt to use a more consistent and complete definition of aleatoric and epistemic uncertainty as a starting point. This is particularly relevant because the Pointwise Risk perspective seems to try to give an alternative precise definition. Consider for a source [3], Section 2.4.1.\n- It is unclear why the energy-based models are discussed, as they seem to have little relevance. Please clarify the relevance or remove this section. \n\n[1] Hofman, Paul, Yusuf Sale, and Eyke Hüllermeier. \"Quantifying Aleatoric and Epistemic Uncertainty with Proper Scoring Rules.\" arXiv preprint arXiv:2404.12215 (2024).\n\n[2] Schweighofer, Kajetan, et al. \"Introducing an improved information-theoretic measure of predictive uncertainty.\" arXiv preprint arXiv:2311.08309 (2023).\n\n[3] Gawlikowski, Jakob, et al. \"A survey of uncertainty in deep neural networks.\" Artificial Intelligence Review 56.Suppl 1 (2023): 1513-1589.\n\n\n## Additional Feedback\n\n- In Section 6.4 it would be good to clearly point to where we can find evidence for which claim (table, row and column). This mainly applies to Lines 477-482. I cannot find the results that show R_exc(3,1) is better than both energy-based methods for misclassification detection, nor that R_Exc(3,1) is preferred over energy based measures for Soft-OOD. \n- Line 456 seems overstated “all instances of excess risk should perform worse [than Bayes/Total risk on misclassification detection]”. This is true if there is more (separable) aleatoric uncertainty than epistemic uncertainty, but this is not true in general. For example in low data or high dimensional problems, this might not hold. Adding the word “typically” or “usually” could be sufficient. \n- Lines 104-107 are repetitive with Lines 36-38. \n- The whole paper assumes Deep Ensembles as the BNN, but different behavior may be observed with different models. It could be good to show behavior with MC-Dropout or Flipout.\n- Table 5 mentioned on Line 475 is actually in the appendix, but the text presents it as part of the main body. \n- In the tables it should be clear whether the plus-minus indicates standard deviation, variance or standard error. If appropriate, it may be helpful to indicate the which parts of the tables relate to conclusions drawn (highlighting in bold may be useful). \n- The notation of e.g. R_Exc^{(1, 3)} is hard to keep track of as a reader. Would it be acceptable to rename “Excess” risk to epistemic risk, or Bayes risk to Aleatoric Risk? Perhaps the indices may also be substituted with abbreviations so the connection to the approximations is clear?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cRD1Gv5ym", "reviewer": "Reviewer_cjsf", "summary": "The authors propose a family of uncertainty measures based on point-wise risks which allows to decompose the uncertainty into aleatoric (Bayes risk) and epistemic (Excess risk) parts. Plugging-in proper scoring rules into the derived uncertainty measures, they obtain expressions for computing the Bayes and Excess risk for the specific case of proper scoring rules. Finally, the authors compare three different methods for estimating the risks based on the Bayesian formulation, which they allows to derive many of the existing uncertainty measures as special cases.", "questions": "1. In Section 4.1 and Appendix E, the authors arrive at the conclusion that choosing the best estimate is often impossible in practice. namely, in Section 4.1, the authors say that exact relationship of the central estimate, $r(\\bar\\eta)$ with the other ones is not known so it's not clear which choice of the estimate for Bayes risk is best; while in Appendix E, the authors conclude that knowing which estimate to choose for the Excess is impossible to know apriori. Similarly, there is little guidance on the choice of the proper scoring rules. Overall, this leaves me with a 4 (proper scoring rules) * 9 (estimates) possible ways to get the Excess risk. Similarly, for the Bayes risk, we have another 12 possible choices. Are there any general or maybe domain-specific advice on how to choose the best UQ measure from among the proposed ones? Is it possible to have results on the error rate of each estimate for some common distribution families? In Appendix E, the authors note that none of the estimated yield a lower / upper on the Excess risk, which raises a question if it's possible to derive such an estimated bound? \n2. I think there are important points missing from the experiments description (Section 6). For example, how are the UQ measures computed given the outputs of the deep ensemble model? And more generally, given a model that only allows access samples from the posterior, are we resorting to a regular Monte Carlo? \n3. In Section 6, what is meant by 5 groups of ensembles? Do all of them have the same architecture? I think the wording here is confusing.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Many parts of the work have appeared in some form before, but combining them is a novel idea. For example, generating the uncertainty measure from the point-wise risks has been explored in [1] and [2] as the authors note, but also in [3]. Proper scoring rules were explored in the context of the uncertainty quantification in [4]. Finally, the various Bayesian risk estimation techniques have existed in the literature for a long time. However, the authors do a good job connecting these pieces and showing how various uncertainty measures naturally arise and correspond to different ways of estimating the risk for a proper scoring rule. The authors supplement this with a deeper investigation of the relationships between the estimates and exploring which estimates are better under specific conditions. \n\n[1] Kotelevskii, Nikita, et al. \"Nonparametric uncertainty quantification for single deterministic neural network.\" Advances in Neural Information Processing Systems 35 (2022): 36308-36323.\n[2] Lahlou, Salem, et al. \"Deup: Direct epistemic uncertainty prediction.\" Transactions on Machine Learning (2021).\n[3] Liu, Jeremiah, et al. \"Accurate uncertainty estimation and decomposition in ensemble learning.\" Advances in neural information processing systems 32 (2019).\n[4] Gruber, Sebastian, and Florian Buettner. \"Better uncertainty calibration via proper scores for classification and beyond.\" Advances in Neural Information Processing Systems 35 (2022): 8618-8632.", "weaknesses": "While the work as a whole appears sound, I have a few concerns:\n1. How are the proposed risks affected by the model misspecification, e.g., prior misspecification of $p(\\theta)$? I think since the method is Bayesian in nature, it's important to outline the assumptions and highlight possible shortcomings.  \n2. No simulations. Assuming, we know the true $\\eta$, it would be interesting to see how the different estimators of the risk fare under (1) various distribution shapes (2) misspecification (of prior) (3) error from approximate posterior (if full posterior is not available). \n3. Experimental section is narrowly focused and not well-motivated. The authors devote their experiments to testing whether the aleotoric and epistemic uncertainties are captured well. For this, the authors compute the proposed Total, Bayes, and Excess risks and classify the samples into out-of-distribution and misclassified based on these values. They then test the accuracy of this classification. Since Excess risk is connected to the epistemic uncertainty that is connected to the out-of-distribution classification, the out-of-distribution classification accuracy must be better based on the Excess risk. Similarly for the Bayes risk and the accuracy of identifying misclassification, both of which are connected to the aleatoric uncertainty. First, these experiments hinge on the assumption that indeed the misclassification and out-of-distribution are related to the aleatoric and epistemic uncertainties -- a largely untestable assumption. Second, from the tables, I see very little difference between using the Total, Bayes, and Excess risks across all datasets and tasks: the differences while being statistically significant appear almost negligible, which raises questions whether the suggested measures and/or experiments are meaningful. \n4. No comparison with other UQ measures. The authors do not compare their proposed UQ measures with the existing approaches, which leaves the question of why they should be used in practice. \n5. (Minor gripe)  Limited practical utility. It appears to compute any of the proposed uncertainty measures, one needs an access to the posterior $p(\\theta|X,Y)$. Ultimately, this limits the set of methods to which the proposed approach could be applied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdsUDaHW9d", "reviewer": "Reviewer_Y7My", "summary": "The authors use a decomposition of statistical pointwise risk into components, associated with aleatoric and epistemic uncertainties. Together with Bayesian methods, applied as an approximation, they build a framework that allows to generate different predictive uncertainty measures. Experiments are shown that support the theoretical claims.", "questions": "See Weaknesses. The authors should cite https://arxiv.org/abs/2302.09656 and https://link.springer.com/chapter/10.1007/978-3-031-57963-9_1#:~:text=In%20their%20seminal%201990%20paper,bound%20to%20hold%20with%20equality when discussing about the mentioned problem (and possibly in the related work section). They may also cite other approaches that may be considered in the future, such as credal sets ones, studied by Yusuf Sale, Eyke Hülleremeier, Paul Hofman, Michele Caprio, Viktor Bengs, Sebastien Desterke, Fabio Cuzzolin, Thierry Denoeux, Alessio Benavoli, and Cassio de Campos.\n\nAlso, I think there's a typo in line 198: shouldn't it be $\\eta_{\\theta \\mid D_{tr}}$ instead of $\\eta_{\\theta}\\mid D_{tr}$?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a nice and interesting paper that addresses a cogent problem in the literature, that is, the existence of many aleatoric and epistemic uncertainty measures based on the total risk decomposition, and how they're related to one another.", "weaknesses": "Taking a Bayesian route has a fundamental drawback, that was highlighted e.g. by Hullermeier and Waegeman (2021, already cited in the paper) and Caprio et al. (2024, Credal Bayesian Deep Learning). Consider the *posterior predictive distribution*,\n\n$$p(\\tilde y \\mid \\tilde x , D)=\\int_\\Theta p(\\tilde y \\mid \\tilde x, \\theta) p(\\theta \\mid D) \\text{d}\\theta = \\mathbb{E}_{\\theta \\sim p(\\theta \\mid D)} [p(\\tilde y \\mid \\tilde x, \\theta)],$$\n\nwhere $p(\\tilde y \\mid \\tilde x, \\theta)$ is the model distribution, and $p(\\theta \\mid D)$ is the parameter posterior. Posterior predictive $p(\\tilde y \\mid \\tilde x , D)$ tells us \"how likely\" output $\\tilde y$ is to be the \"correct one\" for input $\\tilde x$, given the knowledge encapsulated in the data $D$ we collected, which enters the computation via the posterior probability $p(\\theta \\mid D)$. Oftentimes, scholars claim that, in a Bayesian setting, the distribution on the parameters $\\theta$ captures (or is linked to) the epistemic uncertainty (EU) faced by the agent. This is a somehow agreeable premise, akin to a second-order distribution reasoning. If we accept this assertion, though, we see how EU at the predictive level is not quantifiable any more, since it gets washed away by taking the expectation $\\mathbb{E}_{\\theta \\sim p(\\theta \\mid D)} [\\cdot ]$. \n\nThis conceptual problem is relevant for this paper, especially because it is related to the ideas of Bayesian averaging of risk and Central label. I'd like the authors to add a discussion on this matter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)