Rethinking Aleatoric and Epistemic Uncertainty

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-05 20:21:45

Abstract: The ideas of aleatoric and epistemic uncertainty are widely used to reason about the probabilistic predictions of machine-learning models. We highlight that the common understanding of these ideas ("the standard view") makes a number of spurious associations, including between the data-generating process and a model's predictive uncertainty, between parameter stochasticity and the reducibility of predictive uncertainty, and between subjective uncertainty estimates and objective measures of predictive performance. To address this we derive a simple, principled framework for reasoning about sources of predictive uncertainty.

Main paper: 
 setup: 
 Our work considers generic prediction of y|x where x ∈ X is an input and y ∈ Y is an output. This setup encompasses scenarios ranging from a standard coin toss (X = ∅ and Y = {heads, tails}) to image classification (X = R H×W ×3 and, eg, Y = {cat, dog}) and predicting the next word in a sentence (X = V l and Y = V where V is a vocabulary and l is the number of observed words).We assume we have some data, d 1:n ∼ p data (d 1:n ). This does not need to correspond directly to the predictive task at hand: it could consist of either inputs or outputs alone (ie, d i ∈ X or d i ∈ Y), input-output pairs (ie, d i ∈ X × Y) or observations in some altogether separate space (eg, d i ∈ Z).The model of interest, p n (y|x), is a parametric model trained on d 1:n . We use p ∞ (y|x) to denote the model we converge to in the limit of n → ∞. While not required for all of the uncertainties we consider, there can be stochastic model parameters, θ ∼ p n (θ), defined such that p n (y|x) = E pn(θ) [p(y|x, θ) ]. Here p n (θ) could for example be a Bayesian posterior given d 1:n .While this work primarily considers the subjective uncertainties of our model, we often want to measure how well the model performs in practice. To formalise this we use the notion of a target distribution, p targ (y|x), which represents a system that we compare to in evaluations. 
background: 
 Bayesian experimental design (Lindley, 1956; Rainforth et al, 2024) provides a rigorous basis for reasoning about learning and uncertainty. Let ψ denote a generic random variable of interest, and let p(y, ψ|x) = p(y|x, ψ)p(ψ) denote a joint model over ψ and output observations, y, associated with a given input, x. The information gain (IG) in ψ due to observing new data, (x, y), is the reduction in entropy (Shannon, 1948) that results from a Bayesian update from p(ψ) to p(ψ|x, y) ∝ p(y, ψ|x):IG ψ (x, y) = H[p(ψ)] -H[p(ψ|x, y)].(1)Given an input, x, we can compute the expected information gain (EIG) over possible realisations of y simulated using p ψ (y|x) = E p(ψ) [p(y|x, ψ) ], the marginal predictive distribution: [H[p(y|x, ψ) ]] = I(y; ψ|x).EIG ψ (x) = E p ψ (y|x) [IG ψ (x, y)] = H[p ψ (y|x)] -E p(ψ)(2)The standard view on aleatoric and epistemic uncertainty in machine learning (Gal, 2016; Gal et al, 2017; Kendall & Gal, 2017) is often summarised as an interpretation of the EIG in the parameters of a model, also known as the BALD score (Houlsby et al, 2011) :EIG θ (x) epistemic = H[p n (y|x)] total -E pn(θ) [H[p(y|x, θ)]] aleatoric .(3)Gal (2016) stated the "total = aleatoric + epistemic" relationship and the correspondence between p n (y|x) and the total uncertainty; Gal et al (2017) made the explicit link between these and the information-theoretic quantities in Equation 3; and Kendall & Gal (2017) articulated an intuitive interpretation of these ideas that has since become the de-facto reference on the standard view. 
issues with the standard view: 
 At its core the standard view gets at a valuable insight: if a model's prediction of y|x is uncertain, it can be useful to know whether that prediction is fundamentally uncertain for the given model class or instead due to a lack of data. This distinction is crucial in the context of active learning, where we seek new data that will lead to better predictions (Bickford Smith et al, 2023) . But it can also be useful in other cases, such as where a model's prediction is being used to inform a person's decision.We argue the standard view went some way to formalising this insight but fell short. Perhaps the easiest way to see this is to note that both aleatoric and epistemic uncertainty as discussed in the standard view refer to multiple quantities (Figure 1 ), introducing multiple spurious associations."captures noise inherent in the observations"H[p data (d 1:n )] or H[p targ (y|x)]"cannot be reduced even if more data were to be collected"H[p ∞ (y|x)]Conditional predictive entropy (Equation 3)E pn(θ) [H[p(y|x, θ)]] 1 2 3 Aleatoric uncertainty"uncertainty in the model parameters"H[p n (θ)]"can be explained away given enough data"H[p n (θ)] or H[p n (y|x)] -H[p ∞ (y|x)]Expected information gain in θ (Equation 3)H[p n (y|x)] -E pn(θ) [H[p(y|x, θ)]] 1 2 3 Epistemic uncertaintyFigure 1 The standard view on aleatoric and epistemic uncertainty attaches multiple quantities to each term. Some of these quantities can coincide in particular cases but in the general case they are distinct. The quotations here are from Kendall & Gal (2017) ; the interpretation of Equation 3 is due to Gal (2016) and Gal et al (2017) .The competing definitions of aleatoric uncertainty conflate a model's irreducible predictive uncertainty,H[p ∞ (y|x)],a key quantity in our framework (Section 5), with three separate quantities:(a) H[p data (d 1:n )], the entropy of the data. Issue: suppose d 1:n represents the outcomes of n coin tosses and the task is to predict the number of heads in the next m tosses (ie, X = ∅ and [H[p(y|x, θ) ]], the expected information gain in the model parameters. Issue:Y = {0, 1, . . . , m}); then H[p n (y|x)] → 0 as n → ∞ while H[p data (d 1:n )] = n log 2. (b) H[p targ (y|x)],for finite n the parameter EIG is only an estimator ofH[p n (y|x)] -H[p ∞ (y|x)] (Section 5).Other sources of confusion in the standard view include an incorrect association between a model's subjective uncertainty and objective measures of performance, such as classification accuracy (Figure 2 in Kendall & Gal (2017) ), along with misleading implications about how a model's uncertainty will behave with varying n (Figure 6 .11-6.12 in Gal (2016) and Table 3 in Kendall & Gal (2017) )and varying distance from the training data ("Aleatoric uncertainty does not increase for out-of-data examples. . . whereas epistemic uncertainty does" in Kendall & Gal (2017) ). 
a new framework: 
 We return to the original motivation for discussing aleatoric and epistemic uncertainty, now using principles from modern statistics. If our model has been trained on data d 1:n and has some predictive uncertainty, H[p n (y|x)], we want to know where that uncertainty comes from. The prediction could be fundamentally uncertain for the given model class; it could be uncertain because we do not know which model is best within the model class; or it could be uncertain for a mixture of these reasons.From a Bayesian perspective, if we do not know which model is the best within a given model class, this stems from a lack of data: it is a result of not observing d (n+1):∞ (Fong et al, 2023) . This tells us we need to think of a model's behaviour as n → ∞ in order to precisely define the model's sources of uncertainty. More concretely, using the language of Bayesian experimental design (Rainforth et al,Total uncertaintyData d 1:nActual reducible uncertaintyH[p n (y|x)] -H[p ∞ (y|x)]Estimated reducible uncertaintyH[p n (y|x)] -E pn(θ) [H[p(y|x, θ)]]Figure 2 Our framework focuses on the predictive uncertainty of a given model, pn(y|x), trained on data d1:n. This can be decomposed into irreducible and reducible components, which reflect how the model's uncertainty changes as n → ∞. Given finite n, these irreducible and reducible components must be estimated. This is made possible by using stochastic model parameters, θ ∼ pn(θ), where pn(y|x) = E pn(θ) [p(y|x, θ)].information gain, which is defined as its reduction in predictive entropy from updating on d (n+1):∞ :EQUATION)Given finite n, the actual irreducible predictive uncertainty must be estimated. We show the conditional predictive entropy is an optimal estimator, providing a link between Equations 3 and 4.Proposition 1 A model's conditional predictive entropy, E pn(θ) [H[p(y|x, θ) ]], is a Bayes estimator of its actual irreducible predictive uncertainty, H[p ∞ (y|x)].Proof Let h be an estimator of H[p ∞ (y|x)], and let p n (θ) be our beliefs over which θ produces p(y|x, θ) = p ∞ (y|x). The Bayes estimator is the minimiser of the expected loss,EQUATIONSolving this gives h = E pn(θ) [H[p(y|x, θ) ]], the conditional predictive entropy.Thus we see that the information-theoretic quantities often used within the standard view are correctly viewed not as direct measures of irreducible and reducible uncertainty, but instead as finite-data estimators of those uncertainties. As such, while they are principled estimators, they can still be highly inaccurate. Figure 2 in Bickford Smith et al (2024) provides some evidence for this: a model's irreducible-uncertainty estimates at small n can be far from its estimates at large n.The framework we propose (Figure 2 ) combines the decomposition from Equation 4with the estimation from Proposition 1. Crucially it avoids the standard view's spurious associations. An example of this is its decoupling of the uncertainty decomposition from the use of a stochastic model: Equation 4 applies to deterministic models; only when we consider estimation does parameter stochasticity become relevant. Another example is that the actual irreducible uncertainty, even though it depends on the data-generating process, is not equal to the entropy of the data, H[p data (d 1:n )].In addition to the subjective uncertainties that our framework focuses on, objective measures of a model's predictive performance are important in practice. Counter to Kendall & Gal (2017) , we emphasise that subjective uncertainties are not a substitute for explicitly comparing the model to a target distribution, p targ (y|x). A standard quantity of interest is H[p targ (y|x) ∥ p n (y|x)], the cross entropy between the target distribution and the model's predictive distribution. 
conclusion: 
 We have identified sources of confusion in a standard view on uncertainty in machine learning and, to deal with this, we have introduced an alternative framework. A key fact underlying this work is the extent of the subjectivity of the uncertainties we have discussed. Our framework makes transparent the dependence on the model class and the amount of data. But it abstracts away dependencies on other things, such as what exactly the data is and what algorithm is used to learn from the data. While we believe this abstraction is useful for the high-level understanding we have aimed to provide here, a precise analysis would need to account for these dependencies. Another possible extension of this work would be to consider nonparametric models, which might require extra care. Understanding the implications of the current work for designing practical methods could also be productive.Lindley (1956) . On a measure of the information provided by an experiment. Annals of Mathematical Statistics.Liu, Zhang, & Barnes (2022). Modeling aleatoric uncertainty for camouflaged object detection.Winter Conference on Applications of Computer Vision. (1992a) . The evidence framework applied to classification networks. Neural Computation. Seeböck, Orlando, Schlegl, Waldstein, Bogunović, Klimscha, Langs, & Schmidt-Erfurth (2019) . 
: 
 ideas that have been wrongly associated in the literature. Our hope is that embracing this framework in future work will stimulate clearer thinking, helping the field more quickly achieve its goals. 
 
 Appendix: Exploiting epistemic uncertainty of anatomy segmentation for anomaly detection in retinal OCT.Transactions on Medical Imaging. , Bösner, Dembczyński, Haasenritter, Hirsch, Donner-Banzhoff, & Hüllermeier (2014) . Reliable classification: learning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences. (1978) . Non-additive probabilities in the work of Bernoulli and Laplace. Archive for History of Exact Sciences. (1948) . A mathematical theory of communication. The Bell System Technical Journal. 



Reviews: review: title: title: Review
main_review: main_review: This paper notes systemic issues with standard definitions/decompositions of epistemic and aleatoric uncertainty as used in modern machine learning literature and proposes a new framework for properly decomposing uncertainty into reducible and irriducible elements. This framework is largely based on ideas from Fong's work on martingale posteriors as well as Rainforth's work on modern Bayesian experimental design.

This new framework is thought provoking and likely controversial. The authors' math and arguments are sound, correct, and rooted in modern literature. I imagine this submission will make for great discussion at the workshop and so I recommend that it is featured as a spotlight presentation.
title: title: Good paper on estimation of aleatoric and epistemic uncertainty
main_review: main_review: This paper is a critique and alternate formulation for the common interpretation and estimation of aleatoric and epistemic uncertainty.

The authors argue that commonly used formulations for aleatoric and epistemic uncertainty are often wrong or contradictory, and they propose a partially new formulation (based on the well known information theoretical formulation for disentanglement). The paper only makes a theoretical argument. The paper is strong and should be accepted.

**Strengths**: The paper is very well written and argued, notation is very clear. The paper tackles an important open problem, as while there are very clear definitions for aleatoric and epistemic uncertainty (irreducible vs reducible uncertainties), there are many ways to estimate them, some more correct than others, and this paper make arguments on why some estimations are not completely correct, while providing a more correct formulation and a proof about estimating irreducible (aleatoric) uncertainty.

This paper is definitely interesting and would trigger interesting discussion in the workshop.

**Weaknesses**: The paper does not seem to have results, only the formulation, validation would have been nice, in particular with settings where training set size is varied.