DEUP: Direct Epistemic Uncertainty Prediction
Authors: Salem Lahlou
Moksh Jain
Hadi Nekoei
Victor I Butoi
Paul Bertin
Jarrid Rector-Brooks
Maksym Korablyov
Yoshua Bengio
Venue: Accepted by TMLR ()

Abstract: Epistemic Uncertainty is a measure of the lack of knowledge of a learner which diminishes with more evidence. While existing work focuses on using the variance of the Bayesian posterior due to parameter uncertainty as a measure of epistemic uncertainty, we argue that this does not capture the part of lack of knowledge induced by model misspecification. We discuss how the excess risk, which is the gap between the generalization error of a predictor and the Bayes predictor, is a sound measure of epistemic uncertainty which captures the effect of model misspecification. We thus propose a principled framework for directly estimating the excess risk by learning a secondary predictor for the generalization error and subtracting an estimate of aleatoric uncertainty, i.e., intrinsic unpredictability. We discuss the merits of this novel measure of epistemic uncertainty, and highlight how it differs from variance-based measures of epistemic uncertainty and addresses its major pitfall. Our framework, Direct Epistemic Uncertainty Prediction (DEUP) is particularly interesting in interactive learning environments, where the learner is allowed to acquire novel examples in each round. Through a wide set of experiments, we illustrate how existing methods in sequential model optimization can be improved with epistemic uncertainty estimates from DEUP, and how DEUP can be used to drive exploration in reinforcement learning. We also evaluate the quality of uncertainty estimates from DEUP for probabilistic image classification and predicting synergies of drug combinations.

Published in Transactions on Machine Learning Research (02/2023)
DEUP: Direct Epistemic Uncertainty Prediction
Salem Lahlou'
lahlosal@vmilla.co quebec
Mila Quebec Al Institute, Université de Montréal
Moksh Jain"
moksh.jainn@nia @rreila quebee
Mila Quebec Al Institute, Université de Montréal
Hadi Nekoci
nekovihhe Avreila @wreila.quebeer
Mila Quebec Al Institute, Université de Montréal
Victor Ion Butoi!
obutoi@mit.c edu
Massachusetts Institute of Technology
Paul Bertin
bertinpa Eireila. qucbec
Mila Quebec Al Institute, Université de Montréal
Jarrid Rector-Brooks
jarrid. brooks @vmila. quebec
Mila Quebec Al Institute, Université de Montréal
Maksym Korablyov
korablymm@melll. quecbe
Mila Quebec AI Institute, Université de Montréal
Yoshua Bengio
yoshua, bengio @milla queebec
Mila Quebec Al Institute, Université de Montréal
CIFAR Fellow
Reviewed on OpenReview: https: openreview. net/ forum? id= VVLdVRvuf
Abstract
Epistemic Uncertainty is measure of the lack of knowledge of learner which diminishes with more evidence. While existing work focuses on using the variance of the Bayesian posterior due to parameter uncertainty as a measure of epistemic uncertainty, we argue that this does not capture the part of lack of knowledge induced by model misspecification We discuss how the excess risk, which is the gap between the generalization error of predictor and the Bayes predictor, is sound measure of epistemic uncertainty which captures the effect of model misspecification. We thus propose principled framework for directly estimating the excess risk by learning secondary predictor for the generalization estimate of aleatoric uncertainty, i.e., intrinsic unpredictability. We
error and subtracting discuss the merits of this novel measure of epistemic uncertainty, and highlight how it differs from variance- based measures of epistemic uncertainty and addresses its major pitfall. Our framework, Direct. Epistemic Uncertainty Prediction (DEUP) is particularly interesting in interactive learning environments, where the learner allowed to acquire novel examples in each round. Through wide of experiments, illustrate how existing methods in sequential model optimization be improved with epistemic uncertainty estimates from DEUP, and how DEUP can be used to drive exploration in reinforcement learning. We also evaluate the quality of uncertainty estimates from DEUP for probabilistic image classification and predicting synergies of drug combinations.
"Equal Contribution
Work done during internship Mila.

---

Published in Transactions on Machine Learning Research (02/2023)
1 Introduction
A remaining great challenge in machine learning research is purposeful knowledge-seeking by learning agents, which can benefit from estimation of epistemic uncertainty (EU), i.e., a measure of lack of knowledge that an active learner should minimize. Unlike aleatoric uncertainty, which refers to an intrinsic notion of randomness and is irreducible by nature, epistemic uncertainty can potentially be reduced with additional information (Kiureghian & Ditlevsen, 2009; Hüllermeier & Waegeman, 2019).
In machine learning, EU estimation is already a key ingredient in interactive decision making settings such as active lcarning (Aggarwal et al., 2014; Gal et al., 2017), soquential model optimization (SMO, Frazier (2018); Garnett (2022)) as well as exploration in reinforcement learning (RL, Kocsis Szepesvári (2006); Tang et al. (2017)), as EU estimators can inform the learner about the potential information gain from collecting data around particular area of state-space or input data space.
What is uncertainty and how to quantify it? In the subjective interpretation of the concept of Probability (De Morgan, 1847; Ramsey, 1926), probabilities are representations of an agent's 's subjeclive degree of confidence (Hájek, 2019). It is thus unsurprising that probabilities have been widely used to represent. uncertainty. Yet, there is not a generally agreed upon definition of uncertainty, let alone a way of quantifying it mumber, confirmed by meta-analysis performed by Zidek & Van Eeden (2003). In supervised learning, Bayesian methods, which aim at predicting a conditional probability distribution on the variable to prodict, called the posterior predictive, are natural candidates for providing uncertainty estimates. Bayesian learning is however more computationally expensive than its frequentist counterpart, and generally relies on approximations such as MCMC (Brooks et al., 2011) and Variational Inference (Blei et al., 2017) methods. Prominent uncertainty estimation methods, such as MC-Dropout (Gal & Ghahramani, 2016) and Deep Ensembles (Lakshminarayanan et al., 2017), both of which are approximate Bayesian methods (Hoffmann & Flster, 2021), use the posterior predictive variance as measure of uncertainty: if multiple neural nets that, are all compatible with the data make different predictions at z, the discrepancy between these predictions is a strong indicator of epistemic uncertainty at x.
Pitfalls of using the Bayesian posterior to estimate uncertainty: Epistemic uncertainty in a predictive model, seen as measure of lack of knowledge, consists of approrimation uncertainty due to the finite size of the training dataset, and model uncertainty- due to model misspecification (Hällermeier & Waegeman, 2019), also called bias. The latter is not accounted for when using variance or entropy as a measure of EU, but has been shown to be important for generalization (Masegosa, 2020) as well as within interactive learning settings like optimal design (Zhou et al., 2003). Approximate Bayesian methods that are widcly used in machine learning often suffer from model misspecification, for instance due to the implicit bias induced by SGD (Kale et al., 2021) and the finite computational time. Fig. illustrates with a Gaussian Process (GP) the suboptimal behavior of the GP to estimate EU under model misspecification. The confidence intervals provided by the GP are underestimated when the model is misspecified, which can lead to confidently wrong decisions and poor decisions (see Fig. 1, bottom left).
As a first contribution, we systematically analyze the sources of uncertainty and misspecification, and analyze the pitfalls of using discrepancy- based measures of EU (such as variance of the Bayesian posterior predictive), given that they miss out on model uncertainty, which we define as a component of the excess risk, i.c. the gap between the risk (or out-of-sample loss) of the predictor at a point x and that of the Bayes predictor at (the one with the lowest expected loss, that no amount of additional data could reduce).
As second contribution, uncertainty as lack of knowledge, and based on this, we propose to estimate the excess risk as a mcasure of EU. This leads us to our proposed framework DEUP, for Direct Epistemic Uncertainty Prediction, where we train a secondary model, called the error predictor, with an appropriate objective and appropriate data, to estimate the point-wise generalization error (the risk), and then subtract an estimate of alcatoric uncertainty if available, or provide an upper bound on EU otherwise. Fig. 1 illustrates in a toy task in which held-out is available, that the epistemic uncertainty estimated by DEUP is more useful to an interactive
take step back by considering the fundamental notion of epistemic
2

---

Published in Transactions on Machine Learning Research (02/2023)
learner than those provided by misspecified GP. It is important to note that, in these settings of interest, DEUP does not use any additional held-out data.
Ground truth GP
2
0-
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
Ground truth GP mean
Ground truth
2
5.0
GP mean
2.5
0
0.0
-2.
-0.5 0.0
0.5
1.0
1.5
2.0
2.5
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
Figure Top. A GP is fit on (dark blue) points in [0,0.5] U [1.5, 2]. The shaded area represents the GP standard deviation, often used as a measure of epistemic uncertainty: note how it fails badly to enclose the ground function in [0.5, 1.5]. Bottom left. A second GP fit on the same points, with extra points (green crosses). This second GP predicts almost 0 standard deviation everywhere, even though the first GP significantly underestimated the uncertainty in [0.5, 1.5], because no signal is given to the learner that the region [0.5, 1.5] should be explored more. Bottom right. DEUP learns to map the inderestimated variances of the first GP to the L2 errors made by that GP, and yields more reasonable uncertainty estimates that should inform the learner of what area to explore.
Accounting for bias when measuring EU is particularly useful to an interactive learner whose effective capacity depends on the training data, for instance neural network, it reduced with additional training data, especially in regions of the input space we care about, i.e. where EU is large. DEUP is agnostic to the lype of predictors used, and in interactive settings, it is agnostic to the particular search method still needed to select points with high EU in order to propose new candidates for active lcarning (Aggarwal et al., 2014; Nguyen et al., 2019; Bengio et al., 2021), SMO (Kushner, 1964; Jones et al., 1998; Snoek et al., 2012) or exploration in RL (Koesis Szepesvári, 2006; Osband et al., 2016; Janz et al., 2019).
A unique advantage of DEUP, compared with discrepancy- measures of EU, is that it can be explicitly trained to care about, and calibrate for estimating the uncertainty for examples which may come from a distribution slightly different from the distribution of most of the training examples. Such distribution shifts (referred to as feedback covariate shift in Fannjiang et al. (2022)) arise naturally in interactive contexts such as RL, becanse the learner explores new areas of the input space. In these non-stationary settings, we typically want to retrain the main predictor as we acquire new training data, not just because more training data is generally better but also to better track the changing distribution and generalize to yet unseen but upcoming out-of-distributton (OOD) inputs. This setting makes it challenging train the main predictor but it also entails second non-stationarity for the training data seen by the error predictor: large error initially made at point æ before is incorporated in the training set (along with an outcome y) will typically be greatly reduced after updating the main predictor with (I, To cope with this non-stationarity in the targets of the error predictor, we propose, as a third contribution, to use additional features as input to the error predictor, that are informative of both the input point and the dataset used to obtain the current predictor. We mainly use density estimates and model variance estimates, that sometimes come at no additional cost.
The remainder of the paper is divided as follows:
3

---

Published in Transactions on Machine Learning Research (02/2023)
In Sec. we describe theoretical framework for analysing sources of uncertainty and describe pitfall of discrepancy-based measures of epistemic uncertainty.
In Sec. 3, we present DEUP; a principled model-agnostic framework for estimating epistemic uncertainly, and describe how it can be applied in a fixed dataset as well as interactive learning settings, and propose mcans address the non-stationaritics arising in the latter.
In Sec. 4, discuss related work uncertainty estimation well as error predictors in machine learning. In Sec. 5, we experimentally validate that EU estimates from DEUP can improve upon existing SMO methods, drive exploration in RI, and evaluate the quality of these uncertainty estimates in probabilistic image classification and in a regression lask predicting synergies of drug combinations.
2 Excess Risk, Epistemic Uncertainty, and Model Misspecification
In this section, we analyze the sources of lack of knowledge through the lens of the excess risk of a predictor, define model misspecification, and discuss the pitfall of using diserepancy-based measures of uncertainty when the model is misspecified. The section is divided as follows:
In Sec. 2.1, we define the mathematical framework and notations used in the analysis.
In Sec. 2.2, we characterize the sources of lack of knowledge, and argue that estimating the excess risk is a sound measure of epistemic uncertainty. This the main rationale behind our framework, DEUP, presented in Sec. 3.
In Sec. 2.3, briefly analyze the sub-optimal behavior of the Bayesian posterior when the model is misspecified, and explain why in practice models are generally biased, even non-parametric ones.
2.1 Notations and Background
Predictive models tackle the problem of learning to predict outputs y given inputs E X S Rd using function f X - A estimated from training dataset 2 := (Z1,..., zN) E 2N where Z = X x ) and 2, = (xt, 3t), and the unknown ground-truth generative model is defined by P(X,Y) = P(X) X). In decision theory, the set is called the action space, and typically either equal to y for regression problems, or A()), the set of probability distributions on V, for classification problems. Given a loss function : y x A + (e.g., ((y, a) lly - alI for regression, and (ly.a) a) log a(y) for classification), the point-wise risk (or expected loss) of predictor f at x E X is defined as:
R(f x) = Ep(rxx=>)(00 [(,,f()))
(1)
We define the risk (or total expected loss) of a predictor as the marginalization of (1) over r:
R(f)
(2)
Given a hypothesis space H, subset of F(X,A), the set of functions algorithm (or learner) L is to find predictive function h" E H with the lowest possible risk:
A, the goal of any learning
h = arg min R(h).
(3)
NCH
Naturally, the search for clusive given that P(X,Y) Y) is usually unknown to the learner. Instead, the learner maps finite training dataset N to predictive function L(=N) = hgN minimizing an approximation of (2), called the empirical risk:
1 Σ (yn,)((s))) R.x(h) = N
where z; (xii, y:)
(4)
byN arg min R2, (h). KEW
(5)
The datasct N need not be used solely define the empirical risk as in (4). The learner can for example use a subset of N as a validation to tune its hyperparameters and thus its effective capacity (Arpit et al., 2017).
4

---

Published in Transactions on Machine Learning Research (02/2023)
2.2 Sources of lack of knowledge
Clearly, high value of R(hgx .)) indicates lack of knowledge around æ. Similar to Hillermeier & Waegeman (2019), we char- acterize the three sources this lack of knowledge, illustrated in Fig. 2, as follows:
A)
Model uncerairty Bass
(1) has a fundamental limiting lower bound, usually reached at function f called the Bayes predictor*:
H
(x) = arg min Bpo a)).
(6)
Unrwtenty
eEA
lug8
R(f",z) 0 indicates irreducible risk due to the inherent randomness of P(Y X x). R(f",x) is thus measure of aleatoric uncertainty at x. Note that there might be more than one Bayes predictor, but by definition, they all have the same point-wise risk, which we denote as A:
Figure 2: Graphical representations of the two components of epistemic uncertainty. Inspired by Fig. 4 of Hüllermeier & Waege man (2019).
A(x) = 2)
(7)
Minimizing the risk over H rather than F(X,A) induces discrepancy between the predictor that is optimal in H, h", and the Bayes predictor f*, usually referred to as model uncertainty (our hypothesis space could be improved). This can be seen as a form of bias, as the optimization is limited to functions in H.
Minimizing the empirical risk instead of the risk induces a discrepancy between h_ and h", called the approximation uncertainty, due both to the finite training set and finite computational resources for training.
Borrowing terminology from Futami al. (2022), we use the Bayes predictor to define the excess risk follows:
Definition 1 The excess risk of a predictor f: X + A at x is the gap between the point pose wise risk and its fundamental limit:
BR(J, R(J,x) A(x)
Both model uncertainty and approximation uncertainty are indicative the epistemic state of the learner, and can be reduced with better choices (of H) and more data respectively. This implies that an estimator of the excess risk of at (Def. 1) can be used as measure of epistemic uncertainty. ER is intimately linked to the minimum excess risk (Xu & Raginsky, 2020), measure of the gap between the minimum expected loss attainable by learning from data and A(z), which was shown to decrease and converge towards 0 as the training datasct size grows to infinity, which is a desirable property for EU mcasures. Using an estimator of ER as a measure of EU is the main idea behind DEUP, and will be expanded upon in Sec. 3.
The following examples illustrate the concepts introduced thus far:
Example 1 Consider a univariate regression problem with Gaussion ground truth, i.e. y IR and there exist functions μ and a such that P(Y | = x) NN (((; with the squared loss U(y, a) = (y - a)². Then, the Bayes predictor is f* μ,, which has a point-wise risk + A(z) (z)), and for any predictor f X > RR and for every x. E X:
) - ( xxx)) ER(J,x)= = -f(x) - (J(x))u(x))
Example 2 Consider a classification problem with K classes, i.e. y = fl....,K'' K} and there exists a function p: X + A K where A K is probability simplex of dimension K 1, such that P(Y X = x) is the categorical
Xx=
An equivalent definition is f" = arg minginneeFe... R(.)
5

---

Published in Transactions on Machine Learning Research (02/2023)
distribution with probability mass function j(æx), with the log loss ly,,a) - log aly). Then, the Bayes predictor is - n, which has point-wise risk x + A(x) = p())))===) [Y]] the entropy of the ground-trulh label distribution, and for any predictor f + A and for every I E X:
R(f, x = ( f(x)) ER(f.z) = (kL((())
where H(, .) and DELG: .) denote respectively the cross entropy and the Kullback- Leibler (KL) divergence between two distribulions.
It is clear from the definitions above, and from Fig. 2 that the choice of the subset H can contribute to higher epistemic uncertainty, as it introduces misspecification (Masegosa, 2020; Hong & Martin, 2020; Cervera et al., 2021):
Definition 2 learner with hypothesis space H. _ F(X, is said to be learning under model misspecification if f & H, where f* is the Bayes predictor.
While there no agreed upon mcasure of misppcciication, some authors (Masegosa, 2020; Hong Martin, 2020) focus on Bayesian or approximate Bayesian learners, which maintain a distribution over predictors, and use discrepancy measure between the best reachable posterior predictive p(Y X) defined by functions h E H. and the ground-truth likelihood P(Y X). Alternatively, and assuming the function space F(X, AA) is endowed with a metric, misspecification (or bias) can be defined as the distance between h* and f*.
Additionally, and as argued by many authors (see c.g. Cervera et al. (2021); Knoblauch et al. (2022)), a common implicit assumption in (approxinate) Bayesian methods is correct model specification (i.e., there is no bias). In the next subsection, we will briefly review why this assumption is rarely satisfied in practice, and what it entails in terms of uncertainty modelling.
2.3 Bayesian uncertainty under model misspecification
Consider a parametric model p(Y X: f) and learner maintaining distribution over parameters 0 E θ each corresponding to a predictor h in parametric set of functions H. possibly starting from a prior p(0) that would lead to posterior distribution p(0 =>) upon observing data N Clearly, the fact that multiple O's and corresponding values of h are compatible with the data and the prior indicates lack of knowledge. Because the lack of knowledge indicates where an interactive learner should acquire more information, this justifies the usage of dispersion measures, such as the variance or the entropy of the posterior predictive, as measures of EU.
While such dispersion measures are indicative of approximation uncertainty, they do not account for model misspecification, or bias. The sub-optimal behavior of Bayesian methods with misspecified models has already been studied in terms of the quality of predictions (Grünwald, 2012; Walker, 2013; Grünwald & Van Ommen, 2017) as well as their generalization properties (Masegosa, 2020). These works show that even though the Bayesian posterior predictive concentrates around the best model p(Y xx with minimum KL divergence to the ground truth P(Y X) within H, it does not fit the data distribution perfectly. Additionally, Kleijn & van der Vaart (2012) provide a theoretical analysis of the sub-optimalitt of the uncertainty estimates under model misspecification. More specifically, the authors derive a misspecified version of the Bernstein- Von Mises theorem, which implies that the Bayesian credible sets are valid confidence sets when the model is well specified, but prove that under misspecification, the credible sets may over-cover under-cover, and are thus not valid confidence sets. More recently, Cervera el al. (2021) showed that a Bayesian treatment of a misspecified model leads to unreliable epistemic uncertainty estimates in regression settings, while D' Angelo & Hemning (2021) showed that uncertainty estimates in misspecified models lead to undesirable behaviors when used to detect OOD examples.
Sources of model misspecification: In the Bayesian framework, the hypothesis class H and the corre- sponding set of posterior predictive distributions p(Y X) are only implicitly defined, through the choices of the prior p(0), the likelihood functions p(Y X)0), and the computation budget. Even if in theory, the model
6

---

Published in Transactions on Machine Learning Research (02/2023)
is well specified, in the sense that that there exists 0 E Q such that p(Y | X;0))= = P(Y x)2, in practice, the combination of the prior and the finite computational budget limits the set of reachable 0, resulting in a systematic bias (Knoblauch et al., 2022). This behavior is exacerbated in low data regimes, even with modern machine learning models such as ncural networks, because the learner may not use all of its capacity due to the implicit (and not fully understood) regularization properties of stochastic gradient descent (SGD, Kale et al. (2021)), explicit regularization, carly stopping, or a combination of these. "All models are wrong, but some are useful" Box (1976).
The importance of bias in interactive settings: Naturally, bad uncertainty estimates also have adverse effects on the performances of interactive learners, that use said estimates to guide the acquisition and evaluation of more inpuls (Zhou et al., 2003). In fact, it has long been known (Box & Draper, 1959) that model uncertainty (also called bias in Box & Draper (1959)) is just as important as, if not more than, approximation uncertainly (interestingly referred to as variance in Box & Draper (1959)), for the problem of optimal design. Therefore, a learner using approximation uncertainty alone, measured by the variance of the posterior predictive, can be confidently wrong. When the acquisition of more inputs is guided by these predictions, this can slow down the interactive learning process. In Deep Ensembles (Lakshminarayanan et al., 2017) for example, if all the networks in the ensemble tend to fail in a systematic (i.e. polentially redueíble) way, this aspect of prediction failure will not be captured by variance, flexible models like neural networks however, the hypothesis space H. is defined not only through the chosen architecture of the nelwork, but also through the pre-defined training procedure (e.g. the hyperparameters of the optimizer, the stopping criterion based on the performance on a validation set chosen from the training data.... This means that even though the learning strategy is data-independent,, the hypothesis space H depends on the training data. Arpit et al. (2017) formalized this subtle distinction using the notion of "effective capacity". As a consequence, the learner can incorporate new training examples in areas of the input space where the bias is large, in order to change the hypothesis space H to H', which can be closer to the Bayes predictor, thus reducing the bias, and de facto the excess risk ER.
approximation uncertainty. With
As a consequencee of the ubiquity of misspecified models, it is important to question the status quo of relying on discrepancy-baser measures of epistemic uncertainty in machine learning, and explore alternative ways of measuring EU, such as DEUP, which we introduce in the next section.
3 Direct Epistemic Uncertainty Prediction
As argued in the previous section, the excess risk (8) of a predictor f at a point x can be used as a measure of EU that is robust against model misspocification. Estimating the excess risk ER(f, x) requires an estimate of the expected loss R(f,x), and an estimate of the aleatoric uncertainty A(x) (7).
DEUP (Direct Epistemic Uncertainty Prediction) uses observed out-of-sample errors in order to train error predictor - e(f,,) .r) estimating - R(f,z). These may be in-distribution or out-of-distribation errors, depending on what we care about and the kind of data that is available. Given a predictor + a(x) of aleatoric uncertainty, u(J,z) defined by:
- =(f,x)
becomes an estimator of ER(f, x) and as a consequence, an estimator of the epistemic uncertainty of f at x. Before delving into the details of DEUP, we briefly discuss how one can obtain an estimator of aleatoric uncertainty.
How to estimate aleatoric uncertainty? We consider three possible scenarios for the aleatoric uncer- tainty:
1. If we know that A(x)= = 0, i.c. that the data-generating process is noiscless, then u(f,)) is an estimate of R(fx), and alx) can safely be set to O. This is the case in the experiments present in Sec. 5.1.
Note how we use upper case P for the ground truth posterior and lower case p for the likelihood and the Bayeslan posterior predictive estimator.
7

---

Published in Transactions on Machine Learning Research (02/2023)
2. In regression settings with the squared loss, where A(x) equals the variance of P(Y X x) (Ex. 1); if we have access to an oracle that samples Y given a query a from the environment P(Y x)) (e.g., in active learning or SMO), then A(x) can be estimated using the empirical variance of different outcomes of the oracle at the same input 2. It is common practice for example to perform replicate experiments in biological assays and use variation across replicates to estimate aleatoric uncertainty (Lee et al., 2000; Schurch et al., 2016).
More formally, if we have multiple independent outcomes yI,.... 5/K P(Y x)) for each input point x, then training predictor with the squared loss on (input, target) examples k- - Var(y......
K
where Var denotes the empirical variance, yields an estimator of the alcatoric uncertainty.
Naturally, this estimator is asymptol unbiased if the learning algorithm ensures asymptolic convergence to Bayes predictor. This is due to the known fact that
K Var(yl ...... MK) Varp(y)))) [X)])
(10)
K-1
We illustrate how such estimator could be used to obtain EU estimates from the error predictor in Appendix E.
3. In cases where it is not possible to estimate the alcatoric uncertainty, we can use the expected loss estimate e(f,z) 2) as a pessimistic (i.e. conservative) proxy for u(f.z), i... set a(x) to 0 in (9). This is particularly relevant. for settings when uncertainty estimates are used only to rank different data-points, and which there is no reason to suspect that there significant variability in aleatoric uncertainty across the input space. This is actually the implicit assumption made whenever the variance entropy of the posterior predictive (which, in principle, accounts for aleatoric uncertainly) is used to measure epistemic uncertainty. This is the case in the experiments we present in Sec. 5.3.
In real life settings, an estimator x + a(z) of aleatoric uncertainty can be available (e.g. margins of errors of instruments used in a wet lab experiment), and can thus readily be plugged into (9). In the following subsections, we thus assune that x a(x) is available, whether it is identically equal to 0 or not, and focus on estimating the point-wise risk or expected loss R) In Sec. 3.1, we present DEUP in a fixed training set setting, when the learner has access to a held-out validation set. In Sec. 3.2, we demonstrate how the challenges brought about by interactive settings actually bypass the need for the validation set required to train the error predictor.
3.1 Fixed Training Set
Using the notations introduced Sec. 2, we first consider the scenario where we are interested in estimating EU for one predictor hgM, trained on a given training set N of certain size N, and held-out validation set _'K is available. Recall that the goal is to obtain an estimator + e(he8, of the point-wise risk + R(h_x, 2) ((C
Each 2 (z/, v in the validation datasct can be used to compute an out-of-sample error e = 1(b1. ha (at)). Training a secondary predictor e on input-target pairs (x/, e(), for E (1,..., , K} yields the desired estimator x + e(h,,,,), The procedure is summarized in Algorithm 1. It is used for the example provided in Fig. 1
Algorithm DEUP with a fixed training set
Inputs: h h22, trained predictor. a, an estimator of aleatoric uncertainty. zk {(1), 14
validation set.
Outputs: u: X + R. estimator of the EU of h.
Training:
Create input-error dataset D. = {(1/, 11%/ h(xl)))aee........ befL.....K Fit an estimator I + e(x) on De, using the squared loss l(e,e') = cee2
Return:
The predictor x + u(x) = e(x) a(x).
8

---

Published in Transactions on Machine Learning Research (02/2023)
3.2 Interactive Settings
Interactive setlings, in which EU estimates are used to guide the acquisilion of new examples, provide more interesting use case for DEUP. However, they bring their own challenges, as the main predictor is retrained multiple times with the newly acquired points. We discuss these challenges below, along with simple ways to mitigate them.
First, as the growing training set z² = {(r,, Ui Wi))e( for the main predictor hgx changes at cach acquisition step (as z becomes z by incorporating the pair (IN+1, MN+1) for example), it is necessary to view the risk estimate e(h,x,2) as function of the pair (z, >M rather than a function of only, unlike the fixed training set setting. The error predictor e, used to guide acquisition, needs to provide accurate uncertainty estimates as soon as the main predictor changes from lug to hgN11 11 (for clarity, we assunne that only one point is acquired at each step, but this is not a requirement for DEUP). This means that e needs lo generalize not only over input data 2, but also over training datasets 2 e should thus be trained using a datasct De of input-target pairs ((z, ), My, hgw (I))), where (a.y) not part of 2 This would make the error predictor robust. to feedback covariate shift (Pannjiang et al., 2022), iee. the distribution shift resulting from the acquired points being function of the training data. Conditioning address concerns from Bengs et al. (2022) which shows that directly estimating uncertainty using second order predictor trained simply with input data using L2 error yields poor uncertainty estimates. A learning algorithm with good generalization properties (such as modern neural notwork)), would in principle be able to extrapolate from De, and estimate the errors made by a predictor h on points I E X not seen so far, i.e. belonging to what we call the frontier of knowledge.
the dataset explicitly also
Second, the learner usually does not have the luxury of having access to a held-out validation set, given that the goal of such interactive learner is to learn the Bayes predictor f using little data as possible. This makes Algorithm 1 inapplicable to this setting, While it might be tempting to replace the held-out set z in Algorithm 1 with the training set 2 this would lead to an error predictor trained with in-sample errors rather that out-of-sample errors, thus severely limiting its ability to generalize its resulting uncertainty estimates out-of-sample and generally underestimating EU.
Denoting by No 2 the number of initially available training points before any acquisition, and observing that for i >> No. the pair (r,, vi) is not used to train the predictors h Nott h_.-1, we propose to use the future acquired points out-of-sample examples for the past predictors, in order to build the training dataset D. for the error estimator. At step M No, i.e. after acquiring (M No) additional input-output. pairs (I, y) and obtaining the predictor hyM, De is equal to:
M U N= N=No
D.
(11)
Using De (11) requires storing in memory all versions of the main predictor de (i.e. heMos.... ltar M) which is impractical. Additionally, requires using predictors that take as an input a dataset of arbitrary size, which might lead to overfitting issues as the dataset size grows. Instead, we propose the following two approximations of De:
We embed each input pair (xa, = in a feature space o; and replace each such pair in D. with the feature vector dex (z), hereafter referred to as the stationarizing features of the dataset I at x.
2. To alleviate the need of storing multiple versions of h, we make each pair (a; Ui) contribute to De once rather than i i- No times, by replacing the inner union of (11) with the singleton Said differently, for each predictor hgN, only the next acquired point (14+1,9N+1) is used to populate D...
[((x,, (iara hat
These approximations result in the following training dataset of the error estimator at step M:
D. {(ds: (2xz), (yr, bas 1
(12)
9

---

Published in Transactions on Machine Learning Research (02/2023)
Algorithm 2 Pre-filling the uncertainty estimator training dataset Dz
De -0
while |Del < Npretroin do
Split: Dinit = =No into K random subsets , of equal size. Define D = =UK= Dr. Fit a new predictor he on D, and fit the features o5 on D.
De + De DU,N)ED ((tp)))((((((()))))))))))))")))) (z))))))))
end
In this paper, we explored (2N(x) = x 8, ((x >)), V(C, x*)))) or a subset of these 4 features, where ((x )>a) is a density estimate from data N at x; 8= if x is part of NN and otherwise 0, É a learner that produces a distribution over predictors, c.g. a GP or an ensemble of neural networks (Lakshminarayanan el al., 2017)), and V(Z, =,,,) is an estimate of the model variance of [(**) at x. Note that E can be chosen to be the same as L (Sec. 2). For numerical reasons, we found preferable to use log 9 and log V instead of or V. as input features. q can be obtained by training density estimator (such as Kernel Density Estimator or flow-based deep network (Rezende & Mohamed, 2015)). Like the other predictors, the density estimator also needs to be fine-tuned when new data is added to the training set. While these features are not required per se to train DEUP, they provide clues to help training the uncertainty estimator, and one can play with the trade-off of computational cost versus usefulness of each clue. They also provide DEUP the flexibility to incorporate useful properties. For instance, the density can help in incorporating distance awareness which Liu et al. (2020) show is critical for uncertainty cstimation. They sometimes come at no greater cost, if our main predictor is the mean prediction of the learner's output distribution, and if we use the corresponding variance as the only extra feature, as is the case in the experiments of Sec. 5.1 with GPs. As (12) is merely an approximation of (11), not all subsets of features are expected to be useful in all settings. In the ablations presented in Appendices A and C, we experimentally confirm that x alone is not sufficient, and the stationarizing features are critical for reliable uncertainty estimates. Additionally, we observe that having x as part of ø2N can help in some tasks and hurt in others. We note that the choice of features here is a design choice, and the usage of other features could be investigated in future work.
Pre-training the error predictor: If the learner cannot afford to wait for a few rounds of acquisition in order to build a dataset D, large enough to train the error predictor (e.g. when the prediction target y is the output of a costly oracle), it is possible to pre-fill D. using the No initially available training points Noo only, following a strategy inspired by K-lold cross validation. We present one such strategy in Algorithm 2. The procedure stops when the training dataset for the secondary learner, Du; contains at least Npretrain elements. In our experiments, we choose Npretrain to be small multiple of the number of initial training points.
Putting all these things together yiclds the pscudo-code for DEUP in interactive learning settings provided in Algorithm& 3. In practice, in addition to out-of-sample errors, the secondary predictor can be trained with in-sample errors, which inform the error predictor that the uncertainty at the training points should be low. The stopping criterion is defined by the interactive learning task. can be a fixed number of iterations for example, a criterion defined by a metric evaluated on a validation set, or a criterion defined by the optimal value reached (maximum or mimumim) in SMO. The algorithm is agnostic to the acquisition function policy T(. h, u), the acquisition machinery that proposes new input points from X, using the current predictor h and its corresponding EU estimator u. Examples of acquisition functions in the context of SMO include Expected Improvement (Močkus, 1975) and Upper Confidence Bound (UCB, Srinivas et al. (2010)).
4 Related Work
Bayesian Learning. Bayesian approaches have received significant. attention as they provide a natural way of representing epistemic uncertainty in the form of the Bayesian posterior distributions. Gaussian Processes (Williams & Rasmussen, 1995) are a popular way to estimate EU, as the variance among the functions in the posterior (given the training data) can be computed analytically. Beyond GPs, efficient MCMC-based approaches have been used for approximating samples from the Bayesian posterior on large
10

---

Published in Transactions on Machine Learning Research (02/2023)
Algorithm 3 Training procedure for DEUP in an Interactive Learning setting
Inputs: Dinit = NNo {zi, Vi) (e........... a an estimator of aleatoric uncertainly. o, the embedding
space (i.e. the chosen stationarizing features). т, the acquisition machinery.
De + 0, training dataset for the error predictor e
D + Pinst: dataset of training points for the main predictor
Tary + 0, Y/cg +- 0
Optional: Pre-fill De using Algorithm 2 while stopping criterion not reached do Fit predictor hp and features ΦD on D
Fit a predictor o + e(ő) on D.
Lacy IT. h e(op(x)) a(x)) (can be either single point, or batch of points) Sample outcomes from the ground truth distribution: Yaeq ~ P(. Faeg)
D. + De U (()D( [yaog liD ((tcg))))
+ DU (Tacq. }
end
datasets (Welling & Teh, 2011; Zhang et al., 2020; Vadera et al., 2020a). In the deep learning context, Blundell et al. (2015); Kendall & Gal (2017); Depeweg et al. (2018) use the posterior distribution of weights in Bayesian Neural Networks (BNNs) (MacKay, 1992) to caplure EU. SWAG (Maddox et al., 2019) fits a Gaussian distribution on the first moments of SGD iterates building upon SWA (Izmailov et al., 2018) to define the posterior over the neural network weights. This distribution then used as posterior over the neural network weights. Dusenberry et al. (2020) parametrize the BNN with a distribution on rank-1 subspace for each weight. matrix. Other techniques that rely measuring the discrepancy between different. predictors a proxy for EU include MC Dropout (Gal & Ghahramani, 2016), that interprets Dropout (Hinton et al., 2012) as variational inference technique in BNNs. These approaches, relying on sampling of weights or dropout masks at inference time, share some similarities with ensemble-based meth that include bagging (Breiman, 1996) and boosting (Efron & Tibshirani, 1994), where multiple predictors are trained and their outputs are averaged to make a prediction, although the latter measure variability due to the training set instead of the spread of functions compatible with the given training set, as in Bayesian approaches. Deep Ensembles (Takshminarayanan et al., 2017) are closer to the Bayesian approach, using ensemble of neural networks that differ because of randomness in initialization and training (as you would have with MCMC, albeit in a more heuristic way). Wen et al. (2020) present memory-efficinn way of implementing deep ensembles, by using one shared matrix and a rank-1 matrix for the parameters per member, while Vadera et al. (2020b); Malinin et al. (2020) improve the efficiency of ensembles by distilling the distribution of predictions rather than the average, thus preserving the information about the uncertainty captured by the ensemble. Classical work on Query by commiltte (Seung et al., 1992; Freund al., 1992; Bachh-Bacchachhhh et al., 2005; Burbidge et al., 2007) also studied the idea of using discrepancy as a measure for information gain for the design of experiments. Tagasovska & Lopez-Paz (2019) propose using orthonormal certificates which capture the distance between a test. sample and the dataset. Liu et al. (2020) further establish the importance of this notion of distance awareness for uncertainty estimation, and along with proceeding methods DUE (van Amersfoort et al., 2021), and DDU (Mukhoti et al., 2021) combine feature representations learnt by deep neural networks with exact Bayesian inference methods like GPs and Gaussian Discriminant Analysis. This line of work falls in the umbrella of Deep Kernel Learning (Wilson et al., 2016, DKL:). van Amersfoort et al. (2020) present an alternative instantiation of DKL using RBF networks (LeCun al., 1998). DUN (Antoran et al., 2020) uses the disagreement between the outputs from intermediate layers measure of uncertainty. Another line of work that has received attention in the community recently is that of evidential uncertainty ostimation (Malinin & Gales, 2018; Sensoy et al., 2018; Amini et al., 2020) which estimates EU based on a parameteric estimate of the model variance, which has been shown to have poor uncertainty estimales in recent work by Bengs et al. (2022). Finally, Tran et al. (2022) combine several of these techniques in the context of large neural networks. Also closely related to the stationarizing features introduced in Soc. 3.2, Morningstar et al. (2021); Haroush al. (2021) use estimators of statistical features to capture the uncertainty for OOD detection.
11

---

Published in Transactions on Machine Learning Research (02/2023)
Fistrbbution-Free Uncertainty Estimation. Conformal Prediction (Vovk et al., 2005; Shafer & Vovk, 2008; Angelopoulos & Bates, 2021) is an alternative to Bayesian methods for uncertainty estimation. Conformal prediction involves building statistically rigorous uncertainty sets and intervals for model predictions, which are guaranteed to contain the ground truth with a specified probability. It is also closely linked to the statistical paradigm of hypothesis lesting (Angelopoulos & Bates, 2021). Conformal prediction an appealing alternative to Bayesian approaches as it can be applied on top of existing models, and does not require any special training procedure. Recent work has demonstrated the efficacy of applying conformal prediction with neural network models for time series (Lin et al., 2022; Zaffran et al., 2022) and image data (Angelopoulos et al., 2020). Fannjiang et al. (2022) study conformal prediction within an active learning setting. While DEUP can broadly be categorized as a distribution-fiee uncertainty estimation methods, it differs from Conformal Prediction as it does not require a pre-defined degree of confidence before outputling a prediction set.
Loss Prediction. Kull & Flach (2015) present several docompositions of the total expected loss, including the decomposition into the epistemic and irreducible (aleatoric) loss. They present additive adjustments that. reduce the scoring rules like the log-loss and Brier score, but do not tackle the general problem of uncertainty estimation. More closely related to DEUP, Yoo & Kweon (2019) propose a loss prediction module for lcarning to predict the value of the loss function. Hu et al. (2020) also propose using a separate network that learns to predict the variance of an ensemble. These methods, however, are trained only to capture the in-sample error, and do not capture the out-of-sample error which is more relevant for scenarios like active learning where we want to pick x where the reducible generalization error is large. EpiOut (Umlauft et al., 2020; Hafner et al., 2019) propose learning binary output that simply distinguishes between low or high EU..
5 Experiments
Through the experiments described below, we aim to provide evidence for the following key claims: (C1) Epistemic uncertainty measured by DEUP leads to significant improvements in downstream decision making tasks compared to established baselines, and (C2) The error predictor learned by DEUP can generalize to unseen samples. We emphasize that in order to make fair comparisons, DEUP does not have access to any additional OOD data during training, in all experiments presented in this section. Instead, when required, we use Algorithm 2 to generate the OOD data used for training the error predictor. Finally, note that in terms of computational cost, training DEUP with density and model variance as stationarizing features is on-par with training an ensemble of networks
5.1 Sequential Model Optimization
Sequential model optimization is a form of interactive learning, where at each stage, the learner chooses query examples to label, looking for examples with high value of the unknown oracle function. Such examples are selected so they have a high predicted value (to maximize the unknown oracle function) and a large predicted uncertainty (offering the opportunity of discovering yct higher valuos). Acquisition functions, such as Expected Improvement (EI, Močkus (1975)) trade-off exploration and exploitation, and one select the next candidate by looking for x's maximizing the acquisition function. We combine ET with DEUP (and call the method DEUP-EI, by analogy to GP-EI, a common SMO baseline), treating the main predictor and DEUP EU predictions at x respectively as mean and standard deviation of a Gaussian distribution for the learner's guess of the value of the oracle at x. Similarly, when using MC-Dropout or Deep Ensembles as prodictors, we refer to the resulting methods as MCDropout-EI and Ensembles-RR EI respectively.
We first consider in Fig. 3 (Left) synthetic one-dimensional function with multiple local maxima, and starting from initial dataset of 6 pairs (z, y), we compare the maximum values reached by each of DEUP-EL, GP-EI, MCDropout-EI, and Ensembles-Ei EI to a random acquisition strategy (which is an unsurprisingly strong baseline given the nunber of local optima of the target function). Because MCDropout and Ensembles are trained on sample data only, they are unable to generalize their uncertainty estimates, which makes them bad candidates for Sequential Model Optimization, because they are easily stuck in local minima, and require many iterations before the acquisttion function gives more weight to the predicted uncertainties than the current, maximum. For DEUP-FI, the main predictor is neural network, and the error predictor is GP
12

---

Published in Transactions on Machine Learning Research (02/2023)
regressor. The stationarizing features (x) used are the input ø itself and the variance of a GP fit the available data at every step. More details are provided in Appendix A. In Appendix A.2, we show a similar experiment on a two-dimensional function. In Appendix A.4, we provide an ablation study of different subsets of the features (x)), confirming the nccessity of extra features and the usefulness of the input I as well.
1.00
GP-EI
0.9
Random acquisition
0.50
Maan
Masimun valua ruached
DEUP-EI
McDroputtEl
0.25
EnsembleEl
True maximum
-0.25
0.6
0.5
-0.55
-1.0
-0.5
0.0
0.5
2.0
1.5
2.0
0
10
20
30
50
Number of additionl finction cals
Figure 3: Left. Synthetic function to optimize. Right. Maximum value reached by the different methods the synthetic function. The shaded areas represent the standard error across 5 different runs, with different. initial sets of 6 pairs. For clarity, the shaded arcas are omitted for the two worst performing methods. In each run, all the methods hods start with the same initial set of 6 points. GP-EI lends to get stuck in local optima and requires more than 50 steps, on average, to reach the global maximum.
Next, we showcase how using DEUP to calibrate GP variances (used as the only input for the error predictor) allows for better performances higher-dimensional optimization tasks. Specifically, we compare DEUP-EI TuRBO-EI (Eriksson et al., 2019), a state-of-the-art method for sequential optimization, that fits collection of local GP models instead of global one in order to perform efficient high-dimensional optimization, the Ackley function (Ackley, 2012), common benchmark for optimization algorithms. This function can be defined for arbitrary dinensions, and has many local minima.
300
-6
GPEE
-10
TUABO
DEUP
Mastmum
-12
GP
DEUP-B
TUABO-DEUP-E
-14
T-RRC-EI
-34
True maxthut
-16
T,880 REUPE
Addday
Figure Left. Max. value reached by the different optimization methods, for the 10 dimensional Ackley function. In each run, all the methods start with the same initial 20 points. Shaded arcas reprosent the standard error across 3 runs. Right. Max. value reached in the budget-constrained setting, on the Ackley functions of different dimensions. Error bars represent the standard error across 3 different runs, with different initial sets of 20 pairs. The budget is 120 function calls in total. Higher is better and TuRBO-DEUP-EI is less hurt by dimensionality.
In Fig. 4 (Left), we compare the different methods on the 10-D Ackley function, and observe that while GP-EI gets stuck in local optima, DEUP-EI is able to reach the global maximum consistently. In Fig. (Right), we show that for budget--onsttaiieed optimization problems, adapting DEUP TuRBO (called TuRBO-DEUP-EI) consistently outperforms regular TuRBO-EI, especially in higher dimensions. More details about the Ackley function and this experiment are provided in Appendix A.3. See also Appendix A for ID and 2D SMO tasks where DEUP-EI outperforms GP-EI (Bull, 2011), as well as neural networks with MC-Dropout or Ensembles. The experiments presented in this section thus validate our experimental claim C1.
5.2 Reinforcement Learning
13

---

Published in Transactions on Machine Learning Research (02/2023)
1429
Similar to SMO, key challenge in RL is efficient ex- ploration of the input state space. To investigate the effectiveness of DEUP's uncertainty estimates in the con- text of RL, we present a proof-of-concepy for incorporating epistemic uncertainties predicted by DEUP to DQN (Mnih et al., 2013), which we refer to as DEUP-DQN. Specifi- cally, we train the uncertainty predictor with the objective to predict the TD-error, using log-density estimates a stationarizing feature. The predicted uncertainties are then used as an exploration bonus in the Q-values. As explained in Sec. 3.2, it is the acquired points, before they are used to retrain the main predictor, that act as the oul-of-sample examples to train DEUP. In RI, because the targets (e.g. of Q-Learning) are themselves estimates and moving, data seen at any particular point is normally out-of-sample and can inform the uncertainty cstimator, when the inputs are used with the stationariiing features. Details of the experimental setup are in Appendix B. We evaluate DEUP-DQN on CartPole, classic RL task from bsuite (Osband et al., 2020), against DQN + e-greedy, DQN + MC-Dropout (Gal & Ghahramani, 2016) and Bootstrapped DQN (Osband et al., 2016). Fig. 5 shows that DEUP achieves lower regret faster, compared to all the basclines, which demonstrates the advantage of DEUP's uncertainty estimates for efficient exploration, confirming our claim C1. Future work should investigate ways to scale this method to more complex environments.
DON
DEUF-30R
MCDropout
Figure 5: Average regret on CartPole task. Error bars represent standard error across 5 runs.
5.3
Uncertainty Estimation
5.3.1 Epistemic Uncertainty Estimation for Drug Combinations
We validate DEUP's ability to generalize its uncertainty estimate (claim C2) in a real-world regression task predicting the synergy of drug combinations. While much effort in drug discovery is spent on finding novel small molecules, a potentially cheaper method is identifying combinations of pre-existing drugs which are synergistic (i.e., work well together). However, every possible combination cannot be tested due to the high monetary cost and time required to rum experiments. Therefore, developing good estimates of EU can help practitioners select experiments that are both informative and promising. we used the DrugComb and LINCS L1000 datasets (Zagidullin et al., 2019; Subramanian et al., 2017). DrugComb is dataset consisting of pairwise combinations of anti-cancer compounds tested on various cancer cell lines. For each combination, the dataset provides access to several synergy scores, each indicating whether the two drugs have a synergistic or antagonistic effect on cancerons coll death. LINCS L1000 contains differential gene expression profiles for various cell lines and drugs. Differential gene expressions measure the difference in the amount of mRNA related to set of influential genes before and after the application of drug. Because of this, gene expressions are a powerful indicator of the effect of a single drug at the cellular level. As shown in Fig. 6(c), the out-of-sample error predicted by DEUP correlates better with residuals of the model the test. set in comparison to several other uncertainty estimation methods. Moreover, DEUP better captured the order of magnitude of the residuals as shown in Fig. 6, confirming the claim C2. Details on experiments and metrics are in Appendix D.
5.3.2 Epistemic Uncertainty Predictions for Rejecting Difficult Examples
Epistemic uncertainty estimates can be used to reject difficult examples where the predictor might fail, such as OOD inputs? We thus consider a standard OOD Detection task (Liu et al., 2020; van Amersfoort et al., 2021; Nado et al., 2021), where we train ResNet. (He et al., 2016) model for CIFAR-10 classification (Krizhevsky, 2009) and reject OOD examples using the estimated uncertainty in the prediction. To facilitate rejection of classes other than those in the training set, we use Bernoulli Cross-
Loss for each class following van Amersfoort et al. (2020): v) = - E, ys log f((x) + (1 - 34) log(1 - f(()),, where y is a one-hot vector (v, 1 if i is the correct class, and 0 otherwisc), and f((z) predicted probability for class i. The target for out- of distribbiit data (from other classes) can be represented y 10...., 0}. We use Algorithm 1 with stationarizing features from Sec. 3.2 for training DEUP on this task. At inference time, we can reject an
Be.g. rare but challenging inputs can be directed to human, avoiding costly mistake
14

---

Published in Transactions on Machine Learning Research (02/2023)
Ordered Prediction Intervals
Ordered Prediction Intervals
60
50
Predicted Values and Intervals
Predicted Values and Intervals
60
40
30-
40
20-
10
Predicted Values
Predicted Values
Observed Values
Observed Values
0
10
20
40
50
10
20
30
40
Index (Ordered Observed Value)
Index (Ordered by Observed Value)
(a) Ensemble
(b) DEUP
Model
Corr. w. res.
U. Bound
Ratio
Log Likelihood
MC-Dropout
0.14 1 0.07
0.56 1 0.05
0.25 i 0.12
20.1 1 6.8
Deep Ensemble
0.30 + 0.09
0.59 + 0.04
0.50 = 0.13
-14.3 4.7
DUE
0.12 0.12
0.15 0.03
0.80 + 0.79
-13.0 1 0.52
DEUP
0.47 + 0.03
0.63 + 0.05
0.75 + 0.07
-3.5 - 0.25
(c) Quality of uncertainty estimates from different methods.
Figure 6: Drug Combinations. Predicted mean and uncertainty (error bars) on 50 test examples ordered by increasing value of true synergy score (orange). Model predictions and uncertainties blue. Ensemble (a) (and MC-dropout, not shown) consistently underestimate uncertainty while DEUP (b) captures the right (c) Corr. res. shows correlation between model residuals and predicted uncertainties 0. A best-case Upper Bound on Corr. w. res. is obtained from the correlation between è and true samples from 0). Ratio is the ratio between col. 1 and 2 (larger is better). Log-likelihood: average over 3 seeds of per sample predictive log-likelihood.
order of magnitude.
example based on the estimated epistemic uncertainty. To ascertain how well an epistemic error estimate sorts unseen examples by the above NLL loss, we consider the rank correlation between the predicted uncertainty and the observed generalization error. Note that we can compute the generalization error for OOD examples with y = fo.....0) .} directly since we are using a Binary Cross-Entrops loss. We use examples from SVHN (Netzer et al., 2011) as the OOD examples. This metric focuses on the quality of the uncertainty estimates rather than just their ability to simply classify in- vs oft-o--distribution examples. This metric is also indicator of how accurate the uncertainty estimates are for out-of-distribution examples. We also report the standard AUROC for the OOD detection task. We use MC-Dropout (Gal & Ghahramani, 2016), Deep Ensembles (Takshminarayanaa et al., 2017), DUE (van Amersfoort et al., 2021) and DUQ (van Amersfoort et al., 2020) as the baselines. As the primary focus of our work is estimating epistemic uncertainty, we do not consider specialized methods for OOD detection (Morningstar et al., 2021; Haroush et al., 2021). Additionally, to study the effect of model capacity we consider ResNet-18 and ResNet-50 as the main predictors for all the mothods. Additional training details along with results for ResNet-50 are presented in Appendix C.
Table 1 shows, supporting experimental claim C2, that with the variance from DUE (van Amersfoort et al., 2021) and the density from MAF (Papamakarios et al., 2017) stationarizing features, DEUP provides uncertainty estimates that have high rank correlation with the underlying generalization error on OOD data. We also achieve competitive AUROC with the strong baselines, demonstrating that uncertainty estimates from DEUP result in better performance on the downstream task OOD detection. Additionally, since the error predictor trained scparately from the main predictor, there is no explicit trado-off between the accuracy of the main predictor and the quality of uncertainty estimates. We achieve competitive accuracy of 93.89% for the main predictor. We ignore the effect of aleatoric uncertainty (due to inconsistent human
15

---

Published in Transactions on Machine Learning Research (02/2023)
Model
SRCC
AUROC
MC-Dropout
0.287 + 0.002
0.894 0.008
Deep Ensemble
0,381 + 0.004
0.933 + 0.008
DUQ
0.376 0.003
0.927 + 0.013
DUE
0.378 + 0.004
0.929 + 0.005
DEUP (D+V)
0.426 + 0.009
0.933 + 0.010
Table 1: Spearman Rank Correlation Coefficient (SRCC) between predicted uncertainty and OOD gener- alization error (SVHN); Area under ROC Curve (AUROC) for OOD Detection (SVHN) with CIFAR-10 ResNet-18 models (3 seeds). DEUP significantly outperforms baselines in terms of SRCC and is equivalent to Deep Ensembles but scoring better than the other methods in terms of the coarser AUROC metric.
labelling), which would require a human study to ascertain. We note that we choose the DUE baseline as it
is representative of related methods such as SNGP (Liu et al., 2020) and DDU (Mukhoti et al., 2021), and performs best in our experiments. We present additional results in distribution shift setting in Appendix C. Note that in the pretraining phase of the uncertainly estimator (Algorithm 2), we obtain the subsets by splitting the data based on classes, with each split containing [n/K] classes. So when we train on K - subsets, the [n/K] classes from the remaining subset become out-of-distribution.
6 Conclusion and Future Work
Whereas standard measures of epistemic uncertainty focus variance (due to approximation error), we argue that bias (introduced by misspecification) should also be accounted for as part of the epistemic uncertainty, as it is reducible for predictors like neural networks whose effective capacity is a function of the training data. By analyzing the sources of lack of knowledge, we describe how the excess risk is a sounds measure of epistemic uncertainty. This motivates the DEUP framework, where we train a second model predict the risk of the main predictor. In interactive settings, this nonetheless raises non-stationarity challenges for this estimator, and we propose extra input features to tackle this issuc, and show experimentally their advantages. Future work should investigate ways to improve the computational efficiency of DEUP (for instance by considering different choices of stationarizing features alternate ways of incorporating the dataset as input to the secondary predictor), and ways to estimate aleatoric uncertainty when no estimator thereof is readily available and when one cannot simply query an oracle several times on the same input x. Theoretically studying the properties of DEUP under model misspecification another important direction to explore in future work.
Acknowledgements
The authors would like to thank Tristan Deleu, Anirudh Goyal, Tom Bosc, Chritos Tsirigotis, Léna Néhale Ezzine, Doina Precup, Pierre Luc-Bacon, John Bradshaw, José Miguel Hernández-Toobto as well as the anonymous reviewers for useful comments and feedback. This research was enabled in part by support provided by Compute Canada, the Bill & Mclinda Gates Foundation, IVADO and CIFAR AI Chair. The authors also acknowledge funding from Samsung, IBM, Microsoft.
References
David Ackley. A connectionist machine for genetic hillclimbing, volume 28, Springer Science; Business Media, 2012.
Charu C Aggarwal, Xiangnan Kong, Quanquan Gu, Jiawei Han, and S Yu Philip. Active learning: A survey. In Dala Classification: Algorithms and Applications, pp. 571 605. CRC Press, 2014.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. In Neural Information Processing Systems (NeurIPS), 2020.
16

---

Published in Transactions on Machine Learning Research (02/2023)
Anastasios N Angeloponlos and Stephen Bates. A gentle introduction to conformal prediction and distribution- free uncertainty quantification. arXiv preprint arXiv:2107.07511, 2021.
Anastasios Nikolas Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations, 2020.
Javier Antoran, James Allingham, and José Miguel HernándezzLLoato. Depth uncertainty in neural networks In Neural Information Processing Systems (NeurIPS), 2020.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look memorization in deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 232. PMIR, 06-11 Aug 2017.
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy, BoTorch: Framework for Efficient Monte-Carlo Bayesian Optimization. Advances in Neural Information Processing Systems 83, 2020. URL
www.ariww...oooaaaaeeaeeeaeeeeee (trggabs//991 06403.
Pedro J Ballester and John BO Mitchell. A machine learning approach to predicting protein-ligane binding affinity with applications to molecular docking. Bioinformatics, 26(9):1169-1) 1199-115,7 2010.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. NeurIPS 2021, arXiv:2106. 04399, 2021.
Viktor Bengs, Eyke Hüllermeier, and Willem Wacgeman. On the difficulty of epistemic uncertainty quantifi- cation in machine learning: The case of direct uncertainty estimation through loss minimisation. arXiv preprint arXiv:2203.0611 06100, 2022.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: review for statisticians. Journal of the American statistical Association, 112(518):859-877 877, 2017.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613-1622. PMLR, 2015.
George EP Box. Science and statistics. Journal of the American Statistical Association, (1(56)):791-799 1976.
George EP Box and Norman R. Draper. A basis for the selection of response surface design. Journal of the American Statistical Association, 54(287):662-66)) 654, 1959.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123 140, 1996.
Steve Brooks, Andrew Gelman, Galin Jones, and Xiao-Li Meng. Handbook of markov chain monte carlo. CRC pross, 2011.
Adam D Bull. Convergence rates of efficient global optimization algorithms. Journal of Machine Learning Research, 12(10), 2011.
Robert Burbidge, Jem Rowland, and Ross King. Active learning for regression based on query by committee. In International conference on intelligent data engineering and automated learning, pp. 209 218. Springer, 2007.
Maria R. Cervera, Rafael Dätwyler, Francesco D'Angelo, Hamza Keurti, Benjamin F. Grewe, and Christian Henning. Uncertainty estimation under model misspecification in neural network regression, 2021.
Youngscog Chung, Willie Neiswanger, Ian Char, and Jeff Schneider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. arxio preprint. arXiv:201 1.09588, 2020.
17

---

Published in Transactions on Machine Learning Research (02/2023)
Tomas Cihlar and Marshall Fordyce. Current status and prospects of hiv treatment. Current opinion in virology, 18:50-66 56, 2016.
Francesco D' Angelo and Christian Henning. On out-of-distribution detection with bayesian neural nelworks. arXiv preprint arxi: Aroio-2110.0600 2021.
Augustus De Morgan. Formal logic: or, the calculus of inference, necessory and probable. Taylor and Walton, 1847.
Stefan Depeweg, Josc-Migucl Hornandez-Lobate Finale Doshi-Velez, and Steffen Udluft. Decomposition of uncertainly in bayesian deep learning for elficient and risk-sensitive learning. In International Conference Machine Learning, pp. 1184-1192, PMLR, 2018.
Michael W. Dusenberry, Ghassen Jerfel, Yeming Wen, Y. Ma, Jasper Snoek, K. Heller, Balaji Lakshmi- narayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 1 factors. In International Conference on Machine Learning (ICML), 2020.
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrup. CRC press, 1994.
David Eriksson, Michael Pearce, Jacob R Gardner, Ryan Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. arXie preprinl arXiv:1910.01799 2019.
Clara Fannjiang, Stephen Bates, Anastasios Angelopoulos, Jenuifer Listgarten, and Michael I Jordan. Conformal prediction for the design problem. arXiv preprint arXiv:2202.083: 03610, 2022.
P. Frazier. A tutorial on bayesian optimization. ArXin, tbs/1807.00811, 2018.
Yoav Freund, H Sebastian Seung, Eli Shamir, and Naftali Tishby. Information, prediction, and query by committec. Advances in neural information processing systems, 5, 1992.
Futoshi Futami, Tomoharu Twata, Naonori Ueda, Issei Sato, and Masashi Sugiyama. Fxcess risk analysis for epistemic uncertainty with application to variational inference. arXiv preprint arXiv: Arriv-2206. 20ni0-2006.01666 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (ICML), pp. 1050-1059. PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference Machine Learning, pp. 1183 1192. PMLR, 2017.
Roman Garnett. Bayesian Optimization. Cambridge University Press, 2022. in preparation.
Ran Bachrac, Amir Navot, and Naftali Tishby, Query by committee made real. Advances in neural information processing systems, 18, 2005.
Peter Grünwald. The safe bayesian. In International Conference on Algorithmic Learning Theory, pp. 169-183. Springer, 2012.
Peter Grünwald and Thijs Van Ommen. Inconsistency of bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesion Analysis, 12(4):1069-110 2017.
Danijar Hafner, Dustin Tran, Timothy Lillicrap, Alex Irpan, and James Davidson. Noise contrastive priors for functional uncertainty. Conference on Uncertainty Artificial Intelligence (UAT), 2019.
Matan Haroush, Tzviel Frostig, Ruth Heller, and Daniel Soudry. A statistical framework for efficient out of distribution detection in deep neural networks. In International Conference on Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEFE conference on computer vision and paltern recognition, pp. 770-77 770-778, 2016.
18

---

Published in Transactions on Machine Learning Research (02/2023)
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Leurning Represenlations, 2019.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Πlya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation feature detectors. arXiv preprint 1rx7. 2012.
Lara Hoffmann and Clemens Elster. Deep ensembles from bayesian perspective. arXiv preprint
arXiu:2105.13288 2021.
Liang Hong and Ryan Martin. Model misspecification, bayesian versus credibility estimation, and gibbs posterions. Scandinavion Actuarial Journal, 2020(7):6344 649, 2020.
S. Hu, Nicola Pezzotti, and M. Welling. A new perspective on uncertainty quantification of deep ensembles. arXiv, 2020.
Alan Häjek. Interpretations of Probability. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Fall 2019 edition, 2019.
E. Hüllermeier and W. Waegeman. Alcatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Mach. Learn., 2019. doi: 10.1007/5109944-21-055555
Pavel Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. Wilson. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
David Janz, Jiri Hron, Przemystaw Mazur, Katja Hofmann, José Miguel Hernández-Lobato, and Sebastian Tschiatschek. Successor uncertainties: Exploration and uncertainty in temporal difference learning. In Neural Information Processing Systems (NewrIPS), 2019.
Donald R. Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-492, 1998.
Satyen Kale, Ayush Sckhari, and Karthik Sridharan. Sgd: The role of implicit regularization, batch-size and multiple-epochs. In NewrIPS, 2021.
Alex Kendall and Yarin Gal. What uncertainties do we need bayesian deep learning for computer vision? In Neural Information Processing Systems (NewrIPS), 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference for Learning Representations, 2015.
Armen Der Kiureghian and Ove Ditlevsen. Alcatory or epistemic? docs it matter? Structural Safety, 31 (2): 105 112, 2009. ISSN 0167-4730. doi: Risk Communication.
0008.06.000.0 Risk Acceplance and
Bas JK Kleijn and Aad W van der Vaart. The bernstein-von-mise theorem under misspecification. Electronic Journal of Statistics, 6:354-381, 2012.
Jeremias Knoblauch, Jack Jewson, and Thcodoros Damoulas. An optimizatiion-cnnticc view on bayes' rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 23(122): 109, 2022. URL. httt::/jmmrz org/
html.
Levente Koesis and Csaba. Szepesvári. Bandit based monte-carlo planning. In In: FCML-006 Number 4212 in LNCS, pp. 282 293. Springer, 2006.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Meclis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment precursor to calibration. In Machine Learning and Knowledge Discovery in Databases, 2015.
19

---

Published in Transactions on Machine Learning Research (02/2023)
H. J. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering, 86:97-106, 1964.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Neural Information Processing Systems (NeurIPS), 2017.
Yann LeCun, Lóon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-bascl learning applied to document. recognition. Proceedings of the IEEE, 86(11)22278 2324, 1998.
Mci-Ling Ting Lee, Frank C Kuo, GA Whitmore, and Jeffrey Sklar. Importance of replication in microarray gene expression studies: statistical methods and evidence from repetitive edna hybridizations. Proceedings of the National Academy of Sciences, (1334999),,)3), 2000.
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Conformal prediction with temporal quantile adjustments. ArXio, abs/2205509944m 2022.
Jeremiah Zhe Liu, Zian Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax-Weiss and Balaji Labshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Neural Information Processing Systems (NeurIPS), 2020.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4(3): 448-472, 1992.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson simple baseline for bayesian uncertainty in deep learning. In Neural Information Processing Systems (NeurIPS), 2019.
A Malinin and M Gales. Predictive uncertainty estimation via prior networks. In NIPS'18: Proceedings of the 32nd International Conference on Neural Information Processing Systems, 2018.
Andrey Malinin, Bruno Mlodozenice, and Mark Gales. Ensemble distribution distillation. In International Conference on Learning Representations, 2020. URL http:://pppareersse.ecc////rrrr/
Alina Malyutina, Muntasir Mamun Majumder, Wenyu Wang, Alberto Pessia, Caroline A Heckman, and Jing Tang. Drug combination sensitivity scoring facilitates the discovery of synergistic and efficacious drug combinations in cancer. PLoS computational biology, 15(5):e1006752 2019.
Andres Masegosa. Learning under model misspecification: Applications to variational and ensemble methods. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neurol Information Processing Systems, 2020.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglon, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement. learning. ar preprint arXiv:1312 2.022, 2013.
Jonas Močkus. On bayesian methods for seeking the extremum. In Optimization techniques IFIP technical conference, pp. 400-404. Springer, 1975.
Reza Bayat Mokhtari, Tina S Homayouni, Narges Baluch, Evgeniya Morgatskaya, Sushil Kumar, Bikul Das, and Herman Yeger. Combination therapy in combating cancer. Oncolargel, 8(23):38022, 2017.
Harry Morgan. The generation of a unique machine description for chemical structures-a technique developed chemical abstracts service. Journal of Chemical Documentation, 5(2):107-111) 1965.
Warren Morningstar, Cusuh Ham, Andrew Gallagher, Balaji Takshminarayanan, Alex Alemi, and Joshua Dillon. Density of states estimation for out of distribution detection. In International Conference on Artificial Intelligence and Statistics, pp. 3232-3240. PMLR, 2021.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip H. S. Torr, and Yarin Gal. Deep deterministic uncertainty: A simple baseline, 2021.
20

---

Published in Transactions on Machine Learning Research (02/2023)
Zachary Nado, Neil Band, Mark Collier, Josip Djolonga, Michael Dusenberry, Sebastian Farquhar, Angelos Filos, Marton Havasi, Rodolphe Jenatton, Ghassen Jerfel, Jeremiah Liu, Zelda Mariet, Jeremy Nixon, Shreyas Padhy, Jie Ren, Tim Rudner, Yeming Wen, Florian Wenzel, Kevin Murphy, D. Sculley, Balaji Lakshminarayanan, Jasper Snock, Yarin Gal, and Dustin Tran. Uncertainty Basclines: Benchmarks for uncertainly & robustness in deep learning. ai preprint or urxiv:2106 04015, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Newral Information Processing Systems (NIPS), 2011.
Vu-Linh Nguyen, Sébastien Destercke, and Eyke Hüllermeier. Epistemic uncertainty sampling. In International Conference on Discovery Science, pp. 72 86. Springer, 2019.
World Health Organization and Stop TB Initiative. Treatment of tuberculosis: guidelines. World Health Organization, 2010.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrappod dqn. arxi preprint orXiv:1602.04 04621, 2016.
Ian Osband, Yotam Doron, Mattco Hcssel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreview.uu wttp://wwwweeeeetoeo/reeeeerre/e
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Neural Information Processing Systems (NewrIPS), 2017.
FP Ramsey. Truth and probability. reprinted in. Studies in subjective probabilily, pp. 61-92, 1926.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows In International Conference on Machine Learning (ICML), pp. 1530-1538. PMLR, 2015.
Nicholas Schurch, Pietá Schofield, Marek Gierliáski, Christian Cole, Alexander Sherstnev, Vijender Singh, Nicola Wrobel, Karim Gharbi, Gordon G Simpson, Tom Owen-Hughes, et al. How many biological replicates are needed in an rna-seq experiment and which differential expression tool should you use? Rna, 22(6):
839-851, 2016.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Neural Information Processing Systems (NeurIPS), pp. (193-3193, 2018.
H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee. In Proceedings of the fifth annual workshop on Computational learning theory, pp. 287-294, 1992.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008
Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Neural Information Processing Systems (NeurIPS), 2012.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning (ICML), 2010.
Aravind Subramanian, Rajiv Narayan, Steven M Corsello, David D Pock, Tod E Natoli, Xiaodong Lu, Joshua Gould, John F Davis, Andrew A Tubelli, Jacob K Asiedu, al. next generation connectivity map: L.1000 platform and the first. 1,000,000 profiles. Cell, (61 1452, 2017.
21

---

Published in Transactions on Machine Learning Research (02/2023)
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. In Neural Information Processing Systems (N),r 2019.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. # exploration: study of count-based exploration for deep reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2017.
Dustin Tran, Jeremiah Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint ar Xixx:200.0044 11, 2022.
Jonas Umlauft, Armin Loderer, T. Beckers, and S. Hirche. Real-time uncertainty decomposition for online learning control. ArXiv, abs/2010.02613, 2020.
Meet P. Vadera, Adam D. Cobb, Borhan Jalacian, and Benjamin M Marlin. Ursabench: Comprchensive benchmarking of approximate bayesian inference methods for deep neural networks. ArXiv, abs/2007.00 2020a.
Meet P. Vadera, Borhan Jalaeian, and Benjamin M Marlin. Generalized bayesian posterior expectation distillation for deep neural networks. In UAI, 2020b.
Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. Improving deterministic uncertainly estimation in deep learning for classification and regression. CoRR, abs/2102 1222201111400 1409, 2021. URL https: /ttpe////aaiv..eeeeeeeeeeeeeeaar http://axxvv.orab//////e
11409.
Joost R. van Amersfoort, L. Smith, Y. Teh, and Yarin Gal. Simple and scalable epistemic uncertainty ostimation using single deep deterministic neural network. In International Conference on Machine Learning (ICML), 2020.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer Science Business Media, 2005.
Stephen G Walker. Bayesian inference with misspecified models. Journal of Stalistical Planning and Inference, 143(10):1621 -13113661,1 2013.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, 2011.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: An alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
C. K. Williams and C. Rasmussen. Gaussian processes for regression. In Neural Information Processing Systems (NewrIPS), 1995.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Aolin Xu and M. Raginsky. Minimum excess risk in bayesian learning. ieee transactions on information (heory, 2020. doi: 20.100/0..022..1770000
Donggeun Yoo and I. Kweon. Learning loss for active learning. 2019 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 93 102, 2019.
Margaux Zaffran, Aymeric Dieuleveut, Olivier F'eron Yannig Goude, and Julie Josse. Adaptive conformal predictions for time series. In ICML, 2022.
Bulat Zagidullin, Jehad Aldahdooh, Shuyu Zheng, Wenyu Wang, Yinyin Wang, Joseph Saad, Alina Malyutina, Mohieddin Jafari, Ziaurrchman Tanoli, Alberto Pessia, et al. Drugcomb: an integrative cancer drug combination data portal. Nucleic acids research, 47(W1):W43 W51, 2019.
22

---

Published in Transactions on Machine Learning Research (02/2023)
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In TCLR, 2020.
Shuxing Zhang, Alexander Golbraikh, Scott Oloff, Harold Kohn, and Alexander Tropsha. A novel automated lazy learning qsar (all-qsar) approach: method development, applications, and virtual screening of chemical databases using validated all-qsar models. Journal of chemical information and modeling, 46(5):1984 1995, 2006.
Xiaojie Zhou, Lawrence Joseph, David B Wolfson, and Patrick Bélisle. A bayesian a-optimal and model robust design critcrion. Biometrics, 58(4):1062-1088, 2003.
James V Zidek and Constance Van Eeden. Uncertainty, entropy, variance and the effect of partial information. Lecture Notes-Monograph Series, pp. 155-167, 2003.
A Sequential Model Optimization Experiments
For all our Sequential Optimization algorithms, we use Algorithm 3 to train DEUP uncertainty estimators. We found that the optional step of pre-filling the uncertainty estimator dataset De was important given the low number of available training points. We used half the initial training set (randomly chosen) as in-sample examples (used to train the main predictor and an extra-feature generator) and the other half as out-of-samplc examples to provide instances of high epistemic uncertainty to train an uncortainty predictor; we repeated the procedure by alternating the roles of the two halves of the dataset. We repeated the whole procedure twice using a new random split of the dataset, thus ending up with 4 training points in De for every initial training point in Dinit-
The error predictor is trained with the log targets (i.e. log MSE between predicted and observed error). This helps since the scale of the errors varies over multiple orders of magnitude.
Computationally, the training time of DEUP-EI depends on various choices (e.g. the features used to train the epistemic uncertainty predictor, the dimension of the input, the learning algorithms, etc..). Additionally, the training time for the uncertainty predictor varies at cach step of the optimization. In total, the sequential optimization experiments took about 1 CPU day.
We use 4oTorch (Balandat et al., 2020) as the base framework for our experiments.
A.1 One-dimensional function
For Random acquisition, sampled for different seeds 56 points, and used the (average across the seeds of the) maximum of the first 6 values as the first value in the plots (Figs. 3 and 4). Note that because the function is specifically designed to have multiple local maxima, GP-EI also required more optimization steps, and actually performed worse than random acquistion.
As a stationarizing input feature, we used the variance of a GP fit on the available data at every step. We found that the binary of variance feature and didn't tt improve the performance as captured by the number of additional function calls. We used GP for the DEUP uncertainty estimator. Using a neural net provided similar results, but was computationally more expensive in this 1-D case with few datapoints. We used a 3-hidden layer neural network, with 128 neurons per layer and a ReLU activation function, with Adam (Kingma & Ba, 2015) and a learning rate of 10-3 3 (and default values for the other hyperparameters) to train the main prodictor for DEUP-EI (in order to fit the available data). The same network architecture and learning rate were used for the Dropout and Ensemble baselines. We used 3 networks for the Ensemble baseline, and a dropout probability of 0.3 for the Dropout baseline, with 100 test-time forward passes to compute uncertainty estimates.
feature and density estimates were redundant with the
hhttps/ hhttss///ttch..rg/
23

---

Published in Transactions on Machine Learning Research (02/2023)
A.2 swo-dimensional function
To showcase DEUP's usefulness for Sequential Model Oplimization in with number of dimensions greater than 1, we consider the optimization of the Levi N.13 function, known benchmark for optimization. The function f takes a point (x, in 2D space and returns:
f(z, v) = -ssin (3Tx) + (x - () + sin + (y - 1) sin (2Ty)))
We use the box [-10, 10 as the optimization domain. In this domain, the maximum of the function is O, and it is reached at 11, The function has multiple local maxima, shown in Fig. 7(a)9.
Similar to the previous one-dimensional function, MCDropout and Ensemble provided bad performances and are omilled from the plot in Fig. 7(b). We used the same setting and hyperparameters for DEUP as for the previous function. DEUP-EI is again the only method that reaches the global maximum consistently in under 56 function ovaluations.
LeeyFarsten
299
-10
159
-30
Maximum
GP-EI
Random acquisition
DEUNEE
True maximum
20
Number additinnl fuection calls
(a) Visualization of (x, far
(b) Comparisons with GP-EI and Random acquisition
Figure 7: Sequential Model Optimization the Levi N.13 function
A.3 Additional details for the Ackley function experiment, for synthetic data in higher dimensions
The Ackley function of dimonsion d is defined as:
Ackleya B++
Σ cos(cr.)) A exp(1)
- Aexp - B
+ exp
i=1
where B is a hyperrectangle of IRd, (O...., 0) is the only global optimizer of Ackleya, at which the function is equal to 0. We use BoTorch's default values for A, B, which are 20, 0.2, 2T respectively.
In our experiments, we used = [-10, 15]d for all dimonsions d.
For the TurBO baseline, we use BoTcrs''s 's default implementation, with Expected Improvement as an acquisition function, and batch size of 1 (i.e. acquiring one point per step).
For fair comparisons, for DEUP, only input of the epistemic uncertainty predictor. This means that calibrate the GP variance to match the out-of-sample squared error, using another GP to perform the regression. TurBO-DEUP combination of both, which we perform the variance calibration task for the local GP models TurBO. The uncertainty
Gaussian Process the main model, and use its variance as the
Splot of the function copied from https://www.sfu.ee/ ssurjano/ /leyy 13.html
24

---

Published in Transactions on Machine Learning Research (02/2023)
Matemum value northed
0.0
0.7
911
Maxemum
0.5
Tue maxmrum
Figure 8: We train DEUP-EI to optimize the synthetic function of Fig. 3, with different subsets of the stationarizing features inputs to the error predictor, for 50 iterations. All the instances of the algorithm start with an initial set of 6 (x, v) pairs, (Teft) Maximum value reached at the 40-th iteration for each subset of the stationarizing features. The box plots represent the resulting distributions across 5 different runs. (Right) Evolution of the mcan (across the 5 runs) of the maximum value reached at each itcration, for the subsets of features that surpassed the 0.9 threshold.
predictor, i.e, the GP regressor, is trained with log targets, as in Appendix A.1, but also with log variances as inputs.
Note that only the stationarizing feature is used as input for the uncertainty predictor. When we used the input as well, we found that the GP error predictor overfits on the æ part of the input (z, v), and it was detrimental to the final performances. For all experiments, used 20 initial points.
A.4 Ablation study for the stationarizing features
In order to study the usefulness of each subset of the "xdvb" features (respectively representing the input v, density estimate, a variance estimate, and a bit indicating whether z was part of the training set), we compare in Fig. 8 the different subsets of the features (x) used input to the error predictor. Most notably, this confirms the importance of the stationarizing features besides the imput x, and shows that incorporating æ to the features can help improve the performances of an interactive learner.
B Reinforcement Learning Experiments
For RL experiments, we used bsuûle (Osband et al., 2020), a collection of carefully designed RL environments. bsuite also comes with list of metrics which aim evaluate RL agents from different aspects. We compare the agents based on the basic metric and average regret they capture both sample complexity and final performance. The default DQN agent is used the base of our experiments with 3 layer fully-connected (FC) neural network as its Q-network. For the Bootstrapped DQN baseline, used the default implementation provided by bsuite. To implement DQN + MC-Dropout, following the implementation from Gal & Ghahramani (2016), two dropout layers with dropout probability 0.1 are used before the second and the third FC layers. In order take an action, the agent performs a single stochastic forward pass through the Q-network, which is equivalent to taking a sample from the posterior over the Q-values, as done in Thompson sampling, an alternative e-greedy exploration.
As density estimator, used Kernel Density Estimator (KDE) with Gaussian kernel and bandwidth of to map states to densitics. This KDE is fit after cach 10000 steps (actions) with batch of samples from the replay bufler (which is of size 10000). The uncertainty estimator network (E-nelwork) has the same number of layers as the Q-network, with an additional Softplus layer at the end. All other hyperparameters are the same as the default implementation by Osband al. (2020). One complete training run for the DEUP-DQN with 5 seeds experiments takes about 0.04-0.05 GPU days on V100 GPU. In total RL experiments took about 0.15 GPU days Nvidia V100 GPU.
25

---

Published in Transactions on Machine Learning Research (02/2023)
Algorithm 4 DEUP-DQN
Initialize replay buffer D with capacity V
Qo(s, a): state-action value predictor
Es (log d): uncertainty estimator network, which takes the log-density of the states as the input
d(s): Kernel density estimator (KDE)
K: KDE fitting frequency
W: Number of warm-up episodes
for episode=1 to M do set So as the initial state
for t=1 to maz-sstepsper--pissode do with probability e: take random action, otherwise: if episode S W: store (st, ae, Tt, s(+1) in D
Qo(s1, a), else: max, [Qu(s:; a) + Esllog dl (a)] (naxa[Qu(ss..0)
Sample random minibatch B of transitions (sj, aj ;, 8j+1) from
if 8j is a final state: Y/ = Y; else: yj = rj + ymaza (5t,
Update Q-network:
0 + 0+ aq. Vo ((,,a)=BB [10 - a))
Update E-network:
- ag. Vo
( Vo
(w,-QQ(ss Eoolog Ma)
if mod((otal-stepp) K) = 0: fit the KDE d the states of D
end end
C Rejecting Difficult Examples
We adapt the standard OOD rejection task (van Amersfoort et al., 2020; Liu et al., 2020) and mea- sure the Spearman Rank Correlation of the predicted uncertainty with the true generalization crror, in addition to the OOD Detection AUROC. MC-Dropout and Deep Ensemble baselines are based on https://github..cm com/google////////ooooooooooooee /google/uncertaanty--essee uncertaintt--baselnee, DUQ based on https://gitt https://github.com /omm/yy///etttmmiiiii /dttermiiittic nucertainty-quantiicctae and DUE based on https:/ htt///ggtttt.cooo wtw.tuubb...mm/mmmmmm com/y0ast, DoE.Note that for the ResNet50 DEUP model continue using the ResNet-18 based DUE variance source.
Table 2: Spearman Rank Correlation between predicted uncertainty and the true generalization error on OOD data (SVHN) with ResNet-50 models (3 sceds) trained on CIFAR-10.
Model
ResNet-50
MC-Dropout
0.312 + 0.003
Deep Ensemble
0.401 = 0.004
DUQ
0.399 + 0.003
DEUP (D+V)
0.465 + 0.002
Training The baselines were trained with the CIFAR-10 training set with 10% set aside validation set for hyperparameter tuning. The hyperparameters not specified are set to the default values. For DEUP, we consider the log-density, model-variance estimate and the seen-unseen bit as the features for the error predictor. The density estimator we use is Masked- Autoregressive Flows (Papamakarios el al., 2017) and the variance estimator used is DUE (van Amersfoort et al., 2021). Note that as indicated earlier x, the input image, not used as a feature for the error predictor. We present those ablations in the next sub-scction. For training DEUP, the CIFAR-10 training set divided into folds, with each fold containing 8 unique classes. For each fold, we train an instance of the main predictor, density estimator and model variance estimator on only the corresponding 8 classes. The remaining
presented in Table 3 and Table 4. The hyperparameters
26

---

Published in Transactions on Machine Learning Research (02/2023)
classes act the out-of-distriuttion examples for training the error predictor. Using these folds construct dataset for training the error predictor, a simple feed forward network. The error predictor is trained with the log targets (i.e. log MSE between predicted and observed error). This helps since the scale of the errors varies over multiple orders of magnitude. We then train the main predictor, density estimator and the variance estimator on the entire CIFAR-10 10 dataset, for evaluation. The hyperparameteers are presented Table 4. For all models, we train the main predictor for 75 and 125 epochs for ResNct-18 and ResNet--50 respectively. We use SGD with Momentum (set 0.9), with multi-step learning schedule with decay of 0.2 at epochs [25, 50] and [45, 90] for ResNet-18 and ResNet-50 respectively, One complete training run for DEUP takes about 1.5-2 GPU days on V100 GPU. In total these set of experiments took about 31 GPU days on Nvidia V100 GPU.
Table 3: Left: Hyperparametens for training Deep Ensemble (Lakshminarayanann et al., 2017). Right: Hyperparameters for training MC-Dropout (Gal & Ghahramani, 2016).
Model
Parameters
Model
Parameters
ResNett18
ResNet-50
ResNet-18
ResNet-50
Number of samples
50
50
Number of members
Dropout Rate
0.15
Learning
0.05
0.01
L2 Regularization Coefficient
6e-6
8e-4
Learning Rate
0.05
0.01
Table 4: Left: Hyperparametors for training DUQ (van Amersfoort et al., 2020). Right: Hyperparamotors
for training DUE (van Amersfoort al., 2021).
Model
Parameters
Model
ResNet-18
Parameters
ResNet-18
ResNet-50
Inducing Points
50
Gradient Penalty
0.5
0.65
Kernel
RBF
Centroid Size
512
512
Lipschitz Coefficient
Length scale
0.2
BatchNorm Momentum
0.99
Learning Rate
0.05
0.025
Learning Rate
0.05
Weight Decay
0.0005
Ablations We also perform some ablation experiments to study the effect of each feature for the error predictor. The Spearman rank correlation cocfficient between the generalization error and the variance feature, V, from DUE (van Amersfoori et. al., 2021) alone is 37.84 + 0.04, and the log-density, D, from MAF (Papamakarios et al., 2017) alone is 30.52 + 0.03. With only the image (x) the SRCC is 36.58 + 0.16
Table presents the results for these experiments. We observe that combining all the features performs the best. Also note that using the log-density and variance as features to the error predictor we observe better performance than using them directly, indicating that the error predictor perhaps captures a better target for the epistemic uncertainty. The boolean feature (B) indicating seen examples, discussed in Sec. 3.2, also leads to noticeable improvments.
Table 5: Hyperparameters for training DEUP.
Model
Parameters
Res
Nes-50
Uncertainty Prodictor Architecture
1024 x 5
[0024] x 5
Uncertainty Predictor Epochs
100
100
Uncertainty Predictor LR
0.01
0.01
Main Predictor Learning Rate
0.05
0.01
27

---

Published in Transactions on Machine Learning Research (02/2023)
Table 6: Spearman Rank Correlation between predicted uncertainty and the true generalization error on OOD data (SVHN) with variants of DEUP with different features as input for the uncertainty predictor. D indicates the log-density from MAF (Papamakarios et al., 2017), V indicates variance from DUQ (van Amersfoort et al., 2020) and B indicates bit indicating if the data scen.
Model
Features
ResNet-18
ResNet-50
D+V+B
0.426 + 0.009
0.465 + 0.002
D+V
0.419 = 0.003
0.447 0.003
V+B
0,401 = 0,004
0.419 0.004
D+B
0.403 0.003
0.421 0.002
0.352 + 0.004
0.376
0.001
DV
0.382 = 0.006
0.397 + 0.002
C.1 Predicting Uncertainty under Distribution Shift
We also consider the task of uncertainty estimation in the setting of shifted distributions (Ovadia al., 2019; Hendryeks & Dietterich, 2019). We evaluate the uncertainty predictions of models trained with CIFAR-10, on CIFAR-10-C (Hendrycks & Dietterich, 2019) which consists of images from CIFAR-10 10 distorted using 16 corruptions like gassian blur, impulse noise, among others. Fig. 9 shows that even in the shifted distribution setting, the uncertainty estimates of DEUP correlate much better with the error made by the predictor, compared to the basclines.
Eisembres
5.55
Wwnmmaa cunduuan wonde//nnneeiiieee Colltven
0,25
5.30
Ingiade, Aodee
deborr Aber
passuss Mar
mon Aoer
ataatic Grendore
Blng comprestio
froe
Typ
Figure 9: Spearman Rank Correlation Coefficient between the predicted uncertainty and true error for models trained with CIFAR-10, and evaluated on CIFAR-10-C. DEUP outperforms the basclines on all types of corruptions.
D Drug Combination Experiments
To validate DEUP's uncertainty estimates in real-world setting, we measured its performance on a regression task predicting the synergy of drug combinations. While much effort in drug discovery is spent finding
28

---

Published in Transactions on Machine Learning Research (02/2023)
novel small molecules, potentially cheaper method is identifying combinations of pre-existing drugs which are synergistic (i.e., work well together). Indeed, drug combinations are the current standard-of-care for a number of diseases including HIV, tuberculosis, and some cancers (Cihlar & Fordyce, 2016; Organization & Initiative, 2010; Mokhtari et al., 2017).
However, due to the combinatorial nature of drug combinations, identifying pairs exhibiting synergism is challenging. Compounding this problem is the high monetary cost of running experiments on promising drug combinations, as well as the length of time the experiments take to complete. Uncertainty models could be used by practitioners downstream to help accelerate drug combination treatment discoveries and reduce involved development costs.
To test DEUP's performance on this task we used the DrugComb and LINCS L1000 datasets (Zagidullin et al., 2019; Subramanian et al., 2017). DrugComb is a dataset consisting of pairwise combinations of anti-cancer compounds tested on various cancer cell lines. For each combination, the datasct provides accoss to several synergy scores, each indicating whether the two drugs have a synergistic or antagonistic effect cancerous cell death. LINCS L1000 contains differential gene expression profiles for various cell lines and drugs, Differential gene expressions measure the difference in the amount of mRNA related to set of influential genes before and after the application of a drug. Because of this, gene expressions are a powerful indicator of the effect of a single drug at the cellular level.
In our experiments, each drug is represented by its Morgan fingerprint (Morgan, 1965) (with 1,024 bits and a radius of 3) as well as two differential gene expression profiles (each of dimension 978) from two cell lines (PC-3 and MCF-7). In order to use gene expression features for every drug, we only used drug pairs in DrugComb where both drugs had differential gene expression data for cell lines PC-3 and MCF-7.
We first compared the quality of DEUP's uncertainty estimations to other uncertainty estimation methods on the task of predicting the combination sensitivity score (Malyutina et al., 2019) for drug pairs tested on the cell line PC-3 (1,385 examples). We evaluated the uncertainty mothods using a train, validation, test split of 40%, 30%, and 30%, respectively. The underlying model used by each uncertainty estimation method consisted of a single drug fully connected neural network (2 layers with 2048 hidden units and output of dimension 1024) and combined drug fully connected neural network (2 layers, with 128 hidden units). The embeddings of an input drug pair's drugs produced by the single drug network are summed and passed to the combined drug network, which then predicts final synergy. By summing the embeddings produced by the single drug network, we ensure that the model is invariant to permutations in order of the two drugs in the pair. The models were trained with Adam (Kingma & Ba, 2015), using a learning rate of 1e-4 and weight decay of 1e-5. For MC-Dropout we used a dropout probability of 0.1 on the two layers of the combined drug network and 3 test-time forward passes to compute uncertainty estimates. The ensemble used constituent. models for its uncertainty estimates. Both Ensemble and MC-Dropout models were trained with the MSE loss.
We also compared against DUE (van Amersfoort et al., 2021) which combinos a neural network feature extractor with an approximate Gaussian process. Spectral normalization was added to all the layers of the combined drug network and of the single drug network. Let demb denote the dimension of the output of the combined drug network, which is also the input dimension of the approximate Gaussian process. We conducted a grid-scarch over different values of demb (from 2 to 100), the number of inducing points (from 3 to 200), the learning rate, and the kernel used by the Gaussian process The highest correlation of uncertainty estimates with residuals was attained with demb = 10, 100 inducing points, learning rate of 1e-2, and the Matern12 kernel.
6The Morgan fingerprint represents molecule associating with it boolcan vector specifying its chemical structure. Morgan fingerprints have been used signal of various molecular characteristics to great success (Ballester & Mitchell, 2010; Zhang al., 2006).
29

---

Published in Transactions on Machine Learning Research (02/2023)
Algorithm 5 DEUP for Drug Combinations
Data: D dataset pairwise drug combinations, along with synergy scores ((d1, d2). y)
Initialization:
Split training set into two halves, in-sample Din and oul-of-sample D
fu(d1, d2): predictor which takes pair of drugs as input
(d1, d2): In-sample din error predictor
So (d, d2): Out-of-sample dott error predictor
Training:
while training not finished do In-sample update
Get an in-sample batch (d1,in; d2,,n Yiu)) ~ Din
Predict f = fu(d1, in, d2,5n) and in-sample error Gin = S0 (di,inddg,n
Compute NLI: log(a?n) + (A-Min)
2
26%
Backpropagate through Su and for and update. Out-of sat-ol-sample update
Get an out-of-sample batch (di, out, Yout) ~ Dout
Estimate j = [_u(d1 o2)o and oul-of-sample ervor dout sool (d2 d2,out)
Compute NLL: (uue) + (B-Rou))²
20
Backpropagate through fo and update.
end
The DEUP model we used outputs two heads [6] and is trained with the NLL log(d") +
28 in similar fashion as in (Lakshminaraaaaaa et al., 2017). To obtain predictor of the out-of-sample error, altered our optimization procedure so that the μ and a heads were not backpropagated through at all times. Specifically, we first split the training set into two halves, terming the former the in-sample set Din and the latter the out-of-sample set Dout. We denote as för the in-sample error predictor and for the out-of-sample error predictor. fo is used estimate total uncertainty. Note that this setting, fo predicts the square root of the epistemic uncertainty (Gout) rather than the epistemic uncertainty itself (02ux).
In our experiments, extra bit is added input to the model in order to indicate whether a given batch is from Din or Dont Through this, the same model is used to estimate f and sout with the model estimating få" when the bit indicates an example is drawn from Di, and fout otherwise. When the batch is drawn from Din, both heads are trained using NLL using a single forward pass. However, when the data is drawn from Dout only the head is trained. To do this, we must still predict f in order to compute the NLL. But the á predictor f/ must be agnostic to the difference between Din and Dout. To solve this, we perform two separate forward passes. The first pass computes Å and sets the indicator bit to 0 fa has no notion of Dout, while the second pass computes o, setting the bit to to indicate the true source of the batch. Finally, we backpropagate through the & head only. The training procedure is described in Algorithm 5
We report several measures for the quality of uncertainty predictions on a separate test set in Table 7.
Model
Corr, w. res.
U. Bound
Ratio
Log Likelihood
Coverage Probability
CI width
MC-Dropout
0.14 t 0.07
0.56 + 0.05
0.25 0.12
-20.1 + 6.8
11.4 0.2
3.1 3.1:0.11
Deep Ensemble
0.30 + 0.09
0.59 + 0.04
0.50 0.13
-14.3 4.7
10.8 +
3.4 0.6
DUE
0.12 1 0.12
0.15 : 0.03
0.80 0.79
-13.0 0.52
15.2 + 1.0
3.5 1.5::00 0.1
DEUP
0.47 + 0.03
0.63 1 0.05
0.75 + 0.07
-3.5 + 0.25
36.1 + 2.5
13.1 + 0.9
Table 7: Drug combinations: quality of uncertainty estimates from different. methods. Corr. w. res. shows correlation between model residuals and predicted uncertainties à. A best-case Upper Bound on Corr. w. res. is obtained from the corrclation between à and true samples from (0,0)). Ratio is the ratio between col. 1 and 2 (larger is better). Log-likelihood: average over seeds of per sample predictive log- likeliooio Coverage Probabilily: Percentage of test samples which are covered by the 68% confidence interval. CT width: width of the 86% confidence interval.
30

---

Published in Transactions on Machine Learning Research (02/2023)
For each model, report the per sample predictive logllikelhood, coverage probability and confidence interval width, averaged over 3 seeds.
We also computed the correlation between the residuals of the model | and the predicted uncertainties (z,). We noted that the different uncertainty estimation methods lead to different distributions p(o(z)). For example, predicted uncertainties obtained with DUE always have a similar magnitude. By contrast, DEUP yields wide range of different predicted uncertainties.
These differences between the distributions p()(2)) obtained with the different methods may have an impact the correlation metric, possibly biasing the comparison of the different methods. In order to account for differences in the distribution p()(x)) across methods, we report another metric which is the ratio between the observed correlation Corr(lā(x) - yl. (x)) and the maximum achievable correlation given a specific distribution p()(x)).
This maximum achievable corrclation (refered to as the upper bound) is not per se a comparison metric, and is estimated (given a specific p()()))) as follows: we assume that, for each example (æi, ye), the predictive distribution of the model (a((zz),, (x.)) corresponds exactly to the distribution of the target, i.e. Vi ~ (N(()z)) (())). Under this assumption, the residual of the mcan predictor follows distribution We can then estimate the upper bound by computing the correlation between the predicted uncertainties (x;) and samples from the corresponding Gaussians NO0, (x;)). samples were drawn from each Gaussian for our evaluation. This upper bound is reported in the Table.
Finally, we reported our comparison metric: the ratio between the correlation Corr([A(z) - y) (x))) and the upper bound. The higher the ratio is, the closer the observed correlation is to the estimated upper bound and the better the method is doing.
It is interesting to note that the upper bound is much lower for DUE compared to other methods, as its predicted uncertainties lie within a short range of values.
Predicted f and uncertainly estimates can be visualized in Fig. 10 for different models. MC-dropout, Ensemble and DUE consistently underestimate uncertainty, while the out-of-sample uncertainties predicted by DEUP are much more consistent with the order of magnitude of the residuals. Morcover, observed that DUE predicted very similar uncertainties for all samples, resulting in lower upper-bound for the correlation between residuals and predicted uncertainties compared to other methods. We observed a similar pattern when experimenting with the other kernels available in the DUE package, including the standard Gaussian kernel.
Finally, we note that the context of drug combination experiments, aleatoric uncertainty could be estimated by having access to replicates of a given experiment (c.f. Sec. 3), allowing us to subtract the aleatoric part from the out-of-sample uncertainty, leaving us with the epistemic uncertainty only.
31

---

Published in Transactions on Machine Learning Research (02/2023)
Ordered Tredicion intevall
Orreced Preduction Inbervlls
Ordered Predict on Inferven
Ordered Tredition Intevvals
That cnd Natar art plece
heaitted Nalas Charree Wan
Noustted Nalucc Chanreer
Inkes Sittora Stearod
inder Snsoeeeeae
tol Oesenno Walee
(a) MC-dropout
(b) Ensemble
(c) DUE
(d) DEUP
Figure 10: Predicted mean and uncertainty for different models on a separate test set. 50 examples from the test set are ordered by increasing value of true synergy score (orange). Model predictions and uncertainties are visualized in blue. MC-Dropout, Ensemble and DUE consistently underestimate the uncertainty while DEUP seems to capture the right order of magnitude. Figures made using The Uncertainty Toolbox (Chung et al., 2020).
One complete training run for the drug combination experiments lakes about 0.01 GPU days on a V100 GPU. In total these set of experiments took about 0.2 GPU days on a Nvidia V100 GPU,
E DEUP in the presence of aleatoric uncertainty
We consider a scenario similar to that of Fig. 1, but in which Gaussian noise is added to the ground truth oracle before providing training examples. Because of the noisy training dataset, GP conflates epistemic and aleatoric uncertainty, which makes the gap between the predicted epistemic uncertainty (as measured by the GP variance) and the true epistemic uncertainty (as measured by the mean squared error between the GP mcan and the noiscless ground truth function) higher than in the deterministic setting of Fig. 1. The goal of this experiment is to illustrate that using the noisy training set allows learning an estimator of aleatoric uncertainty (AU), which could be subtracted from DEUP's error predictor e to obtain an estimator of epistemic uncortainty. The estimator of AU is obtained using (10). A simple linear regressor used to estimate AU, to ovoid overfitting issues.
A key distinction of this setting, is that DEUP's training data (the errors of the main predictor) are themselves noisy, which makes it important to use more out-of-sample data obtain reasonable total uncertainty estimates (from which we subtract the estimates of the aleatoric uncertainty).
32

---

Published in Transactions on Machine Learning Research (02/2023)
Ground truth
-2
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
10
Epistemic uncerfalnty GP variacee
Eptstemic uncertainty DEUP's prediction
Figure 11: Top. A GP is trained to regress function using noisy samples. GP uncertainty (model standard deviation) shaded in blue. Bottom left. Using GP variance as a proxy for epistemic uncertainty misses out on more regions of the input space, when compared to Figure out-of-sample data in low density regions, a second GP trained to predict the generalization error of the first GP (total uncertainty). Using sccond samples from the oracle for each the training points, a lincar regressor fits the training pairs (x, ( - y))²) to estimate the point-wise aleatoric uncertainly. Note that constraint is imposed on DEUP's outputs, which explains the predicted negative values for uncertainties. In practico, if these predicted uncertainties were to be used, (soft) clipping should be used.
Bottom right. Using additional
33


Reviews:
[{"review_id": "dmLoDzP2Rb", "reviewer": "Reviewer_iZLw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nAuthors took great care to:\n- establish the core ideas.\n- empirically validate the approach.\n\nWeaknesses:\n- In general \"fit-the-fit\" does not seem practically ideal as a proposed solution. If on in-data one finds the fit to be poor, then this suggests the underlying model could be improved. If one knows the definition of OOD then one could train a classifier directly to train it.\n- If one's goal is OOD, ideas such as \"Density of States Estimation for Out-of-Distribution Detection\" are SOTA, somewhat similar, but are not referenced. (I note that the referenced work does not succomb to the criticsm of the first point because it presumes no access to OOD data nor that error can be interpretted except insomuch as a fingerprint for \"typical error.\")\n- Perhaps I missed it, but I was disheartened to not see treatment of the model itself as a random variable and how it is a measure of uncertainty. While a GP is in principle doing this, it does so through the choice of kernel which itself could also be regarded as a rv and whose posterior variance would in principle indicate if theirs sufficient evidence for a particular belief.  Confusingly, the authors mention these ideas eg in \"Pitfalls of using the Bayesian posterior to estimate uncertainty\" but then fail to adequately compare to them.\n- Ab advantage of DEUP over \"discrepancy-based measures of EU\" is claimed that it could be adapted to use additional features. I take logical issue with this claim because why would we permit the DEUP model access features withheld from the main model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- A pedagogical depiction of sample/population risk, aleatoric/epistemic uncertainty, and Bayesian interpretations thereof.\n- \"Fit the fit\" procedure to estimate error adapted to in/out distribution and/or fixed/interactive data.\n- Experimental validation.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "We do not generally find \"fit-the-fit\" approaches to be a compelling research contribution and were not compelled to feel otherwise from this exposition. That said we'd deeply welcome being moved from this position as it is a research area of vital importance especially as ML techniques become increasingly \"launched\" in the real world.\n\nHere are two examples which we have found motivating; we list them in the hope of inspiring you to invent better alternatives:\n\n- Masegosa '20 operates under the premise that we want to use the Bayes loss for its posterior interpretation but recognize doing so introduces suboptimal posterior predictive accuracy in the presence of model misspecifcation. Ie: the main loss is desirable and we wish merely to \"tilt\" it toward some objective which ideally only different in the presence of model misspecification.\n- The \"Density of States Estimation for Out-of-Distribution Detection\" work offered a gamut of SOTA empirical results demonstrating that it was adequate to not fit the error (which is as hard as the core task) but merely fit the main model's _predictions_ and without withholding main model data or access to OOD.\n\nAdditionally, I'd like to see more effort dedicated to comparing to / showing the inadequacy of standard assumption validation practices eg Bayesian model criticsm.\n\nHowever our main concern remains the same: why wouldn't a systematic discrepancy a fit-the-fit uncovers not be something I'd want to always pass \"upstream\" to improve the main model? If error isnt iid then doesn't this simply mean training is broken, ie failed this diagnostic test? Isnt this idea precisely what Bayesian model criticsm and Masegosa's predictive spin do already?\n\nRegarding the overall presentation: I found the exposition to be lacking in \"new idea density\". The prose spent several pages on background and related work and in a way which I subjectively felt was at times too short and other times too long.  Figure 2 seemed particularly egregiously verbose; I believe left and right differ only in the inlusion of a \"prime\" differing lengths of arrows. Here's a few nits to offer some color on my concerns about the presentation:\n\nRegarding the abstract, the first two sentences triggered a bit of knee-jerk reaction. \n\nRe:first abstract sentence (and _not_ a problem in Intro definition of EU nor elsewhere): I agree it is a measure but I worry this is unecessarily vague/confusing to the uninitiated since noise in the data itself (aleatory) also imbues lack of knowledge in the learner.  As an alternative, how about \"Epistemic Uncertainty is a measure of the lack of knowledge of a learner which diminishes with more\nevidence.\"\n\nRe:second abstract sentence:\nI don't think its accurate to say that parameter uncertainty is a proxy. If you happened to know the model--because someone \"in the know\" told you--then it would be the appropriate definition. I also think the second sentence of the abstract could be\ninterpreted to imply that epistemic uncertainty cannot capture model misspecification error. In fact, if we treated the model as also a random variable then in principle we'd know how much evidence there is for a particular model. That we don't is probably because of computation and not lack of principle.\n\n\n--------------\n\n\n\"...shown to decrease and converge towards 0 as the training dataset size grows\nto infinity, which is a desirable property for EU measures.\"\nI think this statement could be improved considerably with a reference or some additional detail.\nPerhaps something along the lines of \"a self-evidently reasonable property of a measure of uncertainty owing to lack of training data is that it tends to zero as training data tends to infty.\"?\n\n-----\n\nI dont think its necessary to use `x\\mapsto` in `x \\mapsto A(x) = sigma^2(x)` since its convention to assume the relationshoip `A(x)=\\sigma^2(x)` holds for all `x`. (Indeed this is the assumption made everywhere else in the paper.)\n\n-----\n\n\n", "broader_impact_concerns": "No ethical concerns are present. The paper is a theory oriented paper and with intentions to make models more robust/transparent to invalid assumptions.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mGlhBrQ8ad", "reviewer": "Reviewer_MM5Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general, I think the paper is very well written. Its empirical validation seems sound, and a preliminary code review of the supplementary material is very promising. That is, I think the code looks really good and useful and might provide a good base for additional research. In its current state, the paper will be of interest to readers in the field and provide inspiration for further research questions. However, some clarifications and changes might be necessary to either explain or make sure that all claims are backed up by evidence and put into context correctly.\n\nThe following are not necessarily weaknesses but things that might be improved via editing. Please see these as coming from a different perspective (and not having read all prior art).\n\n## Active Learning\n\nFirst, the paper mentions active learning as a setting that we can use to evaluate epistemic uncertainty:\n\n1. It does not refer to original literature on active learning or earlier lit surveys. [Gal 2017](https://arxiv.org/abs/1703.02910) is one of the earlier papers that used EU for AL, though QBC (Query-by-Committee) from the 1990s might also count towards that depending on interpretation.\n2. There are no active learning experiments after setting these expectations: SMO, which seems to me to be the non-Bayesian setting that is also dealt with by Bayesian Optimization, is not active learning by itself.\n3. Algorithm 3's caption reads to be for the \"active learning\" setting, but it is generally used, while Algorithm 1 does not seem to be used by the paper experiments at all.\n\n## Figure 2\n\nFigure 2 (right) seems \"wrong\": H' should stay the same or become smaller with more data. If H is the universe of possible model functions, then it should stay the same. If we look at it from a Bayesian perspective (as a level set in posterior space) then it should become smaller. Imo, the important part is that $h_{z^M}$ should move closer to $h'^*$ as more data is used for training. (I still need to read Arpit 2017, but it might cause less cognitive dissonance if H' just remained as H.)\n\n## Definition of Epistemic Uncertainty\n\nThe definition of epistemic uncertainty as approximation uncertainty + model uncertainty/bias seems to contradict the definition of epistemic uncertainty as reducible uncertainty: commonly, it is reducible by gaining access to more data, not by also changing the model. Hence, the model bias is assumed as fixed usually, is it not? Malinin et al have also decomposed (total) uncertainty into aleatoric, epistemic and model uncertainty in one of their early papers (you are probably better aware of it than I). \n\nAs such, you might want to consider that you do not actually predict epistemic uncertainty but are looking at proxies for the generalization error.\n\n## DEUP as EU\n\nIn §3.1, the paper enumerates different examples of how to estimate aleatoric uncertainty. It would seem that in the cases where aleatoric uncertainty is set to 0, DEUP would not actually compute EU but total uncertainty or rather predict the full error. This is not epistemic uncertainty.\n\n*I know this is much to ask (so I won't ask explicitly in the requested changes), but I will wholeheartedly ask you to consider the following request here:\nCould you consider reframing the paper as Direct Reducible Error Prediction or similar? I believe the connection to that is much stronger evidenced and fits the narrative that is put forward better than claiming that it only predicts epistemic uncertainty.*\n\n## Theory $\\rightarrow$ Implementation\n\nFor me, the step from theory (11) to implemented method (12) is not clear, and a huge step: in particular, how important is $x$ in the feature vector that is used to train the error predictor? \n\nIt would seem that it should not be too important because: given that we train the predictor on the future dataset, training samples close to $x$ should be exactly where the model will have a better performance after retraining. Yet, usually, we add the most informative samples to the dataset (with the highest EU), so the error will be large for those samples, which means the error predictor will be trained to expect high losses, even though after retraining the model should have a small loss for the newly acquired data.\n\nAre the losses updated using the newly trained models? How does this square?\n\n## OOD via Statistical Tests\n\nThe proposed method seems related to [\"A statistical framework for efficient out of distribution detection in deep neural networks\"](https://arxiv.org/abs/2102.12967) and [\"Density of States Estimation for Out-of-Distribution Detection\"](https://arxiv.org/abs/2006.09273), in that it uses several different statistics to predict OOD. (The proposed approach except for including $x$ could be seen as predicting the error given statistics of estimates, like feature-space density or prediction variance.)\n\n## SMO Experiment\n\nWhy do all other SMO methods perform worse than random? Given that the function is sufficiently smooth, a GP should perform better. Was the length scale chosen correctly?\n\n---\n\nThese are a lot of questions. Answers will help me understand the paper better and come up with a recommendation.\n\nI would like to reiterate that the paper is really well written overall. I particularly liked §2 and specifically §2.3 on model misspecification as well as the related work in §4. It's great to see these connections and explanations written down clearly.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a framework, DEUP=Direct Epistemic Uncertainty Prediction, to predict epistemic uncertainty.\n\nThe paper discusses different forms of uncertainty and looks at model misspecification as a reason why Bayesian epistemic uncertainty might not be sufficient.\n\nEmpirically, the paper shows that their proposed approach works well in several different settings and on several different tasks (Sequential Model Optimization, Reinforcement Learning, and Uncertainty Estimation for Drug Combinations, OOD Detection & Rejection Classification).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Given my current state of knowledge:\n\n## For Acceptance\n\n1. Algorithm 1 (non-interactive) does not seem to be used in any of the experiments. Please rename it to make clear that is educational only to illustrate the original concepts. Algorithms 2 and 3 seem to be DEUP. (Otherwise, experiments for Algorithm 1 would be missing.)\n\n2. I am not convinced that the motivation and first sections in Section 3 (DEUP) are material to why DEUP works. As such, either ablations are needed to show (11) -> (12) - which I am not sure can be done tbh - or it should be made clear that this is speculative.\n\n    For the latter, for acceptance, I would suggest restructuring the paper such that the exposition of DEUP starts with (12) and Algorithms 2 and  3. The preceding sections are then motivation and inspiration but not theoretical grounding.\n\n3. I would like to ask for an ablation or investigation of how (11)/(12) compares to the losses of the model trained with the new data because I don't understand how it can be a good error predictor around training data when $x$ is included in the features $\\phi(x)$. \n\n## Improvement Suggestions\n\n1. Actual active learning experiments, given that it is mentioned a lot, and AL != SMO (or edit the paper to reduce the reader's expectations to find these).\n2. Update Figure 2 according to the concerns detailed in the previous section.\n3. Mention earlier in the paper that you do not require held-out/OOD data for DEUP. The initial sections make it sound like this is a necessity.\n4. On page 10, there are two mentions of Algorithm 3. It looks like accidental text duplication. Please reword these.\n5. Figure 6: please share the Y-axis between the two plots. Having different scaling makes it harder to compare them qualitatively.\n6. How is the SRCC computed between ID and OOD data? It would seem that all OOD is equal and should have the same \"rank\". What is the OOD generalization error between CIFAR-10 and SVHN? (OOD is always wrong?) Please clarify this in the text if possible.\n7. Appendix sections B and C and potentially others don't use `\\citep` but `\\cite'. Please fix this too.\n\nI want to remind the authors that novelty and impact do not concern us reviewers for TMLR. The requested & suggested changes are about making the claims as clear and evidenced as possible. If I misunderstood or made other mistakes, my apologies, and please correct me and take this as hopefully useful signal for making edits that will help the next reader with similar experience understand the paper better. ", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CA6ItEZ4el", "reviewer": "Reviewer_TayB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The major weakness of the paper lies in the formulation of the proposed estimator as an epistemic uncertainty estimator. The method proposed by the authors serves as a convincing model for the excess risk consisting of an approximation uncertainty and a bias. Here, the former exists due to a lack of data/computational resources, while the latter exists because the model is restricted to a hypothesis space $H$ instead of an $F$ that includes the true $f^*$. \nHowever, while a model of the uncertainty introduced due to this restriction is important, it is wrong in my opinion to formulate this as part of epistemic uncertainty, as if this epistemic uncertainty were a universal property. Rather, epistemic and aleatoric uncertainty are always conditioned on the specific hypothesis space the model lives in. To quote from the papers the authors cite in their introductory paragraph.  \n\n> \"...after a long debate between us, we came to the conclusion that these concepts only make unambiguous sense if they are defined within the confines of a model of analysis. In one model an addressed uncertainty may be aleatory, in another model it may be epistemic.\" -- Kiureghian & Ditlevsen (2009)\n\n> \"What this example shows is that aleatoric and epistemic uncertainty should not be seen as absolute notions. Instead, they are context-dependent in the sense of depending on the setting $(X,Y,H,P)$. Changing the context will also change the sources of uncertainty: aleatoric may turn into epistemic uncertainty and vice versa. Consequently, by allowing the learner to change the setting, the distinction between these two types of uncertainty will be somewhat blurred (and their quantification will become even more difficult).\" -- Hüllermeier & Waegeman (2019)\n\nIn that sense the proposed model accounts for the error, which should be properly modelled, that is due to the model being restricted to a space H, but for the model itself, this bias is not epistemic uncertainty. Rephrasing it therefore as epistemic is not correct, everything belonging to the bias is irreducibly conditioned on this space. As the quotes so nicely argue, one person's epistemic uncertainty is another one's epistemic one. In that sense, one could argue that the bias-introduced uncertainty is epistemic on the meta-level of the user applying the proposed algorithms. He/she could increase H to a space that reduces this bias up to the next level of underlying inherent aleatoric uncertainty, thereby reducing his/her epistemic uncertainty. But this applies only to this meta-level, not the current formulation of the paper.\n\n### Minor\n- Sec 2.3 \"Clearly, the fact that multiple,...\" -> why this clearly? If the posterior is not unimodal, but has some permutation symmetries, we do not lack information\n- In (11) and later, I would suggest to change the blue color to something that overlaps less with the quotation blue\n- Please follow the TMLR style guide for tables. Captions belong above the table (and below the figure)\n- Table 1: DEUP (D + V) should actually include a +B following the appendix\n- Typo: Table 1 column U. Bound lacks boldness\n- Typo: Alg 1 in D_e specification misses a closing bracket\n- Typo in Section 1: In the pitfalls paragraph SGD is used but only later introuced", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors of this paper argue that sticking to posterior variance as a measure of epistemic uncertainty is lacking in that it does not account for model misspecification. To follow their nomenclature, it only considers the approximation uncertainty, not the model uncertainty/bias (see Fig 2). They argue that this bias should also be modelled and create a model for it and, formulating the whole task as one of estimating the excess risk, propose to learn an error estimator which, together with a preexisting aleatoric uncertainty estimator, gives them an estimate for the excess risk. The usefulness of this excess risk estimator is then convincingly demonstrated in a series of four different experiments.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Apart from the typos mentioned in the minor section above, the main request is the change of the discussion/formulation of the model discussed at length in the weakness section. This requires a reformulation of the relevant parts to follow the common nomenclature as, e.g., summarized in the two quotes above. The simplest and, in my opinion, the best way would be to just refocus the approach on the excess risk formulation which is what the authors are relying upon anyway instead of redefining it and mixing it with terms that have other meanings. \n\nWhile this might seem a minor point compared to the rest of the paper (for which I would recommend acceptance), it is an important one as the terms aleatoric and epistemic uncertainty keep getting reused and redefined seemingly at random by every other paper.\n", "broader_impact_concerns": "There are none.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I want to thank the authors and reviewers for engaging in substantive discussion, despite the review for this paper taking place over the holiday period.  It seems as though the paper was improved by the process.
Decision: Accept (poster)