DEUP: Direct Epistemic Uncertainty Prediction
Authors: Moksh Jain
Salem Lahlou
Hadi Nekoei
Victor I Butoi
Paul Bertin
Jarrid Rector-Brooks
Maksym Korablyov
Yoshua Bengio
Venue: ICLR 2022 Submitted (2022)
Keywords: deep learning
uncertainty estimation

Abstract: Epistemic uncertainty is the part of out-of-sample prediction error due to the lack of knowledge of the learner. Whereas previous work was focusing on model variance, we propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty, i.e., intrinsic unpredictability. This estimator of epistemic uncertainty includes the effect of model bias (or misspecification) and is useful in interactive learning environments arising in active learning or reinforcement learning. In addition to discussing these properties of Direct Epistemic Uncertainty Prediction (DEUP), we illustrate its advantage against existing methods for uncertainty estimation on downstream tasks including sequential model optimization and reinforcement learning. We also evaluate the quality of uncertainty estimates from DEUP for probabilistic classification of images and for estimating uncertainty about synergistic drug combinations. 

Under review as a conference paper at ICLR 2022
DEUP: DIRECT EPISTEMIC UNCERTAINTY PREDIC- TION
Anonymous authors Paper under double-blind review
ABSTRACT
Epistemic uncertainty is the part of out-of-sample prediction error due the lack of knowledge of the learner. Whereas previous work was focusing on model variance, we propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization eгror and subtracting an estimate of aleatoric uncertainty, i.e., intrinsic unpredictability. This estimator of epistemic uncertainty includes the effect of model bias (or misspecification) and is useful in interactive learning environments arising in active learning or reinforcement learning. addition to discussing these properties of Direct Epistemic Uncertainty Predic- tion (DEUP), we illustrate its advantage against existing methods for uncertainty estimation on downstream tasks including sequential model optimization and re- inforcement learning. We also evaluate the quality of uncertainty estimates from DEUP for probabilistic classification of images and for estimating uncertainty about synergistic drug combinations.
INTRODUCTION
A remaining great challenge for machine learning research is purposeful knowledge-seekee by learning agents, which can benefit from estimation of epistemic uncertainty, i.e., a measure of lack of knowledge, which could potentially be eliminated with enough data if the learner converges to a Bayes-optimal predictor. Epistemic uncertainty estimation is already a key ingredient in active learning and Bayesian optimization (Aggarwal al., 2014; Frazier, 2018) as well as exploration in Reinforcement Learning (Kocsis & Szepesvári, 2006; Tang et al., 2017). Epistemic uncertainty (EU) thus tells us how much could be gained from learning around particular area of state-space or input data space. But how should we even quantify it? Much previous work has focused on model variance (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017) as a proxy thereof, i.c., how different are the functions compatible with the learner's 's preferences and the data. However, this approach does not take into account model bias (or misspecification) due to the learner's 's finite capacity preferences, self-imposed to avoid overfitting reduce computational costs. Recent work (Masegosa, 2020) has also shown that Bayesian methods are suboptimal for learning predictive models when the model is misspecified. Fig. illustrates with Gaussian Processes (GP) that a model's posterior variance is not necessarily a good predictor of the reducible generalization error.
Motivated by interactive learn- ing settings arising for example in active learning, black-box optimiza- tion and reinforcement learning, we study as an alternative or complement the quantification of epistemic uncer- tainty in terms of the loss function. The total expected out-of-sample loss at point can be decomposed into an epistemic (reducible) part and aleatoric (irreducible) part. Aleatoric uncertainty is output noise, the part of the error that is intrinsic to the distribution of interest (and independent of the choice of the
Sebrent arottctss
vattrcaa
01
-0.3
2.0
-3.5
0.5
Figure 1: Top. A GP trained on the blue points. GP uncertainty (model standard Left. Using variance proxy for important regions of the impat space from which useful information can he acquired. Right. With extra out-of-sample points, DEUP learns estimator captures that information. Using this estimator make decisions (s.g. active learning) would focus on where more data could most reduce the loss. The experiments show realistic of DEUP in which less (ur no) additional data is required to obtain reasonable uncertainty estimators.
deviation) in

---

Under review as a conference paper at ICLR 2022
learner), and it is also the error which may be achieved by Bayes-optimal predictor. Consider the case of a learning system with universal approximation properties, like modern deep learning, whose effective capacity can be increased with more data, either via model selection, early stopping or other means. In such cases, the part of the error which is not aleatoric is reducible with more data. For this reason, we propose define epistemic uncertainty the difference between the expected generalization error and alcatoric uncertainty. Estimating it is useful to an active or interactive learner because it could be reduced with enough data, especially in regions of the input space we care about, i.e., where EU is large. In order to estimate EU, we propose DEUP, for Direct Epistemic Uncertainty Prediction, where we train side learner (the uncertainty predictor) with an appropriate objective and training data in order to predict this generalization error. An estimate of EU can thus be obtained by subtracting an estimate of aleatoric uncertainty.
In this paper we thus focus on two predictors: the main predictor (for the learning task of original interest) and an error predictor which can be used to predict epistemic uncertainty of the main predictor at given points, given some knowledge or estimate of the alcatoric uncertainty. The proposed methodology can be seen as an end-to-end approach to epistemic uncertainty estimation, ie., to help estimate where a learner would acquire the most information in terms of out-of-sample loss reduction, without relying on the brittle relationship between model variance and generalization error, which can get murky in the presence of misspecification (bias), as we argue below. In order to select points with high EU, one would still need some kind of search (in the space of x), e.g., to propose new candidates for active learning (Aggarwal et al., 2014; Nguyen et al., 2019; Bengio et al.. 2021) or sequential model optimization (Kushner, 1964; Jones et al., 1998; Snoek et al., 2012 or exploration in RL (Kocsis & Szepesvári, 2006; Osband al., 2016; Janz et al., 2019).
An interesting potential advantage of the proposed ap- proach, compared with model variance estimators, is that DEUP error predictors can be explicitly trained to care about, and calibrate for estimating the generalization er- ror for examples which may come from a distribution different from the distribution of most of the training ex- amples, i.e., an out-of-distribution (OOD) setting. Such non-stationarities arise naturally in contexts like active learning and reinforcement learning (RL) because the learner explores new areas of the input space. In these non-stationary settings, we typically want to retrain the main predictor as we acquire new training data, not just because more training data is generally better but also to better track the changing distribution and generalize to yet unseen but upcoming OOD inputs. This setting makes it challenging to train the main predictor but it also entails an even greater non-stationarity for the training data seen by the error predictor: large error initially made at point æ before x is incorporated in the training set (along with outcome y) will typically be greatly reduced af- ter updating the main predictor with (I, y). We propose several strategies cope with this non-stationarity, the most important being to rely on additional features (such as log-density estimates and model variance estimates at x). These features account for variations in prediction error arising due to updates of the main predictor, making DEUP's training distribution more stationary.
noise 41
cift
AJ f "Siid. dev.
Figure 2: Illustrating noise and bias. Ob- served is independent noise plus true E|Y[=] = fx), itself best approxi- mated by unknown f(x) in parametric set S (orange), e.g., using Bayesian pos- terior distribution (grey) over parame- ters. ff is the closest function in to fr, leading to a bias b(x) = (x) f(x). e (() = (xx) is the reducible error of the main predictor ff (e.g. pos- terior mean), whose square corresponds to the epistemic uncertainty or lack of knowledge, that DEUP aims at estimat- ing. With f the unknown ideal predic- tor and f the actual (e.g. mean) pre- dictor, the square of Af = f(x) f(x) induces variance in the posterior. e( (() = b(x)) Af(x) indicates that using the variance as proxy for lack of knowledge misses out non-negligible quantity: the bias b(x).
The experiments presented here explore first the scenario of purely trying to estimate epistemic uncertainty in terms of the recoverable generalization error, and second how this can be used in contexts of sequential model optimiza- tion (or black-box optimization) and exploration in RL (where the issue of non-stationarity arises). The main contributions of this paper are thus the following:
2

---

Under review as a conference paper at ICLR 2022
1. Novel end-to-end approach for directly estimating the epistemic uncertainty as the lack of knowledge about the true (x) = [x) the Direct Epistemic Uncertainty Prediction (DEUP) approach.
2. Experimental evidence that this direct estimation of epistemic uncertainty can lead to more precise estimates the lack of knowledge than variance-based methods.
mean mitigate the issue of non-stationarity arising in training the uncertainty estimator in interactive settings.
4. A successful validation of the above in the contexts of (a) sequential model optimization and (b) exploration in RL.
2 ALEATORIC AND EPISTEMIC UNCERTAINTIES, MODEL VARIANCE
Consider supervised learning algorithm (or learner) L mapping a dataset D to a predictive function f f= L(D). L tries to minimize the expected value of a supervised learning loss 1(f(x), v) E IR under unknown PaY[z). In this section, we focus on regression tasks with the squared loss 1(7 y) (ÿ y)², with E TR. In Appendix B, we provide the corresponding results for general loss function.
Definition 2.1 The expected loss of a predictor f at x is defined as:
u(f = (f()
Note how the expected loss an unknown quantity, because we generally do have access the true data generating distribution P(Y[z). The errors made by any predictor f at x are due to both inherent randomness of [z) (aleatoric uncertainty) and the lack of knowledge of the predictor that can be tackled by acquiring more information around x (epistemic uncertainty). Next, we will define these concepts more formally. Because of this natural decomposition of the expected loss, will also refer to it as total uncertainty, and we will use the two terms interchangeably.
Bayes-optimal predictors f" satisfy the following equation at every
R f v) & f |( (U-w))dPyvl
Vğ E
They depend on the underlying data distribution only and not on learner L or trained predictor f. Additionally, all Bayes-optimal predictors have the same total uncertainty for all æ. Aleatoric uncertainty is the irreducible expected error, i.e., that made by a Bayes-optimal predictor:
Definition 2.2 We define alcatoric uncertainty at x as the total uncertainty of any Bayes-optimal predictor f at a:
A(x) = =)))
and we note that by definition A(x) S. u(fxx), x), vf Vx.
We now propose a definition of epistemic uncertainty of predictor f as the gap between the error of at and the lowest possible error at i.e., the reducible part of the loss, given more knowledge.
Definition 2.3 We define epistemic uncertainty E(f.z) of predictor f at x as
E(f x) =)] A(z)) =
-U(f"xz)
(3)
Using these definitions, can present the main result about the regression setting.
Proposition 1 In regression task with ground truth P(y|z) No;;; (*)x,, (²()))
E(f,x) (f(z) - (x))²
u(f,z) = -Epp(())(())) - -) -f(x) 2 +z((x)
A(x) = (f - - = -z(x)
3

---

Under review as a conference paper at ICLR 2022
The proof is given in Appendix C. Additionally, we present in Appendix B a similar result for the negative log- ligeeiielhood loss, with discrete or continuous outputs.
We will discuss in the remainder of this section the relation between this new definition of epistemic uncertainty and the more traditional way of modelling it through model variance or entropy.
Consider a parametric model p( (x,0) 8) and a learner maintaining distribution over parameters E O, each corresponding to a predictor f in a parametric set of functions S, possibly starting from prior p(f) that would lead to a posterior distribution D)) Clearly, the fact that multiple 0's and corresponding values of / are compatible with the data and the prior indicates lack of knowledge. Because the lack of knowledge indicates where interactive learner should acquire more information, this justifies the usage of dispersion measures, such as the variance or the entropy of the posterior predictive, as proxies for epistemic uncertainty.
However, the limited capacity of S the prior p(0) may pull the optimal f in S away from the Bayes-optimal predictor fa We can refer to these self- iiffsidp constraints as a form of bias. in the sense that the learner usually biased towards the prior preferences, e.g,, towards smoother predictors. In particular, this issue can arise when training neural networks with limited data, where the network may not use all of its capacity due to implicit (and not fully understood) regularization properties of SGD (Kale et al., 202 1), explicit regularization, early stopping which can induce a preference on the functions it learns. In Fig. 2, we illustrate this gap with the bias function 6(x)= = (z) - fzx). Because of this bias, model variance cannot be an accurate measure of lack of knowledge E {ff. (f(x) (x))² in the general case, as we have shown with the results in Fig. 1. In Deep Ensembles (Lakshminrrayanan et al., 2017) e.g., if all the networks in the ensemble tend to fail in systematic ways, this aspect of prediction failure will not be captured by variance. Whereas Deep Ensembles variance provide us uncertainty regarding which of the networks we could draw is correct, this does not tell us how poor that network is even in a noise-free setting. On the other hand, with flexible models like neural networks, adding examples around x where b(x)) is large may allow to increase capacity around and reduce b(z)²
combination of these,
In section 5, we will provide more experimental evidence supporting our claim that directly estimating the epistemic uncertainty as the reducible part of the loss can lead to better measures of lack of knowledge, compared to variance-based methods, especially in interactive learning settings.
3 DIRECT EPISTEMIC UNCERTAINTY PREDICTION
DEUP (Direct Epistemic Uncertainty Prediction) uses observed out-of-sample errors in order to train an error predictor which can be used to estimate epistemic uncertainty elsewhere, as suggested directly by definitions 2. 1-2.3. These may be in-distribution or out-of-distribution errors, depending on whichever we care about. The pseudo-code of DEUP in interactive settings is provided below. The pseudo-code for the simpler version with a fixed training set is given in Appendix E.
3.1 FIXED TRAINING SET
Consider the simpler scenario with fixed training set D and learning algorithm L yielding predictor f. We assume that N out-of-sample (validation) points have been set aside and can thus be used for training an error predictor.
Training an error predictor with (input, target) pairs (z, (f(x) - v)²) and squared loss yields an estimator of the total uncertainty. To see that this is the right training set for the secondary predictor e, we can assess what happens at the limit of infinite data: the estimator is asymptotically unbiased, i.e.
lim (xx) u(f E((Í(x) Y)²] for all ær if the learning algorithm ensures asymptotic predictor. Examples of such learners include k-nearest neighbors with À increasing at proper rate slower than n., and neural networks whose size and regularization are hyper-parameters optimized on a growing validation set.
convergence to a Bayes-
If A(x) = 0 then e(x) is an estimate of & (ʃi, ( x) as well as (f. If an estimator a(x) of aleatoric uncertainty is available, then e(x) a(x) becomes an estimate of the epistemic uncertainty of at x. Alternatively, one could train e, constrained be positive (e.g. with softplus(w) log(exp(1 u)) output non- lineariir, ) with examples (z. (f(x) - y)² - -(x)) and also obtain an estimator of epistemic uncertainty. When we have access to oracle that samples Y given query from the environment
4

---

Under review as a conference paper at ICLR 2022
a(Y]z) (e.g., in active learning or black-box optimization), then A(x) can be estimated using the empirical variance of different outcomes of the oracle at the same input æ; see Appendix D.
Note that because UI and & are functions of the predictor f as well (and f is itself derived from the training data D), the error predictor can also be trained on features that are informative of the pair (f.z) rather than z only. We will mainly use variance estimates (V), obtained from a separate model (e.g, a GP), and log-density estimates (D), in addition to the input (z). The usage of these features is particularly important in non-stationary settings, as discussed next.
3.2 DEUP IN INTERACTIVE SETTINGS
Interactive settings, in which epistemic uncertainty estimation is used in order to guide information acquisition and iteratively gather new examples, provide more interesting use case for DEUP (as shown in Figures 4 and 5, where good epistemic uncertainty estimation translates in faster learning curves). These settings are however more challenging: the main predictor is going to be retrained multiple times, with the new examples being added to the observed dataset each time. Two kinds of non-stationarities arise here:
First, the new examples are not drawn i.i.d. (since their choice depends on the epistemic uncertainty estimator obtained from earlier examples). This makes the stream of data seen by the main predictor non-stationary. For example in sequential model optimization or RL, the new examples may concen- trate more and more towards regions where the reward might be higher. Whereas is not clear how variance- based predictors can handle this, DEUP in principle cope with it by virtue of having the error predictor trained not on out-of-sample and in-distribution examples but on out-of-sample and out-of-distribution examples of the kind care about (e.g., as obtained from the interactive learning process). The сπтоr predictor then learns to predict the error which would be made on typical new examples coming from what we may call by analogy the "frontier of knowledge", while this frontier continuously expands due to exploration and interactive learning: we expect such out-of-distribure errors to be larger than in-distribution out-of- -aampl errors.
Second, the eтror predictor itself faces even more distributional change, even if the data stream seen by the main predictor was i..dd. Indeed, after example a was selected, the main predictor will be retrained using that additional data point, and x: would now have much lower epistemic uncertainty. This means that the targets of the error predictor also change, as the main predictor is updated with newly acquired points.
Because the dataset D that is used to train the main predictor changes at each step of an active learning process, it is necessary to see ll and E as functions of the dataset as well: u(f = (((CC) x). meaning that we can no longer use (input, output) pairs (z, f(z) y)²) as mentioned in section 3.1 to train an estimator of the total uncertainty, but should principle use (input, output) pairs ((x, D), (f)))) where f L(D). However, being a very high-dimensional object, with size growing with the number of acquired points, we may face severe overfitting issues when training an uncertainty estimator using (x, D) as inputs. Hence, in this paper. we propose to use stationarizing features of the dataset D at that we denote by dD(I), as inputs to the error predictor instead of (x. D).
In addition to the input itself, we explored the use of two scalar features stationarize the training set of the error predictor: a model variance estimate, and log-density estimate. Both have been used as proxies for epistemic uncertainty in the literature. Furthermore, we considered binary feature 8 to incorporate in @D(()) that specifies whether x was already used to train the main predictor (or equivalently, D) or not.
More formally, we use @b(x)) (I, log ((x|D), ((C,)) where ((xr(D) is density estimate
from data D at x s i E D otherwise 0, 2 a learner that produces distribution over predictors, which can chosen to be the same as L, and ((,,D, D, is an estimate of the model variance of Ž trained on 'D at xr ÿ is obtained by training a third model (besides the main predictor and the error predictor) as a density estimator (such as a Kernel Density Estimator or flow-based deep network (Rezende & Mohamed, 2015), in our case). Like the other predictors, the density estimator also needs to be fine-tuned when new data is added to the training set.
In our experiments, we found that using inputs @D(x), even subset of the 4 possible features, is sufficient train uncertainty estimator with targets (f(x) ))
5

---

Under review as a conference paper at ICLR 2022
The pseudo-code is provided in Algorithm 1. @D (Xacg) is different each time because Zacy first is and then is not yet in D.
Note that De is incremented twice with Facy but
3.2.1 KICKSTARTING THE ERROR PREDICTOR
In practical applications, whether in fixed dataset setting or in interactive settings, it is preferable to use all the available data to train the main predictor, rather than keeping subset to train the error predictor. This would entail inaccurate uncertainty estimates once real out-of-sample unlabelled data comes in. To tackle this, we propose a scheme inspired by standard K- fold cross-validation in order to kickstart the error predictor with the initially available data. Multiple versions of the main predictor, trained on different subsets of the training data D, are used to define targets for the error predictor, whereas the subsets themselves provide the input features. The scheme summarized in the Optional Cross- calss-Vaiiationn step of Algorithm 1. Note that in the case of multi-class classification tasks (with classes), we obtain the folds by splitting the data based classes, with each fold containing n K classes. So when we train on K 1 folds, the [n/K] KJ classes from the remaining fold outoff-saple.
Algorithm 1 Training procedure for DEUP in an Active Learning setting
Data: Danit initial training dataset with pairs (z, 3) E X
a: X + R. estimator of aleatoric uncertainty
X + I, main predictor (of y given x))
X + R. total uncertainty estimator
Φ H R chosen stationarizing features
acq: acquisition machinery that proposes new input points eq from X, using the current f and Training:
Initialize empty dataset train the uncertainty estimator, D. Initialize D = Dinis, the dataset of training points seen so far kickstart_erron prrddiiicrrrrrredctttrr then
Cross-validation step: Pre-fill D. using training emros on initial training data Dinit: while stopping criterion reached do
Split into K random subsets Di,..., DDx x of equal size. Define D -U-- DE Fit f on D, and fit the features Φ5 on
D. + D. U UU
(0)(z), (v (Ce.neel(((b))) {x)))²)
Fit e on De
end end
Eacy t Ø. Yacy Ø
while stopping criterion not reached do
Optional (or every few iterations only): Fit a on D (e.g. see Appendix D)
Fit f on D
Fit features in ΦD on D (e.g. density estimation)
D. + fDe U (D (Facg); (yacy - f(zoog)2)) if Facg + Ø
Fit on D.
Eacy &- (C' f.e e a) (can be either a single point, or a batch of points) Sample outcomes from the ground truth distribution: Yoog ~ P(. (Focg)
D. DeU U. ( (Faco))))) (yloy - f (xxco)2))
D -- DU ((Icqy Yacy))
end
RELATED WORK
Kiureghian & Ditlevsen (2009) characterized the sources of uncertainty as aleatoric (inherent noise) and epistemic (incomplete knowledge). Gaussian Processes GPs (Williams & Rasmussen, 1995), provide a way to capture epistemic uncertainty through the disagreement between the different predictors that fit the data.
In a deep learning context, Blundell et al. (2015); Kendall & Gal (2017); Depeweg et al. (2018) use the posterior distribution of network weights (MacKay, 1992) in Bayesian Neural Networks (BNNs) to capture epistemic uncertainty. Other techniques that rely on measuring the discrepancy between different predictors proxy for epistemic uncertainty include MC-Dropout (Gal & Ghahramani,

---

Under review as a conference paper at ICLR 2022
2016), that interprets Dropout (Hinton et al., 2012) as variational inference technique in BNNs. These approaches, because they rely on sampling multiple sets of weights or dropout masks at inference time, share some similarities with ensemble- 1 1996) and boosting (Efron & Tibshirani, 1994), in which multiple predictors are trained, and used jointly to make a prediction, although the latter measure variability due the training set instead of the spread of functions compatible with the given training set, as in Bayesian approaches. For example, Shaker & Hüllermeier (2020) use Random Forests (Breiman, 2001) to estimate epistemic uncertainty. Deep Ensembles (Lakshminarayaman et al., 2017) are closer to the Bayesian approach, using an ensemble of neural networks that differ because of randomness in initialization and training (as you would have with MCMC, albeit in more heuristic way). In addition to this, several other variants of this central idea of measuring discrepancy between different predictors have been proposed recently (Malinin & Gales, 2018; 8; Tagasovska & Lopez-Paz, 2019; Amini et al., 2020; Liu et al., 2020; van Amersfoort et al., 2020; Wen et al., 2020; Antoran et al., 2020; Kirichenko et al., 2020; Malinin et al., 2020a; van Amersfoort et al., 2021). We discuss these methods in more detail in Appendix A.
methods, that include bagging (Breiman,
More closely related to DEUP, Yoo & Kweon (2019) proposed a loss prediction module for learning to predict the value of loss function. Hu et al. (2020) also propose using separate network that learns to predict the variance of an ensemble. These methods, however, are trained only to capture the in-sample error, and do not capture the out-of-sample error which is more relevant for scenarios like active learning where we want to pick & where the reducible generalization error is large. EpiQut (Umlauft et al., 2020; Hafner al., 2019) propose learning binary output that simply distinguishes between low or high epistemic uncertainty.
5
EXPERIMENTS
5.1
UNCERTAINTY ESTIMATION
5.1.1
IMPORTANCE OF EPISTEMIC UNCERTAINTY CALIBRATION
In tasks that use uncertainty estimates to make decisions, and when the resources are limited, it is important to have well calibrated uncertainty estimates, avoid exploring uninteresting areas of the search space, as we will observe in the following subsections. We aim at comparing DEUP, which presumably takes into account both bias and variance, with established methods that are based on variance. Hence, we design a function with varying degrees of smoothness (as measured by the absolute values of its derivatives), and use fewer training points in the less smooth regions, where epistemic uncertainty will thus be higher. We first consider the scenario described in Section 3.1, where the task is to regress a known oneddimensional function.
In Figure 1, we a GP predictor, and compare the GP's variance to the epistemic uncertainty as per per 3 (the square loss between the GP's predicted mean and the ground truth, as per Proposition 1). The figure shows that, naturally, in regions of the input space where little or no data is available, and especially in the less smooth regions, the predicted variance underestimates the epistemic uncertainty (squared error) by a higher margin compared to the other regions. We see that training another GP to directly estimate this squared loss, using a small held out out-of-sample data set for which the ground truth value is known, allows to close the gap between the predicted uncertainty and the true epistemic uncertainty. Note that this second predictor can be trained using any chosen learning algorithm L.
Naturally, if we had access a held out set, it would be wiser to use it as part of the training set of the main predictor, rather than the uncertainty predictor. Alternatively, we could use cross-valie voss-vaaiiotion splits (Section 3.2.1). at the expense of computational efficiency. to avoid discarding precious data. In active learning, it is the acquired points, before they are used to retrain the main predictor, that also act as the out-of-sample examples to train DEUP. Using the stationarizing features introduced in Section 3.2, these acquired points should be informative enough for DEUP to generalize its uncertainty estimates to unknown regions of the search space. In RI., because the targets (e.g. of Q-Learning) are themselves estimates and moving, data seen at any particular point is normally out-of-sample and can inform the uncertainty estimator, when the inputs are used with the stationarizing features.
5.1.2 EPISTEMIC UNCERTAINTY PREDICTIONS FOR REJECTING DIFFCULT EXAMPLES
Epistemic uncertainty estimates can be used to reject difficult examples where the predictor might fail, such as OOD inputs'. We thus consider standard OOD Detection task (van Amersfoort et al., 2020; 2021), where we train a ResNet-18 (He et al., 2016) for CIFAR-10 classification (Krizhevsky, 2009)
c.g. rare but challenging inputs be directed to human, avoiding costly mistake
7

---

Under review as a conference paper at ICLR 2022
Table 1: Spearman Rank Correlation Coefficient (SRCC) between predicted uncertainty and OOD generalization error (SVHN); Area under ROC Curve (AUROC) for OOD Detection (SVHN) with CIFAR-IO ResNet-18 models (3 seeds). DEUP significantly outperforms baselines.
Model
SRCC
AUROC
MC-Dropout Gal & Ghahramani (2016)
0.287
0.002
0.894 + 0.008
Deep Eusemble Lakshminaraaanan ct al. (2017)
0.381
0.004
0.933 + 0.008
DUQ van Amersfoort et al. (2020)
0.376
0.003
0.927 + 0.013
DUE van Amersfoort et al. (2021)
0.378
0.004
0.929 1: 0.005
DRUP (D+V)
0.426 + 0.009
0.933 0.010
and reject OOD examples using estimated uncertainty in the prediction. To facilitate rejection of classes other than those in the training set, we use a Bernoulli Cross-Entropy Loss for each class (van Amersfoort et al., 2020): ()) = Σ; yt log fi(x)) + (1 - y:) log(1 - f(())), where y is a one-hot vector (y/: = 1 if i is the correct class, and 0 otherwise), and fi(x) predicted probability for class i, so the target for cut-of-distribution data (from other classes) is y = (o......0) 0}. To ascertain how well an epistemic error estimate sorts non-training examples by the above NLL loss, we consider the rank correlation between the predicted uncertainty and the observed OOD generalization error on SVHN examples (Netzer et al., 2011). This metric focuses on the quality of the uncertainty estimates rather than just their ability to simply classify in- vs out-of-distribution. We also report the AUC for the OOD detection task; more details in Appendix F. Table 1 shows that with the variance from DUE (van Amersfoort et al., 2021) and the density from MAF (Papamakarios et al., 2017) as DEUP inputs, we obtain uncertainty estimates that have high rank correlation with the underlying generalization errors and competitive AUROC, compared with the baselines. In addition, since the error predictor trained separately from the main predictor there is no explicit trade-off between the accuracy of the main predictor and the quality of uncertainty estimates. We achieve competitive accuracy of 93.89% for the main predictor. We ignore the effect of alcatoric uncertainty (due to inconsistent human labelling), which would require human study to ascertain. We present additional results in distribution shift setting in Appendix F.
5.1.3 EPISTEMIC UNCERTAINTY ESTIMATION FOR DRUG COMBINATIONS
We validate DEUP in real-world regression task predicting the synergy of drug combinations. While much effort in drug discovery is spent on finding novel small molecules, a potentially cheaper method is identifying combinations of pre-existing drugs which are synergistic (i.e., work well together). However, every possible combination cannot be tested due to the high monetary cost and time required to experiments. Therefore, developing good estimates of epistemic uncertainty can help practitioners select experiments that are both informative and promising. As shown in Table 3(c), the out-of-sample error predicted by DEUP correlates better with residuals of the model on the test in comparison to several other uncertainty estimation methods. Moreover, DEUP better captured the order of magnituee of the residuals as shown in Figure 3. Details on experiments and metrics are in Appendix J.
Model
Corn. w.
Bound
Ratio
Log Likelihood
MC Dropout
0.14 2: 0.07
0.56 2 0.05
0.25 0.12
20.1 6.8
Deep Ensemble
0.30 + 0.00
0.59 + 0.04
0.50 0.13
14.3 + 4.7
DUE
0.15
0.50 4:
-13.0
DEUP
0.47 0.08
0.63
+ 0.05
0.75 4 0.07
-3.5 0.25
(c) Quality of uncertainty estimates from different methods.
(a) Ensemble
(b) DEUP
Figure 3: Drug Combinations. Predicted mean and uncertainty on test set. 50 test examples are ordered by increasing value of true synergy score (orange). Model predictions and uncertainties in blue. Ensemble (a) (and MC-dropout, not shown) consistently underestimate uncertainty while DEUP (b) captures the right order of magnitude. (c) Corr. w. res. shows correlation between model residuals and predicted uncertainties à. A best-case Upper Bound on Corr. W. res. is obtained from the correlation between â and true samples from (0, Ø). Ratio is the ratio between col. and 2 (larger is better). Log-likelihood: average over 3 seeds of per sample predictive log-likelihood.
5.2 SEQUENTIAL MODEL. OPTIMIZATION
Sequential model optimization or black-box optimization are forms of active learning. At each stage, the learner chooses query examples to label, looking for examples with a high value of the unknown oracle function. Such examples are selected so they have a high predicted value (to maximize the unknown oracle function) and large predicted uncertainty (offering the opportunity of discovering yet higher values). Acquisition functions, such as Upper Confidence Bound (UCB, Srinivas et al. (2010)) and Expected Improvement (EI, Močkus (1975) trade-off exploration and exploitation, and
8

---

Under review as a conference paper at ICLR 2022
one can select the next candidate by looking for z's maximizing the acquisition function. We combine UCB and EI with DEUP (DEUP-UCB and DEUP-EI) to perform active learning, treating the main predictor and DEUP epistemic uncertainty prediction at z respectively as mean and variance of a Gaussian distribution for the learner's ' guess of the value of the oracle at x. We showcase how using DEUP to calibrate GP variances (used input for DEUP) allows for better performances in higher-dimennional optimization tasks, Specifically, we compare DEUP-EI to TuRBO-EI (Eriksson al., 2019), state-of-the-art method for sequential optimization, that fits collection of local GP models instead of a global one in order to perform efficient high-dimensionas optimization, on the Ackley function (Ackley, 2012) as oracle, common benchmark for optimization algorithms. The oracle function can be defined for arbitrary dimensions, and has many local minima.
In Figure 4, we compare different methods on the Ackley-10 10 function, in addition to the optimum reached in budget-constrained optimization problems for different oracle input dimensions, and we find that adapting DEUP to TuRBO consistently outperforms regular TuRBO, especially in higher dimensions. See also Appendix H for ID and 2D black-box optimization tasks where DEUP-EI outperforms GP-EI (Bull, 2011), as well as neural networks with MC-Dropout or Ensembles. We find GP- EI getting stuck in local optima whereas DEUP-EI was able reach the global maximum consistently. Experimental details are provided in Appendix H.3.
Mosmmum atter 240
GP-E1
GP-B)
1U18O.0EUP-E1
EI TURSO-DEUP-
Truc maximum
-16
50
100
150
200
250
103
Number of addE onal fundtion cate
Dimerston of Ariday function
Figure 4: Left. value reached by the different optimization methods, for the 10 dimensional Ackley function. In each run, all the methods start with the same initial 20 points. The shaded areas represent the standard error across 3 runs Right. maximum value reached in the budget-constraing setting, on the Ackley functions of different dimensions. The error bars represent the standard error across 3 different runs, with different initial sets of 20 pairs. The budget is 120 function calls in total. Higher is better and TURBO-DEUU-EI is less hurt by dimensionality. 5.3 REINFORCEMENT LEARNING
3293
Similar to sequential model optimization, a key challenge in reinforcement learning (RL) is ef- ficient exploration of the input state space. To investigate the effectiveness of DEUP's uncer- tainty estimates in the context of RL, we in- corporate epistemic uncertainties predicted by DEUP to DQN (Mnih et al., 2013), which we refer to as DEUP-DQN. Specifically, we train the uncertainty predictor with the objective to predict the TD-error, using log-density as a stationarizing input. The predicted uncertainties are then used as exploration bonus in the Qaluues. Details of the experimental setup are in Appendix I. We evaluate DEUP-DQN on CartPole, a classic RL task from bsuite (Osband et al., 2020), against DQN + c-greedy, DQN + MC-Dropout (Gal & Ghahramani, 2016) and Bootstrapped DQN (Osband et al., 2016). Figure 5 shows that DEUP achieves lower regret faster, compared to all the baselines, which demonstrates the advantage of DEUP's uncertainty estimates for efficient exploration.
603
1:
Figure 5: Average regret on CartPole task. Error bars for standard error across 5 runs.
CONCLUSION AND FUTURE WORK
Whereas standard measures of epistemic uncertainty focus on variance, we argue that bias (misspecifi- cation) can also be reduced with predictors like neural nets, e.g. using early stopping. In a regression setup, the expected out-of-sample squared error minus aleatoric uncertainty thus captures all the uncertainty about E|Y [æ]] that more data can allow to reduce. This motivates the DEUP estimator, using a second network trained to predict the errors of the first. In interactive settings, this nonetheless raises non-stationarity challenges for this estimator and we propose input features reduce this issue and show experimentally their advantages. Future work should investigate ways to estimate aleatoric uncertainty when one cannot simply query the oracle several times the same
9

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
The authors do not foresee any negative social impacts of this work, but of course the accumulation of improvements in ML could be misused may give more power nefarious agents.
REPRODUCIBILITY STATEMENT
We provide the code to reproduce the results along with the submission. In addition the code, we also outline all the details of the procedure to ensure reproducibility of the results. The general procedure for the method in the fixed-data and interactive setting provided in Algorithm 2 and Algorithm respectively. We also provide pseudocode for the specific instances of DEUP used for the Brug-Combination and Reinforcement Learning experiments in Algorithm 4 and Algorithm. 3. Detailed discussion about the methodology and hyperparameters are provided in the appendix in Appendix F, Appendix H, Appendix and Appendix We provide proofs for all the propositions in Appendix C, with appropriate assumptions.
REFERENCES
David Ackley, connectionist machine for genetic hillclimbing, volume 28. Springer Science &amp; Business Media, 2012.
Charu C Aggarwal, Xiangnan Kong, Quanquan Gu, Jiawei Han, and S Yu Philip. Active learning: A survey. In Data Classification: Algorithms and Applications, pp. 571 571-605. CRC Press, 2014.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. In Neural Information Processing Systems (NeurIPS), 2020.
Javier Antoran, James Allingham, and José Miguel Hernández-Lobato. Depth uncertainty in neural networks. In Neural Information Processing Systems (NeurIPS), 2020.
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL http://arxiis org/ abs /1 1910.111. 06403.
Pedro Ballester and John BO Mitchell. machine learning approach predicting binding affinity with applications to molecular docking. Bioinformatics, (6(9):1169-11) -175, 2010.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation. NeurlPS 2001, arXiv:2/06.00 04399, 2021.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp. 1613-1622. PMLR, 2015.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Adam D Bull, Convergence rates of efficient global optimization algorithms. Journal of Machine Learning Research, 12(10), 2011.
Youngseog Chung. Willie Neiswanger, Ian Char, and Jeff Schneider. Beyond pinball loss: Quantile methods for calibrated uncertainty quantification. arXiv preprint arXiv:2011.09588 2020.
Tomas Cihlar and Marshall Fordyce. Current status and prospects of hiv treatment. Current opinion
in virology, 18:50-56, 2016.
Stefan Depeweg, Jose- Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Decom- position of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In International Conference Machine Learning, pp. 1184-1193, PMLR, 2018.
10

---

Under review as a conference paper at ICLR 2022
Michael W. Dusenberry, Ghassen Jerfel, Yeming Wen, Y. Ma, Jasper Snoek, K. Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International Conference Machine Learning (ICML), 2020.
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
David Eriksson, Michael Pearce, Jacob R Gardner, Ryan Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. arXiv preprint arXiv:1910.0173 2019.
Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when fix it. ICLR 2021, arXiv:2101.11665, 2021.
Peter I. Frazier. tutorial on bayesian optimization, 2018.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (ICML), pp- 1050-1059. PMLR, 2016.
Danijar Hafner, Dustin Tran, Timothy Lillicrap, Alex Irpan, and James Davidson. Noise contrastive priors for functional uncertainty. In Conference on Uncertainty in Artificial Intelligence (UAI), 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0580 2012.
S. Hu, Nicola Pezzotti, and M. Welling, A new perspective uncertainty quantification of deep ensembles. arXiv, 2020.
Pavel Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. Wilson. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence (UAI), 2018.
David Janz, Jiri Hron, Przemystaw Mazur, Katja Hofmann, José Miguel Hernández-Lobace and Se- bastian Tschiatschek. Successor uncertainties: Exploration and uncertainty in temporal difference learning. Neural Information Processing Systems (NeurIPS), 2019.
Donald Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 1344) 1998.
Satyen Kale, Ayush Sekhari, and Karthik Sridharan. Sgd: The role of implicit regularization, batch-size and multiple-epochs. In NewrlPS, 2021.
Alex Kendall and Yarin Gal. What uncertainties we need bayesian deep learning for computer vision? In Neural Information Processing Systems (NeurIPS), 2017.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference for Learning Representations, 2015.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect out-of diito--dittrbution data. In Neural Information Processing Systems (NeurIPS), 2020.
Armen Der Kiureghian and Ove Ditlevsen. Aleatory epistemic? does it matter? Structural Safery, 31(2):105-112, 12, 2009. ISSN 0167-4730. doi: https://doi.org// 100 10.111001111... o0t 000000000000 Risk Acceptance and Risk Communication.
11

---

Under review as a conference paper at ICLR 2022
Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In In: ECML-06. Number 4212 in LNCS, pp. 282-293. Springer, 2006.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Meelis Kull and Peter Flach. Novel decompositions of proper scoring rules for classification: Score adjustment as precursor to calibration. In Machine Learning and Knowledge Discovery in Databuses, 2015.
H. J. Kushner. A new method of locating the maximum point of an arbitrary multipeak curve in the presence of noise. Journal of Basic Engineering, 86:97-106, 1964.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Neural Information Processing Systems (NeurIPS), 2017.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEFF, 86(11) (0(11):2278-23) 1998.
Jeremiah Zhe Liu, Zian Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax- Bedrax-Weiss, and Balaji Laksh- minarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Neural Information Processing Systems (NeurIPS), 2020.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computa- tion, 4(3)::48-477) 1992.
Wesley Maddox, Timur Garipov, Pavel Izmailov, Dmitry Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. Neural Information Processing Systems (NeurIPS), 2019.
A Malinin and M Gales. Predictive uncertainty estimation via prior networks. In NIPS' Proceed- ings of the 32nd International Conference on Neural Information Processing Systems, 2018.
Andrey Malinin, Sergey Chervontsev, Ivan Provilkov, and Mark Gales. Regression prior networks, 2020a.
Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. In International Conference on Learning Representations, 2020b. URL //RLLtttp///oeeeeerreee fort/form?? id BddddBygPPee
6Vtvv. 6Vtvr.
Alina Malyutina, Muntasir Mamun Majumder, Wenyu Wang, Alberto Pessia, Caroline A Heckman, and Jing Tang. Drug combination sensitivity scoring facilitates the discovery of synergistic and efficacious drug combinations in cancer. PLoS computational biology. 1506722 2019.
Andres Masegosa. Learning under model misspecification: Applications to variational and ensemble methods. In Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, arXiv preprint 13X 2013.
Jonas Močkus. On bayesian methods for seeking the extremum. In Optimization techniques IFIP technical conference, pp. 400-404. Springer, 1975.
Reza Bayat Mokhtari, Tina S Homayouni, Narges Baluch, Evgeniya Morgatskaya, Sushil Kumar, Bikul Das, and Herman Yeger. Combination therapy in combating cancer. Oncotarget. 8(23): 38022, 2017.
Harry L Morgan. The generation of a unique machine description for chemical structures-a technique developed chemical abstracts service. Journal of Chemical Documentation, 5(2):107-113,, 1965.
12

---

Under review as a conference paper at ICLR 2022
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NIPS), 2011.
Vu-Linh Nguyen, Sébastien Destercke, and Eyke Hüllermeier. Epistemic uncertainty sampling. In International Conference Discovery Science, pp. 72-86. Springer, 2019.
World Health Organization and Stop TB Initiative. Treatment of tuberculosis: guidelines. World Health Organization, 2010.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. arXiv preprint arXiv:1602.0462, 2016.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2020. URL Ad=rygf-kSYwH.
net/ forum?
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshninarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. In Neural Information Processing Systems (NeurIPS), 2017.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference on Machine Learning (ICML), pp. 1530-1538. PMLR, 2015.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Neural Information Processing Systems (NeurIPS), pp. 3183-3193, 2018.
Mohammad Hossein Shaker and Eyke Hüllermeier. Aleatoric and epistemic uncertainty with random forests. International Symposium on Intelligent Data Analysis, pp. 444 456. Springer, 2020.
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Neural Information Processing Systems (NeurIPS), 2012.
Niranjan Srinivas, Andreas Krause, Sham M Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In International Conference on Machine Learning (ICML), 2010.
Aravind Subramanian, Rajiv Narayan, Steven M Corsello. David D Peck, Ted E Natoli, Xiaodong Lu, Joshua Gould, John F Davis, Andrew Tubelli, Jacob Asiedu, et A next generation connectivity map: L1000 platform and the first 1,000,000 profiles. Cell, 171(6):1437-1455
2017.
Natasa Tagasovska and David Lopez-Paz. Single-model uncertainties for deep learning. In Neural Information Processing Systems (NeurIPS), 2019.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbecl. exploration: A study of count-based exploration for deep reinforcement learning. In Neural Information Processing Systems (NeurIPS), 2017.
Jonas Umlauft, Armin Lederer, Beckers, and Hirche. Real-time uncertainty decomposition for online learning control. ArXiv, abs/2010.02613, 2020.
Meet P. Vadera, Adam D. Cobb, Borhan Jalaeian, and Benjamin M Marlin. Ursabench: Comprehen- sive benchmarking of approximate bayesian inference methods for deep neural networks. ArXiv, abs/2007. 04466, 2020a.
Meet P. Vadera, Borhan Jalacian, and Benjamin M Marlin. Generalized bayesian posterior expectation distillation for deep neural networks. In UAI, 2020b.
13

---

Under review as a conference paper at ICLR 2022
Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. Improving determinis- tic uncertainty estimation in deep learning for classification and regression. CoRR, abs/2102.111499 2021. URL https: httpaaxxiiiv../ org/ wtt.a//.........................
Joost R. van Amersfoort, L. Smith, Y. Teh, and Yarin Gal. Simple and scalable epistemic uncertainty estimation using single deep deterministic neural network. In International Conference on Machine Learning (ICML), 2020.
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference International Conference Machine Learning, 2011.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: An alternative approach to efficient ensemble and lifelong learning. In International Conference Learning Representations, 2020.
C. K. Williams and C. Rasmussen. Gaussian processes for regression. In Neural Information Processing Systems (NeurIPS), 1995.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning. In Arthur Gretton and Christian Robert (eds.), Proceedings of the 19th International Conference Artificial Intelligence and Statistics (AISTATS), volume 51 of Proceedings of Machine Learning Research, pp. 370-378, Cadiz, Spain, 09-11 May 2016. PMLR. URL http: htatt:::/pproeeeee /ppoceedings. mlr. wal.spress//e 5 wwwsonlf html.
Donggeun Yoo and I. Kweon. Learning loss for active learning. 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 93-102, 2019.
Bulat Zagidullin, Jehad Aldahdooh, Shuyu Zheng, Wenyu Wang. Yinyin Wang, Joseph Saad, Alina Malyutina, Mohieddin Jafari, Ziaurrehman Tanoli, Alberto Pessia, al. Drugcomb: an integrative cancer drug combination data portal. Nucleic acids research, W75 2019.
Ruqi Zhang, Chumyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. ICLR, 2020.
Shuxing Zhang, Alexander Golbraikh, Scott Oloff, Harold Kohn, and Alexander Tropsha. A novel automated lazy learning qsar (all-qsar) approach: method development, applications, and virtual screening of chemical databases using validated all all-qsar models. Journal of chemical information and modeling, 46(5):1984 1995, 2006.
14

---

Under review as a conference paper at ICLR 2022
APPENDICES
A RELATED WORK
Recently, several novel deep learning-bard based techniques to estimate uncertainty with a single model have been proposed. For example, Deep Evidential Regression Amini et al. (2020) is method for estimating epistemic uncertainty that is based on a parametric estimate of model variance. This is in line with previous work using evidential uncertainty estimation (Malinin & Gales, 2018; Sensoy et al., 2018). Orthonormal Certificates, a set of learned features with a suitable loss function, are used in Tagasovska & Lopez-Paz (2019). These certificates capture the distance to the training set to learn an estimate of the epistemic uncertainty. This is further studied in Liu et al. (2020) who formalize distance awareness, which captures the model's ability to quantify the distance of a test sample from the training data manifold, as necessary condition for uncertainty estimation. This distance awareness can be captured with weight normalization step in training. in addition to using GP as the output layer. DUE (van Amersfoort et al., 2021) is an instance of Deep Kernel Learning (Wilson et al., 2016), which is defined as a GP with deep feature extractor inside the kernel. DUE improves upon SNGP by using an inducing point GP, and bi-lipschitz constraints on the feature extractor, giving better test set accuracies as well as improved training efficiency. DUN (Antoran et al., 2020) uses the disagreement between the outputs from intermediate layers measure of uncertainty. DUQ (van Amersfoort et al., 2020) on the other hand uses two-sided Jacobian regularization on RBF networks (LeCun et al., 1998) for reliable uncertainty estimates.
Wen et al. (2020) present an efficient way of implementing ensembles of neural networks, by using one shared matrix and a rank-1 I matrix per member. The weights for each member are then computed as the Hadamard product of the shared matrix and the rank-1 matrix of the member. There has also been extensive work in scaling up Bayesian Neural Networks for high-dimensional data to capture epistemic uncertainty. SWAG (Maddox et al., 2019) fits a Gaussian distribution capturing the SWA (Izmailov et al.. 2018) mean and covariance matrix representing the first two moments of SGD iterated. This distribution is then used as a posterior over the neural network weights. Dusenberry et al. (2020) parametrize the BNN with a distribution on a rank- 1 subspace for each weight matrix, inspired by BatchEnsembles (Wen et al., 2020). Vadera et al. (2020b); Malinin et al. (2020b) propose approaches to improve the efficiency of ensembles by distilling the distribution of predictions rather than the average, thus preserving the information about the uncertainty captured by the ensemble.
There also a large body of work on methods for approximating samples from the Bayesian posterior on large datasets with efficient MCMC based approaches. (Welling & Teh, 2011: 1; Zhang et al., 2020; Vadera et al., 2020a). The variance of this posterior distribution can then be computed and used as an uncertainty estimate. However, as have discussed in the sections above, this does not account for model misspecification and thus is not an accurate estimate for the lack of knowledge of the predictor.
Kull & Flach (2015) present several decompositions of the total expected loss, including the decom- position into the epistemic and irreducible (aleatoric) loss. They present additive adjustments that reduce the scoring rules like the log-loss and Brier score, but do not tackle the general problem of uncertainty estimation.
There are also interesting connections between the problem of -ustr--distributionn generalization arising in sequential model optimization and Bayesian optimization, discussed here, and the possibility of reweighing examples, see Farquhar et al. (2021).
B EPISTEMIC UNCERTAINTY IN A GENERAL LOSS FUNCTION SETTING
We consider the setting presented in section 2, but with a general loss function 1. We use the same notations as in section 2.
Definition B.1 The total uncertainty of f at x is defined as:
u(f = / (1f(2)
(4)
15

---

Under review as a conference paper at ICLR 2022
Definition B.2 For a learning algorithm L which produces distributtio P=(f(=)(D,) over possible solutions f(x) at æ, the model variance at is defined as
Vr, /
(5)
with f(z) = arg min f(x) I [(f((),, (())dP Note that for a loss function that is differ- ent from the square loss, the semantics of variance (such as its non-negane non-negativity) might lost.
Let us consider the special cases of the negative log-likelihood loss in general (for outputs which may be discrete or continuous) and that the squared error loss (which ends up being a special case of the former for normally distributed outputs). Below we see Q(Y{z) probability mass or density function (over y), which also a function x.
Definition B.3 The negative log- aggliikl (NLL) loss takes as first argument Q(Y(z) probability mass or density function and returns
INLI (Q((())) log Q(Y = y|æ)..
(6)
Proposition 2 For the NLL loss with ground truth P(Y(x) and predictor Q(Y (z), the total uncer- tainty ({Q((((=))) x)) is cross-entropy. i.e.
M(Q(Y))),s -C(((()))))))(())))))
= -- dP log Q(ylz)
The proposition is shown by applying the definitions.
Proposition 3 For the NLL loss with ground truth P(Y(z), the aleatoric uncertainty A(x) in this setting is the entropy H[P(Y{=:) of the ground truth conditional:
A(r) f dP(ylz))
- P(y|z)
The proposition is shown from the cross-entropy CE(P(Y)=)]IQ0) being minimized when Q ==
Proposition 4 For the NLL loss with ground truth P(Y|æ) and predictor (x), the epistemic uncertainty E(Q(Y)z); x) is the Kullback-Liehle divergence between and Q (with P as the reference):
E(Q(Y) .). KL(P(Y 1Q)
= / dP(ylæ) log
P(y|æ)
Q()(a)
The proposition is shown by combining the above two propositions and the definition of epistemic uncertainty.
To move towards the MSE loss, consider the special case of NLL with a conditionally Normal output density for both P and Q.
Proposition 5 For the NLL loss with a conditionally Normal output density for both P and Q, with respective means (x) and f(x) and respective variances (z)(x) and (z). the epistemic uncertainty is
E(Q(Y).),a) (x)
I IMSE(f())
+ KUPPL))))(((())))
(10)
where Q( (æ) is obtained by shifting Q( |z) towards P(. (z) (i.e., Q |x) is Gaussian with mean
(x) and variance (2(()) (x) ), and the Bayes-optime mean predictor is f fx) = Ep[Y [2]. Note that faap- aq then the KI. term is zero.
16

---

Under review as a conference paper at ICLR 2022
The proof is presented in Appendix C. We compare with the MSE loss (which assumes constant variance 5 = ap = σ②)) and obtain the same result up to variance scaling.
C PROOFS
C. PROPOSITION
It is well known result that, because f*xx) is the mean of P(I2z), it is also the minimizer of ý f'Ú -) dP f is thus Bayes Optimal predictor.
By definition of the total uncertainty:
u(f,²) - (1(()) w) [((((x) =)
Hence, by definition of aleatoric uncertainty:
A(r) = = r)
and by definition epistemic uncertainty
E(..) = [[[(x) - y)² - (x) v)
f(x)² - (x)² - 2(((x) - (x))))
(f(x) (x)))
Which concludes the proof.
C.2 PROPOSITION 5
From Equation 9, we get:
E(Q(Y) ),=) ((P(())))QQ())))
xo(x) log or(z)
xz(x) + (f(x) (()))
20z(2)
2
(() log σQ(x)
+z(x)+ + (f)z)) 1
(z)
0p(z) 20z(2)
2
(x)
Which concludes the proof
D ESTIMATING ALEATORIC UNCERTAINTY WITH ACCESS TO AN ORACLE
In scenarios like active learning, one has access an oracle from which we can obtain samples of Y ~ paY at any given point x. In that case, one can train an estimator a(z) of aleatoric uncertainty by obtaining K samples ./.... /KK at the same x, for of representative r's.
More formally. if we have multiple independent outcomes y1...., MK input point x, then training a predictor with the squared loss on (input, target) examples Vor wk)) where Var denotes the empirical variance, yields estimator of the aleatoric uncertainty.
P(Y for each
Naturally, this estimator is asymptotically unbiased, if the learning algorithm ensures asymptotic convergence to a Bayes-Optimal predictor.
- Varly..... ..... .uk) V aar x)) which according to
This is due the fact that Ey.....p Proposition I is equal A(x).
17

---

Under review as a conference paper at ICLR 2022
E PSEUDO CODES
Algorithm illustrates the training procedure when held-out validation set is available. We focus on y E R in this paper because it makes sense for active learning applied black-box optimization (where we want to maximize it) but the algorithms can trivially be applied to the case where R for any Similarly, the algorithms can be generalized to other losses besides the square loss, following the generalized theory presented above in Appendix B.
Algorithm 2 DEUP with a fixed training set: Training procedure to obtain estimates of epistemic uncertainty
Data: the training dataset with pairs (z, y) with x E X, E IR; Dout the out-of-sample dataset with pairs y), train the uncertainty estimator X, the input/search space
a: a: X + R. trained estimator of aleatoric uncertainty
f: + R, main predictor, trained on D
e: e: R. total uncertainty estimator (estimates error of f)
Training:
Initialize empty dataset of errors De Cross-validation step (Optional, if using other features for the error predictor): Similar to the corresponding step of Algorithm for every pair (Ir, y) in D U Dout do
| De - De U (x, (y f(x))²)} end
Fit on De Evaluation: For every input x, return e(x) - a(x) as an estimator of epistemic uncertainty at
F REJECTING DIFFICULT EXAMPLES
We adapt the standard OOD rejection task (van Amersfoort al., 2020; Liu al., 2020) measure the Spearman Rank Correlation of the predicted uncertainty with the true generalization error, in addition to the OOD Detection AUROC. We use MC-Dropout (Gal Ghahramani, 2016), Deep En- semble (Lakshminarayanaa et al., 2017). DUE(van Amersfoort et al., 2021) and DUQ van Amersfoort et al. (2020) as the baselines 2, We use these baselines as representatives for the major approaches for uncertainty estimation in recent literature. For all the methods, including DEUP we consider two architectures for the main predictor, ResNet- 18 and ResNet-50 (He et al., 2016) (Table 2) to study the effect of model capacity. Note that for the ResNet50 DEUP model we continue using the ResNet- 18 based DUE as variance source.
Table 2: Spearman Rank Correlation between predicted uncertainty and the true generalization error on OOD data (SVHN) with ResNet-50 models (3 seeds) trained on CIFAR-10.
Model
ResNet-50
MC-Dropout
0.312 + 0.003
Deep Ensemble
0.401 + 0.004
DUQ
0.399 + 0.003
DEUP (D+V)
0.465 0.002
Training The baselines were trained with the CIFAR-10 training set with 10% set aside as validation set for hyperparameter tuning. The hyperparameters are presented in Table and Table The hyperparameters not specified are set to the default values. For DEUP, we consider the log-density, model-variance estimate and the seen-unseen bit as the features for the error predictor. The density
MCC-Dropout and Deep Ensemble baselines are based on https://github.cc http::///hhbbboooooooog///eeeeee and DUE based
baselines, DUQ based on http://gnhbbooooooo//eeeeemeemee http:://gthhub.coottoo/DDDUD
18

---

Under review as a conference paper at ICLR 2022
estimator use is Yusked-Autoregrees estimator used DUE (van Amersfoort et al., 2021). Note that as indicated earlier z, the input image, is not used as feature for the error predictor. We present those ablations in the next sub-section. For training DEUP, the CIFAR-10 10 training set is divided into 5 folds, with each fold containing 8 unique classes. For each fold, model variance estimator on only the corresponding 8 classes. The remaining classes act as the out-of-distribution examples for training the error predictor. Using these folds we construct a dataset for training the error predictor, a simple feed forward network. The error predictor is trained with the log targets (i.e. log MSE between predicted and observed error). This helps since the scale of the errors varies over multiple orders of magnitude. We then train the main predictor, density estimator and the variance estimator on the entire CIFAR-I0 dataset, for evaluation. The hyperparameters are presented in Table 4. For all models, we train the main predictor for 75 and 125 epochs for ResNet-18 18 and ResNet-50 respectively. We use SGD with Momentum (set to 0.9), with a multi-step learning schedule with decay of 0.2 at epochs [25, 50) and [45, 90] for ResNet-1e 18 and ResNet-50 respectively. One complete training run for DEUP takes about 1.5-2 GPU days on a V100 GPU. In total these set of experiments took about 31 GPU days Nvidia V100 GPU.
Flows (Papamakarios al., 2017) and the variance
train instance of the main predictor, density estimator and
Table 3: Left: Hyperparameters for training Deep Ensemble (Lakshminarayanan al., 2017). Right: Hyperparameters for training MC-Dropout (Gal & Ghahramani, 2016).
Model
Parameters
Model
Parameters
ResNet-18
Nea-et-50
ResNet-18
ResNet-S0
Number of sumples
50
50
Number of members
5
Dropout Rate
0.15
0.1
Learning Rate
0.05
0.01
L2 Regularization Coefficient
6e-5
8c-4
Learning Rate
0.05
0.01
Table Left: Hyperparameters for training DUQ (van Amersfoort al., 2020). Right: Hyperpa- rameters for training DUE (van Amersfoort et al., 2021).
Model
Parameters
Model
Parameters
ResNet-18
ResNet-18
ResNet-50
Inducing Points
50
Gradient Penalty
0.5
0.65
Kernel
RBF
Centroid Size
512
512
Lipschitz Coefficient
Lenghh scale
0.1
0.2
BatchNorm Momentum
0.99
Learning Rate
0.05
0.025
Learning Rate
0.05
Weight Decay
0.0005
Ablations We also perform some ablation experiments study the effect of each feature for the error predictor. The Spearman rank correlation coefficient between the generalization еπгоr and the variance feature, V. from DUE van Amersfoort al. (2021) alone is 37.84 + 0.04, and the log-density. D, from MAF Papamakarios et al. (2017) alone is 30.52 + 0.03. With only the image (x) the SRCC is 36.58 + 0.16
Table 6 presents the results for these experiments. We observe that combining all the features performs the best. Also note that using the log-density and variance as features to the error predictor we observe better performance than using them directly, indicating that the error predictor perhaps captures
Table 5: Hyperparameters for training DEUP.
Model
Parameters
ResNet-18
ResNet-50
Uncertainty Predictor Architecture
[1024] x 5
[1024] x
Uncertainty Predictor Epochs
100
100
Uncertainty Predictor LR
0.01
0.01
Main Predictor Learning Rate
0.05
0.01
19

---

Under review as a conference paper at ICLR 2022
better target for the epistemic uncertainty. The boolean feature (B) indicating examples, discussed in Section 3.2, also leads to noticeable improvments.
Table 6: Spearman Rank Correlation between predicted uncertainty and the true generalization error on OOD data (SVHN) with variants of DEUP with different features as input for the uncertainty predictor. D indicates the log-density from MAF Papamakarios et al. (2017), V indicates variance from DUQ van Amersfoort al. (2020) and indicates bit indicating if the data is seen.
Model
Features
ResNet-18
ResNet-50
D+V+B
0.426 + 0.009
0.465
0.002
D+V
0.419 + 0.003
0.447
0.003
V+B
0.401 = 0.004
0.419
0.004
D+B
0.403 = 0.003
0.421 + 0,002
F.1 PREDICTING UNCERTAINTY UNDER DISTRIBUTION SHIFT
We also consider the task of uncertainty estimation in the setting of shifted distributions (Ovadia et al., 2019; Hendrycks & Dietterich, 2019). We evaluate the uncertainty predictions of models trained with CIFAR-10, CIFAR-IO-C (Hendrycks & Dietterich, 2019) which consists of images from CIFAR- 10 distorted using 16 corruptions like gassian blur, impulse noise, among others. Figure shows that even in the shifted distribution setting, the uncertainty estimates of DEUP correlate much better with the error made by the predictor, compared to the baselines.
G DEUP IN THE PRESENCE OF ALEATORIC UNCERTAINTY
In the presence of aleatoric uncertainty, have seen that DEUP's s predictor e is an estimate of the total uncertainty, rather than the epistemic uncertainty. However, if we have access to to an estimator of aleatoric uncertainty a, then show the difference in behavior of DEUP when there is alcatoric uncertainty, we consider, modified version of the experiment in Section 5. 1. with a non-deterministic oracle (ground truth function). Because of the noisy training dataset, GP conflates epistemic and aleatoric uncertainty, which makes the gap between the predicted epistemic uncertainty (as measured by the GP variance) and the true epistemic uncertainty (as measured by the MSE between the GP mean and the noiseless ground truth function) higher than the deterministic setting of Section 1.
becomes estimator of epistemic uncertainty. To
Similarly, in order to train DEUP's uncertainty estimator, more out-of-sample data is needed compared the noiseless setting. figure 7, we consider the setting described in Section D, and train separate predictor on the targets ....... 3/ to estimate the alcatoric uncertainty (which boils down to the variance of the ground-trute function). Because these targets are themselves noisy, we chose simple linear regressor the estimator of aleatoric uncertainty, avoid overfitting to the noise.
key distinction of this setting, is that DEUP's ' training data (the errors of the main predictor) are themselves noisy, which makes it important to use more out- data to obtain reasonable total uncertainty estimates (from which we subtract the estimates of the aleatoric uncertainty).
H SEQUENTIAL MODEL OPTIMIZATION EXPERIMENTS
We use BoTorch (Balandat et al., 2020) the base framework for experiments.
For all our Sequential Optimization algorithms, we use Algorithm to train DEUP uncertainty estimators. We found that the optional step of pre-filling the uncertainty estimator dataset D, was important given the low number of available training points. We used half the initial training set (randomly chosen) in-sample examples (used to train the main predictor and an extra-fcature generator) and the other half as out-of-sample examples to provide instances of high epistemic
hhups///htorchh.org
20

---

Under review as a conference paper at ICLR 2022
Ensembles Dropout
DUE
DEUP
0.35
Spearman Rank Correlation Coefficient
0.30
0.25
0.20
0.10
gaussian reeiee
brightress
contrast
pizmlate
speckle noise
shot noise
impuise neise
deforus blur
gaussian blur
zoom blur
rog
elastic transform
ipeg compression
sparter
saturate
frost
Comuption Type
Figure 6: Spearman Rank Correlation Coefficient between the predicted uncertainty and true error for models trained with CIFAR-10 10, and evaluated on CIFAR-1O-C. DEUP outperforms the baselines on all types of corruptions.
uncertainty to train uncertainty predictor; we repeated the procedure by alternating the roles of the two halves of the dataset. We repeated the whole procedure twice using a new random split of the dataset, thus ending up with 4 training points in D. for every initial training point in D
The error predictor trained with the log targets (i.e. log MSE between predicted and observed error). This helps since the scale of the errors varies over multiple orders of magnitude.
Computationally, the training time of DEUP-EL depends on various choices (e.g. the features used to train the epistemic uncertainty predictor, the dimension of the input, the learning algorithms, etc..). Additionally, the training time for the uncertainty predictor varies at each step of the optimization. In total, the sequential optimization experiments took about CPU day.
H.1 ONE-DIMENSIONAL FUNCTION TOY EXAMPLE
In Figure 8, we show the results of DEUP-EI, compared to GP-EI, MCDropout-Ei and Ensembles-FI, in the task of optimizing synthetic one-dimensional function. Because MCDropout and Ensembles are trained on in-sample data only, they are unable to generalize their uncertainty estimates, which makes them bad candidates for Sequential Model Optimization, because they easily stuck in local minima, and require many iterations before the acquisition function gives more weight to the predicted uncertainties than the current maximum.
For Random acquisition, we sampled for different seeds 56 points, and used the (average across the seeds of the) maximum of the first 6 values the first value in the plots (Figures 4 and 8). Note that
21

---

Under review as a conference paper at ICLR 2022
Ground truth
2
mean
-2
-0.5
0.0
0.5
1.0
1.5
2.0
2.5
10
Epistemic uncertainty GP variance
Epistemic uncertainty DEUP's prediction
10
A
0
0.5 0.0
0.5
1.0
1.5
2.0
2.5
0.5 0.0
0.5
1.0
1.5
2.0
2.5
Figure 7: Top. A GP is trained to regress function using noisy samples. GP uncertainty (model standard deviation) is shaded blue. Bottom left. Using GP variance as proxy for epistemic uncertainty misses out more regions of the input space, when compared to Figure 1 Bottom right. Using additional sut-ol-sample data in low density regions, a second GP is trained to predict the generalization error of the first GP (total uncertainty). Using second samples from the oracle for each of the training points, linear regressor fits the training pairs (z, 2 (3) y2)²² to estimate the pointwise aleatoric uncertainty (constant in this case). The aleatoric uncertainty is subtracted from DEUP's (second GP) predictions to obtain more accurate of epistemic uncertainty. Note that no constraint is imposed on DEUP's outputs, which explains the predicted negative values for uncertainties. In practice, if these predicted uncertainties were to be used, (soft) clipping should be used.
because the function is specifically designed to have multiple local maxima, GP-EI also required more optimization steps, and actually performed worse than random acquistion
As stationarizing input feature, we used the variance of a GP fit on the available data at every step. feature and density estimates were redundant
We found that the binary wosaapp//ouooo-saaapppap with the variance feature and didn't improve the performance captured by the number of additional function calls. We used a GP for the DEUP uncertainty estimator. Using a neural net provided similar results, but was computationally more expensive in this 1-D case with few datapoints. We used a 3-hidden layer neural network, with 128 neurons per layer and a ReLU activation function, with Adam (Kingma & Ba, 2015) and learning rate of 10-3 (and default values for the other hyperparameters) to train the main predictor for DEUP-EI (in order to fit the available data). The same network architecture and learning rate used for the Dropout and Ensemble baselines. We used 3 networks for the Ensemble baseline, and a dropout probability of 0.3 for the Dropout bascline, with 100 test-time forward passes compute uncertainty estimates.
22

---

Under review as a conference paper at ICLR 2022
1.00
GMB
0.75
Random aequstion
0.50
BRUPEI
0.25
Ersentle B
True makrium
0.00
Nestuu
0.23
-0.50
-0.75
-1.00
-1.0
2.0
bumberafarddio function rals
Figure 8: Left. Synthetic function to optimize. Right. Maximum value reached by the different methods the synthetic function. The shaded areas represent the standard error across different runs, with different initial sets of 6 pairs. For clarity, the shaded areas are omitted for the two worst performing methods. In each run, all the methods start with the same initial set of points. GP-EI tends to get stuck local optima and requires more than 50 steps, on average, to reach the global maximum.
H.2 TWO-DIMENSIONAT. FUNCTION
To showcase DEUP's usefulness for Sequential Model Optimization in with number of dimensions greater than 1, we consider the optimization of the Levi N.13 function, a known benchmark for optimization. The function f takes point (z, y) in 2D space and returns:
f(x, v) = (sin" (3Tx) + (x - 1) + sin?(3 (y - 1) + sin (2my)))
We use the box [-10, 10 as the optimization domain. In this domain, the maximum of the function and it reached (1,1). 1). The function has multiple local maxima, as shown in Figure 9()
Similar to the previous one-onneeeeee mieddiensio function, MCDropout and Ensemble provided bad perfor- mances and are omitted from the plot in 9(b). We used the same setting and hyperparameters for DEUP as for the previous function. DEUP-EI is again the only method that reaches the global maximum consistently under 56 function evaluations.
-30
-20
-30
440
GP-E
Random auquiztion
30
DEJF-B
300
True mox mum
10
Namber Toadtiors Aundio
50
(a) Visualization of (z, v) + - f(x, v)
(b) Comparisons with GP-EI and Random acquisition
Figure 9: Sequential Model Optimization on the Levi N.13 function
*Pot of the function copied from hutps//www.sfu.cc// ssurjano/levy 11..o//lllyy111.mmmmm 13.hmll
23

---

Under review as a conference paper at ICLR 2022
H.3 ADDITIONAL DETAILS FOR THE ACKLEY FUNCTION EXPERIMENT, FOR SYNTHETIC DATA IN HIGHER DIMENSIONS
The Ackley function of dimension d is defined as:
Ackleya B + R
1 < cos cos()))))
+ Aexp
-B
+exp
ex1xx(1
A exp(1)
where B is a hyperrectangle Rdd (O...., 0) is the only global optimizer of Ackleya, at which the function is equal to O. We use BoTorrhhss s default values for A, B, c, which are 20, 0.2, 2x respectively.
In our experiments, we used B = 10 15) for all dimensions d.
For the TurBO baseline, we use BoTorch's 's default implementation, with Expected Improvement as an acquisition function, and a batch size of (i.e. acquiring one point per step).
For fair comparisons, for DEUP, we use a Gaussian Process as the main model, and use its variance as the only input of the epistemic uncertainty predictor. This means that we calibrate the GP variance to match the out-of-sample squared error, using another GP to perform the regression. TurBO-DEUP is a combination of both, in which we perform the variance calibration task for the local GP models of TurBO. The uncertainty predictor, i.e. the GP regressor, is trained with log targets, as in Appendix H.1, but also with log variances as inputs.
Note that only the stationarizing feature is used input for the uncertainty predictor. When used the input & as well, we found that the GP error predictor overfits on the æ part of the input (x, v), and it was detrimental the final performances. For all experiments, used 20 initial points.
I REMENT LEARNING EXPERIMENTS
For RL experiments, used bsuite (Osband et al., 2020), collection of carefully designed RL environments. bsuite also comes with a list metrics which aim to evaluate RL agents from different aspects. We compare the agents based the basic metric and average regret they capture both sample complexity and final performance. The default DQN agent is used as the base of our experiments with 3 layer fully-connected (FC) neural network as its Q-network. For the Bootstrapped DQN baseline, we used the default implementation provided by bsuite, To implement DQN MC-Dropout, following the implementation from Gal & Ghahramani (2016), two dropout layers with dropout probability of 0.1 are used before the second and the third FC layers. In order take an action, the agent performs single stochastic forward pass through the Q-network. which equivalent to taking sample from the posterior over the Q-values, as done in Thompson sampling, an alternative to é greedy exploration.
As a density estimator, we used a Kernel Density Estimator (KDE) with Gaussian kernel and bandwidth to map states to densities. This KDE is after each 10000 steps (actions) with a batch of samples from the replay butfer (which is of size 10000). The uncertainty estimator network (E-network) has the same number of layers as the Q-network, with an additional Softplus layer at the end. All other hyperparameters One complete training run for the DEUP-DQN with 5 seeds experiments takes about 0.04-0.05 GPU days V100 GPU. In total RL experiments took about 0.15 GPU days Nvidia V 1000 GPU.
the same as the default implementation by Osband al. (2020).
24

---

Under review as a conference paper at ICLR 2022
Algorithm 3 DEUP-DQN
Initialize replay buffer 'D with capacity N
Qa(s, a)): state-action value predictor
Eg (logd)): uncertainty estimator network, which takes the log-density of the states as the input d(s): Kernel density estimator (KDE)
K: KDE fitting frequency
W: Number of warm-up episodes
for episode=/ to M do
set 80 as the initial state
for /=/ to muxxsttepx-er--episode do
with probability e: take random action, otherwise:
if episode I W:
Qo a),, else: mmz. (Qo(s;; + x Eollog di (a)]
store (s,, ft, St+1) 1 in 'D
Sample random minibatch B transitions (5), Ug. Tyy 4)41) from
if sj is final state: Y/j Tj. else: M/ -r + ymaaa a) Update Q-network:
0 0 aq. V (s [WV) (ooss
Update E-network:
[(0) Qo(s,a))² (log dl ((s)))
- a-otag- V (e,0)=BB
if mod(total-steps, K) 0: fit the KDE d on the states of 2>
end end
DRUG COMBINATION EXPERIMENTS
To validate DEUP's uncertainty estimates in real-world setting, we measured its performance on regression task predicting the synergy of drug combinations. While much effort in drug discovery is spent on finding novel small molecules, a potentially cheaper method identifying combinations of pre-existing drugs which are synergistic (i.e., work well together). Indeed, drug combinations are the current standard-of-care for a number of diseases including HIV, tuberculosis, and some cancers Cihlar & Fordyce (2016); Organization & Initiative (2010); Mokhtari et al. (2017).
However, due to the combinatorial nature of drug combinations, identifying pairs exhibiting synergism is challenging. Compounding this problem is the high monetary cost of running experiments on promising drug combinations, as well as the length of time the experiments take to complete. Uncertainty models could be used by practitioners downstream help accelerate drug combination treatment discoveries and reduce involved development costs.
To test DEUP's performance on this task used the DrugComb and LINCS L1000 datasets Za- gidullin et al. (2019); Subramanian et al. (2017). DrugComb is a dataset consisting of pairwise combinations of anti-cancer compounds tested various cancer cell lines. For each combination, the dataset provides access to several synergy scores, each indicating whether the two drugs have synergistic antagonistic effect on cancerous cell death. LINCS L1000 contains differential gene expression profiles for various cell lines and drugs. Differential gene expressions measure the difference in the amount of mRNA related set of influential genes before and after the application of drug. Because of this, gene expressions are a powerful indicator of the effect of single drug at the cellular level.
In our experiments, each drug is represented by its Morgan fingerprint Morgan (1965) (with 1,024 bits and a radius of 3) as well as two differential gene expression profiles (each of dimension 978) from two cell lines (PC-3 and MCF-7). In order to use gene expression features for every drug, only used drug pairs in DrugComb where both drugs had differential gene expression data for cell lines PC-3 and MCF-7.
'The Morgan fingerprint represents molecule by associating with it boolean vector specifying its chemical structure. Morgan fingerprints have been used a signal of various molecular characteristies to great success Ballester& & Mitchell (2010); Zhang et al. (2006).
25

---

Under review as a conference paper at ICLR 2022
We first compared the quality of DEUP's uncertainty estimations to other uncertainty estimation methods on the task of predicting the combination sensitivity score Malyutina et al. (2019) for drug pairs tested on the cell line PC-3 (1,385 examples). We evaluated the uncertainty methods using train, validation, test split of 40%, 30%, and 30%, respectively. The underlying model used by each uncertainty estimation method consisted of single drug fully connected neural network (2 layers with 2048 hidden units and output of dimension 1024) and a combined drug fully connected neural network (2 layers, with 128 hidden units). The embeddings of an input drug pair's drugs produced by the single drug network are summed and passed to the combined drug network, which then predicts final synergy. By summing the embeddings produced by the single drug network, we ensure that the model is invariant to permutations in order of the two drugs in the pair. The models were trained with Adam Kingma & Ba (2015), using a learning rate of le-4 and weight decay of 1e-5. For MC-Dropout we used dropout probability of 0.1 on the two layers of the combined drug network and 3 test-time forward passes to compute uncertainty estimates. The ensemble used 3 constituent models for its uncertainty estimates. Both Ensemble and MC-Dropout models were trained with the MSE loss.
We also compared against DUE (van Amersfoort et al., 2021) which combines a neural network added to all the
feature extractor with an approximate Gaussian process. Spectral normalization layers of the combined drug network and of the single drug network. Let demb denote the dimension of the output of the combined drug network, which is also the input dimension of the approximate Gaussian process. We conducted a grid-search over different values of demh (from 2 to 100), the number of inducing points (from 3 to 200), the learning rate, and the kernel used by the Gaussian process. The highest correlation of uncertainty estimates with residuals was attained with demb = 10, 100 inducing points, a learning rate of 1e-2, and the Matern12 kernel.
Algorithm 4 DEUP for Drug Combinations
Data: D dataset of pairwise drug combinations, along with synergy scores ((d2, d2). y)
Initialization:
Split training set into two halves, in-sample Din and out-of-sample Dout fa(di, d2): á predictor which takes pair of drugs as input
(d1, d2): In-sample ôin error predictor
fo (d1,d2) d2): Out-of-sample dott error predictor
Training:
while training not finished do
In-sample update
Get an in-sample batch (di,in; d2,in: Win) in
Predict ja [p(dd,ing dz and in-sample error oin = f (di,in, dn.
Compute NLL: log(ā?.) (B-wi=)
20
Backpropagate through Sy and foD and update.
Out-of-samppl update
Get out-of-sample batch (d1,out, d2, out, Yous) Dout
Estimate ja = fu(ch outt, out and out out-off samp-le error dout = f (di, dz
28 one Compute NLL: log(e" (p-
2
Backpropagate through roul and update.
end
The DEUP model we used outputs two heads [ and is trained with the NLL log(d?) + (A=y)² similar fashion as in Lakshminarayanan al. (2017). To obtain a predictor of the oft-of-sample error, we altered our optimization procedure so that the μ and heads were not backpropagated through at all times. Specifically, we first split the training set into two halves, terming the former the in-sample set D.,. and the latter the oft-of-sample set Dout- We denote as fü" the in-sample error predictor and fout the out-of-sample error predictor. fout is used to estimate total uncertainty. Note that this setting, fout predicts the square root of the epistemic uncertainty (ōout) rather than the epistemic uncertainty itself i.....
26t in
In our experiments, an extra bit is added as input to the model in order indicate whether given bathh is from Din or D Dout- Through this, the same model is used to estimate für and fout with the model estimating fin when the bit indicates example is drawn from D.,, and fout otherwise. When
26

---

Under review as a conference paper at ICLR 2022
the batch is drawn from Din. both heads are trained using NLL using single forward pass. However, when the data is drawn from Dont only the & head is trained. To do this, we must still predict ji in order to compute the NLL. But the f predictor fp must be agnostic the difference between Din and D Dout- To solve this, we perform two separate forward passes. The first pass computes ji and sets the indicator bit to 0 so J has no notion of Dout. while the second pass computes à, setting the bit to 1 to indicate the true source of the batch. Finally, we backpropagate through the ē head only. The training procedure is described in Algorithm 4
We report several measures for the quality of uncertainty predictions on a separate test set in Table 7.
Model
Corn. w, res.
C. Bound
Ratio
Log Likelihood
Coverage Probability
CI width
MC-Dropout
0.14 + 0.07
0.56 = 0.05
0.25 1 0.12
20.1 6.8
11.4
3.1 0.1
Deep Finsemble
0.30 + 0.09
0.59 + 0.04
0.50 + 0.13
-14.3 4.7
10.8 + 1.4
3.4 + 0.6
DUE
0.12 2 0.12
0.15 :: 0.03
0.80
0.79
-13.0 0.52
15.2 13.2 1.0
3.5 1::::00 0.1
DEUP
0.47 = 0.03
0.63
0.75 0.07
-3.5 0.25
36.1 1.1:2.55
13.1 +
Table 7: Drug combinations: quality of uncertainty estimates from different methods. Corr w. res. shows correlation between model residuals and predicted uncertainties à. A best-case Upper Bound on Corr. w. res. obtained from the correlation between ē and true samples from /N o). Ratio is the ratio between col. I and 2 (larger is better). Log-likelihood: average over 3 seeds of per sample predictive log-likelihood. Coverage Probability: Percentage of test samples which are covered by the 68% confidence interval. CI width: width of the 86% confidence interval.
For each model, we report the per sample predictive log-likelihood, coverage probability and confi- dence interval width, averaged over 3 seeds.
We also computed the correlation between the residuals of the model [,(x;)) yil and the predicted uncertainties å (2xz). We noted that the different uncertainty estimation methods lead to different distributions p(ő(z)) For example, predicted uncertainties obtained with DUE always have a similar magnitude. By contrast, DEUP yields a wide range of different predicted uncertainties.
These differences between the distributions p(o(z)) obtained with the different methods may have an impact on the correlation metric, possibly biasing the comparison of the different methods. order to account for differences in the distribution p( (z)) across methods, we report another metric which is the ratio between the observed correlation Corr([ā(z) - yl. ā (x)) and the maximum achievable correlation given specific distribution (1)).
This maximum achievable correlation (refered to as the upper bound) is per se a comparison metric, and is estimated (given a specific p(ă (()z))) )) as follows: we assume that, for each example (I v/). the predictive distribution of the model (f; o((;)) corresponds exactly to the distribution of the target, iee. Yi ((((ar)), ā (a:)). Under this assumption, the residual of the mean predictor follows distribution (0, ā (zz)). We can then estimate the upper bound by computing the correlation between the predicted uncertainties (xx;) and samples from the corresponding Gaussians (O, & (x;)). samples were drawn from each Gaussian for our evaluation. This upper bound is reported in the Table.
Finally, we reported our comparison metric: the ratio between the correlation Corr(IA(z) y) (x)) and the upper bound. The higher the ratio is, the closer the observed correlation is to the estimated upper bound and the better the method is doing.
is interesting to note that the upper bound is much lower for DUE compared to other methods, as its predicted uncertainties lie within a short range of values.
Predicted f and uncertainty estimates can be visualized in Figure 10 for different models. MC-dropout, Ensemble and DUE consistently underestimate uncertainty, while the out-of-sample uncertainties predicted by DEUP are much more consistent with the order of magnitude of the residuals. Moreover, observed that DUE predicted very similar uncertainties for all samples, resulting in lower upper-bound for the correlation between residuals and predicted uncertainties compared to other methods. We observed similar pattern when experimenting with the other kernels available in the DUE package, including the standard Gaussian kernel.
Finally, we note that in the context of drug combination experiments, aleatoric uncertainty could be estimated by having access to replicates a given experiment (c.f. Section D), allowing us to subtract the aleatoric part from the out-of-sample uncertainty, leaving us with the epistemic uncertainty only.
27

---

Under review as a conference paper at ICLR 2022
Orcane Prdicaee monsaa
Proditton menoa
Occenel Pacidton Iraena
Oscenel Prec redidinn sneoak
Coarrod 60.00
Oasiniod Viloci
actemn by CANEAE ORUR)
MC-dropout
Ensemble
(C) DUB
(d) DEUP
Figure 10: Predicted mean and uncertainty for different models separate test set. 50 examples from the test set are ordered by increasing value of true synergy score (orange). Model predictions and uncertainties the uncertainty while DEUP seems to capture the right order of magnitude, Figures made using The Uncertainty Toolbox (Chung al., 2020).
visualized in blue. MC-Dropout, Ensemble and DUE consistently underestimate
One complete training run for the drug combination experiments takes about 0.01 GPU days on V100 GPU, In total these of experiments took about 0.2 GPU days on a Nvidia V100 GPU.
28


Reviews:
[{"review_id": "sPNQdWe4qTv", "reviewer": "Reviewer_Scim", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It's possible that there are some interesting ideas in this paper, but at this stage, I find it difficult to understand this paper.\n\nThe paper argues a different definition of epistemic uncertainty for the interactive learning settings but did not provide a clear argument on why their definition is more compelling than the existing ones. I suggest discussing how interactive learning is different from the standard setting, then motivating their definition.\n\nThe authors also seem to use out-of-sample generalization and out-of-distribution generalization interchangeability. For example, on page 1, it says ``The total expected out-of-sample loss at a point can be decomposed into an epistemic (reducible) part and an aleatoric (irreducible) part\". Then on page 2, it says ``DEUP error predictors can be explicitly trained to care about, and calibrate for estimating the generalization error for examples which may come from a distribution different from the distribution of most of the training examples, i.e., an out-of-distribution (OOD) setting\". out of sample generalization is about generalizing to the same distribution. Out of distribution generalization is about how to generalize to a different one. These are two very different concepts.\n \nIt's unclear what contributions 3  and 4 are supposed to mean.\n\nProposition 1 is not a result. It should not be a proposition. Is it supposed to be an illustrative example? The paragraph after proposition 1 is half-page long. I am guessing the goal is to discuss some mathematical insights? But it is impossible to parse. \n\nThe algorithm itself is also difficult to understand not at all self-contained. What is the stopping criterion? What does the ``optional (or every few interactions only)\" statement supposed to mean? What is x_{acq}?  If a and e are estimators, what does a-e mean? \nThis is one of the main results of the paper. Yet it is not at all self-contained. A reader should not have to go to the appendix to understand the algorithm.\n\n", "summary_of_the_paper": "This paper uses out-of-sample prediction error as a measurement of epistemic uncertainty.\nUsing this definition, it develops an estimator: direct epistemic uncertainty prediction. \nThe idea is to have a main predictor to learn the task, and an error predictor to predict the generalization error.\nEmpirical studies show that their proposed estimator produces better estimation on downstream tasks such as sequential model optimization and reinforcement learning. ", "summary_of_the_review": "It is possible that there are some interesting ideas, but the writing is extremely unclear. I cannot understand this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iUQhZxnRuff", "reviewer": "Reviewer_4Lh6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper provides a relatively novel approach to uncertainty estimation, which explicitly includes estimation of model bias.  With a few exceptions, I found the paper to be well written. I found it to be an enjoyable read, as it made me think about and question some of the assumptions traditionally made in the deep learning uncertainty estimation literature. For this, I think that this work can present a relevant contribution to the ICLR community. Having said this, I have some doubts/concerns which I outline below:\n\n**Major doubts**\n\n1. The paper proposes to redefine uncertainty in terms of generalisation error instead of predictive variance. I am not sure how to feel about this:\n\nPros:\n\n* The definition is clearly stated. The authors clearly put effort into avoiding confusion regarding this point. Additionally, the new definition clearly motivates the proposed method: a sensible alternative to existing methods.\n*  The authors provide clear intuition for this definition in proposition 1, relating their definition of uncertainty to more traditional estimates of uncertainty like variance for specific choices of data generative process.\n\t\nCons:\n  * This redefinition could cause confusion in the deep learning uncertainty quantification field, which is an already not very rigorous space. \n  * The new definition is not always intuitive or does it provide interpretable uncertainty estimates. I think this is due to there not being an explicit predictive distribution. Consider a binary disease diagnostic setting: The primary model outputs a probability of disease of 87%. The secondary model outputs a risk estimate of 4.7. What does this mean?\n\t\nI look forward to discussing this point with the authors and other reviewers.\n\n\n2.  The authors motivate their approach by contrasting the sub optimality of the Bayesian update under model misspecification with the asymptotic unbiasedness off large neural networks:\n\n\t* The GP shown in the first toy example of the paper clearly has a strong preference over some functions over others. However, the rest of the examples in the paper use NNs. As stated by the authors these are asymptotically unbiased. In part, this is what makes Bayesian deep learning so difficult. As a result, it is not clear to me why bias would be such a large concern in the deep learning setting. \n\t* In section 3.1, the authors claim that DEUP has asymptotically optimal properties (for large enough secondary models and held out sets). It is not clear to me how this is a relevant property: under this setting the main model would have no uncertainty and we would not need an error predictor. In the finite data setting, if the error predictor incorporates less assumptions than Bayesian Methods it will have larger variance. I dont think it is clear enough from the paper that there is no free lunch in this regard. \n\nAs a result, I think that the claim made at the end of section 2 is unfounded: “directly estimating the epistemic uncertainty as the reducible part of the loss is superior to existing variance-based predictions of epistemic uncertainty.” I think the claim in the introduction is much more reasonable ”direct estimation of epistemic uncertainty can lead to more precise estimates than variance-based ones.” I encourage the authors to explicitly discus the above and relax their claims. \n\n3. I found section 3.2 to not be very clear. The authors state that variance-based uncertainty estimators can not deal with non-stationarity in the observation distribution arising from an active learning setting. It is not clear to me why this is the case: as more data is observed, independently of what distribution it comes from, the range of possible functions that fit all the datapoints decreases and so does the variance. What am I missing here? In fact, I found this section to be written in a somewhat confusing manner overall. I am not sure I understood the problem. I encourage the authors to re-write this section.\n\t\n\n\n**Minor concerns** \n\n\n1. At the end of the introduction, the authors mention dropout and ensembles as approaches to approximate the Bayesian predictive variance.  While the former is a rather crude approximation, the latter does not target the Bayesian posterior at all. Perhaps HMC is a better example. \n\n2. It took some effort to understand Fig 2 because it kind of looks like the grey area corresponds to posterior variance while orange is bias. In fact, the bias and variance labels are for the straight and dashed arrows. Does it make sense to have this diagram? Is it needed to understand the rest of the paper? Perhaps It would be more useful if it were made more relevant to the proposed method DEUP: for example, the authors could state how DEUP targets \\epsilon instead of \\Delta in the plot. \n\n\n**Experiments** \n\n\n1. The proposed method is somewhat elaborate: It combines a k-fold error estimation, with a NN variance estimator (DUQ specifically) and a deep generative model to produce uncertainty estimates. It is somewhat unsurprising that this combination is able to outperform a variance estimator on its own.\n\n2. Apart from the toy GP experiment, I do not think that any of the experiments of the paper target the main claim of DEUP capturing error from bias in addition to variance.\n\n3. On the GP experiment: As the authors state, the GP is not allowed access to the held out data, while DEUP is. I understand this choice was made to drive home a point. However, it would have been interesting to see the setting in which the GP does have access to this data, as this setting better matches that of the rest of the paper. The Bayesian marginal likelihood could also be seen as a proxy for generalisation error and it would be interesting to see it compared against DEUP.\n\n4. In the image classification OOD rejection experiment, it is not obvious to me why we see such a difference in results when using Spearman-rank correlation and AUROC as metrics. Do the authors have any intuition for that?\n\n5. The remaining are very cool but non-standard for this type of paper. It's hard to say much about them as I am not familiar with the settings. \n\t* Their diversity is a definite plus\n\t* They seem to be relatively toy tasks however. I am somewhat surprised by this as I would have guessed the neural-network based approach to uncertainty estimation allows for good scalability. \n\n6.  Why is the vanilla DQN baseline better than uncertainty aware methods in Fig 5?", "summary_of_the_paper": "Given some supervised task, this paper redefines the uncertainty of a solution as its generalisation risk. The authors then propose to learn a secondary function to estimate the generalisation risk of the first. This is done by using held-out points as training data for the secondary model. In turn, this held out used to estimate the primary model error comes from a k-fold split.  The inputs to the secondary model are the data points being evaluated, estimates of the predictive variance of the primary model, density estimates from a generative model and whether the point has been observed by the primary model.\n\nThe authors posit that the advantage of their method is that it can capture uncertainty due to model selection bias, something omitted by existing work, which focuses on the variance of the learnt estimator. The authors provide a diverse range of experiments: OOD rejection in image classification, active learning for drug discovery, function optimisation and exploration in reinforcement learning. ", "summary_of_the_review": "This paper provides a novel view on uncertainty estimation and is an interesting read. I believe it is held back by some overly ambitions claims and a lack of depth in discussion at some points. Experimentally the paper is again interesting but fails to drive home its main points. \n\nI currently can not recommend acceptance. If the authors improve their discussion and claims I will raise my score to a weak accept. If the authors provide some experimental evidence for the existence of bias in a realistic setting and DEUP accurately estimating that bias while variance-based methods to fail, I will raise my score to a strong accept.\n\n**update**\n\nAfter a discussion with the authors I have raised my score from 5 to 6. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "32MHuDVxdOV", "reviewer": "Reviewer_zaec", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths of the paper:\n\n- Attempts to estimate the epistemic uncertainty in a creative way (total minus aleatoric).\n- Methods cover both static and interactive use cases\n- \n\nWeaknesses of the paper:\n\n- Dependence on the Oracle for estimating Aleatoric uncertainty makes its practical use very limited IMO. While the approach bypasses some of the limitations of existing epistemic uncertainty techniques (e.g vulnerability to misspecification), the dependence on having an aleatoric uncertainty estimator or Oracle can be a major limitation as well.\n\n- The generalization of the secondary model (the one that predicts total generalization error) itself feels to be on shaky grounds. It appears that the main difference in the secondary model (compared to similar works that employ a secondary model) is that it uses labels from out-of-sample examples. However those examples are (by definition) something already available at training time, and it's not clear how or why we somehow ended up with the ability to predict total error in OOD settings?\n\n- The overall narrative / presentation is hard to follow. There is too much of back and forth between the main paper and Appendix. Some of the key elements of the paper (e.g Algorithm 2, which is likely of high interest to a majority of readers) is down in the appendix.", "summary_of_the_paper": "This paper proposes a method to estimate the epistemic uncertainty (uncertainty due to lack of knowledge/data) at a new model input. The paper takes an indirect approach towards this goal by a) first estimate the generalization error at the new input, b) next estimate the aleatoric error (inherent uncertainty in the data distribution / irreducible error), and c) subtract aleatoric error estimate from the total generalization error estimate to obtain the epistemic uncertainty estimate.\n\nThe authors claim that this method captures both the uncertainty due to lack of data, as well as model misspecification in the process - while other/existing methods focus mostly on the variance of the posterior distribution (or its approximation) as a measure of epistemic uncertainty (and thereby implicitly assume the model is well specified.)\n\nEstimating the total generalization error itself is performed with a second neural network model, which uses residuals obtained from the primary model as its labels. Estimating aleatoric error either assumes the presence of an Oracle or \n\nThe authors apply their technique on both static (fixed data set) as well as interactive (active learning) settings, though mostly focused on the mean squared error loss function.", "summary_of_the_review": "The main factors influencing the recommendation are the strengths and weaknesses highlighted above.\n\nThe paper can be improved significantly by:\n\na) Adding narrative to help justify why the secondary network generalized to OOD settings where the primary network couldn't. Perhaps present results that shine light on just this sub-aspect.\n\nb) Improving the overall flow and presentation.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-LOIr9T5p6d", "reviewer": "Reviewer_iSGN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n1. The paper focuses on estimating epistemic uncertainty of machine learning models. Estimating predictive uncertainty of black box models is an important research area with promises on enabling risk aware decision making, which will enable safe deployment of such models to production.\n2. The proposed approach is practical, which is important for its adoption by others. Interesting extensions are proposed (e.g. data density estimation and model variance features) that helps with improving DEUP's performance. It also does not affect the original model training, which is a positive.\n3. Experiments are conducted in different settings with compelling performance compared to the state of the art methods like deep ensembles and MC Dropout.\n\nCons:\n\n1. The paper compares DEUP's uncertainty estimates to variance estimates obtained from other techniques. Variance (and standard deviation) provides intuitive explanations on what they are capturing (i.e. the dispersion of predicted values). Furthermore, they have nice links to prediction intervals, which also provides nice intuitions on the estimated parameters with their probability. The other techniques (e.g. ensembles and MC Dropout) that are indicated as yielding variance estimates can also be directly used to compute such intervals from their empirical predictive distributions. I think the proposed uncertainty estimates would benefit a lot from such intuitions. For instance, by training the error predictor to predict the square error loss, my intuition is that DEUP is predicting extreme quantiles. I would love to hear authors' intuition as well as how the uncertainty estimates map to prediction intervals.\n2. Since the approach relies on a predictive model to estimate the errors, do the authors believe that DEUP will be susceptible to issues predictive models face (e.g. how important is model fit, how to prevent overfitting)? For instance, in the case of fixed training set where the set used does not capture OOD dataset and density features are not used, is DEUP's error prediction able to generalize to OOD?\n3. Questions on experiments:\n    1. Section 5.1.2: I like the OOD analysis conducted. I have some minor comments:\n        1. CIFAR-10 with SVHN provides a very strong OOD dataset, which is also evident from high AUROC values. In such a setting, I am interested to hear why the authors think the rank correlation metric between predicted uncertainty and OOD generalization error is a good metric. I actually would be interested to see an analysis where OOD definition is more subtle/gradual. For instance, Ovadia et al. 2019 provides such an analysis with increasing distribution shifts. I would be very interested to see how the method performs in such settings.\n    2. Section 5.1.3: The metrics used here can be improved (except log likelihood). For instance, coverage probabilities and CI widths would be great metrics to use here, which will also help with the intuition point made above. I am not able to fully grasp what Upper Bound on Corr. w. res. metric is capturing (as well as the Ratio metric).\n\nMinor comments:\n- Algorithm 1 requires an estimator of aleatoric uncertainty. Does the authors use that in any of the experiments and how do they define it?\n- Appendix pages 19, 20, 26 has missing links (indicated with ??'s).\n- Appendix Table 6: I would be very interested to see the performance of features by themselves in this table.", "summary_of_the_paper": "This paper proposes a new approach for computing epistemic uncertainty. The proposed approach, DEUP, builds a new model (in addition to the original model) which predicts epistemic uncertainty, defined as generalization error minus aleatory uncertainty. I highlight some features of DEUP below:\n- DEUP works with a hold-out dataset that is used for training the error predictor. \n- In case there does not exist a hold-out set or in interactive settings (like RL or active learning), DEUP is extended to be used in a cross-validation setting and the features used to fit the error predictor is extended to include data density estimates and model variance.\nDEUP is evaluated on different settings including OOD data, sequential model optimization and RL.", "summary_of_the_review": "Overall I vote for accepting the paper. I like the approach of directly estimating uncertainty. I have some concerns on providing intuition on the estimated uncertainty, which I believe could be addressed by some additional discussion and some more analyses. Hopefully the authors can address my concern in the rebuttal period. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript develops a novel method for uncertainty prediction that can be used in the context of active or reinforcement learning problems. They consider experiments such as an OOD Detection task wherein a ResNet is trained on CIFAR10 and predictions must subsequently be made for in- versus out-of-distribution (SVHN) data. 
The work develops an approach based on directly estimating epistemic (as opposed to a aleatoric) uncertainty by learning to predict generalization error and then subtracting estimated aleatoric uncertainty.
Reviewers found the essential approach to be novel and creative. However, there were several issues raised by reviewers that are not well addressed by responses by the authors. For example, Zaec worries about the dependence of the approach on an oracle for estimating aleatoric uncertainty. Multiple reviewers were concerned that this would make the approach unsuitable for many situations and thus limit the applicability of the ideas. 
Multiple reviewers also found the manuscript to be difficult to understand. I agree with the sentiment. While there may indeed be an interesting and important idea here, the text and explication of the algorithm and approach are complicated and leave the reader unsure about the contribution. I would recommend that the authors invest time and effort in simplifying and streamlining the narrative and presenting the technical innovation so that it is easier to judge. In it's current form, the manuscript is premature for publication.
Decision: Reject