DEUP: DIRECT EPISTEMIC UNCERTAINTY PREDIC-
Year: 2021

Abstract:
Epistemic uncertainty is the part of out-of-sample prediction error due to the lack of knowledge of the learner. Whereas previous work was focusing on model variance, we propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty, i.e., intrinsic unpredictability. This estimator of epistemic uncertainty includes the effect of model bias (or misspecification) and is useful in interactive learning environments arising in active learning or reinforcement learning. In addition to discussing these properties of Direct Epistemic Uncertainty Prediction (DEUP), we illustrate its advantage against existing methods for uncertainty estimation on downstream tasks including sequential model optimization and reinforcement learning. We also evaluate the quality of uncertainty estimates from DEUP for probabilistic classification of images and for estimating uncertainty about synergistic drug combinations.

heading: 1 INTRODUCTION
text: A remaining great challenge for machine learning research is purposeful knowledge-seeking by learning agents, which can benefit from estimation of epistemic uncertainty, i.e., a measure of lack of knowledge, which could potentially be eliminated with enough data if the learner converges to a Bayes-optimal predictor. Epistemic uncertainty estimation is already a key ingredient in active learning and Bayesian optimization (Aggarwal et al., 2014; Frazier, 2018) as well as exploration in Reinforcement Learning (Kocsis & Szepesvári, 2006; Tang et al., 2017). Epistemic uncertainty (EU) thus tells us how much could be gained from learning around a particular area of state-space or input data space. But how should we even quantify it? Much previous work has focused on model variance (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017) as a proxy thereof, i.e., how different are the functions compatible with the learner’s preferences and the data. However, this approach does not take into account model bias (or misspecification), due to the learner’s finite capacity or preferences, self-imposed to avoid overfitting or reduce computational costs. Recent work (Masegosa, 2020) has also shown that Bayesian methods are suboptimal for learning predictive models when the model is misspecified. Fig. 1 illustrates with Gaussian Processes (GP) that a model’s posterior variance is not necessarily a good predictor of the reducible generalization error.
0.5 0.0 0.5 1.0 1.5 2.0 2.5
0
2 Ground truth GP mean
0.5 0.0 0.5 1.0 1.5 2.0 2.5 0
2
4 Epistemic uncertainty GP variance
0.5 0.0 0.5 1.0 1.5 2.0 2.5 0
2
4 Epistemic uncertainty DEUP's prediction
learner), and it is also the error which may be achieved by a Bayes-optimal predictor. Consider the case of a learning system with universal approximation properties, like modern deep learning, whose effective capacity can be increased with more data, either via model selection, early stopping or other means. In such cases, the part of the error which is not aleatoric is reducible with more data. For this reason, we propose to define epistemic uncertainty as the difference between the expected generalization error and aleatoric uncertainty. Estimating it is useful to an active or interactive learner because it could be reduced with enough data, especially in regions of the input space we care about, i.e., where EU is large. In order to estimate EU, we propose DEUP, for Direct Epistemic Uncertainty Prediction, where we train a side learner (the uncertainty predictor) with an appropriate objective and training data in order to predict this generalization error. An estimate of EU can thus be obtained by subtracting an estimate of aleatoric uncertainty.
In this paper we thus focus on two predictors: the main predictor (for the learning task of original interest) and an error predictor which can be used to predict epistemic uncertainty of the main predictor at given points, given some knowledge or estimate of the aleatoric uncertainty. The proposed methodology can be seen as an end-to-end approach to epistemic uncertainty estimation, i.e., to help estimate where a learner would acquire the most information in terms of out-of-sample loss reduction, without relying on the brittle relationship between model variance and generalization error, which can get murky in the presence of misspecification (bias), as we argue below. In order to select points with high EU, one would still need some kind of search (in the space of x), e.g., to propose new candidates for active learning (Aggarwal et al., 2014; Nguyen et al., 2019; Bengio et al., 2021) or sequential model optimization (Kushner, 1964; Jones et al., 1998; Snoek et al., 2012), or exploration in RL (Kocsis & Szepesvári, 2006; Osband et al., 2016; Janz et al., 2019).
An interesting potential advantage of the proposed approach, compared with model variance estimators, is that DEUP error predictors can be explicitly trained to care about, and calibrate for estimating the generalization error for examples which may come from a distribution different from the distribution of most of the training examples, i.e., an out-of-distribution (OOD) setting. Such non-stationarities arise naturally in contexts like active learning and reinforcement learning (RL) because the learner explores new areas of the input space. In these non-stationary settings, we typically want to retrain the main predictor as we acquire new training data, not just because more training data is generally better but also to better track the changing distribution and generalize to yet unseen but upcoming OOD inputs. This setting makes it challenging to train the main predictor but it also entails an even greater non-stationarity for the training data seen by the error predictor: a large error initially made at a point x before x is incorporated in the training set (along with an outcome y) will typically be greatly reduced after updating the main predictor with (x, y). We propose several strategies to cope with this non-stationarity, the most important being to rely on additional features (such as log-density estimates and model variance estimates at x). These features account for variations in prediction error arising due to updates of the main predictor, making DEUP’s training distribution more stationary.
The experiments presented here explore first the scenario of purely trying to estimate epistemic uncertainty in terms of the recoverable generalization error, and second how this can be used in contexts of sequential model optimization (or black-box optimization) and exploration in RL (where the issue of non-stationarity arises). The main contributions of this paper are thus the following:
1. Novel end-to-end approach for directly estimating the epistemic uncertainty as the lack of knowledge about the true f∗(x) = E[Y |x], the Direct Epistemic Uncertainty Prediction (DEUP) approach.
2. Experimental evidence that this direct estimation of epistemic uncertainty can lead to more precise estimates of the lack of knowledge than variance-based methods.
3. A mean to mitigate the issue of non-stationarity arising in training the uncertainty estimator in interactive settings.
4. A successful validation of the above in the contexts of (a) sequential model optimization and (b) exploration in RL.

heading: 2 ALEATORIC AND EPISTEMIC UNCERTAINTIES, MODEL VARIANCE
text: Consider a supervised learning algorithm (or learner) L mapping a dataset D to a predictive function f̂ = L(D). L tries to minimize the expected value of a supervised learning loss l(f̂(x), y) ∈ R under unknown P (Y |x). In this section, we focus on regression tasks with the squared loss l(ŷ, y) = (ŷ−y)2, with y ∈ R. In Appendix B, we provide the corresponding results for a general loss function.
Definition 2.1 The expected loss of a predictor f̂ at x is defined as: U(f̂ , x) = ∫ (f̂(x)− y)2dP (y|x). (1)
Note how the expected loss is an unknown quantity, because we generally do not have access to the true data generating distribution P (Y |x). The errors made by any predictor f̂ at x are due to both the inherent randomness of P (Y |x) (aleatoric uncertainty) and the lack of knowledge of the predictor that can be tackled by acquiring more information around x (epistemic uncertainty). Next, we will define these concepts more formally. Because of this natural decomposition of the expected loss, we will also refer to it as total uncertainty, and we will use the two terms interchangeably.
Bayes-optimal predictors f∗ satisfy the following equation at every x: ∀ỹ ∈ R ∫ (f∗(x)− y)2dP (y|x) ≤ ∫ (ỹ − y)2dP (y|x).
They depend on the underlying data distribution only and not on learner L or trained predictor f̂ . Additionally, all Bayes-optimal predictors have the same total uncertainty for all x. Aleatoric uncertainty is the irreducible expected error, i.e., that made by a Bayes-optimal predictor:
Definition 2.2 We define aleatoric uncertainty at x as the total uncertainty of any Bayes-optimal predictor f∗ at x:
A(x) = U(f∗, x). (2)
and we note that by definition A(x) ≤ U(f, x), ∀f ∀x.
We now propose a definition of epistemic uncertainty of predictor f̂ as the gap between the error of f̂ at x and the lowest possible error at x, i.e., the reducible part of the loss, given more knowledge.
Definition 2.3 We define epistemic uncertainty E(f̂ , x) of a predictor f̂ at x as
E(f̂ , x) = U(f̂ , x)−A(x) = U(f̂ , x)− U(f∗, x). (3)
Using these definitions, we can present the main result about the regression setting.
Proposition 1 In a regression task with ground truth P (y|x) = N (y; f∗(x), σ2(x)),
E(f̂ , x) = (f̂(x)− f∗(x))2
U(f̂ , x) = Ey∼P (.|x)[(f̂(x)− y)2] = (f̂(x)− f∗(x))2 + σ2(x) A(x) = Ey∼P (.|x)[(f∗(x)− y)2] = σ2(x)
The proof is given in Appendix C. Additionally, we present in Appendix B a similar result for the negative log-likelihood loss, with discrete or continuous outputs.
We will discuss in the remainder of this section the relation between this new definition of epistemic uncertainty and the more traditional way of modelling it through model variance or entropy.
Consider a parametric model p(Y |x, θ) and a learner maintaining a distribution over parameters θ ∈ Θ, each corresponding to a predictor f in a parametric set of functions S, possibly starting from a prior p(θ) that would lead to a posterior distribution p(θ|D). Clearly, the fact that multiple θ’s and corresponding values of f are compatible with the data and the prior indicates lack of knowledge. Because the lack of knowledge indicates where an interactive learner should acquire more information, this justifies the usage of dispersion measures, such as the variance or the entropy of the posterior predictive, as proxies for epistemic uncertainty.
However, the limited capacity of S or the prior p(θ) may pull the optimal f̃ in S away from the Bayes-optimal predictor f∗. We can refer to these self-imposed constraints as a form of bias, in the sense that the learner is usually biased towards the prior preferences, e.g., towards smoother predictors. In particular, this issue can arise when training neural networks with limited data, where the network may not use all of its capacity due to implicit (and not fully understood) regularization properties of SGD (Kale et al., 2021), explicit regularization, early stopping or a combination of these, which can induce a preference on the functions it learns. In Fig. 2, we illustrate this gap with the bias function b(x) = f∗(x)− f̃(x). Because of this bias, model variance cannot be an accurate measure of lack of knowledge E(f̂ , x) = (f̂(x) − f∗(x))2 in the general case, as we have shown with the results in Fig. 1. In Deep Ensembles (Lakshminarayanan et al., 2017) e.g., if all the networks in the ensemble tend to fail in systematic ways, this aspect of prediction failure will not be captured by variance. Whereas Deep Ensembles variance provide us uncertainty regarding which of the networks we could draw is correct, this does not tell us how poor that network is even in a noise-free setting. On the other hand, with flexible models like neural networks, adding examples around x where b(x)2 is large may allow to increase capacity around x and reduce b(x)2.
In section 5, we will provide more experimental evidence supporting our claim that directly estimating the epistemic uncertainty as the reducible part of the loss can lead to better measures of lack of knowledge, compared to variance-based methods, especially in interactive learning settings.

heading: 3 DIRECT EPISTEMIC UNCERTAINTY PREDICTION
text: DEUP (Direct Epistemic Uncertainty Prediction) uses observed out-of-sample errors in order to train an error predictor which can be used to estimate epistemic uncertainty elsewhere, as suggested directly by definitions 2.1-2.3. These may be in-distribution or out-of-distribution errors, depending on whichever we care about. The pseudo-code of DEUP in interactive settings is provided below. The pseudo-code for the simpler version with a fixed training set is given in Appendix E.

heading: 3.1 FIXED TRAINING SET
text: Consider the simpler scenario with fixed training set D and learning algorithm L yielding predictor f̂ . We assume that N out-of-sample (validation) points have been set aside and can thus be used for training an error predictor.
Training an error predictor e with (input, target) pairs (x, (f̂(x)− y)2) and squared loss yields an estimator of the total uncertainty. To see that this is the right training set for the secondary predictor e, we can assess what happens at the limit of infinite data: the estimator is asymptotically unbiased, i.e. limN→∞ e(x) = U(f̂ , x) = E[(f̂(x)− Y )2] for all x, if the learning algorithm ensures asymptotic convergence to a Bayes-optimal predictor. Examples of such learners include k-nearest neighbors with k increasing at a proper rate slower than n, and neural networks whose size and regularization are hyper-parameters optimized on a growing validation set.
IfA(x) = 0 then e(x) is an estimate of E(f̂ , x) as well as of U(f̂ , x). If an estimator a(x) of aleatoric uncertainty is available, then e(x)− a(x) becomes an estimate of the epistemic uncertainty of f̂ at x. Alternatively, one could train e, constrained to be positive (e.g. with a softplus(u) = log(exp(1 + u)) output non-linearity), with examples (x, (f̂(x)−y)2−a(x)) and also obtain an estimator of epistemic uncertainty. When we have access to an oracle that samples Y given query x from the environment
P (Y |x) (e.g., in active learning or black-box optimization), then A(x) can be estimated using the empirical variance of different outcomes of the oracle at the same input x; see Appendix D.
Note that because U and E are functions of the predictor f̂ as well (and f̂ is itself derived from the training data D), the error predictor can also be trained on features that are informative of the pair (f̂ , x) rather than x only. We will mainly use variance estimates (V), obtained from a separate model (e.g. a GP), and log-density estimates (D), in addition to the input (x). The usage of these features is particularly important in non-stationary settings, as discussed next.

heading: 3.2 DEUP IN INTERACTIVE SETTINGS
text: Interactive settings, in which epistemic uncertainty estimation is used in order to guide information acquisition and iteratively gather new examples, provide a more interesting use case for DEUP (as shown in Figures 4 and 5, where good epistemic uncertainty estimation translates in faster learning curves). These settings are however more challenging: the main predictor is going to be retrained multiple times, with the new examples being added to the observed dataset each time. Two kinds of non-stationarities arise here:
First, the new examples are not drawn i.i.d. (since their choice depends on the epistemic uncertainty estimator obtained from earlier examples). This makes the stream of data seen by the main predictor non-stationary. For example in sequential model optimization or RL, the new examples may concentrate more and more towards regions where the reward might be higher. Whereas it is not clear how variance-based predictors can handle this, DEUP can in principle cope with it by virtue of having the error predictor trained not on out-of-sample and in-distribution examples but on out-of-sample and out-of-distribution examples of the kind we care about (e.g., as obtained from the interactive learning process). The error predictor then learns to predict the error which would be made on typical new examples coming from what we may call by analogy the "frontier of knowledge", while this frontier continuously expands due to exploration and interactive learning: we expect such out-of-distribution errors to be larger than in-distribution out-of-sample errors.
Second, the error predictor itself faces even more distributional change, even if the data stream seen by the main predictor was i.i.d.. Indeed, after example x was selected, the main predictor will be retrained using that additional data point, and x would now have much lower epistemic uncertainty. This means that the targets of the error predictor also change, as the main predictor is updated with newly acquired points.
Because the datasetD that is used to train the main predictor changes at each step of an active learning process, it is necessary to see U and E as functions of the dataset as well: U(f̂ , x) = U(L(D), x), meaning that we can no longer use (input, output) pairs (x, f̂(x) − y)2) as mentioned in section 3.1 to train an estimator of the total uncertainty, but should in principle use (input, output) pairs ((x,D), (f̂(x)−y)2)), where f̂ = L(D). However,D being a very high-dimensional object, with size growing with the number of acquired points, we may face severe overfitting issues when training an uncertainty estimator using (x,D) as inputs. Hence, in this paper, we propose to use stationarizing features of the dataset D at x, that we denote by φD(x), as inputs to the error predictor instead of (x,D). In addition to the input x itself, we explored the use of two scalar features to stationarize the training set of the error predictor: a model variance estimate, and a log-density estimate. Both have been used as proxies for epistemic uncertainty in the literature. Furthermore, we considered a binary feature s to incorporate in φD(x), that specifies whether x was already used to train the main predictor (or equivalently, x ∈ D) or not.
More formally, we use φD(x) = ( x, s, log q̂(x|D), V̂ (L̃,D, x) ) , where q̂(x|D) is a density estimate
from data D at x, s = 1 if x ∈ D otherwise 0, L̃ a learner that produces a distribution over predictors, which can chosen to be the same as L, and V̂ (L̃,D, x) is an estimate of the model variance of L̃ trained on D at x. q̂ is obtained by training a third model (besides the main predictor and the error predictor) as a density estimator (such as a Kernel Density Estimator or a flow-based deep network (Rezende & Mohamed, 2015), in our case). Like the other predictors, the density estimator also needs to be fine-tuned when new data is added to the training set.
In our experiments, we found that using inputs φD(x), or even a subset of the 4 possible features, is sufficient to train an uncertainty estimator with targets (f̂(x)− y)2).
The pseudo-code is provided in Algorithm 1. Note that De is incremented twice with xacq but φD(xacq) is different each time because xacq first is and then is not yet in D.

heading: 3.2.1 KICKSTARTING THE ERROR PREDICTOR
text: In practical applications, whether in a fixed dataset setting or in interactive settings, it is preferable to use all the available data to train the main predictor, rather than keeping a subset to train the error predictor. This would entail inaccurate uncertainty estimates once real out-of-sample unlabelled data comes in. To tackle this, we propose a scheme inspired by standard K− fold cross-validation in order to kickstart the error predictor with the initially available data. Multiple versions of the main predictor, trained on different subsets of the training data D, are used to define targets for the error predictor, whereas the subsets themselves provide the input features. The scheme is summarized in the Optional Cross-Validation step of Algorithm 1. Note that in the case of multi-class classification tasks (with n classes), we obtain the folds by splitting the data based on classes, with each fold containing bn/Kc classes. So when we train on K − 1 folds, the bn/Kc classes from the remaining fold are out-of-sample.
Algorithm 1 Training procedure for DEUP in an Active Learning setting Data: Dinit initial training dataset with pairs (x, y) ∈ X × R a : X 7→ R, estimator of aleatoric uncertainty f̂ : X 7→ R, main predictor (of y given x) e : X 7→ R, total uncertainty estimator φ : X 7→ Rk: chosen stationarizing features acq: acquisition machinery that proposes new input points xacq from X , using the current f̂ and e Training: Initialize empty dataset to train the uncertainty estimator, De Initialize D = Dinit, the dataset of training points seen so far if kickstart_error_predictor then
Cross-validation step: Pre-fill De using training errors on initial training data Dinit: while stopping criterion not reached do
Split D into K random subsets D1, . . . ,DK of equal size. Define D̃ = ⋃K−1 k=1 Dk Fit f̂ on D̃, and fit the features φD̃ on D̃ De ← De ∪ ⋃ (x,y)∈D{ ( φD̃(x), (y − f̂(x))2 ) }
Fit e on De end
end xacq ← ∅, yacq ← ∅ while stopping criterion not reached do
Optional (or every few iterations only): Fit a on D (e.g. see Appendix D) Fit f̂ on D Fit features in φD on D (e.g. density estimation) De ← De ∪ { ( φD(xacq), (yacq − f̂(xacq)2 ) } if xacq 6= ∅
Fit e on De xacq ← acq(X , f̂ , e− a) (can be either a single point, or a batch of points) Sample outcomes from the ground truth distribution: yacq ∼ P ( . |xacq) De ← De ∪ { ( φD(xacq), (yacq − f̂(xacq)2 ) }
D ← D ∪ {(xacq, yacq)} end

heading: 4 RELATED WORK
text: Kiureghian & Ditlevsen (2009) characterized the sources of uncertainty as aleatoric (inherent noise) and epistemic (incomplete knowledge). Gaussian Processes or GPs (Williams & Rasmussen, 1995), provide a way to capture epistemic uncertainty through the disagreement between the different predictors that fit the data.
In a deep learning context, Blundell et al. (2015); Kendall & Gal (2017); Depeweg et al. (2018) use the posterior distribution of network weights (MacKay, 1992) in Bayesian Neural Networks (BNNs) to capture epistemic uncertainty. Other techniques that rely on measuring the discrepancy between different predictors as a proxy for epistemic uncertainty include MC-Dropout (Gal & Ghahramani,
2016), that interprets Dropout (Hinton et al., 2012) as a variational inference technique in BNNs. These approaches, because they rely on sampling multiple sets of weights or dropout masks at inference time, share some similarities with ensemble-based methods, that include bagging (Breiman, 1996) and boosting (Efron & Tibshirani, 1994), in which multiple predictors are trained, and used jointly to make a prediction, although the latter measure variability due to the training set instead of the spread of functions compatible with the given training set, as in Bayesian approaches. For example, Shaker & Hüllermeier (2020) use Random Forests (Breiman, 2001) to estimate epistemic uncertainty. Deep Ensembles (Lakshminarayanan et al., 2017) are closer to the Bayesian approach, using an ensemble of neural networks that differ because of randomness in initialization and training (as you would have with MCMC, albeit in a more heuristic way). In addition to this, several other variants of this central idea of measuring discrepancy between different predictors have been proposed recently (Malinin & Gales, 2018; Tagasovska & Lopez-Paz, 2019; Amini et al., 2020; Liu et al., 2020; van Amersfoort et al., 2020; Wen et al., 2020; Antoran et al., 2020; Kirichenko et al., 2020; Malinin et al., 2020a; van Amersfoort et al., 2021). We discuss these methods in more detail in Appendix A.
More closely related to DEUP, Yoo & Kweon (2019) proposed a loss prediction module for learning to predict the value of loss function. Hu et al. (2020) also propose using a separate network that learns to predict the variance of an ensemble. These methods, however, are trained only to capture the in-sample error, and do not capture the out-of-sample error which is more relevant for scenarios like active learning where we want to pick x where the reducible generalization error is large. EpiOut (Umlauft et al., 2020; Hafner et al., 2019) propose learning a binary output that simply distinguishes between low or high epistemic uncertainty.

heading: 5 EXPERIMENTS

heading: 5.1 UNCERTAINTY ESTIMATION

heading: 5.1.1 IMPORTANCE OF EPISTEMIC UNCERTAINTY CALIBRATION
text: In tasks that use uncertainty estimates to make decisions, and when the resources are limited, it is important to have well calibrated uncertainty estimates, to avoid exploring uninteresting areas of the search space, as we will observe in the following subsections. We aim at comparing DEUP, which presumably takes into account both bias and variance, with established methods that are based on variance. Hence, we design a function with varying degrees of smoothness (as measured by the absolute values of its derivatives), and use fewer training points in the less smooth regions, where epistemic uncertainty will thus be higher. We first consider the scenario described in Section 3.1, where the task is to regress a known one-dimensional function.
In Figure 1, we use a GP as a predictor, and compare the GP’s variance to the epistemic uncertainty as per Eq. 3 (the square loss between the GP’s predicted mean and the ground truth, as per Proposition 1). The figure shows that, naturally, in regions of the input space where little or no data is available, and especially in the less smooth regions, the predicted variance underestimates the epistemic uncertainty (squared error) by a higher margin compared to the other regions. We see that training another GP to directly estimate this squared loss, using a small held out out-of-sample data set for which the ground truth value is known, allows to close the gap between the predicted uncertainty and the true epistemic uncertainty. Note that this second predictor can be trained using any chosen learning algorithm L. Naturally, if we had access to a held out set, it would be wiser to use it as part of the training set of the main predictor, rather than the uncertainty predictor. Alternatively, we could use cross-validation splits (Section 3.2.1), at the expense of computational efficiency, to avoid discarding precious data. In active learning, it is the acquired points, before they are used to retrain the main predictor, that also act as the out-of-sample examples to train DEUP. Using the stationarizing features introduced in Section 3.2, these acquired points should be informative enough for DEUP to generalize its uncertainty estimates to unknown regions of the search space. In RL, because the targets (e.g. of Q-Learning) are themselves estimates and moving, data seen at any particular point is normally out-of-sample and can inform the uncertainty estimator, when the inputs are used with the stationarizing features.

heading: 5.1.2 EPISTEMIC UNCERTAINTY PREDICTIONS FOR REJECTING DIFFICULT EXAMPLES
text: Epistemic uncertainty estimates can be used to reject difficult examples where the predictor might fail, such as OOD inputs1. We thus consider a standard OOD Detection task (van Amersfoort et al., 2020; 2021), where we train a ResNet-18 (He et al., 2016) for CIFAR-10 classification (Krizhevsky, 2009)
1e.g. rare but challenging inputs can be directed to a human, avoiding a costly mistake
and reject OOD examples using estimated uncertainty in the prediction. To facilitate rejection of classes other than those in the training set, we use a Bernoulli Cross-Entropy Loss for each class (van Amersfoort et al., 2020): l(f̂(x), y) = − ∑ i yi log f̂i(x) + (1 − yi) log(1 − f̂i(x)), where y is a one-hot vector (yi = 1 if i is the correct class, and 0 otherwise), and f̂i(x) = predicted probability for class i, so the target for out-of-distribution data (from other classes) is y = {0, . . . , 0}. To ascertain how well an epistemic error estimate sorts non-training examples by the above NLL loss, we consider the rank correlation between the predicted uncertainty and the observed OOD generalization error on SVHN examples (Netzer et al., 2011). This metric focuses on the quality of the uncertainty estimates rather than just their ability to simply classify in- vs out-of-distribution. We also report the AUC for the OOD detection task; more details in Appendix F. Table 1 shows that with the variance from DUE (van Amersfoort et al., 2021) and the density from MAF (Papamakarios et al., 2017) as DEUP inputs, we obtain uncertainty estimates that have high rank correlation with the underlying generalization errors and competitive AUROC, compared with the baselines. In addition, since the error predictor is trained separately from the main predictor there is no explicit trade-off between the accuracy of the main predictor and the quality of uncertainty estimates. We achieve competitive accuracy of 93.89% for the main predictor. We ignore the effect of aleatoric uncertainty (due to inconsistent human labelling), which would require a human study to ascertain. We present additional results in a distribution shift setting in Appendix F.

heading: 5.1.3 EPISTEMIC UNCERTAINTY ESTIMATION FOR DRUG COMBINATIONS
text: We validate DEUP in a real-world regression task predicting the synergy of drug combinations. While much effort in drug discovery is spent on finding novel small molecules, a potentially cheaper method is identifying combinations of pre-existing drugs which are synergistic (i.e., work well together). However, every possible combination cannot be tested due to the high monetary cost and time required to run experiments. Therefore, developing good estimates of epistemic uncertainty can help practitioners select experiments that are both informative and promising. As shown in Table 3(c), the out-of-sample error predicted by DEUP correlates better with residuals of the model on the test set in comparison to several other uncertainty estimation methods. Moreover, DEUP better captured the order of magnitude of the residuals as shown in Figure 3. Details on experiments and metrics are in Appendix J.

heading: 5.2 SEQUENTIAL MODEL OPTIMIZATION
text: Sequential model optimization or black-box optimization are forms of active learning. At each stage, the learner chooses query examples to label, looking for examples with a high value of the unknown oracle function. Such examples are selected so they have a high predicted value (to maximize the unknown oracle function) and a large predicted uncertainty (offering the opportunity of discovering yet higher values). Acquisition functions, such as Upper Confidence Bound (UCB, Srinivas et al. (2010)) and Expected Improvement (EI, Močkus (1975)) trade-off exploration and exploitation, and
one can select the next candidate by looking for x’s maximizing the acquisition function. We combine UCB and EI with DEUP (DEUP-UCB and DEUP-EI) to perform active learning, treating the main predictor and DEUP epistemic uncertainty prediction at x respectively as mean and variance of a Gaussian distribution for the learner’s guess of the value of the oracle at x. We showcase how using DEUP to calibrate GP variances (used as input for DEUP) allows for better performances in higher-dimensional optimization tasks. Specifically, we compare DEUP-EI to TuRBO-EI (Eriksson et al., 2019), a state-of-the-art method for sequential optimization, that fits a collection of local GP models instead of a global one in order to perform efficient high-dimensional optimization, on the Ackley function (Ackley, 2012) as oracle, a common benchmark for optimization algorithms. The oracle function can be defined for arbitrary dimensions, and has many local minima.
In Figure 4, we compare different methods on the Ackley-10 function, in addition to the optimum reached in budget-constrained optimization problems for different oracle input dimensions, and we find that adapting DEUP to TuRBO consistently outperforms regular TuRBO, especially in higher dimensions. See also Appendix H for 1D and 2D black-box optimization tasks where DEUP-EI outperforms GP-EI (Bull, 2011), as well as neural networks with MC-Dropout or Ensembles. We find GP-EI getting stuck in local optima whereas DEUP-EI was able to reach the global maximum consistently. Experimental details are provided in Appendix H.3.
5.3 REINFORCEMENT LEARNING
Similar to sequential model optimization, a key challenge in reinforcement learning (RL) is efficient exploration of the input state space. To investigate the effectiveness of DEUP’s uncertainty estimates in the context of RL, we incorporate epistemic uncertainties predicted by DEUP to DQN (Mnih et al., 2013), which we refer to as DEUP-DQN. Specifically, we train the uncertainty predictor with the objective to
predict the TD-error, using log-density as a stationarizing input. The predicted uncertainties are then used as an exploration bonus in the Q-values. Details of the experimental setup are in Appendix I. We evaluate DEUP-DQN on CartPole, a classic RL task from bsuite (Osband et al., 2020), against DQN + -greedy, DQN + MC-Dropout (Gal & Ghahramani, 2016) and Bootstrapped DQN (Osband et al., 2016). Figure 5 shows that DEUP achieves lower regret faster, compared to all the baselines, which demonstrates the advantage of DEUP’s uncertainty estimates for efficient exploration.

heading: 6 CONCLUSION AND FUTURE WORK
text: Whereas standard measures of epistemic uncertainty focus on variance, we argue that bias (misspecification) can also be reduced with predictors like neural nets, e.g. using early stopping. In a regression setup, the expected out-of-sample squared error minus aleatoric uncertainty thus captures all the uncertainty about E[Y |x] that more data can allow to reduce. This motivates the DEUP estimator, using a second network trained to predict the errors of the first. In interactive settings, this nonetheless raises non-stationarity challenges for this estimator and we propose input features to reduce this issue and show experimentally their advantages. Future work should investigate ways to estimate aleatoric uncertainty when one cannot simply query the oracle several times on the same x.
ETHICS STATEMENT
The authors do not foresee any negative social impacts of this work, but of course the accumulation of improvements in ML could be misused as it may give more power to nefarious agents.
REPRODUCIBILITY STATEMENT
We provide the code to reproduce the results along with the submission. In addition to the code, we also outline all the details of the procedure to ensure reproducibility of the results. The general procedure for the method in the fixed-data and interactive setting is provided in Algorithm 2 and Algorithm 1 respectively. We also provide pseudocode for the specific instances of DEUP used for the Drug-Combinations and Reinforcement Learning experiments in Algorithm 4 and Algorithm 3. Detailed discussion about the methodology and hyperparameters are provided in the appendix in Appendix F, Appendix H, Appendix J and Appendix I. We provide proofs for all the propositions in Appendix C, with appropriate assumptions.

heading: A RELATED WORK
text: Recently, several novel deep learning-based techniques to estimate uncertainty with a single model have been proposed. For example, Deep Evidential Regression Amini et al. (2020) is a method for estimating epistemic uncertainty that is based on a parametric estimate of model variance. This is in line with previous work using evidential uncertainty estimation (Malinin & Gales, 2018; Sensoy et al., 2018). Orthonormal Certificates, a set of learned features with a suitable loss function, are used in Tagasovska & Lopez-Paz (2019). These certificates capture the distance to the training set to learn an estimate of the epistemic uncertainty. This is further studied in Liu et al. (2020) who formalize distance awareness, which captures the model’s ability to quantify the distance of a test sample from the training data manifold, as a necessary condition for uncertainty estimation. This distance awareness can be captured with a weight normalization step in training, in addition to using a GP as the output layer. DUE (van Amersfoort et al., 2021) is an instance of Deep Kernel Learning (Wilson et al., 2016), which is defined as a GP with a deep feature extractor inside the kernel. DUE improves upon SNGP by using an inducing point GP, and bi-lipschitz constraints on the feature extractor, giving better test set accuracies as well as improved training efficiency. DUN (Antoran et al., 2020) uses the disagreement between the outputs from intermediate layers as a measure of uncertainty. DUQ (van Amersfoort et al., 2020) on the other hand uses two-sided Jacobian regularization on RBF networks (LeCun et al., 1998) for reliable uncertainty estimates.
Wen et al. (2020) present an efficient way of implementing ensembles of neural networks, by using one shared matrix and a rank-1 matrix per member. The weights for each member are then computed as the Hadamard product of the shared matrix and the rank-1 matrix of the member. There has also been extensive work in scaling up Bayesian Neural Networks for high-dimensional data to capture epistemic uncertainty. SWAG (Maddox et al., 2019) fits a Gaussian distribution capturing the SWA (Izmailov et al., 2018) mean and a covariance matrix representing the first two moments of SGD iterated. This distribution is then used as a posterior over the neural network weights. Dusenberry et al. (2020) parametrize the BNN with a distribution on a rank-1 subspace for each weight matrix, inspired by BatchEnsembles (Wen et al., 2020). Vadera et al. (2020b); Malinin et al. (2020b) propose approaches to improve the efficiency of ensembles by distilling the distribution of predictions rather than the average, thus preserving the information about the uncertainty captured by the ensemble.
There is also a large body of work on methods for approximating samples from the Bayesian posterior on large datasets with efficient MCMC based approaches. (Welling & Teh, 2011; Zhang et al., 2020; Vadera et al., 2020a). The variance of this posterior distribution can then be computed and used as an uncertainty estimate. However, as we have discussed in the sections above, this does not account for model misspecification and thus is not an accurate estimate for the lack of knowledge of the predictor.
Kull & Flach (2015) present several decompositions of the total expected loss, including the decomposition into the epistemic and irreducible (aleatoric) loss. They present additive adjustments that reduce the scoring rules like the log-loss and Brier score, but do not tackle the general problem of uncertainty estimation.
There are also interesting connections between the problem of out-of-distribution generalization arising in sequential model optimization and Bayesian optimization, discussed here, and the possibility of reweighing examples, see Farquhar et al. (2021).

heading: B EPISTEMIC UNCERTAINTY IN A GENERAL LOSS FUNCTION SETTING
text: We consider the setting presented in section 2, but with a general loss function l. We use the same notations as in section 2.
Definition B.1 The total uncertainty of f at x is defined as: U(f, x) = ∫ l(f(x), y)dP (y|x). (4)
Definition B.2 For a learning algorithmL which produces a distribution PL(f(x)|Dn) over possible solutions f(x) at x, the model variance at x is defined as
V (L,Dn, x) = ∫ l(f(x), f̂(x))dPL(f(x)|Dn)) (5)
with f̂(x) = arg minf̄(x) ∫ l(f(x), f̄(x))dPL(f(x)|Dn)). Note that for a loss function that is different from the square loss, the semantics of variance (such as its non-negativity) might be lost.
Let us consider the special cases of the negative log-likelihood loss in general (for outputs which may be discrete or continuous) and that of the squared error loss (which ends up being a special case of the former for normally distributed outputs). Below we see Q(Y |x) as a probability mass or density function (over y), which is also a function of x.
Definition B.3 The negative log-likelihood (NLL) loss takes as first argument Q(Y |x) a probability mass or density function and returns
lNLL(Q(Y |x), y) = − logQ(Y = y|x). (6)
Proposition 2 For the NLL loss with ground truth P (Y |x) and predictor Q(Y |x), the total uncertainty U(Q(Y |x), x) is a cross-entropy, i.e.,
U(Q(Y | . ), x) = CE(P (Y |x)||Q(Y |x)) = − ∫ dP (y|x) logQ(y|x) (7)
The proposition is shown by applying the definitions.
Proposition 3 For the NLL loss with ground truth P (Y |x), the aleatoric uncertainty A(x) in this setting is the entropy H[P (Y |x)] of the ground truth conditional:
A(x) = − ∫ dP (y|x) logP (y|x) = H[P (Y |x)], (8)
The proposition is shown from the cross-entropy CE(P (Y |x)||Q(Y |x)) being minimized when Q = P .
Proposition 4 For the NLL loss with ground truth P (Y |x) and predictor Q(Y |x), the epistemic uncertainty E(Q(Y |x), x) is the Kullback-Liebler divergence between P and Q (with P as the reference):
E(Q(Y | . ), x) = KL(P (Y |x)||Q(Y |x))
= ∫ dP (y|x) log P (y|x)
Q(y|x) (9)
The proposition is shown by combining the above two propositions and the definition of epistemic uncertainty.
To move towards the MSE loss, consider the special case of NLL with a conditionally Normal output density for both P and Q.
Proposition 5 For the NLL loss with a conditionally Normal output density for both P and Q, with respective means f∗(x) and f̂(x) and respective variances σ2P (x) and σ 2 Q(x), the epistemic uncertainty is
E(Q(Y | . ), x) = 1 2σ2Q(x) lMSE(f̂(x), f ∗(x))
+KL(P (Y |x)||Q̃(Y |x)), (10)
where Q̃( . |x) is obtained by shifting Q( . |x) towards P ( . |x) (i.e., Q̃( . |x) is Gaussian with mean f∗(x) and variance σ2Q(x) ), and the Bayes-optimal mean predictor is f
∗(x) = EP [Y |x]. Note that if σP = σQ, then the KL term is zero.
The proof is presented in Appendix C. We can compare with the MSE loss (which assumes a constant variance σ = σP = σQ) and obtain the same result up to variance scaling.

heading: C PROOFS
text: C.1 PROPOSITION 1
It is a well known result that, because f∗(x) is the mean of P (.|x), it is also the minimizer of ŷ 7→ ∫ (ŷ − y)2dP (y|x). f∗ is thus a Bayes Optimal predictor.
By definition of the total uncertainty: U(f, x) = ∫ (f(x)− y)2dP (y|x) = E[(f(x)− Y )2].
Hence, by definition of aleatoric uncertainty:
A(x) = U(f∗, x) = E[(f∗(x)− Y )2].
and by definition of epistemic uncertainty E(f, x) = E [ (f(x)− y)2 − (f∗(x)− y)2 ] = f(x)2 − f∗(x)2 − 2(f(x)− f∗(x))f∗(x) = (f(x)− f∗(x))2.
Which concludes the proof.
C.2 PROPOSITION 5
From Equation 9, we get:
E(Q(Y | . ), x) = KL(P (Y |x)||Q(Y |x))
= log σQ(x) σP (x) + σ2P (x) + (f(x)− f∗(x))2 2σ2Q(x) − 1 2
= 1
2σ2Q(x) lMSE(f(x), f
∗(x)) + log σQ(x) σP (x) + σ2P (x) + (f ∗(x)− f∗(x))2 2σ2Q(x) − 1 2
= 1
2σ2Q(x) lMSE(f(x), f
∗(x)) +KL(P (Y |x)|Q̃(Y |x))
Which concludes the proof

heading: D ESTIMATING ALEATORIC UNCERTAINTY WITH ACCESS TO AN ORACLE
text: In scenarios like active learning, one has access to an oracle from which we can obtain samples of Y ∼ P (Y |x) at any given point x. In that case, one can train an estimator a(x) of aleatoric uncertainty by obtaining K > 1 samples y1, . . . yK at the same x, for a set of representative x’s.
More formally, if we have multiple independent outcomes y1, . . . , yK ∼ P (Y |x) for each input point x, then training a predictor a with the squared loss on (input, target) examples( x, KK−1V ar(y1, . . . , yK) ) , where V ar denotes the empirical variance, yields an estimator of the aleatoric uncertainty.
Naturally, this estimator is asymptotically unbiased, if the learning algorithm ensures asymptotic convergence to a Bayes-Optimal predictor.
This is due to the fact that Ey1,...,yK [ K K−1V ar(y1, . . . , yK) ] = V arP (y|x)[y|x], which according to Proposition 1 is equal to A(x).

heading: E PSEUDO CODES
text: Algorithm 2 illustrates the training procedure when a held-out validation set is available. We focus on y ∈ R in this paper because it makes sense for active learning applied to black-box optimization (where we want to maximize it) but the algorithms can trivially be applied to the case where y ∈ Rd for any d. Similarly, the algorithms can be generalized to other losses besides the square loss, following the generalized theory presented above in Appendix B.
Algorithm 2 DEUP with a fixed training set: Training procedure to obtain estimates of epistemic uncertainty Data: D the training dataset with pairs (x, y) with x ∈ X , y ∈ R; Dout the out-of-sample dataset with pairs (x, y), to train the uncertainty estimator X , the input/search space a : X 7→ R, trained estimator of aleatoric uncertainty ( f : X 7→ R, main predictor, trained on D e : X 7→ R, total uncertainty estimator (estimates error of f ) Training: Initialize empty dataset of errors De Cross-validation step (Optional, if using other features for the error predictor): Similar to the corresponding step of Algorithm 1. for every pair (x, y) in D ∪Dout do De ← De ∪ {(x, (y − f(x))2)} end Fit e on De Evaluation: For every input x, return e(x)− a(x) as an estimator of epistemic uncertainty at x

heading: F REJECTING DIFFICULT EXAMPLES
text: We adapt the standard OOD rejection task (van Amersfoort et al., 2020; Liu et al., 2020) to measure the Spearman Rank Correlation of the predicted uncertainty with the true generalization error, in addition to the OOD Detection AUROC. We use MC-Dropout (Gal & Ghahramani, 2016), Deep Ensemble (Lakshminarayanan et al., 2017), DUE(van Amersfoort et al., 2021) and DUQ van Amersfoort et al. (2020) as the baselines 2. We use these baselines as representatives for the major approaches for uncertainty estimation in recent literature. For all the methods, including DEUP we consider two architectures for the main predictor, ResNet-18 and ResNet-50 (He et al., 2016) (Table 2) to study the effect of model capacity. Note that for the ResNet50 DEUP model we continue using the ResNet-18 based DUE as variance source.
Training The baselines were trained with the CIFAR-10 training set with 10% set aside as a validation set for hyperparameter tuning. The hyperparameters are presented in Table 3 and Table 4. The hyperparameters not specified are set to the default values. For DEUP, we consider the log-density, model-variance estimate and the seen-unseen bit as the features for the error predictor. The density
2MC-Dropout and Deep Ensemble baselines are based on https://github.com/google/uncertaintybaselines, DUQ based on https://github.com/y0ast/deterministic-uncertainty-quantification and DUE based on https://github.com/y0ast/DUE
estimator we use is Masked-Autoregressive Flows (Papamakarios et al., 2017) and the variance estimator used is DUE (van Amersfoort et al., 2021). Note that as indicated earlier x, the input image, is not used as a feature for the error predictor. We present those ablations in the next sub-section. For training DEUP, the CIFAR-10 training set is divided into 5 folds, with each fold containing 8 unique classes. For each fold, we train an instance of the main predictor, density estimator and model variance estimator on only the corresponding 8 classes. The remaining 2 classes act as the out-of-distribution examples for training the error predictor. Using these folds we construct a dataset for training the error predictor, a simple feed forward network. The error predictor is trained with the log targets (i.e. log MSE between predicted and observed error). This helps since the scale of the errors varies over multiple orders of magnitude. We then train the main predictor, density estimator and the variance estimator on the entire CIFAR-10 dataset, for evaluation. The hyperparameters are presented in Table 4. For all models, we train the main predictor for 75 and 125 epochs for ResNet-18 and ResNet-50 respectively. We use SGD with Momentum (set to 0.9), with a multi-step learning schedule with a decay of 0.2 at epochs [25, 50] and [45, 90] for ResNet-18 and ResNet-50 respectively. One complete training run for DEUP takes about 1.5-2 GPU days on a V100 GPU. In total these set of experiments took about 31 GPU days on a Nvidia V100 GPU.
Ablations We also perform some ablation experiments to study the effect of each feature for the error predictor. The Spearman rank correlation coefficient between the generalization error and the variance feature, V , from DUE van Amersfoort et al. (2021) alone is 37.84 ± 0.04, and the log-density, D, from MAF Papamakarios et al. (2017) alone is 30.52 ± 0.03. With only the image (x) the SRCC is 36.58± 0.16 Table 6 presents the results for these experiments. We observe that combining all the features performs the best. Also note that using the log-density and variance as features to the error predictor we observe better performance than using them directly, indicating that the error predictor perhaps captures
a better target for the epistemic uncertainty. The boolean feature (B) indicating seen examples, discussed in Section 3.2, also leads to noticeable improvments.
F.1 PREDICTING UNCERTAINTY UNDER DISTRIBUTION SHIFT
We also consider the task of uncertainty estimation in the setting of shifted distributions (Ovadia et al., 2019; Hendrycks & Dietterich, 2019). We evaluate the uncertainty predictions of models trained with CIFAR-10, on CIFAR-10-C (Hendrycks & Dietterich, 2019) which consists of images from CIFAR-10 distorted using 16 corruptions like gassian blur, impulse noise, among others. Figure 6 shows that even in the shifted distribution setting, the uncertainty estimates of DEUP correlate much better with the error made by the predictor, compared to the baselines.

heading: G DEUP IN THE PRESENCE OF ALEATORIC UNCERTAINTY
text: In the presence of aleatoric uncertainty, we have seen that DEUP’s error predictor e is an estimate of the total uncertainty, rather than the epistemic uncertainty. However, if we have access to to an estimator of aleatoric uncertainty a, then e− a becomes an estimator of epistemic uncertainty. To show the difference in behavior of DEUP when there is aleatoric uncertainty, we consider, a modified version of the experiment in Section 5.1.1, with a non-deterministic oracle (ground truth function). Because of the noisy training dataset, GP conflates epistemic and aleatoric uncertainty, which makes the gap between the predicted epistemic uncertainty (as measured by the GP variance) and the true epistemic uncertainty (as measured by the MSE between the GP mean and the noiseless ground truth function) higher than in the deterministic setting of Section 5.1.1.
Similarly, in order to train DEUP’s uncertainty estimator, more out-of-sample data is needed compared to the noiseless setting. In figure 7, we consider the setting described in Section D, and train a separate predictor on the targets y1, . . . , yk to estimate the aleatoric uncertainty (which boils down to the variance of the ground-truth function). Because these targets are themselves noisy, we chose a simple linear regressor as the estimator of aleatoric uncertainty, to avoid overfitting to the noise.
A key distinction of this setting, is that DEUP’s training data (the errors of the main predictor) are themselves noisy, which makes it important to use more out-of-sample data to obtain reasonable total uncertainty estimates (from which we subtract the estimates of the aleatoric uncertainty).

heading: H SEQUENTIAL MODEL OPTIMIZATION EXPERIMENTS
text: We use BoTorch3 (Balandat et al., 2020) as the base framework for our experiments.
For all our Sequential Optimization algorithms, we use Algorithm 1 to train DEUP uncertainty estimators. We found that the optional step of pre-filling the uncertainty estimator dataset De was important given the low number of available training points. We used half the initial training set (randomly chosen) as in-sample examples (used to train the main predictor and an extra-feature generator) and the other half as out-of-sample examples to provide instances of high epistemic
3https://botorch.org/
uncertainty to train an uncertainty predictor; we repeated the procedure by alternating the roles of the two halves of the dataset. We repeated the whole procedure twice using a new random split of the dataset, thus ending up with 4 training points in De for every initial training point in Dinit. The error predictor is trained with the log targets (i.e. log MSE between predicted and observed error). This helps since the scale of the errors varies over multiple orders of magnitude.
Computationally, the training time of DEUP-EI depends on various choices (e.g. the features used to train the epistemic uncertainty predictor, the dimension of the input, the learning algorithms, etc..). Additionally, the training time for the uncertainty predictor varies at each step of the optimization. In total, the sequential optimization experiments took about 1 CPU day.
H.1 ONE-DIMENSIONAL FUNCTION TOY EXAMPLE
In Figure 8, we show the results of DEUP-EI, compared to GP-EI, MCDropout-EI and Ensembles-EI, in the task of optimizing a synthetic one-dimensional function. Because MCDropout and Ensembles are trained on in-sample data only, they are unable to generalize their uncertainty estimates, which makes them bad candidates for Sequential Model Optimization, because they are easily stuck in local minima, and require many iterations before the acquisition function gives more weight to the predicted uncertainties than the current maximum.
For Random acquisition, we sampled for different seeds 56 points, and used the (average across the seeds of the) maximum of the first 6 values as the first value in the plots (Figures 4 and 8). Note that
because the function is specifically designed to have multiple local maxima, GP-EI also required more optimization steps, and actually performed worse than random acquistion
As a stationarizing input feature, we used the variance of a GP fit on the available data at every step. We found that the binary (in-sample/out-of-sample) feature and density estimates were redundant with the variance feature and didn’t improve the performance as captured by the number of additional function calls. We used a GP for the DEUP uncertainty estimator. Using a neural net provided similar results, but was computationally more expensive in this 1-D case with few datapoints. We used a 3-hidden layer neural network, with 128 neurons per layer and a ReLU activation function, with Adam (Kingma & Ba, 2015) and a learning rate of 10−3 (and default values for the other hyperparameters) to train the main predictor for DEUP-EI (in order to fit the available data). The same network architecture and learning rate were used for the Dropout and Ensemble baselines. We used 3 networks for the Ensemble baseline, and a dropout probability of 0.3 for the Dropout baseline, with 100 test-time forward passes to compute uncertainty estimates.
H.2 TWO-DIMENSIONAL FUNCTION
To showcase DEUP’s usefulness for Sequential Model Optimization in with a number of dimensions greater than 1, we consider the optimization of the Levi N.13 function, a known benchmark for optimization. The function f takes a point (x, y) in 2D space and returns:
f(x, y) = − ( sin2(3πx) + (x− 1)2(1 + sin2(3πy)) + (y − 1)2(1 + sin2(2πy)) )
We use the box [−10, 10]2 as the optimization domain. In this domain, the maximum of the function is 0, and it is reached at (1, 1). The function has multiple local maxima, as shown in Figure 9(a)4.
Similar to the previous one-dimensional function, MCDropout and Ensemble provided bad performances and are omitted from the plot in 9(b). We used the same setting and hyperparameters for DEUP as for the previous function. DEUP-EI is again the only method that reaches the global maximum consistently in under 56 function evaluations.
4Plot of the function copied from https://www.sfu.ca/ ssurjano/levy13.html
H.3 ADDITIONAL DETAILS FOR THE ACKLEY FUNCTION EXPERIMENT, FOR SYNTHETIC DATA IN HIGHER DIMENSIONS
The Ackley function of dimension d is defined as:
Ackleyd : B → R
x 7→ A exp −B √√√√1 d d∑ i=1 x2i + exp(1 d d∑ i=1 cos(cxi) ) −A− exp(1)
where B is a hyperrectangle of Rd. (0, . . . , 0) is the only global optimizer of Ackleyd, at which the function is equal to 0. We use BoTorch’s default values for A,B, c, which are 20, 0.2, 2π respectively.
In our experiments, we used B = [−10, 15]d for all dimensions d. For the TurBO baseline, we use BoTorch’s default implementation, with Expected Improvement as an acquisition function, and a batch size of 1 (i.e. acquiring one point per step).
For fair comparisons, for DEUP, we use a Gaussian Process as the main model, and use its variance as the only input of the epistemic uncertainty predictor. This means that we calibrate the GP variance to match the out-of-sample squared error, using another GP to perform the regression. TurBO-DEUP is a combination of both, in which we perform the variance calibration task for the local GP models of TurBO. The uncertainty predictor, i.e. the GP regressor, is trained with log targets, as in Appendix H.1, but also with log variances as inputs.
Note that only the stationarizing feature is used as input for the uncertainty predictor. When we used the input x as well, we found that the GP error predictor overfits on the x part of the input (x, v), and it was detrimental to the final performances. For all experiments, we used 20 initial points.

heading: I REINFORCEMENT LEARNING EXPERIMENTS
text: For RL experiments, we used bsuite (Osband et al., 2020), a collection of carefully designed RL environments. bsuite also comes with a list of metrics which aim to evaluate RL agents from different aspects. We compare the agents based on the basic metric and average regret as they capture both sample complexity and final performance. The default DQN agent is used as the base of our experiments with a 3 layer fully-connected (FC) neural network as its Q-network. For the Bootstrapped DQN baseline, we used the default implementation provided by bsuite. To implement DQN + MC-Dropout, following the implementation from Gal & Ghahramani (2016), two dropout layers with dropout probability of 0.1 are used before the second and the third FC layers. In order take an action, the agent performs a single stochastic forward pass through the Q-network, which is equivalent to taking a sample from the posterior over the Q-values, as done in Thompson sampling, an alternative to −greedy exploration. As a density estimator, we used a Kernel Density Estimator (KDE) with a Gaussian kernel and bandwidth of 1 to map states to densities. This KDE is fit after each 10000 steps (actions) with a batch of samples from the replay buffer (which is of size 10000). The uncertainty estimator network (E-network) has the same number of layers as the Q-network, with an additional Softplus layer at the end. All other hyperparameters are the same as the default implementation by Osband et al. (2020). One complete training run for the DEUP-DQN with 5 seeds experiments takes about 0.04-0.05 GPU days on a V100 GPU. In total RL experiments took about 0.15 GPU days on a Nvidia V100 GPU.
Algorithm 3 DEUP-DQN Initialize replay buffer D with capacity N Qθ(s, a): state-action value predictor Eφ(log d): uncertainty estimator network, which takes the log-density of the states as the input d(s): Kernel density estimator (KDE) K: KDE fitting frequency W: Number of warm-up episodes for episode=1 to M do
set s0 as the initial state for t=1 to max-steps-per-episode do
with probability : take a random action, otherwise: if episode ≤W: a = maxaQθ(st, a), else: a = maxa [ Qθ(st, a) + κ× Eφ(log d(st))(a) ] store (st, at, rt, st+1) in D Sample random minibatch B of transitions (sj , aj , rj , sj+1) from D if sj is a final state: yj = rj , else: yj = rj + γmaxaQ(st, a) Update Q-network: θ ← θ + αQ.∇θ E(s,a)∼B [( yj −Qθ(s, a)
)2] Update E-network:
φ← φ+ αE .∇φ E(s,a)∼B [[( yj −Qθ(s, a) )2 − Eφ(log d(st))(a)]2] if mod(total-steps, K) = 0: fit the KDE d on the states of D
end end

heading: J DRUG COMBINATION EXPERIMENTS
text: To validate DEUP’s uncertainty estimates in a real-world setting, we measured its performance on a regression task predicting the synergy of drug combinations. While much effort in drug discovery is spent on finding novel small molecules, a potentially cheaper method is identifying combinations of pre-existing drugs which are synergistic (i.e., work well together). Indeed, drug combinations are the current standard-of-care for a number of diseases including HIV, tuberculosis, and some cancers Cihlar & Fordyce (2016); Organization & Initiative (2010); Mokhtari et al. (2017).
However, due to the combinatorial nature of drug combinations, identifying pairs exhibiting synergism is challenging. Compounding this problem is the high monetary cost of running experiments on promising drug combinations, as well as the length of time the experiments take to complete. Uncertainty models could be used by practitioners downstream to help accelerate drug combination treatment discoveries and reduce involved development costs.
To test DEUP’s performance on this task we used the DrugComb and LINCS L1000 datasets Zagidullin et al. (2019); Subramanian et al. (2017). DrugComb is a dataset consisting of pairwise combinations of anti-cancer compounds tested on various cancer cell lines. For each combination, the dataset provides access to several synergy scores, each indicating whether the two drugs have a synergistic or antagonistic effect on cancerous cell death. LINCS L1000 contains differential gene expression profiles for various cell lines and drugs. Differential gene expressions measure the difference in the amount of mRNA related to a set of influential genes before and after the application of a drug. Because of this, gene expressions are a powerful indicator of the effect of a single drug at the cellular level.
In our experiments, each drug is represented by its Morgan fingerprint Morgan (1965)5 (with 1,024 bits and a radius of 3) as well as two differential gene expression profiles (each of dimension 978) from two cell lines (PC-3 and MCF-7). In order to use gene expression features for every drug, we only used drug pairs in DrugComb where both drugs had differential gene expression data for cell lines PC-3 and MCF-7.
5The Morgan fingerprint represents a molecule by associating with it a boolean vector specifying its chemical structure. Morgan fingerprints have been used as a signal of various molecular characteristics to great success Ballester & Mitchell (2010); Zhang et al. (2006).
We first compared the quality of DEUP’s uncertainty estimations to other uncertainty estimation methods on the task of predicting the combination sensitivity score Malyutina et al. (2019) for drug pairs tested on the cell line PC-3 (1,385 examples). We evaluated the uncertainty methods using a train, validation, test split of 40%, 30%, and 30%, respectively. The underlying model used by each uncertainty estimation method consisted of a single drug fully connected neural network (2 layers with 2048 hidden units and output of dimension 1024) and a combined drug fully connected neural network (2 layers, with 128 hidden units). The embeddings of an input drug pair’s drugs produced by the single drug network are summed and passed to the combined drug network, which then predicts final synergy. By summing the embeddings produced by the single drug network, we ensure that the model is invariant to permutations in order of the two drugs in the pair. The models were trained with Adam Kingma & Ba (2015), using a learning rate of 1e-4 and weight decay of 1e-5. For MC-Dropout we used a dropout probability of 0.1 on the two layers of the combined drug network and 3 test-time forward passes to compute uncertainty estimates. The ensemble used 3 constituent models for its uncertainty estimates. Both Ensemble and MC-Dropout models were trained with the MSE loss.
We also compared against DUE (van Amersfoort et al., 2021) which combines a neural network feature extractor with an approximate Gaussian process. Spectral normalization was added to all the layers of the combined drug network and of the single drug network. Let demb denote the dimension of the output of the combined drug network, which is also the input dimension of the approximate Gaussian process. We conducted a grid-search over different values of demb (from 2 to 100), the number of inducing points (from 3 to 200), the learning rate, and the kernel used by the Gaussian process. The highest correlation of uncertainty estimates with residuals was attained with demb = 10, 100 inducing points, a learning rate of 1e-2, and the Matern12 kernel.
Algorithm 4 DEUP for Drug Combinations Data: D dataset of pairwise drug combinations, along with synergy scores ((d1, d2), y) Initialization: Split training set into two halves, in-sample Din and out-of-sample Dout fµ(d1, d2): µ̂ predictor which takes a pair of drugs as input f inσ (d1, d2): In-sample σ̂in error predictor foutσ (d1, d2): Out-of-sample σ̂out error predictor Training: while training not finished do
In-sample update Get an in-sample batch (d1,in, d2,in, yin) ∼ Din Predict µ̂ = fµ(d1,in, d2,in) and in-sample error σ̂in = f inσ (d1,in, d2,in) Compute NLL: log(σ̂ 2 in)
2 + (µ̂−yin)2
2σ̂2in
Backpropagate through fµ and f inσ and update. Out-of-sample update Get an out-of-sample batch (d1,out, d2,out, yout) ∼ Dout Estimate µ̂ = fµ(d1,out, d2,out) and out-of-sample error σ̂out = foutσ (d1,out, d2,out) Compute NLL: log(σ̂ 2 out)
2 + (µ̂−yout)2
2σ̂2out
Backpropagate through foutσ and update. end
The DEUP model we used outputs two heads [ µ̂ σ̂ ] and is trained with the NLL log(σ̂ 2) 2 + (µ̂−y)2 2σ̂2 in a similar fashion as in Lakshminarayanan et al. (2017). To obtain a predictor of the out-of-sample error, we altered our optimization procedure so that the µ and σ heads were not backpropagated through at all times. Specifically, we first split the training set into two halves, terming the former the in-sample set Din and the latter the out-of-sample set Dout. We denote as f inσ the in-sample error predictor and foutσ the out-of-sample error predictor. f out σ is used to estimate total uncertainty. Note that in this setting, foutσ predicts the square root of the epistemic uncertainty (σ̂out) rather than the epistemic uncertainty itself (σ̂2out).
In our experiments, an extra bit is added as input to the model in order to indicate whether a given batch is from Din or Dout. Through this, the same model is used to estimate f inσ and foutσ with the model estimating f inσ when the bit indicates an example is drawn fromDin and foutσ otherwise. When
the batch is drawn from Din, both heads are trained using NLL using a single forward pass. However, when the data is drawn from Dout only the σ̂ head is trained. To do this, we must still predict µ̂ in order to compute the NLL. But the µ̂ predictor fµ must be agnostic to the difference between Din and Dout. To solve this, we perform two separate forward passes. The first pass computes µ̂ and sets the indicator bit to 0 so fµ has no notion of Dout, while the second pass computes σ̂, setting the bit to 1 to indicate the true source of the batch. Finally, we backpropagate through the σ̂ head only. The training procedure is described in Algorithm 4
We report several measures for the quality of uncertainty predictions on a separate test set in Table 7.
For each model, we report the per sample predictive log-likelihood, coverage probability and confidence interval width, averaged over 3 seeds.
We also computed the correlation between the residuals of the model |µ̂(xi)− yi| and the predicted uncertainties σ̂(xi). We noted that the different uncertainty estimation methods lead to different distributions p(σ̂(x)). For example, predicted uncertainties obtained with DUE always have a similar magnitude. By contrast, DEUP yields a wide range of different predicted uncertainties.
These differences between the distributions p(σ̂(x)) obtained with the different methods may have an impact on the correlation metric, possibly biasing the comparison of the different methods. In order to account for differences in the distribution p(σ̂(x)) across methods, we report another metric which is the ratio between the observed correlation Corr(|µ̂(x)− y|, σ̂(x)) and the maximum achievable correlation given a specific distribution p(σ̂(x)).
This maximum achievable correlation (refered to as the upper bound) is not per se a comparison metric, and is estimated (given a specific p(σ̂(x))) as follows: we assume that, for each example (xi, yi), the predictive distribution of the model N (µ̂(xi), σ̂(xi)) corresponds exactly to the distribution of the target, i.e. yi ∼ N (µ̂(xi), σ̂(xi)). Under this assumption, the residual of the mean predictor follows a distributionN (0, σ̂(xi)). We can then estimate the upper bound by computing the correlation between the predicted uncertainties σ̂(xi) and samples from the corresponding Gaussians N (0, σ̂(xi)). 5 samples were drawn from each Gaussian for our evaluation. This upper bound is reported in the Table.
Finally, we reported our comparison metric: the ratio between the correlation Corr(|µ̂(x)− y|, σ̂(x)) and the upper bound. The higher the ratio is, the closer the observed correlation is to the estimated upper bound and the better the method is doing.
It is interesting to note that the upper bound is much lower for DUE compared to other methods, as its predicted uncertainties lie within a short range of values.
Predicted µ̂ and uncertainty estimates can be visualized in Figure 10 for different models. MC-dropout, Ensemble and DUE consistently underestimate uncertainty, while the out-of-sample uncertainties predicted by DEUP are much more consistent with the order of magnitude of the residuals. Moreover, we observed that DUE predicted very similar uncertainties for all samples, resulting in a lower upper-bound for the correlation between residuals and predicted uncertainties compared to other methods. We observed a similar pattern when experimenting with the other kernels available in the DUE package, including the standard Gaussian kernel.
Finally, we note that in the context of drug combination experiments, aleatoric uncertainty could be estimated by having access to replicates of a given experiment (c.f. Section D), allowing us to subtract the aleatoric part from the out-of-sample uncertainty, leaving us with the epistemic uncertainty only.
One complete training run for the drug combination experiments takes about 0.01 GPU days on a V100 GPU. In total these set of experiments took about 0.2 GPU days on a Nvidia V100 GPU.

Review summaries:
This paper uses out-of-sample prediction error as a measurement of epistemic uncertainty. Using this definition, it develops an estimator: direct epistemic uncertainty prediction.  The idea is to have a main predictor to learn the task, and an error predictor to predict the generalization error. Empirical studies show that their proposed estimator produces better estimation on downstream tasks such as sequential model optimization and reinforcement learning. 

Given some supervised task, this paper redefines the uncertainty of a solution as its generalisation risk. The authors then propose to learn a secondary function to estimate the generalisation risk of the first. This is done by using held-out points as training data for the secondary model. In turn, this held out used to estimate the primary model error comes from a k-fold split.  The inputs to the secondary model are the data points being evaluated, estimates of the predictive variance of the primary model, density estimates from a generative model and whether the point has been observed by the primary model.  The authors posit that the advantage of their method is that it can capture uncertainty due to model selection bias, something omitted by existing work, which focuses on the variance of the learnt estimator. The authors provide a diverse range of experiments: OOD rejection in image classification, active learning for drug discovery, function optimisation and exploration in reinforcement learning. 

This paper proposes a new approach for computing epistemic uncertainty. The proposed approach, DEUP, builds a new model (in addition to the original model) which predicts epistemic uncertainty, defined as generalization error minus aleatory uncertainty. I highlight some features of DEUP below: - DEUP works with a hold-out dataset that is used for training the error predictor.  - In case there does not exist a hold-out set or in interactive settings (like RL or active learning), DEUP is extended to be used in a cross-validation setting and the features used to fit the error predictor is extended to include data density estimates and model variance. DEUP is evaluated on different settings including OOD data, sequential model optimization and RL.

This paper proposes a method to estimate the epistemic uncertainty (uncertainty due to lack of knowledge/data) at a new model input. The paper takes an indirect approach towards this goal by a) first estimate the generalization error at the new input, b) next estimate the aleatoric error (inherent uncertainty in the data distribution / irreducible error), and c) subtract aleatoric error estimate from the total generalization error estimate to obtain the epistemic uncertainty estimate.  The authors claim that this method captures both the uncertainty due to lack of data, as well as model misspecification in the process - while other/existing methods focus mostly on the variance of the posterior distribution (or its approximation) as a measure of epistemic uncertainty (and thereby implicitly assume the model is well specified.)  Estimating the total generalization error itself is performed with a second neural network model, which uses residuals obtained from the primary model as its labels. Estimating aleatoric error either assumes the presence of an Oracle or   The authors apply their technique on both static (fixed data set) as well as interactive (active learning) settings, though mostly focused on the mean squared error loss function.