Quantification of Uncertainty with Adversarial Models
Authors: Kajetan Schweighofer
Lukas Aichberger
Mykyta Ielanskyi
Günter Klambauer
Sepp Hochreiter
Venue: NeurIPS 2023 poster (2023)
Keywords: uncertainty
uncertainty quantification
predictive uncertainty
epistemic uncertainty
out of distribution
mc dropout
deep ensembles
sg-mcmc
adversarial model
adversarial model search
imagenet
TL;DR: We introduce QUAM to quantify predictive uncertainty with adversarial models (not adversarial examples!). Adversarial models identify important posterior modes that are missed by current uncertainty methods.

Abstract: Quantifying uncertainty is important for actionable predictions in real-world applications. A crucial part of predictive uncertainty quantification is the estimation of epistemic uncertainty, which is defined as an integral of the product between a divergence function and the posterior. Current methods such as Deep Ensembles or MC dropout underperform at estimating the epistemic uncertainty, since they primarily consider the posterior when sampling models. We suggest Quantification of Uncertainty with Adversarial Models (QUAM) to better estimate the epistemic uncertainty. QUAM identifies regions where the whole product under the integral is large, not just the posterior. Consequently, QUAM has lower approximation error of the epistemic uncertainty compared to previous methods. Models for which the product is large correspond to adversarial models (not adversarial examples!). Adversarial models have both a high posterior as well as a high divergence between their predictions and that of a reference model. Our experiments show that QUAM excels in capturing epistemic uncertainty for deep learning models and outperforms previous methods on challenging tasks in the vision domain.

Quantification of Uncertainty with Adversarial Models
Kajetan Schweighofer' Lukas Aichberger" Mykyta Ielanskyi' Günter Klambauer Sepp Hochreiter
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning,
Johannes Kepler University Linz, Austria Joint first authors
Abstract
Quantifying uncertainty is important for actionable predictions real-world ap- plications. A crucial part of predictive uncertainty quantification is the estimation of epistemic uncertainty, which defined as an integral of the product between a divergence function and the posterior. Current methods such Deep Ensembles or MC dropout underperform at estimating the epistemic uncertainty, since they primarily consider the posterior when sampling models. We suggest Quantification of Uncertainty with Adversarial Models (QUAM) to better estimate the epistemic uncertainty. QUAM identifies regions where the whole product under the integral is large, not just the posterior. Consequently, QUAM has lower approximation error of the epistemic uncertainty compared previous methods. Models for which the product is large correspond to adversarial models (not adversarial examples!), Adversarial models have both a high posterior as well a high divergence between their predictions and that of reference model. Our experiments show that QUAM excels in capturing epistemic uncertainty for deep learning models and outperforms previous methods on challenging tasks in the vision domain.
Introduction
Actionable predictions typically require risk assessment based on predictive uncertainty quantification [Apostolakis, 1991]. This is of utmost importance in high stake applications, such medical diagnosis or drug discovery, where human lives or extensive investments are at risk. such settings, even single prediction has far-reaching real-world impact, thus necessitating the most precise quantification of the associated uncertainties. Furthermore, foundation models or specialized models that are obtained externally are becoming increasingly prevalent, also in high stake applications. It crucial to assess the robustness and reliability of those unknown models before applying them. Therefore, the predictive uncertainty of given, pre-selected models at specific test points should be quantified, which we address in this work.
We consider predictive uncertainty quantification (see Fig. 1) for deep neural networks [Gal, 2016, Hüllermeier and Waegeman, 2021]. According to Vesely and Rasmuson [1984], Apostolakis [1991 I]. Helton [1993], McKone [1994], Helton [1997]. predictive uncertainty can be categorized into two types. First, aleatoric (Type A, variability, stochastic, true, irreducible) uncertainty refers to the variability when drawing samples or when repeating the same experiment. Second, epistemic (Type B, lack knowledge, subjective, reducible) uncertainty refers to the lack of knowledge about the true model. Epistemic uncertainty result from imprecision in parameter estimates, incompleteness in modeling, or indefiniteness in the applicability of the model. While aleatoric uncertainty cannot be reduced, epistemic uncertainty about the problem. We follow Helton [1997] and consider epistemic uncertainty as the imprecision variability of parameters that determine the predictive distribution. Vesely and Rasmuson [1984]
be reduced by more data, better models, more knowledge
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Figure 1: Adversarial models. For the red test point, the predictive uncertainty is high as it is far from the training data. High uncertainties are detected by different adversarial models that assign the red test point to different classes, although all of them explain the training data equally well. As result, the true class of the test point remains ambiguous.
calls this epistemic uncertainty "parameter uncertainty", which results from an imperfect learning algorithm or from insufficiently many training samples. Consequently, we consider predictive uncertainty quantification as characterizing a probabilistic model of the world. In this context, aleatoric uncertainty refers the inherent stochasticity of sampling outcomes from the predictive distribution of the model and epistemic uncertainty refers to the uncertainty about model parameters.
Current uncertainty guantification methods such Deep Ensembles [Lakshminarayanan al., 2017] or Monte-Carre (MC) dropout [Gal and Ghahramani, 2016] underperform at estimating the epistemic uncertainty [Wilson and Izmailov, 2020, Parker-Holder et al., 2020, Angelo and Fortuin, 2021], since they primarily consider the posterior when sampling models. Thus they are prone to miss important posterior modes, where the whole integrand of the integral defining the epistemic uncertainty is large. We introduce Quantification of Uncertainty with Adversarial Models (QUAM) to identify those posterior modes. QUAM searches for those posterior modes via adversarial models and uses them to reduce the approximation етrоr when estimating the integral that defines the epistemic uncertainty.
Adversarial models are characterized by large value of the integrand of the integral defining the epistemic uncertainty. Thus, they considerably differ to the reference model's prediction at a test point while having similarly high posterior probability. Consequently, they counterexamples of the reference model that predict differently for a new input, but explain the training data equally well. Fig. shows examples of adversarial models which assign different classes to test point, but agree on the training data. A formal definition is given by Def. 1. It is essential to note that concept that is to be distinguished from other concepts that include the term 'adversarial' in their naming, such as adversarial examples [Szegedy et al., 2013, Biggio et al., 2013], adversarial training [Goodfellow et al., 2015], generative adversarial networks [Goodfellow et al., 2014] or adversarial model-based RL [Rigter et al., 2022].
adversarial models are
Our main contributions are:
We introduce QUAM as a framework for uncertainty quantification. QUAM approximates the integral that defines the epistemic uncertainty substantially better than previous methods, since it reduces the approximation error of the integral estimator.
We introduce the concept of adversarial models for estimating posterior integrals with non-negath nog-negttive integrands. For given test point, adversarial models have considerably different predictions than reference model while having similarly high posterior probability.
We introduce a new setting for uncertainty quantification, where the uncertainty of a given, pre- selected model is quantified.
2 Current Methods to Estimate the Epistemic Uncertainty
Definition of Predictive Uncertainty. Predictive uncertainty quantification is about describing probabilistic model of the world, where aleatoric uncertainty refers to the inherent stochasticity of sampling outcomes from the predictive distribution of the model and epistemic uncertainty refers to the uncertainty about model parameters. We consider two distinct settings of predictive uncertainty quantification. Setting (a) concerns with the predictive uncertainty all plausible models given the training dataset [Gal, 2016, Hüllermeier and Waegeman, 2021]. This definition of uncertainty comprises how differently possible models predict (epistemic) and how
new test point expected under

---

confident each model is about its prediction (aleatoric), Setting (b) concerns with the predictive uncertainty a new test point for given, pre-selected model. This definition of uncertainty comprises how likely this model is the true model that generated the training dataset (epistemic) [Apostolakis, 1991, Helton, 1997] and how confident this model is about its prediction (aleatoric).
As an example, assume we have initial data from an epidemic, but we do not know the exact infection rate, which is a parameter of prediction model. The goal is to predict the number of infected persons at a specific time in the future, where each point in time is test point. In setting (a), we are interested in the uncertainty of test point predictions of all models using infection rates that explain the initial data. If all likely models agree for given new test point, the prediction of any of those models can be trusted, otherwise we can not trust the prediction regardless of which model selected in the end. In setting (b), we have selected a specific infection rate from the initial data as parameter for our model to make predictions. We refer this model as the given, pre-selected model. However, we do not know the true infection rate of the epidemic. All models with infection rates that are consistent with the initial data are likely to be the true model. If all likely models agree with the given, pre-selected model for given new test point, the prediction of the model can be trusted.
2.1 Measuring Predictive Uncertainty
We consider the predictive distribution of a single model pyy x, w), which is a probabílistic model of the world. Depending on the task. the predictive distribution of this probabilistic model can be a categorical distribution for classification or Gaussian distribution for regression. The Bayesian framework offers principled way to treat the uncertainty about the parameters through the posterior p( w (D) x P(D w)) w))(w) for given dataset D. The Bayesian model average (BMA) predictive distribution is given by p(y x; D) = =Iw
"wy p(y x; u ( D)dw. Following Gal [2016], Depeweg et al. [2018], Smith and Gal [2018), Hüllermeier and Waegeman [2021], the uncertainty of the BMA predictive distribution is commonly measured by the entropy H H[p(y xz D)] It refers to the total uncertainty, which can be decomposed into an aleatoric and an epistemic part. The BMA predictive entropy is equal to the posterior expectation of the cross- coss-entropy CE(- between the predictive distribution of candidate models and the BMA, which corresponds to setting (a). In setting (b), the cross-entropy is between the predictive distribution of the given, pre-selected model and candidate models. Details about the entropy and cross-entropy as measures of uncertainty are given in Sec. B.1.1 in the appendix. In the following, we formalize how measure the notions of uncertainty in setting (a) and (b) using the expected cross-entropy over the posterior.
Setting (a): Expected uncertainty when selecting model. We estimate the predictive uncertainty at test point æ when selecting a model in given a training dataset D. The total uncertainty is the expected cross-entropy between the predictive distribution of candidate models p(y æ. i) and the BMA predictive distribution p(y æ, D), where the expectation is with respect to the posterior:
/, CElp(y ae, u)) ply a. D)) p(
D) dw = H'ply D)]
Hip(y x,, ]6 p D) dw
+ I[Y; W D]
/ ww
Hppyy x. i p( D) du +
DRL(
x, w)) | x. D)) p(u
| (D) dw
ww
alcatoric
epistemic
The aleatoric uncertainty characterizes the uncertainty due the expected stochasticity of sampling outcomes from the predictive distribution of candidate models p(y x. w). The epistemic uncertainty characterizes the uncertainty due the mismatch between the predictive distribution of candidate models and the BMA predictive distribution. It is measured by the mutual information I[: -]. between the prediction Y and the model parameters W for a given test point and dataset, which is equivalent to the posterior expectation of the KL KL-divergence DRL. (- |I.) between the predictive distributions of candidate models and the BMA predictive distribution. Derivations are given in appendix Sec. B.
Setting (b): Uncertainty of given, pre-selected model. We estimate the predictive uncertainty of a given, pre-selected model w at test point ær We assume that the dataset D is produced according to the true distribution p(y x, w parameterized by w". The posterior p(t D) is an estimate of
3

---

how likely to match w". For epistemic uncertainty, we should measure the difference between the predictive distributions under w and w", but unknown. Therefore, we measure the expected difference between the predictive distributions under w and ır. In accordance with Apostolakis [1991] and Helton [1997], the total uncertainty is therefore the expected cross-entropy between the predictive distributions of given, pre-selected model w and candidate models in, any of which could be the true model wt according to the posterior:
J.m CElp(y x, w) p(y æx (r, p(w (D) du
H[p(y w)] DKL(p(y w) ply x, ()) p(aō D) dw
aleatoric
epistemic
The alcatoric uncertainty characterizes the uncertainty due to the stochasticity of sampling outcomes from the predictive distribution of the given, pre-selected model p(y æ, w). The epistemic uncertainty characterizes the uncertainty due to the mismatch between the predictive distribution of the given, pre-selected model and the predictive distribution of candidate models that could be the true model. Derivations and further details are given in appendix Sec. B.1.
2.2 Estimating the Integral for Epistemic Uncertainty
Current methods for predictive uncertainty quantification suffer from underestimating the epistemic uncertainty [Wilson and Izmailov, 2020, Parker-Holder et al., 2020, Angelo and Fortuin, 2021]. The epistemic uncertainty given by the respective terms in Eq. (1) for setting (a) and Eq. (2) for our new setting (b). To estimate these integrals, almost all methods use gradient descent the training data. Thus, posterior modes that are hidden from the gradient flow remain undiscovered and the epistemic uncertainty is underestimated [Shah al., 2020, Angelo and Fortuin, 2021]. An illustrative example is depicted in Fig. 2. Posterior expectations as Eq. (1) and Eq. (2) that define the epistemic uncertainty generally approximated using Monte Carlo integration. A good approximation of posterior integrals through Monte Carlo integration requires to capture all large values of the non- negative integrand [Wilson and Izmailov, 2020], which not only large values of the posterior, but also large values of the Kl.-divergence.
Variational inference [Graves, 2011 Blundell et al., 2015, Gal and Ghahramani, 2016] and ensemble methods [Lakshminarayanan et al., 2017] estimate the posterior integral based on models with high posterior. Posterior modes may be hidden from gradient descent based techniques they only discover mechanistically similar models. Two models are mechanistically similar they rely on the same input attributes for making their predictions, that is, they invariant to the same input attributes [Lubana et al., 2022]. However, gradient descent will always start by extracting input attributes that are highly correlated to the target as they determine the steepest descent in the error
Deep Ensembles
(b) MC dropout
Training data test point
Figure 2: Model prediction analysis. Softmax outputs (black) of individual models of Deep Ensembles (a) and MC dropout (b), as well as their average output (red) on probability simplex. Models were selected on the training data, and evaluated on the new test point (red) depicted in (c). The background color denotes the maximum likelihood of the training data that achievable by model having predictive distribution (softmax values) equal to the respective location on the simplex. Deep Ensembles and MC dropout fail find models predicting the orange class, although there would be likely models that do so. Details on the experimental setup are given in the appendix, Sec. C.2.

---

landscape. These input attributes create large basin in the error landscape into which the parameter vector is drawn via gradient descent. Consequently, other modes further away from such basins are almost never found [Shah et al., 2020, Angelo and Fortuin, 2021]. Thus, the epistemic uncertainty is underestimated. Another reason that posterior modes may hidden from gradient descent is the presence of different labeling hypotheses. If there is more than one way to explain the training data, gradient descent will use all of them as they give the steepest error descent [Scimeca et al., 2022].
Other work focuses on MCMC sampling according to the posterior distribution, which is approxi- mated by stochastic gradient variants [Welling and Teh, 2011, Chen et al., 2014] for large datasets and models. Those are known to face issues to efficiently explore the highly complex and multimodal parameter space and escape local posterior modes. There are attempts to alleviate the problem [Li et al., 2016, Zhang et al., 2020]. However, those methods do not explicitly look for important posterior modes, where the predictive distributions of sampled models contribute strongly to the approximation of the posterior integral, and thus have large values for the KI.-divergence.
3 Adversarial Models to Estimate the Epistemic Uncertainty
The epistemic uncertainty in Eq. (1) for setting (a) compares possible models with the BMA. Thus, the BMA is used reference model. The epistemic uncertainty in Eq. (2) for our new setting (b) compares models that are candidates for the true model with the given, pre-selected model. Thus, the given, pre-selected model is used as reference model. If the reference model makes some prediction at the test point, and if other models (the adversaries) make different predictions while explaining the training data equally well, then one should be uncertain about the prediction. Adversarial models are plausible outcomes of model selection, while having a different prediction at the test data point than the reference model. In court, the same principle is used: if the prosecutor presents a scenario but the advocate presents alternative equally plausible scenarios, the judges become uncertain about what happened and rule in favor of the defendant. We use adversarial models to identify locations where the integrand of the integral defining the epistemic uncertainty in Eq. (1) or Eq. (2) large. These locations are used to construct a mixture distribution that is used for mixture importance sampling to estimate the desired integrals. Using the mixture distribution for sampling, we aim to considerably reduce the approximation error of the estimator of the epistemic uncertainty.
Intuition.
Mixture Importance Sampling. We estimate the integrals of epistemic uncertainty in Eq. (1) and in Eq. (2). In the following, we focus on setting (b) with Eq. (2). but all results hold for setting (a) with Eq. (1) as well. Most methods sample from a distribution q(σ)) to approximate the integral:
w = / DKL(P(y | x, w) p(y x, ()) p(u D) deō = / u(x, w. w))
q( ④ dw
(3)
q(w)
where u(xx, w. 1 DKL (p(y x, w) ll p x, p | D). As with Deep Ensembles or MC dropout, posterior sampling is often approximated by sampling distribution q( (w) that is close to p(u D). Monte Carlo (MC) integration estimates v by
N
1
u(a a. w. wn)
wn ~ g(w)
(4)
N
n=1
If the posterior has different modes, the estimate under unimodal approximate distribution has high variance and converges very slowly [Steele et al., 2006]. Thus, we use mixture importance sampling (MIS) [Hesterberg, 1995]. MIS utilizes a mixture distribution instead of the unimodal distribution in standard importance sampling [Owen and Zhou, 2000]. Furthermore, many MIS methods iteratively enhance the sampling distribution by incorporating new modes [Raftery and Bao, 2010]. In contrast to the usually applied iterative enrichment methods which find new modes by chance, we have a much more favorable situation. We can explicitly search for posterior modes where the KL divergence is large, as we can cast it as supervised learning problem. Each of these modes determines the location of mixture component of the mixture distribution.
Theorem 1. The expected mean squared error of importance sampling with q can be bounded by
(
[u(fx ur ())
2 4
Egt,a) [B - u)?] < Ey((0)
(5)
q( ④
N
5

---

Proof. The inequality Eq. (5) follows from Theorem 1 in Akyildiz and Miguez [2021], when considering 0 & u(x w, w) as an unnormalized distribution and setting y 1.
Approximating only the posterior p( ( (D) as done by Deep Ensembles or MC dropout is insufficient to guarantee low expected mean squared error, since the sampling variance cannot be bounded (see appendix Sec. B.2).
Corollary 1. With constant e, Eg((n) [ v)²] < 4c2 holds if ff(x, w, w) < c c(
Consequently, ( must have modes where u(æ, w, w)) has modes even if the q-modes are factor c smaller. The modes of u(æ, w, t0) are models with both high posterior and high KL-divergence. We are searching for these modes to determine the locations Vy of the components of mixture distribution q( ()):
K
g( = Σ ak P(u: wkk 0)
(6)
k=1
with ag = 1/KK for K such models WH that determine a mode. Adversarial model search finds the locations tip of the mixture components, where the is adversarial model. The reference model does not define a mixture component, as it has zero KL-divergence to itself. We then sample from a distribution P at the local posterior mode with mean tiş and set of shape parameters O. The simplest choice for P is a Dirac delta distribution, but one could use e.g. local Laplace approximation of the posterior [MacKay, 1992], or a Gaussian distribution in some weight-subspace [Maddox al., 2019]. Furthermore, one could use Uhg as starting point for SG-MCMC chains [Welling and Teh, 2011, Chen al., 2014, Zhang al., 2020, 2022]. More details regarding MIS are given in the appendix in Sec. B.2. In the following, we propose an algorithm to find those models with both high posterior and high KL-divergence to the predictive distribution of the reference model.
Adversarial Model Search. Adversarial model search is the concept of searching for a model that has large distance / divergence to the reference predictive distribution and at the same time high posterior. We call such models "adversarial models" as they act as adversaries to the reference model by contradicting its prediction. A formal definition of an adversarial model is given by Def. 1:
Definition 1. Given are a new test data point x, a reference conditional probability model p'y x; w) from model class parameterized by w, divergence or distance measure D(-, :) for probability distributions, Y > O, A >> O, and a dataset D. Then a model with parameters w that satisfies the inequalities log (D) - logp(w [ D) < 7 and D(p(y x. (,,W), w)yp(y x, w)) 2 A is called an model.
Adversarial model search corresponds to the following optimization problem:
max D(p(y x, w)),,, p(y x, w + 6)) s.t. logp(w D) log p(w + 8 D < 7 SEA
(7)
We are searching for a weight perturbation that maximizes the distance D(., to the reference distribution without decreasing the log posterior more than 7. The search for adversarial models is restricted to 8 E A, for example by only optimizing the last layer of the reference model or by bounding the norm of 5. This optimization problem can be rewritten as:
max Dpp(y) x, w), p(y x, w + 8)) + c (logp(w + 8 I - logp(w | D) + 7) (8) SEA
where is hyperparameter. According to the Karush- Kuhn-Tucker (KKT) theorem [Karush, 1939, Kuhn and Tucker, 1950, May, 2020, Luenberger and Ye, 2016]: If 8* the solution to the problem Eq. (7), then there exists c* 2 0 with VoL(8" c²) = 0 (C is the Lagrangian) and c" (log p(w w (D) - logp(w + 8* D) - 7) = O. This is a necessary condition for an optimal point according to Theorem on Page 326 of Luenberger and Ye [2016].
We solve this optimization problem by the penalty method, which relies on the KKT theorem [Zangwill, 1967]. A penalty algorithm solves a series of unconstrained problems, solutions of which converge to the solution of the original constrained problem (see e.g. Fiacco and McCormick [1990]). The unconstrained problems are constructed by adding weighted penalty function measuring the constraint violation to the objective function. At every step, the weight of the penalty is increased, thus the constraints are less violated. If exists, the solution to the constraint optimization problem is an adversarial model that is located within posterior mode but has different predictive distribution compared to the reference model. We summarize the adversarial model search in Alg. 1.

---

Algorithm Adversarial Model Search (used in QUAM)
Supplies: Adversarial model ib with maximum Lady and L Len & 0
Requires: Test point æ, training dataset D = {(Ex. yk) ] reference model w, loss function I, loss of reference model the training dataset Lief -=== Up(y Ig, w), yk), minimization procedure MINIMIZE, number of penalty iterations M, initial penalty parameter cn. penalty parameter increase scheduler 1/, slack parameter y, distance / divergence measure D(:;"" :)
1: i +- un; ti <- w: 3- C0
2: for m t to M do
3: Lepnn < 1 >n
K k= Up(y Dg. w) yk) - (Lref + 7)
4: Ladv t- (pp 2 W) p(y x,, x,
5:
L f Ladv + L.pen
6:
MINIMIZE(L((E))
7:
if Ludy larger than all previous and Lpen < 0 then
8:
4 < (Tc)
9:
10:
return ti
Practical Implementation. Empirically, we found that directly executing the optimization pro- cedure defined in Alg. tends to result in adversarial models with similar predictive distribution for given input across multiple searches. The vanilla implementation of Alg. 1 corresponds to an untargeted attack, known from the literature on adversarial attacks [Szegedy et al., 2013, Biggio et al., single solution, we optimize the cross-entropy
2013]. To prevent the searches from converging loss for one specific class during each search, which corresponds to a targeted attack. Each resulting adversarial model represents a local optimum of Eq. (7). We execute as many adversarial model searches as there are classes, dedicating one search to each class, unless otherwise specified. To the predictive distributions p(y w) of all models w encountered during each penalty iteration of all scarches, weighted by their posterior probability. The posterior probabil- ity is approximated with the negative exponential training loss, the likelihood, of models w. This approximate posterior probability is scaled with a temperature parameter, set as a hyperparameter. Further details are given in the appendix Sec. C.I.
compute Eq. (4),
4 Experiments
In this section, we compare previous uncertainty quantification methods and our method QUAM in a set of experiments. First, we assess the considered methods on synthetic benchmark, on which it feasible to compute a ground truth epistemic uncertainty. Then, we conduct challenging out-of- distribution (OOD) detection, adversarial example detection, misclassification detection and selective prediction experiments in the vision domain. We compare (1) QUAM, (2) cyclical Stochastic Gradient Hamiltonian Monte Carlo (cSG-HMC) [Zhang et al., 2020], (3) an efficient Laplace approximation (Laplace) [Daxberger et al., 2021], (4) MC dropout (MCD) [Gal and Ghahramani, 2016] and (5) Deep Ensembles (DE) [Lakshminarayanan et al., 2017] their ability to estimate the epistemic uncertainty. Those baseline methods, especially Deep Ensembles, are persistently among the best performing uncertainty quantification methods across various benchmark tasks [Filos al., 2019, Ovadia et al., 2019, Caldeira and Nord, 2020, Band et al., 2022]
4.1 Epistemic Uncertainty on Synthetic Dataset
We evaluated all considered methods on the two-moons dataset, created using the implementation of Pedregosa al. [2011], To obtain the ground truth uncertainty, we utilized Hamiltonian Monte Carlo (HMC) [Neal, 1996]. HMC is regarded as the most precise algorithm to approximate posterior expectations [Izmailov et al., 2021], but necessitates extreme computational expenses to applied to models and datasets of practical scale. The results are depicted in Fig. 3. QUAM most closely matches the uncertainty estimate of the ground truth epistemic uncertainty obtained by HMC and excels especially on the regions further away from the decision boundary such as in the top left and bottom right of the plots. All other methods fail to capture the epistemic uncertainty those regions as gradient descent on the training set fails to capture posterior modes with alternative predictive distributions those parts and misses the important integral components. Experimental details and results for the epistemic uncertainty in Eq. (2) are given in the appendix Sec. C.3.
7

---

M
2
V
(a) Ground Truth HMC
(b) cSG-HMC
(c) Laplace
V
(d) MC dropout
(c) Deep Ensembles
(f) Our Method QUAM
Figure 3: Epistemic uncertainty as Eq. (1) for two-moons. Yellow denotes high epistemic
uncertainty. Purple denotes low epistemic uncertainty. HMC is considered ground truth [lzmailov et al., 2021] I and is most closely matched by QUAM. Artifacts for QUAM arise because it is applied to each test point individually, whereas other methods the same sampled models for all test points.
4.2 Epistemic Uncertainty on Vision Datasets
We benchmark the ability of different methods to estimate the epistemic uncertainty of a given, pre-selected model (setting (b) in Eq. (2)) in the context of (i) out-of-distribution (OOD) detection, (ii) adversarial example detection, (iii) misclassification detection and (iv) selective prediction. In all experiments, assume have access pre-trained model the in-distribution (ID) training dataset, which we refer to as reference model. The epistemic uncertainty is expected to be higher for OOD samples, as they can be assigned multiple ID classes, depending on the utilized features. Adversarial examples indicate that the model is misspecified on those inputs, thus we expect a higher epistemic uncertainty, the uncertainty about the model parameters. Furthermore, we expect higher epistemic uncertainty for misclassified samples than for correctly classified samples. Similarly, we expect the classifier to perform better on subset of more certain samples. This tested by evaluating the accuracy of the classifier on retained subsets of a certain fraction of samples with the lowest epistemic uncertainty [Filos al., 2019, Band al., 2022]. We report the AUROC for classifying the ID vs. OOD samples (i), the ID vs. the adversarial examples (ii), or the correctly classified vs. the misclassified samples (iii), using the epistemic uncertainty as score to distinguish the two classes respectively. For the selective prediction experiment (iv), report the AUC of the accuracy vs. fraction of retained samples, using the epistemic uncertainty to determine the retained subsets.
MNIST. We perform OOD detection the FMNIST [Xiao al., 2017]. KMNIST [Clanuwat et al., 2018], EMNIST [Cohen et al., 2017] and OMNIGLOT [Lake et al., 2015] test datasets as OOD datasets, using the LeNet [LeCun al., 1998] architecture. The test dataset of MNIST [LeCun et al., 1998] is used as ID dataset. We utilize the aleatoric uncertainty of the reference model (as in Eq. (2))
baseline assess the added value of estimating the epistemic uncertainty of the reference model. The results are listed in Tab. 1. QUAM outperforms all other methods on this task, with Deep Ensembles being the runner up method all dataset pairs. Furthermore, that only the epistemic uncertainties obtained by Deep Ensembles and QUAM are able to surpass the performance of using the alcatoric uncertainty of the reference model.
observed,
ImtgeNet-1K. We conduct OOD detection, adversarial example detection, misclassification detec- tion selective prediction experiments on ImageNet- K [Deng et al., 2009]. As OOD dataset, we use ImageNet-O) [Hendryeks et al., 2021], which is a challenging OOD dataset that was explicitly created be classified as an ID dataset with high confidence by conventional ImageNet- IK classifiers. Similarly, ImageNet-A [Hendrycks et al., 2021] is a dataset consisting of natural adversarial exam-

---

Table 1: MNIST results: AUROC using the epistemic uncertainty of a given, pre-selected model (as in Eq. (2)) as score to distinguish between ID (MNIST) and OOD samples. We also report the AUROC when using the aleatoric uncertainty of the reference model (Reference).
Dood FMNIST
Reference
cSG-HMC
Laplace
MCD
DE
QUAM
_8661000
.9771.004
.9781.004
.97810005
18881.001
1904
KMNIST
99644000
2957.000
.9594.006
.9564.
.990
.904
EMNIST
888 007
.869
.877..011
.876..00
.9244.008
.937+.00s
OMNIGLOT
9734.003
.9634.004
.963+.uu3 0033
.9654.003
.9834.001
.992 .001
Table 2: ImageNet-IK IK results: AUROC using the epistemic uncertainty of a given, pre-selected model (as in Eq. (2)) to distinguish between ID (ImageNet-IK) and OOD samples. Furthermore, we report the AUROC when using the epistemic uncertainty for misclassification detection and the AUC of accuracy over fraction of retained predictions the ImageNet- 1K validation dataset. We also report results for all experiments, using the aleatoric uncertainty of the reference model (Reference).
Dona // Task
Reference
cSG-HMC
MCD
DE (LL.)
DE (all)
QUAM
ImageNet-O 62662.001
.677..005
.680+.003
.562+.004
.709=.005
.753+.01
ImageNet-A
.792+.002
.7994.001
.827..002
.6864.001
.8744.004
.8722.003
Misclassífication
8677.007
.7722.011
.796+.u14
.657+.u09
.780=.009
.904
Selective prediction
.958..000
.931 1.003
.9351.006
.911 1.004
.950 =502
.969 1. 1.002
ples, which belong to the ID classes of ImageNet- IK,, but are misclassified with high confidence by conventional ImageNet- 1K classifiers. Furthermore, we evaluated the utility of the uncertainty score for misclassification detection of predictions of the reference model on the ImageNet- IK validation dataset. On the same dataset, we evaluated the accuracy of the reference model when only predicting on fractions of samples with the lowest epistemic uncertainty.
All ImageNet experiments were performed on variations of the EfficientNet architecture [Tan and Le. 2019]. Recent work by Kirichenko et al. [2022] showed that typical ImageNet- 1K classifiers learn desired features of the data even if they rely simple, spurious features for their prediction. Furthermore, they found, that last layer retraining on a dataset without the spurious correlation is sufficient re- re-weight the importance that the classifier places different features. This allows the classifier to ignore the spurious features and utilize the desired features for its prediction. Similarly, we apply QUAM on the last layer of the reference model. We compare against eSG-HMC applied to the last layer, MC dropout and Deep Ensembles. MC dropout was applied to the last layer as well, since the EfficientNet architectures utilize dropout only before the last layer. Two versions of Deep Ensembles were considered. First, Deep Ensembles aggregated from pre- pre-trained EfficientNets of different network sizes (DE (all)) Second, Deep Ensembles of retrained last layers on the same encoder network (DE (LL)). We further utilize the alcatoric uncertainty of the reference model (as in Eq. (2)) as a baseline to assess the additional benefit of estimating the epistemic uncertainty of the reference model. The Laplace approximation was not feasible to compute on our hardware, even only for the last layer.
The results are listed in Tab. 2. Plots showing the respective curves of each experiment are depicted in Fig. C.8 in the appendix. We observe that using the epistemic uncertainty provided by DE (LL) has the worst performance throughout all experiments. While DE (all) performed second best on most tasks, MC dropout outperforms on OOD detection on the ImageNet-O dataset. QUAM outperforms all other methods on all tasks we evaluated, except for ImageNet-A, where it performed on par with DE (all). Details about all experiments and additional results are given in the appendix Sec. C.4.
Compute Efficiency. As an ablation study, we investigate the performance of QUAM under a restricted computational budget. Therefore, the searches for adversarial models were performed on only subset of classes instead of each eligible class, specifically the top N most probable classes according to the predictive distribution of the given, pre-selected model. The computational budget between QUAM and MC dropout was matched by accounting for the number of forward pass equivalents required by each method. In this context, we assume that the backward pass corresponds to the computational cost of two forward passes. The results depicted in Fig. 4 show that QUAM
9

---

considered adieersertal model
considered adverssreel medel
500
1000
500
1000
0.76
0.74
AUROC
AUROC
0.72
MCD
0.84
NCD
QUAM
QUAM
0.82
0.68
7500
7500 passes
forward passes
Figure 4: Inference speed performance. MCD and QUAM evaluated on equal computational budget in terms of forward pass equivalents on ImageNet-O (left) and ImageNet-A (right) tasks.
outperforms MC dropout even under a very limited computational budget. Furthermore, training single additional ensemble member for Deep Ensembles requires more compute than evaluating the entire ImageNer-O and ImageNet-A datasets with QUAM when performed on all 1000 classes.
5 Related Work
Quantifying predictive uncertainty, especially for deep learning models, is active area of research. Classical uncertainty quantification methods such as Bayesian Neural Networks (BNNs) [MacKay, 1992, Neal, 1996] challenging for deep learning, since (i) the Hessian maximum-a paosterrt (MAP) is difficult to estimate and (ii) regularization and normalization techniques cannot be treated [Antoráan et al., 2022]. Epistemic neural networks [Osband et al., 2021] add variance term (the epinet) to the output only. Bayes By Backprop [Blundell et al., 2015] and variational neural networks [Oleksiienko et al., 2022] work only for small models they require considerably more parameters. MC dropout [Gal and Ghahramani, 2016] casts applying dropout during inference as sampling from an approximate distribution. MC dropout was generalized to MC dropconnect [Mobiny et al., 2021]. Deep Ensembles [Lakshminarayanan et al., 2017] are often the best-performing uncertainty quantification method [Ovadia al., 2019, Wursthorn et al., 2022]. Masksembles or Dropout Ensembles combine ensembling with MC dropout [Durasov et al., 2021J. Stochastic Weight Averaging approximates the posterior over the weights [Maddox al., 2019]. Single forward pass methods are fast and they aim to capture different notion of epistemic uncertainty through the distribution or distances of latent representations [Bradshaw et al., 2017, Liu al., 2020, Mukhoti et al., 2021, van Amersfoort al., 2021, Postels et al., 2021] rather than through posterior integrals. For further methods and a general overview of uncertainty estimation see e.g. Hüllermeier and Waegeman [2021]. Abdar et al. [2021] and Gawlikowski et al. [2021].
Conclusion
We have introduced QUAM, novel method that quantifies predictive uncertainty using adversarial models. Adversarial models identify important posterior modes that are missed by previous uncer- tainty quantification methods. We conducted various experiments deep neural networks, for which epistemic uncertainty is challenging to estimate. On a synthetic dataset, we highlighted the strength of method to capture epistemic uncertainty. Furthermore, conducted experiments large-scale benchmarks in the vision domain, where QUAM outperformed all previous methods.
Searching for adversarial models is computationally expensive and has to be done for each new test point. However, more efficient versions can be utilized. One can search for adversarial models while restricting the search to subset of the parameters, c.g. to the last layer as done for the ImageNet experiments, to the normalization parameters, or to the bias weights. Furthermore, there have been several advances for efficient fine-tuning of large models [Houlsby al., 2019, Hu et al., 202 I]. Utilizing those for more efficient versions of our algorithm is an interesting direction for future work.
Nevertheless, high stake applications justify this effort to obtain the best estimate of predictive uncertainty for each new test point. Furthermore, QUAM applicable to quantify the predictive uncertainty of any single given model, regardless of whether uncertainty estimation was considered during the modeling process. This allows to assess the predictive uncertainty of foundation models or specialized models that are obtained externally.
10

---

Acknowledgements
We would like to thank Angela Bitto-Nemling for providing relevant literature, organizing meetings. and giving feedback on this research project. Furthermore, we would like to thank Angela Bitto- Nemling. Daniel Klotz, and Sebastian Lehner for insightful discussions and provoking questions. The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning. are supported by the Federal State Upper Austria. We thank the projects AI-MOTION (LIT-201 8---88-6--00 YOU-222,, DeepFlood (LIT- D019--YYU-2211 Medical Cognitive Computing Center (MC3), INCONTROL-RI (FFG-88 1064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-87 -712))), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235, AL4GreenHcatingrridkse INTEGRATE #FFG-892418, ELISE (H2020-ICT-2019-3 ID: 951847), 4tars4Waters (HORIZON- CL6-2021 - -LMILLI--LMMAT--1-1111 GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.
899943),
We thank AudiJKKU Deep Learning Center, TGW LOGISTICS
References
M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, V. Makarenkov, and S. Nahavandi. A review of uncertainty quantifi- cation in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-299 2021.
A. Adler, R. Youmaran, and W. R. B. Lionheart. A measure of the information content of EIT data. Physiological Measurement, 29(6):S101-S10) 2008
Ö. D. Akyildiz and Míguez. Convergence rates for optimised adaptive importance samplers. Statistics and Computing, 31(12), 2021.
F. D' D'Angolo and V. Fortuin. Repulsive deep ensembles are bayesian. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 3451-3465. Curran Associates, Inc., 2021.
J. Antoráan, D. Janz, J. U. Allingham, E. Daxberger, R. Barbano, E. Nalisnick, and J. M. Hernández- Lobato. Adapting the linearised Laplace model evidence for modern deep learning. ArXiv, 2206.08900, 2022.
G. Apostolakis. The concept of probability if safety assessments of technological systems. Science, 1991.
N. Band, T. G. J. Rudner, Q. Feng, A. Filos, Z. Nado, M. W. Dusenberry, G. Jerfel, D. Tran, and Y. Gal. Benchmarking bayesian deep learning on diabetic retinopathy detection tasks. ArXiv, 2211. 2022.
B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šmdii P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In H. Blockeel, K. Kersting, S. Nijssen, and F. Železný, editors, Machine Learning and Knowledge Discovery in Databuses, pages 387-402, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg.
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1613-1622, 2015.
J. Bradshaw, A. G. de G. Matthews, and Z. Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks. ArXiv, 1707.02476, 2017.
J. Caldeira and B. Nord. Deeply uncertain: comparing methods of uncertainty quantification in deep learning algorithms. Machine Learning: Science and Technology, 2(1 2(0520550 2020.
O. Cappé, A. Guillin, J.-M. Marin, and C. P. Robert. Population Monte Carlo. Journal of Computa- tional and Graphical Statistics, 13(4):907-929, 2004.

---

T. Chen, E. Fox, and C. Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In International Conference on Machine Learning, pages 1683-1691. Proceedings of Machine Learning Research, 2014.
T. Clanuwat, M. Bober-Irizar, A. Kitamoto, A. Lamb, K. Yamamoto, and D. Ha. Deep learning for classical japanese literature. ArXiv, 1812.01718, 2018.
A. D. Cobb and B. Jalaian. Scaling hamiltonian monte carlo inference for bayesian neural networks with symmetric splitting. Uncertainty Artificial Intelligence, 2021.
G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks. IEEE, 2017.
M. Cover and A. Thomas. Elements of Information Theory. Wiley Series in Telecommunications and Signal Processing. Wiley-Interscience, 2nd edition, 2006. ISBN 047124195%
E. Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and P. Hennig. Laplace redux- effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34: 20089-20103, 2021.
Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and paltern recognition. IEEE, 2009.
Depeweg, M. Hernández-Lobato, Doshi-Velez, and S. Udluft. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1192-1201, 2018.
N. Durasov, T. Bagautdinov, Baque, and P. Fua. Masksembles for uncertainty estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15539-13544 2021.
V. Elvira, L. Martino, D. Luengo, and M. F. Bugallo. Efficient multiple importance sampling estimators. IEEE Signal Processing Letters, 22(10):757--161,, 2015.
V. Elvira, L. Martino, D. Luengo, and M. F. Bugallo. Generalized multiple importance sampling. Statistical Science, 34(1), 2019.
R. Eschenhagen, E. Daxberger, P. Hennig, and A. Kristiadi. Mixtures of laplace approximations for improved post-hoc uncertainty in deep learning. arXiv, 2111.03577, 2021.
A. V. Fiacco and G. P. McCormick. Nonlinear programming: sequential unconstrained minimization techniques. Society for Industrial and Applied Mathematics, 1990.
A. Filos, Farquhar, A. Gomez, T. Rudner, Z. Kenton, L. Smith, M. Alizadeh, A. De Kroon, and Y. Gal. A systematic comparison of bayesian deep learning robustness in diabetic retinopathy tasks. ArXiv, 1912. 204811 2019.
S. Fort, H. Hu, and B. Lakshminarayanan. Deep ensembles: A loss landscape perspective. ArXiv, 1912.02757,2000 2019.
Y. Gal. Uncertainty in Deep Learning. PhD thesis, Department of Engineering, University of Cambridge, 2016.
Y. Gal and Z. Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference Machine Learning, 2016.
J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher, M. Shahzad, W. Yang, R. Bamler, and X. Zhu, A survey of uncertainty in deep neural networks. ArXiv, 2107.03342. 2021.
12

---

I. J. Goodfellow, J. Pouget-Abadic M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
I. J. Goodfellow. J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. arXiv, 1412.6572, 2015.
A. Graves. Practical variational inference for neural networks. In J. Shawe- Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 24, Curran Associates, Inc., 2011.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321-1330, 2017.
K. Gustafsson, M. Danelljan, and B. Schön. Evaluating scalable bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
J. Hale. A probabilistic earley parser as a psycholinguistic model. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 1-8. Association for Computational Linguistics, 2001.
J. C. Helton. Risk, uncertainty in risk, and the EPA release limits for radioactive waste disposal. Nuclear Technology, 101(1):18-39 1993.
J. C. Helton. Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty. Journal of Statistical Computation and Simulation, 57(1-4):3-76, 1997.
D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
T. Hesterberg. Weighted average importance sampling and defensive mixture distributions. Techno- metrics, 3752)188-1144 1995.
T. Hesterberg. Estimates and confidence intervals for importance sampling sensitivity analysis. Mathematical and Computer Modelling, 23(8):79-88) -5, 1996.
N. Houlsby, F. Huszar, Z. Ghahramani, and M. Lengyel. Bayesian active learning for classification and preference learning. ArXiv, 1112.5745, 2011.
N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning. Proceedings of Machine Learning Research, 2019.
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. ArXiv, 2106.09685, 2021.
G. Huang, Y. Li, G. Pleiss, Z. Liu, J. E. Hopcroft, and K. Q. Weinberger. Snapshot ensembles: Train 1, get m for free. ArXiv, 1704.00109, 2017.
E. Hüllermeier and W. Wacgeman. Aleatoric and epistemic uncertainty in machine learning: an introduction concepts and methods. Machine Learning, 3(110):477550, 2021.
P. Izmailov, S. Vikram, M. D. Hoffman, and A. G. Wilson. What are Bayesian neural network posteriors really like? In Proceedings of the 38th International Conference on Machine Learning, pages 4629-4640, 2021.
E. T. Jaynes. Information theory and statistical mechanics. Phys. Rev. 106:620-6300 1957.
S. Kapoor, torch-sgld: Sgld as pytorch optimizer: httpe://ggpa..or/pppettrrraats/ 2023. Accessed: 12-05-2023.
13

---

W. Karush. Minima of functions of several variables with inequalities as side conditions. Master's thesis, University of Chicago, 1939.
A. Kendall and Gal. What uncertainties do we need Bayesian deep learning for computer vision?
In Advances in Neural Information Processing Systems, volume 30, 2017.
D. Kingma and J. Ba. Adam: method for stochastic optimization. ArXiv, 1412.6980, 2014.
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In 2nd International Conference on Learning Representations, 2014.
Kirichenko, P. Izmailov, and G. Wilson. Last layer re-training sufficient for robustness to spurious correlations. ArXiv, 2204.02937, 2022.
H. W. Kuhn and A. W. Tucker. Nonlinear programming. In J. Neyman, editor, Second Berkeley Symposium on Mathematical Statistics and Probability, pages 481-492, Berkeley, 1950. University of California Press.
O. Kviman, H. Melin, H. Koptagel, V. Elvira, and J. Lagergren. Multiple importance sampling ELBO and deep ensembles of variational approximations. In Proceedings of the 25th International Conference on Artificial Intelligence and Staristics, volume 151 of Proceedings of Machine Learning Research, pages 10687-10702, 2022.
B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through proba- bilistic program induction. Science, 2015.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems, page 6405-644 64166616.222
Curran Associates Inc., 2017.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233) 1998.
C. Li, C. Chen, D. Carlson, and L. Carin. Preconditioned stochastic gradient Langevin dynamics for deep neural networks. Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI), page 1788-18 1794. AAAI Press, 2016.
J. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 01:7498-75122 2020.
E. S. Lubana, E. J. Bigelow, R. P. Dick, D. Krueger, and H. Tanaka, Mechanistic mode connectivity. ArXiv, 2211.0842,, 2022.
D. G. Luenberger and Y. Ye. Linear and nonlinear programming. International Series in Operations Research and Management Science. Springer, 2016.
D. J. C. MacKay. A practical Bayesian framework for backprop networks. Neural Computation, 4: 448-472, 1992.
W. Maddox, Izmailov, T. Garipov, D. P. Vetrov, and A. G. Wilson. simple baseline for Bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems, 2019.
A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
R. May. A simple proof of the Marush-Kuhne Kuhn-Tucker theorem with finite number of equality and inequality constraints. ArXiv, 1248, 2020.
T. F. McKone. Uncertainty and variability in human exposures to soil contaminants through home- grown food: monte carlo assessment. Risk Analysis, 14, 1994.
A. Mobiny, P. Yuan, S. K. Moulik, Garg, C. C. Wu, and VanNguyen. DropConnect effective in modeling uncertainty of Bayesian deep networks. Scientific Reports, 11:5458, 2021.
14

---

J. Mukhoti, A. Kirsch, J. vanAmersfoort, P. H. S. Torr, and Y. Gal. Deep deterministic uncertainty: A simple baseline. ArXiv, 2102.11588 2021.
R. Neal. Bayesian Learning for Neural Networks. Springer Verlag, New York, 1996.
I. Oleksiienko, D. T. Tran, and A. Iosifidis. Variational neural networks. ArXiv, 2207.01524, 2022.
I. Osband, Z. Wen, S. M. Asghari, V. Dwaracherla, M. Ibrahimi, X. Lu, and B. VanRoy. Weight uncertainty in neural networks. ArXiv, 2107.08922 2021.
Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and
J. Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, volume 32, 2019.
A. Owen and Y. Zhou. Safe and effective importance sampling. Journal of the American Statistical Association, 95(449):135 1433,-113,.7 2000.
Parker-Holder, L. Metz, C. Resnick, H. Hu, A. Lerer, A. Letcher, A. Peysakhovich, A. Pacchiano, and J. Foerster. Ridge rider: Finding diverse solutions by following eigenvectors of the hessian. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 753-765. Curran Associates, Inc., 2020.
A. Paszke, S. Gross, Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pret- tenhofer, R. Weiss, V. Dubourg, et al. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research, 12:2825-2830 2011.
J. Postels, M. Segu, T. Sun, L. Van Gool, F. Yu, and F. Tombari. On the practicality of deterministic epistemic uncertainty. ArXiv, 2107 006499, 2021.
A. E. Raftery and L. Bao. Estimating and projecting trends in HIV/AIDS generalized epidemics using incremental mixture importance sampling. Biometrics, 66, 2010.
M. Rigter, B. Lacerda, and N. Hawes. Rambo-rl: Robust adversarial model-based offline reinforce- ment learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 16082-16097. Curran Associates, Inc., 2022.
L. Scimeca, S. J. Oh, S. Chun, M. Poli, and S. Yun. Which shorteut cues will dans choose? a study from the parameter-space perspective. arXiv. 2110.03095, 2022.
Seidenfeld. Entropy and uncertainty. Philosophy of Science, 53(4))467-491, 1986.
H. Shah, K. Tamuly, A. Raghunathan, P. Jain, and P. Netrapalli. The pitfalls of simplicity bias neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9573-9585. Curran Associates, Inc., 2020.
C. Shannon and C. Elwood. mathematical theory of communication. The Bell System Technical Journal, 27:379 423, 1948.
L. Smith and Y. Gal. Understanding measures of uncertainty for adversarial example detection. In A. Globerson and R. Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, pages 560-569. AUAI Press, 2018.
R. J. Stecle, A. E. Raftery, and M. J. Emond. Computing normalizing constants for finite mixture models via incremental mixture importance sampling (IMIS). Journal of Computational and Graphical Statistics, 15, 2006.
M. Sundararajan, Taly, and Q. Yan. Axiomatic attribution for deep networks. In International Conference Machine Learning, 2017.
15

---

C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. ArXiv, 1312.6199, 2013.
M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-6114 Proceedings of Machine Learning Research, 2019.
M. Tribus. Thermostatics and T'ermodynanics: An Introduction to Energy, Information and States of Matter, with Engineering Applications. University series in basic engineering. Van Nostrand, 1961.
J. van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal. Uncertainty estimation using single deep deterministic neural network. In International Conference on Machine Learning, pages 9690-9700. Proceedings of Machine Learning Research, 2020.
J. van Amersfoort, I.. Smith, A. Jesson, O. Key, and Y. Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. ArXiv, 2102. 1409, 2021.
E. Veach and L. Guibas. Optimally combining sampling techniques for Monte Carlo rendering. In Proceedings of the 22nd Annual Conference Computer Graphics and Interactive Techniques, pages 419-428. Association for Computing Machinery, 1995.
W. E. Vesely and D. M. Rasmuson. Uncertainties in nuclear probabilistic risk analyses. Risk Analysis, 1984.
Weinzierl. Introduction to Monte Carlo methods. ArXiv, hep-ph/0006269, 2000.
M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, page 681-688, Madison, WI, USA, 2011. Omnipress.
A. G. Wilson and P. Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in Neural Information Processing Systems, 33:4697-47708, 2020.
K. Wursthorn, M. Hillemann, and M. M. Ulrich. Comparison of uncertainty quantification methods for CNN-based regression. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XL.II--2-2022::21--11
2022.
H. Xiao, Rasul, and Vollgraf. Fashion-mnin mnnit: novel image dataset for benchmarking machine learning algorithms. ArXiv, 1708.07747, 2017.
W. I. Zangwill. Non-linear programming via penalty functions. Management Science, 13(5):344-358 1967.
R. Zhang, C. Li, Zhang, C. Chen, and A. G. Wilson. Cyclical stochastic gradient MCMC for Bayesian deep learning. In International Conference on Learning Representations, 2020.
R. Zhang, A. G. Wilson, and C. DeSa. Low-precision stochastic gradient Langevin dynamies. ArXiv, 2206.09909 2022.
J. V. Zidek and C. vanEeden. Uncertainty, entropy, variance and the effect of partial information. Lecture Monogrhon Series, 42:155-161 2003.
16

---

Appendix
This is the appendix of the paper "Quantification of Uncertainty with Adversarial Models". It consists of three sections. In view of the increasing influence of contemporary machine learning research the broader public, section A gives societal impact statement. Following to this, section B gives details our theoretical results, foremost about the measure of uncertainty used throughout our work. Furthermore, Mixture Importance Sampling for variance reduction is discussed. Finally, section C gives details about the experiments presented in the main paper, as well as further experiments.
Contents of the Appendix
Societal Impact Statement
18
Theoretical Results
19
B.1
Measuring Predictive Uncertainty
19
B.1.1
Entropy and Cross- Entropy as Measures of Predictive Uncertainty
19
B.1.2
Classification
20
B.1.3
Regression
22
B.2
Mixture Importance Sampling for Variance Reduction
23
C Experimental Details and Further Experiments
27
C.I. Details on the Adversarial Model Search
27
C.2 Simplex Example
28
C.3 Epistemic Uncertainty Synthetic Dataset
29
C.4 Epistemic Uncertainty on Vision Datasets
29
C.4.1 MNIST
31
C.4.2 ImageNet
31
C.5 Comparing Mechanistic Similarity of Deep Ensembles vs. Adversarial Models
38
C.6 Prediction Space Similarity of Deep Ensembles and Adversarial Models
38
C.7 Computational Expenses
38
List of Figures
B.1 Asymptotic variance for multimodal target and unimodal sampling distribution
26
C.1 Illustrative example of QUAM
27
C.2 HMC and Adversarial Model Search on simplex
28
C.3 Epistemic uncertainty (setting (b)) on synthetic classification dataset
30
C.4 Model variance of different methods toy regression dataset
30
C,5 Histograms MNIST
33
C.6 Calibration on ImageNet
34
C.7 Histograms ImageNet
36
C88 Detailed results of ImageNet experiments
37
C.9 Comparing mechanistic similarity of Deep Ensembles vs. Adversarial Models
39
C.10 Prediction Space Similarity of Deep Ensembles Adversarial Models
39
17

---

List of Tables
C.1 Results for additional baseline (MoLA)
31
C.2 Detailed results of MNIST OOD detection experiments
32
C.3 Results for calibration on ImageNet
34
C.4 Detailed results of ImageNet experiments
35
A Societal Impact Statement
In this work, we have focused on improving the predictive uncertainty estimation for machine learning models, specifically deep learning models. Our primary goal is enhance the robustness and reliability of these predictions, which we believe have several positive societal impacts.
1. Improved decision-making By providing more accurate predictive uncertainty estimates, we enable broad range of stakcholders to make more informed decisions. This could have implications across various sectors, including healthcare, finance, and autonomous vehicles, where maaiinnnmmain based on machine learning predictions can directly affect human lives and economic stability.
2. Increased trust in machine learning systems: By enhancing the reliability of machine learning models, our work may also contribute increased public trust in these systems. This could foster greater acceptance and integration of machine learning technologies in everyday life, driving societal advancement.
3. Promotion of responsible machine learning: Accurate uncertainty estimation is crucial for the responsible deployment of machine learning systems. By advancing this area, our work promotes the use of those methods in an ethical, transparent, and accountable manner.
While anticipate predominantly positive impacts, it is important to acknowledge potential negative impacts or challenges.
Even with improved uncertainty estimates, there is risk that these might be misinterpreted or misused, potentially leading incorrect decisions or unintended consequences. It is vital to couple advancements this field with improved education and awareness around the interpretation of uncertainty in Al systems.
Misinterpretation of uncertainty:
2. Increased reliance on machine learning systems: While increased trust in machine learning systems is beneficial, there a risk it could lead to vverreeliance on these systems, potentially resulting in reduced human oversight or critical thinking. It's important that robustness and reliability improvements don't result in blind trust.
3. Inequitable distribution of benefits: As with any technological advancement, there is risk that the benefits might not be evenly distributed, potentially exacerbating existing societal inequalities. We urge policymakers and practitioners to consider this when implementing our findings.
In conclusion, while our work aims to make significant positive contributions to society, we believe essential consider these potential negative impacts and take steps to mitigate them proactively.
18

---

B Theoretical Results
B.1 Measuring Predictive Uncertainty
In this section, we first discuss the usage of the entropy and the cross-entropy as measures of predictive uncertainty. Following this, we introduce the two settings (a) and (b) (see Sec. 2) in detail for the predictive distributions of probabilistic models in classification and regression. Finally, we discuss Mixture Importance Sampling for variance reduction of the uncertainty estimator.
B.1.1 Entropy and Cross-Entronn as Measures of Predictive Uncertainty
Shamnon and Elwood [1948] defines the entropy Hpp = - Ex= Pi log as measure of the amount of uncertainty of discrete probability distribution p (Pi..... Pv) and states that it measures how much "choice" is involved in the selection of class i. See also Jaynes [1957], Cover and Thomas [2006] for an elaboration on this topic. The value log p) has been called "surprisal" [Tribus, 1961] (page 64, Subsection 2.9.1) and has been used in computational linguisties [Hale, 2001]. Hence, the entropy is the expected or mean surprisal. Instead of "surprisal" also the terms "information content", celffinformatton", or "Shannon information" are used.
The cross-entropy CElp, q) = - log qi between two discrete probability distributions p (P1,.... PN) and q = (92,..., UN measures the expectation of the surprisal of q under distri- bution Like the entropy, the cross-entropy is mean of surprisals, therefore can be considered as measure to quantify uncertainty. The higher surprisals are on average, the higher the uncertainty. The cross-entropy has increased uncertainty compared to the entropy since more surprising events are expected when selecting events via p instead of q. Only if those distributions coincide, there is no ad- ditional surprisal and the cross-entropy is equal to the entropy of the distributions. The cross-entropy depends on the uncertainty of the two distributions and how different they are. In particular, high surprisal of gi and low surprisal of p: strongly increase the cross-entropy since unexpected events more frequent, that is, we are more often surprised. Thus, the cross-entropy does not only measure the uncertainty under distribution p. but also the difference of the distributions. The average surprisal via the cross-entropy depends on the uncertainty of p and the difference between p and q:
CE(p, a)
Ps log q:
(9)
N
Σ 1=1
Pt log +
Σ 1=1
ps
log
Pt
H[p] + DKL(pllq)
where the Kullback-Leibler divergence (-
(p q)
Σ ps log Pr
(10)
9
The Kullback- divergence measures the difference in the distributions via their average difference of surprisals. Furthermore, it measures the decrease in uncertainty when shifting from the estimate p to the true [Seidenfeld, 1986, Adler et al., 2008].
Therefore, the cross-entropy can serve to measure the total uncertainty, where the entropy is used as aleatoric uncertainty and the difference of distributions used as the epistemic uncertainty. We assume that q is the true distribution that is estimated by the distribution p. We quantify the total uncertainty of p as the sum of the entropy of p (aleatoric uncertainty) and the Kullback-Leibler divergence to g (epistemic uncertainty). In accordance with Apostolakis [1991] and Helton [1997], the aleatoric uncertainty measures the stochasticity of sampling from p. while the epistemic uncertainty measures the deviation of the parameters p from the true parameters q.
In the context of quantifying uncertainty through probability distributions, other measures such as the variance have been proposed [Zidek and vanEeden, 2003]. For uncertainty estimation in the context of deep learning systems, e.g. Gal [2016), Kendall and Gal [2017], Depeweg et al. [2018] proposed to use the variance of the BMA predictive distribution as a measure of uncertainty. Entropy and variance capture different notions of uncertainty and investigating measures based on the variance of the predictive distribution is an interesting avenue for future work.
19

---

B.1.2 Classification
Setting (a): Expected uncertainty when selecting a model. We assume to have training data D and an input æ. We want to know the uncertainty in predicting a class y from ø when we first choose model to based the posterior p(w (D) an then use the chosen model to choose class for input æ according to the predictive distribution pyy æ, w). The uncertainty in predicting the class arises from choosing a model (epistemic) and from choosing a class using this probabílistic model (aleatoric).
Through Bayesian model averaging, we obtain the following probability of selecting class:
pyy D) = J, p(y x; w) p(rō D) dw.
(11)
The total uncertainty is commonly measured as the entropy of this probability distribution [Houlsby et al., 2011, Gal, 2016, Depeweg et al., 2018, Hüllermeier and Waegeman, 2021]:
Hpp(y x, D)]
(12)
We can reformulate the total uncertainty as the expected cross-entropy:
H Hsp(y| D)] = Σ
ply a. D) logp(y a. D)
(13)
yEy
Σ
logp(y D) / puy x, ()) p(u D) dw
VEY
/ -
Σ p(y x,, w) logp(y x, D) p(rō) D) dw
DEY
CElp(y x. w. pa x, D)] p(w D) dw.
We can split the total uncertainty into the aleatoric and epistemic uncertainty [Houlsby et al., 2011, Gal, 2016, Smith and Gal, 2018]:
J, CE(p(U x. (), p(y x. D)] p(i D) dw
(14)
(Hlp(y a. ()) (KL( x, 10) ll ply x. D))) p( (D) dō
Hlp(y xx, w)] p( D) dw + / DRL(p | a. w)) || p(y æ, D)) p(10 D) dū
[ H[p(y xx w)) p( D) du + I[Y; W x. ar
We verify the last equality in Eq. (14), i.e that the Mutual Information is equal to the expected Kullback-Leibler divergence:
I[; ; W D. = / Σ
p(y. ②
x. D)
p(y,
w x. D)
log
dw
(15)
ply x, D)
p(u D)
yEy
/ Σ
p(y
x, w) p(u D))
p(y x () p( D) log
dw
p
x; D) p( D)
VEy
/ Σp(y
ply
x, w
x p(y 2 w) log
p(
D) dr
p(y x. D)
DEY =/n DRl(p(y w. 1 x, D)) p(u
D)) dw
20

---

This is possible because the label is dependent on the selected model. First, a model is selected, then a label is chosen with the selected model. To summarize, the predictive uncertainty is measured by:
Jw /
H[p(y x; D)]
Hyp(y x x)) p( () dw + I[;; W x; DI
(16)
Hlp(y x. i)) p(: D) dw
w (KL(p(y x w) || ply x, P)) p( ()) dw
ww
/ CE(p(y æ. t)), p(y x. D) p(rō) (D) deb
The total uncertainty is given by the entropy of the Bayesian model average predictive distribution, which showed is equal to the expected cross-entropy between the predictive distributions of candidate models vp selected according to the posterior and the Bayesian model average predictive distribution. The aleatoric uncertainty is the expected entropy of candidate models drawn from the posterior, which can also be interpreted as the entropy we expect when selecting a model according to the posterior. Therefore, if all models likely under the posterior have low surprisal, the aleatoric uncertainty in this setting is low. The epistemic uncertainty is the expected KL divergence between the the predictive distributions of candidate models and the Bayesian model average predictive distribution. Therefore, if all models likely under the posterior have low divergence of their predictive distribution to the Bayesian model average predictive distribution, the epistemic uncertainty this setting is low.
Setting (b): Uncertainty of given, pre-selected model. We assume have training data D, an input x. and a given, pre-selected model with parameters to and predictive distribution p(y æ; w). Using the predictive distribution of the model, class y is selected based æ, therefore there is uncertainty about which is selected. Furthermore, we assume that the true model with predictive distribution p(y x, w²) and parameters w" has generated the training data D and will also generate the observed (real world) y" from æ that we want to predict. The true model is only revealed later, e.g. via more samples or by receiving knowledge about w=" Hence, there is uncertainty about the parameters of the true model. Revealing the true model is viewed as drawing a true model from all possible true models according to their agreement with D. Note, to reveal the true model is not necessary in our framework but helpful for the intuition of drawing true model. We neither consider uncertainty about the model class nor the modeling nor about the training data, summary, there uncertainty about drawing class from the predictive distribution of the given, pre-selected model and uncertainty about drawing the true parameters of the model distribution.
According to Apostolakis [1991] and Helton [1997], the aleatoric uncertainty is the variability of selecting class y via p(y æ, w). Using the entropy, the aleatoric uncertainty is
Hpp(y x; u²)
(17)
Also according to Apostolakis [1991] and Helton [1997]. the epistemic uncertainty is the uncertainty about the parameters w of the distribution, that is, a difference measure between w and the true parameters w", We use as measure for the epistemic uncertainty the Kullback-Leibler divergence:
(py( æ, w)) p(y x. w.)")).
(18)
The total uncertainty is the aleatoric uncertainty plus the epistemic uncertainty, which is the cross- entropy between p(y xx w) and p(y æ., w²):
CE|p(y) xx, w) p(y x, w")] = Hpp(y x, w,w) + DKL(p( x, w,w) || p(y
x, w))) (19)
However, we do not know the true parameters w". The posterior p(u (D) gives us the likelihood of i being the true parameters w². We assume that the true model is revealed later. Therefore we use the expected Kullback-Leibler divergence for the epistemic uncertainty:
J. DKL DKL(P(U x w) p( x, x)) p(u | (D) dro
(20)
Consequently, the total uncertainty is
H[p(y| x. w)] + /,v DKL x, w) |l | x, w)) p(w (D) dw.
(21)
21

---

The total uncertainty can therefore be expressed by the expected cross eross-yntrppy as it was in setting (a) (see Eq. (16)), but between ply x, w) and p x w)):
J. CElp(y a m) ply o úi)) p(u D) dw
(22)
/ (Hlp(y a. + (Kl(p( æ, w) || p(y (.)) p(u D) dı
Hip(y x, w) + I (Ry(p( a w) ll x w)) p(u D) dū
B.1.3 Regression
We follow Depeweg et al. [2018] and measure the predictive uncertainty in a regression setting using the differential entropy H[p(y æ, [.] - Jy p(y x, w) logp(y x, w)dy of the predictive distribution p(y x., wo) of probabilistic model. In the following, we assume that Gaussian distribution, but other continuous probability distributions e.g. a Laplace lead to similar results. The model thus has to provide estimators for the mean p(æ, w) and variance oz w) of the Gaussian. The predictive distribution is given by
modeling
ply x w) (2 o² (2x (x) - exp {
(y pla, w))
(23)
2 202(x w)
The differential entropy of a Gaussian distribution is given by
H[p(y x w)] I ply æ, w) w) logply x, w) dy
(24)
log(o w)) + log(2 + 2
The KL divergence between two Gaussian distributions is given by
DKL(P(y) a, w) p(y x, wi)
(25)
= =- I, ply xx w) log
p(y x, w)
dy
p(y xr, w))
1 log (z) w)
r² (x, w)
+ (u( w) u(x w))
1
2 (2(2
2 0² (x, 10
2
Setting (a): Expected uncertainty when selecting model. Depeweg et al. [2018] consider the differential entropy of the Bayesian model average p(y x. D) = Jw p(y x, ()p(w D)dw which is equal to the expected cross-entropy and can be decomposed into the expected differential entropy and Kullback-Leibler divergence. Therefore, the expected uncertainty when selecting model is given by
J, CElp(y x. () plu ar D)) p( D) dio H'p(y | ar, D)]
(26)
- H[ply æ. x)) p( D) drō + / w
DKI. æ, x0 [plly x, D)) p(w D)) dr
Jw
S/1 2 log(o"(a,, u)) p( DD d + log(2n)
+ S DKi(p(y a. 1p( x. D)) p(
D) dw
22

---

Setting (b): Uncertainty of a given, pre-selected model. the uncertainty of given, pre-selected model given by
Synonymous to the classification setting,
J., CElply x,ww) w) ply x wi]] p(ui) D) du
(27)
= Hip(y x. w)] + I (KL( x, w) |l p x, ()) p(ti D) dw
1
2 log(o²(z,, + log(2x)
/
1 log 2
(z)(xx 20) 0² 02 w)
(o w) + (u(x, w) - (ax 1z)) 2 2o² x w)
p(20 D) dw.
Homoscedastic, Model Invariant Noise.
We assume, that noise homoscedastic for all inputs
X, thus o² w) = o² (w)) Furthermore, most models in regression do not explicitly model the variance in their training objective. For such a model w. we can estimate the variance on validation
dataset D = (Fnn Un ) as
8² (w) ((ww)
Σ (Un fl(zn, w))² N
(28)
If we assume that all reasonable models under the posterior will have similar variances (6² (6²(w) se o² (w) for w ~ p(w D)), the uncertainty of prediction using the given, pre-selected model w is given by
/ CElp(y a, w), ply xx x)) p(w D) dw
(29)
w
1 log(d"(w)) + log(2
~ 2
/ log (2((w))
(ww + (u(æ,, w) - u(xx, 20))
p(u D) dw
w (2(ww)
2 26²(2r)
1
(og(O2(w)) + (ww) J.w (u(m, w)) - u(x, u))) p(u D) dw + 2
1
+ log((n))
2
B.2 Mixture Importance Sampling for Variance Reduction
The epistemic uncertainties in Eq. (1) and Eq. (2) are expectations of KI divergences over the posterior. We have to approximate these integrals.
If the posterior has diffeernnt modes, a concentrated importance sampling function has high variance of estimates, therefore converges very slowly [Steele et al., 2006]. Thus, we use mixture importance sampling (MIS) [Hesterberg, 1995]. MIS uses mixture model for sampling, instead of unimodal model of standard importance sampling [Owen and Zhou, 2000]. Multiple importance sampling Veach and Guibas [1995] is similar to MIS and equal to it for balanced heuristics [Owen and Zhou, 2000J. More details on these and similar methods can be found in Owen and Zhou [2000], Cappé al. [2004], Elvira et al. [2015, 2019], Steele et al. [2006], Raftery and Bao [2010]. MIS has been very successfully applied to estimate multimodal densities. For example, the evidence lower bound (ELBO) [Kingma and Welling, 2014] has been improved by multiple importance sampling ELBO [Kviman et al., 2022]. Using a mixture model should ensure that at least one of its components will locally match the shape of the integrand. Often, MIS iteratively enrich the sampling distribution by new modes [Raftery and Bao, 2010].
In contrast to iterative enrichment, which finds modes by chance, posterior modes, where the integrand of the definition of epistemic uncertainty is large. For each of these modes, we define a component of the mixture from which then sample. We have the huge advantage to have explicit expressions for the integrand. The integrand of the epistemic uncertainty in Eq. (1) and Eq. (2) has the form
able to explicitly search for
D(p(y) x. w), p(v w)) p( D),
(30)
23

---

where D -) is distance or divergence of distributions which is computed using the parameters that determine those distributions. The distance/divergence D(.,)) -) eliminates the aleatoric uncertainty. which is present in pyy w) and p(y m, w.) Essentially, D) reduces distributions to functions of their parameters.
Importance sampling is applied to estimate integrals of the form
= f(x)() p(x) dx /
[(x) p(æ)
g(x)) dæ
(31)
((x)
with integrand fx) and probability distributions p(a:) and (ax), when it is easier to sample according to (()) than p(æ). The estimator of Eq. (31) when drawing 2n according to q(x)) is given by
f(xn) p(æ,)
(32)
g(x,.)
n=1
The asymptotic variance of of importance sampling is given by (see e.g. Owen and Zhou [2000]):
o² f [(x)) p(z)
))
2 ((x)) dæ
8)
(33)
g(x))
f(x) f(x) p(x)
((x) dæ 2²
x
((xr)
and its estimator when drawing from g(x) is given by
1
N
I ((xa) p(ann) -s)
$)
(34)
N
g(xn))
n=1
((xa) p(#n)
5²
q(In))
We observe, that the variance is determined by the term [(=)p()) thus we want ((x) to be proportional
((x)
f())p(()) Most importantly, q(xr) should be close zero for large ((x))(x). To give an
intuition about the severity of unmatched modes, we depict an educational example in Fig. B.I. Now we plug in the form of the integrand given by Eq. (30) into Eq. (31), calculate the expected divergence D(., -) under the model posterior pl i D). This results in
D(p(y x., ()) p(y x. it)) p(u (D)
q i dw
(35)
((w)
with estimate
D(p(y x, w.) p(y x. wn)) p(w D)
(36)
q((.n)
The variance is given by
at Jm
D(r(y w,, ply x. w)) p(u0 D) =)² g( dw
v
(37)
q(
D(p(y æ, ply
a x)) p(u D) ((w) dw 2
w
q(w))
The estimate for the variance is given by
N
D(p(y) xx w) pqy x, (n) p(zōn D)
0.
v
(38)
N
q(w,)
n=1
D(p(y) æ,, w), p(y x, wr p(ron D)
02
N
((w,)
n
24

---

where Wn is drawn according to (())..
The asymptotic (N + oc) confidence intervals are given by
lim Pr (-a. X00
V2 ["
de
dt.
(39)
Thus, â converges with VN to v. The asymptotic confidence interval proofed Weinzierl [2000]
and Hesterberg [1996] using the Lindeberg-LLvv central limit theorem which ensures the asymptotic normality of the estimate The ((w) that minimizes the variance is
( D(p(y) x, w) pvy x. w)) p(u D)
(40)
Thus we want find density q( (b) that is proportional to D(p(y x w) ply æ. w)) p(u D). Only approximating the posterior p( i 'D) as Deep Ensembles or MC dropout is insufficient to guarantee a low expected error, since the sampling variance cannot be bounded, as o could get arbitrarily big if the distance is large but the probability under the sampling distribution is very small. For ((3) x p(to (D) and non-negative, unbounded, but continuous D(., +), the variance of given by Eq. (37) cannot be bounded.
For example, if D(., where the means (xx w), ju(,, ti) and variances (²x w)) o2 ai) are estimates provided by the models, the KL is unbounded. The KL divergence between two Gaussian distributions given by
the KL-divergence and both p(y x. w) and p(y x; w)) are Gaussians
DRI.(p(V) x; w) p(y ())
(41)
J ply x. w) log
ply 2 w)
dy
ply 2. 2))
(0²(x, w 02 w))
0² xx. w)
+ (u(xx w) - u(x w))) 2 o 20)
1
= 2
log
+
2
For oxx w) going towards and non-zero difference of the mean values, the KL-divergence can be arbitrarily large. Therefore, methods that only consider the posterior p(u D) cannot bound the variance of if D is unbounded and the parameters w allow distributions which can make D(.,)) :) arbitrary large.
25

---

0.40
pix)
Asymptolic Variance IS Estimate
q(x)
0.30
0.25
105
Density
0.20
Uo.2
10
0.15
10
0.05
0.00
0.6
0.8
1.0
X
0
Asymptotic Variance of Estimato
Asymplolic Variance of IS Essimalte
10"
10
10
10
10
10
10
10
10'
3
5
2
3
Mp.2
Figure B.1: Analysis of asymptotic variance of importance sampling for multimodal target distribution p(x) and unimodal sampling distribution g(x).. The target distribution is mixture of two Gaussian distributions with means Mp.1, and variances of 17.2 The sampling distribution is a single Gaussian with mean Plg and variance o² q(x) matches one of the modes of p(r), but misses the other. Both distributions are visualized for their standard parameters μp.1 = My = O, μp. = 3 and of = of = o 1, where both mixture components p(x) are equally weighted. We calculate the asymptotic variance (Eq, (33) with f(x) = 1) for different values o. Hp, and o² and show the results in the top right, bottom left and bottom right plot respectively. The standard value for the varied parameter is indicated by the black dashed line. We observe, that slightly increasing the variance of the second mixture component of p(z), which is not matched by the mode of g(z)) ). rapidly increases the asymptotic variance. Similarly. increasing the distance between the center the unmatched mixture component of p(r)) and q(x)) strongly increases the asymptotic variance. On the contrary, increasing the variance of the sampling distribution g(x) does not lead to a strong increase, the worse approximation of the matched mode of p(a)) is counterbalanced by putting probability mass where the second mode of p(x) is located. Note, that this issue even more exacerbated if f(x) non-constant. Then, g(z) has to match the modes of fxx well.
26

---

C Experimental Details and Further Experiments
Our code is publicly available at https:/ggihubb.commm//kuuuuuuu
C.1 Details on the Adversarial Model Search
During the adversarial model search, seek to maximize the KL divergence between the prediction of the reference model and adversarial models. For an example, see Fig. C.I. We found that directly maximizing the KL divergence always leads to similar solutions to the optimization problem. Therefore, we maximized the likelihood a new test point to be in each possible class. The optimization problem is very similar, considering the predictive distribution p(y | æ, w) of reference model and the predictive distribution p(y æ, ú)) of an adversarial model, the model that is updated. The KL divergence between those two is given by
(KL(p(3 æ. w) p(y æ; (0)
(42)
= Σ p(y æ. w) log p(y x, u)
py x, t6)
Σ p(y æ, log (ply x, w)) Σp(p p(y I x. w) log (p(y x ())
Hpp(y æ w)) + CE(p(3 ()) p(u æz in)
Only the cross-entropy between the predictive distributions of the reference model parameterized by w and the adversarial model parameterized by v plays a role in the optimization, since the entropy of Pw stays constant during the adversarial model search. Thus, the optimization target equivalent to the cross-entropy loss, except that Pw is generally not one-hot encoded but an arbitrary categorical distribution. This also relates to targeted untargeted adversarial attacks on the input. Targeted attacks try to maximize the output probability of a specific class. Untargeted attacks try to minimize the probability of the originally predicted class, by maximizing all other classes. We found that attacking individual classes works better empirically, while directly maximizing the KL divergence always leads to similar solutions for different searches. The result often a further increase of the probability associated with the most likely class. Therefore, we conducted as many adversarial model searches for new test point, there are classes the classification task. Thereby, we optimize the cross-entropy loss for specific class in each search.
Input
dog
reference w
adversarial
reference W
adversarial WP
reference
adversarial W9
Model
IP;
W
likellhood
likelihood
likelihood
likelihood
likellhood
likelhood
Ayex
PtD
4pen
Output
pla k).
plar W:)
pOr xx,w)
ply Ix.,
ply ky,
pir (R))
cat
dog
cat
dog
cat
dog
D
low uncertainty
low uncertainty
high uncertainty
Figure C.I: Illustrative example of QUAM. We illustrate quantifying the predictive uncertainty of given, pre-selected model (blue), classifier for images of cats and dogs. For each of the input images, we search for adversarial models (orange) that make different predictions than the given, pre-selected model while explaining the training data equally well (having high likelihood). The adversarial models found for an image a dog or still make similar predictions (low epistemic uncertainty). while the adversarial model found for an image of lion makes highly different prediction (high epistemic uncertainty), as features present in images both cats and dogs can be utilized classify the image of a lion.
27

---

For regression, we add a small perturbation to the bias of the output linear layer. This is necessary to ensure gradient in the first update step, as the model to optimize is initialized with the reference model. For regression, we perform the adversarial model search two times, the output of an adversarial model could be higher or lower than the reference model if we assume scalar output. We force, that the two adversarial model searches get higher or lower outputs than the reference model respectively. While the loss of the reference model on the training dataset Leef is calculated on the full training dataset (as it has to be done only once), approximate Lpen by randomly drawn mini-batches for each update step. Therefore, the boundary condition might not satisfied on the full training set, even if the boundary condition is satisfied for the mini-batch estimate.
As described in the main paper, the resulting model of each adversarial model search used to define the location of a mixture component of sampling distribution ( (Eq. (6)). The epistemic uncertainty estimated by Eq. (4), using models sampled from this mixture distribution. The simplest choice of distributions for each mixture distribution is delta distribution at the location of the adversarial model zig. While this performs well empirically, we discard lot of information by not utilizing predictions of models obtained throughout the adversarial model search. The intermediate solutions the adversarial model search allow to assess how easily models with highly divergent predictive distributions to the reference model can be found. Furthermore, the expected mean squared N with the number of samples and the expected variance of the estimator (Eq. (38)) decreases with VN- Therefore, using more samples beneficial empirically, even though we potentially introduce a bias the estimator.
error (Eq. (5)) decreases with 1
Consequently, we utilize all sampled models during the adversarial model search empirical sampling distribution for our experiments. This is the same as how members of an ensemble can be seen an empirical sampling distribution [Gustafsson al., 2020] and conceptually similar to Snapshot ensembling [Huang et al., 2017]. To compute Eq. (4), we use the negative exponential training loss of each model to approximate its posterior probability p( in (D). Note that the training loss is the negative tog-likelihood which in turn is proportional to the posterior probability. Note
we temperature-scal the approximate posterior probability by p(u D with the temperature parameter T set as a hyperparameter.
C.2 Simplex Example
We sample the training dataset = {(0*,, y1)) from three Gaussian distributions (21 datapoints
(0, /2)" and the same two-dimensional covariance with ṭ² 1.5 on both entries of the diagonal and zero on the off oif-oiagsnals. The labels are one- hot encoded vectors, signifying which Gaussian the input
from each Gaussian) at locations = (-4.-2) μ2 = (4, 2)
was sampled from. The new test point æ we evaluate for is located at 1,, 2). To attain the likelihood
(a) HMC
(b) Adversarial Model Search
Training data test point
Figure C.2: Softmax outputs (black) of individual models of HMC (a) as well as their average output (red) on probability simplex. Softmax outputs models found throughout the adversarial model search (b). colored by the attacked class. Left, right and top corners denote 100% probability mass at the blue, orange and green class in (c) respectively. Models were selected on the training data, and evaluated on the new test point (red) depicted in (c). The background color denotes the maximum likelihood of the training data that achievable by model having equal softmax output as the respective location on the simplex.
28

---

for each position on the probability simplex, we train two-layer fully connected neural network (with parameters w) with hidden size of 10 on this dataset. We minimize the combined loss
K Σ L = K
Uply kk, w), uv) + Up((y) x. w)
(43)
where is the cross-entropy loss function and ÿ is the desired categorical distribution for the output of the network. We report the likelihood on the training dataset upon convergence of the training procedure for ý the probability simplex. To average over different initializations of w and alleviate the influence of potentially bad local minima, we use the median over 20 independent runs to calculate the maximum.
For all methods, we utilize the same two-layer fully connected neural network with hidden size of 10; for MC dropout we additionally added dropout with dropout probability 0.2 after every intermediate layer. We trained 50 networks for the Deep Ensemble results. For MC dropout we sampled predictive distributions using 1000 forward passes.
Fig. C.2 (a) shows models sampled using HMC, which is widely regarded as the best approximation to the ground truth for predictive uncertainty estimation. Furthermore, Fig. C.2 (b) shows models obtained by executing the adversarial model search for the given training dataset and test point depicted in Fig. C.2 (c). HMC also provides models that put more probability mass on the orange class. Those are missed by Deep Ensembles and MC dropout (see Fig. 2 (a) and (b)). The adversarial model search used by QUAM helps to identify those regions.
C.3 Epistemic Uncertainty on Synthetic Dataset
We create the two-moons dataset using the implementation of Pedregosa al. [2011]. All exper- iments were performed on a three-layer fully connected neural network with hidden size 100 and ReLU activations. For MC dropout, dropout with dropout probability of 0.2 was applied after the intermediate layers. We assume to have a trained reference model w of this architecture. Results of the same runs as in the main paper, but calculated for the epistemic uncertainty setting (b) (see Eq. (2)) are depicted in Fig. C.3. Again, QUAM matches the ground truth best.
Furthermore, conducted experiments on a synthetic regression dataset, where the input feature & is drawn randomly between [-T, π] and the target is = sin(z) + with e ~ NO 0.1). The results are depicted in Fig. C.4. As for the classification results, the estimate of QUAM is closest to the ground truth provided by HMC.
The HMC implementation of Cobb and Jalaian [2021] was used to obtain the ground truth epistemic uncertainties. For the Laplace approximation, we used the implementation of Daxberger et al. [2021]. For SG-MCMC used the python package of Kapoor [2023].
C.4 Epistemic Uncertainty Vision Datasets
Several vision datasets and their corresponding OOD datasets are commonly used for benchmarking predictive uncertainty quantification in the literature, e.g. in Blundell et al. [2015], Gal and Ghahra- mani [2016], Malinin and Gales [2018], Ovadia et al. [2019], van Amersfoort et al. [2020]. Mukhoti al. [2021], Postels et al. [2021] Band et al. [2022]. Our experiments focused on two of those: MNIST [LeCun et al., 1998] and its OOD derivatives as the most basic benchmark and ImageNet K [Deng al., 2009] to demonstrate our method's ability perform larger scale. Four types of experiments were performed: (i) OOD detection (ii) adversarial example detection, (iii) misclassi- fication detection and (iv) selective prediction. Our experiments on adversarial example detection did not utilize a specific adversarial attack on the input images, but natural adversarial examples [Hendrycks al., 2021], which are images from the ID classes, but wrongly classified by standard ImageNet classifiers. Misclassification detection and selective prediction was only performed for Imagenet1K, K, since MNIST classifiers easily reach accuracies of 99% on the test set, thus hardly misclassifying any samples. In all cases except selective prediciton, we measured AUROC, FPR at TPR of 95% and AUPR of classifying ID vs. OOD, non-adversarial vs. adversarial and correctly classified vs. misclassified samples (on ID test set), using the epistemic uncertainty estimate provided by the different methods. For selective prediction, we utilized the epistemic uncertainty estimate to select subset of samples on the ID test set.
29

---

(a) Ground Truth HMC
(b) cSG-HMC
(c) Laplace
(d) MC dropout
(e) Deep Ensembles
(i) Our Method QUAM
Figure C.3: Epistemic uncertainty as in Eq. (2). Yellow denotes high epistemic uncertainty. Purple denotes low epistemic uncertainty. The black lines show the decision boundary of the reference model w. HMC is considered to be the ground truth epistemic uncertainty. The estimate of QUAM closest to the ground truth. All other methods underestimate the epistemic uncertainty in the top left and bottom right corner, as all models sampled by those predict the same class with high confidence for those regions.
(a) Ground Truth
HMC
(b) cSG-HMC
(c) Laplace
(d) MC dropout
(c) Deep Ensembles
(f) Our Method QUAM
Figure C.4: Variance between different models found by different methods on synthetic sine dataset. Orange line denotes empirical mean of the averaged models, shades denote one, two and three standard deviations respectively. HMC is considered to be the ground truth epistemic uncertainty. The estimate of QUAM is closest the ground truth. All other methods fail capture the variance between points well the variance left outside the region ([-T,, x)) datapoints are sampled from.
30

---

Table C.I: Additional baseline MoLA: AUROC using the epistemic uncertainty of a given, pre- selected model as a score to distinguish between ID (MNIST) and OOD samples. Results for additional baseline method MoL.A, comparing to Laplace approximation, Deep Ensembles (DE) and QUAM. Results are averaged over three independent runs.
Dood FMNIST
Laplace
MoLA
DE
QUAM
.978..00
986420022
.988+.001
.994_ =
KMNIST
4.09
.984..000
.990+.001
.994_.001
EMNIST
.8771.011
1900
.9241.003
2.3720008
OMNIGLOT
.9634.003
.9794.000
.9834.001
.992..001
C.4.1 MNIST
OOD detection experiments were performed on MNIST with FaSHionMNIST (FMNIST) [Xiao et al., 2017], EMNIST [Cohen et al., 2017]. KMNIST [Clanuwat et al., 2018] and OMNIGLOT [Lake et al., 2015] as OOD datasets. In case of EMNIST, we only used the "letters" subset, thus excluding classes overlapping with MNIST (digits). We used the MNIST (test set) vs FMNIST (train set) OOD detection task to tune hyperparameters for all methods. The evaluation was performed using the complete test sets of the above-mentioned datasets (n 10000).
For each seed, a separate set of Deep Ensembles was trained. Ensembles with the size of 10 were found to perform best. MC dropout was used with a number of samples set to 2048. This hyperparameter setting was found to perform well. A higher sampling size would increase the performance marginally while increasing the computational load. Noteworthy is the fact, that with these settings the computational requirements of MC dropout surpassed those of QUAM. Laplace approximation was performed only for the last layer, due to the computational demand making it infeasible on the full network with our computational capacities. Mixture of Laplace approximations Eschenhagen et al. [2021] was evaluated as well using the parameters provided in the original work. Notably, the results from the original work suggesting improved performance compared to the Deep Ensembles on these tasks could not be reproduced. Comparison is provided in Table C.I. 1. SG-HMC was performed on the full network using the Python package from Kapoor [2023]. Parameters were set in accordance with those of the original authors [Zhang et al., 2020]. For QUAM, the initial penalty parameter found by tuning was G = 6, which was exponentially increased (C++1 = 770)) with n 2 every 14 gradient steps for a total of two epochs through the training dataset. Gradient steps were performed using Adam [Kingma and Ba, 2014] with learning rate of 5.e-3 and weight decay of 1.e-3, chosen equivalent to the original training parameters of the model. A temperature of 1.e-3 was used for scaling the cross-entropy loss, an approximation for the posterior probabilities when calculating Eq. (4). Detailed results and additional metrics and replicates of the experiments
be found in Tab. C.2. Experiments were performed three times with seeds: (42, 142, 242) to provide confidence intervals. Histograms of the scores on the ID dataset and the OOD datasets for different methods are depicted in Fig. C.5.
C.4.2 ImageNet
For ImageNetIK K [Deng et al., 2009], OOD detection experiments were performed with ImageNet-O (Hendryeks et al., 2021], adversarial example detection experiments with ImageNet-A [Hendrycks et al., 2021], and misclassification detection as well as selective prediction experiments on the official validation set of ImageNetlK. For each experiment, we utilized a pre-trained EfficientNet [Tan and Le, 2019] architecture with 21.5 million trainable weights available through Py Torch [Paszke et al., 2019], achieving a 1 accuracy of 84.2% as well as a top-5 accuracy of 96.9%.
cSG-HMC was performed on the last layer using the best hyperparameters that resulted from a hyperparameter search around the ones suggested by the original authors [Zhang et al., 2020]. The Laplace approximation with the implementation of [Daxberger et al., 2021] was not feasible to compute for this problem on our hardware, even only for the last layer. Similarly to the experiments in section 1,4.1, we compare against Deep Ensemble consisting of 10 pre-trained EfficientNet architectures ranging from 5.3 million to 66.3 million trainable weights (DE (all)). Also, we retrained the last layer of 10 ensemble members (DE (LL)) given the same base network. We also compare
31

---

Table C.2: Detailed results MNIST OOD detection experiments, reporting reprottin AUROC, AUPR and FPR TPR=95% for individual seeds.
OOD dataset
Method
Seed
AUPR
1 AUROC
+ FPR @ TPR=95%
42
0.8859
0.8823
0.5449
cSG-HMC
142
0.8714
0.8568
0.8543
242
0.8797
0.8673
0.7293
42
0.8901
0.8861
0.5273
Laplace
142
0.8762
0.8642
0.7062
242
0.8903
0.8794
0.6812
42
0.9344
0.9239
0.4604
EMNIST
Deep Ensembles
142
0,9325
0.9236
0,4581
242
0.9354
0.9267
0,4239
0.8854
0.8787
0.5636
MC dropout
142
0.8769
0.8630
0.6718
242
0.8881
0.8751
0.6855
42
0.9519
0.9454
0.3405
QUAM
142
0.9449
0.9327
0.4538
242
0.9437
0.9317
0.4325
42
0.9532
0,9759
0.0654
cSG-HMC
142
0.9610
0.9731
0.0893
242
0.9635
0.9827
0,0463
42
0.9524
0.9754
0,0679
Laplace
142
0.9565
0.9739
0.0788
242
0.9613
0.9824
0.0410
42
0.9846
0.9894
0.0319
FMNIST
Deep Ensembles
142
0.9776
0.9865
0.0325
242
0.9815
0.9881
0.0338
42
0.9595
0.9776
0.0644
MC dropout
142
0.9641
0.9748
0.0809
242
0.9696
0.9848
0.0393
42
0.9896
0.9932
0.0188
QUAM
142
0.9909
0.9937
0.0210
242
0.9925
0.9952
0,0132
0.9412
0.9501
0.2092
cSG-HMC
142
0.9489
0.9591
0.1551
242
0.9505
0.9613
0.1390
42
0.9420
0.9520
0.1915
Laplace
142
0.9485
0.9617
0.1378
242
0.9526
0.9640
0.1165
42
0.9885
0.9899
0.0417
KMNIST
Deep Ensembles
142
0.9875
0.9891
0.0458
242
0.9884
0.9896
0.0473
42
0.9424
0,9506
0,2109
MC dropout
142
0.9531
0.9618
0.1494
242
0.9565
0.9651
0.1293
0.9928
0.9932
0.0250
QUAM
142
0.9945
0.9952
0,0194
242
0.9925
0,9932
0.0260
42
0.9499
0.9658
0.1242
cSG-HMC
142
0.9459
0.9591
0.1498
242
0,9511
0.9637
0.1222
42
0,9485
0.9647
0.1238
Laplace
142
0,9451
0,9597
0.1345
242
0.9526
0.9656
0.1077
0.9771
0.9822
0.0621
OMNIGLOT
Deep Ensembles
142
0.9765
0.9821
0,0659
242
0.9797
0.9840
0,0581
42
0.9534
0.9663
0.1248
MC dropout
142
0.9520
0.9619
0.1322
242
0.9574
0.9677
0.1063
42
0.9920
0.9930
0.0274
QUAM
142
0.9900
0.9909
0.0348
242
0,9906
0.9915
0,0306
32

---

802
274
600
14000
0401100
1.00
5.0
1300
056077
064050
16/10
800
Figure C.5: MNIST: Histograms of uncertainty scores calculated for test set samples of the specified datasets.
33

---

Table C.3: Calibration: expected calibration error (ECE) based on the weighted average predictive distribution. Reference refers to the predictive distribution of the given, pre-selected model. Experi- ment was performed on three distinct splits, each containing 7000 ImageNer- K validation samples.
Reference
cSG-HMC
MCD
DE
QUAM
.1559..00 1.044
.3641.001
.166 1.004
11941.004
.096 1.006
00
08
00
82 34
64
00 04
16
03 06
64 C4
Carfernss
(a) Reference
(b) cSG-HMC
(c) MCD
(d) DE
(e) QUAM
Figure C.6: Calibration: confidence vs. accuracy based on (weighted) average predictive distribution
of different uncertainty quantification methods. Point size indicates number of samples in the bin.
against MC dropout used with 2048 samples with a dropout probability of 20%. The EfficientNet architectures utilize dropout only before the last layer. The adversarial model search for QUAM was performed on the last layer of the EfficientNet, which has 1.3 million trainable parameters. To enhance the computational efficiency, the output of the second-to-last layer was computed once for all samples, and this output was subsequently used as input for the final layer when performing the adversarial model search. We fixed co to 1 and exponentially updated it at every of the 256 update steps. Also, weight decay was fixed to 1.e-4 for the Adam optimizer [Kingma and Ba, 2014].
Two hyperparameters have jointly been optimized on ImageNet-O and ImageNet-A using a small grid search, with learning rate a E (5.e-3, 1.e-3, 5.e-4, L.e-4] and the exponential schedule update constant 7) E {1.15, 1.01, 1.005, 1.001]. The hyperparameters a 1.e-3 and 1) = 1.01 resulted in the overall highest performance and have thus jointly been used for each of the three experiments. This implies that cy increases by 1% after each update step. We additionally searched for the best temperature and the best number of update steps for each experiment separately. The best temperature for scaling the cross-entropy loss when calculating Eq. (4) was identified as 0.05, 0.005, and 0.0005, while the best number of update steps was identified as 50, 100, and 100 for ImageNet-O) OOD detection, ImageNet-A adversarial example detection, and ImageNet1K K misclassification detection, respectively. Selective prediction was performed using the same hyperparameters as misclassification detection. We observed that the adversarial model search is relatively stable with respect to these hyperparameters.
The detailed results on various metrics and replicates of the experiments can be found in C.4. Histograms of the scores on the ID dataset and the OOD dataset, the adversarial example dataset and the correctly and incorrectly classified samples are depicted in Fig. C.7 for all methods. ROC curves, as well as accuracy over retained sample curves, are depicted in Fig. C.8. To provide confidence intervals, we performed all experiments on three distinct dataset splits of the IDD datasets, matching the number of OOD samples. Therefore we used three times 2000 ID) samples for Imagenet-O) O and three times 7000 ID samples for Imagenet-A and misclassification detection as well as selective prediction.
Calibration. Additionally, we analyze the calibration of QUAM compared to other baseline methods. Therefore, we compute the expected calibration error (ECF) [Guo et al., 2017] on the ImageNet-1 K validation dataset using the expected predictive distribution. Regarding QUAM, the predictive distribution was optained using the same hyperparameters as for misclassification detection reported above. We find that QUAM improves upon the other considered baseline methods, although it was not directly designed to improve the calibration of the predictive distribution. Tab. C.3 states the ECE of considered uncertainty quantification methods and in Fig. C.6 the accuracy and number of samples (depicted by the size) for specific confidence bins is depicted.
34

---

Table C.4: Detailed results of ImageNet OOD detection, adversarial example detection and misclassi- fication experiments, reporting AUROC, AUPR and FPR @ TPR=95% for individual splits.
OOD dataset task
Method
Split
AUPR
AUROC
FPR @ TPR=95%
I
0.615
0.629
0.952
Reference
II
0.600
0.622
0,953
III
0.613
0.628
0.954
0.671
0.682
0.855
cSG-HMC
0.661
0.671
0.876
0.674
0.679
0.872
0.684
0.681
0.975
MC dropout
II
0.675
0.677
0.974
ImageNet-O
III
0.689
0.681
0,972
0.573
0.557
0.920
Deep Ensembles (LL)
II
0.566
0.562
0.916
III 0.573
0.566
0.928
0.679
0.713
0.779
Deep Ensembles (all)
II
0.667
0.703
0.787
III 0.674
0.710
0.786
0.729
0.758
0.766
QUAM
0.713
0.740
0.786
0.734
0.761
0.764
0.779
0.795
0.837
Reference
II 0.774
0.791
0.838
III
0.771
0.790
0.844
0.800
0.800
0.785
cSG-HMC
II 0.803
0.800
0.785
III 0.799
0.798
0.783
0.835
0.828
0.748
MC dropout
0.832
0.828
0.740
ImageNet-A
III 0.826
0.825
0.740
0.724
0.687
0.844
Deep Ensembles (LL.)
0.723
0.685
0.840
0.721
0.686
0.838
0.824
0.870
0.385
Deep Ensembles (all)
II
0.837
0.877
0.374
III
0,832
0.875
0.375
0.859
0.875
0.470
QUAM
II 0.856
0.872
0.466
III
0.850
0.870
0.461
0.623
0.863
0.590
Reference
0.627
0.875
0.554
0.628
0.864
0.595
I 0.478
0.779
0.755
cSG-HMC
0.483
0.779
0.752
III
0.458
0.759
0.780
0.514
0.788
0.719
MC dropout
II
0.500
0.812
0.704
Misclassification
III
0.491
0.788
0.703
I
0.452
0.665
0.824
Deep Ensembles (LL)
II
0.421
0.657
0.816
III 0.425
0.647
0.815
0.282
0.770
0.663
Deep Ensembles (all)
0.308
0.784
0.650
0.310
0.786
0.617
0.644
0,901
0.451
QUAM
0.668
0,914
0.305
III
0.639
0.898
0.399
35

---

212
1400
1706
QUAM
.02
600
203
400
40
183
leaplell 000
1750
1549
6-0
1250
300
1083
100
200
830
Inegeter
2000
DE
4:2
2009
1300
1803
240
0.6
39.3
10 CD
000
1000
TenseNI TM
.000
200
2000
VDD
2003
1003
503
1299
302
Inagilas
Imagee 000
800
301
50- BNC
0000
302
J42
100
Rervenso
200
400
30
203
300
Figure C.7: ImageNet: Histograms of uncertainty scores calculated for test set samples of the specified datasets.
36

---

1.0
100
0.8
0.8
True Positive E.800
0.4
CUAN IAUC: 0.75)
0.4
CUAN [AUC) O 0.87)
DE (al) (AUC: 0.71)
DE (AUC: 0.87)
[UJ] LAUC,
c5S HMC (AUC:0.68)
0.2
c5G HMC (AUC: 0.80)
Reference 0.631
Reference (AUC: 0.151
Rendor (AIIC: 0.50)
Rendar (AUIC: 0.50)
Positive
False Positive foae
(a) OOD detection
(b) Adversarial example detection
0.95
0.6
Accurscy
(al) 0.78)
CE IUL] (AUC) 0,58)
DE (L.] (AUC) 0.901
0.2
eSG-IIM
MCD (AUC: 0.80)
0.85
eSG-INE (ALIC 0.93)
MCD (AUC: 0.931
(AUIC
0.0
0.4 False Positive Rate
0.6
1.0
Q2
0.6
Percentage retained samples
(c) Misclassification
(d) Selective prediction
Figure C.8: ImageNet- IK OOD detection results ImageNet-O, adversarial example detection results on ImageNet-A, misclassification detection and selective prediction results on the validation dataset. ROC curves using the epistemic uncertainty of given, pre-selected model (as in Eq. (2)) to distinguish between (a) the ImageNet- IK validation dataset and ImageNet-O, (b) the ImageNet- KK validation dataset and ImageNet-A and (c) the reference model's correctly and incorrectly classified samples. (d) Accuracy of reference model on subset composed of samples that exhibit lowest epistemic uncertainty.
37

---

C,5 Comparing Mechanistic Similarity of Deep Ensembles vs. Adversarial Models
The experiments were performed on MNIST, EMNIST, and KMNIST test datasets, using 512 images of each using Deep Ensembles, and the reference model w. trained on MNIST. Results are depicted in Fig. C.9. For each image and each ensemble member, gradients were integrated over 64 steps from 64 different random normal sampled baselines for extra robustness [Sundararajan et al., 2017]. Since the procedure was also performed on the OOD) sets as well as our general focus on uncertainty estimation, no true labels were used for the gradient computation. Instead, predictions of ensemble members for which the attributions were computed were used as targets. Principal Component Analysis (PCA) was performed for the attributions of each image separately, where for each pixel the attributions from different ensemble members were treated as features. The ratios of explained variance, which are normalized to sum up one, are collected from each component. If all ensemble members would utilize mutually exclusive features for their prediction, all components would be weighted equally, leading to straight line in the plots in the top row in Fig. C.9. Comparatively high values of the first principal component to the other components in the top row plots in Fig. C.9 indicate low diversity in features used by Deep Ensembles.
The procedure was performed similarly for an ensemble of adversarial models. The main difference was that for each image an ensemble produced as a result of an adversarial model search on that specific image was used. We observe, that ensembles of adversarial models utilize more dissimilar features, indicated by the decreased variance contribution of the first principal component, This is especially strong for ID data, but also noticeable for OOD data.
C.6 Prediction Space Similarity of Deep Ensembles and Adversarial Models
In the following, ensembles members and adversarial models are analyzed in prediction space. We used the same Deep Ensembles as the one trained on MNIST for the OOD detection task described in Sec. C.4.1. Also, 10 adversarial models were retrieved from the reference model w and a single OOD sample (KMNIST), following the same procedure as described in Sec. C.4.1.
For the analysis, PCA was applied to the flattened softmax output vectors of each of the 20 models applied to ID validation data. The resulting points represent the variance of the model's predictions across different principal components [Fort et al., 2019]. The results in Fig. C.10 show, that the convex hull of blue points representing adversarial models, in general, is much bigger than the convex hull of orange points representing ensemble members across the first four principal components, which explain 99, 99.99% of the variance in prediction space. This implies that even though adversarial models achieve similar accuracy as Deep Ensembles on the validation set, they are capable of capturing more diversity in prediction space.
C.7 Computational Expenses
Experiments on Synthetic Datasets The example in Sec. C.2 was computed within half an hour on a GTX 1080 Ti. Experiments on synthetic datasets shown in Sec. C.3 were also performed on single GTX 1080 Ti. Note that the HMC baseline took approximately 14 hours on 36 CPU cores for the classification task. All other methods except QUAM finish within minutes. QUAM scales with the number of test samples. Under the utilized parameters and 6400 test samples, QUAM computation took approximately hours on a single GPU and under one hour for the regression task, where the number of test points is much smaller.
Experiments on Vision Datasets Computational Requirements for the vision domain experiments depend a lot on the exact utilization of the baseline methods. While Deep Ensembles can take long time to train, depending on the ensemble size, we utilized either pre-trained networks for ensembling or only trained last layers, which significantly reduces the runtime. Noteworthy, MC-dropout can result in extremely high runtimes depending on the number of forward passes and depending on the realizable batch size for inputs. The same holds for SG-HMC. Executing the QUAM experiments on MNIST (Sec. C.4.1) took a grand total of around 220 GPU-hours on a variety of mostly older generation and www-oower GPUs (P40, Titan V. T4), corresponding to roughly 4 GPU-scconds per sample. Executing the experiments on ImageNet (Sec. C.4.2) took about 100 GPU-hours on a mix of A100 and A40 GPUs, corresponding to around 45 GPU-scconds per sample. The experiments presented in C.c.C.5 and C.6 took around 2 hours each on 4 GTX 1080 Ti.
38

---

0.8
QUAM
0.8
QUAM
Deep Ensemble
Deep Ensemble
relative var contribution
0.2
0.2
0.0
0.0
0.1
0.1
0.0
0.0
-0.1
00.
contribution
0.2
0.2
-0.3
-0.4
-0.5
Figure C.9: The differences between significant component distribution are marginal on OOD data but pronounced the ID data. The ID data would be subject to optimization by gradient descent during training, therefore the features are learned greedily and models are similar to each other mechanistically. We observe, that the members Deep Ensembles show higher mechanistic similarity than the members of ensembles obtained from adversarial model search.
CUJA
Dees Encembres Pre-selected model
PC1
AC1
PC1
M.C.
TC2
PC2
PC3
Figure C.10: Convex hull the droj-projected softmax output from 10 Ensemble Members (orange) well 10 adversarial models (blue). PCA is used for down-projectie all combinations of the first four principal components (99.99% variance explained) are plotted against each other. Softmax outputs obtained batch of 10 random samples from the ID validation dataset. The black cross marks the given, pre-selected model w.
39


Reviews:
[{"review_id": "3hyGt5xjt9", "reviewer": "Reviewer_T3Ph", "summary": "\nThis paper is about uncertainty estimation using adversarial models (not examples!).  The authors propose a new uncertainty estimation method, called QUAM, which performs a search of an adversarial model, which is one that fits the training set but has predictions far away from the predefined model, with the idea that a point is uncertain if multiple models explain it very differently, while still fitting the original training set.\n\nQUAM has the potential to estimate epistemic uncertainty better than other uncertainty models, and experimental results point in this direction.\n\nContributions are:\n- The QUAM framework for uncertainty quantification using adversarial model search.\n- The new concept of an adversarial model, different from an adversarial example, which is a model (set of weights) that explains the training set well while having maximum difference with the orignal model in a new test point.\n- The proposed method can estimate epistemic uncertainty for any model, including a pretrained model, which is an advantage over usual uncertainty methods that require modifications to the training process or model retraining.", "questions": "Questions:\n- How were the baselines selected? The baseline UQ methods are not state of the art for epistemic uncertainty estimation. I already mentioned in weaknesses that DUQ has better epistemic uncertainty estimation in the two moons dataset.\n- In Figure 3f, QUAM has artifacts or non-smoothness in the uncertainty, in the top-left and bottom-right corners, while other models do not have such artifacts, is there an explanation or intuition about this?\n\nPost-rebutal, both of these questions have been answered to my satisfaction by the authors.\n\nSuggestions:\n- The DE (LL) setting is the same as the concept of sub-ensembles [2], maybe it could be mentioned or cited.\n- In Tables 1 and 2, the aleatoric uncertainty of the model is claimed to be used, but I am not sure what this means, as most models actually output predictive uncertainty, which is the combination of aleatoric and epistemic uncertainty. Even a model without any uncertainty quantification (i.e without using MC-Dropout, Ensembling, BNNs, etc) still has a non-zero degree of aleatoric and epistemic uncertainty, so I believe this claim should be changed to predictive uncertainty and not aleatoric uncertainty.\n\n[2]: Deep Sub-Ensembles for Fast Uncertainty Estimation in Image Classification\nby Matias Valdenegro-Toro. arXiv 1910.08168.", "limitations": "The paper properly discusses the limitations of the proposed method in the conclusion sections, including the fact that for each new test point, a model search has to be performed which is computationally very expensive.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is very well written and easy to understand.\n- The paper touches an important topic, uncertainty estimation methods fail in samples far from the training set, usually producing overconfident uncertainties that are not useful to detect out of distribution samples. This is well known from Ovadia et al and other papers in the literature.\n- This paper defines a new kind of meta-model, the adversarial model for uncertainty quantification, where to make a prediction in a new test point, new model parameters are found that fit the training set while providing alternative explanations for the new test point, producing higher quality epistemic uncertainty. This is of course very computationally expensive and it is mentioned as a limitation in the conclusions.\n- Experimental results show that QUAM outperforms the baselines on the task of out of distribution detection on MNIST vs FMNIST/KMINST/EMNIST/OMNIGLIT and  ImageNet-1K vs Imagenet-O/ImageNet-A.\n- I believe the proposed method QUAM is a good contribution to the field of uncertainty quantification, as it proposed a new framework for uncertainty estimation that is model agnostic and provides high quality epistemic uncertainty estimates.", "weaknesses": "- I have some serious doubts about the concept/interpretation of aleatoric uncertainty in this paper. In the literature and widely agreed concepts, aleatoric uncertainty is about the data, it is a property of the data, like stochasticity in measuring processes or noise, but the paper makes claims that \"aleatoric uncertainty is the stochasticity of the model and epistemic uncertainty is the uncertainty about model parameters.\" (Lines 39/40), here the  claim is made that aleatoric uncertainty is about stochasticity of the model, which I do not believe it is correct. Just as an example, there are non-stochastic models (like ensembles) that can estimate aleatoric uncertainty without using stochastic components. And I mention previously, aleatoric uncertainty is mainly a property of the data, not of the model, even though a model can estimate the aleatoric uncertainty of the data. I believe here the paper should clarify or simply fix the definition.\n\n~~- Another issue I have is about selection of baselines, in the two moons setting, DUQ [1] is a strong baseline that actually approximates epistemic uncertainty much closer than QUAM in the two moons dataset (see Figure 1 of the DUQ paper), and I believe this also points that an issue in epistemic uncertainty estimation is the model's inductive bias. DUQ uses a RBF layer to predict classes, based on distance to class centroids, unlike a separating hyperplane in a highly dimensional space, so epistemic uncertainty is very different. I believe in this paper DUQ and other single network uncertainty estimation method (like Direct Uncertainty Estimation, DDU), should also be compared.~~", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful assessment of our work. Regarding the explicitly stated remarks and questions:\n- **Aleatoric uncertainty:** Your remark is correct, aleatoric uncertainty is indeed a property of the data, stemming from the stochasticity / noise in the measurement process as we point out in line 29/30. Fundamentally, aleatoric uncertainty is due to the stochasticity of the conditional distribution $p(\\boldsymbol{y} \\mid \\boldsymbol{x})$ (often referred to as predictive distribution) that assigned labels to the samples in the dataset. However, we do not know the underlying conditional distribution, but use a model to approximate it, resulting in an approximation of the conditional distribution $p(\\boldsymbol{y} \\mid \\boldsymbol{x}, \\boldsymbol{w}) \\approx p(\\boldsymbol{y} \\mid \\boldsymbol{x})$. Therefore, we consider quantifying the aleatoric uncertainty as characterizing this stochasticity of the model, in accordance with e.g. [1] or [2]. Nevertheless, you are right that lines 39/40 should be improved. We suggest “Consequently, we consider uncertainty quantification as characterizing a stochastic model of the world. Here, aleatoric uncertainty stems from the stochasticity of the model and epistemic uncertainty from the variability of plausible model parameters.”, but would appreciate suggestions.\n- **Choice of baselines:** We certainly acknowledge the recent advances in single network uncertainty estimation like DDU [3], DUQ [4], DUE [5], SNGP [6], and others, and their ability to provide meaningful epistemic uncertainty estimates based on the location of new samples in the feature space. We also agree that they provide uncertainty estimates akin to those of HMC in the two-moon example. However, we did not compare our approach to these for the following reasons:\n  1) *Model Constraints:* The aforementioned single forward pass methods require the regularization of the feature space to assess uncertainty (either a two-sided gradient penalty or spectral normalization in the examples above), which has to be applied during training of the model. Therefore, it is not applicable under our newly introduced setting (b), where we assume a given, pre-selected model without any constraints of how it was obtained.\n  2) *Different Uncertainty Perspectives:* Single forward pass methods capture a different notion of epistemic uncertainty than Bayesian methods. They capture epistemic uncertainty through the location of new test samples in the latent feature space, whereas we consider capturing epistemic uncertainty through posterior integrals, thus through sampling different models. We solely claim that QUAM enhances the quantification of epistemic uncertainty for the specific notion given by the respective terms in equations (1) [setting (a)] and (2) [setting (b)] of the main paper. We thus selected the best known baselines that also aim to quantify this notion of uncertainty. A comprehensive exploration of the relation between these two different notions of epistemic uncertainty remains an exciting topic for future research, which hasn’t been deeply explored yet to the best of our knowledge.\n- **Artifacts / non-smoothness:** QUAM finds adversarial models for every input $\\boldsymbol{x}$. Thus, to create Figure 3, we applied our method to each point of an input-mesh used to show the uncertainty over the whole input space. For all other methods, we applied the same sampled models to the whole input-mesh at once, thus the higher smoothness. Note that if e.g. MC dropout is applied to each test input individually instead of to all of them at once, the same non-smoothness will be observed due to the different sampled models for different test points.\n\nThank you for pointing out the connection of the last layer ensemble to [7], we will reference it accordingly in the final version of the paper. \nRegarding the aleatoric uncertainty in Tables 1 and 2, we used the estimate of aleatoric uncertainty as in equation (2). This is the entropy of the predictive distribution under the single given, pre-selected model $\\mathrm{H}[p(\\boldsymbol{y} \\mid \\boldsymbol{x}, \\boldsymbol{w})]$ and therefore solely an estimate of aleatoric uncertainty. \n\nWe greatly appreciate your insightful feedback, which has helped us to refine and clarify our work. Should you have any further questions or need additional explanations, we look forward to provide them. Thank you for your time and consideration.\n\n---\n\n[1] Helton, J. C. (1997). Uncertainty and sensitivity analysis in the presence of stochastic and subjective uncertainty. journal of statistical computation and simulation, 57(1-4), 3-76.\n\n[2] Kendall, A., & Gal, Y. (2017). What uncertainties do we need in bayesian deep learning for computer vision?. Advances in neural information processing systems, 30.\n\n[3] Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H., & Gal, Y. (2021). Deep deterministic uncertainty: A simple baseline. arXiv preprint arXiv:2102.11582.\n\n[4] Van Amersfoort, J., Smith, L., Teh, Y. W., & Gal, Y. (2020, November). Uncertainty estimation using a single deep deterministic neural network. In International conference on machine learning (pp. 9690-9700). PMLR.\n\n[5] van Amersfoort, J., Smith, L., Jesson, A., Key, O., & Gal, Y. (2021). On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2102.11409.\n\n[6] Liu, J., Lin, Z., Padhy, S., Tran, D., Bedrax Weiss, T., & Lakshminarayanan, B. (2020). Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33, 7498-7512.\n\n[7]: Valdenegro-Toro, M. (2019). Deep sub-ensembles for fast uncertainty estimation in image classification. arXiv preprint arXiv:1910.08168."}, {"review_id": "J9wEQ2nJgW", "reviewer": "Reviewer_EcJb", "summary": "This paper introduces Quantification of Uncertainty with Adversarial Models (QUAM). Building on the claim that previous epistemic uncertainty estimation methods (e.g. MC dropout, Deep Ensembles) underestimate the epistemic uncertainty by only considering the posterior distribution when sampling models, the authors introduce an Adversarial Model Search algorithm to identify models located within a posterior mode but with different predictive distributions compared to the reference model.", "questions": "Overall, I find the paper unclear and lacking some important details and ablations. I believe the contribution to be relevant, but the paper would benefit from rewriting for clarity and from additional experiments based on the comments in the Weaknesses section.\n\nI would change my opinion if the authors would convincingly address my comments in the Weaknesses section and, most importantly, optimize the paper for clarity. Providing the definition of adversarial models only at page 6 makes very hard understanding the first 5 pages.", "limitations": "The authors pointed out the limitations of their work.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "Previous epistemic uncertainty estimation methods (i) require changing the training procedure to account for uncertainty estimation by e.g. adding dropout layers or training multiple models, (ii) underestimate the epistemic uncertainty by only sampling models from the posterior distribution. \n\nThe proposed method QUAM addresses (i) by searching for adversarial models at test time for each test sample, thus working with any pre-trained model. Moreover, it solves (ii) by designing an adversarial model search algorithm that  allows to cover more posterior modes by identifying models located within a posterior mode but with different predictive distributions compared to the reference model.\n\n", "weaknesses": "## Lack of clarity\nAlthough the method is interesting and addresses some important limitations of previous epistemic uncertainty estimation approaches, the paper writing should be revised.\n\nToo often important details are omitted in the main paper or disclosed too late. In particular, \n1. **Definition of adversarial models.** the definition of adversarial models introduced in this paper is completely new and has nothing to do with previous definitions of generative adversarial models [a, b] or adversarial attacks [c]. However, the first definition of what the authors mean by adversarial models is provided only at page 6, making it very hard to understand the paper before reading line 226. This paper would **greatly benefit** from earlier definitions of adversarial models in both abstract and introduction. Moreover, writing \"(not adversarial examples!)\" is the least elegant solution.\n2. **Missing details.** Implementation details are solely reported in the supplement, but some details are essential in the main paper for better understanding. In particular, the number of adversarial models, number of MC samples, and the number of models in the deep ensemble should be clearly stated in the main paper.\n3. **Settings (a) and (b)** It is not clear from the writing how the settings (a) and (b) defined in Section 2 relate to the experiments and whether the proposed QUAM presents any significant advantage on either task in particular compared to previous methods.\n\n## Unsupported claims\nSome claims are unsupported. They should either be supported by quantitative or theoretical analysis in the paper or some reference to other papers should be provided. In particular,\n1. The claim that previous epistemic uncertainty estimation methods underestimate the uncertainty (line 59, line 143) is not backed up by empirical results or references to other papers. \n2. \"Adversarial models are characterized by a large value of the integrand of the integral that defines he epistemic uncertainty\" (line 65). Without a definition of adversarial models in the introduction, this claim is not possible at all to understand.\n\n## Missing ablation on number of adversarial models\nThe main paper does not specify the number of adversarial models learned through algorithm 1. According to the supplement, it seems that 10 are used. The paper would benefit from an ablation on the number of adversarial models\n\n## Missing explanation of why the search algorithm does not always converge to the same solution\nThe search algorithm is not constrained to avoid finding always the same solution. Since it is an optimization problem, it is likely that similar solutions to the same problem are found if starting from the same initialization. The paper would benefit from an explanation of why the search does not converge always to the same adversarial model. Perhaps, enforcing that solutions must be different could improve the performance.\n\n## Missing ablation on trade-off inference speed vs. uncertainty estimation performance at different number of samples\nAs pointed out by the authors, the proposed method sacrifices inference speed for uncertainty estimation performance. Although this undermines the practicality of the proposed approach, the theoretical insights gained from QUAM could outweigh this limitation.\n\nHowever, an ablation on the trade-off between inference speed and uncertainty estimation performance at different number of samples for MC dropout, Deep Ensembles and QUAM is needed. I would expect MC dropout to be significantly more efficient for low number of samples, still obtaining reasonable performance at low inference time. Moreover, QUAM should have a higher uncertainty estimation performance upper bound than the competitors, if computational resources allow to tolerate the high inference time. This comparison would allow practitioners to understand which method to use based on their computation budget and needs.\n\n\n## References\n[a] Wang, Kunfeng, et al. \"Generative adversarial networks: introduction and outlook.\" IEEE/CAA Journal of Automatica Sinica 4.4 (2017): 588-598.\n[b] Li, Guofu, et al. \"Security matters: A survey on adversarial machine learning.\" arXiv preprint arXiv:1810.07339 (2018).\n[c] Chakraborty, Anirban, et al. \"Adversarial attacks and defences: A survey.\" arXiv preprint arXiv:1810.00069 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for this thoughtful feedback and critical assessment of our work, as well as the concrete suggestions to improve clarity. We will change the manuscript according to the reviewer’s suggestions for the final version as follows: \n\n- **Clarity:** \n1) *Definition of adversarial models:* It is true that adversarial models are formally defined only at page 6, although they are a novel concept. In the abstract and the introduction, we only loosely define adversarial models as having both a high posterior as well as a high divergence between their predictions and that of a reference model, consequently being counterexamples of the reference model, predicting differently for a new input while explaining the training data equally well (Lines 12-13 and 65-70). This is because we deliberately have chosen to first focus on presenting the problem, only then followed by the solution, namely the search for adversarial models. However, to improve clarity, we will refine and expand these early definitions and add the formal definition to the introduction (after Line 70) in the final version of the paper.\n2) *Missing details:* Thank you for pointing this out. We will move the number of adversarial models, number of MC samples, and the number of models in the Deep Ensemble from the appendix to the end of section 3 as a new subsection to further improve clarity.\n3) *Settings (a) and (b):* QUAM is a method to estimate the integral defining the epistemic uncertainty in both setting (a) and (b), which are posterior expectations of divergences between predictive distributions. We evaluated the synthetic dataset experiments on both settings (a) and (b). Regarding the large scale experiments in the vision domain, we focused on our newly introduced setting (b). This is stated in the first sentence introducing this set of experiments (line 278/279) and in the table headers.\n- **Unsupported claims:** \n1) Thank you for raising this point, we added references to similar claims in the literature as follows: [1] points out that adequate sampling of those posterior regions that make the greatest contribution to the posterior expectation of the predictive distribution are of greater interest for the purpose of estimating this posterior expectation than obtaining a generally good posterior representation. Those regions are the ones with the highest KL divergence in our setting (b) that only QUAM explicitly searches for. Similarly, [2] claims that Deep Ensembles do not provide enough functional diversity; we argue the same applies to MC dropout. [3] claims this is due to gradient descent always finding similar “easy” solutions.\nEmpirically we investigated this issue on a toy dataset, results are shown in Figure 2, the ground truth (HMC) to compare to is shown in Figure C.2 in the appendix. Furthermore, we observe that epistemic uncertainty is underestimated in the experiments on the two-moon dataset (Figure 3) when considering a test point that is for instance in the upper left or lower right corner. Similarly, we find that epistemic uncertainty is underestimated for inputs outside the range of data points (Figure C.4 in the appendix) in a regression task.\n2) We did not perceive this as a claim but solely as a loose version of the definition of adversarial models. Nonetheless, you are right that it is hard to understand at this stage in the paper. We will improve it by including the definition in the introduction as suggested before.\n- **Number of adversarial models and convergence:** We use 10 adversarial models for the MNIST experiments and 10 to 1000 for the ImageNet-1k experiments (see ablation study below). As you pointed out, directly maximizing the KL-divergence to the prediction of the reference model would indeed converge to very similar solutions, which we empirically observed during early experiments. Therefore, we instead minimize the cross-entropy towards one out of all possible classes at a time. This way we attain diverse solutions that, as you pointed out, lead to improved performance. This is discussed in detail in appendix section C.1. We will move these implementational details to the main paper at the end of section 3 as a new subsection to make it more explicit.\n- **Ablation on inference speed vs. performance:** QUAM requires updating the last layer for up to 100 steps for ImageNet-1k OOD detection, which is equivalent to 15 forward passes on the full network required by MC dropout (assuming that the last layer is 5\\% of all parameters and that the backward pass is twice as expensive as the forward pass). In a new ablation study, we searched for adversarial models only on a subset of classes based on the highest softmax probabilities assigned by the given, pre-selected model. The results are listed in Table 1 and Figure 1 of the PDF document attached to the global answer. For instance, when searching adversarial models only for the 10 most probable classes (QUAM$\\_{top1\\\\%}$), the number of full forward passes per sample is reduced from 15k (QUAM$\\_{all}$) to 150, an inference speed reduction by a factor of 100. Still, QUAM outperforms MC dropout in terms of performance. Also, training a single additional ensemble member requires more computational cost than evaluating all ImageNet-1k OOD samples with QUAM. We will provide more details on inference speed vs. performance in the final version of the paper.\n\nWe hope to have addressed your open questions and concerns. Your insightful review has guided us in enhancing the clarity of our paper, and we believe these changes will contribute positively to the overall impact of our work.\n\n---\n[1] Wilson, A. G., & Izmailov, P. (2020). Bayesian deep learning and a probabilistic perspective of generalization. NeurIPS.\n\n[2] D'Angelo, F., & Fortuin, V. (2021). Repulsive deep ensembles are bayesian. NeurIPS.\n\n[3] Parker-Holder, J., & ... & Foerster, J. (2020). Ridge rider: Finding diverse solutions by following eigenvectors of the hessian. NeurIPS.\t"}, {"review_id": "yawqChCuWT", "reviewer": "Reviewer_DS26", "summary": "The paper introduces Quantification of Uncertainty with Adversarial Models (QUAM), a novel approach for epistemic uncertainty estimation in deep learning. Well-known uncertainty quantification approaches, such as Deep Ensembles or variational inference, underestimate the epistemic uncertainty by sampling from posterior modes found by gradient descent, which might prevent some modes to be found due to the shape of the loss landscape. To overcome this limitation, QUAM introduces adversarial models: plausible models with a high posterior that differ in the prediction from a reference one. Mixture importance sampling, with the adversarial models as modes, is then used to estimate the epistemic uncertainty. Experiments on both synthetic and vision data confirmed the validity of the approach in improving epistemic uncertainty estimation.", "questions": "I have just two minor observations:\n- The notation used in equation 1 (like the mutual information) was introduced only in the Appendix.\n- Figure C.1 (Appendix): maybe wrong colour assignment of the adversarial model (yellow instead of blue).", "limitations": "The authors addressed potential negative societal impact of their work in the Appendix.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is very well-written, all the design choices are motivated by references to existing work, and the experiments are reproducible and robust. The topic is of great significance since in relates to important topics such as Out-Of-Distribution detection and Responsible AI, as also stated by the authors in the Societal Impact Statement. Overall, I really liked reading this work and I rate it as a strong accept.", "weaknesses": "I have no major remarks against this version of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for this very positive assessment of our work. \n\nIndeed we missed out on formally introducing the symbols for the cross-entropy, the KL-divergence and the mutual information in equation (1) in the main paper. Thank you for pointing out we will correct this in the final version of the paper.\n\nFurthermore, thank you for pointing out the wrong color assignment in the caption of Figure C.1 in the appendix. We changed the color of the adversarial model part from yellow to blue for better readability when preparing the final version of the supplementary material, but did not change the caption accordingly. We will correct that in the final version of the paper.\n"}, {"review_id": "jSgwrpTknv", "reviewer": "Reviewer_rwwy", "summary": "The authors propose a new method for estimating epistemic uncertainty. They propose to adversarially search for modes of the posterior distribution. Empirically, they demonstrate that their uncertainty performs well on OOD detection.", "questions": "see weaknesses", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The empirical results of this approach on OOD detection are very promising.", "weaknesses": "- The method performs well on ood detection. Is the resulting uncertainty also well calibrated?\n- l60 : The argument as to why ensembles and BNNs underperform is not clear to me. The authors say that they miss important posterior modes. Caen the authors elaborate on this? \n- L142: can the authors demonstrate that the current methods underestimate uncertainty? Just because some methods dont explore all modes doesn mean that they underestimate uncertainty. E.g. they may overestimate the width of other modes.\n- Lack of comparison with other ensemble methods, e.g. [1]. [1] in particular shows similar results on the two moon example. This makes me wonder whether the main impact of the method comes from the mixture of gaussians, as done in [1], or from the adversarial search. I guess it would be important to quantify that.\n\n[1] Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for this assessment of our work. Regarding the stated weaknesses:\n- **Calibration:** Thank you for proposing this interesting direction. Your intuition was right, we found that our method indeed improves upon the other considered baseline methods, although it was not directly designed to improve calibration of the predictive distribution. The detailed results measuring the expected calibration error (ECE) and the calibration plots for the ImageNet-1K validation dataset are given in Table 2 and Figure 2 of the PDF document attached to the global answer. The results for QUAM were obtained using the same hyperparameters as for misclassification detection reported in the appendix of the main paper. Future work should explore methods based upon QUAM that specifically target to improve the calibration.\n- **Why do prior methods underestimate epistemic uncertainty:** In line 60 we raise the concern that Deep Ensembles and MC dropout fail to capture important posterior modes and thus underestimate epistemic uncertainty. This is inspired by [1], where the authors raise attention to the fact that “Ultimately, the goal is to accurately compute the predictive distribution in Eq. (1), rather than find a generally accurate representation of the posterior. In particular, we must carefully represent the posterior in regions that will make the greatest contributions to the BMA integral.”. The regions that contribute most to the posterior integrals defining the epistemic uncertainty in equations (1) and (2) in our paper are those where predictions differ a lot to the reference model, thus exhibit high KL-divergence. QUAM explicitly searches for those, while Deep Ensembles and MC dropout do not have an explicit mechanism to enforce variety in the predictive distributions on a new test sample, resulting in low values of KL-divergences. An additional reference in the literature would be [2], which also discusses the problem of functionally similar ensemble members for estimating the BMA predictive distribution. As pointed out also in [3], the issue is that Deep Ensembles maximize the posterior probability of the ensemble models through gradient descent, which is prone to yield functionally similar “easy” solutions. We are thus not concerned that prior methods do not find all modes, but modes that yield functionally different solutions that predict differently, thus underestimate the epistemic uncertainty. The same arguments apply also to the dropout models used in MC dropout.\n- **Empirical evidence that prior methods underestimate epistemic uncertainty:** This issue is empirically exemplified for Deep Ensembles and MC dropout on a toy dataset, where results are shown in Figure 2 in the main paper. They underestimate the epistemic uncertainty due to their lack of finding functionally different solutions that yield different predictions, compared to the ground truth (HMC) shown in Figure C.2 in the appendix. Furthermore, we observe the same issue in the experiments on the two-moon dataset (Figure 3) when considering a test point that is for instance in the upper left or lower right corner, compared to the ground truth (HMC). Here we directly observe that the epistemic uncertainty is too low. Similarly, we find that epistemic uncertainty is underestimated for inputs outside the range of data points (Figure C.4 in the appendix) in a regression task.\n- **Lack of comparison to other ensemble methods:** Thank you for pointing us to this interesting paper [4]. Unfortunately, the authors did not provide the original code for their proposed method MoLA. We gave our best effort to reproduce their results on the two-moon example using the reported pipeline, but found the results to only marginally differ from the single network Laplace approximation reported in our paper that uses the same implementation for the Laplace approximation as [4]. However, we also evaluated our reimplementation of MoLA on the MNIST OOD detection task. The results are stated in Table 3 in the attached pdf to our global answer. We found that MoLA improves upon the Laplace approximation of a single model, but leads to worse performance than just using the underlying Deep Ensemble and consequently does not outperform QUAM as well. We will release an updated version of our code including this new baseline and add the additional results to Table 1 for the final version of the paper.\n\nThank you once again for your thoughtful assessment and valuable feedback. We have worked diligently to address your concerns, and we hope that our responses clarify the issues you raised. Please do not hesitate to reach out if you have any further questions or require additional information.\n\n---\n\n[1] Wilson, A. G., & Izmailov, P. (2020). Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems, 33, 4697-4708.\n\n[2] D'Angelo, F., & Fortuin, V. (2021). Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems, 34, 3451-3465.\n\n[3] Parker-Holder, J., Metz, L., Resnick, C., Hu, H., Lerer, A., Letcher, A., ... & Foerster, J. (2020). Ridge rider: Finding diverse solutions by following eigenvectors of the hessian. Advances in Neural Information Processing Systems, 33, 753-765.\n\n[4] Eschenhagen, R., Daxberger, E., Hennig, P., & Kristiadi, A. (2021). Mixtures of Laplace approximations for improved post-hoc uncertainty in deep learning. arXiv preprint arXiv:2111.03577."}]

Author rebuttal:
We thank all reviewers again for the time and effort they have invested in order to provide their high quality feedback. 


All reviewers found our approach novel and relevant, and acclaimed the general applicability of the method as well as the empirical performance. Nevertheless, reviewers were not unanimous about the paper's clarity of writing and requested additional ablations and evaluations, as well as further baselines for our OOD detection experiments.


We are confident to have addressed all concerns in the individual responses and will incorporate them in the final version of the paper. To summarize:

- We have supported the claim that prior methods underestimate uncertainty by citing several papers and further elaborated on the empirical evidence provided in our paper.
- We have improved the clarity of writing based on the reviewers feedback, e.g. by shifting implementational details from the appendix to the main paper and introducing the formal definition of adversarial models earlier. 
- We have performed three additional experiments/ablations (see attached PDF document) that further confirm the strong performance of QUAM:
  1) Table 1 and Figure 1: We compared QUAM to other baselines in terms of inference speed (*as requested by reviewer EcJb*). We discovered that the inference time of QUAM can be reduced by a factor of 100, while outperforming baselines like MCD in both performance and speed.
  2) Table 2 and Figure 2: We evaluated QUAM in terms of expected calibration error (*as requested by reviewer rwwy*). Although QUAM was not directly designed to be well calibrated, we discovered that it outperforms our baseline methods in being able to provide a more calibrated output prediction. 
  3) Table 3: We added a new baseline (MoLA) to our OOD detection experiment (*as requested by reviewer rwwy*). QUAM outperforms the proposed method in OOD detection on the FMNIST, KMNIST, EMNIST and OMNIGLOT test datasets. 

We hope that this clarifies all questions and concerns and that the additional experiments provide convincing evidence of the effectiveness of our method. Thank you for your time and efforts!


Meta-review: This paper receives 4 reviews,  including 1 strong accept, 1 accept, 1 reject, and 1 borderline reject, with an average rating of 5.5.  Reviewers find the idea of uncertainty estimation using adversarial search model novel, with improved performance over current epistemic uncertainty estimation methods.  Before rebuttal, they also identify some deficiencies with this work.  Their main concerns include unclear presentation, missing ablation study, missing important details, and lack of comparison with other ensemble methods.  The authors rebuttal was effective, largely addressing reviewers concerns.  Some minor concerns remain.  The authors are strongly advised to carefully revise the paper as per reviewer comments, in particular improving its presentation clarity and strengthening the experiments.
Decision: Accept (poster)