SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kajetan Schweighofer* Lukas Aichberger* Mykyta Ielanskyi* Günter Klambauer Sepp Hochreiter
ELLIS Unit Linz and LIT AI Lab, Institute for Machine Learning, Johannes Kepler University Linz, Austria *Joint first authors
Quantifying uncertainty is important for actionable predictions in real-world applications. A crucial part of predictive uncertainty quantification is the estimation of epistemic uncertainty, which is defined as an integral of the product between a divergence function and the posterior. Current methods such as Deep Ensembles or MC dropout underperform at estimating the epistemic uncertainty, since they primarily consider the posterior when sampling models. We suggest Quantification of Uncertainty with Adversarial Models (QUAM) to better estimate the epistemic uncertainty. QUAM identifies regions where the whole product under the integral is large, not just the posterior. Consequently, QUAM has lower approximation error of the epistemic uncertainty compared to previous methods. Models for which the product is large correspond to adversarial models (not adversarial examples!). Adversarial models have both a high posterior as well as a high divergence between their predictions and that of a reference model. Our experiments show that QUAM excels in capturing epistemic uncertainty for deep learning models and outperforms previous methods on challenging tasks in the vision domain.
Actionable predictions typically require risk assessment based on predictive uncertainty quantification [Apostolakis, 1991]. This is of utmost importance in high stake applications, such as medical diagnosis or drug discovery, where human lives or extensive investments are at risk. In such settings, even a single prediction has far-reaching real-world impact, thus necessitating the most precise quantification of the associated uncertainties. Furthermore, foundation models or specialized models that are obtained externally are becoming increasingly prevalent, also in high stake applications. It is crucial to assess the robustness and reliability of those unknown models before applying them. Therefore, the predictive uncertainty of given, pre-selected models at specific test points should be quantified, which we address in this work.
We consider predictive uncertainty quantification (see Fig. 1) for deep neural networks [Gal, 2016, Hüllermeier and Waegeman, 2021]. According to Vesely and Rasmuson [1984], Apostolakis [1991], Helton [1993], McKone [1994], Helton [1997], predictive uncertainty can be categorized into two types. First, aleatoric (Type A, variability, stochastic, true, irreducible) uncertainty refers to the variability when drawing samples or when repeating the same experiment. Second, epistemic (Type B, lack of knowledge, subjective, reducible) uncertainty refers to the lack of knowledge about the true model. Epistemic uncertainty can result from imprecision in parameter estimates, incompleteness in modeling, or indefiniteness in the applicability of the model. While aleatoric uncertainty cannot be reduced, epistemic uncertainty can be reduced by more data, better models, or more knowledge about the problem. We follow Helton [1997] and consider epistemic uncertainty as the imprecision or variability of parameters that determine the predictive distribution. Vesely and Rasmuson [1984]
Figure 1: Adversarial models. For the red test point, the predictive uncertainty is high as it is far from the training data. High uncertainties are detected by different adversarial models that assign the red test point to different classes, although all of them explain the training data equally well. As a result, the true class of the test point remains ambiguous.
calls this epistemic uncertainty "parameter uncertainty", which results from an imperfect learning algorithm or from insufficiently many training samples. Consequently, we consider predictive uncertainty quantification as characterizing a probabilistic model of the world. In this context, aleatoric uncertainty refers to the inherent stochasticity of sampling outcomes from the predictive distribution of the model and epistemic uncertainty refers to the uncertainty about model parameters.
Current uncertainty quantification methods such as Deep Ensembles [Lakshminarayanan et al., 2017] or Monte-Carlo (MC) dropout [Gal and Ghahramani, 2016] underperform at estimating the epistemic uncertainty [Wilson and Izmailov, 2020, Parker-Holder et al., 2020, Angelo and Fortuin, 2021], since they primarily consider the posterior when sampling models. Thus they are prone to miss important posterior modes, where the whole integrand of the integral defining the epistemic uncertainty is large. We introduce Quantification of Uncertainty with Adversarial Models (QUAM) to identify those posterior modes. QUAM searches for those posterior modes via adversarial models and uses them to reduce the approximation error when estimating the integral that defines the epistemic uncertainty.
Adversarial models are characterized by a large value of the integrand of the integral defining the epistemic uncertainty. Thus, they considerably differ to the reference model's prediction at a test point while having a similarly high posterior probability. Consequently, they are counterexamples of the reference model that predict differently for a new input, but explain the training data equally well. Fig. 1 shows examples of adversarial models which assign different classes to a test point, but agree on the training data. A formal definition is given by Def. 1. It is essential to note that adversarial models are a new concept that is to be distinguished from other concepts that include the term 'adversarial' in their naming, such as adversarial examples [Szegedy et al., 2013, Biggio et al., 2013], adversarial training [Goodfellow et al., 2015], generative adversarial networks [Goodfellow et al., 2014] or adversarial model-based RL [Rigter et al., 2022].
- We introduce QUAM as a framework for uncertainty quantification. QUAM approximates the integral that defines the epistemic uncertainty substantially better than previous methods, since it reduces the approximation error of the integral estimator.
- We introduce the concept of adversarial models for estimating posterior integrals with non-negative integrands. For a given test point, adversarial models have considerably different predictions than a reference model while having similarly high posterior probability.
- We introduce a new setting for uncertainty quantification, where the uncertainty of a given, pre-selected model is quantified.
2 Current Methods to Estimate the Epistemic Uncertainty
Definition of Predictive Uncertainty. Predictive uncertainty quantification is about describing a probabilistic model of the world, where aleatoric uncertainty refers to the inherent stochasticity of sampling outcomes from the predictive distribution of the model and epistemic uncertainty refers to the uncertainty about model parameters. We consider two distinct settings of predictive uncertainty quantification. Setting (a) concerns with the predictive uncertainty at a new test point expected under all plausible models given the training dataset [Gal, 2016, Hüllermeier and Waegeman, 2021]. This definition of uncertainty comprises how differently possible models predict (epistemic) and how
confident each model is about its prediction (aleatoric). Setting (b) concerns with the predictive uncertainty at a new test point for a given, pre-selected model. This definition of uncertainty comprises how likely this model is the true model that generated the training dataset (epistemic) [Apostolakis, 1991, Helton, 1997] and how confident this model is about its prediction (aleatoric).
As an example, assume we have initial data from an epidemic, but we do not know the exact infection rate, which is a parameter of a prediction model. The goal is to predict the number of infected persons at a specific time in the future, where each point in time is a test point. In setting (a), we are interested in the uncertainty of test point predictions of all models using infection rates that explain the initial data. If all likely models agree for a given new test point, the prediction of any of those models can be trusted, otherwise we can not trust the prediction regardless of which model is selected in the end. In setting (b), we have selected a specific infection rate from the initial data as parameter for our model to make predictions. We refer to this model as the given, pre-selected model. However, we do not know the true infection rate of the epidemic. All models with infection rates that are consistent with the initial data are likely to be the true model. If all likely models agree with the given, pre-selected model for a given new test point, the prediction of the model can be trusted.
2.1 Measuring Predictive Uncertainty
We consider the predictive distribution of a single model  $p(\boldsymbol{y} \mid \boldsymbol{x}, \boldsymbol{w})$ , which is a probabilistic model of the world. Depending on the task, the predictive distribution of this probabilistic model can be a categorical distribution for classification or a Gaussian distribution for regression. The Bayesian framework offers a principled way to treat the uncertainty about the parameters through the posterior  $p(\boldsymbol{w} \mid \mathcal{D}) \propto p(\mathcal{D} \mid \boldsymbol{w})p(\boldsymbol{w})$  for a given dataset  $\mathcal{D}$ . The Bayesian model average (BMA) predictive distribution is given by  $p(\boldsymbol{y} \mid \boldsymbol{x}, \mathcal{D}) = \int_{\mathcal{W}} p(\boldsymbol{y} \mid \boldsymbol{x}, \tilde{\boldsymbol{w}}) p(\tilde{\boldsymbol{w}} \mid \mathcal{D}) \mathrm{d}\tilde{\boldsymbol{w}}$ . Following Gal [2016], Depeweg et al. [2018], Smith and Gal [2018], Hülmermeier and Waegeman [2021], the uncertainty of the BMA predictive distribution is commonly measured by the entropy  $\mathrm{H}[p(\boldsymbol{y} \mid \boldsymbol{x}, \mathcal{D})]$ . It refers to the total uncertainty, which can be decomposed into an aleatoric and an epistemic part. The BMA predictive entropy is equal to the posterior expectation of the cross-entropy  $\mathrm{CE}[\cdot, \cdot]$  between the predictive distribution of candidate models and the BMA, which corresponds to setting (a). In setting (b), the cross-entropy is between the predictive distribution of the given, pre-selected model and candidate models. Details about the entropy and cross-entropy as measures of uncertainty are given in Sec. B.1.1 in the appendix. In the following, we formalize how to measure the notions of uncertainty in setting (a) and (b) using the expected cross-entropy over the posterior.
Setting (a): Expected uncertainty when selecting a model. We estimate the predictive uncertainty at a test point  $\pmb{x}$  when selecting a model  $\tilde{\pmb{w}}$  given a training dataset  $\mathcal{D}$ . The total uncertainty is the expected cross-entropy between the predictive distribution of candidate models  $p(\pmb{y} \mid \pmb{x}, \tilde{\pmb{w}})$  and the BMA predictive distribution  $p(\pmb{y} \mid \pmb{x}, \mathcal{D})$ , where the expectation is with respect to the posterior:
$$
\begin{array}{l} \int_ {\mathcal {W}} \operatorname {C E} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}), p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} = \mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) ] \tag {1} \\ = \int_ {\mathcal {W}} \mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} + \mathrm {I} [ Y; W \mid \boldsymbol {x}, \mathcal {D} ] \\ = \underbrace {\int_ {\mathcal {W}} \mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x} , \tilde {\boldsymbol {w}}) ]   p (\tilde {\boldsymbol {w}} \mid \mathcal {D})   \mathrm {d} \tilde {\boldsymbol {w}}} _ {\text {a l e a t o r i c}} + \underbrace {\int_ {\mathcal {W}} \mathrm {D} _ {\mathrm {K L}} (p (\boldsymbol {y} \mid \boldsymbol {x} , \tilde {\boldsymbol {w}}) \| p (\boldsymbol {y} \mid \boldsymbol {x} , \mathcal {D}))   p (\tilde {\boldsymbol {w}} \mid \mathcal {D})   \mathrm {d} \tilde {\boldsymbol {w}}} _ {\text {e p i s t e m i c}}. \\ \end{array}
$$
The aleatoric uncertainty characterizes the uncertainty due to the expected stochasticity of sampling outcomes from the predictive distribution of candidate models  $p(\boldsymbol{y} \mid \boldsymbol{x}, \tilde{\boldsymbol{w}})$ . The epistemic uncertainty characterizes the uncertainty due to the mismatch between the predictive distribution of candidate models and the BMA predictive distribution. It is measured by the mutual information  $\mathrm{I}[\cdot ; \cdot]$ , between the prediction  $Y$  and the model parameters  $W$  for a given test point and dataset, which is equivalent to the posterior expectation of the KL-divergence  $\mathrm{D}_{\mathrm{KL}}(\cdot \| \cdot)$  between the predictive distributions of candidate models and the BMA predictive distribution. Derivations are given in appendix Sec. B.1.
Setting (b): Uncertainty of a given, pre-selected model. We estimate the predictive uncertainty of a given, pre-selected model  $\pmb{w}$  at a test point  $\pmb{x}$ . We assume that the dataset  $\mathcal{D}$  is produced according to the true distribution  $p(\pmb{y} \mid \pmb{x}, \pmb{w}^*)$  parameterized by  $\pmb{w}^*$ . The posterior  $p(\tilde{\pmb{w}} \mid \mathcal{D})$  is an estimate of
how likely  $\tilde{w}$  match  $w^{*}$ . For epistemic uncertainty, we should measure the difference between the predictive distributions under  $\boldsymbol{w}$  and  $w^{*}$ , but  $w^{*}$  is unknown. Therefore, we measure the expected difference between the predictive distributions under  $\boldsymbol{w}$  and  $\tilde{w}$ . In accordance with Apostolakis [1991] and Helton [1997], the total uncertainty is therefore the expected cross-entropy between the predictive distributions of a given, pre-selected model  $\boldsymbol{w}$  and candidate models  $\tilde{w}$ , any of which could be the true model  $w^{*}$  according to the posterior:
$$
\begin{array}{l} \int_ {\mathcal {W}} \operatorname {C E} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \boldsymbol {w}), p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} \tag {2} \\ = \underbrace {\mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x} , \boldsymbol {w}) ]} _ {\text {a l e a t o r i c}} + \underbrace {\int_ {\mathcal {W}} \mathrm {D} _ {\mathrm {K L}} (p (\boldsymbol {y} \mid \boldsymbol {x} , \boldsymbol {w}) \| p (\boldsymbol {y} \mid \boldsymbol {x} , \tilde {\boldsymbol {w}}))   p (\tilde {\boldsymbol {w}} \mid \mathcal {D})   \mathrm {d} \tilde {\boldsymbol {w}}} _ {\text {e p i s t e m i c}}. \\ \end{array}
$$
The aleatoric uncertainty characterizes the uncertainty due to the stochasticity of sampling outcomes from the predictive distribution of the given, pre-selected model  $p(\boldsymbol{y} \mid \boldsymbol{x}, \boldsymbol{w})$ . The epistemic uncertainty characterizes the uncertainty due to the mismatch between the predictive distribution of the given, pre-selected model and the predictive distribution of candidate models that could be the true model. Derivations and further details are given in appendix Sec. B.1.
2.2 Estimating the Integral for Epistemic Uncertainty
Current methods for predictive uncertainty quantification suffer from underestimating the epistemic uncertainty [Wilson and Izmailov, 2020, Parker-Holder et al., 2020, Angelo and Fortuin, 2021]. The epistemic uncertainty is given by the respective terms in Eq. (1) for setting (a) and Eq. (2) for our new setting (b). To estimate these integrals, almost all methods use gradient descent on the training data. Thus, posterior modes that are hidden from the gradient flow remain undiscovered and the epistemic uncertainty is underestimated [Shah et al., 2020, Angelo and Fortuin, 2021]. An illustrative example is depicted in Fig. 2. Posterior expectations as in Eq. (1) and Eq. (2) that define the epistemic uncertainty are generally approximated using Monte Carlo integration. A good approximation of posterior integrals through Monte Carlo integration requires to capture all large values of the nonnegative integrand [Wilson and Izmailov, 2020], which is not only large values of the posterior, but also large values of the KL-divergence.
Variational inference [Graves, 2011, Blundell et al., 2015, Gal and Ghahramani, 2016] and ensemble methods [Lakshminarayanan et al., 2017] estimate the posterior integral based on models with high posterior. Posterior modes may be hidden from gradient descent based techniques as they only discover mechanistically similar models. Two models are mechanistically similar if they rely on the same input attributes for making their predictions, that is, they are invariant to the same input attributes [Lubana et al., 2022]. However, gradient descent will always start by extracting input attributes that are highly correlated to the target as they determine the steepest descent in the error
(a) Deep Ensembles
(b) MC dropout
Figure 2: Model prediction analysis. Softmax outputs (black) of individual models of Deep Ensembles (a) and MC dropout (b), as well as their average output (red) on a probability simplex. Models were selected on the training data, and evaluated on the new test point (red) depicted in (c). The background color denotes the maximum likelihood of the training data that is achievable by a model having a predictive distribution (softmax values) equal to the respective location on the simplex. Deep Ensembles and MC dropout fail to find models predicting the orange class, although there would be likely models that do so. Details on the experimental setup are given in the appendix, Sec. C.2.
(c) Training data + new test point
landscape. These input attributes create a large basin in the error landscape into which the parameter vector is drawn via gradient descent. Consequently, other modes further away from such basins are almost never found [Shah et al., 2020, Angelo and Fortuin, 2021]. Thus, the epistemic uncertainty is underestimated. Another reason that posterior modes may be hidden from gradient descent is the presence of different labeling hypotheses. If there is more than one way to explain the training data, gradient descent will use all of them as they give the steepest error descent [Scimeca et al., 2022].
Other work focuses on MCMC sampling according to the posterior distribution, which is approximated by stochastic gradient variants [Welling and Teh, 2011, Chen et al., 2014] for large datasets and models. Those are known to face issues to efficiently explore the highly complex and multimodal parameter space and escape local posterior modes. There are attempts to alleviate the problem [Li et al., 2016, Zhang et al., 2020]. However, those methods do not explicitly look for important posterior modes, where the predictive distributions of sampled models contribute strongly to the approximation of the posterior integral, and thus have large values for the KL-divergence.
3 Adversarial Models to Estimate the Epistemic Uncertainty
Intuition. The epistemic uncertainty in Eq. (1) for setting (a) compares possible models with the BMA. Thus, the BMA is used as reference model. The epistemic uncertainty in Eq. (2) for our new setting (b) compares models that are candidates for the true model with the given, pre-selected model. Thus, the given, pre-selected model is used as reference model. If the reference model makes some prediction at the test point, and if other models (the adversaries) make different predictions while explaining the training data equally well, then one should be uncertain about the prediction. Adversarial models are plausible outcomes of model selection, while having a different prediction at the test data point than the reference model. In court, the same principle is used: if the prosecutor presents a scenario but the advocate presents alternative equally plausible scenarios, the judges become uncertain about what happened and rule in favor of the defendant. We use adversarial models to identify locations where the integrand of the integral defining the epistemic uncertainty in Eq. (1) or Eq. (2) is large. These locations are used to construct a mixture distribution that is used for mixture importance sampling to estimate the desired integrals. Using the mixture distribution for sampling, we aim to considerably reduce the approximation error of the estimator of the epistemic uncertainty.
Mixture Importance Sampling. We estimate the integrals of epistemic uncertainty in Eq. (1) and in Eq. (2). In the following, we focus on setting (b) with Eq. (2), but all results hold for setting (a) with Eq. (1) as well. Most methods sample from a distribution  $q(\tilde{w})$  to approximate the integral:
$$
v = \int_ {\mathcal {W}} \mathrm {D} _ {\mathrm {K L}} \left(p (\boldsymbol {y} \mid \boldsymbol {x}, \boldsymbol {w}) \| p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}})\right) p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} = \int_ {\mathcal {W}} \frac {u (\boldsymbol {x} , \boldsymbol {w} , \tilde {\boldsymbol {w}})}{q (\tilde {\boldsymbol {w}})} q (\tilde {\boldsymbol {w}}) \mathrm {d} \tilde {\boldsymbol {w}}, \tag {3}
$$
where  $u(\pmb{x}, \pmb{w}, \tilde{\pmb{w}}) = \mathrm{D}_{\mathrm{KL}}(p(\pmb{y} \mid \pmb{x}, \pmb{w}) \| p(\pmb{y} \mid \pmb{x}, \tilde{\pmb{w}})) p(\tilde{\pmb{w}} \mid \mathcal{D})$ . As with Deep Ensembles or MC dropout, posterior sampling is often approximated by a sampling distribution  $q(\tilde{\pmb{w}})$  that is close to  $p(\tilde{\pmb{w}} \mid \mathcal{D})$ . Monte Carlo (MC) integration estimates  $v$  by
$$
\hat {v} = \frac {1}{N} \sum_ {n = 1} ^ {N} \frac {u (\boldsymbol {x} , \boldsymbol {w} , \tilde {\boldsymbol {w}} _ {n})}{q \left(\tilde {\boldsymbol {w}} _ {n}\right)}, \quad \tilde {\boldsymbol {w}} _ {n} \sim q (\tilde {\boldsymbol {w}}). \tag {4}
$$
If the posterior has different modes, the estimate under a unimodal approximate distribution has high variance and converges very slowly [Steele et al., 2006]. Thus, we use mixture importance sampling (MIS) [Hesterberg, 1995]. MIS utilizes a mixture distribution instead of the unimodal distribution in standard importance sampling [Owen and Zhou, 2000]. Furthermore, many MIS methods iteratively enhance the sampling distribution by incorporating new modes [Raftery and Bao, 2010]. In contrast to the usually applied iterative enrichment methods which find new modes by chance, we have a much more favorable situation. We can explicitly search for posterior modes where the KL divergence is large, as we can cast it as a supervised learning problem. Each of these modes determines the location of a mixture component of the mixture distribution.
Theorem 1. The expected mean squared error of importance sampling with  $q(\tilde{w})$  can be bounded by
$$
\mathrm {E} _ {q (\tilde {\boldsymbol {w}})} \left[ (\hat {v} - v) ^ {2} \right] \leqslant \mathrm {E} _ {q (\tilde {\boldsymbol {w}})} \left[ \left(\frac {u (\boldsymbol {x} , \boldsymbol {w} , \tilde {\boldsymbol {w}})}{q (\tilde {\boldsymbol {w}})}\right) ^ {2} \right] \frac {4}{N}. \tag {5}
$$
Proof. The inequality Eq. (5) follows from Theorem 1 in Akyildiz and Miguez [2021], when considering  $0 \leqslant u(\pmb{x},\pmb{w},\tilde{\pmb{w}})$  as an unnormalized distribution and setting  $\varphi = 1$ .
Approximating only the posterior  $p(\tilde{w} \mid \mathcal{D})$  as done by Deep Ensembles or MC dropout is insufficient to guarantee a low expected mean squared error, since the sampling variance cannot be bounded (see appendix Sec. B.2).
Corollary 1. With constant  $c$ ,  $\operatorname{E}_{q(\tilde{\boldsymbol{w}})}\left[(\hat{v} - v)^2\right] \leqslant 4c^2 / N$  holds if  $u(\boldsymbol{x}, \boldsymbol{w}, \tilde{\boldsymbol{w}}) \leqslant cq(\tilde{\boldsymbol{w}})$ .