SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Consequently,  $q(\tilde{w})$  must have modes where  $u(\pmb{x},\pmb{w},\tilde{\pmb{w}})$  has modes even if the  $q$ -modes are a factor  $c$  smaller. The modes of  $u(\pmb{x},\pmb{w},\tilde{\pmb{w}})$  are models  $\tilde{\pmb{w}}$  with both high posterior and high KL-divergence. We are searching for these modes to determine the locations  $\tilde{\pmb{w}}_k$  of the components of a mixture distribution  $q(\tilde{w})$ :
$$
q (\tilde {\boldsymbol {w}}) = \sum_ {k = 1} ^ {K} \alpha_ {k} \mathcal {P} (\tilde {\boldsymbol {w}}; \breve {\boldsymbol {w}} _ {k}, \boldsymbol {\theta}), \tag {6}
$$
with  $\alpha_{k} = 1 / K$  for  $K$  such models  $\breve{w}_k$  that determine a mode. Adversarial model search finds the locations  $\breve{w}_k$  of the mixture components, where  $\breve{w}_k$  is an adversarial model. The reference model does not define a mixture component, as it has zero KL-divergence to itself. We then sample from a distribution  $\mathcal{P}$  at the local posterior mode with mean  $\breve{w}_k$  and a set of shape parameters  $\pmb{\theta}$ . The simplest choice for  $\mathcal{P}$  is a Dirac delta distribution, but one could use e.g. a local Laplace approximation of the posterior [MacKay, 1992], or a Gaussian distribution in some weight-subspace [Maddox et al., 2019]. Furthermore, one could use  $\breve{w}_k$  as starting point for SG-MCMC chains [Welling and Teh, 2011, Chen et al., 2014, Zhang et al., 2020, 2022]. More details regarding MIS are given in the appendix in Sec. B.2. In the following, we propose an algorithm to find those models with both high posterior and high KL-divergence to the predictive distribution of the reference model.
Adversarial Model Search. Adversarial model search is the concept of searching for a model that has a large distance / divergence to the reference predictive distribution and at the same time a high posterior. We call such models "adversarial models" as they act as adversaries to the reference model by contradicting its prediction. A formal definition of an adversarial model is given by Def. 1:
Definition 1. Given are a new test data point  $\pmb{x}$ , a reference conditional probability model  $p(\pmb{y} \mid \pmb{x}, \pmb{w})$  from a model class parameterized by  $\pmb{w}$ , a divergence or distance measure  $\mathrm{D}(\cdot, \cdot)$  for probability distributions,  $\gamma > 0$ ,  $\Lambda > 0$ , and a dataset  $\mathcal{D}$ . Then a model with parameters  $\tilde{\pmb{w}}$  that satisfies the inequalities  $|\log p(\pmb{w} \mid \mathcal{D}) - \log p(\tilde{\pmb{w}} \mid \mathcal{D})| \leqslant \gamma$  and  $\mathrm{D}(p(\pmb{y} \mid \pmb{x}, \pmb{w}), p(\pmb{y} \mid \pmb{x}, \tilde{\pmb{w}})) \geq \Lambda$  is called an  $(\gamma, \Lambda)$ -adversarial model.
Adversarial model search corresponds to the following optimization problem:
$$
\max  _ {\delta \in \Delta} \mathrm {D} (p (\boldsymbol {y} \mid \boldsymbol {x}, \boldsymbol {w}), p (\boldsymbol {y} \mid \boldsymbol {x}, \boldsymbol {w} + \delta)) \quad \text {s . t .} \log p (\boldsymbol {w} \mid \mathcal {D}) - \log p (\boldsymbol {w} + \delta \mid \mathcal {D}) \leqslant \gamma . \tag {7}
$$
We are searching for a weight perturbation  $\delta$  that maximizes the distance  $\mathrm{D}(\cdot, \cdot)$  to the reference distribution without decreasing the log posterior more than  $\gamma$ . The search for adversarial models is restricted to  $\delta \in \Delta$ , for example by only optimizing the last layer of the reference model or by bounding the norm of  $\delta$ . This optimization problem can be rewritten as:
$$
\max  _ {\delta \in \Delta} \mathrm {D} (p (\boldsymbol {y} \mid \boldsymbol {x}, \boldsymbol {w}), p (\boldsymbol {y} \mid \boldsymbol {x}, \boldsymbol {w} + \delta)) + c (\log p (\boldsymbol {w} + \delta \mid \mathcal {D}) - \log p (\boldsymbol {w} \mid \mathcal {D}) + \gamma). \tag {8}
$$
where  $c$  is a hyperparameter. According to the Karush-Kuhn-Tucker (KKT) theorem [Karush, 1939, Kuhn and Tucker, 1950, May, 2020, Luenberger and Ye, 2016]: If  $\delta^{*}$  is the solution to the problem Eq. (7), then there exists a  $c^{*} \geq 0$  with  $\nabla_{\delta} \mathcal{L}(\delta^{*}, c^{*}) = 0$  ( $\mathcal{L}$  is the Lagrangian) and  $c^{*} (\log p(\boldsymbol{w} \mid \mathcal{D}) - \log p(\boldsymbol{w} + \delta^{*} \mid \mathcal{D}) - \gamma) = 0$ . This is a necessary condition for an optimal point according to Theorem on Page 326 of Luenberger and Ye [2016].
We solve this optimization problem by the penalty method, which relies on the KKT theorem [Zangwill, 1967]. A penalty algorithm solves a series of unconstrained problems, solutions of which converge to the solution of the original constrained problem (see e.g. Fiasco and McCormick [1990]). The unconstrained problems are constructed by adding a weighted penalty function measuring the constraint violation to the objective function. At every step, the weight of the penalty is increased, thus the constraints are less violated. If exists, the solution to the constraint optimization problem is an adversarial model that is located within a posterior mode but has a different predictive distribution compared to the reference model. We summarize the adversarial model search in Alg. 1.
Algorithm 1 Adversarial Model Search (used in QUAM)
Supplies: Adversarial model  $\pmb{w}$  with maximum  $\mathrm{L}_{\mathrm{adv}}$  and  $\mathrm{L}_{\mathrm{pen}} \leqslant 0$
Requires: Test point  $\pmb{x}$ , training dataset  $\mathcal{D} = \{(x_k, y_k)\}_{k=1}^K$ , reference model  $\pmb{w}$ , loss function  $l$ , loss of reference model on the training dataset  $\mathrm{L}_{\mathrm{ref}} = \frac{1}{K} \sum_{k=1}^{K} l(p(\pmb{y} \mid x_k, \pmb{w}), \pmb{y}_k)$ , minimization procedure MINIMIZE, number of penalty iterations  $M$ , initial penalty parameter  $c_0$ , penalty parameter increase scheduler  $\eta$ , slack parameter  $\gamma$ , distance / divergence measure  $\mathrm{D}(\cdot, \cdot)$ .
1:  $\breve{w} \gets w$ ;  $\tilde{w} \gets w$ ;  $c \gets c_0$
2: for  $m\gets 1$  to  $M_{x}\mathbf{d}\mathbf{o}$
3:  $\mathrm{L}_{\mathrm{pen}}\gets \frac{1}{K}\sum_{k = 1}^{K}l(p(\boldsymbol {y}\mid \boldsymbol{x}_k,\tilde{\boldsymbol{w}}),\boldsymbol {y}_k) - (\mathrm{L}_{\mathrm{ref}} + \gamma)$
4:  $\mathrm{L}_{\mathrm{adv}}\gets -\mathrm{D}(p(\boldsymbol {y}\mid \boldsymbol {x},\boldsymbol {w}),p(\boldsymbol {y}\mid \boldsymbol {x},\tilde{\boldsymbol{w}}))$
5:  $\mathrm{L}\gets \mathrm{L}_{\mathrm{adv}} + c\mathrm{L}_{\mathrm{pen}}$
6:  $\tilde{w} \gets \text{MINIMIZE}(\mathrm{L}(\tilde{w}))$
7: if  $\mathrm{L}_{\mathrm{adv}}$  larger than all previous and  $\mathrm{L}_{\mathrm{pen}} \leqslant 0$  then
8:  $\breve{w} \gets \tilde{w}$
9:  $c\gets \eta (c)$
0: return  $\check{w}$
Practical Implementation. Empirically, we found that directly executing the optimization procedure defined in Alg. 1 tends to result in adversarial models with similar predictive distribution for a given input across multiple searches. The vanilla implementation of Alg. 1 corresponds to an untargeted attack, known from the literature on adversarial attacks [Szegedy et al., 2013, Biggio et al., 2013]. To prevent the searches from converging to a single solution, we optimize the cross-entropy loss for one specific class during each search, which corresponds to a targeted attack. Each resulting adversarial model represents a local optimum of Eq. (7). We execute as many adversarial model searches as there are classes, dedicating one search to each class, unless otherwise specified. To compute Eq. (4), we use the predictive distributions  $p(\boldsymbol{y} \mid \boldsymbol{x}, \tilde{\boldsymbol{w}})$  of all models  $\tilde{\boldsymbol{w}}$  encountered during each penalty iteration of all searches, weighted by their posterior probability. The posterior probability is approximated with the negative exponential training loss, the likelihood, of models  $\tilde{\boldsymbol{w}}$ . This approximate posterior probability is scaled with a temperature parameter, set as a hyperparameter. Further details are given in the appendix Sec. C.1.
In this section, we compare previous uncertainty quantification methods and our method QUAM in a set of experiments. First, we assess the considered methods on a synthetic benchmark, on which it is feasible to compute a ground truth epistemic uncertainty. Then, we conduct challenging out-of-distribution (OOD) detection, adversarial example detection, misclassification detection and selective prediction experiments in the vision domain. We compare (1) QUAM, (2) cyclical Stochastic Gradient Hamiltonian Monte Carlo (cSG-HMC) [Zhang et al., 2020], (3) an efficient Laplace approximation (Laplace) [Daxberger et al., 2021], (4) MC dropout (MCD) [Gal and Ghahramani, 2016] and (5) Deep Ensembles (DE) [Lakshminarayanan et al., 2017] on their ability to estimate the epistemic uncertainty. Those baseline methods, especially Deep Ensembles, are persistently among the best performing uncertainty quantification methods across various benchmark tasks [Filos et al., 2019, Ovadia et al., 2019, Caldeira and Nord, 2020, Band et al., 2022]
4.1 Epistemic Uncertainty on Synthetic Dataset
We evaluated all considered methods on the two-moons dataset, created using the implementation of Pedregosa et al. [2011]. To obtain the ground truth uncertainty, we utilized Hamiltonian Monte Carlo (HMC) [Neal, 1996]. HMC is regarded as the most precise algorithm to approximate posterior expectations [Izmailov et al., 2021], but necessitates extreme computational expenses to be applied to models and datasets of practical scale. The results are depicted in Fig. 3. QUAM most closely matches the uncertainty estimate of the ground truth epistemic uncertainty obtained by HMC and excels especially on the regions further away from the decision boundary such as in the top left and bottom right of the plots. All other methods fail to capture the epistemic uncertainty in those regions as gradient descent on the training set fails to capture posterior modes with alternative predictive distributions in those parts and misses the important integral components. Experimental details and results for the epistemic uncertainty as in Eq. (2) are given in the appendix Sec. C.3.
(a) Ground Truth - HMC
(b) cSG-HMC
(c) Laplace
(d) MC dropout
(e) Deep Ensembles
(f) Our Method - QUAM
Figure 3: Epistemic uncertainty as in Eq. (1) for two-moons. Yellow denotes high epistemic uncertainty. Purple denotes low epistemic uncertainty. HMC is considered as ground truth [Izmailov et al., 2021] and is most closely matched by QUAM. Artifacts for QUAM arise because it is applied to each test point individually, whereas other methods use the same sampled models for all test points.
4.2 Epistemic Uncertainty on Vision Datasets
We benchmark the ability of different methods to estimate the epistemic uncertainty of a given, pre-selected model (setting (b) as in Eq. (2)) in the context of (i) out-of-distribution (OOD) detection, (ii) adversarial example detection, (iii) misclassification detection and (iv) selective prediction. In all experiments, we assume to have access to a pre-trained model on the in-distribution (ID) training dataset, which we refer to as reference model. The epistemic uncertainty is expected to be higher for OOD samples, as they can be assigned to multiple ID classes, depending on the utilized features. Adversarial examples indicate that the model is misspecified on those inputs, thus we expect a higher epistemic uncertainty, the uncertainty about the model parameters. Furthermore, we expect higher epistemic uncertainty for misclassified samples than for correctly classified samples. Similarly, we expect the classifier to perform better on a subset of more certain samples. This is tested by evaluating the accuracy of the classifier on retained subsets of a certain fraction of samples with the lowest epistemic uncertainty [Filos et al., 2019, Band et al., 2022]. We report the AUROC for classifying the ID vs. OOD samples (i), the ID vs. the adversarial examples (ii), or the correctly classified vs. the misclassified samples (iii), using the epistemic uncertainty as score to distinguish the two classes respectively. For the selective prediction experiment (iv), we report the AUC of the accuracy vs. fraction of retained samples, using the epistemic uncertainty to determine the retained subsets.
MNIST. We perform OOD detection on the FMNIST [Xiao et al., 2017], KMNIST [Clanuwat et al., 2018], EMNIST [Cohen et al., 2017] and OMNIGLOT [Lake et al., 2015] test datasets as OOD datasets, using the LeNet [LeCun et al., 1998] architecture. The test dataset of MNIST [LeCun et al., 1998] is used as ID dataset. We utilize the aleatoric uncertainty of the reference model (as in Eq. (2)) as a baseline to assess the added value of estimating the epistemic uncertainty of the reference model. The results are listed in Tab. 1. QUAM outperforms all other methods on this task, with Deep Ensembles being the runner up method on all dataset pairs. Furthermore, we observed, that only the epistemic uncertainties obtained by Deep Ensembles and QUAM are able to surpass the performance of using the aleatoric uncertainty of the reference model.
ImageNet-1K. We conduct OOD detection, adversarial example detection, misclassification detection and selective prediction experiments on ImageNet-1K [Deng et al., 2009]. As OOD dataset, we use ImageNet-O [Hendrycks et al., 2021], which is a challenging OOD dataset that was explicitly created to be classified as an ID dataset with high confidence by conventional ImageNet-1K classifiers. Similarly, ImageNet-A [Hendrycks et al., 2021] is a dataset consisting of natural adversarial exam
Table 1: MNIST results: AUROC using the epistemic uncertainty of a given, pre-selected model (as in Eq. (2)) as a score to distinguish between ID (MNIST) and OOD samples. We also report the AUROC when using the aleatoric uncertainty of the reference model (Reference).
Dood	Reference	cSG-HMC	Laplace	MCD	DE	QUAM
FMNIST	.986±.005	.977±.004	.978±.004	.978±.005	.988±.001	.994±.001
KMNIST	.966±.005	.957±.005	.959±.006	.956±.006	.990±.001	.994±.001
EMNIST	.888±.007	.869±.012	.877±.011	.876±.008	.924±.003	.937±.008
OMNIGLOT	.973±.003	.963±.004	.963±.003	.965±.003	.983±.001	.992±.001
Table 2: ImageNet-1K results: AUROC using the epistemic uncertainty of a given, pre-selected model (as in Eq. (2)) to distinguish between ID (ImageNet-1K) and OOD samples. Furthermore, we report the AUROC when using the epistemic uncertainty for misclassification detection and the AUC of accuracy over fraction of retained predictions on the ImageNet-1K validation dataset. We also report results for all experiments, using the aleatoric uncertainty of the reference model (Reference).
Dood//Task	Reference	cSG-HMC	MCD	DE(LL)	DE(all)	QUAM
ImageNet-O	.626±.004	.677±.005	.680±.003	.562±.004	.709±.005	.753±.011
ImageNet-A	.792±.002	.799±.001	.827±.002	.686±.001	.874±.004	.872±.003
Misclassification	.867±.007	.772±.011	.796±.014	.657±.009	.780±.009	.904±.008
Selective prediction	.958±.003	.931±.003	.935±.006	.911±.004	.950±.002	.969±.002
plies, which belong to the ID classes of ImageNet-1K, but are misclassified with high confidence by conventional ImageNet-1K classifiers. Furthermore, we evaluated the utility of the uncertainty score for misclassification detection of predictions of the reference model on the ImageNet-1K validation dataset. On the same dataset, we evaluated the accuracy of the reference model when only predicting on fractions of samples with the lowest epistemic uncertainty.
All ImageNet experiments were performed on variations of the EfficientNet architecture [Tan and Le, 2019]. Recent work by Kirichenko et al. [2022] showed that typical ImageNet-1K classifiers learn desired features of the data even if they rely on simple, spurious features for their prediction. Furthermore, they found, that last layer retraining on a dataset without the spurious correlation is sufficient to re-weight the importance that the classifier places on different features. This allows the classifier to ignore the spurious features and utilize the desired features for its prediction. Similarly, we apply QUAM on the last layer of the reference model. We compare against cSG-HMC applied to the last layer, MC dropout and Deep Ensembles. MC dropout was applied to the last layer as well, since the EfficientNet architectures utilize dropout only before the last layer. Two versions of Deep Ensembles were considered. First, Deep Ensembles aggregated from pre-trained EfficientNets of different network sizes (DE (all)). Second, Deep Ensembles of retrained last layers on the same encoder network (DE (LL)). We further utilize the aleatoric uncertainty of the reference model (as in Eq. (2)) as a baseline to assess the additional benefit of estimating the epistemic uncertainty of the reference model. The Laplace approximation was not feasible to compute on our hardware, even only for the last layer.
The results are listed in Tab. 2. Plots showing the respective curves of each experiment are depicted in Fig. C.8 in the appendix. We observe that using the epistemic uncertainty provided by DE (LL) has the worst performance throughout all experiments. While DE (all) performed second best on most tasks, MC dropout outperforms it on OOD detection on the ImageNet-O dataset. QUAM outperforms all other methods on all tasks we evaluated, except for ImageNet-A, where it performed on par with DE (all). Details about all experiments and additional results are given in the appendix Sec. C.4.
Compute Efficiency. As an ablation study, we investigate the performance of QUAM under a restricted computational budget. Therefore, the searches for adversarial models were performed on only a subset of classes instead of each eligible class, specifically the top  $N$  most probable classes according to the predictive distribution of the given, pre-selected model. The computational budget between QUAM and MC dropout was matched by accounting for the number of forward pass equivalents required by each method. In this context, we assume that the backward pass corresponds to the computational cost of two forward passes. The results depicted in Fig. 4 show that QUAM
Figure 4: Inference speed vs. performance. MCD and QUAM evaluated on equal computational budget in terms of forward pass equivalents on ImageNet-O (left) and ImageNet-A (right) tasks.
outperforms MC dropout even under a very limited computational budget. Furthermore, training a single additional ensemble member for Deep Ensembles requires more compute than evaluating the entire ImageNet-O and ImageNet-A datasets with QUAM when performed on all 1000 classes.
Quantifying predictive uncertainty, especially for deep learning models, is an active area of research. Classical uncertainty quantification methods such as Bayesian Neural Networks (BNNs) [MacKay, 1992, Neal, 1996] are challenging for deep learning, since (i) the Hessian or maximum-a-posteriori (MAP) is difficult to estimate and (ii) regularization and normalization techniques cannot be treated [Antoráan et al., 2022]. Epistemic neural networks [Osband et al., 2021] add a variance term (the epinet) to the output only. Bayes By Backprop [Blundell et al., 2015] and variational neural networks [Oleksiienko et al., 2022] work only for small models as they require considerably more parameters. MC dropout [Gal and Ghahramani, 2016] casts applying dropout during inference as sampling from an approximate distribution. MC dropout was generalized to MC dropconnect [Mobiny et al., 2021]. Deep Ensembles [Lakshminarayanan et al., 2017] are often the best-performing uncertainty quantification method [Ovadia et al., 2019, Wursthorn et al., 2022]. Masksembles or Dropout Ensembles combine ensembling with MC dropout [Durasov et al., 2021]. Stochastic Weight Averaging approximates the posterior over the weights [Maddox et al., 2019]. Single forward pass methods are fast and they aim to capture a different notion of epistemic uncertainty through the distribution or distances of latent representations [Bradshaw et al., 2017, Liu et al., 2020, Mukhoti et al., 2021, van Amersfoort et al., 2021, Postels et al., 2021] rather than through posterior integrals. For further methods and a general overview of uncertainty estimation see e.g. Hüllermeier and Waegeman [2021], Abdar et al. [2021] and Gawlikowski et al. [2021].
We have introduced QUAM, a novel method that quantifies predictive uncertainty using adversarial models. Adversarial models identify important posterior modes that are missed by previous uncertainty quantification methods. We conducted various experiments on deep neural networks, for which epistemic uncertainty is challenging to estimate. On a synthetic dataset, we highlighted the strength of our method to capture epistemic uncertainty. Furthermore, we conducted experiments on large-scale benchmarks in the vision domain, where QUAM outperformed all previous methods.
Searching for adversarial models is computationally expensive and has to be done for each new test point. However, more efficient versions can be utilized. One can search for adversarial models while restricting the search to a subset of the parameters, e.g. to the last layer as was done for the ImageNet experiments, to the normalization parameters, or to the bias weights. Furthermore, there have been several advances for efficient fine-tuning of large models [Houlsby et al., 2019, Hu et al., 2021]. Utilizing those for more efficient versions of our algorithm is an interesting direction for future work.
Nevertheless, high stake applications justify this effort to obtain the best estimate of predictive uncertainty for each new test point. Furthermore, QUAM is applicable to quantify the predictive uncertainty of any single given model, regardless of whether uncertainty estimation was considered during the modeling process. This allows to assess the predictive uncertainty of foundation models or specialized models that are obtained externally.
We would like to thank Angela Bitto-Nemling for providing relevant literature, organizing meetings, and giving feedback on this research project. Furthermore, we would like to thank Angela Bitto-Nemling, Daniel Klotz, and Sebastian Lehner for insightful discussions and provoking questions. The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. We thank the projects AI-MOTION (LIT-2018-6-YOU-212), DeepFlood (LIT-2019-8-YOU-213), Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), EPILEPSIA (FFG-892171), AIRI FG 9-N (FWF-36284, FWF-36235), AI4GreenHeatingGrids(FFG-899943), INTEGRATE (FFG-892418), ELISE (H2020-ICT-2019-3 ID: 951847), Stars4Waters (HORIZONCL6-2021-CLIMATE-01-01). We thank Audi.JKU Deep Learning Center, TGW LOGISTICS GROUP GMBH, Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, GLS (Univ. Waterloo) Software Competence Center Hagenberg GmbH, TÜV Austria, Frauscher Sensonic, TRUMPF and the NVIDIA Corporation.