SlowGuess/NeurIPS_2023_OCR huggingface dataset:
M. Abdar, F. Pourpanah, S. Hussain, D. Rezazadegan, L. Liu, M. Ghavamzadeh, P. Fieguth, X. Cao, A. Khosravi, U. R. Acharya, V. Makarenkov, and S. Nahavandi. A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 76:243-297, 2021.
A. Adler, R. Youmaran, and W. R. B. Lionheart. A measure of the information content of EIT data. Physiological Measurement, 29(6):S101-S109, 2008.
Ö. D. Akyildiz and J. Miguez. Convergence rates for optimised adaptive importance samplers. Statistics and Computing, 31(12), 2021.
F. D'Angelo and V. Fortuin. Repulsive deep ensembles are bayesian. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 3451-3465. Curran Associates, Inc., 2021.
J. Antoráan, D. Janz, J. U. Allingham, E. Daxberger, R. Barbano, E. Nalisnick, and J. M. Hernández-Lobato. Adapting the linearised Laplace model evidence for modern deep learning. *ArXiv*, 2206.08900, 2022.
G. Apostolakis. The concept of probability if safety assessments of technological systems. Science, 250(4986):1359-1364, 1991.
N. Band, T. G. J. Rudner, Q. Feng, A. Filos, Z. Nado, M. W. Dusenberry, G. Jerfel, D. Tran, and Y. Gal. Benchmarking bayesian deep learning on diabetic retinopathy detection tasks. *ArXiv*, 2211.12717, 2022.
B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, and F. Roli. Evasion attacks against machine learning at test time. In H. Blockeel, K. Kersting, S. Nijssen, and F. Železný, editors, Machine Learning and Knowledge Discovery in Databases, pages 387-402, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg.
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1613-1622, 2015.
J. Bradshaw, A. G. de G. Matthews, and Z. Ghahramani. Adversarial examples, uncertainty, and transfer testing robustness in gaussian process hybrid deep networks. *ArXiv*, 1707.02476, 2017.
J. Caldeira and B. Nord. Deeply uncertain: comparing methods of uncertainty quantification in deep learning algorithms. Machine Learning: Science and Technology, 2(1):015002, 2020.
O. Cappé, A. Guillin, J.-M. Marin, and C. P. Robert. Population Monte Carlo. Journal of Computational and Graphical Statistics, 13(4):907-929, 2004.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. J. Goodfellow, and R. Fergus. Intriguing properties of neural networks. *ArXiv*, 1312.6199, 2013.
M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-6114. Proceedings of Machine Learning Research, 2019.
M. Tribus. Thermostatics and Thermodynamics: An Introduction to Energy, Information and States of Matter, with Engineering Applications. University series in basic engineering. Van Nostrand, 1961.
J. van Amersfoort, L. Smith, Y. W. Teh, and Y. Gal. Uncertainty estimation using a single deep deterministic neural network. In International Conference on Machine Learning, pages 9690-9700. Proceedings of Machine Learning Research, 2020.
J. van Amersfoort, L. Smith, A. Jesson, O. Key, and Y. Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. *ArXiv*, 2102.11409, 2021.
E. Veach and L. J. Guibas. Optimally combining sampling techniques for Monte Carlo rendering. In Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, pages 419-428. Association for Computing Machinery, 1995.
W. E. Vesely and D. M. Rasmuson. Uncertainties in nuclear probabilistic risk analyses. Risk Analysis, 4, 1984.
S. Weinzierl. Introduction to Monte Carlo methods. ArXiv, hep-ph/0006269, 2000.
M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, page 681-688, Madison, WI, USA, 2011. Omnipress.
A. G. Wilson and P. Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in Neural Information Processing Systems, 33:4697-4708, 2020.
K. Wursthorn, M. Hillemann, and M. M. Ulrich. Comparison of uncertainty quantification methods for CNN-based regression. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, XLIII-B2-2022:721-728, 2022.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. *ArXiv*, 1708.07747, 2017.
W. I. Zangwill. Non-linear programming via penalty functions. Management Science, 13(5):344-358, 1967.
R. Zhang, C. Li, J. Zhang, C. Chen, and A. G. Wilson. Cyclical stochastic gradient MCMC for Bayesian deep learning. In International Conference on Learning Representations, 2020.
R. Zhang, A. G. Wilson, and C. DeSa. Low-precision stochastic gradient Langevin dynamics. ArXiv, 2206.09909, 2022.
J. V. Zidek and C. vanEeden. Uncertainty, entropy, variance and the effect of partial information. Lecture Notes-Monograph Series, 42:155-167, 2003.
This is the appendix of the paper "Quantification of Uncertainty with Adversarial Models". It consists of three sections. In view of the increasing influence of contemporary machine learning research on the broader public, section A gives a societal impact statement. Following to this, section B gives details of our theoretical results, foremost about the measure of uncertainty used throughout our work. Furthermore, Mixture Importance Sampling for variance reduction is discussed. Finally, section C gives details about the experiments presented in the main paper, as well as further experiments.
A Societal Impact Statement 18
B Theoretical Results 19
B.1 Measuring Predictive Uncertainty 19
B.1.1 Entropy and Cross-Entropy as Measures of Predictive Uncertainty 19
B.1.2 Classification 20
B.1.3 Regression 22
B.2 Mixture Importance Sampling for Variance Reduction 23
C Experimental Details and Further Experiments 27
C.1 Details on the Adversarial Model Search 27
C.2 Simplex Example 28
C.3 Epistemic Uncertainty on Synthetic Dataset 29
C.4 Epistemic Uncertainty on Vision Datasets 29
C.4.1 MNIST 31
C.4.2 ImageNet 31
C.5 Comparing Mechanistic Similarity of Deep Ensembles vs. Adversarial Models 38
C.6 Prediction Space Similarity of Deep Ensembles and Adversarial Models 38
C.7 Computational Expenses 38
B.1 Asymptotic variance for multimodal target and unimodal sampling distribution 26
C.1 Illustrative example of QUAM 27
C.2 HMC and Adversarial Model Search on simplex 28
C.3 Epistemic uncertainty (setting (b)) on synthetic classification dataset 30
C.4 Model variance of different methods on toy regression dataset 30
C.5 Histograms MNIST 33
C.6 Calibration on ImageNet 34
C.7 Histograms ImageNet 36
C.8 Detailed results of ImageNet experiments 37
C.9 Comparing mechanistic similarity of Deep Ensembles vs. Adversarial Models 39
C.10 Prediction Space Similarity of Deep Ensembles vs. Adversarial Models 39
List of Tables
C.1 Results for additional baseline (MoLA) 31
C.2 Detailed results of MNIST OOD detection experiments 32
C.3 Results for calibration on ImageNet 34
C.4 Detailed results of ImageNet experiments 35
A Societal Impact Statement
In this work, we have focused on improving the predictive uncertainty estimation for machine learning models, specifically deep learning models. Our primary goal is to enhance the robustness and reliability of these predictions, which we believe have several positive societal impacts.
1. Improved decision-making: By providing more accurate predictive uncertainty estimates, we enable a broad range of stakeholders to make more informed decisions. This could have implications across various sectors, including healthcare, finance, and autonomous vehicles, where decision-making based on machine learning predictions can directly affect human lives and economic stability.
2. Increased trust in machine learning systems: By enhancing the reliability of machine learning models, our work may also contribute to increased public trust in these systems. This could foster greater acceptance and integration of machine learning technologies in everyday life, driving societal advancement.
3. Promotion of responsible machine learning: Accurate uncertainty estimation is crucial for the responsible deployment of machine learning systems. By advancing this area, our work promotes the use of those methods in an ethical, transparent, and accountable manner.
While we anticipate predominantly positive impacts, it is important to acknowledge potential negative impacts or challenges.
1. Misinterpretation of uncertainty: Even with improved uncertainty estimates, there is a risk that these might be misinterpreted or misused, potentially leading to incorrect decisions or unintended consequences. It is vital to couple advancements in this field with improved education and awareness around the interpretation of uncertainty in AI systems.
2. Increased reliance on machine learning systems: While increased trust in machine learning systems is beneficial, there is a risk it could lead to over-reliance on these systems, potentially resulting in reduced human oversight or critical thinking. It's important that robustness and reliability improvements don't result in blind trust.
3. Inequitable distribution of benefits: As with any technological advancement, there is a risk that the benefits might not be evenly distributed, potentially exacerbating existing societal inequalities. We urge policymakers and practitioners to consider this when implementing our findings.
In conclusion, while our work aims to make significant positive contributions to society, we believe it is essential to consider these potential negative impacts and take steps to mitigate them proactively.
B.1 Measuring Predictive Uncertainty
In this section, we first discuss the usage of the entropy and the cross-entropy as measures of predictive uncertainty. Following this, we introduce the two settings (a) and (b) (see Sec. 2) in detail for the predictive distributions of probabilistic models in classification and regression. Finally, we discuss Mixture Importance Sampling for variance reduction of the uncertainty estimator.
B.1.1 Entropy and Cross-Entropy as Measures of Predictive Uncertainty
Shannon and Elwood [1948] defines the entropy  $\mathrm{H}[\pmb{p}] = -\sum_{i=1}^{N} p_i \log p_i$  as a measure of the amount of uncertainty of a discrete probability distribution  $\pmb{p} = (p_1, \dots, p_N)$  and states that it measures how much "choice" is involved in the selection of a class  $i$ . See also Jaynes [1957], Cover and Thomas [2006] for an elaboration on this topic. The value  $-\log p_i$  has been called "surprisal" [Tribus, 1961] (page 64, Subsection 2.9.1) and has been used in computational linguistics [Hale, 2001]. Hence, the entropy is the expected or mean surprisal. Instead of "surprisal" also the terms "information content", "self-information", or "Shannon information" are used.
The cross-entropy  $\mathrm{CE}[p, q] = -\sum_{i=1}^{N} p_i \log q_i$  between two discrete probability distributions  $\pmb{p} = (p_1, \dots, p_N)$  and  $\pmb{q} = (q_1, \dots, q_N)$  measures the expectation of the surprisal of  $\pmb{q}$  under distribution  $\pmb{p}$ . Like the entropy, the cross-entropy is a mean of surprisals, therefore can be considered as a measure to quantify uncertainty. The higher surprises are on average, the higher the uncertainty. The cross-entropy has increased uncertainty compared to the entropy since more surprising events are expected when selecting events via  $\pmb{p}$  instead of  $\pmb{q}$ . Only if those distributions coincide, there is no additional surprisal and the cross-entropy is equal to the entropy of the distributions. The cross-entropy depends on the uncertainty of the two distributions and how different they are. In particular, high surprisal of  $q_i$  and low surprisal of  $p_i$  strongly increase the cross-entropy since unexpected events are more frequent, that is, we are more often surprised. Thus, the cross-entropy does not only measure the uncertainty under distribution  $\pmb{p}$ , but also the difference of the distributions. The average surprisal via the cross-entropy depends on the uncertainty of  $\pmb{p}$  and the difference between  $\pmb{p}$  and  $\pmb{q}$ :
$$
\begin{array}{l} \operatorname {C E} [ \boldsymbol {p}, \boldsymbol {q} ] = - \sum_ {i = 1} ^ {N} p _ {i} \log q _ {i} \tag {9} \\ = - \sum_ {i = 1} ^ {N} p _ {i} \log p _ {i} + \sum_ {i = 1} ^ {N} p _ {i} \log \frac {p _ {i}}{q _ {i}} \\ = \mathrm {H} [ \boldsymbol {p} ] + \mathrm {D} _ {\mathrm {K L}} (\boldsymbol {p} \| \boldsymbol {q}), \\ \end{array}
$$
where the Kullback-Leibler divergence  $\mathrm{D}_{\mathrm{KL}}(\cdot \| \cdot)$  is
$$
\mathrm {D} _ {\mathrm {K L}} (\boldsymbol {p} \| \boldsymbol {q}) = \sum_ {i = 1} ^ {N} p _ {i} \log \frac {p _ {i}}{q _ {i}}. \tag {10}
$$
The Kullback-Leibler divergence measures the difference in the distributions via their average difference of surprisals. Furthermore, it measures the decrease in uncertainty when shifting from the estimate  $\pmb{p}$  to the true  $\pmb{q}$  [Seidenfeld, 1986, Adler et al., 2008].
Therefore, the cross-entropy can serve to measure the total uncertainty, where the entropy is used as aleatoric uncertainty and the difference of distributions is used as the epistemic uncertainty. We assume that  $q$  is the true distribution that is estimated by the distribution  $p$ . We quantify the total uncertainty of  $p$  as the sum of the entropy of  $p$  (aleatoric uncertainty) and the Kullback-Leibler divergence to  $q$  (epistemic uncertainty). In accordance with Apostolakis [1991] and Helton [1997], the aleatoric uncertainty measures the stochasticity of sampling from  $p$ , while the epistemic uncertainty measures the deviation of the parameters  $p$  from the true parameters  $q$ .
In the context of quantifying uncertainty through probability distributions, other measures such as the variance have been proposed [Zidek and vanEeden, 2003]. For uncertainty estimation in the context of deep learning systems, e.g. Gal [2016], Kendall and Gal [2017], Depeweg et al. [2018] proposed to use the variance of the BMA predictive distribution as a measure of uncertainty. Entropy and variance capture different notions of uncertainty and investigating measures based on the variance of the predictive distribution is an interesting avenue for future work.
B.1.2 Classification
Setting (a): Expected uncertainty when selecting a model. We assume to have training data  $\mathcal{D}$  and an input  $\pmb{x}$ . We want to know the uncertainty in predicting a class  $\pmb{y}$  from  $\pmb{x}$  when we first choose a model  $\tilde{\pmb{w}}$  based on the posterior  $p(\tilde{\pmb{w}}\mid \mathcal{D})$  and then use the chosen model  $\tilde{\pmb{w}}$  to choose a class for input  $\pmb{x}$  according to the predictive distribution  $p(\pmb{y}\mid \pmb{x},\tilde{\pmb{w}})$ . The uncertainty in predicting the class arises from choosing a model (epistemic) and from choosing a class using this probabilistic model (aleatoric).
Through Bayesian model averaging, we obtain the following probability of selecting a class:
$$
p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) = \int_ {\mathcal {W}} p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}}. \tag {11}
$$
The total uncertainty is commonly measured as the entropy of this probability distribution [Houlsby et al., 2011, Gal, 2016, Depeweg et al., 2018, Hüllermeier and Waegeman, 2021]:
$$
\operatorname {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) ]. \tag {12}
$$
We can reformulate the total uncertainty as the expected cross-entropy:
$$
\begin{array}{l} \operatorname {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) ] = - \sum_ {\boldsymbol {y} \in \mathcal {Y}} p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) \log p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) \tag {13} \\ = - \sum_ {\boldsymbol {y} \in \mathcal {Y}} \log p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) \int_ {\mathcal {W}} p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} \\ = \int_ {\mathcal {W}} \left(- \sum_ {\boldsymbol {y} \in \mathcal {Y}} p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) \log p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D})\right) p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) d \tilde {\boldsymbol {w}} \\ = \int_ {\mathcal {W}} \operatorname {C E} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}), p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}}. \\ \end{array}
$$
We can split the total uncertainty into the aleatoric and epistemic uncertainty [Houlsby et al., 2011, Gal, 2016, Smith and Gal, 2018]:
$$
\begin{array}{l} \int_ {\mathcal {W}} \operatorname {C E} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}), p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} \tag {14} \\ = \int_ {\mathcal {W}} \left(\mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) ] + \mathrm {D} _ {\mathrm {K L}} (p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) \| p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}))\right) p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} \\ = \int_ {\mathcal {W}} \mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} + \int_ {\mathcal {W}} \mathrm {D} _ {\mathrm {K L}} (p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) \| p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D})) p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} \\ = \int_ {\mathcal {W}} \mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} + \mathrm {I} [ Y; W \mid \boldsymbol {x}, \mathcal {D} ]. \\ \end{array}
$$
We verify the last equality in Eq. (14), i.e. that the Mutual Information is equal to the expected Kullback-Leibler divergence:
$$
\begin{array}{l} I [ Y; W \mid \boldsymbol {x}, \mathcal {D} ] = \int_ {\mathcal {W}} \sum_ {\boldsymbol {y} \in \mathcal {Y}} p (\boldsymbol {y}, \tilde {\boldsymbol {w}} \mid \boldsymbol {x}, \mathcal {D}) \log \frac {p (\boldsymbol {y} , \tilde {\boldsymbol {w}} \mid \boldsymbol {x} , \mathcal {D})}{p (\boldsymbol {y} \mid \boldsymbol {x} , \mathcal {D}) p (\tilde {\boldsymbol {w}} \mid \mathcal {D})} d \tilde {\boldsymbol {w}} \tag {15} \\ = \int_ {\mathcal {W}} \sum_ {\boldsymbol {y} \in \mathcal {Y}} p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \log \frac {p (\boldsymbol {y} \mid \boldsymbol {x} , \tilde {\boldsymbol {w}}) p (\tilde {\boldsymbol {w}} \mid \mathcal {D})}{p (\boldsymbol {y} \mid \boldsymbol {x} , \mathcal {D}) p (\tilde {\boldsymbol {w}} \mid \mathcal {D})} d \tilde {\boldsymbol {w}} \\ = \int_ {\mathcal {W}} \sum_ {\boldsymbol {y} \in \mathcal {Y}} p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) \log \frac {p (\boldsymbol {y} \mid \boldsymbol {x} , \tilde {\boldsymbol {w}})}{p (\boldsymbol {y} \mid \boldsymbol {x} , \mathcal {D})} p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) d \tilde {\boldsymbol {w}} \\ = \int_ {\mathcal {W}} \mathrm {D} _ {\mathrm {K L}} \left(p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) \| p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D})\right) p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}}. \\ \end{array}
$$
This is possible because the label is dependent on the selected model. First, a model is selected, then a label is chosen with the selected model. To summarize, the predictive uncertainty is measured by:
$$
\begin{array}{l} \mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) ] = \int_ {\mathcal {W}} \mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} + \mathrm {I} [ Y; W \mid \boldsymbol {x}, \mathcal {D} ] \tag {16} \\ = \int_ {\mathcal {W}} \mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} \\ + \int_ {\mathcal {W}} \mathrm {D} _ {\mathrm {K L}} (p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}) \| p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D})) p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}} \\ = \int_ {\mathcal {W}} \operatorname {C E} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \tilde {\boldsymbol {w}}), p (\boldsymbol {y} \mid \boldsymbol {x}, \mathcal {D}) ] p (\tilde {\boldsymbol {w}} \mid \mathcal {D}) \mathrm {d} \tilde {\boldsymbol {w}}. \\ \end{array}
$$
The total uncertainty is given by the entropy of the Bayesian model average predictive distribution, which we showed is equal to the expected cross-entropy between the predictive distributions of candidate models  $\tilde{w}$  selected according to the posterior and the Bayesian model average predictive distribution. The aleatoric uncertainty is the expected entropy of candidate models drawn from the posterior, which can also be interpreted as the entropy we expect when selecting a model according to the posterior. Therefore, if all models likely under the posterior have low surprisal, the aleatoric uncertainty in this setting is low. The epistemic uncertainty is the expected KL divergence between the the predictive distributions of candidate models and the Bayesian model average predictive distribution. Therefore, if all models likely under the posterior have low divergence of their predictive distribution to the Bayesian model average predictive distribution, the epistemic uncertainty in this setting is low.
Setting (b): Uncertainty of a given, pre-selected model. We assume to have training data  $\mathcal{D}$ , an input  $\pmb{x}$ , and a given, pre-selected model with parameters  $\pmb{w}$  and predictive distribution  $p(\pmb{y} \mid \pmb{x}, \pmb{w})$ . Using the predictive distribution of the model, a class  $\pmb{y}$  is selected based on  $\pmb{x}$ , therefore there is uncertainty about which  $\pmb{y}$  is selected. Furthermore, we assume that the true model with predictive distribution  $p(\pmb{y} \mid \pmb{x}, \pmb{w}^{*})$  and parameters  $\pmb{w}^{*}$  has generated the training data  $\mathcal{D}$  and will also generate the observed (real world)  $\pmb{y}^{*}$  from  $\pmb{x}$  that we want to predict. The true model is only revealed later, e.g. via more samples or by receiving knowledge about  $\pmb{w}^{*}$ . Hence, there is uncertainty about the parameters of the true model. Revealing the true model is viewed as drawing a true model from all possible true models according to their agreement with  $\mathcal{D}$ . Note, to reveal the true model is not necessary in our framework but helpful for the intuition of drawing a true model. We neither consider uncertainty about the model class nor the modeling nor about the training data. In summary, there is uncertainty about drawing a class from the predictive distribution of the given, pre-selected model and uncertainty about drawing the true parameters of the model distribution.
According to Apostolakis [1991] and Helton [1997], the aleatoric uncertainty is the variability of selecting a class  $\pmb{y}$  via  $p(\pmb{y} \mid \pmb{x}, \pmb{w})$ . Using the entropy, the aleatoric uncertainty is
$$
\mathrm {H} [ p (\boldsymbol {y} \mid \boldsymbol {x}, \boldsymbol {w}) ]. \tag {17}
$$