Credal Bayesian Deep Learning
Authors: Michele Caprio
Souradeep Dutta
Kuk Jin Jang
Vivian Lin
Radoslav Ivanov
Oleg Sokolsky
Insup Lee
Venue: Accepted by TMLR ()

Abstract: Uncertainty quantification and robustness to distribution shifts are important goals in machine learning and artificial intelligence. Although Bayesian Neural Networks (BNNs) allow for uncertainty in the predictions to be assessed, different sources of uncertainty are indistinguishable. We present Credal Bayesian Deep Learning (CBDL). 
Heuristically, CBDL allows to train an (uncountably) infinite ensemble of BNNs, using only finitely many elements. This is possible thanks to prior and likelihood finitely generated credal sets (FGCSs), a concept from the imprecise probability literature. Intuitively, convex combinations of a finite collection of prior-likelihood pairs are able to represent infinitely many such pairs. After training, CBDL outputs a set of posteriors on the parameters of the neural network. At inference time, such posterior set is used to derive a set of predictive distributions that is in turn utilized to distinguish between aleatoric and epistemic uncertainties, and to quantify them. The predictive set also produces either (i) a collection of outputs enjoying desirable probabilistic guarantees, or (ii) the single output that is deemed the best, that is, the one having the highest predictive lower probability -- another imprecise-probabilistic concept. CBDL is more robust than single BNNs to prior and likelihood misspecification, and to distribution shift. We show that CBDL is better at quantifying and disentangling different types of uncertainties than single BNNs and ensemble of BNNs.
In addition, we apply CBDL to two case studies to demonstrate its downstream tasks capabilities: one, for motion prediction in autonomous driving scenarios, and two, to model blood glucose and insulin dynamics for artificial pancreas control. We show that CBDL performs better when compared to an ensemble of BNNs baseline.

Published in Transactions on Machine Learning Research (10/2024)
Credal Bayesian Deep Learning
Michcle Caprio Deportment of Computer Science, University of Monchester
michele. michele.caprio wwwprio@@mmnccsstrrrrr...
Souradeep Dutta
souradeep@ece.uke wbc.:pe..eeeee........oooo ubc. ca
Department Computer Science, University of British Columbia
Kuk Jin Jang
(eaggkjjscas.upeenne edu
Department of Computer and Information Science, University of Pennsyleania
Vivian Lin
vilin www.ssuueann.nncc. edu
Depariment of Computer and Information Science, University of Pennsyluania
Radoslav Ivanov
ivanor@rpi.co edu
Department of Computer Science,
Rensselaer Polytecknic Institute
Oleg Sokolsky
sokolsky@seas upenn. edu
Department of Computer and Information Science, Universily of Pennsyluania
Insup Lee
lee Ceeass upeenn edu
Department Computer and Information Science, University of Pennsyluania
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id= 4NHF9AC5us
Abstract
Uncortainty quantification and robustness to distribution shifts are important goals in ma- chine learning and artificial intelligence. Although Bayesian Neural Networks (BNNs) allow for uncertainty the predictions be assessed, different sources of predictive uncertainty cannot be distinguished properly. We present Credal Bayesian Deep Learning (CBDL). Heuristically, CBDL allows to train (uncountably) infinite ensemble of BNNs, using only finitely many elements. This is possible thanks to prior and likelihood finitely generated credal sets (FGCSs), a concept from the imprecise probability literature. Intuitively, convex combinations of a finite collection of prior-likelihood pairs are able to represent infinitely many such pairs. After training, CBDL outputs a set of posteriors on the parameters of the neural network. At inference time, such posterior set used to derive a set of predic- tive distributions that is in turn utilized to distinguish between (predictive) aleatoric and epistemic uncertainties, and to quantify them. The predictive set also produces either (i) collection of outputs enjoying desirable probabilistic guarantees, (ii) the single output that is decmed the best, that is, the one having the highest predictive lower probability another iuprecise-probaabilstt concept. CBDL is more robust than single BNNs to prior and likelihood misspecification, and to distribution shift. We show that CBDL is better at quantifying and disentangling different types of (predictive) uncertaintics than single BNNs and ensemble of BNNs. In addition, we apply CBDL to two case studies to demonstrate its downstream tasks capabilities: one, for motion prediction in autonomous driving scenarios,

---

Published in Transactions on Machine Learning Research (10/2024)
and two, to model blood glucose and insulin dynamics for artificial pancreas control. We show that CBDL performs better when compared to an ensemble of BNNs baseline.
1 Introduction
One of the greatest virtues an individual can have is arguably being aware of their own ignorance, and acting cautiously as a consequence. Similarly, an autonomous system using neural networks (NNs) would greatly benefit from understanding the probabilistic properties of the NN's output (for example, its robustness to distribut shift), in order to incorporate them inlo any further decision-making. This paper collocates on the path of giving a machine such desirable quality.
In the last few years, there has been proliferation of work on calibrating (classification) NNs, in order to estimate the confidence in their outputs (Guo et al. 2017) or to produce conformal sets that are guaranteed to contain the true label, in a probably approximately correct (PAC) sense (Park et al. 2020). While such methods are a promising first step, they require a calibration set (in addition to the original training set) and cannot be directly used on Out-Of-Distribution data without further examples.
Bayesian Neural Networks (BNNs) offer one approach to overcome the above limitations. The Bayesian paradigm provides a rigorous framework to analyze and train uncertainly-aware neural networks, and more generally to support the development of learning algorithms (Jospin et al. 2022). In addition, it overcomes some of the drawbacks of deep learning models, namely that they are prone to overfitting, which adversely affects their generalization capabilities, and that they tend to be overconfident about their predictions when they provide a confidence interval. BNNs, though, are trained using a single prior, which may still suffer from miscalibration and robustness issues Lenk & Orme 2009
In this work we introduce Credal Bayesian Deep Learning (CBDL), a procodure that draws on concepts from the imprecise probability (IP) literature (Augustin al. 2014 Troffaes & de Cooman 2014 Walley 1991). Unlike other techniques in the fields of artificial intelligence (Al) and machine learning (ML) involving imprecise probabilities- that typically only focus on classification problems CBDL can be used for both classification and regression. It captures the ambiguity the designer faces when selecting which prior to choose for the parameters of a neural network, and which likelihood distribution to choose for the training data at hand.
CBDL can be thought of as a NN trained using prior and likelihood finitely generated credal sets (FGCSs), Pprior and Pikk, respectively They are convex sets of probability measures having finitely many extreme elements (the elements that cannot be written as a convex combination of one another), exPprior and exPiik respectively (see also Remark 2 A very simple example of a prior FGCS P Pprior is the collection of all the convex combinations of two one-dimensional Normal distributions N(H,, of) and and M(H2, oz), ie..
P Pprior P P= (N((1),7) + +1 (32 oz), for all B E [0, 1]}.
Consequently, in this toy example we have that exP exPprior prior = ( FGCSs are further
examined section 2.2
Given the use of finitely generated credal sets, CBDL can also be seen as a non-condensed (uncountably) infinite ensemble of BNNs each BNN corresponding to a pair L) of prior from the prior FGCS Perior and likelihood L from the likelihood FGCS Pik- Such infinite ensemble, though, is carried out using only finitely many elements, that is, only pairs (Pex, Iox) of components of the sets of extreme elements exPprier and exPiik This because every element P of Pprior can be obtained from convex combinations of the elements of exP prior, and similarly for the likelihood FGCS. After training, CBDL produces a posterior FGCS Prost on the parameters the neural network. At inference time, Ppost is used to derive a predictive FGCS Ppred, that is, given a new input, a set of plausible distributions over the space of outputs In turn,
IItuutivvly, the larger these credal scts, the higher prior and likelihood ambiguity the user faces. AAs we shall see in section 3.1 since computing the posteriors and the predictive distributions is oftentimes an intractable problem, we approximate the elements of P Posst and of Preed using Variational Inference (VI). As a consequence, we denote the VT-approximated posterior and predictive credal sets as Ppoet and P5 Ppred: respectively.
2

---

Published in Transactions on Machine Learning Research (10/2024)
P Preed generates set of outputs or single output, depending on the user's needs that enjoys desirable probabilistic guarantees, even when the elements of credal set Prred are approximated, e.g. using variational inference. CBDL also gives a way of quantifying and disentangling different types of uncertainties within Ppred-
We use a credal set approach to overcome some of the drawbacks of single BNNs. In particular, CBDL allows to counter the criticism to the practice in (standard) Bayesian statisties of (i) using a single, arbitrary prior to represent the initial state of ignorance of the agent, (fi) using non-informative priors to model ignorance, and (fii) using& single, arbitrary likelihood to represent the agent's knowledge about the sampling model As a consequence, credal sets make the analysis more robust to prior and likelihood misspecification. In addition, they make it possible to quantify and distinguish between epistemic and aleatoric uncertainties (EU and AU, respectively). This is desirable in light of several areas of recent ML research, such as Bayesian deep learning (Depeweg et al.} 2018 Kendall & Gal 2017). adversarial example detection (Smith Gal, 2018), and data augmentation in Bayesian classification (Kapoor et al. 2022).
AU refers to the uncertainty that is inherent to the data generating process; as such, it is irreducible. Think, for example, of a coin toss. No matter how many times the coin is tossed, the stochastic variability of the experiment cannot be eliminated. EU, instead, refers to the lack of knowledge about the data generating process; as such, it is reducible. It can be lessened on the basis of additional data. For example, after only a few tosses, we are unable to gauge whether a coin is biased or not, but if we repeat the experiment long enough, this type of uncertainty vanishes. We note in passing that predictive EU cannot be adequately captured using a single BNN (Hüllermeier & Waegeman 2021 Fellaji & Pennerath, 2024]. One reason for this is that selecting a unique prior and unique likelihood implicitly assumes perfect knowledge around the true prior and the true data generating process. Another, more subtle one, is explored in the last paragraph of section 2.1 Methods that disentangle between the two types of (predictive) uncertainties that are based on single distribution are ad-hoc, but are not theoretically well-justified. EU can be typically reduced by retraining the model using an augmented training set (Lin ct al., 2024) (c.g. via semantic preserving transformations (Kaur el al., 2023). Puzzle Mix (Kim el al., 2020), etc.). On the other hand, since AU is irreducible, there is an increasing need for ML techniques that are able to detect and flag its excess, so that the user can "proceed with caution".
Remark 1. EU should not be confused with the concept of epistemic probability (de Finetti 1974 1975 Walley 1991). In the subjective probability literature, epistemic probability con be captured by a single distribution. Its best definition can be found in Walley [1991, Sections 1.3.2 and 2.11 2.11.2). There, the author specifies how epistemic probabilities model logical or psychological degrees of partial belief of the agent. We remark, though, how de Finetti and Walley work with finitely additive probabilities, while in this paper we use countably additive probabilities.
The motivation for working with credal sets is threefold: (i) it allows to be robust against prior and likelihood misspecification; (ii) unlike when using a single probability distribution, it permits to represent ignorance in the sense of lack of knowledge; (iii) it allows to quantify and disentangle between EU and AU, A more in-depth discussion can be found in Appendix A In addition, we point out that despite a hierarchical Bayesian model (HBM) approach seems to be a viable alternative to one based on credal sets, these latter are better justified philosophically, and do not suffer from the same thcoretical shortcomings of HBM procedures (Bernardo, 1979 Hüllermeier & Waegemann 2021 Jellreys| 1946; Walley] 1991). A more detailed explanation can be
found in Appendix B Let mention that only compare CBDL against Bayesian techniques because comparison against non- mon-Bayesian ones is either difficult to justify, or unfair. Indeed, the initial knowledge component, captured by the prior distribution (or the set of priors, in our case), has no direct comparisons with other approaches. Take conformal prediction (Vork et al. 2022 Shafer & Vovk 2008 Gibbs et al. 2003) Barber et al. 2023) for instance. It is a model-free method, which means that no prior knowledge around the experiment at hand is required. The only choice the user makes is which non-conformity score to use. How would we compare this approach with the choice of different priors, which conveys the idea that the user knows that a true distribution over the space of neural network parameters exists, but it is not fully known? In addition, despite the ubiquitous claim that conformal prediction (CP) is an uncertainty quantification tool, it is actually not. CP is an uncertainty representation tool. Indeed, CP represents uncertainty via
&rriticisms (1) and (ii)) are also pointed out in Manchingal & Cuzzolin 2022 Section 2.2).
3

---

Published in Transactions on Machine Learning Research (10/2024)
the conformal prediction region (on the other hand, in this paper we represent uncertainty via credal sets). It does not quantify it: there is no real value attached to any kind of predictive uncertainty (aleatoric or epistemic). Some claim that the diameter of the conformal prediction region quantifies the uncertainty, but even in that case, it is unable to distinguish between AU and EU. Indeed, the diameter is a positive function of both: increases as both increase, and hence it cannot be used to distinguish between the two.
We summarize our contributions next: (I) We present CBDL, and develop the theoretical tools and the algorithm required to use it in practice. (2) We show that CBDL approach is more robust than single BNNs to prior and likelihood misspecification, and to distribution shifts. We also explain how, during inference, the credal set of posteriors Poost on the network parameters obtained during training is used to derive a credal set of predictive distributions Ppred, and in turn a set of outcomes or a single outcome that enjoys probabilistic guarantees 4
(3) We show how CBDL is better at quantifying and disentangling predictive AU and EU than single BNNs and ensemble of BNNs F
We also apply CBDL to two safety critical systems to demonstrate its downstream tasks capabilitics. One, motion prediction for autonomous driving, and two, the human insulin and blood glucose dynamics for artificial pancreas control. We demonstrate improvements in both these settings with respect to ensemble of BNNs methods, the reason being that better uncertainty quantification leads to positive impact on the decisions made using them.
Before moving on, let us point out that while CBDL pays a computation ional price coming from the use of credal sets, it is able to quantify both (predictive) EU and AU, and to do so in a principled manner, unlike single and ensembles of BNNs. In addition, it requires less stringent assumptions on the nature of the prior and likelihood ambiguity faced by the agent than other cuprecise-probabiiee billiiee--asseeeee techniques, as explained in Appendices
B and L We also stress that if the user prioritizes computational efficiency, they should use non-Bayesian mods they are faster to implement than Bayesian techniques. In saiety-critial situations instead - where using mnoertainty-inforeed methods is crucial for quantifying the types of uncertainties, but also for the outcome of the analysis at hand CBDL is a natural choice.
Structure of the Paper. Section 2 presents the needed preliminary concepts, followed by section 3 that introduces and discusses the CBDL algorithm, together with its theoretical properties. We present our experimental results in section 4 and we examine the related work in section Section c concludes our work. In the appendices, we give further theoretical and philosophical arguments and prove our claims.
Notation Summary. In this paper, we make extensive use of different notation and acronyms. We summarize them in Table T to help the reader navigate through them.
2 Background and Preliminaries
In this section, we present the background notions that are needed to understand our main results. In section 2.1 we introduce Bayesian Neural Networks. Section 2.2 discusses (finitely generated) credal sets, upper and lower probabilities, and imprecise highest density regions. Section 2.3 introduces the concepts of upper and lower entropy, which are used to quantify and disentangle AU and EU, The reader familiar with these concepts can skip to section 3
2.1 Bayesian Neural Networks
In line with the recent survey BNNs by Jospin et al. (002), Bayes' theorem can be stated as P(H D) [PDD| H))())))P( = P(D, H)/ P(D, HH', where H is hypothesis about which the agent holds some prior beliefs, and D is the data the agent uses to update their initial opinion. Probability distribution P(D Ht) represents how likely it to observe data likelihood, while probability distribution P(H) represents the agent's 's initial opinion around the plausibility
hypothesis were to be true, and is called
*Once again, as shall see in section 3.1 we approximate the elements of the posterior and the predictive credal sets using Variational Inference (VI).
We note in passing how recently Mucsányy et al. (2024) show that (fimite) deep ensemble are the current state-of-the-art methods for quantifying and discntangling difterent types of uncertaintics on ImageNet. Our experiments show that CBDL improves on (finite) cnsembles of BNNs, both uncertainty quantification and disentanglemenn, and in dowustream tasks performances. Together with the findings in Mucsányi al. 2024 this is an additional argument in favor of the effectiveness of our methodology.
4

---

Published in Transactions on Machine Learning Research (10/2024)
Notation and Acronyms FGCS
Meaning
Finitely Generated Credal Set
CBDL
Credal Bayesian Deep Learning
Upper case latin letters, such as P and Lower case latin letters, such as p and l
Probability measures, e-g. prior or likelihood
The pdf/pmit associated to probability measures
A finite collection of probability measures
Conv(.)
The convex hull operator
The convex hull of II, i.e.
TT'
IT' = CCIv
P prior
A prior FGCS
Pikk
A likelihood FGCS
P Post
A postcrior FGCS
VI
Variational Inference
A posterior FGCS whose extreme elements are the VI approximations of the "correct" ones
P post
P prod
A predictive FGCS
A predictive FGCS whose extreme elements are the VI approximations of the "correct" ones
P Prred
exP
The extreme elements of an FGCS P
BNN
Bayesian Neural Network
Training set; Dx is the set of training inputs,
D = Dx x Dy
and Dy is the set of training outputs
EBNN
Ensemble of Baycsian Neural Networks
I + ((x)
Function representing the neural network architecture
P. P
Lower and upper probabilities, respectively
HDR
Highest Density Region
THDR
Imprecise Highest Density Region
Imprecise Highest Density Region of level a associated with FGCS P
(P)(P)
H(P)
Entropy of probability measure P
H(P), H(P)
Upper and lower entropy associated with a credal set
EU(P)
Epistemic Uncertainty encoded in an FGCS P
AU(P)
Alcatoric Uncertainty encoded in an FGCS P
TU(P)
Total Uncertainly encoded in an FGCS P
#A
Cardinality of a generic set A
The act of computing a postcrior and a postcrior predictive distributions, respectively
post, pred
BMA
Bayesian Model Averaging
Table 1: Nolation and acronyms that we use throughout the paper, together with their meaning.
5

---

Published in Transactions on Machine Learning Research (10/2024)
of hypothesis H, and is called prior. The evidence available is encoded in P(D) = P(D, H')dH', while posterior probability P(H | D) represents the agent's updated opinion. Using Bayes' theorem to train a predictor can be understood as learning from data D: the Bayesian paradigm offers an established way of quantifying uncertainty in deep learning models.
BNNs are stochastic artificial neural networks (ANNs) trained using a Bayesian approach (Caprio & Mukher-
jee
Goan & Fookes 2020] Jospin al. 2022 Lampinen & Vehtari 2001 Titterington 2004| Wang & Yeung 2021). The goal of ANNs is to represent an arbitrary function y/ = +(z). Let & represent the parameters of the network, and call θ the space f belongs to. Stochastic neural networks are type of ANN built by introducing stochastic components to the network. This is achieved by giving the network either a stochastic activation or stochastic weights to simulate multiple possible models with their associ- ated probability distribution. This can be summarized as 0 ~ p(0), y = Φo(z) +E, where & depends on to highlight the stochastic nature of the neural network, p the density of probability measure P on o and & represents random noise to account for the fact that function by is just an approximation. The connection between the BNN notation and the general one of the Bayes' theorem is explained in the next two paragraphs.
2023b
To design BNN, the first step is to choose a deep neural network architecture, that is, functional model Φn. Then, the agent specifies the slochaslic model, that is, a prior distribution over the possible model parametrization p(0), and a prior confidence in the predictive power of the model ply x, 0). Given the usual assumption that multiple data points from the training set are independent, the product [( p(y xx 0) represents the likelihood of outputs y E Dy given inputs x E Dx and parameter 8, where (a) D = Dx x Dy is the training set; (b) Dx = {a;)7_ 1 is the collection of training inputs, which is a subset of the space X of inputs; (c) Dy = }= is the collection of training outputs, which a subset of the space y of outputs.
The model parametrization can be considered to be hypothesis H. Following Jospin et al. (002), we assume independence between model paramcters and training inputs Dx, in formulas Dx 11 8. Hence, Bayes' theorem can be rewritten as
p(0 D) = p(Dy Dx, 0)p(0)
p(Dy Dx 0' ((0)ddr x x((Dy Dx, O)p()).
Notice that the equality comes from having assumed Dx 1L A. Posterior density p(0 D) is typically high dimensional and highly nonconvex (Tzmailov et al., 2021c; Jospin et al. 2022), so computing it and sampling from it is difficult task. The first issue is tackled using Variational Inference (VI) procedures, while Markov Chain Monte Carlo (MCMC) methods address the second challenge. Both are reviewed - in the context of machine learning- - in Jospin et al.] (2022, Section V), where the authors also inspect their limitations. BNNs can be used for both regression and classification (Jospin et al. 2022, Section II); besides having a solid theoretical justification, there are practical benefits from using BNNs, as presented in Jospin et al. 2022 Section III).
Suppose now we collect a new input à of interost, and that want to predict its correct output g. Then, the posterior distribution p(0 D) over the network parameters comes to the rescue, as it allows us to compute the so-called posterior predictive distribution,
/ p(ū E. O)(0 D)do Eo Ee-poo a,0)].
p(ū P(
F., D)
where p(ū ( 0) is the model distribution we specified before. Posterior predictive p(ū a. D) tells us "how likely" output ÿ is to be the "correct one" for input ă, given the knowledge encapsulated in the data D we collected, which enters the computation via the posterior probability p(0 D). Let us add a remark here: oftentimes, scholars claim that, in a Bayesian setting, the distribution on the parameters f captures the epistemic uncertainty (EU) faced by the agent. This is a somehow agreeable premise, akin to a second-order distribution reasoning. we accept this assertion, though, we see how EU at the predictive level is not
We can write p as the Radon Nikodym derivative of P with respect some finite dominating measure H, that is,
p= dP/dp.
6

---

Published in Transactions on Machine Learning Research (10/2024)
quantifiable any more, since it gets washed away by taking the expectation E As we shall see later, CBDL allows to overcome this conceptual shortcoming.
2.2 Imprecise Probabilities
As CBDL is rooted in the theory of imprecise probabilities (IPs), in this section we give gentle introduction to the IP concepts we will use throughout the paper.
CBDL is based on the Bayesian sensilivily analysis (BSA) approach to IPs, that in turn is grounded in the dogma of ideal precision (DIP) [Berger 1984). WWlley 1991 Section 5.9). The DIP posits that any
problem there an ideal probability model which procise, but which may not be preciscly known. We call this condition ambiguity (Ellsberg
1961 Gilboa & Marinacci 2013
Facing ambiguity can be represented mathematically by set Pprior of priors and set Pikk of likelihoods that seem "plausible" or "fit" to express the agent's 's beliefs the parameters of interest and their knowledge of the data generating process (DGP). Generally speaking, the farther apart the "boundary elements" of the sets (i.e., their infimum and supremum), the higher the agent's 's ambiguity. Of course, if Pprior and Pikk are singletons we go back to the usual Bayesian paradigm.
A procedure based on sets Pprior and Pikk yields results that are more robust to prior and likelihood mis- specification than regular Bayesian method. In the presence of prior ignorance and indecisiveness about the sampling model, it is better to give answers in the form of intervals sets, rather than arbitrarily select prior and likelihood, and then update. Sets Pprior and Pikk allow to represent indecision, thus leading to less informative but more robust and valid conclusions.
Remark 2. Throughout the paper, we denote by 11 = [PI..... Pxh. k E E, a finite set of probabilities on a generic space Q2 such that for all j E (1..... other k - components of II. We denote by II' ils conver hull II' =\t Conv((I), i.e., the set of probabililies Q on S that can be written Q(A) = Σ 3, (A), for all A C Ω where the 3, 's are elements of [0, 1] that sum up to 1. In the lileralure, il is referved to as a Finilely Generated Credal Set (FGCS, Levi (1880); Cozman (00066). Notice then that the extreme elements of II' correspond to the elements of I1, in formulas e II. Simple graphical representations of finitely generated credol sets are given Figures and
.......); k}, Pj canmot be written as a conver combination of the
P((w5)
P
P1
Pa
P
P
P((w)))
P((02))
Figure 1: Suppose we are 3-class classification setting, so Ω = fwr, w2, w.). Then, any probability measure P on Ω can be seen probability vector. For example, suppose P([w1)) 0.6, P([w)) = 0.3, and P([ws)) = 0.1. We have that = (0.6, 0.3, 0.1)". Since its elements are positive and sum up to probability vector belongs the unit simplex, the purple triangle in the figure. Then, that IT' = Conv(II) the orange pentagon. It is a convex polygon with finitely many extreme elements, and it is the geometric representation of a finitely generated credal set.
specify II = (P.,... Ps}, and obtain consequence
Let us now introduce the concepts of lower and upper probabilities. The lower probability P associated with IT is given by P(A) = ren P(A), for all A C Ω. The upper probability P associated with IT is defined as the conjugate to P, that is, P(A) == - P(A") = sup P(A) [ for all A C Q. These definitions hold even if II is not finite. Then, we have the following important result.
Here, A=

---

Published in Transactions on Machine Learning Research (10/2024)
Proposition 3. P is the upper probability for II if and only if it is also the upper probability for II'. That is, ((A) suppen P(A) s (()), for all A C C. The some holds for the lower probobility.
A version of Proposition 8 proven in Dantzig 1963). while variant for finitely additive probability measures can be found in Walley 1991 Section 3.6). A simple graphical representation of upper and lower
probabilities for a set A is given in Figure 2
We now use lower probability P to define the a-level Imprecise Highest Density Region (IHDR), for some a E [0.1].
Definition 4. (Coolen 1992 Section 2) Let a be any value in [0, 1]. Then, set TR. (TT') Q is called (1 a) Imprecise Highest Density Region (IHDR) if
Pl[w E
2. (IR, de is minimum. If Q is ot most countable, we replace (IR,(TT) du with # #IR, (II',, where # denotes the cordinality operator.
Ps(w)
P(())
9(w)
@
-0.8 -0.6
-.46
42
0.2
0.4 0.6
0.8
Figure 2: In this figure, replica of Flint et al. distributions whose probability density functions (pdf's and are given by the dashed blue and brown curves, respectively. Their convex hull IT' Conv(IT) {Q Q BP + (1 P2 for all B E [0, 1]]. The pdf of element of II' is depicted by solid black curve. In addition, let A 0.8, -.44.. Then, P(A) f p2(w)dw 0, while P(A) is given by the red shaded area under Pl, that is, P(A) fo, pi (w)dw.
2017
Figure 1), 11 P3), where Pt and P2 two Normal
Definition 4 holds also if II exll' is not finite. Notice that condition 2 is needed so that IR, (II') is the subset of Q having the lowest possible cardinality, which still satisfies condition 1. By the definition of lower probability, Definition implics that P'[fw E IRa 2 1- a, for all P' E IT'. Here lies the appeal the IHDR concept. Let us give a simple example, borrowed from Caprio el al. (2024a). Suppose Ω (wi,...,45), IT (P1, P2, Pa}, and a 0.1. The numerical values for Px((w,)) are given Table 2 for all E (1,2,3) and all j (l,...,5) 5). Then, from Proposition 3 and Definition 4 we have that (II' [Wi, Wb, w2, w
W2
W4
W6
P1
0.7
0.25
0.03
0.01
0.01
P
0.6
0.2
0.1
0.05
0.05
P3
0.5
0.3
0.15
0.025
0.025
Table 2: Numerical values for our example. It is easy to see that the smallest subset of Ω that is assigned a probability of least 0.9 by all the elements II is (w1,W2;ws) ).
An operative way of building the IHDR is to consider the union of the (precise) Highest Density Regions (HDRs) of the elements of Il exTT Let us define them formally.
Here, by "operative" we mean that this procedure compute the THDR, easy carry out in practice.
8

---

Published in Transactions on Machine Learning Research (10/2024)
Definition 5. CCoolen 1992. Section 1) Pick any P3 II, JE {l....,k). Let be any value in |0, 1]. Then, set R., (P))) C is called (1 aa)-Hghhest Densily Region (HDR) for Pj is
Pillw 2a(3))2210 and J (RaIP))) du is a minimum.
If Q is at most countable, we replace SR.(P,) du with (P)( Equivalently Hyndman 1996
R. (2(P)) = {w 8: p)(w) 2 pj} C.
where Pj is the pdf or the probability mass function (pmf) of Pj. and pj is constant value. In particular, it is the largest constant such that Pjl[ E Ra(P)) 2 1 a.
In dimension 1, (P)) be interpreted as the smallest collection of elements of Ω (interval or union of intervals) to which distribution P3 assigns probability of at least - a. As we can sec, HDRs are a Bayesian counterpart of confidence intervals g We give a simple visual example in Figure
0.20
0.10
Pj
0.25
()))
@
75% HDR
Figure 3: The 0.25-HDR from a Normal Mixture density. This picture is a replica of Hyndman
1996 Figure 1).
The geometric representation of "75% probability according to P," is the area between the pdf curve ps (w) and the horizontal bar corresponding to 0.25 higher probability coverage (according to P2) would correspond to a lower constant, so p3 < kgg 0.25 for all a < 0.25. In the limit, we recover 100% coverage at p. =.
As we mentioned earlier, an operative way obtaining IHDR IR, (II') is by putting (RI( = Bo Thanks to Proposition by taking the union of the HDRs, we ensure that all the probability measures the credal set IT' Conv assign probability of least. - to the event {w TR, (IT))- In turn, this implies that Pfw IRa between the upper and lower probabilities of (w (II)) is bounded by a. To see this, notice that Pllw E IR, (IL' s so Pl[w e (II')) P[( IRa (RT)) a.
min.e((1....N) Pu IRa (II))) 2 1-a a. We also have that the difference
2.3 Quantifying and Disentangling Aleatoric and Epistemic Uncertainties
Recall that, given a probability measure P on a generic space 12 the (Shannon) entropy of P. is defined H(P) = E[- log p] most countable, we have that H(P) = EweR P((w)) log[P((w])] As pointed out by Dubois & Hüllermeier (2007): Hüllermeier & Wacgeman (2021) the entropy primarily captures the shape of the distribution, namely its "peakedness" or non-uniformity, and hence informs about the predictability of the outcome of a random experiment: the higher its value, the lower the predictability. Then, we can define the imprecise versions of the Shannon entropy as proposed by Abellán et al. (2006;; Hüllermeier Waegeman (2021) ((P) suppeI H(P) and H(P) == inf Penr H(P), called the upper and lower Shannon entropy, respectively Notice that. these definitions hold for all sets of probabilities, not just for (finitely generated) credal sets.
Sa loglp(w)) if Ω is uncountable, where p denotes the pdf of P. If Ω is
The upper entropy is measure of total uncertainty since it represents the minimum level of predictability
associated with the clements of II' In Abellán et al. (2006); Hüllermeier & Waegeman 2021 ) the authors
Heenee standard choices for the value of are 0.1, 0.05, 0.01.
10Tn Appendix we provide bounds the values of upper and lower entropy.

---

Published in Transactions on Machine Learning Research (10/2024)
posit that it can be decomposed as sum of alcatoric and epistemic uncertainties, and that this latter be specified as the difference between upper and lower entropy, thus obtaining
H(P') = H(P)) + H(P') - H(P)]] TU(I))
AU(TT)
BU(I)
where TUII)) denotes the total uncertainty associated with set II' AU(IT') is the AU associated with II', and EU(TT) represents the EU associated with II'. As we can see, if IT' is a singleton, then TU(IT' = AU(I)) and (I()))) = 0. This captures the idea that, in general, a single distribution can only gauge aleatoric uncertainty We have the following proposition.
Proposition 6. Let II, be sels of probabilily measures the ones considered in Remark 2 Then, supren (P) = (P) s H(P)) sup H(P)) and inf pen H H(P) H(P)) inf HPP)).
Proposition G tells us that the upper entropy of the extreme elements in exll' is a lower bound for the upper entropy of the whole credal set II', and that the lower entropy of the extreme elements in II is equivalent to the lower entropy of the whole credal set IT'. These facts imply that AU(ITT) AU(IT), and that (U(I)' 2 EU(.).. In addition, as consequence of (Smieja & Tabor 2012 Theorem 111.1), we have that EU(II' & ETI + log(#II). In turn, we have that EU(IT' E EUU(T) + log(#II))-
We briefly mention that other uncertainty measures based on credal sets are also available (see Bronevich & Rozenberg (021)); Hofman al. (2024); Hillermeier & Waaggmaan (202I) or Appendix D for few examples) and they be used in place of upper and lower entropy to quantily EU and AU within our credal region P, long the measure chosen for the total uncertainty is bounded.
Let us also add small remark. Although EU can be reduced with an increasing amount of data, it is very seldom the case that it goes to zero when a finite amount of data is collected. When that happens, it means that the initial uncertainty was very low the first place. Typically, EU goes to zero only asymptotically, as finite amount of data almost never enough to overcome initial uncertainty (Walley 1991 Wimmer et al. 2023).
3
Our Procedure and Its Properties
This the main portion of the paper. section theoretical properties are derived in section 3.2
3.1
we present and discuss the CBDL algorithm. Its
3.1 CBDL algorithm
Recall that D = Dx x Dy denotes the training set, where Dx = {x,);=1 C X is the collection training inputs, Dy = [ y is the collection of training outputs, and X' and y denote the input and output spaces, respectively. We then denote by P generic prior on the parameters and by L= L,,0 generic likelihood the posterior from prior P and likelihood L using BNN is designated by post[P, L], and the act of deriving the posterior predictive from posterior P(. D) and likelihood L is designated by pred | D), L]. The CBDL procedure is presented in Algorithu and it is discussed in the following paragraphs.
of BNN having pdf p, the space y of outputs having pdf ( = Ey The act of computing
During training, in Step the user specifies K priors on the parameters of the neural network, that constitute the extrema exPprior the prior FGCS Pprior- Similarly, in Step they clicit S likolihoods capturing the possible architectures of the neural network, that correspond to the extrema exPiik of the likelihood FGCS Pik- Let us give an example. Outlined in Jospin et. al. (0222 Sections TV-B and IV-C1), for classification, the standard process for BNNs involves
It can gauge EU, though, if is sccond-order distribution, if it the result of ensemble of probabilities. Whhhich, although easy compute, can sometimes suffer from shortcomings coming from lacking the monotonicity property. That is, there are credal seta that, although nested into each other, have the same upper and/or for lower entropy (Hüllermeier & Waegeman 2021
10

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 1 Credal Bayesian Deep Learning (CBDL) Training and Inference
During Truining
Step 1 Specify K priors exP exPprior prior = { K
[k=]
Step 2 Specify S likelihoods exPlik = [[g]] 15
1==1
Step Compute Ps. D) = post Pkx Lgd. for all is and all s b % Approximated via VI by Pk, % During Inference
Inputs: New input ã E X
Parameters: Confidence parameter a E [0,1]
Outputs: Predictive Aleatoric and Epistemic Uncertainties, a-level IHDR
Step Compute ppred = pred L.d), for all k and all
> % Approximated via VI by ppredy %
k.s
k,s
Step Compute and return (U prea) and the bounds for BU (roaa))
Step Compute and return the (1 - II-I IR, (Ppred)
A Normal prior with zero mean and diagonal covariance o²1 on the coefficients of the network, that is, p(0)= = o2 In the context of CBDL, we could specify, e.g. exP = {P p(0) N(H, I²1), μ E {p-, 0, p4): o² E (3.7]] That is, the extreme elements of the prior credal set are five independent Normals having different levels of "fatness" of the tails, and centered at a vector J4 having positive entries, a vector μl- having negative entries, and a vector 0 having entries equal to 0. They capture the ideas of positive bias, negative bias, and no bias of the coefficients, respectively. This is done to hedge against possible prior misspecification.
A Categorical likelihood, p(y | xr, 0) = (x))), whose parameter is given by the output of a functional model &g. In the context of CBDL, we could specify the set of extreme elements of the likelihood credal set as exPii. = {L. (v. = Cat (f.,o(z))) 8 E (1)....S)). Specifying set exPlik. then, corresponds to eliciting a finite number S possible (parametrized) architectures for the neural network $s,0, s E fl..... 5), and obtain, as a consoquence, S categorical distributions {Cat(# (f.,) This captures the ambiguity around the true data generating process faced by the agent, and allows them to hedge against. likelihood misspecification.
More in general, we can use the priors and likelihoods that better fit the type of analysis we are performing 16 For example, for the choice of the priors we refer to Fortuin et al. (2021). where the authors study the problem of selecting the right type of prior for BNNs.
Step 3 performs an element-wist wise application of Bayes' rule for all the elements of exP prior and exPlik- Each posterior is approximated using the Variational Inference (VI) method (Jospin et al.) 2022 Section V). That is, we project every posterior (- D) onto set S of "well-behaved distributions (e.g. Normals) using the KL divergence. In formulas, Pr.s = arg minges | D)J. By "well-behaved", we mean that they have to satisfy the conditions in Zhang & Gao 2200 Sections 2 and 3) This ensures that as the sample size goes to infinity, the approximated posteriors converge to the true data generating process. We also point out how despite using a VI approximation for the exact posteriors, as we shall see in the next section, the credal set of approximated posteriors is closer to the "oracle" posterior po D) than any of its elements taken individually. As a consequence, working with credal sets leads to VI posterior approximations that are better than the ones resulting from a single BNN, or an ensemble of BNNs, where several BNNs are combined into one.
Remark 7. Although highly unlikely in practice, it is theoretically possible that the VI approrimation of the (finite) set (Pr.o(- Di} of posteriors a singleton, see Figure 4 While the conditions in Zhang 8 Gao (2020) guarantee that asymptotically the approzimated posteriors coincide with the true data generating pro- cess, this lypically does not happen with finile-dimennsionall dolasels. As consequence, oblaining a singleton when projecting (P D)),, onto S may result in an underestimation of the uncertainties faced by the user. In that case, we either consider different set - whose elements still satisfy the conditions in Zhang
Chhoosing to 5 priors and likelihoods usually enough to safely hedge against. prior and likelihood misspecification.
"e assume that the conditions on priors and the likelihoods given in Zhang & Gao [2020] are satisfied.
11

---

Published in Transactions on Machine Learning Research (10/2024)
E Gao 2020, Sections 2 and 3) on which to project (Pk,al- D)}k, according to the KL divergence, or use different "projection operator", that is, divergence differrnn from the KL. For example, Rényi and x² divergences, or Hellinger and total variation metrics are suggested by Zhang & Gao (2020). Alternatively, we con consider a different approzimation strategy altogether, for instance the Laplace approzimation Ritter
ol. 2018).
A
S
I
KI
P((-DD D)) P2(-1D) D))
Figure 4: Let Ae denote the space of probability measures on θ Suppose that in the analysis at hand specified three priors and only one likelihood, so S 1 and we can drop the index. Let {P&(- D)) i the collection of exact posteriors, that the black segment represents the exact posterior FGCS. Then, if we project the elements of (Pe(- D)} onto S₁ via the KL. divergence, we obtain the same distribution P. This detrimental the analysis because such approximation underestimates the (posterior) epistemic (and possibly also aleatoric) uncertainty faced by the agent. Then, the user could specify a different set S of "well-behaved" distributions onto which project the elements of (- D)} In the figure, Pr. Pr, and P3. The convex hull of these latter, captured by the red shaded triangle, represents the variational upproximation of the exact posterior FGCS.
that they projected onto 8, via the KL divergence to obtain
After Step 3, we obtain finite set {Pr, of VI approxxintto of the posteriors on the network parameters, whose cardinality is K x S. Its convex hull constitutes Ppastt that is, the VT-approximated posterior FGCS. We assume that exP post = [Pr,s This is an assumption because it may well be that due to the approximation procedure some of the elements of (PR,*)e, are not independent of one another. We defer to future work the design of a procedure that finds the elements of {PA,s]s.s that cannol be written as a convex combination of one another.
Being combinatorial task, Step 3 is computational bottleneck of Algorithm I We have to calculate K x S VT approximations to as many posteriors, but this allows us to forego any additional assumptions on the nature of the lower and upper probabilities that are oftentimes required by other seerecise-probabiliie based techniques 1b Clearly, CBDL is simplified if either Pprior or Pik are singletons. Notice that in the case
that Prrior and Pikk are both not singletons, for all A C θ the interval [P(A), P(A)] is wider than the case when one or the other is a singleton. the limiting case where both are singletons, we retrieve the usual Bayesian updating, so the interval shrinks down to point.
Before moving to inference time, let remark difference between Bayesian Model Averaging (BMA) and CBDL. In BMA, the user specifies distribution on the models. Translated in the nolation we use in this work, this means having discrete distribution Q over the K x S prior-likelhhodd combinations, that is, over the elements of exP post. Such Q is then used to select an element pa from the (Vl-approximated) posterior FGCS P Ppost as
P Pront 3 pr
k.N
18ff we willing make such assumptions, Theorem 10 only upper prior and upper likelibood.
Appendix
shows how compute upper posterior using
12

---

Published in Transactions on Machine Learning Research (10/2024)
where Q((Pk,,t) [0.1] for all I and all 8, and Ex.e Q((Pk,s) = (Caprio & Mukheejee 2023a]. For example, if Q is the discrete uniform distribution on fl...., K x 5}, then p is the so-called center of grovity of the credal set Prost (Miranda & Montes 2023 Section 3.2). Instead, CBDL. does not select a unique distribution from Ppost- Rather, distributions are kept separate so to be able to derive a predictive FGCS in Step 4 of Algorithm 1 that is in turn used to quantify and disentangle predictive uncertainties, and to compute the predictive IHDR, that is, a collection of outputs having a high probability of being the correct ones for a new input 2.
During inference, new imput # provided. In Step 4, every element of exp post is used derive a predictive distribution k,s = pred[ Pk.., Lgd] on the output space y In particular, for every h and every 8, the pdf pred of ppred is obtained as
Pk. (Ø ž, 21, y1
In Ini
( (ū (2 0) pk, 21 M1.............................. Fn yr )d0
( (ū 0t 0) (0)d00
where 6x is the pdf of likelihood L Pk.s is the pdf of the true posterior Pt D), Pk.s is the pdf of
the VT-approximated posterior Pk.s, and ÿ is the output associated to the new input t (see Appendix F for more details). Each predictive distribution pered is approximated to A (e.g. using Normals) via
Variational Inference. The convex hull of the collection [Ppreed hR, having cardinality K x S constitutes the
Vlaaprooimmttdd predictive FGCS P Prred. Similarly to what we did for exPpost: (Prred) JR,s-
post. we assume that exP exP pred
In Step 5, building on the results in section 2.3 and in particular on Proposition 6 compute and return AU(Pproa) and the bounds for EU(P prea). In particular, we have
(U pred) 2o AU(Pprea) = 11 (Dprodd)
(2)
and
EU(P prea) R FU(P prea) E (Ppredd) - H (ppeed) H (Pprodd) H ((Pprod)) + log( K x S)],
(3)
where (i) ((Ppred) = ming, H ((Prrr)); (ii) ((prre)) = max H ((Ppre)); and (iii) Ppred is the "true" k.s prodictive FGCS, that is, the one we would have obtained had we boen able to compute the exact posteriors in Step and the exact predictive distributions in Step 4. Let point out a salient feature of CBDL. The AU and the (bounds for) the EU associated with the V1-approximated predictive FGCS Ppxed embed uncertainty comparable to uncountably infinite ensemble of BNNs, i.e., an ensemble of BNNs of cardinality N1, despite the simple and intuitive mathematics over the finite set exP They are not merely pessimistic results on the uncertainties associated with a finite ensemble of BNNs.
We observe that we compute the AU and EU associated with the (Vl-approximatec) predictive credal set P Ppred, rather than those related the (VI-approximated) posterior credal set Ppost- We do so because we are ultimately interested in reporting the uncertainty around the predicted outputs given new input in the problem at hand, more than the uncertainty on the parameters of the NN.
Before commenting on the next step, let us pause here and add a remark. While a "bad choice" of priors and likelihoods may lead to large values upper and lower entropy - H ((Ppred) and H ((ppred), respectively this is not a risk confined to our procedure. Poor modeling choices are an unavoidable risk in model-basod techniques. This gave rise to the famous adage by George Box "essentially, all models are wrong but some are useful" (Box 1976 [6 We maintain that our method is indeed useful, since it overcomes some of the shortcomings of traditional Bayesian techniques as explained in Appendices A and B As for "regular" Bayesian methods, though, for our approach too the designer will need to make "plausible" choices for priors
16 Curioussy similar motivation was brought forward by SecuooDionyssun the Arcopagite in favor of the use of sucred images in the Christian tradition (Migne 1857 While they do not capture the essence of God, these defective approximations help the believer's 's thought to elevate.
13

---

Published in Transactions on Machine Learning Research (10/2024)
and likelihoods. A further positive aspect of working with credal sets is that they are able to "self-regulate" That is, in the case of prior-likelhhood conflict - which happens e.g. if the prior set is ill-specified (Reimherr et al., 2021) the posterior credal set, and in turn the predictive credal set too, will be wider. This will be reflected in the uncortainty measures which will register an excess of posterior, and in turn predictive, epistemic uncertainties.
Finally, in Step 6, we compute and return the a-level Imprecise Highest Density Region IR (Porea) for the V1-approximated predictive FGCS P Ppred, which approximates the IHDR for the "true" predictive FGCS Prred. It is the smallest subset of 3 such that [ E (Preaa) 2 1 - a, for all ppred E Ppeed and for some a E [o, I]. It can be interpreted as the smallest collection of outputs ÿ that have a high probability of being the correct ones for the new input ž, according to all the distributions in Ppred- Or equivalently, we can say that the correct output for the new imput È belongs to IHDR. (Ppea)) with lower probability of at least 1 - a, a probabilistic guarantee for the set of outputs generated by our procedure. Notice also that the size of I (Ppreed is an increasing function of both (predictive) AU and EU. As a consequence, it is related, but it is not oqual, to the (prodictive) AU the agent faces. If we want to avoid to perform the procedure only to discover that IR (Ppred) is "too large", then we can add an "AU check" after Step This, together with computing I Ra. (Proo)) in a classification setting, is explored in Appendices G and
As we have seen in section 2.2 we find IR, (Pprea) by taking the union of the K x S HDRs Ra the extreme elements exPp exPpred of P Ppreed The HDR of a well-known distribution (for instance, a Normal) can be routinely obtained in R, e.g. using package HDInterval (Juat et al. 2022).
R (Ppred) of
We conclude this section with two remarks. First, we point out how CBDL does not depend on the meth used to approximate the posterior and the predictive distributions: Variational Inference be substituted by other approaches. CBDL can also be easily adapted to other TU, AU, and EU measures, as long as the measure chosen for the total uncertainty is bounded. Second, Step 6 can be effortlessly modified so that CBDL produces the collection of outputs having the highest lower density. This is in line with the Naive Credal Classifier theory (Cozman 2000a Zaffalon 2002). In this case, we forego control on the accuracy level 1 a of the output region produced by CBDL. It is implemented as follows. In the modified version of Step 6, we compute
arg max min spred
(7);
JEY
k,s
where ppree is the pdf of the V1-approximated predictive distribution ppred If such arg max is not a singleton, ki and the user is set on CBDL outputting a unique value ÿ for the new input ã, they can select one element uniformly at random from the arg max.
3.2 Theoretical Properties of CBDL
Working with credal sets makes CBDL more robust to distribution misspecification and shifts than single BNNs. To see this, we present the following general result, and then we apply it to our case.
Let II' be an FGCS as in Remark 2 and consider a probability measure such that II..
Proposition 8. Call d any metric and div any divergence on the space of probabilily measures of interest. Let d(II', v == inf d(P', V and ) == inf div(P")[V) Then, for all p' E II', d(II', V) < d(P', 4) and div(IT', V) < (P))))
Proposition holds II' is any set of probabilitics, not just an FGCS In Appendix I we show that the above result still holds if the elements of IT' and vi are defined on Faclidean spaces having different dimensions Caa & Lim 2022 Caprio 2022).
Let now apply Proposition 8 to CBDL. Suppose that, when designing single BNN, an agent chooses likelihood L, while when implementing CBDL, they specify in Step 2 a finite set of likelihoods exPıik { a= 1, S 2 2, and then let the induced credal set Pik Conv(exPua)) represent their uncertainty around
More in general, Proposition holds for any type of function f. not just metrics divergences, because Proposition essentially only applies the definition of the infimum.
14

---

Published in Transactions on Machine Learning Research (10/2024)
the sampling model. Assume that L E exPikk This means that when designing the single BNN, the agent chooses arbitrarily which of the elements of exPiik to use. Suppose also that the "oracle" data generating process L is different from L, L * L, so that we are actually in the presence of likelihood misspecification. Then, we have two cases. (1) If the true sampling model L' belongs to Plik, then the distance - mcasured via a metric or divergence between Pikk and L is O, while that, between L and L is positive. (2) If L E Plak, then the distance between Pikk and L is no larger than the distance between and L no matter (i) which metric or distance we use (Proposition 8), (ii) whether or not L and the elements of Pilk are defined on the same Euclidean space (Appendix Lemma 17p. A visual representation is given in Figure A similar argument holds if part of the training dataset D is generated by one distribution Lg. and the remaining part is generated by Lg: a situation that we can categorize as distribution shift. Even if in the case of a single BNN the user were able to elicit exactly L = Li, the likelihood would still be misspecified for the data that come from Lg. This shortcoming is overcome by a credal set approach, in which either both oracle distributions Ls and Lg belong to the credal sct Plik, or the distance between Pikk and cither of them is no larger than that between the single chosen one L, and Li and Lg.
A y
d. 1k
Plik
d1..1")
Figure 5: CRDL is more robust to distribution shifts than single BNNs. Here Pikk is the convex hull of five plausible likelihoods, and d denotes generic metric on the space Ay of probabilities on V. We see how d(Pus. I.) < d(L, I.)): if we replace metric d by a generic divergence div, the inequality would still hold.
Let add remark here. Assume that the "oracle" prior po is in the prior credal set Pprior and that the "oracle" likelihood 7 is in the likelihood credal set Pik- Then, it is immediate see that the "oracle" posterior (. D) belongs to the posterior credal set Poost- Naturally, this does not imply that the posterior credal set collapses to P'(- D). In general, it is unlikely that a finite amount of data is able to completely annihilate all the epistemic uncertainty faced by the agent. What may happen is that if the training set is large enough, Ppast may be inscribed in a ball of small radius around (. D). This does not mean that epistemic uncertainty. Rather, the relative
we suffer from under-confidence due to larger-than-necessee epistemic uncertainty (measured by the difference between prior and posterior uncertainty, divided by the prior uncertainty) drops significantly. In addition, working with sets of prior and likelihoods allows to hedge against prior and likelihood misspecification, consequence of Proposition S
4 Experiments
From the previous sections, it is clear that CBDL improves on the uncertainty quantification capabilities of single BNN 15 CBDL allows a better quantification of predictive AU because of its robustness to misspeci- fication stemming from Proposition g In a sense, the predictive AU quantificd by a single BNN is a function of the choices of prior and likelihood made by the user. In addition, as we pointed out before (e.g. in the last paragraph of section 2.1). predictive EU cannot be obtained in a theoretically principled manner from single BNN (Hüllermeier & Wacgeman 2021 Fellaji & Pennerath 2024
Here, we show that these theoretical arguments are also backed by empirical evidence. In particular, in section we compare CBDL to the method proposed in Krishnan & Tickoo (2020), in which a single Bayesian Neural Network's output is utilized to estimate both predictive epistemic and aleatoric uncertainties. We also show that CBDL improves on the ensemble of BNNs (EBNN) proposed in Cobb et al. (2019), that we
18TTis includes empirical Bayes heoholgggeess KKishhan
2020
15

---

Published in Transactions on Machine Learning Research (10/2024)
treat as an ablation that is naive extension of (Krishnan & Tickoo 2020 to multiple BNNs. In section 4.2 we analyze the downstrream task performance of CBDL. We show that it performs better than an ensemble of BNNs (EBNN) (Cobb et al., 2 To demonstrate the utility of our method, we study the behavior of certain safety-critical sottings under distribution shifts and its ramifications. One, for motion prediction in autonomous driving scenarios, and two, to model blood glucose and insulin dynamics for artificial pancreas control.
4.1 (Predictive) Uncertainty Quantification
Distribution shifts can introduce uncertainties in a system, which in turn can render the predictions mean- ingless. This can be due to naturally occurring corruptions, as introduced in Hendrycks & Dietterich (2019)
for image datasets. The authors introduced 18 different noise types, which can be varied across 5 different
severity levels, ranging from low (severity = 1), to medium (severity = 2, 3), and high (severity = 4. 5). The intuition is that in the current, context, increasing the noise severity should generally result. in higher uncertainty. We evaluate our CBDL method on standard image datasets, CIFAR-10 10 (Krizhevsky et al. 2009), SVHN (Netzer et al. 2011), Fashion-MNNI MNIST (Xiao et al. 2017), and MNIST (Lecun et al. 1998) We use a slightly different set of perturbations than those introduced in Mu & Gilmer (2019) for gray-scale images like MNIST and Fashion-MNIST. Additionally, we perform cross domain testing for each dataset, where we expect the predictive uncertainties to be higher. We implement and train Resnet-20 Bayesian Neural Network model inside the library Bayesian-torch (Krishnan et al. 2019). For each dataset, we train 4 different networks initialized with different seeds on the prior and with the same architecture. This cor- responds to eliciting a prior FGCS Pprior such that exPptior = {Pex ,..,, ) so that K in Step 1 of Algorithum] 1 is equal to 4, and a likelihood FGCS that is singleton, Pikk = exPiik = (L), so that S in Step 2 of Algorithm ! is equal to 1. We use a learning-rate rate of 0.001, batch-size of 128, and train the networks using Mcan-Ficld Variational Inference for 200 epochs. The inference is carried out by performing multiple forward passes through parameters drawn from the posterior distribution. We used 20 Monte-Carlo samples in the experiments.
Baselines. In Krishnan & Tickoo 2020), for a single BNN output, the predictive distribution is obtained through multiple stochastic forward passes on the network while sampling from the weight posteriors using Monte Carlo estimators. In their work, they define the overall entropy and the entropy of the predictive distribution as the predictive entropy (Krishnan & Tickoo 2020 (C.2)). This quantity captures a combination of predictive alcatoric and epistemic uncertainties. Additionally, they define the mutual information between weight posterior and predictive distribution as the predictive epistemic uncertainty. Finally, the predictive alcatoric uncertainty is defined as the expected entropy. The predictive epistemic and alcatoric uncertainties sum to the predictive entropy. In our experiments, we use these definitions to compute the respective quantities.
In Cobb et al. 2019)) the authors consider different BNNs, but instead of kcoping them scparate and use them to build predictive credal set, they average them out. Similar to theirs, we elicit the following procedure, that call ensemble of BNNs (EBNN).
Consider R E N22 different BNNs, and compute the posterior distribution on the parameters. They induce R predictive distributions on the output space y, each having mean Mr and variance of r E (1,..., [R.... R}. We call EBNN distribution Pens Normal having mean plena 1/R D == Pr and covariance matrix o where a = =/ of 1/(R- - 1) ER (1r - )2 In section 4.2 we use the a-level HDR Ro (Pens)
associated with Pens as a baseline for the IHDR I IRa. (Ppred) computed at Step 6 of Algorithm 1
Following Cobb et al. (2019), for EBNN we posit that 1/R of captures the (predictive) alcatoric uncer- tainty associated with Pens. and L/(R- 1/(R-1) Ei- (H, - Meus)² captures the (predictive) epistemic uncertainty associated with Prmss 19 For CBDL, we look at the value of AU prod = H(Ppred) from equation 2 and at the lower bound H (predd (pppeed) for (U(Ppr)d) from equation
It is enough to focus on the lower
Nottic that in this case we retain the assumption that the EBNN distribution Pens on has mean pens and covariance matrix 2 P J, but we do not require that is Normal. That because in the four image datasets that we consider, the output. space y is finite.
16

---

Published in Transactions on Machine Learning Research (10/2024)
bound because the upper bound is given by H ((Ppred) ((Prred) log(4), (),, and log(4) R 0.6 is fixed value. Hence, the trend of both lower and upper bounds for E.U(P prea) as the severity of corruption changes is the samee.
Summary of our Results. Two scenarios are evaluated: an In-Distribution (ID) scenario with corruption by noises of increasing severity, and an Out-Oii-istributioon (OOD) assessment. The following main points summarize our findings:
(ID Evaluation) For both our proposed method and that of Krishnan Tickoo (020), for in- creasing severities of noise corruption, predictive epistemic and aleatoric uncertainties increase. In comparison, the baseline EBNN method demonstrates counterintuitive result, where the predictive aleatoric uncertainty decreases the severity of corruption increases. The table of values can be found in Appendix N
(ID) Evaluation) To assess the utility of our proposed method in downstream tasks, we analyze the accuracy versus rejection rate for all methods. Table 3 summarizes the results. Across MNIST-C and Fashion-MNIST--C, CBDL exhibits higher average accuracy across the range of rejection rales. For the remaining datasets, it demonstrates results comparable to the baseline (see section 4.1.1 for ovaluation dctails)
3. (OOD Evaluation) To evaluate OOD detection, models trained on particular dataset are tested on the other three datasets with the goal of analyzing the behavior of the predictive uncertainties. Tables and summarize the results. Across all the evaluated datasets, CBDL had the most consistent behavior of exhibiting a large increase in predictive AU, relative to the increase in predictive EU when tested on an OOD dataset. comparison, EBNN shows a decrease in predictive AU, and the single BNN shows inconsistent behavior depending on the dataset.
COC
MNIST-C
Fashion MNIST-C
SVHN
Severity
BNN
Ensemble
CBDL
BNN
Ensemble
CBDL
BNN
Ensemble
CBDL
BNN
Ensemble
CEDL
0.754
0.983
0.982
0.645
0.919
0.929
0.525
0.730
0.748
0.698
0.913
0.909
0.761
0.947
0.946
0.680
0.879
0.896
0.537
0.655
0.641
0.703
0.843
0.835
3
0.719
0.906
0.907
0.617
0,801
0.811
0.142
0.198
0.529
0.669
0.764
0.758
0.733
0.639
0.718
0.737
0.332
0.421
0.489
0.608
0.586
U.588
0.661
0.734
0.734
0.404
0.495
0.539
0.300
0.351
0.391
0.455
0.378
0.385
Moan
0.732
0.882
0.881
0.597
0.762
0.788
0.433
0.531
0.560
0.627
0.697
0.695
Table 3: In-Distribution evaluation on noise corruptions of increasing severity. Average area under accuracy rejection rate curve. Higher numbers indicate overall better accuracy over a range of rejection rates.
4.1.1 In-distribution Evaluation
Overall, CBDL demonstrates the most consistent behavior in terms of increasing levels of predictive epistemic and alcatoric uncertaintics as the severity of noise corruption increases. The full table of results can be found in Appendix N In our experiments, for CIFAR- 10 the single BNN also exhibits similarly consistent behavior. In MNIST and Fashion-MNIST, the BNN seems to be more consistent than CBDL, whereas for SVHN it seems to show a more inconsistent behavior. These results allow us to conclude that CBDL is comparable if not better than BNN in terms of In-Distribution behavior.
Accuracy vs Rejection Rate. As the quantitics pertaining to different approaches are not comparable, we perform an additional evaluation comparing the accuracy vs the rejection rate of given method. In this task, for a given uncertainty threshold, if the uncertainty quantity exceeds such threshold, then the prediction is rejected. Of those not rejected, the accuracy is computed. A range of uncertainty thresholds is tested for each method and plotted. Figure 2 depicts an example of the plot. There, for lower severities (c.g. 1-2) of Gaussian blur noise, CBDL exhibits better performance and converges to 100% accuracy at lower rejection rate. Intuitively, for the same range of rejection rates, a higher curve signifies better performance, as less samples are rejected while achieving a higher accuracy. However, for higher severities (above 3), the single BNN seems to show better performance. Even so, the overall rejection rate is high to achieve such accuracy. The entirety of the results be found in Appendix P To quantify the differences, the areas
17

---

Published in Transactions on Machine Learning Research (10/2024)
CBDL
Ensemble
BNN
Epistoric
Aleatoric:
Spistemic
Aleatoric
Epistemic
Alcasoric
In Clen-
0.102
0.031
0,008
0.076
0.151
0.179
CIFAR
MNIST
0.184 (1.801]
0.142 (4.626)
0.021 (2.723)
0.044 (0.572)
0.445 (2.56]
0.772 (4.312)
Fashion
0.183 (1.791)
0.147 (4.808
0.022 (2.822)
0.042 (0.067)
0.431 (2.861)
0.811 (4.63)
MNIST
SVHN
0.183 [1.789]
0.141 (4.606)
(2.421)
(0.608)
0.394 (2.612)
0.647 (3.616)
Epistemic
Aleatoric:
Epistemic
Alsatoric
Epistemic
Aleatoric
In Dist-Clean
0.188
0.057
0.012
0.063
0.198
0.071
CIFAR
0.185 [0.986]
0.167 (2.94)
0.023 (1.918)
0.042 (0.67)
0.821 (4. (35)
0.372 (5. (6.1
MNIST
Fashion
0.162 [0.866]
0.187 (3.287)
0.019 (1.618)
0.035 (0.564)
0.919 (4.654)
0.461 (6.496)
MNIST
SVHN
0.190 (1.016]
0.108 (2.963)
0.022 (L847)
0.643 (0.69)
0.923 (4.674)
0.297 (4.183)
Epistemic
Aleatoric
Epistemic
Aleatoric
Epiatemic
Aleatoric
Fashion
In Dist-Clean
0.121
0.029
0.007
0.075
0.197
0.351
MNIST
CIFAR
0.205 (1.705)
0.104 (3.561)
(3.178)
(0.667)
(2.073)
629)
MNIST
0.165 [1.367]
0.182 192)
0.021 (3.069)
0.033 (0. (434
0.909 (4.618)
0.328 (0.932)
SVHN
0.218 (1.81)
0.109 (3.742)
0.026 (3.827)
0.046 (0.618)
0.398 (3.036)
0.303 (0.867)
Epistemic
Aleatoric
Epistemic
Aleatoric
Eplatemic
Aleatoric
In Dist-Clean
0.061
0.01
0.004
0.083
0.097
0.107
SVHN
CIFAR
0.181 [2.992]
0,121 (11.778)
0.026 (6.029)
0.040 (0.479)
0.635 (6.765)
0.716(6.69)
MNIST
0.198 [3.272]
0.063 (6. 154)
0.022 (5.495)
0.056 (0.675)
0.329 (3.3933
0.336 (3. (39)
Fashion
0.139 (3.258)
0.113 (10.984)
0.026 (6.002)
0.041 (0.491)
0.103 (6.375)
0.125 (6.178)
MNIST
Table 4: Distri-Distributionn evaluation. Across all datasets, when tested on datasets outside of the training dataset, CBDL demonstrated the most consistent behavior in terms of predictive uncertainty quantification. The quantity in the parenthesis the magnitude increase relative to the quantity when tested on the clean dataset.
CBDL
Ensemble
BNN
Epi. AUROC
Alea. AUROC
Epi. AUROC
Ales. AUROC
Epi. AUROC
Alea, AUROC
MNIST
0.905
0.826
0.805
0.104
0.8G0
0.929
CIBAR
Fashion
MNIST
SVHN
CBDL
Ensemble
BNN
Epi. AUROC
AUROC
Epi. AUROC 0.827
AUROC
AUROC
AUROC
CIFAR
0.930
0,912
0.040
0.835
0,882
MNIST
Pashion
0.958
0.335
0.804
0.030
0.929
0.960
MNIST
SVHN
0.971
0,926
0.808
0.033
0.874
0.898
CBDL
Ensembll
BNN
Epi. AUROC
Alea. AUROC
Epi. AUROC 0.770
Alea. AUROC
Epi. AUROC 0.962
Alea. AUROC
CIFAR
0.724
Fashion MNIST
MNIST
0.926
0.881
0.085
0.933
SVHN
0.972
0.830
0.865
0.066
0.855
0.901
CBDL
Ensemble
BNN
Epi. AUTOO
AUROC
Epi. AUROC 0.890
AUROC
Epi. AUROC
AUROC
CIFAR
0.979
0.941
0.014
0.964
0.952
SVHN
MNIST
0.781
0.679
0.777
0.205
0.728
0.709
Fashion
0.967
0.881
0.917
MNIST
Table 5: Out-OO--istributtio evaluation. AUROC for OOD detection is computed using both aleatoric and epistemic predictive uncertainty measures of the different approaches. Each model trained the training partition of the dataset and tested on the respective testing set of the OOD dataset. When using predictive AU, single BNN performs relatively better than CBDL for OOD detection, though CBDL still comparable and outperforms the single BNN when trained Pashion-MNIST. When using predictive EU, CBDL outperforms the other approaches on most datasets.
under the curve computed and averaged for given noise severity level. Finally, this quantity is averaged across all severity levels. The results are shown in Table As see, when EBNN outperforms CBDL in specific instances, the difference is minor, and when the opposite happens, the difference is significant. Over all datasets, CBDL outperforms a single BNN.
18

---

Published in Transactions on Machine Learning Research (10/2024)
CIFAR-10C Gaussian blur
Gaussian blur Level:1
Gausstan blur Level:2
Gaussian blur Level::
Gaussian blur Level:d
Gaussian blur -Lavel:5
1.0
1.0
1.0
1.0
1.0
0.8-
0.8
Nocuracy
Accurncy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.8-
0.4
0.2
0.2-
0,75
0.0
0,75
0.0
13.5
0.0
00.5
0.5
Rojoction Hate
Rojection Hate
Rejection
Rejection Rate
Rejection Hate
BNN CBDL
Figure 6: Accuracy Rejection Rate- CIFAR1OC Gaussian blur. For lower severities (e.g. 1-2) of Gaussian blur noise, CBDL exhibits beller performance and converges to 100% accuracy at a lower rejection rate. For higher severities (above 3), the single BNN seems show better performance. Even so, the overall rejection rate is high achieve such accuracy. The ensemble method (ENS) exhibits comparable performance in this specific instance.
4.1.2 Out-of-distribution Evaluation
In this final evaluation, we analyze the behavior of the three approaches (CBDL, EBNN, and BNN) when lested on an OOD dataset. For given dataset, for example CIFAR, model is trained the datasel, and evaluated on the remaining datasets, iee. MNIST, SVHN, and Rashion-MNISTI The relative increase in the average predictive EU and AU is computed compared to when approach is tested the In- In-Distribution testing set. The In-iistribution uncertainty quantities for each dataset are shown in Table 4 The relative magnitude of the increase in each quantity is shown in the parenthesis for each respective OOD dataset.
The baseline EBNN exhibits a counterintuitive behavior, viz. a decrease in predictive AU. Although this behavior is consistent across the different dataset combinations, this would be an undesirable quality for OOD detection scenarios.
The single BNN exhibits inconsistent behavior in terms of the relative increase or decrease in magnitude. For CIFAR-10, the predictive AU increases by magnitude larger than the predictive EU, while for SVHN the increase is smaller. Moreover, for Fashion-MNIST the predictive AU decreases, and for MNIST the behavior is inconsistent.
Only our proposed CBDL method demonstrates a stable, consistent increase in predictive AU, which is greater in magnitude relative to the increase in predictive EU, These results demonstrate clear improvement in performance by our proposed approach.
We also give results pertaining AUROCs for the OOD detection performance, presented in Table 5 Fach
group of results shows which dataset the models are trained on, and which they are tested on. We report AUROC when using both the predictive EU and AU of each approach. In general, the baseline EBNN method performs the worst in all cases. When using predictive AU, single BNN performs relatively better than CBDL for OOD detection, though CBDL is still comparable and outperforms when trained on Fashion- MNIST. When using predictive EU, CBDL clearly outperforms the other approaches on most datasets. This is in line with other works where predictive EU is considered important for OOD detection (Kendall & Gal 2017
We conjecture that this may be due to the fact that single BNN is not able to gauge predictive EU properly. Hence, the single BNN flags an instance as OOD when comes from the "tail" of the distribution; this is well-captured by (predictive) alcatoric uncortainty. On the contrary, CBDL able to gauge predictive EU properly, and hence is able to capture when OOD happens by looking at the "disagreement" between the elements of the predictive credal set, captured by the difference between upper and lower entropy.
19

---

Published in Transactions on Machine Learning Research (10/2024)
4.2 Downstream Tasks Performance
As we have shown in the previous section, CBDI. better than single BNNs and ensemble of BNNs at quantifying and disentangling predictive AU and EU. In this section, we show with two applications motion prediction in autonomous driving scenarios, and blood glucose and insulin dynamics for artificial pancreas control that CBDL has better downstream tasks capability than EBNN. We do not compare CBDL against belief tracking techniques because these latter require extra assumptions that CBDL do not, as we further expand on in Appendix K
4.2.1 Motion Prediction for Autonomous Racing
In this case study, we demonstrate the utility of CBDL for motion prediction in autonomous driving scenarios. An important challenge in autonomous driving is understanding the intent of other agents and predicting their future trajectories allow for safety-aware planning. In autonomous racing, where control is pushed to the dynamical limits, accurate and robust predictions are even more essential for oulperforming opponent agents while assuring safety. CBDL provides a straightforward method for quantifying uncertainty and deriving robust prediction regions for anticipating an agent's s behavior.
We use the problem settings in Tunn et al. 20223 to define the problem of obtaining prediction sets for future positions of an autonomous racing agent. Our results show that the prediction regions have improved coverage when compared EBNN. These results hold both In-Distribution and Out-Of-Distribution settings, which are described below.
Problem. Let ('tt,)) = O = {mi....... mf} denote the i-th trajectory instance of an agent at time t, consisting of the observed positions from time | L- up to time I. Let then Ca be a lime-invarianl context variable. Let also ht, F = [ | } be the collection of the next h future positions. We wish obtain model M that prodicts region Ra with probabilistic guarantees. the a-level HDR (Pens) of Pans; that [Fn R. (Pans)] 2 1-a, while for CBDL R. (P a, or equivalently, powood [F' (Paee
particular, for EBNN Ra is
that P pred
IF E (Ra( 2 1-a,
a, for all ppred P Prred
The dataset consists of instances of F*) divided into training set Dtrain and testing set Drest- We train uncertainty-aware model on Dirain that computes the triplet (F7 F F.) = M('', C*) where Fj Fi, Fx are the lower, upper, and mean predictions of the future positions, respectively.
The dataset Daul is created by collecting simulated trajectories of autonomous race cars in the F1Tenth-yym OO'Kelly et al. 2020);; for details, see Tumu et al. (2023). As shown Figure 7 different racing lines were utilized including the center, right, left, and optimal racing line for the Spielberg track.
FeTenth Spielberg Track
Center
Left
Right
Race Line
Prediction
Figure 7: Motion Prediction for FlTenth-Gyrn Gym Environment various racing lines the Spielberg Track.
O'Kelly et al.
2020
Data is collected by simulating
20

---

Published in Transactions on Machine Learning Research (10/2024)
In-Distribution Results
Ensemble
CBDL
1-a a
0.9
0.95
0.99
0.9
0.95
0.99
One-step
0.962
0.980
0.992
0.992
0.995
0.997
Multi-step
0.638
0.826
0.937
0.914
0.948
0.979
Out-of-Distribution Results
Ensemble
CBDL
1 a
0.9 0.95
0.99
0.9
0.95
0.99
One-step
0.919 0.950
0.980
0.979
0.988
0.995
Multi-step 0.532
0.703
0.860
0.825
0.884
0.943
Table 6: FiTenth coverage results. We report one-step coverage and multi-step coverage across different values of a. CBDL exceed coverage of EBNNs in all settings.
We denote these by Deenter: Dright, Diet, and Draces respectively. Position a a vector = (a, b, uT) where a and b are coordinates a 2-dimensional Euclidean space, and & and v are the heading and speed, respectively. In total, the Dall consists of 34686 train instances, 4336 validation instances, and 4336 test instances.
In- In-Distribution vs Out-Of-Distribution. We consider the prediction task to be In-Distribution when Drrain, Drest C Dan It is Out-Of--Distribution (OOD) when Dirain C Deentor UDightt Diet and Drest C Drace-
Metrics. We Irain the ensemble BNNs and the CBDL models, Mens and MCBDL respectively, using the same architecture and different seeds. As for section for CBDL. this corresponds to having a non- singleton prior FGCS, and singleton likelihood FGCS. We compare the performance with respect to the test set by computing the single-step coverage, where each prediction time-step is treated independently, and the multi-step coverage, which considers the entire h-step prediction.
Figure (a) depicts a sample of the In-Distribution evaluation for each of the models. For given trajectory, the red boxes indicate when the prediction region did not cover the actual trajectory at that time-step. Qualitatively, MCBDL has less missed time steps when compared to Mens- Table t shows that CBDL, performs better in terms of both one-step and multi-step coverage. Similar results can be obsorved for the OOD scenario. There, all models were trained racing lines which are predominantly parallel to the track curvature. As consequence, when the test set consists of instances with higher curvatures, the overall coverage of all models degrades. This can be soen in Figure 8 (b), where the prediction of the models (orange) tends to be straight while the actual trajectory is more curved (green). Despite this, the figure and the coverage metrics in Table 6 show how CBDL exhibits a more robust behavior.
4.2.2 Artificial Pancreas Control
Overall Sctup. In this next case study we consider the problem of data-driven control of human blood glucose-insulin dynamies, using an artificial pancreas system, see Figure g
External insulin delivery is accomplished by using insulin pump controlled by the artificial pancreas software, which attempts to regulate the blood-glucose (BG) level of the patient within the euglycemic range of [70, 180] mg/dl (Kushner el al. 2018). Levels below 70 mg/dl lead to hypoglycemia, which can lead to loss of consciousness, coma or even death. On the other hand, levels above 300 mg/dl lead to a condition called ketoacidosis, where the body can break down fat due to lack of insulin, and lead to build up of ketones. In order to treat this situation, patients receive external insulin delivery through insulin pumps. Artificial Pancreas (AP) systems remedy this situation by measuring the blood glucose level,
21

---

Published in Transactions on Machine Learning Research (10/2024)
In-distribution Predictions
Out of Distribution Results
CBCL XY
Enserible XY
CBBL XY
Enserbbee XY
-7.5
23.0
-8.2
22.5
22.0
-8.5
21.5
Input
nout
21.0
-34
19.5
20.0
21.0
19.0
20.0 203
21.5
(a)
(b)
Figure 8: In both pictures, the red boxes indicate when the prediction region did not the actual trajectory that time-step. Left: FiTenth In-Distribution results. Given an input past observations, CBDL exhibits better coverage of the future target trajectory. Predictions which do not cover the target within the desired 1 level are indicated in red. Right: FITenth Out-Of-Distributiin (OOD) results. Robust performance is exhibited by CBDL when compared EBNN OOD settings.
BNN
BNN
Cumbins
MPCC-Cntrol Algoriti
Issalin
BKN
Bloed Gluoess
Patient
Figure 9: The Bayesian Neural Networks predict future blood glucose value. These individual predictions are combined to get a robust estimate of the true value as an interval. This used by the Model Predictive Control (MPC) algorithu recommend insulin dosage for the patient. The patient block in experiment is simulated using the virtual patient models from the UVa-Padova simulator.
and automatically injecting insulin into the blood stream. Thus, we define the unsafe regions of the space as G(I) E (-x;, 70) U (300, oo), where G(t) is the BG value at time This the shaded region Figure 10
Ketonsidoels Levdl
Wloo Churoe (mag
G
Hypoglyeniic Level
Time (ntin)
Figure 10: Starting from an initial glucose value, the task of the artificial pancreas controller is to maintain the blood glucose value within safe operating limits using insulin mode of control.
Neural Network Models and Controller. Deep Neural Networks are effective in capturing the BG- insulin dynamics for personalized medical devices (Kushner et al. 20188). This allows for improved device performance. Even though standard Feedforward Neural Networks be used, Bayesian Neural Networks
22

---

Published in Transactions on Machine Learning Research (10/2024)
(BNNs), and especially collection of multiple BNNs, offer a better alternative towards uncertainty aware predictions. Here, we test the ramifications of these prediction sets, when used inside an online receding horizon control scheme for insulin delivery. We use the standard MPC control scheme for this purpose, well-known in the literature (Dutta et al.] 2018). More formally, let G(t) and I(t) = I, be the blood- glucose and insulin values at time I, respectively. We denote the finite length trajectory of length H as (t(() == [G(t- H + .......GG())) and T (t()) [I(t H H 1)................. [()]]. An uncertainty aware model M computes the triplet (Gi(t+ (), (t (1, G.(+ = (M - (t), I (t)), where Gm is the mean prediction output, and Gi, G.. are the lower and upper predictions of the glucose value, respectively. By design, it is true that G1 S G. S Gu. A model predictive control algorithm whose cost function we denote by J solves arg min.g............... Jo...... Ek- J(M( 1), ' (t
After every time step, the control algorithm picks the first insulin input To as the insulin bolus for the patient. and discards the rest. Cost function J takes into account three factors, (i) Distance of the mean prediction level Gm at cach time step from a target value of 120 mg/ dl, (ii) Distance of upper and lower predictions (Gu, and G1) from the unsafe regions of the state space G(L) > 300 and G(t) < 70, and (iii) Total insulin injected E/ 4.
Starting with some initial glucose value G(O), we measure the performance of the artificial pancreas controller as the fraction of time it spends in the unsafe regions,
1 T Σ (G(()) E (- 7 (13000,,)))
tunsafe
where 1 denotes the indicator function. A lower value is more desirable. We compare EBNN and CBDL as different realizations of the model M.
Distribution Shift using Meals. A well known problem with learned models is distribution shift. Bayesian Neural Networks can address this issue by apprising the end user of the increased uncertainty. For regression models of the lype described above, this appears as larger prediction intervals [Gr, G.]. The artificial pancreas controller can run into this situation in the following way: the insulin-glucose time series data collected for training the data-driven model M can be without meals, while at test time the patient can have meals. This creates a distribution shift between the training and test time data, Fortunately, the UVa-Padova simulator (Dalla Man et al.) 2013) allows to create datasets with and without meal inputs. In this case study, the training data was obtained by randomly initializing the BG value in the range [120, 190), and simulating the patient for 720 minutes. The controller was execuled at 5 minutes intervals. At test time the palient was supplied meals at specific time intervals (for details, see Appendix J]. This creates a significant distribution shift since meals are effectively an unknown variable which can affect the system state. However, from the controller's perspective this is practical, since patients can have unannounced meals.
Results and Discussion. To capture the difference in performance between EBNN and CBDL, we compute Perfait (EENN (CBD) / Both +EBNN and (CBDL, depend on interval [G,, G.ll; for EBNN, this latter corresponds to the a-level HDR R. (Paus) associated with EBNN distribution Peus, while for CBDL it corresponds to the IHDR IR. (P (Ppred). We consider one case in which CBDL. is trained using a credal prior set and only one likelihood (we choose different seeds which initialize the prior distributions but we keep the same architecture for the BNNs), and another case in which we do the opposite (we use the same seed and different architectures).
We report Perfair, across different choices in Table We observe that for lower values of (or equivalently, for larger values of 1-a, e.g. 0.95 and 0.99) the gains of CBDL are more pronounced. This means that when larger significance levels need to be ensured, CBDL is to be preferred to ensemble of BNNs. As discussed before, the CBDL procedure considers all the infinitely many possible priors that can be expressed as a convex combination of the priors that the user specifies at the beginning of the analysis. The same holds for the likelihoods. While this results in a more conservative estimate as compared to the EBNN framework, CBDL produces controllers which respect the safety limits better. To see that CBDL more conservative than EBNN, notice that when combining predictive distributions from multiple BNNs, CBDL combines the predictions via finitely generated credal set (FGCS) whose extrema are the individual distributions. On
23

---

Published in Transactions on Machine Learning Research (10/2024)
the contrary, an EBNN takes average of the individual distributions to compute the ensemble distribution P Pens. The union of the HDRs of the predictive distributions is more conservative (i.e., broader) than the
HDR of the single ensemble distribution. While the approach by EBNN seems like a reasonable choice on the surface, it falls short in capturing the uncertainty necessary for the downstream task. For more details on this case study, see Appendix J
0.9
0.95
0.99
Varying Sceds
-5.8%
5.5%
0.6%
Varying Architectures
-6.6%
0.9%
0.3%
Table 7: We report the performance improvements when using IBNNs as compared to EBNNs across 3 different values of Row corresponds to the case where the individual BNNs are trained with different seeds for the prior distribution; and Row 2 is the case when the BNNs have different architectures.
5 Related Work
In Corani et al. (2012), the authors introduce credal classifiers (CCs) as a generalization of classifiers based on Bayesian networks. Unlike CCs, CBDL does not require independence assumptions between non-descendant, non-parent variables. In addition, CBDL avoids NP-hard complexity issues of scarching for optimal structure in the space of Bayesian networks (Chickering et al.) 2004).. In Manchingal & Cuzzolin (2022), an epistemic convolutional neural network (ECNN) is developed that explicitly models the epistemic uncertainty induced by training data of limited size and quality. A clear distinction is that ECNNs measure uncertainty in target- level representations whereas CBDL identifies the uncertainty mcasure on the output space V. Despite the merit of their work, we believe CBDL achieves greater generality, since it is able to quantify both aleatoric and epistemic predictive uncertainties, and is applicable to problems beyond classification. For a review of the state of the art concerning the distinction between EU and AU we refer to Hüllermeier & Waegeman (2021) and Manchingal & Cuzzolin (2022). We also point out how CBDL has been recently used to solve prior-likelihood conflicts in Bayesian statistics (Marquardt. et al. 2023 Further references be found in Appendix L
We also point out how there exist other efficient methods which perform approximate Variational Inference via dropouts in deep neural networks (Kendall & Gall 2017 Gal & Ghahramani 2016). As mentioned at the end of section 3.1 we can casily adapt CBDL to use such dropout approximations in Steps 3-4 of Algorithm Since our contribution centered around how different predictions can be combined via an FGCS, we used the de-facto standard for performing inference on BNNs, which is based on off- the-shehh VI techniques. In the future, we plan to study the effect on computational complexity and uncertainty quantification capability of CBDL procedure that approximates posterior and predictive distributions via dropout.
We do not consider Bayesian Model Averaging (BMA) as baseline for CBDL for two main reasons. First, BMA applied to deep learning needs to implement full batch Hamiltonian Monte Carlo in order to get to the true posterior (Izmailov et al. 2021b Given the number of parameters in modern deep learning architectures in the order of millions- this realistically possible an experimental level only to labs with access industry scale computational resources. In order to be practically relevant, we limit our experiments to the more well-understood realm of Variational Inference pitfalls of BMA in the context of Bayesian Neural Networks, further reason not to Region resulting from BMA as a bascline for CBDL. Such pitfalls are related to the fact that BMA can be seen as a model featuring second-order distributions, i.e. distributions over distributions. In particular, the distribution Q in equation is a second order distribution. These types of models have been recently shown to suffer from major pitfalls when used to quantify predictive EU due to their sonsitivity to regularization parameters, and underestimate predictive AU (Bengs al.) 2022 Pandey & Yu. 2023 Juergens et al. 2024).
BNNs. In addition, Izmailov et al. (2021a) show the Highest Density
24

---

Published in Transactions on Machine Learning Research (10/2024)
6 Conclusion
We presented CBDL, procedure that can be seen as a non-condensed, uncountably infinite ensemble of BNNs, carried out using only finitely many elements. It allows to distinguish between predictive AU and EU, and to quantify them. We showed how it can be used to specify a set of outputs - the THDR - that enjoys probabilistic guarantees. We showed empirically that improves on the Bayesian state of the art at ganging predictive AU and EU, and we also demonstrated its downstream tasks capabilitics.
We point out how a region that improves on TR.. (P (Ppred), meaning that it would be tighter, is
IR, (P = E E y: p 2
where gured := ming,s Phied and _ is the largest constant such that p ly IR, (Ppra)) 2 a. The problem with IR, is that, while the highest density regions (a associated with the predictive distributions in exPpred pred can be computed using off-the-shelf tools, calculating gured and lº would have been much more computationally expensive. In addition, it would have required to come up with a new technique to find g and p". We defer studying this to future work.
We also plan to apply CBDL to continual learning to overcome the curse of dimensionality and to capture agent's 's preference over the lasks to perform, similarly to Lu al. (2023), and to active learning, to be able to sample from the regions of the state space exhibiting the highest epistemic uncertainty, similarly to Dutta et al. (2023).
Furthermore, we intend to relate CBDL to Bayesian Model Selection (BMS) (Ghosh et al. 2019 This latter suffers from the same problem as "regular" Bayesian inference. That is, while it tries to come up with a sophisticate prior that induces shrinkage, it still relies on the "correctness" of that prior, L.e. on correctly specifying the prior's parameters. In the future, an interesting way of combining CBDL with BMS will be to use finite number of regularized horseshoe priors, as suggested by Ghosh al. 2019 Section 3.2), extreme elements of the prior credal set.
We also call attention to the fact that CBDL is a model-based approach. The relationship with model-free approaches such as conformal prediction (Shafer & Vovk 2008) will be the object of future studies. particular, we are interested in finding in which cases IHDRs are narrower than conformal regions, and vice versa, and which credal sets give rise IHDRs enjoying the same probabilistic guarantees as conformal regions.
Finally, point out how one possible way of casing the burden of the combinatorial task in Step 3 of Algorithm
I is to specify prior credal set whose size strikes the perfect balance between being "vague enough" so that we do not underestinate the EU, and being "small enough" so that CBDL is actually implementable. We suspect conjugacy of the priors may play a key role in this endeavor. Because of its centrality, we defer the study of "optimal prior credal sets" " to future work. We also point out how an evidential approach (Amini et al., 2020; Charpentier et al., 2020: Denceux 2022 2023 Sensoy et al.] 2018)) at least in classification problems could allows imprecise probabilistic approach, with some tricks resulting from smartly choosing our priors.
bypass the bottleneck in Algorithm [ by merging an
A CBDL-adjacent rescarch question of great interest pertaining ensemble learning, then, is how do single components of an ensemble contribute to the quantification of the "global predictive EU" faced by the agent. When the uncertainty captured by an ensemble is distilled into only one distribution, there could indeed be "uncertainty spills", like when one pours water in a glass too hastily, and some finishes on the table instead of in the glass. This remains an unexplored venure in the uncertainty quantification community.
25

---

Published in Transactions on Machine Learning Research (10/2024)
References
Joaquín Abellán and Serafin Moral. A non-specificity measure for convex sets of probability distributions. International Journal of Uncertainty, Fuzziness and Knowledge-Basse Systems, 8(3):357-367, 2000.
Joaquín Abellán, George Jifí Klir, and Scrafin Moral. Disaggregated total uncertainty mcasure for credal
sets. International Journal of Generul Systems, 1(35):29 44, 2006.
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14927-14937. Curran Associates, Inc., 2020.
Thomas Augustin, Frank P.A. Coolen, Gert De Cooman, and Matthias C.M. Troffaes (eds.). Introduction to imprecise probabilities. Wiley Series in Probability and Statistics. John Wiley and Sons, 2014.
David Avis and Komei Fukuda. Reverse search for enumeration. Discrete Applied Mathematics, 65:21 46, 1996.
Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas, and Ryan J. Tibshirani. Conformal prediction beyond exchangeabilttyy The Annals of Statislies, (2):8811 845, 2023. doi: 10.1214/23-AOSS AOS2276. URL
https://doi.ors htt:///oi.orr////
orw/oo..r....... 10. 211///23--022266
Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. Pitfalls of epistemic uncertainty quantification through loss minimisation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL https://openreviie: wtw.:eerrrreeeeeeeffuruuuuunniea net/forum?id=
[RZW5|
James O. Berger. The robust Bayesian viewpoint. In Joseph B. Kadane (ed.), Robustness of Boyesian Analyses. Amsterdam: : North-Holland, 1984.
Jose M. Bernardo. Reference posterior dist ribbutios for Bayesian inference. Journal of the Royal Statistical Society; Series B, 41(2):113 128, 1979.
George E. P. Box. Science and statistics. Journal of the American Statistical Association, 71(356):791-799 799, 1976.
Andrey G. Bronevich and Igor N. Rozenberg. Generalized Hartley measures on credal scts. In Andrés Cano, Jasper De Bock, Enrique Miranda, and Serafin Moral (eds.), Proceedings of the Twelveth International Symposium on Imprecise Probability: Theories and Applications, volume 147 of Proceedings of Machine Learning Research, pp. 32-41. PMLR, 2021.
Yuhang Cai and Lek-Heng Lim. Distances between probability distributions of different dimensions. IEEE Transactions on Information Theory, 68(6):4020 4031, 2022.
Michele Caprio. Refined Pinsker's and reverse Pinsker's inequalities for probability distributions of different dimensions. IEEE Access, 10:116425-116411, 2022.
Michele Caprio. Improcisc markov semigroups and their orgodicity, 2024. URL https://arxiv.orrg /bs//
2405.00081]
Michele Caprio and Ruobin Gong. Dynamic precise and imprecise probability kinematies. In Enrique Miranda, Ignacio Montes, Erik Quaeghebeur, and Barbara Vantaggi (eds.), Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, volume 215 of Proceedings of Machine Learning Research, pp. 72 83. PMLR, 11 14 Jul 2023.
Michele Caprio and Sayan Mukherjee. Finite admixture models: bridge with stochastic geometry and choquet theory. Available at arXiv:2002.08 084009 2023a.
Michele Caprio and Sayan Mukherjee. Concentration incqualities and optimal number of layers for stochastic deep neural networks. IEF. Access, 11:38458 -84470, 2023b. doi: 10.1109/ ACCOSS/ 2023.3268034.
26

---

Published in Transactions on Machine Learning Research (10/2024)
Michele Caprio and Sayan Mukherjee. Ergodic theorems for dynamic imprecise probability kinematics. International Journal of Approzimate Reasoning, 152:325-343) -43,, 2023c.
Michele Caprio and Sayan Mukherjee. Exlended probubililies and their application slatistienl inference,
chapter 2. World Scientific, 2023d.
Michele Caprio and Teddy Seidenfeld. Constriction for sets of probabilities. In Enrique Miranda, Ignacio Montes, Erik Quaeghebeur, and Barbara Vantaggi (eds.), Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, volume 215 of Proceedings of Machine Learning Research, pp. 84-95. PMLR, 11-14 Jul 2023. URT httpps://proceeddnngg mlr http://tnnnn.m.....eeseeeeee press//215/ caprio23b. html
Michele Caprio, Kuk Jin Jang, Souradeep Dutta, Shireen Manchingal, Fabio Cuzzolin, Oleg Sokolsky, and insup Lee. Credal and interval deep evidential classification. Technicol Reports of the PRECISE Center, 2024a.
Michele Caprio, Yusuf Sale, Eyke Hüllerreier, and Insup Lee. A Novel Bayes' Theorem for Upper Probabil- ities. In Fabio Cuzzolin and Maryam Sultana (eds.), Epistemic Uncertainty in Artificial Intelligence, pp. 1 12, Cham, 2024b. Springer Nature Switzerland.
Michele Caprio, Maryam Sultana, Eleni Elia, and Fabio Cuzzolin. Credal learning theory, 2024c. URL https://arxize
Simone Cerreia- Viireia,Vioglo, Fabio Maccheroni, and Massimo Marinacci. Ergodic theorems for lower probabilities. Proceedings of the American Mathematical Society, 144:3381-3399 -396, 2015.
Bertrand Charpentier, Daniel Zügner, and Stephan Günnnemann. Posterior network: Uncertainty estimation without OOD samples via density-based pseudo-counts. In H. Larochelle, M. Ranzato, R. Hadselll M.F. Balcan, and H. Lin (ods.), Advances in Neural Information Processing Systems, volume 33, pp. 1356-1367. Curran Associates, Inc., 2020.
David M. Chickering, David Heckerman, and Christopher Meek. Large-sample learning of Bayesian networks is Nardard. The Journal of Machine Learning Research, 5:1287-1330, 2004.
Adam D. Cobb, Michael D. Himes, Frank Soboczenski, Simone Zorzan, Molly D. O'Beirne, Atilım Günes Baydin, Yarin Gal, Shawn D. Homagal-Goldmaan Giada N. Arney Daniel Angerhausen, and 2018 NASA FDL Astrobiology Team II. An ensemble of Bayesian neural networks for exoplanctary atmospheric retrieval. The Astronomical Journal, 158(1):33 2019.
Frank P. A. Coolen. Imprecise highest densily regions related to intervals of measures. Memorandum COSOR, 9254, 1992.
Giorgio Corani, Alessandro Antonucci, and Marco Zaffalon. Bayesian Networks with Imprecise Probabil- ities: Theory and Application to Classification, chapter 4 of Data Mining: Foundations and Intelligent Paradigms: Volume 1: Clustering, Association and Classification, pp. 49-93. Berlin, Germany: Springer, 2012.
Fabio G. Cozman. Credal networks. Artificial Intelligence, 120(2):199 233, 2000a. ISSN 0004-3702. doi: httt:://0...oy///100000/00000000 atticlle/p1//00004700000023
URL httpasss/wwwww..ceeeeeeeee.....t
Fabio Gagliardi Cozman. Credal networks. Artificial Intelligence, 120:199 233, 2000b.
Chiara Dalla Man, Francesco Micheletto, Dayu Lv, Mare D. Breton, Boris Kovatchev, and Claudio Cobelli. The UVA AWAAPAAOVA type 1 diabetes simulator: New features. Journal of Diabetes Science and Technology, 8(1):26-34 34, 2013.
George B. Dantzig. Linear Programming and Extensions. Princeton University Press, 1963. URL http: //// ort.//table/jj.cttcc33gg
27

---

Published in Transactions on Machine Learning Research (10/2024)
Bruno de Finetti. Theory of Probability, volume 1. New York : Wiley, 1974.
Bruno de Finetti. Theory of Probability, volume 2. New York : Wiley, 1975.
Thierry Denceux. An evidential neural network model for regression based on random fuzzy numbers. In Sylvie Le Hégarat-Mascle, Isabelle Bloch, and Emanuel Aldea (eds.), Belief Functions: Theory and Applications, pp. 57-66, Cham, 2022. Springer International Publishing.
Thierry Denoeux. Quantifying prediction uncertainty in regression using random fuzzy sets: the ENNreg model. IEEE Transactions on Fuzzy Systems, pp. 1-10, 2023.
Stefan Depeweg, Josc-Miguel Bernandez-Lobate Finale Doshi- Velez, and Steffen Udluft. Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning. In International Conference on Machine Learning, pp. 1184 1193. PMLR, 2018.
Doshi-Velez,
Didier Dubois and Eyke Hüllermeier. Comparing probability measures using possibility theory: A notion of
relative peakedness. International Journal of Approzimate Reasoming, 45(2):364-385, 2007. Eighth Euro- pean Conference on Symbolic and Quantitative Approaches to Reasoning with Uncertainty (ECSQARU 2005).
Souradeep Dutta, Taisa Kushner, and Sriram Sankaranarayanan. Robust data-driven control of artificial pan- creas systems using neural networks. In Milan Češka and David Šafránck (ods.), Computational Methods in Systems Biology, pp. 183 202, Cham, 2018. Springer International Publishing. ISBN 978-3-319-99429-11
Souradeep Dutta, Michele Caprio, Vivian Lin, Matthew Cleaveland, Kuk Jin Jang, Ivan Ruchkin, Oleg Sokolsky, and Insup Lee. Distributionally robust statistical verification with imprecise neural networks. arXiv preprint Xrx::2330.. 14815, 2023.
Daniel Ellsberg. Risk, ambiguity, and the Savage axioms. The Quarterly Journal of Economics, 75(4): 643-669, 1961.
Mohammed Fellaji and Frédéric Pennerath. The epistemic uncertainty hole: an issuc of bayesian neural networks. Available at (07.0:22477.01985 2024.
Emlyn Flint, Florence Chikurunhe, and Anthony Seymour. Regime- tactical allocation for equity factors and balanced portfolios. SSRN Electronic Journal, 01 2017.
Benoît Fortin, Samir Hachour, and François Delmotte. Multi-target PHD tracking and classification using imprecise likelihoods. Intervational Journal of Approrimate Reasoning, 90:17 36, 2017.
Vincent Fortuin, Adrià Carriga-Alonso, Mark van der Wilk, and Laurence Aitchison. BNNpriors: A library for bayesian neural network inference with different prior distributions. Software Impacts, 9:100079, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainly in deep learning, 2016.
Sounya Ghosh, Jiayu Yao, and Finale Doshi- Velez. Model selection in Bayesian neural networks via horseshoe priors. Journal of Machine Learning Research, 20(182):1-46, 2019. URL http://jmlr. httpimmmr.rgggpp//eee org/papers/v22 v 19- 236. hlml.
Isaac Gibbs, John J. Cherian, and Emmanuel J. Candès. Conformal prediction with conditional guarantees. Available arXic:2305.1161 12616 2023.
Ttzhak Gilboa and Massimo Marinacci. Ambiguity and the Bayesian paradigm. In Daron Acemoglu, Mannel Arellano, and Eddie Dekel (eds.), Advences in Economics and Econometrics, Tenth World Congress, volume 1. Cambridge: Cambridge University Press, 2013.
Ethan Goan and Clinton Fookes. Bayesian Newral Networks: An Introduction and Survey, pp. 45-87. Cham, Switzerland : Springer International Publishing, 2020.
28

---

Published in Transactions on Machine Learning Research (10/2024)
Ruobin Gong and Xiao-Li Meng. Judicious judgment meets unsettling updating: dilation, sure loss, and Simpson's paradox. Statistical Science, 36(2):169 160090, 2021.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks, In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Dan Hendrycks and Thomas G. Dietterich. Benchmarking neural network robustness to commin corruplions and perturbations. CoRR, 00ss/1903.1220 2019.
Paul Hofman, Yusuf Sale, and Eyke Hüllermeier. Quantifying alcatoric and epistemic uncertainty: A credal approach. In ICML 2024 Workshop on Structured Probabilistic Inference (18) Generalive Modeling, 2024.
Peter J. Huber and Elvezio M. Ronchetti. Robust statistics. Wiley Series in Probability and Statistics. Hoboken, New Jersey : Wiley, 2nd edition, 2009.
Rob J. Hyndman. Computing and graphing highest density regions. 120-126, 1996.
The American Statistician, 50(2):
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 3(110):45--06, 2021.
Pavel Izmailov, Patrick Nicholson, Sanae Lotfi, and Andrew G Wilson. Dangers of bayesian model averaging under covariate shift. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vanghan (ods.), Advances in Neural Information Processing Systems, volume 34, pp. 3309 3322. Curran Associates, Inc., 2021a.
Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, and Andrew Gordon Wilson. What are bayesian neural network posteriors really like? CoRR, 222/22104.442,,, 2021b. URL http:///axxvvv.rg//e//
2104. 14421
Pavel Izmailov, Sharad Vikram, Matthew D Hoffman, and Andrew Gordon Gordon Wilson. What are Bayesian neural network posteriors really like? In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139, pp. 4629-4644 PMLR, 2021c.
Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society A, 186(1007):453-465) -41--61, 1946.
Laurent Valentin Jospin, Hamid Laga, Farid Boussaid, Wray Buntine, and Mohammed Bennamoun. Hands- on Bayesian neural networks A tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2):29 48, 2022.
Ngumbang Juat, Mike Meredith, and John Kruschke. Package 'hdinterval', 2022. URL attps://cran http:////pppaaaaa//iiaaaaa///aaa
r-
pdf Accessed on May 9, 2023.
Mira Juergens, Nis Meinert, Viktor Bengs, Eyke Hüllermeier, and Willem Waegeman. Is epistemic uncer- tainty faithfully represented by evidential deep learning methods? In Forty-first International Conference on Machine Learning, 2024. URL httpps://openreview net wttpa//wwww.eeeeeto/////emmaa/ma /oor
Sanyam Kapoor, Wesley Maddox, Pavel lzmailov, and Andrew Gordon Wilson. On uncertainty, tempering, and data augmentation in bayesian classification. arXiv preprint 0rXiv:2208.16481, 2022.
Ramneet Kaur, Xiayan Ji, Souradeep Dutta, Michele Caprio, Yahan Yang, Elena Bernardis, Oleg Sokol- sky, and Insup Lee. Using semantic information for defining and detecting OOD inputs. Available at arXiv:2302.1100 11019 2023.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
29

---

Published in Transactions on Machine Learning Research (10/2024)
lang-Hyun Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. In Hal Daumé ITT and Aarti Singh (eds.), Proceedings of the 97th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5275 5285. PMLR, 13-18 Jul 2020.
John Klein, Christèle Lecomte, and Pierre Miché. Hierarchical and conditional combination of belief functions induced by visual tracking. International Journal of Approzimate Reasoning, (144):110-42) 2010.
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncertainty optimization. In Advances in Neural Information Processing Systems, volume 33, pp. 18237-18248. Cur- ran Associates, Inc., 2020. URL https:// B468445502444277555eedllaa222222 Abstratttttbtttttttt..c.
Ranganath Krishnan, Mahesh Subedar, and Omesh Tickoo. MOPED: efficient priors for scalable variational inference in bayesian deep neural networks. CoRR, abss/106.00 1906.05323, 2019.
Ranganath Krishnan, Mahesh Subedar, and Omesh Tickoo. Specifying weight priors in Bayesian deep neural networks with empirical Bayes. Proceedings of the AAAI Conference Artificial Intelligence, 34 (04):447744484 April 2020. doi: 10.1609/ /ta//aai.ava v34504.58877
URL https://ojs. www.ssaaaaioogg......... hrg/index. php/
MAAT/articleevvee//555
Alex Krizhevaky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (Canadian Institute for Advanced Research). Available http://www.. es. www.ssststttooooooooooooo edu/
httml, 2009.
Meelis Kull and Peter A. Flach. Reliability Maps: A Tool to Enhance Probability Estimates and Improve Classification Accuracy. In Toon Calders, Floriana Esposito, Eyke Hüllermeier, and Rosa Meo (eds.), Machine Learning and Knowledge Discovery in Databases, Lecture Notes in Computer Science, pp. 18-33, Berlin, Heidelberg, 2014. Springer. ISBN 978-3-662-14851-9.
Taisa Kushner, David Bortz, David M. Maahs, and Sriram Sankaranarayanann A data-driven approach to ar- tificial pancreas verification and synthesis. Proceedings of the 9th ACM/TEEE International Conference on Cyber-Physical Systems, ICCPS '18, pp. 242 252. IEEE Press, 2018. ISBN 9781538653012.
Jouko Lampinen and Aki Vehtari. Bayesian approach for neural networks review and case studies. Newral Networks, 4:257-274, 2001.
Daniel Lassiter. Representing credal imprecision: from sets of measures to hierarchical Bayesian models. Philosophical Studies, 177(6):1463 -485,, 2020.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Peter Lenk and Bryan Orme. The value of informative priors in Bayesian inference with sparse data. Journal of Marketing Research, 46(6):832-88) 845, 2009. ISSN 00222437.
Isaac Levi. The Enterprise of Knowledge. London, UK MIT Press, 1980.
Vivian Lin, Kuk Jin Jang, Souradeep Dutta, Michele Caprio, Oleg Sokolsky, and Tnsup Tee. DC4T: Distribu- tion shift recovery via data-driven control for deep learning models. In Alessandro Abate, Mark Cannon, Kostas Margellos, and Antonis Papachristoodolln (eds.), Proceedings of the 6th Annual Learning for Dy- namics and Control Conference, volume 242 of Proceedings of Machine Leorning Research, pp. 1526-1538 PMLR, 15-17 Jul 2024. URL htttps:/
mlr. www.mmrr.rrsssee press/ v242 11n244b..b.bb... hhmlll.mmmm1111mm11
Pengyuan Lu, Michele Caprio, Eric Eaton, and Insup Lec. Zero-shot task preference addressing enabled by imprecise Bayesian continual learning. Available or (orXiv2305.14782 2023.
Shireen Kudukkil Manchingal and Fabio Cuzzolin. Epistemic deep learning. Available at arziv:2206.07609 2022.
30

---

Published in Transactions on Machine Learning Research (10/2024)
Massimo Marinacci and Luigi Montrucchio. Introduction to the mathematics of ambiguity. In Itzhak Gilboa (ed.); Uncerlainty in economic theory: collection of essays in honor of David Schmeidler's 's 65th birthday. London : Routledge, 2004.
Alexander Marquardt, Julian Rodemann, and Thomas Augustin. An empirical study of prior-data conflicts in Bayesian neural networks. |osterr presented at 19th International Symposium on Imprecise Probabilities: Theories and Applications (ISIPTA) 2023, 2023.
Jacques Paul Migne. De coelesti hierarchia. In Patrologia Graeca, volume 3. Paris: Imprimerie Catholique, 1857.
Enrique Miranda and Ignacio Montes. Centroids of the core of exact capacities: a comparative study, Annals of Operations Research, 321:409-449, 2023.
Norman Mu and Justin Gilmer. MNIST-C: A robustness benchmark for computer vision. CoRR, abs/1906.0233/, 02000. 2019.
Bálint Muesányi, Michael Kirchhof, and Seong Joon Oh. Benchmarking uncertainty disentanglement: Spe-
cialized uncertainties for specialized tasks. Available at drXiv:2402.19460 2024.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupercised Feature Learning 2011, 2011.
Matthew O'Kelly, Hongrui Zheng, Dhruv Karthik, and Rahul Mangharam. FITENTH: An Open-source Evaluation Eavironment for Contimuous Control and Reinforcement Learning. In Proceedings of the NewrIPS 2019 Competition and Demonstration Track, pp. 77 89. PMLR, 2020.
Deep Shankar Pandey and Qi Yu. Learn to accumulate evidence from all training samples: Theory and practice. In Andreas Krause, Emma Brunskill, Kyunghyu Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (ods.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 26963 26989. PMLR, 23 29 Jul 2023. URL attps://proceeding mlr. www.preess/evv.. 2toss//2222/pandy22844 html
Sangdon Park, Osbert Bastani, Nikolai Matni, and Insup Lee. PAC confidence sets for deep neural networks via calibrated prediction. Tn 8th International Conference Learning Representations 2020.
Luis Raul Pericchi. Sets of prior probabilites and bayesian robustness. Documentation Section on the website of the Society for Imprecise Probability Theory and Applications (SIPTA), 1998.
Matthew Reimherr, Xiao-Li Meng, and Dan L. Nicolae. Prior sample size extensions for assessing prior impact and prior-likelihood discordance. Journal of the Royol Statistical Society: Series B (Statistical Methodol- ogy), 831 437, 2021, doi: https:/ /toss///oooorr.//// ottp//..g///////////////////.. 10.1111/rssb..111 12414. URL https://rss http::::/rrss.oonnnellllll onlinelibrary.
wiley. hac.:y.commm/o//// 1111:....s.bb.....e... 12414
Hippolyt Ritter, Alcksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In International Conference on Learning Representations, 2018.
Yusuf Sale, Michele Caprio, and Eyke Hüllermeier. the volume of a credal set good measure for epistemic uncertainty? In Robin J. Evans and llya Shpitser (eds.), Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 1795- 1804. PMLR, 31 Jul 04 Aug 2023. URL
httpeeeedinns.m....rrssssssseeee mlr
html
Yusuf Sale, Viktor Bengs, Michele Caprio, and Eyke Hüllermeier. Second-order uncertainty quantification: A distance-based approach. In Ruslan Salakhutdinov,, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 43060-43076. PMLR, 21-27 Jul 2024. URL https:: dtttps:/pprceedding: mlr. preas/v235/sall 202233//aaae222. html
31

---

Published in Transactions on Machine Learning Research (10/2024)
Robin Senge, Stefan Bösner, Krzysztof Dembezyússki Jörg Haasenritter, Oliver Hirsch, Norbert Donner- Banzhoff, and Eyke Hüllermeier. Reliable classification: Tearning classifiers that distinguish aleatoric and epistemic uncertainty. Information Sciences, 255:16 29, January 2014. ISSN 0020-0255.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, Н. Larochelle, K. Grauman, N. Desa-Bianchi, and R. Garnett (ods.), Aduances Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Glenn Shafer and Vladimir Vovk. A tutorial conformal prediction. Journal of Machine Learning Research, 9:371-421, 2008
Marek Smieja and Jacek Tabor. Entropy of the mixture of sources and entropy dimension. IEEE Transactions on Information Theory, (8(5):2719-2722 2012.
Lewis Smith and Yarin Gal. Understanding measures of uncertainty for adversarial example detection. arXiv preprint 1XXt:1803.08533 2018.
Bjørnar Tessem. Interval probability propagation. International Journal of Approzimate Reasoning, 7(3): 95-120, 1992.
D. Michael Titterington. Bayesian methods for neural networks and related models. Statistical Science, 19 (1):128 139, 2004.
Krasymyr Tretiak, Georg Schollmeyer, and Scott Ferson Neural network model for imprecise regression with interval dependent variables. Available at arXiv:2206.02467 02467, 2022.
Matthias C.M. Troffaes and Gert de Cooman. Lower Previsions. Chichester, United Kingdom : John Wiley and Sons, 2014.
Remukanandan Tumn, Lars Lindemann, Truong Nghiem, and Rahul Mangharam. Physics Constrained Motion Prediction with Uncertainty Quantification, 2023. arXiv:230.001060.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Cham Springer, second edition, 2022.
Peter Walley, Coherent lower (and upper) probabilities. Technical report, University of Warwick, Coventry, 1981.
Peter Walley, Statistical Reasoning with Imprecise Probabilities, volume 42 of Monographs on Statistics and Applied Probability, London Chapman and Hall, 1991.
Hao Wang and Dit Yeung. A survey on Bayesian deep learning. ACM Computing Surveys, 53(5):1-37 2021.
Larry A. Wasserman and Joseph B. Kadane. Bayes' theorem for Choquet capacities. The Annals of Statistics, 18(3):1328-133 -1339, 1990.
Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Robin J. Evans and llya Shpitser (eds.), Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 2282- 2292. PMLR, 31 Jul-04 Aug 2023. URL
.ml.. http::awww.....ee..eeeeeeemeeeee ptwps//216/ http2///wwwmmmm.....mmm. html
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image datasct for benchmarking machine learning algorithms. CoRR, Abs//1708.07747, 2017.
Marco Zaffalon. The naive credal classifier. Journal of Statistical Planning and Inference, 105(1): 5 21, 2002. ISSN 0378-3758. doi: hhtps://ttii.org/10000166, 10011110000000007777770100000000
URL https:
//www sciencedirect. and their Applications.
Imprecise Probability Models
Fengshuo Zhang and Chao Gao. Convergence rates of variational posterior distributions. The Annals of Slatistics, 48(4):2180 2207, 2020.
32

---

Published in Transactions on Machine Learning Research (10/2024)
A
Why do we need IPs?
The main motivations for working with crodal sets are two. Let F)) be the measurable space of interest.
(i) A single probability distribution does not suffice to represent ignorance in the sense of lack of knowledge; this is well documented in the literature, see e.g. Hüllermeier & Waegeman (2021) | and references therein. Consider the example of complete ignorance (CI) in the case of a finite state space Q (Hüllermeier & Wacgeman 2021 Section 3.3). In standard Bayesian analysis, CI is modeled in terms of the uniform distribution Uni(( this is justified by Laplace's "principle of indifference". Then, however, it is not possible to distinguish between precise probabilistic knowledge about a random event called prior indifference; think of the tossing of fair coin and complete lack of knowledge due to an incomplete description of the experiment called prior ignorance. Another problem is given by the additive nature of probability distributions. Consider again the example of a uniform distribution. First, let us observe that it is not invariant under reparametrization. In addition, if we model the ignorance about the length r of the side of a cube in R3 via a uniform measure on the interval [1, u] C R, then this does not yield a uniform distribution of x on [t w²), which suggests some degree of informedness about the cubc's volume. Finally, pointed out in Walley (1991), if we ask a subject even an expert about their opinion regarding some events, it is much more likely that they will report interval of probabilities rather than single values.
(ii) Working with credal sets allows to achieve prior and likelihood robustness: realistically large sets Pprior of priors and Pikk of likelihoods are elicited. Using credal sets, the agent recognizes that prior beliefs and knowledge about the sampling model are limited and imprecise. Combining each pair of functions in Pprior and Puk using Bayes' rule, a class of posterior distributions reflecting the updated state of uncertainty is formed. If the available information is not sufficient to idcntify unique posterior distribution, a set of posteriors whose diameter is small, credal sets allow to represent indecision, thus leading to a less informative but more robust conclusions 20
B On the use of credal sets
Let us address a critique raised against the use of credal sets. Lassiter (2200) argues against the use of sets of probabilities to model an agent's prior beliefs and their know ledgledge of the sampling model, while dlebating in favor of using hicrarchical Bayesian models. As reported in Hüllermeier Waegeman (2021) Secton 4.6.2), the argument against credal sets that is more cogent for the machine learning literature is that modeling a lack of knowledge in a set-based manner may hamper the possibility of inductive inference, up to point where learning from empirical data is not possible any more. With this, we mean the following. As PPericci (1998) points out, the natural candidate for a class of priors to represent complete ignorance is the class Pan of all distributions. When this class leads to non-vacuous and useful conclusions, these are quile compelling and uncontroversial. It turns out that the posterior probabilities obtained from this class are vacuous, that is, their lower and upper bounds are 0 and 1: no finite sample is enough to annihilate a sufficiently extreme prior belief. There is then a compromise to be made, and this is the compromise of near-ignorunce. The near- ignorance class should be vacuous priori in some respects, typically the ones that are the most important for the analysis at hand. This way of proceeding is labeled as arbitrary by Lassiter (2020), who instead advocates for the use of hicrarchical Bayesian procedures. We find this critique not compelling, as during the analysis the job of the agent. is to model reality: as pointed out in Hüllermeier & Waegeman (2021) Secton 5), statistical inference is not possible without underlying assumptions, and conclusions drawn from data are always conditional on those assumptions. If we were to work every time with the maximum level of generality, we would hardly be able to reach any conclusions. For example, in a statistical analysis we never consider the state Ω of opparently possible states (Walley 1991 section 2.1.2), that is, the one that contains all the states w that are logically consistent with the available information. If we consider coin toss, we let the state space be Ω = [heads, tails), certainly not Ω = [heads, tails, coin landing on its edge, coin braking into pieces on landing, coin disappearing down a crack in the floor). The same holds for sets
Here "diameter" has he understood the distance between upper and lower probability of event A, for all A E F.
33

---

For this reason we prefer to Published in Transactions on Machine Learning Research (10/2024)
of probabilities: it makes much more sense to work with near-ignorance credal sets than to work with Pan- A final reason to rebut the point in Lassiter (2020) is that the problems indicated in (i) in section A that make the use of the uniform prior distribution often interpreted as representing epistemic uncertainty in
standard Bayesian inference at least dobatable are inherited by hicrarchical Bayesian modeling, as specified
in Bernardo 1979); Hüllermeier & Waegeman (2021); Jellreys (1946). Furthermore, a single distribution that is the result of hicrarchical Bayesian procedure is unable to gauge epistemic uncertainty (Hüllermeier
& Waegeman )
B.1 Further notes on credal sets
As pointed out in Corani ct al. 2012 Section 3.3), there is way of obtaining credal sets starting from sets of probability intervals; in addition, standard algorithms can compule the extreme elements of a credal set for which a probability interval has been provided (Avis & Fukuda, 1996). However, the resulting number
of extrema is exponential in the size of the possibility space (Tessem 1992 21
specify prior and likelihood finitely generated credal sets instead.
The way credal sets behave after conditioning new available evidence has been recently studied in Caprio & Seidenfeld (2023). and way of using credal sets in the open-world scenario (that is, when the support of the elements of the credal set can become larger as more data become available) is explored in Caprio & Mukherjee (2023d) Furthermore, the use credal sets in statistical learning theory has been studied in Caprio et al. 2024c), and in computer vision in Caprio 2024
C A further IP concept: the core
Let again (2, be the measurable space of interest. Because of the conjugacy property of upper and lower probabilities let us focus on upper probabilities only. We say that upper probability P is concave if P(AUB) & P(B))))()(P measures (2, F). Upper probability P is compatible with the convex set (Gong & Meng 2021
for all A, B F. Recall that A(2,F) denotes the set of all probability
core(P) [P A(2, P(A) S P(A), VAEF)
[P E A(,, P(A) S P(A) P(A), VAEF)
=
where the second equality is charactorization Correia- Correia-Vioglio et al. 2015 Page 3389). Notice that the core is convex (Marinacci & Montrucchio 20004, Section 2.2). We assume it is nonempty. Then, it is weak*-compact
result of Marinacci & Montrucchio (2004 Proposition 3) 22
Since the core is convex, the set ex(core P)] of extreme points of the core is well defined. It contains all the elements of the core that cannot be written as a convex combination of one another. The following important result is a consequence of Walley [1991] Theorem 3.6.2).
Theorem 9. Suppose core(P) is nonempby. Then, the following holds.
(a) exlcore(P)] 0 #0.
(b) (P) is the closure in the weak' topology of the convex hull of exlcoree (P)].
(c) P(A) sup (P) P(A), for all A E F, then P(A) sup Pee P(A), for all A F.
So in order to define an upper probability P that setwise dominates the elements of core P) it is enough to specify the extreme points of the core.
21 Recall that the possibility space of random variable the space of the values can take on. 22 Recall that in the weak? topology, net (Pa)aer converges and only if Pa(A) P(A), for all A E.F.
34

---

Published in Transactions on Machine Learning Research (10/2024)
D A new Bayes' theorem for IPs
We present Theorem 10 result that although appealing - does not lend itself well to be applied to the CBDL procedure. An extension of Theorem 10 is given in Caprio et al. (2024b).
Call è the parameter space of interest and assume is Polish, that is, the topology for θ is complete, separable, and metrizable. This ensures that the set A(O, B) of probability measures on θ is Polish as well, where denotes the Borel o-algebra for O. Let 2" be the set of all bounded, non-negative, B-measurable functionals on O. Call 9 X y the sample space endowed with the product a-algebra A = Ax x Ay: where Ax is the o-algebra endowed to X' and Ay is the o-algebra endowed to y. Let the agent elicit Lo := (Po E A) A 6}. Assume that each Po E Lg has density L(0) = p(D 6) with respect to some o-finite dominating measure v on (9, this represents the likelihood function for 0 having observed data D C 9. We assume for now that L E 2,, for all D C 9.
Let the agent specify set P of probabilities on (). Then, compute P, and consider pmm := core (P: it represents the agent's initial beliefs 23 We assume that every E pro has density p with respect some dp o-finite dominating measure μ on (O, that is, p du We require the agent's beliefs to be represented by the core for two main reasons. The first, mathernttical, one is to ensure that the belief set. is compatible with the upper probability. The second, philosophical, one is the following (Caprio & Gong 2023 Caprio & Mukherjec 2023cl. A criticism brought forward by Walley (1991 Section 2.10.4.(c)) is that, given an upper probability P, there no cogent reason for which the agent should choose a specific Pr that is dominated by P, or for that maiter a collection of "plausible" probabilities. Because the core considers all (countably additive) probability measures that are dominated by P, it the perfect instrument to reconcile Walley's behavioral and sensitivity analysis interpretations.
Let the agent compute Po, and consider L/ == core( Po); it represents the set of plausible likelihoods. Let
L L: { -
dl' de
Pa co }
(4)
and denote by I(0)= == supLes ((0) and by L(0) = inf st L(0), for all 0 O. Call
dPD dp
1(0)p(0)
P Pd P
= p(0 D) Jo L(0)p(0)d0)
dP
Pe pco, L dPo du
p du
Po E Ceo
the class of posterior probabilities when the prior is in poo and the likelihood is in 48, and let (A) suppoepe (A),, for all A E B. Then, the following is a generalization of Bayes' theorem in Wasserman & Kadane 1990).
Theorem 10. Suppose pm, Cg are nonempty. Then for all A E B,
(D(A) suppep" So (0))
(5)
c
provided that the ratio is well defined. Here, inf Seg ((0) and 1Λ denotes the indicator function for A E B. In addition, if P is concave, then the inequality in equation an equality for all A E B.
suppEP So L(0) (0)
This result. particularly appealing because, given some assumptions, it allows to perform a (generalized) Bayesian update of prior upper probability (PUP) by carrying out only one operation, even when the likelihood is ill specified so that set of likelihoods is needed. We also have the following.
Lemma 11. Suppose pco Lgo are nonempty. Then, if P is concave, we have that PD D is concave as well.
23 Supprrserrpt "co" stands for convex and core.
35

---

Published in Transactions on Machine Learning Research (10/2024)
This lemma is important because it tells us that the generalized Bayesian update of Theorem 10 preserves concavity, and so it can be applied to successive iterations. Tf at time the PUP is concave, then the PUP at time t+1 that is, the posterior upper probability at time -ill be concave too. Necessary and sufficient conditions for generic upper probability to be concave are given in Marinacci & Montrucchio 2004 Section 5).
In the future, these results can be generalized the case in which the elements of L are unbounded using techniques in Troffaes & de Cooman (2014), and to the case in which the elements of 2. are -vluuuded, for some d N, since never used specific properties of R in our proofs.
Despite being attractive, the generalized Bayesian update of Theorem T0 hinges upon three assumptions, namely that pco and Cg are both cores of an upper probability, that they are nonemply, and that the prior upper probability is concave. As the proverb goes, there no free lunch. Having to check these assumptions, together with computing a supremum, an infinum, and the integrals in equation 5 makes Theorem 10 inadequate be applied in the context of CBDL.
Bounds on upper and lower entropy E
In this section we find an upper bound for HP)) and a lower bound for ((P) that are extremely interesting. We first need to introduce three concepts.
Definition 12. Consider a set P of probabilities on generic measurable space (1, We say that lower probability P is conver if P(AUB)> P(A) P(B) P(An)), for all A,B F.
Then, let P be either an upper or lower probabilily, and consider a generic bounded mensurable function f on F), that is, f E B(2).. We define the Choquet integral of f with respect to P as follows
J. [W)P(())) = P(WWER: ((W) 2 21) di [ [P ((w) 2 03) P({2)] dt.
where the right hand side integrals are (improper) Riemann integrals. If P is additive, then the Choquet integral reduces to the standard additive integral.
Finally, if S2 is uncountable, define for all w E Ω
I(w) = inf dP PE dp (w) and F(w) = sup dP Pes dp (w).
We call them lower and upper densities, respectively.
The following theorem gives the desired bounds. Theorem 13. Consider set P of probabilities on generic measurable space (9, If is uncountable, assume that every P P is dominated by a a-finite measure and that the Madon-Nikodyye derivatives dp are continuous and bounded, for all P E P. Define
dp
log [ P(de)
if Q is uncountable
H(P)
Ewer P((w}) log[P([w]]]
if Ω is at most countable
and similarly
H(P) = lo log [[)) P(dw) Ewen P((w)) log ((())
Q is uncountable
if Ω is at most countable"
Then, H(P) S H() and H(P) 2 H(P). In addition, if is concave, the first bound tighter, and if P is conver, the second bound tighter.
Remark 14. In Hullerneier & Waegeman 2021 Section 4.6.1), the authors point out that Abellán & Moral (200)) presents generalization of the Hartley measure, called generalized Hartley measure GH(P), that can be used to disaggregate the tolal uncerlainly captured by H H(P) into aleatoric and epistemic uncertainlies.
36

---

Published in Transactions on Machine Learning Research (10/2024)
We prefer not to introduce it in the present work because GH(P) is defined based on the mass function of belief function [Gong & Meng, 2021, Definition 2 This enlails that the authors assume that lower probability P. associated with the set of probabilities of interest is belief function, and so that for every collection (A, A such that C As, the following holds
2
E (-1)*1-PP(re)
(+IC()....))
for all & E N. As it is immediate to see, this assumption is not needed in the contert of CBDL. Other ways of disentangling and quantifying AU and EU can be found e.g. in Sale et al. 22023 2024
F How to derive a predictive distribution
Suppose we performed a Bayesian updating procedure so to obtain posterior pdf p(0 2 y1...,, In Vn. Recall that {TT (:) J==1 E (X' y)" denotes the training set. We obtain the predictive distribution p(ū 2, 21, V1 ................................ ., Fni Vn) on Y' as follows
p 11, V1.......................... Zn yn)
p(ū.0 0 1..1,1....... Ens In )d0
plū 0, 0,2,01,01.... tn, Ln,yn) ) p(0 I. 21, 21, Ln, Ba )l0
p(ū (.0) p(0 x1, ,1....,, Nn )n,
where p(ū ž, 0) is the likelihood used to derive the posterior. Notice that the last equality comes from output y only depending on input 7 and parameter 0, and from having assumed Dx Il. (see section 2.1 From an applied point of view, sample from p(ū 3, UI, ... Tna
En, Va) is obtained follows:
1. specify input 2;
2. sample a parameter 0 from the posterior, 0 ~ p(0 wi 1, Ma);
plug Ø the likelihood and sample ÿ ~ ply 0, .)..
G Aleatoric uncertainty check for a-level IHDR
The diameter of (P (Prrea) function of the predictive aleatoric uncertainty (AU) faced by the agent If we want to avoid to perform the computation Step of Algorithm [ only to discover that I IR, (P (Ppred) is "too large", then we can add "predictive AU check".
At the beginning of the analysis, compute the lower entropy H( ((Ppred) ming,s H ((prred) associated with the set exP exPpred prod of extreme elements of the Vi-approximated predictive credal set Prred- By equation 2 it is equal to the predictive alcatoric uncertainty encoded in Å pred We then verify whether the lower entropy H(pored) is "too high". That is, ((prre)) > y, for some o > 0, we want our procedure abstain. This means that if the predictive aleatoric uncertainty in set Ppred is too high, then our procedure does not return any output set for input ã. The value of & can be set equal to the entropy of the probability measures that are typically used in the context the agent works in. For example, in medical applications the agent may consider the entropy of Normal distribution, while in financial applications the entropy of distribution with fatter tails, such as a t-distribution or a Cauchy. We call these reference ip values.
If we add this "predictive AU check", at inference time (that is, before Step of Algorithm the agent needs to specify the pair of parameters (4,, a).
A belief function is mathematical concept that should not be confused with the term "belicf" we used throughout the paper address the ageut's knowledge.
25 the diameter is metric concept, we assume that can find well-defined metric dy on If that is not the case, substitute the diameter with the notion cardinality.
37

---

Published in Transactions on Machine Learning Research (10/2024)
H a-level IHDR in Classification setting
In classification problems, BNNs compute the probability vector
Σ
B
=
#
OEG
where we write Pojd to highlight the fact that @ sampled from posterior p(0 D), and then select the most likely class 0 = arg max, Wj, where the wy's are the elements of w.
When applied to classification setting, the general procedure introduced in Algorithm | becomes the following. Recall that we denote by K the cardinality of exP prior. and by S the cardinality of exP Assume that y = (y1,....,U.1) UJ}, that is, there are J E Nz2 possible labels. Then, a VT-approximated predictive distribution k,s in exP prod can be seen as J-dimensional probability vector p (Dhred ( ....... T where phase = k.s (fyr H), for all & E (1,..., K}. S1, and j E (1......}}
Now fix any and any s, and define the partial order Ix,a on y as y1 Šk.a y: - Pe.ed 2 Plaes and VI Kk,s Y/ = Phisd > Plaass where i,7 (1.....JJ) J}, i * This mcans that we can order the labels according to the probability that ppred assigns them: the first label will be the one having highest probability according to pered, the second label will have the second- highest probability according to ppred and on.
Now order the label space y according to In.e so to obtain
ym = {81 k.s ................................ VJ
7.
This means that ofth Sa. Dj R.s for all j E ,2,..., J), U/2 k. Ik,a Dj k.* for all j E J3 (but with Ik. ;2:"), and so on. That is, we order the labels from the most to the least likely according to ppred k
Then, we call aelevel credible set according to pored a E [0, 1], the set k.N
CSa (P = 81
k.
ppred
(1vii1)) E a, -at-aa,-a+s,1 Ke
1=1
(6)
j'
and 33 <j: Σ
((3, (p k()) a, ( a a
ppred k,
E
for some > 0. It corresponds to the a-level HDR Ra (Ppred) Notice that we require El pored ki
k.s (fu ")) e
[I a1-a, a, 1 - a s]] because we may need to go slightly above level 1 a. Just as a toy example, we may have 7 labels, of which would give a 0.945 coverage, while would give a coverage of 0.953. If we are interested in
the a 0.05-level credible set, we ought to include the fourth label, thus yielding a coverage slightly higher than a 0.95. The interpretation to (PPred) is the following: it consists of the smallest collection of labels to which Ppred assigns probability of at least 1 (that is, those having the highest probability of being the correct one for the new input T).
Finally, we call a-level imprecise credible set, a E [0, 1], the set
(P == UCS (Ppred)
8,5
In turn, we have that prod Hī E (Pred))) 21
Remark 15. Notice that is credible set of level ss is enough, then we can replace the left endyoint of the interval in equation 0 with 1- 1-(a + for some Ek,s > 0. Strictly speaking, in this case we obtain an (a + (x.s)) credible set, which we denole by CSa.... (Ppred) (Se.,.,(PPrrd where Okks == a + Ek,s. Going back to our loy 0.945 ze 0.95 1-a,
example, we will have credible set with 3 labels that yields coverage of of so Ck, = 0.005. In turn, this implies that the imprecise credible set will have a coverage of 1 + that is, it will have level a + maxk,s Sk,s- We denote it by (CISa(Prod)) where ā a + max Ek,--
maxk, Ck,,),
38

---

Published in Transactions on Machine Learning Research (10/2024)
Distance between a set of distributions P and a single distribution P having different dimensions
Many concepts this section are derived from Cai Lim (2022) Let m, n E such that m S n, and let p [L, oo]. Call MP(R') the set of probability measures on R' having finite p-th moment, and Ma(R)) the set of probability measures {m,n) n). Let (m,,n) := {V E m= VVT fm), where Im denotes the m-dimensional identity matrix, and for any V E O(m,n) and any b E R,, define the following function
R' having density with respect to some o-finite dominating mcasure μ4,
PV, R + R"" I x +v,ola) == Vx +b.
Let B(R²) be the Borel o-algebra on R", and for any Q E A(R", B(R))) define vv:((Q) := Qo fv. the pushforward measure. Consider then two generic probability measures Q, such that Q E M)((R)) and S E MP(R"), and call
$ (Q. == fa MP(R") (v.((a) = Q, for some V E (m, n), E R").
(Q, = {ax E Ma(R") (V. = Q, for some V E ((m, n), b E R")), Φ (S. = {8 (R") (P) 3, for some V E O n), R")..
Recall now the definition of Wasserstein metric between two generic distributions defined on the same Euclidean space. Let P1, P2 E (RR"", for some n E N and some [1, Then, the p- p-Wasscrstein distance between them is defined as
|| v123(d(xr, u)) 1/P
W WP =
inf her P2). R2n
where ||- denotes the Euclidean distance, 0o is interpreted as the essential supremum, and I(P,, Pa) : {7 E A(R2n, B(R22)) projf (+) = P2, (n) Pı} is the set of couplings between P1 and P2, where projī' is the projection onto the first n coordinates, and projž is the projection to the last n coordinates.
Recall then the definition of f-divergence between two generic distributions defined on the same Euclidean space. Let Pı, Pg A (TR" ())")), for some E N, and assume P1 < P2. Then, for any convex functional f IR such that f(1) = 0, the f-divergence between P1 and P2 is defined
divy = f S
dP 3 aP2 (r)) P2(dx).
Aside from the Rényi divergence, the f-divergence includes just about every known divergences as special case CCai & Lim 2022). The following are the main results of this section.
Lemma 16. Let m, E N such that m S n, and let E oo] and f be any convex functional on IR such that f(0) = 1. Consider a generic P c MP("") and P' MP(R"). Let D (P,n) = Upe $ (P,n) and
Φ [ n) = UP (P, Define
W"' =
: inf (Pm) W,(a, P"), for all P E ;;
dinj = inf neo (Px) diny (PI P'), for all P E P;
W P') = inf a (.. W Wp(a, P');
dio inf a (x dia, (PI
Then, for all P D the following holds
WTP P') < WP,PPP) and d < dict(P|11 (P))
39

---

Published in Transactions on Machine Learning Research (10/2024)
Lemma 17. Let N such that n, and let p [o] and be any conver functional IR such that f(0) = 1. Consider generic P C MP(R") and p E MP(RT)). Let Φ- ( = U (P, m). Define
(P, == inf (Pm) W, P"), for all P E P;
( = inf (Pm)
for all P ;;
WF infac (m.)) Wy(a, P!);
divy (*) inf.ee (m) dins P P'));
Then, for all P 9 the following holds
W PT) S (P,PPP) and div S die (P
A visual representation of the application of Lemma 17 in the context of CBDL given Figure
MP(R")
Pik
MP(R")
W L9
(
(L.
W.,
Figure 11: We assume that and that the oracle distribution L belongs to M"(R"",, while likelihood FGCS Pik subset (R")") for some finite p 2 We also assume that is one the extreme elements Pik- We see how W, (Pik, L?) W,,L, (I, LP); if we replace metric W, by generic f-divergence div,, the inequality would still hold thanks to Lemma 17
J Details on Artificial Pancreas Example
Artificial Pancreas Model. An important factor when designing the controller for an artificial pancreas is adapt the insulin delivery algorithm to the particular details of the patient. This because patients display wide range of variability in their response to insulin, depending on age, Body Mass Index (BMI), and other physiological parameters. The Bayesian Neural Network models have 2 hidden layers, with 10
40

---

Published in Transactions on Machine Learning Research (10/2024)
neurons cach, for the case study with 4 different seeds. This choice was informed by the experiments in Dutta et al. (2018). For the case study with different architectures, we trained BNNs with 4 different. widths: 10, 20, 30, and 40. The horizon length is H = 10 time steps, and the prediction horizon is 5 steps into the future. The neural networks were trained for 200 time steps, with a learning rate of 0.001, and batch size of 128 using Mean-Field Variational Inference. The training dataset consisted of 28400 training samples, recorded without meals.
We implemented simple model predictive controller, using off-the-shelf implementation of the covariance matrix adaptation evolution strategy (CMA-ES). The model predictive control planning horizon was k = 5, with fixed seed for the randomized solver.
Controller.
K Other possible baselines for CBDL
Our experiments may seem like a type of belief tracking (Fortin et al. 2017 Klein et al.) 2010 Two comments are in order. First, this line of literature does not use deep learning techniques. Second, in these works the authors rely on Dempster- Shafer theory, field in imprecise probability theory where lower probabilities are assumed to be belief functions, see Remark 14 work.
We do not rely on this assumption in our
L Further related work
Modeling uncertainty has been longstanding goal of ML/AI research and a variety of approaches have been developed for doing (Guo et al. 2017 Park et al. 2020 Jospin al. 2022). Recently, emphasis has been placed discerning between aleatoric and epistemic uncertainties (Senge et al. 2014 Kull & Flach 2014 Kendall & Gal 2017). In Tretiak et al. (2022), the authors present an IP-based neural network which uses a regression technique based on probability intervals. Contrary to CBDL, their NN is rooted in the frequentist approach to imprecise probabilities (Huber & Ronchetti 2009).
In Manchingal & Cuzzolin (2022] Sections 2.1, 2.3), the authors focus on belief-functions- based classification methods. CBDL cannot be directly compared with these methodologies because (i) they do not require that the user expresses their knowledge via a belief function, but rather through credal set; (ii) they be used for regression and classification; (iii) they are rooted in Bayesian theory, as opposed to Dempster- Shafer theory. Other works in ML. using belief function approach are those from the field of evidential
machine learning (EML), see e.g. (Amini et al.) [2200] [Charpentier et al. 20200] Denceux 2022 2023 Sensoy et al. 2018) and references thercin. Existing models mainly address clustering, classification, and regression problems. The reasons why CBDL camnot be directly compared with methods from the EML literature are (i) and (iii) above, the fact that EML methods are not (derived from) Bayesian ones, and that their definitions of AU and EC are slightly different from the canonical ones in ensemble deep learning.
M Proofs
Proof of Proposition 3 If P(A) = supPrem ('(A), for all A F, then it is immediate to see that P(A)
supren P(A)= = supp'exx11' P'(A) = supp''m P((A), for all A F, since II II'
Suppose now that P(A) supren P(A), for all A E F. Then, we have that for all P and all A E F. P(A) < P(A). Pick now any P' E II'. We can write it as p = = where By E [0, 1]. for all
jE (1,...,kk) kj,, E Br 1, and (P)} = IT. Pick then any A E F; we have
P'(A) = Σ ()) &
Σ
j=1
1=1
So P(A) 2 P((A). Because this holds for all Pl II' and all A E F, the claim is proven.
7
41

---

Published in Transactions on Machine Learning Research (10/2024)
Proof of Proposition 0 The lower bound for the upper entropy of II' comes immediately from II' being superset of IT.
Let us now prove the lower entropy equality. Let II = PaJ and II' = Convil. Pick any P E II'. By the definition of II', there exists collection of non-negative reals {3,} such that Σ 3, 1 and Σj= B B P By the concavity of the entropy, we have that
h
H(P') = H
Σ
P 2 Σ (PH(P) Σ - = inf HPP).
Petl
Since p' was chosen arbitrarily and II is finite, this implies that
inf PEttt
H(P) =: H(P) 2 H(P).
In addition, we have that, since C II, HP) v (P)) Combining this with the above result, we obtain
H(P") H(P).
Proof of Proposition Pick any metric on the space Da = A(2,F) of probabilities on (Q, and any P II'. Because p' belongs to II', r' d(P", V) can only be either equal or smaller than d(P', V). By the definition of d(II', V), if inf d(P", V) = d(P', V)), then d(II', V) d(P', vr). If instead inf d(P', W) d(P', V), then d(II', D d(P', V). The proof is similar for generic divergence div on A)
Proof of Theorem 10 Assume poo, Lg are nonempty. Then, by Marinacei & Montrucchio 2004 Proposition 3), they are weak* teak*-compact Pick any A B. Recall that we can rewrite the usual Bayes updating rule as
Io (0))
Po(A) = J0 1 (0) + fo x0)
I. ((0)1a=(0)P(00)
+
Jo (())())))
which is maximized when
Se 1(0)1 (O))P((O)
Ss (0))LA)))
is minimized. But
So (()))))(0)P((O)
inf S0 (0)) suprepso Js (())
Sg ((d)))
which proves the inequality equation 5 Assume now that P is concave. By Wasserman Kadane Lemma 1), we have that there exists E pro such that
1990
sup PEPco J ((0)1a = - (1))(a)))(())),
(7)
for all L E L. In addition, by Wasserman & Kadane
1990 Lemma 4), we have that for all X 2 and all > 0, there exists a non-negative, upper semi-continuous function h < X such that
/ x(0))(d0)
sup PEP e
- - sup PEPM
h(d))
(8)
sup PePco
((O))
42

---

Published in Transactions on Machine Learning Research (10/2024)
Let now X L1A- Notice that since Lg is weak* compact, by equation so is L. This implies that L, T. E L, since compact set always contains its boundary, T. E L well, and in turn I1A E 2. Fix then any L E L and put h = LlA. It is immediate to see that h is non-negative and upper semi-continuous. Then, by equation we have that for all c
sup PEP [(0))a(0)P((0) -E<
(9)
(0))0)))))( S sup (())))())P(d)). PEPeo je
sup PEp
Combining equation andequation
g wC obtain
(10)La())P(())
sup PEP
(10)
(()) S sup PEP Je
(())))(0)P(dB),
for all L L. Pick now any > 0 and put
= sup PEP
((0)La(0)P(d0)
e
inf PEP
((0)LA-(0)P(al)) 0.
10 have that [suppep 1e (0) 8 < So (())))) and that pepe (() P(dO)] + 6 So (O))(d).) Recall that
Choose any I. L and (k) By equation
с c :u=reepeee (0)) infrep So (0) and define d (0)) + (0)) Then,
PD(A)= = So (0))()))))
d suppeE- Se [(d())
5 5 suppep'o So (0) 6
T.
suppepoo So A
Since this holds for all c > 0, we have that
suppepeo So (0))
sup PDEPe
PD(A)
concluding the proof.
Proof of Lemma 11 Walley (981); Wasserman Kadane 19900) show that concave upper probabili- ties are closed with respect the generalized Bayes' rule. In particular, this means that, we let b = suppepm Sg ((d()) + inf So (0) for any fixed B, P is concave, then for all L L
PD(A)= = suppep. le I( (0)
(11)
b
is concave. But since Cg is weak* -ommact (by our assumption and Marinacci & Montrucchio (2004 Proposition 3)), by equation so is L. This implies that L, To E L, since compact always contains its
43

---

Published in Transactions on Machine Learning Research (10/2024)
boundary. Call then L' = I1.++ + L1A-. It is immediate to that L' E L. Then, by equation 11 we have that if we call b' == supp ( + in fe (0) it follows that
suppep Se [ Pb(A)=
b' suppepo Se (. (() P(d0)
is concave, concluding the proof.
Proof of Theorem 13 Suppose Q is uncountable. First notice that, since we assumndd dP to be continuous dp
and bounded for all E P, then so log dP for all E P, since composing a continuous function dp' with a contimuous and bounded one gives us a continuous and bounded function. This entails that the Choquet integrals of log off with respect to P. and P are both well defined. In addition, being continuous de
and bounded, both dP and log da attain their infima and suprema thanks to Weierstrass' extreme value du theorem, for all P E 3 Hence, all the Choquet integrals used in this proof are well defined.
Then, we have the following
dP (w) P(aw) du
H(P) sup H(P)
= sup PEP
log
PE
dP
sup
(-log)
dp (w)
P(dw)
PE n
dp (w)) } P(dw)
dP
sup PE .1 PE
sup
(-log)
(- log)
(12)
dP
d/ (w) } P(iww
sup n PE
(- (olg))
(13)
dP
inf
log
(w) P(dw)
(14)
9 PE3*
du
dP
log
inf
(w) P(dw)
(15)
PE
du
log [T(W)) P(dw) (P)..
The inequality in equation 12 is true because for all w E Ω
dP du
dP (-llg)
sup P
(- log)
(w)
(ap (w)
The incquality in equation 13 property of Choquet integrals taken with respect to upper probabilitics
(Marinacci & rucchio 2004). The equality in equation 14 is true because for generic function f, we
have that sup-f- inf f. Finally, the equality in equation 15 is true because the logarithm is a strictly increasing function. By Marinacci & Montrucchio (2004 Theorem 38), if P is concave, then inequality equation 13 holds with an equality, and so the bound is tighter.
The proof for H(P) 2 H(P) is similar; we use the facts that
inf (- log)
dP
du (w)) }
(- log) di (w)) for all w E Ω
by Marinacci & Montrucchio (2004),
( inl
{ (- log)
dP du
(w) } P(dw)
inf PE
Jg PE
(16)
2 inf { (-log)
logg
dP
(w) P(dw);
5 PES
dp
44

---

Published in Transactions on Machine Learning Research (10/2024)
for generic function f, inf -f sup
by Marinacci & Montrucchio (2004 Theorem 38), if P. is convex, then equation 16 holds with an equality.
Suppose now Z is at most countable; in this case, do not need any assumptions to make the Choquet integrals well defined, since we will not deal with density functions. The following holds
H(P) = sup H(P) PED
sup PE
Σ
P((W)) log [P(+W))]
WEB
Σ sup
[P([W))
PEΦ WEn
sup {P([w])(- [P({)))) [P([W))]
(17)
PE
WE
Σ P([w}) sup (- log) [P(()))
(18)
PED
WEL
Σ P((w)) inf log (((w))]
(19)
PES
WE
Σ P([W]) log
PEP inf P((w))
(20)
Σ WED
P((w))log (P(W)) H(P).
The incquality equation comes from the well known fact that the sum of the suprema is least equal to the supremum of the sum. The inequality in equation [18] comes from the fact that for differentiable functions, the product of the suprema is least equal to the supremum of the product. The equality in equation
19 is true because for a generic function f, we have that sup sup-f = inf f. Finally, the equality in equation 20 is true because the logarithm is strictly increasing function. By Marinacci & Montrucchio (2004 Theorem 38), if P is concave, then inequality equation 17 holds with an equality, and so the bound is tighter.
The proof for H(P) 2 (P)) is similar; we use the facts that
the sum of the infima is at most equal to the infimum of the sum;
for differentiable functions, the product of the infima is at most equal to the infimum of the product;
for generic function f, inf ff sup
Theorem 38), if P is convex, then inf Ewen P((w))(- log) [P(w)) Even inf (P((w))(- log)
by Marinacci & Montrucchio 20044
[P[) [P([W))]}
E
16 Fix any p [1, oo] and pick any P. Because (, C (* then inf aea; Wa,a, P') can only be cither equal or smaller than inf ae (P.)) Wp(a, P'). Now, if inf infee (P) Wa P') = inf (P Wp(a, P'), then (Tt, inf aeet Wa P') < inf anfeettP... Wy(a, then W PP) < WP,,PP" P"). This concludes the first part of the proof. Fix then any convex functional ʃ on R such that S(0) = 1; the proof is similar for f-divergences.
Proof of Lemma
W (P.
If instead
Proof of Lemma
The proof is very similar to that of Lemma T6
45

---

Published in Transactions on Machine Learning Research (10/2024)
N All Results: In-distribution and Out-of-distrbuutio Evaluation
N.1 CIFAR-10 Results
CBDL
Ensemble
BNN
Epistemic
Aleatoric
Epistemic
Aleatoric
Epistemic
Aleatoric
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
gaussian
0.145
0.169
0.176
0.066
0.099
0.102
0.012
0.016
0.016
0.065
0.056
0.055
0.268
0.359
0.38
0.292
0.388
0.412
shot hoise
0.129
0.158
0.174
0.003
0.084
0.104
0.01
0.014
0.016
0.069
0.061
0.066
0.224
0.315
0.378
0.245
0:342
0.400
speckle noise
0.128
0.158
0.174
0.003
0.102
0.014
0.016
0.069
0.061
0.055
0.224
0.31
0.375
0.241
0.336
0.407
impulse
0.134
0.161
0.174
0.062
0.085
0.118
0.011
0.015
0.017
0.069
0.039
0.051
0.231
0.338
0.423
0.257
0.364
0.454
delocus blur
0.105
0.129
0.176
0.095
0.01
0.076
0.154
gaussian hlur
0.105
0.154
0.
1.8
0.032
0.069
0.11
0.008
0.013
0.017
0.076 0.004
0.053
0.154
0.255
0.348
0. 185
0.322
0.497
motion blur
0.137
0.166
0.175
0.055
0.085
0.1
0.011
0.015
0.018
0.068 0.06
0.056
0.227
0.322
0.36
0.268
0.385
0.446
blur
0.145
0.161
0.176
0.063
0.077
0.1
0.012
0.014
0.016
0.066 0.062
0.056
0.253
0,287
0.333
0.318
0.336
0.457
CIFAR-10C
Show
0.131
0.157
0.163
0.00
0.077
0.086
0.01
0.014
0.015
0.07 0.062
0.069
0.222
0.31
0.337
0.242
0.326
0.355
0.104
0.122
0.154
0.033
0.008
0.076
0.06
0.155
0.199
brightness
0.103
0.108
0.124
0.031
0.034
0.043
0.008
0,008
0.01
0.076 0.075
0.072
0.153
0.164
0.195
0.179
0.188
0.225
contrast
0.107
0.145
0.035
0.066
0.148
0.005
0.012
0.075 0.065
0.046
0.162
0.265
0.69
clastic
0.134
0.144
0.168
0.003
0.059
0.089
0.011
0.012
0.015
0.069 0.06Y
0.068
0.219
0.238
0.33
0.273
0.207
0.38
pixclate
0.116
0.135
0.162
0.042
0.085
0.098
0.003
0.011
0.015
0.073 0.066
0.058
0.181
0.259
0.342
0.215
0.275
0.38
Jpeg
0.13
0.147
0.156
0.049
0.084
0.075
0.01
0.012
0.013
0.07
0.006
0.063
0.217
0.282
0.200
0.246
0.203
0.331
spatter
0.117
0.145
0.147
0.041
0.065
0.062
0.009
0.012
0.012
0.073
0.065
0.006
0.191
0.279
0.206
0.216
0.299
0.291
saturate
0.113
0.131
0.074
0.07
0.183
Frost
0.124
0.153
0.166
0.047
0.075
0.099
0.01
0.013
0.015
0.07 0.063
0.056
0.208
0.299
0.361
0.23
0.326
0.403
Epistemic
Alcatoric
Epistemic
Alcatoric
Epistemic
Aleatoric
In Dist-Clean
0.102
0.031
0.008
0.076
0.151
0.179
MNIST
0.184 (1.801)
0.142 (2.626)
0.021 (2.723)
0.044 (0.572)
0.445 (2.95)
0.772 (4.312)
Dataset
Fashion
(1.791)
0.147 (4.808)
(2.822)
(0.557)
(2.861)
(4.53)
MNIST
SVHN
0.183 (1.789)
0.141 (4.606)
0.019 (2.421)
0.046 (0.608)
0.394 (2.612)
0.647 (3.616)
Table 8: CIFAR-10 Results. The 4 BNNs trained have the following accuracies 90, 89,90, 90, 89 in percentage terms and rounded to the nearest whole number. For different categories of corruptions, increasing severity leads to higher levels of aleatoric uncertainty for CBDL. When exposed to completely unseen datasets, this reaches its peak. In contrast, the Ensemble has reverse trend. For the single BNN, the network with the highest accuracy selected.
46

---

Published in Transactions on Machine Learning Research (10/2024)
N.2 MNIST Results
CIRDI.
Ensemble
Epistemic
Alestoric
Epistemic
Aleatoric
Epistemic
Aleatoric
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
brightness
0.21
0.246
0.195
0.128
0.087
0.158
0.023
0.026
0.024
0.045
0.05
0.046
0.323
0.399
0.707
0.11
0.229
0.301
canny edges
0.162
0.158
0.162
0.174
0.176
0.174
0.016
0.015
0.016
0.042
0.042
0.042
0.611
0.658
0.651
0.296
0.289
0.296
dutted line
0.173
0.176
0.102
0.102
0.013
0.055
0.055
0.055
0.375
0.377
0.376
0.15
log
0.185
0.187
0.19
0.163
0.167
0.166
0.023
0.023
0.023
0.043
0.042
0.013
0.749
0.827
0.826
0.326
0.359
0.361
glass blur
0.161
0.145
0.135
0.181
0,206
0.2
0.016
0.016
0.02
0.038
0.033
0.033
0.634
0.706
0.751
0.323
0.389
0,386
Impulse noise
0.186
0.172
0.181
0.083
0.132
0.158
0.013
0.015
0.021
0.057
0.049
0.042
0.299
0.489
0.766
0.122
0.211
0.379
motion blur
0.184
0.161
0.137
0.123
0.185
0.195
0.016
0.018
0.019
0.049
0.035
0.029
0.404
0.765
0.918
0.188
0.444
0.542
MNIST-C
rotate
0.167
0.142
0.072
0.196
0.013
0.014
0.015
0.059
0.0488
0.033
v0.247
0.451
0.749
0.096
0.379
scale
0.196
0.169
0.121
0.08
0.152
0.232
0.014
0.015
0.013
0.057 0.044
0.025
0.269
0.478
0.859
0.103
0.217
0.541
0.177
0.153
0.061
0.219
0.346
0.365
shot noise
0.188
0.179
0.18
0.061
0.08
0.113
0.012
0.012
0.014
0.062 0.0.5
0.052
0.213
0.247
0.376
0.078
0.001
0.162
spatter
0.188
0.174
0.176
0.074
0.132
0.127
0.013
0.016
0.018
0.059 0.047
0.049
0.247
0.455
0.421
0.098
0.226
0.195
stripe
0.18
0.184 0.182
0.165
0.163
0.161
0.021
0.02
0.021
0.04 0.04
0.00
0.745
0.76
0.76
0.225
0.22
0.229
translate
0.191
0.192
0.192
0.001
0.086
0.128
0.013
0.016
0.019
0.061 0.036
0.046
0.214
0.292
0.474
0.078
0.112
0.204
zigzag
0.179 0.176
0.123
0.051
0.05
0.176
0.477
0.476
0.201
Epistemic
Aleatoric
Epistemic
Aleatoric
Epistemic
Aleatoric
In Dist-Clean
0.188
0.057
0.012
0.063
0.198
0.071
CIFAR
0,185 (0.986)
0.107 (2.94)
0.023 (1.988)
0.042 [0.67]
0.821 (4.155)
0.372 (5.124)
Dataset
Fashion
0.162 (0.866)
0.187 (3.287)
0.019 (1.618)
0.035 (0.564)
0.919
(1.654)
0.461 (6.496)
MNIST
SVIN
0.190 (1.015)
0.168 (2.563)
0.022 (1887)
0.043 [0.69)
0.923 (4.674)
0.297 [4.183]
Table 9: The BNNs trained have the following accuracy: 99, 99,99,,90 99, 98 in percentage terms and rounded the nearest whole number. These are the probabilities of the most likely label to be the correct one according to the 4 different BNNs. For the single BNN, the network with the highest accuracy was selected. For different categories of corruptions, increasing severity leads to higher levels of aleatoric uncertainty for IBNNs. When exposed to completely unseen datasels, this gets close to the highest aleatoric uncerlainty. The same is not true EBNN. The epistemic uncertainly for IBNNs shows less consistent trend in this case.
47

---

Published in Transactions on Machine Learning Research (10/2024)
N.3 SVHN Results
CHRDI.
Ensemble
Epistemic
Alestoric
Epistemic
Aleatoric
Epistemic
Aleatoric
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
brightness
0.062
0.074
0.102
0.011
0.019
0.053
0.003
0.000
0.008
0.083
0.08
0.07
0.113
0.161
0.293
0.136
0.198
0.405
contrast
0.077
0.099
0.143
0.022
0.048
0.174
0.006
0.008
0.018
0.08
0.072
0.04
0.17
0.261
0.37
0.247
0.487
1.212
blur
0.109
0.138
0.009
0.017
0.072
0.272
0.486
elastic
0.182
0.177
0.192
0.14
0.122
0.102
0.023
0.021
0.022
0.01
0.046
0.05
0.562
0.195
0.489
0.84
0.779
0.639
log
0.157
0.171
0.181
0.095
0.124
0.135
0.016
0.02
0.023
0.056
0.047
0.042
0.448
0,334
0.599
0.602
0.709
0.704
Trost
0.089
0.118
0.131
0.022
0.052
0.063
0.007
0.01
0.012
0.078
0.069
0.066
0.2
0.319
0.372
0.233
0.388
0.442
gaussian blur
0.085
0.131
0.14
0.017
0.087
0.208
0.005
0.012
0.019
0.081
0.06
0.032
0.143
0.38
0.646
0.176
0.603
1.151
gaussian
0.116
0.182 0.163
0.031
0.061
0.108
0.018
0.025
0.061 0.074
0.014
0.251
0.429
0.655
0.251
0.41
0.586
SVHN-C
noise
Impulse Holse
0.169
0.035
0.06
0.102
0.012
0.018
0.43
0.589
Jpeg
0.07
0.083
0.129
0.014
0.018
0.041
0.005
0.006
0.012
0.082
0.08
0.071
0.124
0.153
0.287
0.133
0.6
0.306
motion blur
0.1
0.157
0.167
0.027
0,078
0.149
0.008
0.015
0.02
0.077 0.06
0.041
0.233
0.433
0.605
0.272
0.568
0,889
pixclate
0.081
0.123
0.18
0.011
0.036
0.095
0.004
0.011
0.02
0.083
0.073
0.052
0.102
0.222
0.516
0.114
0.255
0.591
saturate
0.063
0.065
0.141
0.011
0.012
0.075
0.005
0.005
0.014
0.083
0.083
0.062
0.106
0.108
0.376
0.116
0.12
0.415
shot noise
0.119
0.183
0.067
0.106
0.019
0.074
0.014
0.255
0.462
0.659
0.256
0.581
Show
0.131
0.16
0.16
0.043
0.076
0.096
0.012
0.017
0.017
0.07 0.059
0.055
0.3
0.435
0.492
0.31
0.443
0.519
spatter
0.088
0.127
0.06
0.007
0.012
0.079
0.169
0,404
0.191
0.300
0.403
speckle noise
0.1
0.143
0.181
0.023
0.046
0.086
0.008
0.016
0.023
0.077 0.007
0.002
0.201
0.345
0.600
0.207
0.343
0.516
X00m blur
0.061
0.061
0.064
0.011
0.012
0.013
0.004
0.004
0.005
0.083 0.083
0.083
0.105
0.108
0.121
0.117
0.123
0.136
Epistemic
Aleatoric
Epistemic
Aleatoric
Eplatemic
Aleatoric
In Dist. Cleta
0.061
0.01
0.004
0.083
0.097
0.107
CIFAR
0.181 (2,992)
0.121 (11.778)
0.026 (6.029)
0.040 (0.479)
0.655 (6.765)
0.716(6.69)
Datasct
MNIST
0.198 (3.272)
0.063 (6. 154)
0.024 (3.4955
0.056 (0.675)
0.329 (3.333)
0.336 (3.139)
Fashion
0.199 (3.258)
0.113 (I0.984)
0.028 (6.002)
0.041 (0.4
0.103 (6.376)
0.126 178)
MNIST
Table 10: The BNNs trained have the following accuracy : 95, 95, 96, 95 in percentage terms and rounded to the nearest whole number. These the probabilities of the most likely label to be the correct one according to the 4 different BNNs. For the single BNN, the network with the highest accuracy was selected. For different categories of corruptions, increasing severity leads to higher levels of aleatoric uncertainty for IBNNs. When exposed to completely unseen datasels, this reaches its peak. The same is not true for EBNN. For the epistemic uncertainly as well, there is clear trend with increasing corruption severity.
48

---

Published in Transactions on Machine Learning Research (10/2024)
N.4 Fashion MNIST
Haseline
Epistemic
Aleatoric
Epistemic
Aleatoric
Epistemic
Aleatoric
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
Low
Med
High
brightness
0.199
0.212
0.205
0.108
0.106
0.094
0.021
0.024
0.022
0.051
0.048
0.051
0.568
0.519
0.476
0.261
0.294
0.262
canny edges
0.18
0.175
0.177
0.157
0.164
0.16
0.021
0.02
0.02
0.043
0.012
0.043
0.729
0.731
0.753
0.216
0.215
0.217
dotted line
0.167
0.051
0.051
0.012
0.013
0.067
0.066
0.335
0.36
0.131
Tog
0.19
0.185
0.187
0.128
0.132
0.129
0.021
0.02
0.021
0.046
0.015
0.046
0.489
0.463
0.418
0.264
0.25
0.237
glass blur
0.189
0.189
0.193
0.114
0.127
0.133
0.017
0.019
0.022
0.052
0.048
0.045
0.517
0.641
0.777
0:209
0.224
0.334
Impulse noise
1.1
0.196
0.202
0.052
0.00
0.111
0.013
0,019
0.02
0.066
0.055
0.051
0.304
0.498
0.675
0.118
0.203
0.276
Fashion
motion blur
0.182
0.171
0.163
0.106
0.137
0.174
0.017
0.02
0.019
0.053
0.004
0.038
0.505
0.616
0.641
0.216
0.200
0.285
MNIST-C
rotate
0.173
0.162
0.074
0.164
0.184
0,016
0.06
0.035
0.392
0.754
0.797
0.338
0.341
scale
0.154
0.172
0.127
0.054
0.123
0.223
0.01
0.015
0.014
0.067
0.05
0.028
0.302
0.637
0.917
0.11
0.209
0.398
shear
0.172
0.177
0.182
0.13
0.014
0.019
0.02
0.039
0.333
0.609
0.257
shot nnise
0.10
0.178
0.197
0.046
0.059
0.088
0.01
0.013
0.017
0.068
0.004
0.056
0.268
0.344
0.49
0.037
0.121
0.17
spatter
0.15
0.189
0.191
0.044
0.082
0.074
0.01
0.018
0.017
0.07
0.057
0,059
0.279
0.479
0.459
0.112
0.22
0.19
stripe
0.21
0,205
0.2008
0.125
0.126
0.125
0.025
0.025
0.025
0.041
0.041
0.041
0.686
0.698
0.697
0.349
0.352
0.341
translate
1.3
0.189
0.17
0.046
0.094
0.158
0.01
0.018
0.019
0.069
0.064
0.041
0.27
0.499
0.721
0.102
0.21
0.317
zigzag
0.187
0.066
0.016
0.016
0.016
0.061
0.061
0.061
0.401
0.411
0.151
0.152
Epistemic
Aleatoric
Epistemic
Aleatoric
Epistemic
Aleatoric
In Dist-Clean
0.121
0.029
0.007
0.075
0.197
0.35
CIFAR
0.205 (1.705)
0.104 (3.561)
0.022
(3.178)
0.050 (0.667)
0.408 (2.073)
0.221 (0.629)
Dataset
MNIST
0.165 (1.367)
0.182 (6.192)
0.021 (3.069)
0.033 (0:413)
0.909 (4.618)
0.328 (0.932)
SVHN
0.218 (1.81)
(3.742)
(3.827)
0.046 [0.618]
(3.036)
0.305 (0.867)
Table 11: The 4 BNNs trained have the following accuracies: 93, 92, 92, 92 in percentage terms and rounded to the nearest whole number. These the probabilities of the most likely label to be the correct according to the 4 different BNNs. For the single BNN, the network with the highest accuracy was selected. For different categories of corruptions, increasing severity leads higher levels of aleatoric uncertainty for IBNNs. Which is high when exposed to completely unseen datasets as well. The same is not true for EBNN.
49

---

Published in Transactions on Machine Learning Research (10/2024)
O Evaluation Summary by Noise
0.1 CIFAR-10 Results
BNN
Ensembble
CIDL
gaussian noise
0.613
0.116
0.703 = 0.116
0.715 + 0.168
shot noise
0.609 0.077
0.794 0.077
0.793
impulse noise
0.66 = 0.132
0.776
= 0.132
0.791 + 0.191
speckle noise
gausssaa blur
0.741
0.073
0.836
1 0.073
0.825
= 0.191
defocus blur
0.750
0.044
0.919
0.044
0.912
= 0.105
blur
0.813
0.036
0.902
0.038
0.889
0.067
20om blur
0.78 = 0.020
0.899
Σ 0.029
0.888
= 0.006
CIFAR
Tog
0.753
0.034
10
frost
0.745 0.025
0.884
+ 0.025
0.884
= 0.062
SHOW
0.039
0.927
0.039
0.044
spatter
0.773 0.036
0.355
0.036
0.956
1 0.019
contrast
0.000
0.100
0.785
0.155
0.782
= 0.234
hrightness
0.737
0.025
0.991
0.025
0.004
saturate
0.751
0.019
0.984
+ 0.019
0.383 = 0.009
jpeg
0.775
pixelate
0.685 0.097
0.848
0.097
0.855 1 0.132
clastic
Results for CIFAR-10C, organized by noise. For different categories of corruptions, increasing severity leads to higher levels aleatoric uncertainty for CBDL. When exposed to completely unseen datasets, this reaches its peak. In contrast, the baseline has trend.
Table 12:
O.2 MNIST Results
CBDL
shot noise
0.597
0.07
0.999
0.07
0.999 + 0.001
Impulse noise
0.56
0.296
0.716
0.298
0.7 0.396
glass blur
0.365
0.217
0.406
0.217
0.41
0.271
motion blur
0.498
0.317
0.65
0.317
0.854 1 0.394
Tog
0.364
0.209
1.
spatter
0.714
0.089
0.991
0.089
0.991 + 0.007
brightuess
0.63 = 0.323
0.803
= 0.323
0.904
+ 0.113
sheur
0.663
0.121
0.916
1 0.121
0.915
= 0.168
MNIST
rolate
0.64 1. 0.066
0.990
= 0.066
0.999
1 0.0
0.618 0.32
0.755
0.32
0.749
0.382
Translate
0.604 0.142
0.992
0.142
0.987
1 0.023
dotted
0.019
0.994
zigzag
0.777 0.023
0.931
= 0.023
0.927
0.014
stripe
0.288 0.279
0.06
0.279
0.106
= 0.08
canny edges
0.84
0.043
0.849 0.043
0.911 1. 0.038
Table 13: Results for MNIST-C, organized by noise. For different categories of corruptions, increasing severity leads
to higher levels of aleatoric uncertainty for CBDL. When exposed completely unseen datasets, this reaches peak. In contrast, the baseline has reverse trend.
50

---

Published in Transactions on Machine Learning Research (10/2024)
0.3 Fashion MNIST Results
BNN
Ensemble
CBDL
shot noise
0.538
0.134
0.636 = 0.131
0.621 = 0.161
impulse
glass blur
0.407
0.164
0.374
0.164
0.385
0.227
motion
0.66 =
0.328
0.001
Tog
0.045
0.031
0.122
0.031
0.312 1 0.094
spatter
0.665
0.043
0.937
1 0.043
0.336 = 0.019
brightness
0.096
0.117
0.213
0.371 - 0.201
Fashion
shear
0.593 0.143
0.691
= 0.143
0.719 + 0.165
MNIST
rotate
0.277
scale
0.437
0.255
0.38
0.259
0.396
= 0.363
Translate
0.580
0.147
0.710
0.762 _ 0.178
datted Time
0.661
0.024
0.345
0.024
0.947 1 0.01
zigzag
0.660
0.019
0.315
= 0.019
0.919 1 0.015
stripe
0.141
0.016
0.071
canny edges
0.143 + 0.088
0.035 = 0.086
0.062 = 0.023
Table 14: Results for Fashion MNIST-C, organized by noise. For different categories corruptions, increasing severity leads higher levels of aleatoric uncertainty for CBDL. When exposed to completely unseen datasets, this reaches its peak. In contrast, the baseline has a reverse trend.
0.4 SVHN Results
BNN
Ensemble
CBDL
Gaussian Noise
0.476
0.304
0.544
00.04
+ 0.339
Shot Noise
0.417
0.298
0.502
0.298
0.504 + 0.332
Impulse Noise
0.478
0.304
0.522
0.304
0.532 1 0.315
Sperkle Noise
0.586
0.211
0.69
0.11
0.693 1 0.28
Gaussian Blur
0.834
0.325
0.627
= 0.325
0.62 : 0.39
Detocus
0.363
0.351
0.405
Motion Blur
0.765
0.094
0.803
4 0.094
0.795
= 0.133
Zoom Blur
0.675
0.02
0.998
0.02
0.998 + 0.002
SVHN
Fog
0.432
0.2644
0.363
0.264
0.34
0.246
Frost
0.79
0.032
0.952
0.032
0.945
2. 0.033
Snow
0.738
0.034
0.779
0.034
0.775
0.08
Spatter
0.714 0.093
0.807
1 0.093
0.798
- 0.188
Contrast.
0.803
Brightness
0.716 0.067
0.992
= 0.067
0.980
= 0.016
Saturate
0.707
0.078
0.93 + 0.078
0.916
0.14
JPEG
0.744 0.051
0.951
1 0.051
0.949
0.073
Pixelate
0.577 0.275
0.721
= 0.275
0.729
1 0.35
Elastic
0.274 0.23
0.213 0.23
0.227
0.208
Table 15: Results for SVHN-C, organized by noise. For different categories of corruptions, increasing severity leads
to higher levels aleatoric uncertainty for CBDL. When exposed completely unseen datasets, this reaches peak. In contrast, the baseline has reverse trend.
51

---

Published in Transactions on Machine Learning Research (10/2024)
P Accuracy vs Rejection Rate All Results
P.1 CIFAR10-C
CIFAR-10C Gaussian noise
Gaussian noise Level:l
Gaussian noise -evell22
Gaussian noise Level:3
Gaussian noise -Level:4
Gaussian noise -eevel:5
1.0
1.0
1.0
1.0
1.0
Accurey
Accurecy
Accurao
Accurecy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.8
0.8
0.4
0.2
0.2
0.2
0.2
0.2
0.75
0.0
n.5
Rejection Rate
Rejection Rate
Rejection
Rejection
Rejection
CBDL
Figure 12: Accuracy Rejection Rate- CIFAR10C ga ussiann noise.
CIFAR-10C Shot noise
Shot Level:1
Shot Level:2
Shot Leve]:3
Shot -Lvell44
Shot -ovell:5
1.0
1.0
1.0
1.0
1.0
Accuracy
Accuracc
Accuracy
Accuravy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.0
0,25
0.5
0.75
0.0
0,25
0.5
0.75
0.25
0.5
0.75
Rejociion Rale
Rejociion Rate
Rejection Rate
Dejection Rale
Rejection Rate
ENS
Figure 13: Accuracy Rejection Rate- CIFARIOC shot noise.
CIPAR-10C- Impulse noise
Impulse Level:1
Impuise Level:2
Impulse
Impulse
Impuise
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acaurry
Accurey
Accursy
Accusey
Accursey
0.6
0.6
0.4
0.4
0.8
0.8
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.0
0
0.25
n.5
0.75
0.0
0.25
0.5
0.75
0.25
0.5
0.75
Rejection Rate
Rejection Bate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
ENS
Figure 14: Accuracy vs Rejection Rate CIFAR10G impulse noise.
52

---

Published in Transactions on Machine Learning Research (10/2024)
CIFAR-10C Speckle noise
Speckle noise Level:1
Speckle noise Level.2
Speckle noise Level.3
Speckle noise Level:4
Speckle noise Level:5
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acourocy
Acourocy
Accuroey
Accuroay
Accurscy
0.6
0.6-
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rato
Rejection Rate
BNN
CBDL
....
BNS
Figure 15: Accuracy vs Rejection Rate CIFARIOC speckle noise.
CIPAR-10C- Coussian blur
Gaussian blur Level:1
Gausstan blur Level:2
Gausslan blur 3
Gaussian blur Level:4
Gaussian blur Level::5
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Aocurary
Accurary
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.225
0.5
0.75
0.0
0
0.25
11.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejoction Hate
Rojection Kate
Rejection Rate
Rejection. Hate
Rejection Hate
BNN
CBDL
ENS
Figure 16: Accuracy vs Rejection Rate - CIFAR1OC gaussian blur.
CIFAR-10C- Defocus blur
Defocus blur 1evel:1
Defocus blur Level.2
Defocus blur Level:3
Defocus blur Level:4
Defocus blur -eveel:
1.0
0.8
0.8
0.8
0.8
0.8
Accurocy
Accurocy
Accurocy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0
0.25
D.5
0.75
0.0 h
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Hate
Rejection Hate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
ENS
Figure 17: Accuracy vs Rejection Rate CIFAR1OC defocus blur.
CIFAR-10C: Motion blur
Motion Level-1
Motion blur Level.2
Motion blur Level:3
Motion blur Level-4
Motion blur Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Aesurnyy
Aecurncy
Accuracy
Accuracy
Accoracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.2
0.2-
0.2-
0.2
0.2-
0.0
0.0
0.0 0
0.0
0.0
0
0.25
0.5
0.75
0
0.25
0.5
0.75
0.25
0.5
0.75
0
0.25
0.5
0.73
0
0.25
U.5
0.73
Rejection Rate
Rejection Rabe
Dejection Rabe
Bejection Rate
Rejection Rate
BNN
CBDL
KNS
Figure 18: Accuracy vs Rejection Rate CIFAR10C motion blur.
53

---

Published in Transactions on Machine Learning Research (10/2024)
CIFAR-10C- Zoom blur
Zoom blur Level:1
Zoom blur Level:2
Zoom blur Level:3
Zoom blur Level:4
Zoom blur Level:
1.0
1.0
0.8
0.8
0.8
0.8
0.85
Acourocy
Acourocy
Accuroey
Accuracy
Accurscy
0.6
0.6-
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
BNS
Figure 19: Accuracy vs Rejection Rate CIFARIOC zoom blur.
CIFAR-10C Fog
Fog Level:1
Fog Level:2
Fog Level:3
Fog Level-4
Fog Level:5
1.0
1.0
1.0
1.0
0.8
0.8-
0.8
0.8-
0.8
Aocurary
Accurary
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.22
0.5
0.75
0.0
0
0.25
11.5
0.75
0.0
0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejoction Mate
Rojection Kate
Rejection Rate
Rejection Hate
Rejection Hate
BNN
CBDL
ENS
Figure 20: Accuracy vs Rejection Rate CIFAR1OC log.
CIFAR-10C. Frost
Frost Level:1
Frost Level:2
Frost Level:3
Frost Level::
Frost Level::5
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accurocy
Accurocy
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0
0.25
D.5
0.75
0.0 h
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Hate
Rejection Hate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
ENS
Figure 21: Accuracy vs Rejection Rate CIFAR1OC frost.
SIFAR-1OC. Snow
Snow
Level:1
Snow Level:2
Snow Level:3
Snow Level:4
Snow Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8-
0.8
Aesurayy
Aecurnry
Accuracy
Accuracy
Accoracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.2
0.2-
0.2-
0.2
0.2-
0.0
0.0
0.0 0
0.0
0.0
0
0.25
0.3
0.75
0
0.25
0.5
0.75
0.25
0.5
0.75
0
0.25
0.5
0.73
0
0.25
0.5
0.73
Rejection Rate
Rejection Rabe
Rejection Rabe
Bejection Rate
Rejection Rate
BNN
CBDL
ENS
Figure 22: Accuracy vs Rejection Rate- CIFARIOC snow.
54

---

Published in Transactions on Machine Learning Research (10/2024)
CIFAR-IOC- Spatter
Spalter -evel:1
Spalter Level:22
Spalter Levell3
Spatler Level:4
Spatter -evell.5
1.0
1.0
0.8
0.8
0.U
0.8
0.8
Acourocy
Acourocy
Accursey
Accuracy
Accurscy
0.6
0.6-
0.6
0.6-
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
BNS
Figure 23: Accuracy vs Rejection Rate- CIEARIOC spatter.
CIFAR-10C- Contrast
Contrast Level:1
Contrast Level:2
Contrast Level:3
Contrast Level-4
Contrast -Level:5
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accurary
Accurary
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0 0
0.225
0.5
0.75
0.0
0
0.25
1..5
0.75
0.0
0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejoction Mate
Rojection Kate
Rejection Rate
Rejection Hate
Rejection Hate
BNN
CBDL
ENS
Figure 24: Accuracy vs Rejection Rate CIFAR1OC contrast.
CIFAR-10C- Brightness
Brightness Level:1
Brightness Level.2
Brightness Level:3
Brightness Level:4
Brightness Level:5
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accurocy
Accurocy
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0
0.25
D.5
0.75
0.0 n
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Hate
Rejection Hate
Rejection Rate
Rejection Rate
Rejection Hate
BNN
CBDL
ENS
Figure 25: Accuracy vs Rejection Rate- CIFAR10C brightness.
CIFAR-10C: Saturate
Saturate
Level:1
Saturate Level:2
Saturate Level:3
Saturate -Level-4
Saturate -Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
fesuracy
Aecurnry
Accuracy
Accuracy
Accoracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.2
0.2-
0.2-
0.2
0.2
0.0
0.0
0.0 0
0.0
0.0
0
0.25
0.3
0.75
0
0.25
0.5
0.75
0.25
0.5
0.75
0
0.25
0.5
0.73
0
0.25
0.5
0.73
Rejection Rate
Rejection Rabe
Rejection Rabe
Bejection Rate
Rejection Rate
BNN
CBDL
KNS
Figure 26: Accuracy vs Rejection Rate CIEARIOC saturate.
55

---

Published in Transactions on Machine Learning Research (10/2024)
CLFAR-10C - Jpeg compresston
Joeg compression Level:1 Jpeg compression
Level:2 Joeg compression Level:3 Jpeg compression Level:: Jpeg compression Level:!
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accureyy
Accurecy
Accurscy
Accureyy
Accursey
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.0
0.25
n.5
0.75
0.0
0.25
0.5
0.75
0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
ENS
Figure 27: Accuracy vs Rejection Rate CIFARIOC jpeg compression.
CIFAR-10C Pixelate
Pixelate Level::1
Pixelate Level:2
Pixelate Level:3
Pixelate Level:d
Pixelate Levell:5
1.0
1.0-
1.0
1.0
1.0
Accurecy
Accurazy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.8
0.4
0.2
0.2
0.2
0.75
0.5
0.5
Rojection
Rejection
Rejection
Rejection
Rejection
Figure 28: Accuracy vs Rejection Rate- CIFARIOC pixelate.
CIFAR-10C- Elastic transform
Elastic transform Level:1
Elastic transform Level:2 Elastic transform Level:3
Elastic Lransform Level-4
Elastic Lransform Level.
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acouray
Accuracy
Accursey
Accusccy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.0 0
0.25
0.75 Rujection Rate
0.5
0.0
0.25
0.75 Rujection Rate
0.5
Rejection Rate
Rejection Rate
Rejection Bate
DNN
CDDL
RNS
Figure 29: Accuracy vs Rejection Rate CIFAR10C elastic transform.
56

---

Published in Transactions on Machine Learning Research (10/2024)
P.2 MNIST-C
MNIST- Shot noise
Shot noise Level:1
Shot noise - -Lvell22
Shot noise - -Lveel33
Shot noise - Level:
Shot noise - -evel:5
1.0
1.0
1.0
1.0
1.0
0.8
0.3
0.3
0.8
0.8
Aocarocy
0.6
0.6
0.6
0.0
0.0
0.4
0.4
0.4
0.4
0.4
0.25 0.5
0.75 Rejection Rate
0.0
0.25 0.5
0,75 Rejection Rate
0.0
0.25 0.5
0.75
0.0
0.25
0.75 Rejection Rate
0.5
0.0
0.25
0.75 Rejertion Rate
0.5
Rejection Rate
BNN
CBDL
ENS
Figure 30: Accuracy vs Rejection Rate MNISTC shot noise.
MNIST-C Impulse noise
Impulse noise Level:1
Impulse noise Level:2
Impulse noise Level:3
Impulse noise -Leee::4
Impulse noise -eeve::5
1.0
0.8
0.8
0.8
0.8
Accuracy
Accuracy
Accwra.y
Accurazy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
1..
0.75
0.0
11.5
Rejoection Rate
Rejoction Rate
Rejoction Rate
Rejoction Rate
Rejection Rate
Figure 31: Accuracy Rejection Rate MNISTC impulse noise.
MNIST-C Glass
Class blur Level.1
Class blur Level.2
Class blur Level.3
Closs blur Level:4
Class blur Level:5
Accuracy
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.75
Regection Rate
Rejoctton Rate
Rejoction Rate
Rejection Rate
Rejection Rate
Figure 32: Accuracy vs Rejection Rate MNISTC glass blur.
57

---

Published in Transactions on Machine Learning Research (10/2024)
MNIST-C Motion blur
Motion blur Level:1
Motion blur Level-2
Motion blur Level:3
Motion blur Level:d
Motion blur Levee::5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acouracy
Aceuracy
Acourazy
Acouracy
Acouracy
0.6
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.25
11.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0
0.25
01.5
0.75
Regoction Rate
Rejoction Rate
Rejoction Rate
Rejoction Rate
Rejoction Rate
BNN
CBDL
ENS
Figure 33: Accuracy vs Rejection Rate MNISTC motion blur.
MNIST-C Fog
Foq Level:1
Fog Levell22
Fog Level:3
Fog -Lvell:4
Fog LLvell:
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
Accurscy
Aocurscy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.5
0.75
0.25 0.5
0.75 Rejection Rate
0.25
0.75 Rejection Rate
0.5
0.5
0.75
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
Figure 34: Accuracy Rejection Rate MNISTC fog.
MNISTC- Spatter
Spatter Level:1
Spatter Level:2
Spatter Level:3
Spatter Level:4
Spatter Level::5
Accurssy
Accursey
Accuracy
Accursscy
Accurssyy
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.5
0.0
0.5
0.0
0
0.5
0.0
0
0.5
0.0
0.5
Rejection
Rejection
Rejection
Rejection
Rejection
Figure 35: Accuracy vs Rejection Rate MNISTC spatter.
58

---

Published in Transactions on Machine Learning Research (10/2024)
MNIST-C Brightness
Brightness Level:1
Brightness Levell2
Brightness Level:3
Brightness Level-4
Brightness Level::5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acouracy
Aceuracy
Acouracy
Acouracy
Acowany
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.25
11.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
Regoction Rate
Rejoction Rate
Rejoction Rate
Rejoction Rate
Rejoction Rate
BNN
CBDL
ENS
Figure 36: Accuracy vs Rejection Rate MNISTC brightness.
MNIST-C Shear
Shear Level:1
Shear Level:2
Shear Level:3
Shear Level:4
Shear -Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Accursey
Aocuracy
Aocuroug
Aocursuc
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.5
0.75
0.25
0.5
0.75
0.25
0.75 Rejection Rate
0.5
0.25
0.75 Rejection Rate
0.5
0.0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
Figure 37: Accuracy Rejection Rate MNISTC shear.
MNIST-C
Rotate -Leeel1
Level:2
Level:3
Level:4
Rotate -Level:5
Accurssy
Accursey
Accuracy
Accurssy
Accurssyy
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.5
0.0
0.5
0.0
0
0.5
0.0
0.5
0.0
0.5
Rejection
Rejection
Rejection
Rejection
Rejection
Figure 38: Accuracy vs Rejection Rate MNISTC rotate.
59

---

Published in Transactions on Machine Learning Research (10/2024)
MNIST-C Scale
Scale Level:1
Scale Level:2
Scale Level:3
Scale -Levll:d
Scale -evel::5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accuracy
Aceuracy
Acourazy
Acourayy
Acourayy
0.6
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.25
11.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0
0.25
11.5
0.75
Regoction Rate
Rejoction Rate
Rejoction Rate
Rejection Rate
Rejoction Rate
BNN
CBDL
---.
ENS
Figure 39: Accuracy vs Rejection Rate MNISTC scale.
MNIST-C Translate
Transiate Level:1
Transiate Level:2
Transiate Level:3
Transiate Level:4
Transiate Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Accursey
Aocuracy
Aocurscy
Apcursscy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.5 0.75
0.25
0.5
0.75
0.25
0.75 Rejection Rate
0.5
0.75 Rejection Rate
0.5
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
Figure 40: Accuracy Rejection Rate MNISTC translate.
Dotted
Dotted Level:1
Dotted line Level:2
Level:3
line Level::4
Dotted line Levvll:5
Accurssy
Accursey
Accurssy
Accurssyy
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.5
0.0
0.5
0.0
0
0.5
0.0
0.5
0.0
0
0.5
Rejection
Rejection
Rejection
Rejection
Rejection
Figure 41: Accuracy vs Rejection Rate MNISTC dotted line.
60

---

Published in Transactions on Machine Learning Research (10/2024)
MNIST-C Zigzag
Zigzag Level:1
Zigzag Level:2
Zigzag Level:3
Zigzag Level:4
Zigzag Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acouracy
Aceuracy
Acouracy
Acourazy
Aceuaay
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.25
11.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.55
0.75
Regoction Rate
Rejoction Rate
Rejoction Rate
Rejection Rate
Rejoction Rate
BNN
CBDL
---.
ENS
Figure 42: Accuracy vs Rejection Rate MNISTC zigzug.
MNIST-C Stripe
Stripe -Lvel:11
Stripe Level.2
Stripe Level:3
Stripe LLvel:4
Stripe -Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
Accursey
Aocurscy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.5
0.75
0.25
0.75 Rejection Rute
0.5
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
Figure 43: Accuracy Rejection Rate MNISTC stripe.
MNISI:C. Canny edges
Canny edges LLvel:1
Canny edges Levell2
Canny edges Level:3
Canny edges Level::4
Canny edges Level::5
Accurssy
Accurseyy
Accuracy
Accurscy
Accursey
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.5
0.0
0.5
0.0
0
0.5
0.0
0.5
0.0
0.5
Rejection
Rejection
Rejection
Rejection
Rejection
Figure 44: Accuracy vs Rejection Rate MNISTC canny edges.
61

---

Published in Transactions on Machine Learning Research (10/2024)
P.3 Fashion MNIST-C
Fashion MNIST-C Shot noise
Shot noise - Level:1
Shot noise - -evell22
Shot noise - Level:3
Shot noise - -evell:4
Shot noise - -evel:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8-
0.8
0.8
Accuracy
Aocurney
0.6
0.6
0.6
0.6
0.0
0.4
0.4
0.4
0.4
0.4
0.25 0.5 0.75
0.0
0.25
0,75 Rejection Rate
0.5
0.0
0.25
0.5
0.75
0.0
0.25
0.75 Rejection Rate
0.5
0.0
0.25 0.5
0.75 Rejertion Rate
Rejection Rate
Rejection Rate
BNN
CBDL
ENS
Figure 45: Accuracy Rejection Rate FashionMNISTC shot noise.
Fashion MNIST-C Impuise noise
Impulse noise Level:1
Impulse noise Level:2
Impulse noise Level:3
Impulse noise
Level:4
Impulse noise -Leve::5
1.0
0.8
0.8
0.8
0.8
0.8
Accuracy
Accuracy
Accurouy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
10.
0.75
1.5
0.75
0.0
0.55
0.0
11.5
0.75
Rejoection Rate
Rejoction Rate
Rejoction Rate
Rejoction Rate
Rejection Rate
CBDL
Figure 46: Accuracy Rejection Rate FashionMNISTC impulse noise.
Fashion MNIST Glass blur
Class blur Level.1
Class blur Level.2
Class blur Level.3
Closs blur Level:4
Class blur Level:
Accuracy
Accuracy
Accuracy
Accurcc
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
18.5
0.5
1..5
0.75
Regection Rate
Rejoctton Rate
Rejoction Rate
Rejection Rate
Rejection Rate
Figure 47: Accuracy vs Rejection Rate MNshionMNISTC glass blur.
62

---

Published in Transactions on Machine Learning Research (10/2024)
Pashion MNIST-C Motion biur
Motion blur Level:1
Motion blur Level:2
Motion blur Level:3
Motion blur Level:d
Motion blur Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acouracy
Aceuracy
Acouracy
Acouaagy
0.6
Acourazy
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.25
11.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.55
0.75
Regoction Rate
Rejoction Rate
Rejoction Rate
Rejoction Rate
Rejoction Rate
BNN
CBDL
ENS
Figure 48: Accuracy vs Rejection Rate FashionMNISTC motion blur.
Pashlon MNIST-C Fog
Foq LLvvl:11
Fog Levell:2
Fog Levell3
Fog LLvel:44
Fog -Lvell:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8-
0.8
0.8
Accursey
Aocuracy
Aocarscy
Aocursuy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.5
0.75
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
Figure 49: Accuracy Rejection Rate MNSSCC fog.
Fashion Spatter
Spatter LLeee:1
Spatter Level:2
Spatter -Level:3
Spatter Level:4
Spatter Level::
Accurssy
Accursey
Accurasy
Accurssy
Accurssyy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.5
0.0
0.5
0.0
0
0.5
0.0
0.5
0.0
0
0.5
Rejection
Rejection
Rejection
Rejection
Rejection
Figure 50: Accuracy vs Rejection Rate Pashion spatter.
63

---

Published in Transactions on Machine Learning Research (10/2024)
Foshion MNIST-C- Brightness
Brightness Level:1
Brightness Levell2
Brightness Level::
Brightness Level-d
Brightness Level::5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acouracy
Acouracy
Acouracy
Acouracy
Acowany
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.22
11.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0155
0.75
Regoction Rate
Rejoction Rate
Rejoction Rate
Regoction Rate
Rejoction Rate
BNN
CBDL
ENS
Figure 51: Accuracy vs Rejection Rate PNsHionMMISTC brightness.
Fashion MNIST-C Shhaa
Shear Level:1
Shear I Level:2
Shear -Lveel33
Shear Level:4
Shear -Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8-
0.8-
Accursey
Aocuracy
Aocarscy
Aocursey
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.5 0.75
0.25 0.5
0.75
0.25 0.5
0.75 Rejection Rate
0.5 0.75 Rejection Rate
0.0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
Figure 52: Accuracy vs Rejection Rate FashionMNISTC shear.
MNIST-C
Rotate -Level:1
Level:2
LLevel3
Rotate Level:4
Rotate -Level:5
Accurssy
Accursey
Accuracy
Accurssyy
Accurssyy
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.5
0.0
0.5
0.0
0
0.5
0.0
0.5
0.0
0.5
Rejection
Rejection
Rejection
Rejection
Rejection
Figure 53: Accuracy vs Rejection Rate FashionMMISSTC rotate.
64

---

Published in Transactions on Machine Learning Research (10/2024)
Fashion MNIST-C Scale
Scale Level::1
Scale Level:2
Scale -Level:3
Scale -Lvveldd
Scale -evee::5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acouracy
Aceuracy
Acouracy
Asouracy
Acowary
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.25
11.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
01.5
0.75
Regoction Rate
Rejoction Rate
Rejoction Rate
Rejection Rate
Rejoction Rate
BNN
CBDL
ENS
Figure 54: Accuracy vs Rejection Rate MNISI scale.
Fashion MNIST-C Translate
Transiate Level:1
Transiate Level:2
Transiate Level:3
Transiate Level:4
Translate LLvel::
1.0
1.0
1.0
1.0
1.0
0.8
Accursey
Aocuracy
Aocurocy
Apcurszc
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.5 0.75
0.25 0.5
0.75
0.25
0.75 Rejection Rate
0.5
0.0
0.75 Rejection Rate
0.5
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
Figure 55: Accuracy Rejection Rate MashionMNISTC translate.
Fashion MNISICC
Dotted Level:1
Dotted line Level:2
Level:3
Dotted Level:4
Dotted line Level::5
Accurssy
Accursey
Accursscy
Accurssyy
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.5
0.0
0.5
0.0
0
0.5
0.0
0.5
0.0
0
0.5
Rejection
Rejection
Rejection
Rejection
Rejection
Figure 56: Accuracy vs Rejection Rate FashionMNISTC dotted line.
65

---

Published in Transactions on Machine Learning Research (10/2024)
Fashion MNIST-C Zigzag
Zigzag Level:1
Zigzag Level:2
Zigzag Level::3
Zigzag Level:d
Zigzag Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accuracy
Aceuracy
Acourazy
Acourazy
Aceurayy
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.25
11.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.55
0.75
Regoction Rate
Rejoction Rate
Rejoction Rate
Rejection Rate
Rejoction Rate
BNN
CBDL
ENS
Figure 57: Accuracy vs Rejection Rate FNshionMNISTC zigzag.
Fashion MNIST-C Stripe
Stripe Level:1
Stripe Level.2
Stripe Level:3
Stripe Level:4
Stripe -Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8-
0.8
Accursey
Aocurscy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.5
0.75
0.25
0.75 Rejection Rate
0.5
0.25
0.5
0.75
0.25
0.5
0.75
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
Figure 58: Accuracy Rejection Rate FNshion stripe.
Fashion Canny edges
Canny edges Level:1
Canny edges Level:2
Canny edges Level:3
Canny edges Level:4
Canny edges Level::
Accurssy
Accurseyy
Accuracy
Accursscy
Accurssyy
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.5
0.0
0.5
0.0
0
0.5
0.0
0.5
0.0
0.5
Rejection
Rejection
Rejection
Rejection
Rejection
Figure 59: Accuracy vs Rejection Rate Fashion canny edges.
66

---

Published in Transactions on Machine Learning Research (10/2024)
P.4 SVHN-C
SVHN-C- Gaussian noise
Gaussian noise Level:1
Gaussian noise Level:2
Gaussian noise Level:3
Gaussian noïse Level-4
Gaussian nojse Level
1.0
1.0
1.0
1.0
1.0
Accuraey
Accurncy
Accurayy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.4
0.2
0.2-
0.2
0.2
0.2
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.0
0,25
0.5
0.75
0.5
0.75
0.25
0.5
0.75
Pejociion Rale
Rejecion Rale
Dejection Rall
Dejection Rale
Rejection Rale
Figure 60: Accuracy vs Rejection Rate SVHNC Gaussian Noise.
SVHN-C Shot noise
Shot Level:1
Shot Level:2
Shot LLvee::3
Shot -Level:
Shot -Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8-
0.8-
Accureyy
Accuracy
Accuracy
Accuracy
Accuracy
0.6
0.6
0.4
0.4
0.4
0.4
0.3
0.2
0.2
0.2
0.2
0.2
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.25
0.5
0.75
Rojection Rate
Rojection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
ENS
Figure 61: Accuracy vs Rejection Rate SVHNC Shot Noise.
SVHN-C- Impulse noise
Impulse noise Level:1
Impulse noise Level-2
Impulse noise Level:3
Impulse noise Level-4
Impulse noise -Lvee--5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accurary
Accurary
Acmursy
Accuracy
Ascuracy
0.4
0.4
0.4
0.2
022
0.2
0.2
0.2
0.0
0.0
0.0 0
0.0 0
0.0
0
0.25
0.5
0.75
0
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0
0.25
0.5
0.75
Rejecsion Rale
Rejection Rale
Dejection Rall
Dejection Rall
Rejection Rale
BNN
CBDL
ENS
Figure 62: Accuracy vs Rejection Rate SVHNC Impulse Noise.
67

---

Published in Transactions on Machine Learning Research (10/2024)
SVHN-C- Speckle noise
Speckle noise Level:1
Speckle noise Level.2
Speckle noise Level.3
Speckle noise Level:4
Speckle noise Level::
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acourocy
Acourocy
Jccurocy
Accuraay
Accurscy
0.6
0.6-
0.6
0.6
0.6
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0 0
0.25
0,5
0.75
0.0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
BNS
Figure 63: Accuracy vs Rejection Rate SVHNC Speckle Noise.
SVHN-C Caussian blur
Gaussian blur Level:1
Gausstan blur Level:2
Gaussian blur 3
Gaussian blur Level:
Gaussian blur Level:5
1.0
1.0
1.0
1.0
0.8
0.8-
0.8
0.8-
0.8
Aocurary
Accurary
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.225
0.5
0.75
0.0
0
0.25
1..
0.75
0.0 0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejoction Mate
Rojection Kate
Rejection Rate
Rejection Hate
Rejection Hate
BNN
CBDL
ENS
Figure 64: Accuracy vs Rejection Rate SVHNC Gaussian Blur.
SVHN-C Defocus blur
Defocus blur Level:1
Defocus blur Level.2
Defocus blur -Level:3
Defocus blur Level:4
Defocus blur Level:5
1.0
1.0
0.8
0.8
0.8
0.8-
0.8
Accurocy
Accurocy
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0
0.25
D.5
0.75
0.0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Hate
Rejection Hate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
ENS
Figure 65: Accuracy vs Rejection Rate - SVHNC Defocus Blur.
SVHN-C Molion blur
Motion blur Level-1
Motion blur Level-2
Motion blur Level:3
Motion blur Level-4
Motion blur Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Aesurnyy
Aecurnry
Accuracy
Accuracy
Accoracy
0.6
0.6
0.6
0.6
0.6
0.4
0.4-
0.2
0.2-
0.2-
0.2
0.2-
0.0
0.0
0.0 0
0.0
0.0
0
0.25
0.3
0.75
0
0.25
0.5
0.75
0.25
0.5
0.73
0
0.25
0.5
0.73
0
0.25
U.5
0.73
Rejoction Rate
Rejection Rabe
Rejection Rabe
Bejection Rate
Rejection Rate
BNN
CBDL
KNS
Figure 66: Accuracy vs Rejection Rate - SVHNC Motion Blur.
68

---

Published in Transactions on Machine Learning Research (10/2024)
SVHN-C- Zoom blur
Zoom blur Level:1
Zoom blur Level:2
Zoom blur Level:3
Zoom blur Level:4
Zoom blur Level:5
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Acourecy
Acourocy
Accursey
Accurocy
Accursey
0.6
0.6
0.6
0.6-
0.6
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0 0
0.25
0,5
0.75
0.0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
BNS
Figure 67: Accuracy vs Rejection Rate SVHNC Zoom Blur.
SVHN-C- Fog
Fog Level:1
Fog Level:2
Fog Level:3
Fog
Level-4
Fog Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8-
0.8
0.8-
0.8
Accurary
Accurary
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0
0
0.25
1..
0.75
0.0 0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
Rejoction Mate
Rojection Kate
Rejection Hate
Rejection. Hate
Rejection Hate
BNN
CBDL
ENS
Figure 68: Accuracy vs Rejection Rate SVHNC Fog.
SVHN-C- Frost
Frost
Level:1
Frost
Level:2
Frost Level:3
Frost Level:d
Frost -Level:5
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accurocy
Accurocy
Accurocy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0
0.25
D.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Hate
Rejection Hate
Rejection Rate
Rejection Rate
Rejection Hate
BNN
CBDL
ENS
Figure 69: Accuracy vs Rejection Rate SVHNC Frost.
SVHN-C- - Snow
Snow Leveel1
Snow
Level:2
Snow Level:3
Snow Level:4
Snow Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Aesurary
Aecurnry
Accuracy
Accuracy
Accoracy
0.6
0.6
0.6
0.6
0.6
0.4
0.2
0.2
0.2-
0.2
0.2-
0.0
0.0
0.0 0
0.0
0.0
0
0.25
0.3
0.75
0
0.25
0.5
0.75
0.25
0.5
0.75
0
0.25
0.5
0.73
0
0.25
U.5
0.73
Rejoction Rate
Rejection Rabe
Rejection Rabe
Bejection Rate
Rejection Rate
BNN
CBDL
KNS
Figure 70: Accuracy vs Rejection Rate SVHNC Snow.
69

---

Published in Transactions on Machine Learning Research (10/2024)
SVHN-C Spatter
Spalter Level:1
Spalter -Level22
Spalter -Level.3
Spatter Level:4
Spatter -evell:
1.0
1.0
0.8
0.8
0.8
0.8
0.85
Acourocy
Acourocy
Accuroey
Accuraayy
Accurscy
0.6
0.6-
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
BNS
Figure 71: Accuracy vs Rejection Rate SVHNC Spatter.
SVHN-C Contrast
Contrast Level:1
Contrast Level:2
Contrast Level:3
Contrast Level:1
Contrast -Level:5
1.0
1.0
1.0
1.0
0.8
0.8-
0.8-
0.8
0.8
Aocurary
Accurary
Accuracy
Accuracy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2-
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.225
0.5
0.75
0.0
0
0.25
11.5
0.75
0.0
0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
Rejoction Hate
Rojection Kate
Rejection Rate
Rejection Hate
Rejection Hate
BNN
CBDL
ENS
Figure 72: Accuracy vs Rejection Rate SVHNC Contrast.
SVHN-C Brightness
Brightness Level:1
Brightness
Level.2
Brightness Level:3
Brightness Level:4
Brightness Level:5
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accurocy
Accurocy
Accurocy
Accuracy
0.6
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0
0.25
n.5
0.75
0.0 h
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Rejection Hate
Rejection Hate
Rejection Rate
Rejection Rate
Rejection Hate
BNN
CBDL
ENS
Figure 73: Accuracy vs Rejection Rate SVHNC Brightness.
SVHN-C Saturale
Saturate
Level:1
Saturate
Level:2
Saturate Level:3
Saturate -Level44
Saturate -Level:5
1.0
1.0
1.0
1.0
1.0
0.8
0.8-
0.8.
0.8-
0.8
fesuracy
Aecurncy
Accuracy
Accuracy
Accoracy
0.6
0.6
0.6
0.6
0.6
0.4
0.2
0.2-
0.2-
0.2
0.2
0.0
0.0
0.0 0
0.0
0.0
0
0.25
0.5
0.75
0
0.25
0.5
0.75
0.25
0.5
0.73
0
0.25
0.5
0.73
0
0.25
U.5
0.73
Rejection Rate
Rejection Rabe
Dejection Rate
Bejection Rate
Rejection Rate
BNN
CBDL
KNS
Figure 74: Accuracy vs Rejection Rate SVHNC Saturate.
70

---

Published in Transactions on Machine Learning Research (10/2024)
SVHN-C -CHN-C--Jpeg
pegg Level:1
peg Level:2
peg
Jpeg Level:d
Jee
1.0
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
Accureyy
Accurocy
Accursey
Accureyy
Accursey
0.6
0.4
0.4
0.6
0.4
0.4
0.2
0.2
0.2
0.2
0.2
0.0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.0
0.25
n.5
0.75
0.0
0.25
0.5
0.75
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
Rejection Rate
BNN
CBDL
ENS
Figure 75: Accuracy vs Rejection Rate SVHNC JPEG.
SVHN-C Pixelate
Pixelate Level:1
Pixelate Level:2
Pixelate Level:3
Pixelate Level:d
Pixelate -eveel:5
1.0
1.0
1.0
1.0
1.0
Accurecy
Accurazy
0.6
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.8
0.8
0.2
0.2
0.2
0.75
0.5
0..
Rojection
Rejection
Rejection
Rejection
Rejection
Figure 76: Accuracy Rejection Rate SVHNC Pixelate.
SVHN-C- Elastic
Elastic Level:1
Elastic Level:2
Elastic Level.3
Elastic Level:4
Elastic Level.5
1.0
0.8
0.8
0.8
0.8
0.8
Acouaay
Accuracy
Accurscy
Accurscy
Accuracy
0.6
0.6-
0.6
0.6
0.6
0.2
0.2
0.2
0.0
0
0.25
0.5
0.75
0.0
0
0.25
0.5
0.75
0.0 0
0.25
0.5
0.75
0.0
0.25
0.75 Rujection Rate
0.5
0.0
0.25
0.5
0.75
Rejection Rate
Rejection Rate
Rejection Bate
Rujection Rate
DNN
CDDL
RNS
Figure 77: Accuracy vs Rejection Rate SVHNC Elastic.
71

---

Published in Transactions on Machine Learning Research (10/2024)
Q
Code and Dataset Availability
Code and datascts for implementation and replication of results will be available at https://githubb.com http://eedaa-aayyelaaaareeeeelle
72


Reviews:
[{"review_id": "ZB4cIfxTs2", "reviewer": "Reviewer_9eVm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The idea makes sense.\n- The theory developed is quite solid.\n\n## Weaknesses\n- The story is not really motivative and convincing. The section to motivate why we need to leverage the imprecise probabilities in Bayesian learning is not well motivated and convinced to me. The authors should rewrite this section.\n- It is still unclear the benefit of the imprecise probabilities, e.g., this enables the computation of  Imprecise Highest Density Region and Aleatoric and Epistemic Uncertainties? Moreover, the definition of $\\bar{H}(P)$ as sup of $H(P)$ is not rigorous because the sup is over $P$. Instead, it should be $H(\\Pi')$? \n- Is the final Imprecise Highest Density Region over the label set is similar to the conformal prediction? If so, the authors should compare with the conformal prediction approaches for regression and classification to demonstrate the merit.\n- The proposed approach seems to be time and memory consuming because we need to do variational approach $K \\times S$ times and store $K \\times S$ deep learning models?\n- The authors should enrich the experiments to compare with the current SOTA approaches in Bayesian Neural Networks. As far as I know, there are recently many works in Bayesian Neural Networks. \n- In Tables 1 and 2, the authors should report the expected calibration error.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes credal Bayesian deep learning. The main idea is to consider of a set of $K$ priors/ its convex hull set and another set of $S$ likelihood distribution/its convex hull set (e.g., different architectures), subsequently variational approach is applied to get $K \\times S$ posteriors (each corresponds to one pair of prior and likelihood distribution), and finally $K \\times S$ posteriors are used to make predictions and compute the Imprecise Highest Density Region over the predictions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please give more motivation for leveraging the imprecise probabilities to Bayesian learning.\n\n- What is the fundamental difference of considering a mixture of some priors and a convex hull over the set of priors and similar question for the likelihood distribution?\n\n- Please make changes based on my comments in the weakness section.", "broader_impact_concerns": "There is no ethical concern of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qtaM80TcIA", "reviewer": "Reviewer_tTAz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses an important point that often comes up with Bayesian neural networks and Bayesian approaches in general: the question of selecting a prior and likelihood to perform Bayesian inference on. Indeed, an often-heard claim against BNNs is that the choice of prior and likelihood can appear arbitrary, and while Bayesian inference is principled and rational, the Bayesian viewpoint does not help much with the former. It provides a good overview and many references for underlying research on uncertainty quantification and Bayesian theory.\n\nThus, the research question is of great importance and the paper will find interest within the community.\n\nAt the same time, there are a few weaknesses:\n\n1. The paper mentions Kendall & Gal and other works that explicitly disentangle epistemic and aleatoric uncertainty with a single BNN but then claims that BNNs cannot disentangle aleatoric and epistemic uncertainty. Thus, related work that is mentioned in this context is not compared or acknowledged properly.\n2. On the face of it, the approach trains an ensemble of BNNs with different architectures and/or priors (see also Wenzel et al., 2020). As such, it would seem necessary to compare to approaches that use meta-priors as this could be viewed as Bayesian inference over meta-priors where we choose a uniform (uninformative) prior over the prior distributions and likelihoods.\n3. The experiments only compare an ensemble of BNNs with the proposed method. While the paper mentions that they do not apply BMA (Bayesian model averaging) due to a paper that found that it does not well for BNNs approximated with certain settings for HMC, a comparison to regular BNNs (not ensembled) and different approximation methods would have been helpful to compare the quality of the proposed method.\n\nThus, I don’t see all the statements as sufficiently evidenced yet.\n\n## Details\n\n> We note in passing that EU cannot be captured using a single BNN (Hüllermeier & Waegeman, 2021).\n\nThis is simply wrong. Epistemic Uncertainty as defined in the same paragraph:\n\n> EU, instead, refers to the lack of knowledge about the data-generating process; as such, it is reducible\n\ncan be measured using the concept of Expected Information Gain (EIG), introduced by Lindley (1956) and revisited in much research since then. This concept is exactly equivalent to the mutual information used to quantify EU in BNNs (see Kendal & Gal, etc.). The paper references these but does not actually compare them. Further, Hüllermeier & Waegeman (2021) is likely not the right citation.\n\n> since AU is irreducible, there is an increasing need for ML techniques that are able to detect an excess of AU and query for human help.\n\nThis seems wrong as well. AU is irreducible, so assuming it is captured correctly by the model, querying a human for help will not help as AU is irreducible. Querying humans for help only helps for high EU. See the success of EU for active learning in BNNs and Bayesian Optimal Experiment Design.\n\n### Ensemble of BNNs\n\nFrom my previous review:\n\n> **The problem with the comparison between EBNNs and CBDL is that for EBNNs, there is not necessarily a clean disentanglement of uncertainties, as each BNN might also capture some of the epistemic uncertainty within itself. That is, a sufficiently powerful model class could learn all possible uncertainty, so the resulting ensemble would express no epistemic uncertainty.**\n\nFurther, I am not aware of *anyone* using EBNNs. The citation (Egele et al., 2021) does not substantiate this: it does not propose EBNNs from my reading.\n\n# IHDR as Union of HDRs\n\nThe paper states that the construction of IHDRs (Def 4) can be operationalized via a union of HDRs (Def 5). I thank the authors for clarifying my earlier misunderstanding about this. However, I fail to see how the second property is always fulfilled: does the union of HDRs really always provide a minimal cover of high density?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Adapted from my previous review (Paper1573 by Reviewer cXJk):\n\nThe paper introduces Credal Bayesian Deep Learning (CBDL), which allows for considering multiple priors and likelihoods and offer a different way of quantifying aleatoric and epistemic uncertainty. In empirical studies, they compare CBDL to ensembles of BNNs (EBNNs) for both regression and classification (in the appendix).\n\nThe paper suggests using multiple likelihoods and priors and simply performing Bayesian inference on all combinations to obtain a set of BNNs.\n\nTo quantify and disentangle aleatoric and epistemic uncertainty for an input the paper suggests determining the highest $\\overline{H}(P)$ and lowest entropy $\\underline{H}(P)$ and then using the decomposition:\n\n$$\n\\underbrace{\\bar{H}(P)}\\_{\\text {total uncertainty }}=\\underbrace{\\underline{H}(P)}\\_{\\text {aleatoric uncertainty }}+\\underbrace{[\\bar{H}(P)-\\underline{H}(P)]}\\_{\\text {epistemic uncertainty}}.\n$$\n\nTo make predictions, the paper defines regions of imprecise high density across the BNNs.\n\n---\n\nComparison to the previous submission: https://draftable.com/compare/sPCFPgnAKMzD", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "From the last review:\n\n1. to add comparisons to single BNNs as a baseline using the mean, stddev output parameterization as the ensembling for EBNNs might not be necessary,\n2. compare to BMA as the cited paper is specific to HMC-based inference, which you do not seem to use anyway.\n\nIn particular, EBNNs are flawed compared to a single (merged) BNN, as detailed above, so please compare them to that. Writing\n\n> From the previous sections, it is clear that CBDL improves on the uncertainty quantification capabilities of single BNNs\n\nas the start of the \"Experiments\" section is not sufficient as evidence and, hence, one of the listed contributions is not evidenced.\n\n### Typos etc\n\np. 2: \"This because\"\np. 4: the last paragraph cites Jospin et al (2022) 5 times\np. 5: the \"(Berger, 1984), (Walley, 1991, Section 5.9)\" citation should be merged.\n\nFinally:\n\nCompress the PDF before uploading please. I was able to compress the 4.3MB PDF down to 900KB.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2HTqQoJYgb", "reviewer": "Reviewer_CP1D", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n* The proposed approach has some theoretical justification and empirically leads to improved uncertainty estimates, especially for the AU component, compared to the eBNN baseline under various settings.\n\n* The problem of uncertainty quantification is important and timely.\n\n* The empirical problems seem interesting and relevant for the topic.\n\n \n### Weaknesses\n\n* The paper is generally a bit hard to read, with plenty of side comments and detours. Improving, clarifying and focusing the writing could make the paper a lot easier to read and understand.\n\n* There is next to no discussion and no empirical comparison against other existing approaches beyond (e)BNNs.\n\n* While the proposed approach does seem to work better than the baseline, many of the results are still not very satisfying (e.g., AU estimates are not consistently increasing when increasing the corruption level).\n\n* Claiming that the proposed approach is theoretically well-justified while, e.g., eBNNs are not seems like a strong claim, since as far as I know, it is not generally clear how disentangling TU into AU and EU should actually be done in a principled manner (see e.g. Sec.3.3 in Hüllermeier et al. 2022 for a discussion).\n\n* One of the main limitations of the proposed approach seems to be the high computational complexity, yet this aspect is not discussed much, especially in connection to the empirical results.\n\n#### References:\n\nHüllermeier et al. 2022: Quantification of Credal Uncertainty in Machine Learning: A Critical Analysis and Empirical Comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on quantifying uncertainty in deep learning. The proposed approach is based on credal sets, i.e., on sets of probability distributions, Bayesian learning, and on disaggregation of bounds on total uncertainty (TU) into aleatoric uncertainty (AU) and epistemic uncertainty (EU) component bounds. The authors compare their approach to (ensembles of) Bayesian neural networks ((e)BNNs), providing theoretical and empirical evidence that their credal Bayesian deep learning (CBDL) approach leads to improved uncertainty estimates under various scenarios. All in all, I think the authors mostly follow up on the claims made by presenting a combination of theoretical arguments and empirical evidence.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions and suggestions roughly in decreasing order of importance:\n\n1) Sec1, p1: \"In this paper, we present a procedure that allows us to give a machine such a desirable quality.\" Please tone down the claim: there are existing methods which do this (even if the proposed method would improve on all of them, which is not clear), and the proposed method also does not work perfectly (see e.g. results in Table 3).\n\n2) Sec1, p2: \"CBDL also gives a way of quantifying and disentangling different types of uncertainties within $P_{pred}$\". As far as I can see, the approach to disentangling AU and EU using TU comes from Abellàn et al. 2006? If so, please cite them also in the Intro, now this reads like you are proposing a novel approach to the disaggregation problem.\n   \n3) Sec1, p3: Comparing CBDL with non-Bayesian techniques: I do not fully understand why comparing to existing methods beyond Bayesian ones is out of scope. Can you explain?\n   \n4) W.r.t. to the \"bad choice\" discussion on p11: How sensitive the resulting uncertainty estimates are on the choice of the set of priors and likelihoods for the propose method? How much better such robustness is compared, e.g., to finite eBNNs? It would be good to have some discussion and empirical experiments to demonstrate this.\n   \n5) Sec4.2.2, p20: \"We observe that for lower values of α the gains of CBDL are more pronounced.\" Looking at Table 7, the performance improvements with lowest alpha level (0.9) are negative, which I interpret as saying that CBDL is worse than eBNNs. Do I misread this (also: is there spelling mistake in Table 7: IBNN should be CBDL?)?\n   \n6) E.g. Sec1, p2: P_pred \"enjoys desirable probabilistic guarantees\", also Sec1, p3. Do all the stated theoretical properties carry over when the posterior/predictive distribution is only approximated, e.g., via VI, instead of calculated exactly? Please state this clearly in the paper.\n   \n7) Sec1, p3: \"CBDL [...] is able to quantify both EU and AU, [...] in a principled manner, unlike ensemble of BNNs.\" As far as I know, disentangling TU into AU and EU like done in this paper is also not without potential problems (see e.g. Sec.3.3 in Hüllermeier et al. 2022). Why is this more principled than if one does something similar with eBNNS? You also state (e.g. in Abstract) that, heuristically, CBDL \"allows to train [...] infinite ensemble of BNNs\". If so, again, why is this more principled than if one uses finite ensemble of BNNs instead?\n\n8) Sec3.2, p12 \"This does not mean that we suffer from under-confidence\". I am not sure if I understand the claim, so just to clarify: e.g. enlarging the set of priors/likelihoods around a true oracle prior/likelihood does not come with any cost in terms of increased uncertainty? How would having a larger set of priors/likelihoods compare to having single correctly specified prior/likelihood?\n   \n9) Sec1, p2: \"This because [sic] selecting a unique prior and a unique likelihood implicitly assumes perfect knowledge around the true prior and the true data generating process. In turn, a unique distribution is only able to retrieve AU\". I would have guessed that single prior and likelihood would result in basically estimating TU (i.e., pretending that EU is zero does not actually make the uncertainty \"go away\"). Can you elaborate?\n   \n10) Abstract:  please clarify the claim: \"Although Bayesian Neural Networks (BNNs) allow for uncertainty in the predictions to be assessed, different sources of uncertainty are indistinguishable\". Do you mean specifically using a single BNN here?\n    \n11) Footnote 11, p9: you might as well state these for convenience at least in some appendix.\n    \n12) Sec4.1, p14: is there some motivation for the scale of severity, or is this just an empirical labeling?\n    \n13) Including a table with the notation and abbreviations might help in reading the paper: it is rather easy to get lost in soup of letters at some point.", "broader_impact_concerns": "I have no broader impact concerns for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Given the additional experiments provided throughout a detailed and fruitful discussion with the reviewers the paper has matured enough for me to recommend acceptance. 

The restriction of a minor revision is to ensure that the additional results are properly included into the final version of the paper.
Decision: Accept (poster)