Credal Learning Theory
Authors: Michele Caprio
Maryam Sultana
Eleni Elia
Fabio Cuzzolin
Venue: NeurIPS 2024 poster (2024)
Keywords: Statistical learning
imprecise probabilities
credal sets
epistemic and aleatory uncertainties
Primary area: learning_theory
TL;DR: We develop Credal Learning Theory, which allows to derive tighter bounds wrt SLT by leveraging a finite sample of training sets

Abstract: Statistical learning theory is the foundation of machine learning, providing theoretical bounds for the risk of models learned from a (single) training set, assumed to issue from an unknown probability distribution. In actual deployment, however, the data distribution may (and often does) vary, causing domain adaptation/generalization issues. In this paper we lay the foundations for a `credal' theory of learning, using convex sets of probabilities (credal sets) to model the variability in the data-generating distribution. Such credal sets, we argue, may be inferred from a finite sample of training sets. Bounds are derived for the case of finite hypotheses spaces (both assuming realizability or not), as well as infinite model spaces, which directly generalize classical results.

Credal Learning Theory
Michele Caprio
Department of Computer Science
University of Manchester, Manchester, UK michele. caprio@manchester. ac.uk
Maryam Sultana
Eleni G. Elia
Fabio Cuzzolin
School of Engineering Computing & Mathematics Oxford Brookes University, Oxford, UK
(msultana, fabio inzzooin//brookes ac.uk
Abstract
Statistical learning theory is the foundation of machine learning, providing the- oretical bounds for the risk of models learned from (single) training set, as- sumed to issue from an unknown probability distribution. In actual deployment, however, the data distribution may (and often does) vary, causing domain adapta- tion/generalization issues. In this paper we lay the foundations for 'credal' theory learning, using convex sets of probabilities (credal sets) to model the variability in the generatiig distribution. Such credal sets, argue, may be inferred from a finite sample of training sets. Bounds are derived for the case of finite hypotheses spaces (both assuming realizability not), well infinite model spaces, which directly generalize classical results.
Introduction
Statistical Learning Theory (SLT) considers the problem of predicting an output E y given an input E X' using mapping X + V, E H. called model or hypothesis, belonging to a model (or hypotheses) space H. The loss function error committed by a model h E H. For instance, the zero-one loss is defined as [((x, y), h) = [/ & hxx)], where II denotes the indicator function, and assigns zero value to correct predictions and one incorrect ones. Input-output pairs are usually assumed to be generated i.i.d. by a probability distribution P*, which is unknown. The expected risk or expected loss of the model h, L(h) = Lp- (h) Ep- (1((,,, y). h)] Jxxx y ((1,, y), (d(x, y)), measures the expected value taken with respect to 2* loss The expected risk minimizer h* e arg min L(h) is any hypothesis in the given model space H that minimizes the expected risk. Given training dataset D ((x1 ,/),.... (In, } whose elements are drawn independently and identically distributed (i.i.d.) from probability distribution Pa the empirical risk of a hypothesis h is the average loss D. The empirical risk minimizer (ERM), i.e., the model h one actually learns from the training set D, is the one minimizing the empirical risk [44). Statistical Learning Theory seeks upper bounds for the expected risk L(h) of the ERM h, and in turn, for the excess risk, that is, the difference between L(h) and the lowest expected risk L(h*). This endeavor is pursued under inereasingly more relaxed assumptions about the nature of the hypotheses space H. Two common such assumptions are that either the model space is finite, that there exists model with zero expected risk (realizabiliry).
(X x y) x H. + IR measures the
In real-world situations, however, the data distribution may (and often does) vary, causing issues of domain adaptation (DA) [U]] generalization (DG) [59] Domain adaptation and generalization are interrelated yet distinctive concepts in machine learning, as they both deal with the challenges of
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

transferring knowledge across different domains. The main goal of DA is to adapt a machine learning model trained on source domains to perform well on target domains. In opposition. DG aims train model that can generalize well to unseen data/domains not available during training. In simple terms, DA works on the assumption that our source and target domains are related to each other, meaning that they somehow follow a similar data- aata-generating probability distribution. DG, instead, assumes that the trained model should be able to handle unseen target data.
Attempts to derive generalization bounds under more realistic conditions within classical SLT have been made (see Section 2 Those approaches, however, are characterized by a lack of generalizability. and the use of strong assumptions. A more detailed account of the state of the art and their limitations discussed Section In opposition all such proposals, our learning framework leverages Imprecise Probabilities (IPs) to provide a radically different solution to the construction of bounds in learning theory.
hierarchy of formalisms aimed at mathematically modeling the 'epistemic' uncertainty induced by sources such lack of data, missing data or data which imprecise in nature [24 62 63], IPs have been successfully employed in the design of neural networks providing both better accuracy and uncertainty quantification to predictions 17
64 73 To date, however, they have never been
considered as tool to address the foundational issues of statistical learning theory associated with data drifting.
This paper provides two innovative contributions: (1) the formal definition of a new learning setting in which models are inferred from (finite) sample of training sets (via either objectivist or subjectivist modeling techniques, as explained in Section rather than a single training set, each assumed to have been generated by single data distribution (as in classical SLT); (2) the derivation of generalization bounds to the expected risk of a model learned in this new learning setting, under the assumption that the epistemic uncertainty induced by the available training sets can be described by a credal set [41) i.c., a convex set of (data generating) probability distributions.
Contributions.
The overall framework is illustrated in Figure I Generalized upper bounds under credal uncertainty are derived under three increasingly realistic sets of assumptions, mirroring classical statistical learning theory treatment: (i) finite hypotheses spaces with realizability, (ii) finite hypotheses spaces without realizability, and (iii) infinite hypotheses spaces. We show that the corresponding classical results in SLT are special cases of the ones derived in the present paper.
Uiserraiitty
modeling
P P[Lo(h) s (6)++ () 21-
Generalization bounds under credal uncertainty
Sample of training sets
Credill dals-generating distributions
Graphical representation of the proposed learning framework. Given available finite sample of training sets, each assumed to be generated by a single data distribution, one can learn credal set of data distributions in either frequentist or subjectivist fashion (Section 3 This allows us to derive generalization bounds under credal uncertainty (Section 4).
Figure 1:
Paper outline. The paper is structured as follows. First (Section 2 we present the existing work addressing data distribution shifts in learning theory. We then introduce our new learning framework (Section B). In Section 4 we illustrate the bounds derived under credal uncertainty and show how classical results can be recovered as special cases. Section 5 concludes and outlines future undertakings. We prove our results in Appendix A and we provide synthetic experiments on our first two main results (Theorems 4.1 and 4.5) in Appendix B
2 Related Work
The standard statistical approach to generalization is based on the assumption that test and training data are i.i.d. according to an unknown distribution. This assumption can fall short real-world applications: as a result, many recent papers have been focusing on the "Out of Distribution" (OOD) generalization problem, also known as domain generalization (DG), to address the discrepancy between test and training distribution(s) [3 40 68). Extensive surveys of existing methods and

---

approaches to DG can be found in Wang et al. (71), Zhou et al. [79]. Although several proposals for learning bounds with theoretical guarantees have been made within DA, only a few attempts have been made in the field of DG [28 60 Most theoretical attempts have focused on kernel methods, starting from the seminal work of Blanchard et al. [1], spanning to a body of later work (see for example Deshmukh et al. [28). Hu et al. [33. Muandet et al. (57)) In this line of work, assumptions related to boundedness of kernels and continuity of feature maps and loss function render the approaches not directly applicable to broader scenarios.
Other work has focused on providing theoretical grounds using Nomain-adversien learning method; in this approach, the authors use convex combination of source domains in order to approximate the target distribution leveraging H-divergence (2). Ye et al. [75] have attempted to relax assumptions to provide more general bounds, focusing on feature distribution characteristies; the authors have introduced terms related to stability and discriminative power to calculate the error bound on unseen domains, through the use of an expansion function. Nonetheless, as the authors acknowledge. practical challenges arise concerning the estimation of the expansion function and the choice of a constraint the top model to improve convergence.
Researchers have also focused on adaptation new domains over time, treating DG as an online game and the model as a player minimizing the risk associated with introducing new distributions by an adversary at each step [T]. However, in scenarios where the training distribution is significantly outside the convex hull of training distributions [2] or because of unmet strong convexity loss function assumption [[], they fall short from achieving robust generalization. Causality principles have been leveraged in this sense, for example by Bellot and Bareinboim [10, Sheth et al. [67 to provide distributional robustness guarantees using causal diagrams and source domain data. However, causal approaches for improving model robustness across varying domains pose important challenges including reliance on domain knowledge. Researchers have also explored generalization bounds for DG based on the Rademacher complexity, allowing for the approach to be applicable to broader range of models 42). Though this simplification has a number of practical benefits, models trained under covariate shift assumptions might suffer in terms of robustness to other distribution shift types. On the empirical analysis side, Gulrajani and Lopez-Paz [3 have provided a comprehensive review of the state of the art. Though simple ERM found to outperform other more sophisticated methods in benchmark experiments (TT, this approach has been criticized for its non- -on-generalizabiiity. In this direction, Izmailov al. 35 have highlighted the importance of searching for flat minima in the training process for improved generalization.
All the aforemennioned approaches take point estimate-like, stance (i.e., assuming single training set) to the derivation of generalization bounds. In this paper, in opposition, we explicitly acknow ledge the uncertainty inherent to domain variation in the form of a sample of training sets, each assumed to be generated by a different distribution, and propose a robust and flexible approach representing the resulting epistemic uncertainty via credal sets. Related works on the computational complexity specific to the use of credal sets are discussed in Appendix
3 Credal Learning
Let us formalize the notion of learning a model from a collection of (training) sets of data, each issued from difffrrent 'domain' characterized by a single, albeit unknown, data-generating probability distribution. Assume that we wish to learn a mapping h : X' + y between an input space X' and an output space )' (where, once again, the mapping h belongs to hypotheses space H), having as evidence a finite sample of training sets, Di...., DN D = { Vt.1)..... (x.. Vi.,,) } Assume also that the data in each set D; has been generated by distinct probability distribution Pt. The question we want to answer is: What sort of guarantees can be derived on the expected risk of model learned from such a sample of training sets? How do they relate classical Probably Approximately Correct (PAC) bounds from statistical learning theory?
3.1 Objectivist Modeling
While classical statistical learning theory results are derived assuming no knowledge about the Lata-generating process, the theorems and corollaries in this paper do require some knowledge, although incomplete, of the true distribution. To be more specific, we will posit that, by leveraging the available evidence Dj...., DN. the agent is able to clicit credal set i.c., closed and convex
3

---

set of probabilities that contains the true data generating process pirue = P++ for new set of data 1 (that we call the test ser), possibly different from D....., DN. . As we shall see in Section though, this extra modeling effort allows us to derive stronger results.
There least ways in which such credal set be derived, that is, via either an objectivist or subjectivist modeling stance. In this section, we present the former. We start by inspecting the frequentist approach objectivist modeling, considering in partieular epsilon-contamination models (Section 3.1.1) and belief functions models (Sections 3. 3.1.3). A further objectivist model based
on fiducial inference
32 is outlined in Appendix
3.1.1 Epsilon-contaminntioo models
In classical frequentist statistics, given the available dataset, the agent assumes the analytical form of likelihood L (not to be confused with the expected loss function, which we denote by Roman letter L), e.g., a Normal small perturbations of the specified likelihood can induce substantial differences in the conclusions drawn from the data. A robust frequentist agent is thus interested in statistical methods that may not be fully optimal under the ideal 'true' likelihood model, but still lead to reasonable conclusions if the ideal model only approximately true g
Gamma distribution. As shown by Huber and Ronchetti [34,, though,
To account for this, the agent specifies the class of e-contaminated distributions
P= = P
vQ),
where e is some positive quantity in (0, 1). and Q is any distribution X' x V. Wasserman and Kadane (74) show that d is indeed a (nonempty) credal set. In view of this robust frequentist goal, then, requiring that the true data generating process belongs P is natural assumption.
In our framework, in which a finite sample N training sets {D is available, one approach to building the desired credal set to specify N many likelihoods {Lr} J.W and cj-contaminate each of them to obtain L, = {P: P = (1 C, QQ, vQ), {I...., Nj. The credal set 'P can then be derived by setting
p = Conv (U Ld),
where Conv(-) denotes the convex hull operator An immediate consequence of Wasserman and Kadane (74) and references therein is that p Cunv(U { = {P P(A) 2 [(A), AA C X' }). where [(A) = (I....N (1
c, (A), for all A C X x y A simple numerical example for such a procedure is given in Appendix
3.1.2 Belief functions lower probabilities
An alternative way to derive a credal set from the sample training evidence can be formulated within the framework of the Dempster- Shaffer theory of evidence [27] 66].
is a set-valued random variable, modeling random experiments in which observations come in the form of sets. In the case of finite sample spaces, they are called belief functions [66]. While classical discrete mass functions assign normalized, non-negative values to the elements Ω of their sample space, belief function independently assigns normalized, non- negative mass values to subsets of the sample space: m(A) 2 O, for all A C Ω Exco m(A) = 1. The belief function associated with a mass function m then measures the total mass of the subsets of each event A, (A) ΣBÇA m(B).
random sel 39 52 55
Crucially, belief function can be seen as the lower probability (or lower envelope) of the credal set
M(Bl) (P: Ω + [0, 1] Bel(A) s P(A), AA (.).
where P is data distribution. The dual upper probability to Bel is PI(A) = 1 - Bel(A"), for all C Ω2 When restricted to singleton elements, is called the contour function, pl(w) = (l((w))
It is easy to see that the set P built this way is indeed credal set. This is because it is (i) convex by detinition, and (ii) closed because it is the union of finitely many closed sets.

---

3.1.3 Inferring belief functions from data
There various ways data, such as a sample of training sets. If a classical likelihood L having probability density or mass function (pdf/pmf)) is available (as assumed in the frequentist paradigm) one can build belief
infer belief (or, equivalently, plausibilltty) function from (partial)
function by using the normalized likelihood as its contour function. That is, pl(w) - sup"ren for all w E Ω, where f = X x y is the space where the training pairs live.
k(w)
As before, in framework in which finite sample of N training {D; 1 is available, can specify N many likelihoods {Ci J j== and their corresponding pdf/pmf's {4.) Then, we can
compute [(w) (I... & (w), for all ws Ω and in turn
pl(+)) 2(w)/ swpp//upwelll//
(1)
for all w E 2 In turn, our credal set is derived as P = (P:dP/dv= pl), where dP/dw
is the pdf/pmff associated with distribution P via its Radon-Nikodym derivative with respect sigma-finite dominating measure v. Such construction means that P includes all distributions whose pdf/pmf's are element-wiss dominated by plausibility contour pl.
Numerical example. Let Q = {Wi, W2, wa). J. where wi (Ig. 21,2,3). Suppose also that we observed four sample training sets Di...., D4 and that specified the likelihood pmf"s lg...., la as Table [ There, we see e.g. how pmf & assigns a probability of 0.3 to the element w1 of the state space and similarly for the other pmf's ' and the other elements of the state space. It is immediate to see that ? (0.4, 0.8, 0.6) Then, by Equation
I we have that
(0.5, 1, 0.75)) T.
We can then derive the lower P. and upper P probabilities of P = {P= dP/dv = < pl} on 2 in Table 2, using the results Augustin al. 8 Section 4.4]. That P(A) max (Ewea P(w),
EwEA= P(w)} and P(A) min {EwEA P(w), 1 -1-Eye EyEat P(w)) [(w) As we can see from the visual representation of P (the yellow convex region in Figure 2), the probability bounds imposed by the credal set are not stringent, and line with the evidence encapsulated in l,...., l,. Hence, the assumption that ptrue = P5 E P is quite plausible.
P
P
W/
0
0.5
(1)
W1
W2
W3
0.25
0.1
0.6
{ws)
0
0.75
0.5
82
0.4
0.2
0.4
83
0.1
0.8
0.1
[wi,
0.25
4
0.15
0.7
0.15
fuiz W/
0.5
(w1, WS)
0.75
0.25
[ (22, w
0/3
0.75
Table 1.
Table
Figure
3.2 Subjectivist Modeling
Another way of specifying credal set is by taking personalistic (or subjectivist) route 14 70) In this approach, let ( {As} } be a finite collection of subsets of Ω = X' x y. The agent first specifies the lower probability P.s on the power set 2 where S -As i.e., the smallest value that the probability of any subset of S can take on. This can be done, for example, as a result of the empirical distribution, as described below.
'Here, pdlfpmf the Badon-Nikodym derivative of C with respect to sigma-finite dominating measure v. II is easy to see that pl is well-defined plausibility contour function. 'In this case, v the counting measure.
We write the upper likelihood in vector form for notational convenience. denotes the transpose.
5

---

In our framework in which a finite sample of N training sets {D;} is available, we have that {As} = {D and so S = - 1=1 D. Recall that we originally denoted by P. the true data generating process for training set D,, E [L,...,N}: the empirical distribution pam is a (non- parametric) estimation of Pt. On the other hand, recall that we denoted by piree = N+1 the true data generating process for the test set of data DN-1-
The lower probability Pss is defined as follows. For every element (a. v) in S = , D,. we let Ps (f v)3) = m (f(x,, v))) P. v)}) 0). Requiring Ps(((x, min; P (1(2, v)}) is not enough because, if the training data sets do not overlap, we would end up having lower probability O for some singleton that we observed at training time, and hence we would be neglecting some collected evidence. The lower probability L's of all the other non-singleton elements B S is computed according to g Equation (4.6a)], that is,
(B max
Σ (R.))EB
Ps(H(e,W))..1
Σ max PTT (((x,)))) (a.))B
(2)
Numerical example. Suppose for simplicity that X' = (x), so that Q = {x) x 3 = y and let y = {I....: 10]. Suppose N = 2, and let D₁ be collection of three 4's, three 5's, and three 6's. Let also D2 be collection of six 5's and two 6's. Then, S {4,5, 6) and 2 {0. [4]. (5). [6]. {4, 5). (5, 6), {4, 6), {4. 6... In turn, (s((4)) = 1/3, (s((5) = 1/3, and (s((6] = 1/4. By ( this implies that ((4,5)) 5]) = 2/3, Ls (5, 6)) = 2/3, and Ps ([4,6)) max 17/12,
max 1/3+1/4,11 max,e P, ({5))} = max(7/12, 1
3/4) = =7/12. Of course, (0) = 0 and (s(S)=1. =1.
3.2.1 Walley's Natural Extension
Once a lower probability Ps 2 is inferred, it can be (coherently uniquely) extended to lower probability P on the whole sigma-algebra endowed to X' x y through an operator called natural extension 669 (70, Sections 3.1.7-3.1.9]. The resulting extended lower probability is such that P(B) (B), for all E 25, and a lower probability value P((A) is assigned to all the other subsets A of X' x y that are not in S. It is also coherent - in Walley's ' terminology - because, in the behavioral interpretation of probability derived from de Finetti (25 (6), its values cannot be used to construct a bet that would make the agent lose for sure, no matter the outcome of the bet itself.
Once P is obtained, the agent can consider the core of P. M(P) (P: P(A) S P(A), AA C X' x y}. i.c., the collection of all the (countably additive) probabilities that set-wise dominate P. Scholars 20 50 have shown that P ((P) indeed (nonempty) credal set.
3.2.2 Properties of the Core
As shown in Amarante and Maccheroni [ Example 1] and Amarante et al. 6 Examples 6. 7, 8). given a generic credal set Q whose lower envelope is Q...e, a credal set Q for which Q(+) = inf Q(-)
we have that M Q)) 2 Q. From an anformation-thherette perspective. this means that the uncertainty encapsulated in the core of a lower probability Q is larger than that in any credal set whose lower In turn, M(Q) is the largest credal set the agent can build which represents their partial knowledge. In our learning framework, given the available evidence D D,...., N that the agent uses to derive P.s. if the agent is confident that pime (B) 2 (B). for all B E 25, then is natural to assume ptrue = P5+1 E M(P).
envelope is Q L3 15 16 18, 30
4 Generalization Bounds under Credal Uncertainty
Consider credal set P on X x y derived as in Section 3 and assume that collect new evidence in the form of a test set of data + = {(IN+1.1 UN +,11., (,, 9N -N++,,+ }}. To ease notation, in the following we refer to the newly acquired evidence (I1, (1), (Zn, Vn).
An assumption that is common to all the results we present this section is that (2), yn).... (In, Vn) ~ P = pune = PN i.i.d., and P E P. This means that either the new evidence comes from one of the distributions that generated DI,..., DN, or that is at least compat- ible with the credal set we built from past evidence [30]. That is, either P set- dominates the

---

lower probability P of P (as in Sections 3.1 probability P of P (induced, e.g., by a contour function as in Section 3.1.3). This is a rather natural assumption, especially when the stream of training sets we collect pertains similar experiments or tasks. For example, this is the case in Continual Learning (CL), where it is customary to assume task similarity 477, that is, to posit that the oracle distributions pertaining all the tasks of interest are all contained in a TV-ball of radius chosen by the user. A more complete discussion on the relation between our credal approach and CL can be found in Appendix F It is also the case in the healthcare setting, where experts' opinions can be incorporated alongside empirical data (plausible probability distributions) to represent the probability uncertainty, for example, for the prognosis of disease given set of patient constructed encapsulates most of the "potential distributions needed", a number of approaches can be taken including incremental learning; in this approach the AI model learns and updates knowledge incrementally. As a result, the credal set can be continuously updated (via incremental learning) as new data become available. In this direction, learning health systems are being implemented in practice. These are health systems "in which internal data and experience are systematically integrated with external evidence, and that knowledge is put into practice).
3.1.2 and 3.2 or it is set-wise dominated by the upper
[65] 'To make sure that the credal set
We note, in passing, that assuming P E P is less stringent than what is typically done in frequentist statistics, where the data teattaagnnratingg process is assumed to be perfectly captured by the likelihood. We instead posit that the true data generating process for the new evidence available belongs to a credal set P. that was derived by the sample of training sets Di...., DN-
Formal ways of checking whether the assumption P E P holds exist, e.g., by following what Cella and Martin [9 Section 7] and Javanmardi et al. (36] do for credal-set-based conformal prediction
methods, or more general approaches I 22 29 46 36 That being said, deriving PAC-like guarantees on the correct distribution / being an element of the credal set P is a desirable objective
a task that we defer future work. Here, we focus on formally deriving what the consequences are in terms of generalization bounds.
4.1 Realizability and Finite Hypotheses Space
Theorem 4.1. Let (o1,9))...., (Fns Un) Piid, where P is any element of the credal set P. Let the empirical risk minimizer be
h E arg min DEH
Σ (((x)))1).))
(3)
Assume that there exists a realizable hypothesis, that is, h² E H such that Lp(h') = 0, and that the model space H. is finite. Let 1 denote the zero-one loss, and fix any ö E (0. Then, PlLp(h) < e" (8)] 1 6, where (8) is well-defined quantity that depends only 8 and on extreme elements exP of P, i.e., those that cannot be written as a convex combination of one another.
Under the assumptions of finiteness and realizability, Theorem 4.1 ggiess us a tight probabilistic bound for the expected risk (h) of the empirical risk minimizer h. The bound holds for any possible distribution P in the credal set P that generated the stream of training data. A slightly looser bound depending on the diameter of credal set P holds if we calculate Lo(h) in place of (h).).
Corollary 4.2. Retain the assumptions of Theorem 4.1] Denote by Q E. Q + P, a generic distribution in P different from P. Let Axxy denote the space of all distributions over X' x y, and endow with the total variation metric drv. Then, pick any 1) E Tk If the diameter of denoted by dianry (P), is equal to n, we have that
(l)g(h) <c + 1-8, 8,
where e" (0) is the same quantity as in Theorem 4.1
Corollary 4.2 gives us probabilistic bound for the expected risk LQ(h) of the empirical risk minimizer h, calculated with respect to a "wrong" distribution Q - that is, any distribution in P
'If' we want to avoid to formally check the assumption that P E P. we need to show on a case-by-case basis that either the credal set covers neglnegli portion of the distribution class of interest, or that even a small credal set is "good enough" for the analysis at hand.
7

---

different from the one generating the new test set of data DN+1- We can also give a looser but easier to compute- bound for Lh(h).
Corollary 4.3. Under the assumptions of Theorem 4.1 ()) & cuB(6) = 1 1/nllog [H] + +og( and
P[Lp(h)= S EUB(6)] 1 VP Axxy.
Notice how cUB (6) is uniform bound, that is, bound that holds for all possible distributions X' x V' not just those in P. Strictly speaking, this means that we do not need to come up with a credal set to find such bound. Observe, though, that the bound E* (d) in Theorem 4.1 is tighter, as it leverages the training evidence encoded in the credal set P. A synthetic experiment confirming this, and studying other interesting properties of (6) and EUB (6), can be found in Appendix B
By the proof of Theorem 4.1 we have that (p(h) behaves as O |oooglUpaceee Up pa. //2), which is faster than the rate Olog /Hl/n) that we find in Corollary 4.3 since Up B C H Roughly, pes the set of "bad hypotheses" according to pex E exP, That is, those hts for which L larger than 0. A formal definition is given in the proof of Theorem [4.] The modeling effort required by producing credal set P is therefore rewarded with a tighter bound and faster rate.
p- (h) is
Notice that Corollary 4.3 corresponds to Liang (44 Theorem 4]: we obtain classical result as special case of our general theorem.
Let us now allow for distribution drift in the new test set of data DN+1-
Corollary 4.4. Consider a natural number k Let (01,91)--- (IK, P1 i.i.d., and (TK+1, 1k+1)...... (In, ~ P2 ii.d., where Pı, P2 are two generic elements of credal set P. Retain the other assumptions of Theorem 4.1 Then,
PP (hr) (hi) + Lo, (12) S (6) k(n k:)
n²
2 1 1
(4)
where e (0) is the same quantity as in Theorem 4.1 and
k
hy E arg min BEA
Σ
h E arg min NEH
Σ imk+1
n
k
Corollary 4.4 gives us bound similar to the one in Theorem 4.1 when distribution drift is allowed.
The price we pay for it is that it is looser. As result of Corollary 4.3 for looser but casier to compute bound, we can substitute e (6) with (U)
4.2 No Realizability and Finite Hypotheses Space
Let us now relax the realizability assumption in Theorem
Theorem 4.5. Let (1).))..... (En, yn) ~ i.i.d., where P is any element of the credal set P. Assume that the model space H is finite. Let be the zero-one loss, h the empirical risk minimizer, and h" the best theoretical model. Fix any 8 E (0, 1). Then, P[Lp(h)- - Lp(h)) S e**) 2 1 - 8, where exx (8) is well deffnee quantity that depends only on and on the elements of exP.
As we did in Scction 4.1
we can also show that the "wrong" expected risk L that is, the expected risk computed according to Q E P different from the generating the new evidence DN+1 - concentrates around the expected risk Lp(h") evaluated at the best theoretical model h²
Corollary 4.6. Retain the assumptions of Theorem 4.5 Denote by Q P, Q # P, generic distribution in P different from P. Pick any E R> if diamrv (Pp = 77, we have that
((( Lp(h)) < 5 (8) + n] 6.
where e**(8) is the same quantity as Theorem
Similarly to Corollary 4.3 we can give a looser but easier to compute bound for Lp(h) (h*).
'Notice how, if P has finitely many extreme elements - which happens, e.g., if we put p = Conv ({C,) 1 another frequentist way of deriving credal set then Upseep Bra finite union, hence easier to compute.
8

---

Corollary 4.7. Retain the assumptions of Theorem 4.5 Then, (7) S
(vn(o) =
V /2(10g [ 76 +og(( In turn, (h) Lp(h") S Evu( 2 8, for all PE Axxy.
The main difference with respect to Theorem 4.1 that in Theorem 4.5
Lp (p(h) (p(ht) behaves as
O(v /oo B''p//n), which is slower than what we had Theorem 4.1 This is due to the relaxation of the realizabilityy hypothesis. Just like before, though, we have that (V [ B/xrl/n) is faster than the rate O Olvlog [H]/)) that find in Corollary 4.7 This is because B&xP S H.
Roughly, B' is the set of "bad hypotheses" according to at least one pa E That is, those hts for which (h) - L (f)] is larger than 0, for at least one ses A formal definition is given in the proof of Theorem 4.5 Notice that Corollary 4.7 corresponds Liang 44 Theorem 7]: obtain classical result as special case of our more general theorem.
Let now allow for distribution drift. To improve notation clarity, in the following let h denote an element of arg minsEH (h),,, for a distribution E P.
Corollary 4.8. Consider a natural number k < n. Let (#1, vi)...,, (Tk. Uss ~ P1 i.idd, and (IK+1, 1k+1)..... (In: ~ P2 i..dd. where Pi, P2 two generic elements of credal set P. Retain the other assumptions of Theorem 4.5 Then,
- (1(p,) (Lp, (Lr Lp (1p,)) S <E (0) k(n- (V (V
k)) 2 1 8,
where exx (8) is the same quantity as in Theorem 4.5 and hy and he are defined as in Corollary 4.4
Corollary 4.8 tells that the excess risk is also bounded in the presence of distribution drift. The price we pay for allowing distribution shift is a looser bound. As a result of Corollary 4.7] for a looser but easier to compute bound, we can substitute (6") with cus
4.3 No Realizability and Infinite Hypotheses Space
We now relax also the finite hypotheses space assumption in Theorem 4.1
Theorem 4.9. Let (I1, (11,81).... (Ins Yn) ~ i.i.d., where P is any element of credal set P. Let I denote the zero-one loss, h the empirical risk minimizer and h" the best theoretical model. Fix any 8 E (0, 1). Then,
P [r((h) Lp(h*) S (()) 2-55
(5)
for all E P. Here, (d) = Pan + v (2/8)) suppaeesp (AA and
= where (a))
1
Ry, pi (A) Epa..
sup NEw n
azl((s v:),
(6)
In (6, O1,..., Un1,({-1,1)), and A = {(x, y) + 1((x, v), h) E H}.
Rm, pes (A) is slight modification of the classical Rademacher complexity of class A given by
(A) = R,, (AA) = Ep
sup he7t 12
a,l((z, 9:),
where the expectation is taken with respect to the same distribution P from which the data points (11, vr).... (In, yn) are drawn. We consider Re,,=<(A (AA) instead of Rn,r(A) because, since P credal set, P can be written as a convex combination of the extreme elements of P, and suppep (A, = cee pe (A). If the credal set is finitely generated, that is, if it has
*Class A is the loss class, and it the composition of the zero-one loss function with each of the hypotheses in H [44] Page 70J. The Rademacher complexity of AA measures how well the best element of H fits random noise (coming from the o, (9) 44 Page 69].

---

finitely many extreme elements (see footnote 7. then it is easier to compute e e**, we only need to
compute a maximum place of supremum.
As we show in Corollary 4.10 Theorem 4.9 generalizes I Liang Theorem 9). This latter focuses only on the "true" probability Ph+1= = aide X' x V/ while our result holds for all the plausible distributions in credal set P. This grants us to hedge against distribution misspecification.
Let us pause here to add clarification. In real-world applications, we effectively cannot compute Pm (A), since the distribution pime is unknown. While Pm (A) can be approximated via
the empirical Rademacher complexity Rn(A) 44 Equation (219)], whose expected value is indeed RR, Rn,,m= (A), doing so has at least two drawbacks: (1) When the number of data points n = RDD is not "large enough". " this may lead to a poor approximation of the classical bound (Equation (10) in Appendix A (2) The test set of data DN+1 {( y:)) may well be a realization from the
tail of distribution piree = +1 The empirical Rademacher complexity Ra, (A), then, would be a poor approximation of Rn, pire (A). In opposition, while Ra, (AA is more conservative, it can be computed explicitly since we know the credal set P and its extreme elements exP - and it leads to bound that, although looser, holds for all P P.
Corollary 4.10. Retain the assumptions of Theorem 4.9 If is the singleton {P (i.e., all the training datasets D....., Dy are generated by the same distribution as the new test set DN+1). we retrieve Liang 44 Theorem 9].
We then derive a more general version of Corollary 4.6
Corollary 4.11. Retain the assumptions of Theorem 4.9 Denote by Q P. + P. a generic distribution in P different from P. Pick any 17 E R>o; if diamrv (P) = 17, we have that
(h) (phh") (mM(()) (5) + 7l 1-5,
where e*n* (5) is the same quantity as in Theorem 4.9
Finally, we once again allow for distribution drift.
Corollary 4.12. Consider a natural number &o < n. Let (Z1, ... (Ik. Ms) ~ P1 ii.dd, and (IR+11 11++),..... (In: ~ P2 ii.dd, where Pı, P2 are two generic elements of credal set P. Retain the other assumptions of Theorem 4.9 and let Chift = (r + Rn- (A)
2106(216) (Vn 1: + VT). Then,
P[[LP, (fa) Lp, (hp (Lp, Lp, (hx,)) S Chilp 2 6,
where hy and h2 are defined as in Corollary 4.4
Similar considerations as the ones after Corollaries 4.4 and 4.8
hold in this more general case as well.
5 Conclusions
In this paper, we laid the foundations of a more general Statistical Learning Theory (SLT), that we called Credal Learning Theory (CLT). We generalized some of the most important results of classical SLT to allow for drift and misspecification of the data-generating process. We did so by considering sets of probabilities (credal sets), instead of single distributions. The modeling effort needed to elicit credal sets is paid off in terms of the tightness of the resulting bounds.
Limitations. (i) We only consider the zero-one loss in our results (we did to be able to directly build on the classical results in Liang [44] Chapter 3J). (ii) We assume that the true distribution which the elements of the new test set D - are sampled from, belongs to the credal set that we derive at training time.
Future work. In the future, we plan to further our undertaking, for instance by (i) modeling the epistemic uncertainty induced by domain variation through random sets rather than credal sets, (ii) comparing our method with robust learning 23 (iii) extending our results to different losses, and (iv) deriving PiC-like guarantees on the correct distribution P being an element of the credal set P. We also intend to validate our findings on real datasets.
10

---

References
[1] Jayadev Acharya, Constantinos Daskalakis, and Gautam Kamath. Optimal Testing for Prop- erties of Distributions. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https: http:ttts//ppoceeeiiiiee http///ooo.c.iiin..ne.u.ur.....p www.ips.cc.cceeeoooe htce/aapr__fileee http:////appp.iiieeeee////////// paper/2015/fi1e. pdf
[2] Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint
arXiv:1911.00804, 2019.
[3] Russell Almond. Fiducial inference and belief functions. Technical report, University of Washington, 1992.
[4] Lama Alssum, Juan Leon Alcazar, Merey Ramazanova, Chen Zhao, and Bernard Ghanem. Just a glimpse: Rethinking temporal information for video continual learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2474-2482 2023.
[5] Massimiliano Amarante and Fabio Maccheroni. When an event makes a difference. Theory and Decision, 122116 2006.
[6] Massimiliano Amarante, Fabio Maccheroni, Massimo Marinacci, and Luigi Montrucchio. Cores of non-atomic market games. International Journal of Game Theory, 34:399-424, 2006.
[7] Ron Amit, Baruch Epstein, Shay Moran, and Ron Meir. Integral probability metrics pac-bayes bounds. Advances in Neural Information Processing Systems, 35:3123-33 3136, 2022.
[8] Thomas Augustin, Frank P. A. Coolen, Gert de Cooman, and Matthias C. M. Troffaes. Introduc- tion to imprecise probabilities. John Wiley & Sons., West Sussex, England, 2014.
[9] Maria-Florina Balcan and Chris Berlind. Rademacher complexity. Lecture notes for the course CS8803 of the Georgia Institute of Technology, 2011.
[10] Alexis Bellot and Elias Bareinboim. Partial transportabilityy for domain generalization. Available at http:://ggerrreeeeeeeeeffcddcaat 2022.
[11] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Percira, and Jen- nifer Wortman Vaughan. A theory of learning from different domains. Machine Learning, 79: 151-175, 2010.
[12] Gilles Blanchard, Gyemin Lee, and Clayton Scott. Generalizing from several related classifica- tion tasks to new unlabeled sample. Advances in neural information processing systems, 24, 2011.
[13] Michele Caprio and Ruobin Gong. Dynamic precise and imprecise probability kinematics. In Enrique Miranda, Ignacio Montes, Erik Quaeghebeur, and Barbara Vantaggi. editors, Pro- ceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, volume 215 of Proceedings of Machine Learning Research, pages 72-83, PMLR. 11-14 Jul 2023.
[14] Michele Caprio and Sayan Mukherjee. Extended probabilities in statistics. Available at arXiv22111.01050 2021.
[15] Michele Caprio and Sayan Mukherjee. Ergodic theorems for dynamic imprecise probability kinematics. International Journal of Approximate Reasoning, 152:325-343, 2023.
[16] Michele Caprio and Teddy Seidenfeld. Constriction for sets of probabilities. In Enrique Miranda, Ignacio Montes, Erik Quaeghebeur, and Barbara Vantaggi, editors, Proceedings of the Thirteenth International Symposium on Imprecise Probability: Theories and Applications, volume 215 of Proceedings of Machine Learning Research, pages 84-95. PMLR, 11-14 Jul html
2023. URL hwwphhttp//// lttapp:///proccedinge mlr. www.mmlw.........mmmmmmmmmmmm
[17] Michele Caprio, Souradeep Dutta, Kuk Jang, Vivian Lin, Radoslav Ivanov, Oleg Sokolsky, and Insup Lee. Credal Bayesian Deep Learning. arXiv preprint arXiv:2002.006555 2023.
11

---

[18] Michele Caprio, Yusuf Sale. Eyke Hüllermeier, and Insup Lee. A Novel Bayes' Theorem for Upper Probabilities. Fabio Cuzzolin and Maryam Sultana, editors, Epistemic Uncertainty in Artificial Intelligence, pages 1-12, Cham, 2024. Springer Nature Switzerland.
[19] Leonardo Cella and Ryan Martin. Valid inferential models for prediction in supervised learning problems. International Journal of Approximate Reasoning, 150:1-18, 2022.
[20] Simone Cerreia-Vioglio Fabio Maccheroni, and Massimo Marinacci. Ergodic theorems for lower probabilities. Proceedings of the American Mathematical Society, 144:3381-3396, 2015.
[21] Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-22418, 2021.
[22] Siu Lun Chau, Anurag Singh, and Krikamol Muandet. Comparing the uncertain with credal two-sample tests, 2024.
[23] Christian Cianfarani, Arjun Nitin Bhagoji, Vikash Sehwag, Ben Zhao, Heather Zheng, and Prateek Mittal. Understanding robust learning through the lens of representation similarities. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 199225.. Curran Associates, Inc., 2022.
[24] Fabio Cuzzolin. The Geometry of Uncertainty. Artificial Intelligence: Foundations, Theory, and Algorithms. Cham Springer, 2020.
[25] Bruno de Finetti. Theory of Probability, volume New York Wiley, 1974.
[26] Bruno de Finetti. Theory of Probability, volume New York : Wiley, 1975.
[27]
Arthur Pentland Dempster. Upper and lower probabilities induced by a multivalued mapping. The Annals of Mathematical Statistics, 38(2):325 339, 1967.
[28] Aniket Anand Deshmukh, Yunwen Lei, Srinagesh Sharma. Urun Dogan, James W Cutler, and Clayton Scott. A generalization error bound for multi-class domain generalization. arXiv preprint arXiv: /905. 10392, 2019.
[29] Rui Gao, Liyan Xie, Yao Xie, and Huan Xu. Robust hypothesis testing using wasserstein uncertainty sets. In S. Bengio, H. Wallach, H. Larochelle. K. Grauman, N. Aasa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018. URL https: htttt::://ppocceeddngg newrsiddddinnnnnnnn...........ee neurips. www.wewwww....pp...reeea. htc/paper_ffles/pppeee
22114111.00622222288777226666662 .pf
[30] Ruobin Gong and Xiao-Li Meng. Judicious judgment meets unsettling updating: dilation, sure loss, and Simpson's 's paradox. Statistical Science, 36(2):169-190 2021.
[31] Ishaan Gulrajani and David Lopez-Paz. In search lost domain generalization. arXiv preprint arXiv:200.001434, 2020.
[32] Jan Hannig. On generalized fiducial inference. Statistica Sinica, pages 491-544, 2009.
[33] Shoubo Hu, Kun Zhang, Zhitang Chen, and Laiwan Chan. Domain generalization via multido- main discriminant analysis. In Uncertainty in Artificial Intelligence, pages 292-302. PMLR, 2020.
[34] Peter J. Huber and Elvezio M. Ronchetti. Robust statistics. Wiley Series in Probability and Statistics. Hoboken, New Jersey Wiley, 2nd edition, 2009.
[35] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018.
[36] Alireza Javanmardi, David Stutz, and Eyke Hüllermeier. Conformalized credal set predictors. arXiv preprint 202 2024.
12

---

[37] Kishaan Jeeveswaran, Elahe Arani, and Bahram Zonooz. Gradual divergence for seamless adaptation: A novel domain incremental learning method. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the s International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 21486-21501. PM 21-27 Jul 2024. URL httpp::/prooceedngg... mlr. wtwpppp//./eeppppppppppp jeeeesaareeeessaaaaa2222a2. http://a.........eeeeeee//eeeeee
html
[38] Praneeth Kacham and David Woodruff. Sketching algorithms and lower bounds for ridge regression. In International Conference on Machine Learning, pages 10539-10556. PMLR, 2022.
[39] David G. Kendall. Foundations of theory of random sets. In E. Harding and D. G. Kendall, editors, Stochastic Geomelry, pages 322-376. Wiley, London, 1974.
[40] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xic, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[41] Isaac Levi. The Enterprise of Knowledge. London, UK: MIT Press, 1980.
[42] Da Li, Henry Gouk, and Timothy Hospedales. Finding lost dg: Explaining domain generaliza- tion via model complexity. arXiv preprint arXiv:2202.00563, 2022.
[43] Shaojie Li and Yong Liu. High probability generalization bounds with fast rates for minimax problems. In International Conference on Learning Representations, 2021.
[44] Perey Liang. Statistical learning theory. Lecture notes for the course CS229T/STAT23! of Stanford University, 2016.
[45] Julian Lienen and Eyke Hüllermeier. Credal self-supervised learning. In M. Ranzato,
A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 14370-14388 14382. Curran Associates, Inc., 2021. URL http:ss: wewrrrseerrppp. ct/ccc//paaa cc/paper.r.. fepper_fille wtt.ffelllpppee//22222 pddf
[46] Xing Liu and Trançois-Xavices Briol. On the Robustness of Kernel Goodness-of-Fit Tests. Available 4rXiv:2408.05854, 2024.
[47] Pengyuan Lu, Michele Caprio, Eric Eaton, and Insup Lee. Zero-shot Model Generation for Task Trade-offs in Continual Learning. arXiv preprint arXiv:2305. /4782, 2024.
[48] Shireen Kudukkil Manchingal and Fabio Cuzzolin. Epistemic deep learning. Available at 2xxiv:2206007609 2022.
[49] Shireen Kudukkil Manchingal, Muhammad Mubashar, Kaizheng Wang, Keivan Shariatmadar, and Fabio Cuzzolin. Random-Set Convolutional Neural Network (RS-CNN) for Epistemic Deep Learning. Available at arxiv: 2023.
[50] Massimo Marinacci and Luigi Montrucchio. Introduction to the mathematics of ambiguity. In Itzhak Gilboa, editor, Uncertainty economic theory: collection of essays in honor of David Schmeidler's 's 65th birthday. London Routledge, 2004.
[51] Radu Marinescu, Debarun Bhattacharjya, Junkyu Lee, Fabio Cozman, and Alexander Gray. Credal marginal map. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 47804-47815, Curran Associates, Inc., 2023. URL https::t//// httttppp://proceedinngss nwwrins. wwwpripps.ccppppefffleele Paper- .dd]]
[52] Georges Matheron. Random sets and integral geometry. Wiley Series in Probability and Mathematical Statisties, New York, 1975.
[53] Denis D. Maua, Cassio Polpo de Campos, Alessio Benavoli, and Alessandro Antonucci. On the complexity of strong and epistemic credal networks, 2013. URLphttps: / http://aawwv...o/////a////
1309. 6845
13

---

[54] Denis Deratani Mauá and Fabio Gagliardi Cozman. Thirty years of credal networks: Spec- ification, algorithms and complexity. International Journal of Approximate Reasoning, 126: 133-157, 2020. ISSN 0888-613X. doi: hutp://dbo..gggll..00fffiii22222
URL
https: scwencedirect...... ott.a/accca..c..........aaaaaaaa
[55] Ilya S Molchanov. Theory of random sets, volume 19. Springer, 2005.
[56] Thomas Mortier, Viktor Bengs, Eyke Hüllermeier, Stijn Luca, and Willem Waegeman. On the calibration of probabilistic classifier sets. In Francisco Ruiz, Jennifer Dy, and Jan-Willem de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 8857-88888888 -8700 PMLR, 25-27 Apr 2023. URL http::tt///ooceedinngg
mlr. prwss/ grpess/v200 wttt///000mmmmme... html
[57] Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. In International conference on machine learning, pages 10-18. PMLR, 2013.
[58] Hung Nguyen. On random sets and belief functions. Journal of Mathematical Analysis and Applications, 65:531-542, 1978.
[59] Fabrizio J. Piva, Daan de Geus, and Gijs Dubbelman. Empirical generalization study: Unsuper- vised domain adaptation vs. domain generalization methods for semantic segmentation in the wild. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 499-508, January 2023.
[60] levgen Redko, Emilie Morvant, Amaury Habrard, Mare Sebban, and Younès Bennani. A survey on domain adaptation theory: learning bounds and theoretical guarantees. arXiv preprint arXiv:2004. 18829, 2020.
[61] Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. interpolation and extrapolation in domain generalization. In International Conference on Artificial Intelligence and Statistics, pages 2641-2657. PMLR, 2022.
An online learning approach to
[62] Yusuf Sale, Viktor Bengs, Michele Caprio, and Eyke Hüllermeier. Seccond-Orde Uncertainty Quantification: A Distance-Based Approach. Available at (arxiv:2312.0099)) 2023.
[63] Yusuf Sale, Michele Caprio, and Eyke Hüllermeier, Is the volume of credal set good measure for epistemic uncertainty? In Robin Evans and llya Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelliencee volume 216 of Proceedings of Machine Learning Research, pages 1795-1804. PMLR, 31 Jul-04 Aug 2023.
[64] Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[65] Silvia Seoni, Vicnesh Jahmunah, Massimo Salvi, Prabal Datta Barua, Filippo Molinari, and U. Rajendra Acharya. Application of uncertainty quantification to artificial intelligence in healthcare: A review of last decade (2013-2023). Computers in Biology and Medicine, 165: 107441, 2023. ISSN 0010-4825, doi: https://doi.co http://....100/oooooooooooee 2023, 107441.
202.4..1. URL
bttps:://wa /ww. www.sscieiieeciieeee www.:/eiiirettttcccciieeeeeeeeee
[66] Glenn Shafer. A mathematical theory of evidence, volume 42. Princcton university press, 1976.
[67] Paras Sheth, Raha Moraffah, K Selçuk Candan, Adrienne Raglin, and Huan Liu. Domain generalization-a causal perspective. arXiv preprint arXiv:2209.1511 2022.
[68] Anurag Singh, Siu Lun Chau, Shahine Bouabid, and Krikamol Muandet. Domain gener- alisation via imprecise learning. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceed- ings of the 41si International Conference on Machine Learning, volume 235 of Proceed- ings of Machine Learning Research, pages 35544-45570 PMLR, 21-27 Jul 2024. URL. https://proceedding mlr, press/ v235 /inn////nnh24....m html
14

---

[69] Matthias C.M. Troffaes and Gert de Cooman. Lower Previsions. Chichester, United Kingdom: John Wiley and Sons, 2014.
[70] Peter Walley, Statistical Reasoning with Imprecise Probabilitiess volume 42 of Monographs on Statistics and Applied Probability. London Chapman and Hall, 1991.
[71] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generaliza- tion. IEEE Transactions on Knowledge and Data Engineering, 2022.
[72] Kaizheng Wang, Fabio Cuzzolin, Keivan Shariatmadar, David Moens, and Hans Hallez. Credal wrapper of model averaging for uncertainty estimation on out-of-distribution detection, 2024.
URL htttttts///xxiii wtt.aawwww....a.ao/. wttp:/og///b/aaoeoeeoo.e http:/aab2240..150000
[73] Kaizheng Wang, Keivan Shariatmadar, Shireen Kudukkil Manchingal, Fabio Cuzzolin, David Moens, and Hans Hallez. CreINNs: Credal-Set Interval Neural Networks for Uncertainty Estimation Classification Tasks. Available at arxiv:2401 1,0043,, 2024.
[74] Larry A. Wasserman and Joseph B. Kadane. Bayes' theorem for Choquet capacities. The Annals of Statistics, 18(3):1328 1339, 1990.
[75] Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, and Liwei Wang. Towards a theoretical framework of out-of-distribttion generalization. Advances in Neural Information Processing Systems, 01--33519-23535 2021.
[76] Yiwen Ye, Yutong Xie, Jianpeng Zhang, Ziyang Chen, Qi Wu, and Yong Xia. Continual self-supervised learning: Towards universal multi-modal medical data representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11114-11 11244 2024.
[77] Jiazuo Yu, Yunzhi Zhuge. Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-expert adapters. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23219-23230, 2024.
[78] Jianchun Zhang and Chuanhai Liu. Dempster-Shafer inference with weak beliefs. Statistica Sinica, 31:475-494 2011.
[79] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
A Proofs
Proof of Theorem 4.7 The proof builds on that of Liang 44 Theorem 4]. Fix any e >> 0, and any
P E P. Assume that the training dataset given by i.i.d. draws from P. We want to bound the probability that (h) > e. Define Bp = (h E H: Lp(h) > e). It is the set of "bad hypotheses" according to distribution P. As a consequence, we can write (h) > c] Plh E Bp). Recall that the empirical risk of the empirical risk minimizer is O, that is, [(h) = o So if the empirical risk minimizer is bad hypothesis according to P. that is, if h Br, then some bad hypothesis (according to P) must have zero empirical risk. In turn,
PÍh E Br] P(3h E Br i.(h) =).
Let us bound (P()(()) = 0] for fixed h E Bp. Given our choice of zero-one loss, on each example, hypothesis h does not with probability 1 Lp(h). Since the training examples i.i.d. and (h(h) > e for all h E Bp, then
(L((()) 0] = L < (1 e)" s exp(-en))
(7)
Indeed, at least L(h²) Lr(h")
15

---

Applying the union bound, we obtain
P/ah Bp L(h) =
Σ Pl 0]
BEBP
Bp exp(-cn)
< UpEp Bpl exp( (n)
Upp Bpul exp(
8.
The penultimate equality comes from P being a credal set, by the Bauer Maximum Principle and the linearity of the expectation operator. Rearranging the terms get
(0) =E= log Upreep Bpde log(1/8)
n
In turn, this implies that (h) S { 2 1- 5.
Proof of Corollary 4.2 Let A₁ = {(,,, y) E X y + h(x)) Axxy. Notice that Lp(h) Ep + M(x)) = P(Ag),
(h( EQq (TV # Mxx)) = Q(A;).
Recall that diamyv (P) = supp,qeP sup AEAXXY P(A) - Q(4)). As a consequence, we have that (h) (h) + (q. where (o a quantity in [- depending on Q. Given our assumption on the diameter, then, (h) + SQ $ (h) + so Lo (h) = (h)). In turn this implies that
[h(h) - 1 [6] 2
[p(h) < c0)
The proof is concluded by noting that (h) - n c* (0) P()o(A)) S (0) + n), and that (lLr(h) e" (8)) 2 - by Theorem 4.1
Proof of Corollary 4.3 Since Upa S H. it is immediate to see that c=(8) < <EUB(O) EOUB()). In turn,
sup Lh(h) < EUR cun( 2 1
PEP
or equivalently, (h) S cuB cuB())) 2 for all P.
Proof of Corollary
4.4 From Theorem 4.1 we have that
log Upacesp Bpal log log(1/5)
Lrg (fn)
and that
P (hz) x log Upa Bpal + (og(1/5)
1 11-6 8.
k The result, then, is an immediate consequence the additivity of the expectation operator and of probability P.
Proof of Theorem 4.5 The proof builds on that of Liang 44 Theorem 7]. Fix any 6 0, and any P E P. Assume that the training dataset is given by i..dd draws from P. By Liang [44] Equations (158) and (186)), we have that
P(Lr(h)- -.(hh)))
< sup WEH
i(h) - LP(h))
5 2)
(8)
|7
2 2cxp
-2n
2
= 8(c).
16

---

Notice though, that we can improve on this bound, since we know that P E P, credal set. Let By = {h E H [E(hh) Lr(h)) > c/2) be the set of "bad hypotheses" according to P. Then, it is
immediate to see that
[(h) - Lp(h)
sup - -(h)-Lp(hh)
sup hEA
he
Notice though that we do not know P: we only know it belongs to P. Hence, we need to consider the set 33p of bad hypotheses according to all the elements of P, that is, B3p = {h H 3P e P. [() - Lp(h)] > c/2] Uppp Since P is credal set. by the Bauer Maximum Principle and the linearity of the expectation operator we have that B'p = B&e {h H. : 3Px E
exP, [h( Lp(h)l c/2} =
Up pe B'p. Hence, obtain
[hh)) Lp(h) sup bEB
[(hh) Lp(h)
sup hEH
In turn
implies that
P[Lp(h)-Lp(M) P[Lp(h) - Lp(ht) > 2] P
L(h) Lp(h))
sup
>
2
he BEB
( 2
|B'xpl 2exp
-2n
2
6ox.
Rearranging, we obtain
2 log Baapl + log
2
daP
(9)
so if is fixed, we can write = c** (5). turn, this implies that P(Lp(h) - Lp(h") > c**(8) or equivalently, (h) - Lp(hh) S (6) 2 6.
Proof of Corollary 4.6 The first part of the proof is very similar to that of Corollary assumption on the diameter, have that (h)(h) Ir(hh) (p((h) (Q T,(hh) where to is
Given our
quantity [-n, v| depending on Q. Then, Lp(h)+ -Cp S Lp(h)+ -1- Lp (p*) so (h)-
7 Lp(hh) < Lp(h) Lp(h)). In turn this implies that PP LQ(h) - 7 Lr(h") 5
c*) = 2
s = The proof is concluded by noting that (h) - 11 Lp(h1) S e0) = (h) Lp(h*) *** (0) + n). and that P()p(h) Lp(h*) S e** (8)] > 1 d by
[h(h) - s
Theorem
Proof of Comolary 4. turn,
7. Since Upa B'pa C H. it is immediate to see that e*' S c (0). In
P
sup PEP
Lp(h)) - Lp(ht)) S (0) -
or equivalently, (h) Lp(hh)) < (b) 2 - 8, for all PEP.
Proof of Corollary 4.8 From Theorem 4.5 we have that
(log B'xpl log (2))
PP Lp, (hi) - Lp, (lip)) s
2 11
k
and that
1 (log B''pl log (2))
P Lz, Lp, (lip)) <
2 21
k
The result, then, is an immediate consequence of the additivity of the expectation operator and of probability P.
17

---

Proof of Theorem 4.9 Fix any 8 E (0, 1). In Liang [44 Theorem 9], the author shows that for fixed probability measure on X' x V, we have that
(2/6)
Lp (h) Lp(h") S $4...4mm, P(A)
(10)
holds with probability at least - ô, where Rx,r is defined analogously as in (6) The result in then, follows from P being a credal set, and the expectation being a linear operator.
S
Proof of Corollary 4.10
Immediate from Theorem 4.9
Proof of Corollary
The proof is very similar to that of Corollary 4.6
Proof of Corollary
4.
12
From Theorem 4.9 we have that
2 (2/(2
P
Lp. (h1) - Lp. (ht) - 4R (A) +
2 1
k
and that
2 (2/5)
P
Log (ha (hz) M (A)
1 8.
n
The result, then, is an immediate consequence of the additivity of the expectation operator and of probability P.
B Synthetic Experiments on Theorems 4.1 and 4.5
In this section, we perform synthetic experiments to show that the bounds we find in Theorems
and 4.5 are indeed tighter than the classical SLT ones reported in Corollaries 4.3 and 4.7 respectively, In recent literature, studies by Amit et al. 77. Kacham and Woodruff [38] Li and Liu 43 have conducted synthetic experiments in similar manner. These works are mainly theoretical in nature, but they also acknowledge the importance of experimental validation with preliminary analysis.
Experiment 1: Let the available training sets be D1, D2, D3. Assume, for simplicity, that X' x y = {x} x RR ~ R. Suppose that we specified the likelihood pdfs & = (-5,, 1), & = N NO 1). and N (5, 1). Call L, L2, L. their respective probability measures, and derive the credal set P as we did in footnote 7 That is, let P = Conv (zn) We determine the credal set in this way because it then easy find its extreme elements exP. Indeed, is immediate notice that exP = 1 Let now DN+1 = D4 be collection of n samples from pine = L2 E P. The hypotheses space H defined as a finite set of simple binary classifiers containing at least one realizable hypothesis, and we consider the zero-one loss as we did in the main portion of the paper.
We need find ce
B Be -Z_1(A (h H Lc. (h) c), where depends on 5 as in the proof of Theorem 4.1 That is, we want those h's for which the expected loss according to Li or L2 or La larger than e. They the collection of "bad hypotheses" according at least one of the extreme elements of our credal set. Recall that
(6) log Upp Bpal + log(1/6)
n
the bound we found in Theorem and that
CuB (8) = log [7| + log(1/5)
is the classical SLT bound, that we reported in Corollary
As we can see from Table B. our bound c* (²)) improves on the classical SLT one (O) Table B. also tells us that bound e (o) tighter than éum (6) when the sample size n [D4] is small, and then Dal increases. This same pattern is observed
(6) becomes progressively closer to FUn (6) as
when the extrema of the credal set are closer to each other. Indeed, in Table B.2 we repeat the experiment and choose extrema of P three measures whose pdf's 's are three Normals (-0.1, 1).
18

---

(o0, 1), and (0.1.. 1) that is, when n = D Is low, Credal Learning Theory able to leverage the evidence encoded in the credal set, and hence to derive tighter bound than classical Statistical Learning Theory. When the sample size is large, that is, when n = P4l is high, the classical bound (un(6) itself very small. This is because the amount of evidence available is large, and so 1/n -i>>> [[ y1), h) well approximates Jx ((x,, v), Pime (d(z. v)). In turn, since (U)(d) is already very small, then the CLT bound c* (6) we derive cannot improve greatly on it. Hence, their values are close together, despite e* (6) being slightly tighter. The code for this experiment available upon request.
10 The reason for this behavior is the following. With few available samples,
# Samples n
<*())
(UB
Eexp pm
H
Realizability
10
0.74500
0.76009
86
100
Yes
100
0.07560
0.07600
96
100
Yes
200
0.03785
0.03800
97
100
Yes
300
0.02526
0.02533
98
100
Yes
400
0.01897
0.01900
99
100
Yes
500
0.01516
0.01520
98
100
Yes
Table B.I: Results of experimental evaluation of our bound tightness. Here the hypotheses space is such that [H]
100, and 8 0.05. The likelihood pdfs l (N(-5, 1), la = N'0 1), and
lg = W5, 1).
# Samples n
<*(o)
(UB(O)
Upx Ppa
H
Realizability
10
0.73777
0.76009
80
100
Yes
100
0.07549
0.07600
95
100
Yes
200
0.03780
0.03800
96
100
Yes
300
0.02520
0.02533
96
100
Yes
400
0.01892
0.01900
97
100
Yes
500
0.01514
0.01520
97
100
Yes
Table B.2: Results of experimental evaluation of our bound tightness. Here the hypotheses space
such that [H| 100, and 8 0.05. The likelihood pdfs 4 (V(-0.1, 1), l2 NO 1). and
f3 = ().
We conducted another synthetic experiment to show that the empirical risk for given distribution is upper bounded by the traditional SLT bound of Corollary [.3 This a sanity check to see whether the environment we used in Experiment 1 is valid one to check our results.
Experiment 2:
For the experiment, we selected a standard Gaussian distribution (0 1) (mean O, standard deviation 1) to generate data. Similarly to Experiment (i) the hypotheses space H is defined as finite set of simple binary classifiers containing at least one realizable hypothesis, and (ii) we assume a zero-one loss function. The latter is used to evaluate the performance of the classifiers. For each run, generated a training set and a test set from the standard Gaussian distribution. At training time, for each hypothesis h in H. we calculate the empirical risk on the training using the zero-one loss and identify the hypothesis h that minimizes such risk (the empirical risk minimizer). At test time, we compute the empirical risk (h) of h on the test set as shown Table B.3
Training Samples
Test Samples
Lp(h) (Test)
e*()) (Test)
EUB(O) (Test)
1000
500
0.000
0.01514
0.01520
1500
1000
0.000
0.00757
0.00760
2000
1500
0.000
0.00506
0.00506
Table B.3: Results of experimental evaluation. The hypotheses space is such that [ = 100, and
5 = 0.05.
We calculate the upper bound cus (6) on the empirical risk based on Corollary 4.3 which is function of the number of hypotheses in H. the value of 8, and the number of training samples. This the
Of course, they are much closer to each other than the Normals A Ao,, 1). and (5 1). e.g. in the Total Variation metrie.
Here P denotes the probability measure whose pdf is the standard Normal density.
19

---

classic SLT bound. The experiment is run 1000 times with specified numbers of training and test samples, and 6 value 0.05 check whether the condition (empirical risk the test is upper bounded by the theoretical bound) is satisfied in any of the 1000 trials. The experimental results in Table [B.3 validate the classical SLT bound by repeatedly testing it on randomly generated data. The code for this experiment is also available upon request.
Experiment 3: In this, we aim to empirically validate Theorem 4.5 which addresses the behavior of the empirical risk minimizer in the presence of a finite hypothesis space and no realizability. We generate synthetic data from Gaussian distributions (with the same parameters as in Experiment 1), with added uniform noise to ensure no realizability. meaning that no hypothesis can perfectly predict the labels. The hypotheses space H. is defined set of threshold based classifiers parameterized by 0. For the experiment, we generate training samples Dj. D2. D3 and test sample D4 from Gaussian distributions with added noise. Labels are created based on the samples, with noise introduced to flip labels randomly, ensuring that no hypothesis in H. can achieve zero loss. We identify the empirical risk minimizer h using the combined training samples D1, D2, D3. We calculate the empirical risk Lp (h) using the test data D4. The theoretical risk (h*) is assumed to be the risk of a perfect classifier. We compute the theoretical bound E (6) and verify whether the difference Lp(h) Lp(h*) is within this bound. The results show that the empirical risk of the empirical risk minimizer is within the bound e** (6) of the best theoretical model h". The difference (p(h) Lp(h*) also satisfies the condition
(h(h) - Lp(h*) S (6).
This validates empirically (in synthetic environment) Theorem 4.5 showing that even under no realizability, the empirical risk minimizer's performance is close to the theoretical best within computable bound. The experimental results are presented in Table B.4 where we also show (i) that
(0) < (ō) = V 2(log (((+)ou(#)) from Corollary 4.7 always holds; and (ii) that by foregoing see, e*) is slightly larger than
realizability, obtain slightly looser bound. Indeed, c* (5) from Table B.I for all the sample size values n that we consider. The code for this experiment also available upon request.
# Samples n
Lp(h)
Lp(h*)
Lp(h) - Lp(h")
Ea)(6)
EUS
10
0.30000
0.10000
0.19999
0.76009
0.84877
100
0.14000
0.10000
0.04000
0.07600
0.26840
200
0.10500
0.10000
0.00499
0.03800
0.18979
300
0.11333
0.10000
0.01333
0.02533
0.15496
400
0.11750
0.10000
0.01749
0.01900
0.13420
500
0.10400
0.10000
0.00399
0.01520
0,12003
Table B.4: Results of experimental evaluation of Theorem 4.5. Here the hypotheses space is such that
|H| 100, 8 = 0.05 and noise level 0.1. The likelihood pdfs f - 1), l2 (0, 1), and
N 1). Let us remark that this experiment we forego the assumption of realizability, that
[h(hh) - (h)) S (8) for every sample size we tested on, and that (**(8) S (ō) for every sample size we tested on.
C A Simple Numerical Example for the chhee-CCntain e-Contamination Model of Section 3.1.1
Just like in Section 1.3 let Ω that we observed four training samples Dı,.... D4 and that we specified the likelihoods Li...., Ls as in the following Table.
(wi ي22 Ws). where wy Uz;, E 21, Suppose also
[w)
(wz)
{wa
L1
0.1
0.6
L
0.4
0.2
0.4
L3
0.1
0.8
0.1
C4
0.15
0.7
0.15
20

---

the credal set P = Conv(U Li). Then, suppose that 0.2, 6 003, = 0.1, and 0.25, so that 0.8L, 0.2Q, AQ E An). L2 (P:
P
0.722 + 0.3Q, VQ An). P
0.9L3 + 0.1Q. vQ & An): and L4 = [P:P = 0.7554 + 0.25Q. VQ E Anj.
J. By Wasserman and Kadane [74 Example 3), we know that, we put P [ L)) the following holds
P(A) min (l.... (1))))((() AA# Ω A Ω
and
AA + 0 m (1...44) (1))(()) + c
P(A)
0
Simple calculations, then, give us the following values
P
P
W1
0.09
0.58
[22)
0.08
0.82
(w3)
0.09
0.68
[wi, W2)
0.32
0.91
(uz, Wl3
0.42
0.91
[wi, W3)
0.18
0.92
(W) W2, wa)
1
As we can see, this example too, the probability bounds imposed by the credal set are not too stringent, and in line with the evidence encapsulated in Li...., C1. Hence, the assumption that pane = P & P is very plausible. For a visual representation the yellow convex region in the next figure; it is very similar to the convex region in Figure 2. This is unsurprising since the evidence used to derive the credal set in Section 3.1.3 the same that use to elicit P here.
61
012
0.99
0008
03g
D A Fiducial Approach to Objectivist Modeling
An alternative objectivist approach the ones presented in Section 3.1 proposed by Dempster and Almond 3. is based on fiducial inference B32 Consider parametric model, i.e., family of conditional probability distributions of the data (f(w w E 22 E θ), where is, again, the observation space and θ is parameter space. If the parametric (sampling) model is supplemented by a suitably designed auxiliary equation i = a(O, u). where u is a "pivot" variable of known a-priori distribution μ. one obtains random set 1 mapping pivot values to subsets
I(u)- {ww 0) E Z θ a(0, w)}
of Ω x O This, in turn, induces a belief function on the product space Ω x defined as
Bel(A) ()
Σ p(u), AcΩx
This can be finally be marginalized to the data space Ω = X x y to generate belief function there. This approach was further extended by Martin, Zhang, and Liu, who used "predictive" random set to express uncertainty on the pivot variable itself, leading to weak belief inference technique (78).
21

---

In our framework, in which a finite sample of N training sets {D.) is available, we can derive N many random sets I before. i [1..... .N.. and consider the N belief functions Bel, on f2 x Q they induce. Then, we can compute their marginalization Bel:je on the data space Ω = X 3/ and compute the minimum Bella = (1...g Bel: 22 It is easy to see that Bella is itself well-defined belief function. Finally, our credal set given by (2 as in Section 3.1.2
E Related Work on the Computational Complexity of Credal Sets
In this section, we discuss some of the analyses existing in the literature of the computational complexity specific to the use of credal sets, particularly in the context of graphical models and probabilistic inference. Such approaches can be implemented for large datasets, but they often require approximation techniques to be computationally feasible [45 53 54]. Despite this, credal set approaches can be implemented for large datasets using techniques like parallel processing, distributed computing, and efficient data structures. Similar to Deep Learning -ased approaches, utilization of gegh-performance computing resources, algorithm optimization, and the computational challenges can be effectively managed. Recent advancements demonstrate the practicality of these approaches. For instance, Credal-Set Interval Neural Networks (CreINNs) have shown significant improvements in inference time over variational Bayesian neural networks [73]. Thus, while the computational demands are comparable to those of deep learning- based methods, the robustness and flexibility of credal sets, as demonstrated in recent research, make them practical and valuable approach [5] 72
adaptations,
F On the Relation Between Credal Sets and Continual Learning
The credal approach in this paper is closely linked to Continual Learning applications, which emphasize the need to handle diverse and sequential datasets to achieve robust and generalizable models. Recent works in continual learning have demonstrated the practical applications and benefits of using multi-dataset setup. For instance, Jeeveswaran et al. (37 introduce a novel method for domain incremental learning, leveraging multiple datasets to adapt seamlessly across different tasks. Another example is Yu et al. [77), who propose a parameter-efficient continual learning framework that dynamically expands pre-trained CLIP model through Mixture-of Mixture-of-Experts (MoE) adapters in response to new tasks. Ye et al. [76) address the challenges of multi-modal medical data representation learning through a continual self sulf-supervieed learning approach. These examples from recent studies demonstrate the practical applications and benefits of using a matti-dataset setup in a continual learning framework. Furthermore, some techniques learning without relying on a specific temporal order. For example, Alssum et al. [4] present a replay mechanism based single frames, arguing that video diversity is more crucial than temporal information under extreme memory constraints. By storing individual frames rather than contiguous sequences, they can maintain higher diversity in the replay memory, which leads to better performance in continual learning scenarios.
multi-dataset setup in continual
22

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1 (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims presented in both the abstract and introduction align well with the contributions and scope of the paper. The abstract succinetly outlines the theoretical framework and the bounds introduced in the paper, laying the foundations for a 'credal' learning theory to model variability in data- mata-generating distributions. Similarly, the introduc- tion provides comprehensive account of the relevant background, the motivation behind the research, and the significance of the proposed learning framework. Throughout the manuscript, there is consistent support for the claims made The results from synthetic experiments conducted, and the theoretical results in the paper, provide detailed insights into the novel learning framework and reinforce the claims by demonstrating the effectiveness and relevance of the proposed framework.
Justification:
abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
23

---

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The main limitations of this paper are two. The first that we only consider the zero-one loss in our results. The other is that we assume that the true distribution which the elements of the new test set Dx+1 are sampled from, belongs to the credal set we derive at training time.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- lant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
The assumptions are thoroughly discussed in the main body of the paper.
Proofs are provided in the Appendix section. We have taken great care to provide complete and correct proofs, structured in a logical and transparent manner, for each theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
24

---

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes] We conducted three synthetic experiments. One serves as a sanity check, while the other demonstrates that the bound established in Theorem 4. is tighter than the classical bound found in Statistical Learning Theory (SLT). The third one validates empirically Theorem 4.5 showing that even under no realizabilitty, the empirical risk minimizer's performance is close to the theoretical best within a computable bound.
Justification: Drawing inspiration from recent literature, including studies by Kacham and Woodruff [38], Amit et al. 77.. Li and Liu 43 we have conducted synthetic experiments in a similar manner. While these works are primarily theoretical, they recognize the importance of experimental validation through preliminary analysis.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The experiments are synthetic and extremely easy to reproduce. Also, the experiments do not require any special libraries or large-scale real-world datasets.
25

---

Guidelines:
necessary for conveying the reliability of The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (httpss:/nnips.ce)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ssiici for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have clearly mentioned all the details of our synthetic experiments. In the Supplementary Section B we provide comprehensive details of our training data and hyperparameter values. Tables B. and [B.2] present the results of our bound tightness experiments alongside the hyperparameter values used. Additionally, Table B.3 displays the outcomes of our sanity check experiment. T.b presentedd the results the experimental evaluation of Theorem 4.5 Together, the details of these experiments enhance the support for our theoretical results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: In our experiments, we primarily rely on sampling from known distributions and calculating the theoretical bounds as detailed in the main text of the paper. Given the theoretical nature of our analysis and the use of these known distributions, traditional error bars measures of statistical significance our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: There no need to discuss the allocation of computer resources, as the
synthetic experiments we conducted can be performed on any standard computer.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neuripp: hwtpphwwwwppppppopcppeeppe.epeee
Answer: [Yes]
Justification: The experiments our work are entirely synthetic. This means that they do not involve real individuals or sensitive data that could raise ethical concerns. The synthetic nature of our experiments ensures that they inherently avoid issues such as privacy breaches misuse of personal data.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
27

---

This paper presents work whose goal is to advance the theoretical foundations of Machine Learning. There are many potential societal consequences our work, none of which we feel must be specifically highlighted here.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for the responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our study does not utilize or produce pretrained models, image generators, or datasets that are derived from scraping, which are typically associated with high risks for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
The above not applicable to our research, as our study exclusively employs synthetic data and eeff-generated models without relying on external assets, special libraries, or models. Therefore, there are no third-party assets involved that would require attribution adherence to licensing terms.
Justification:
28

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The concern regarding the documentation of new assets is not applicable to our work. This is because our research does not introduce any new assets such as datasets, models, or software tools that would require documentation accompanying files. We have focused solely on synthetic experiments, which do not involve creating or releasing new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This is not applicable our case as our study exclusively involves synthetic experiments and does not engage with crowdsourcing methods human subjects. Therefore, there are no participant instructions or compensation issues to report.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
29

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Our research strictly involves synthetic experiments and does not include human participants. Consequently, the need for IRB review or any equivalent ethical oversight does not arise in the context of our research methodology.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "gwUj9sGNTI", "reviewer": "Reviewer_hWdx", "summary": "The paper develops a so called credal learning theory that uses convex sets of probability distributions (also known as credal sets) to model the uncertainty of the data-generating distribution. As in the classical statistical learning theory, the paper derives new theoretical bounds on the risk of the models learned from a collection of datasets instead of a single dataset. These datasets do not necessarily correspond to the same distribution. The new results (i.e., bounds) can be viewed as a generalisation of the classical results.", "questions": "- In my understanding, the results derived in the paper rely on the availability of the credal set that is supposed to contain the true data generating process. How is that credal set elicited?\n\n- In principle, the credal set can be arbitrarily large in the sense that it can have arbitrarily many extreme points. Is there a bound on its size?", "limitations": "The limitations of the proposed approach are clearly discussed in the paper.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper looks at a core problem in machine learning and the credal sets based approach to bounding the expected risk of a machine learning model appears to be quite novel.", "weaknesses": "- In my opinion, the presentation is quite dense, the notation appears to be quite heavy in places and the paper is not easy to follow. I think it's important to illustrate some of the key concepts introduced in sections 3 and 4 with some examples. Tables 1 and 2 as well as Figure 2 are good but they seem somewhat disconnected from the rest of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*In my opinion, the presentation is quite dense, the notation appears to be quite heavy in places and the paper is not easy to follow. I think it's important to illustrate some of the key concepts introduced in sections 3 and 4 with some examples. Tables 1 and 2 as well as Figure 2 are good but they seem somewhat disconnected from the rest of the paper.*\n\nWe thank the reviewer for their insights. In the updated version, we will strive for more clarity. In the present version, we already have an example pertaining to Section 3.1.3, and we will try to bring to the main body of the paper the example pertaining to Section 3.1.1 that is currently confined in Appendix C (it was relegated there because of the 9 pages limitation). \nIn fact, a synthetic example pertaining to section 4 can also be found in Appendix B; it was relegated there because of the page limitation. We will try to add a sketch of it in Section 4 to ease the way the paper reads.\n\nWe will also try to be clearer about Tables 1 and 2. The former tells the reader, for example, that pmf $\\ell_1$ assigns a probability of $0.3$ to the element $\\omega_1$ of the state space $\\Omega$, and similarly for the other pmf’s and the other elements of the state space. Table 2 tells the reader the values that the lower and upper probabilities assign to each element of the power set $2^\\Omega$. They are computed according to [6, Section 4.4]. That is, for all $A\\in 2^\\Omega$,\n\n$$\\underline{P}(A)=\\max \\left\\lbrace{ \\sum_{\\omega \\in A} \\underline{P}(\\omega) , 1 - \\sum_{\\omega \\in A^c} \\overline{P}(\\omega)}\\right\\rbrace$$\n\nand\n\n$$\\overline{P}(A)=1-\\underline{P}(A^c)=\\min \\left\\lbrace{ \\sum_{\\omega \\in A} \\overline{P}(\\omega) , 1 - \\sum_{\\omega \\in A^c} \\underline{P}(\\omega)}\\right\\rbrace.$$\n\nFinally, Figure 1 is a visual representation of the resulting credal set (focusing only on the singletons in the power set $2^\\Omega$). As we can see, $\\omega_3$ has a probability between $0$ and $0.75$, $\\omega_1$ between $0$ and $0.5$, and $\\omega_1$ between $0.25$ and $1$.\n\n*In my understanding, the results derived in the paper rely on the availability of the credal set that is supposed to contain the true data generating process. How is that credal set elicited?*\n\nWe thank the reviewer for the opportunity to clarify on this point. Section 3 presents three ways of eliciting a credal set directly from the available data (see Sections 3.1.1, 3.1.2, 3.1.3, and 3.2), and an extra one is presented in Appendix D. Perhaps this was not clear enough, so we will strive for greater transparency in the new version. \n\nFor the sake of completeness, let us summarize here the three approaches in the main portion of the paper. In the first one (Section 3.1.1) we first perturb the likelihood pertaining to each observed training set $D_i$, and then we take the convex hull of these perturbations. In the second one (Sections 3.1.2 and 3.1.3), we derive a plausibility function from the likelihoods, and we use the latter to characterize a credal set of probability measures whose probability density functions are pointwise dominated by the plausibility function. Finally, in Section 3.2 we illustrate the subjectivist approach, in which the scholar first specifies in a subjective manner – but influenced by the available empirical probabilities – the lower probability of some events of interest, then they extend (via Walley’s extension principle) these values to a lower probability defined over the whole power set $2^\\Omega$, and in turn consider the credal set of probabilities that setwise dominate such an extended lower probability.\n\n*In principle, the credal set can be arbitrarily large in the sense that it can have arbitrarily many extreme points. Is there a bound on its size?*\n\nWe thank the reviewer for this deep question; let us try to answer it. The size of the credal set does not necessarily depend on the number of its extreme elements. Think of a very small ball (whose extreme points are infinitely many) inscribed in a large polygon (having finitely many vertices, and hence finitely many extreme elements). \nRather, one way of capturing the size of a credal set is to look at its diameter. To the best of our knowledge, there is no general recipe to bound it: it depends on how “far” (in some well-defined metric or divergence) the true data generating distributions for each of the collected training sets are. In this paper, we aimed at giving general results, in which one cannot control the diameter of the credal set. However, in many applications this may be indeed possible. In continual learning, for example, one can rely on the task similarity assumption (see e.g. Assumption 1 in https://arxiv.org/abs/2305.14782), that tells us that the data associated with each task are generated i.i.d. from distributions whose distance (e.g. in the TV metric) does not exceed some value $r$ specified by the user.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KNKM2dTXi5", "reviewer": "Reviewer_sJhN", "summary": "This paper introduces a novel learning framework termed “Credal Learning Theory” which extends the traditional statistical learning theory to handle variations in data distributions, especially the topic “domain generalization”. The authors propose using credal sets, which are convex sets of probability distributions, to model the variability in data generating processes. By leveraging multiple training sets generated from different distributions, the framework infers a credit set that allows the model to capture uncertainty and have guarantees in domain generalization. The paper also provides theoretical bounds on the expected risk of models learned within this credal framework under various conditions, including finite and infinite hypothesis spaces, and both realizable and non-realizable scenarios.", "questions": "Please refer to Weaknesses. In addition to the issues mentioned in Weaknesses above, are there any analyses of Computational Complexity that are specific to the use of credal sets as an approach to infer models? Can such an approach be implemented for limited but large datasets?", "limitations": "The authors have discussed the limitation of this paper in Section 5.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper introduces the novel concept of Credal Learning Theory to account for data distribution variability using credal sets. This approach presents a new perspective for analyzing the domain generalization field. The authors innovatively apply techniques from Imprecise Probabilities to infer models from limited multiple sample sets, which is practically significant. The definition of credal sets as the convex closure of a family of distributions is also quite natural. Additionally, the paper provides complete mathematical proofs to support the derived generalization bounds. The use of credal sets to model epistemic uncertainty is robust and well-justified, providing theoretical guarantees.", "weaknesses": "My main concern is whether the use of the credal set introduced in this paper is sufficient for analyzing domain generalization. As the paper describes, the model is inferred from a finite number of datasets, and we are studying cases where each dataset is sampled from one of a potential family of distributions. Domain generalization aims to provide guarantees for generalization across different distributions within this potential family. The paper provides generalization guarantees for all distributions within the credal set constructed by convex combinations of inferred distributions. However, the gap lies in whether the credal set adequately represents the true potential family of distributions. To convincingly address this, the paper should at least consider the following two aspects:\n\n1. Provide real-world examples demonstrating that the constructed credal set encompasses most of the potential distributions we need to consider in these scenarios, thus illustrating the practical relevance of the credal set.\n2. Make assumptions about the potential family of distributions, such as assuming the family follows a certain distribution, and prove that the credal set occupies a significant proportion of the support region with high density for this distribution. This would show that the generalization guarantees given for the credal set are meaningful for real-world distribution scenarios.\n\nThese two points are essential to demonstrate the universality of the credal set. Otherwise, there may be cases where the distributions of multiple datasets used for inferring the model are very close to each other, making their convex combination very small and only covering a tiny part of the support of the true family of distributions. This would mean there is no guarantee for the true potential family of distributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*Provide real-world examples demonstrating that the constructed credal set encompasses most of the potential distributions we need to consider in these scenarios, thus illustrating the practical relevance of the credal set.*\n\nWe thank the reviewer for the question, and for giving us the opportunity to expand on the matter. In general, as the reviewer points out and as we mention in our manuscript, a general method to guarantee (even probabilistically) that the true distribution of the new dataset will be included in the credal set is still to be developed. That being said, in many applications – e.g. to continual learning (CL) – scholars often make assumptions that make the use of credal sets with suitable coverage properties plausible. Think for example of the task similarity assumption in CL https://shorturl.at/UJuiZ. There, it is posited that the oracle distributions pertaining to all the tasks of interest are all contained in a TV-ball of radius $r$ chosen by the user. This captures the idea that the model studied will be used on tasks that are not too different from each other. In this case, the credal set elicited from a finite sample of distributions from the ball will be a good approximation of the ball itself, converging for $N \\rightarrow \\infty$ to the entire ball. In the healthcare setting, experts’ opinions can be incorporated alongside empirical data (plausible probability distributions) to represent the probability uncertainty, for example, for the prognosis of a disease given a set of patient characteristics/biomarkers (see e.g. https://shorturl.at/CTKid). To make sure that the credal set constructed encapsulates most of the `potential distributions needed’, a number of approaches can be taken including incremental learning; in this approach the AI model learns and updates knowledge incrementally. As a result, the credal set can be continuously updated (via incremental learning) as new data become available. In this direction, learning health systems are being implemented in practice. These are health systems “in which internal data and experience are systematically integrated with external evidence, and that knowledge is put into practice” https://shorturl.at/ubSCG. Once again, in the future we will study how to guarantee that the true distribution is an element of the credal set in full generality, and we will apply our findings to real-world datasets.\n\n*Make assumptions about the potential family of distributions, such as assuming the family follows a certain distribution, and prove that the credal set occupies a significant proportion of the support region with high density for this distribution. This would show that the generalization guarantees given for the credal set are meaningful for real-world distribution scenarios.*\n\nWe thank the reviewer for their point, which ties to the one in the previous question. They are right in pointing out that – given that no general way (yet) exists of guaranteeing that the oracle distribution for the $(N+1)$-th dataset belongs to the credal set obtained from the previous $N$ ones (as also pointed out e.g. in Section 7 of https://shorturl.at/zbXiu) – assumptions must be made on a case-by-case basis. Once those assumptions are made, then it should be shown that either the credal set covers a non-negligible portion of the distribution class of interest, or that even a small credal set is \"good enough\". We will make this clear in the new version of the manuscript. \n\nWe note in passing, though, that since our goal is maximal generality, the methods that we presented in Section 3 induce rather large credal sets. Consider for example Section 3.1.1. There, we first $\\epsilon$-perturb all the likelihoods that we elicit for the $N$ \"past\" datasets, and then we take the convex hull of the union of such perturbations. If the $N$ likelihoods are \"diverse enough\", we will be able to cover a \"wide area\" of the distribution class of interest. For example, imagine that the class is \"univariate continuous distributions supported on $\\mathbb{R}$\". If $\\mathcal{L}_1$ is a Normal centered at $\\mu_1$, and $\\mathcal{L}_2$ is a Normal centered at $\\mu_2$, $\\mu_1 \\neq \\mu_2$ and \"far enough\" from each other, both having same (or similar) variance, then by considering the credal set induced by these two, we already cover virtually all distributions in the class. To see this, notice that, as pointed out in the equation between lines 145 and 146, the credal set includes all distributions that setwise dominate the lower probability. In addition, by Example 3 in https://tinyurl.com/3ftw7hrn, we know that the lower probability of any set $A$ in the sigma algebra is given by $\\min_i (1-\\epsilon_i) \\mathcal{L}_i (A)$. Now, since the tails of a Normal distribution decay fastly, this means that  $\\min_i (1-\\epsilon_i) \\mathcal{L}_i (A)$ is very close to $0$, for all $A$. To visualize this, please refer to the second picture in Example 1 of https://shorturl.at/x0VbJ. There, we have two Normals depicted, one in red and one in blue, having the same variance, the former (call it $L_1$) centered at 9, and the latter (call it $L_2$) centered at 20. As we can see,  $\\min_i \\mathcal{L}_i$ is basically zero everywhere, and in turn so will be $\\min_i (1-\\epsilon_i) \\mathcal{L}_i$, for any $\\epsilon_i >0$ that we choose. But then, almost all univariate continuous distributions on $\\mathbb{R}$ will setwise dominate such a lower probability, resulting in a \"virtually all-encompassing\" credal set.\n\n*Are there any analyses of Computational Complexity that are specific to the use of credal sets as an approach to infer models? Can such an approach be implemented for limited but large datasets?*\n\nWe thank the reviewer for their question, which we will answer in the Global Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OoNlvZkGoG", "reviewer": "Reviewer_2ZTx", "summary": "This is a paper on learning theory with a focus on machine learning. The authors consider a setup with several training sets available.  Given an additional (test) set, the authors assume that the distribution generating these new data coincides with one of the distributions generating the training set or is a convex combination of them.  Under a few additional assumptions, the authors can find bounds on the expected loss of the empirical risk minimiser.", "questions": "- In which sense are the results in the paper not a simple corollary of those in Liang's LNs?\n- Is it possible to characterise the impact of the realizability assumption on the bounds? What about considering this case even in the experiments?\n- Is it possible to advocate the multi-dataset setup considered by the authors better? Are there examples of real tasks that cope with such a situation?\nI found the experiments very good for the paper, while the discussion about the credal set learning was not so crucial. What about having the latter in the appendix and the former in the main body of the paper?", "limitations": "The limitations discussed by the authors wrt the experiments are very reasonable and I see no problems with them.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is quite technical and dense. Yet, the proofs gathered in the appendix are relatively easy to follow and, after a preliminary check, correct. I think the generalised setup considered by the authors is quite interesting and challenging. \n\nThe author also presents an experimental evaluation of their bounds. This is very valuable for such a kind of theoretical paper.", "weaknesses": "The setup considered by the authors might appear a bit special and not very common.\n\nMost of the results are a generalisation to the multi-dataset setup of results included in Liang's Lecture Notes. In a sense, the work in this paper might appear as a (straightforward?) extension of those results to the case of multiple datasets.\n\nIt is not very clear whether, relaxing the realizability assumption (something adding a lot of realism to the modelling) might have a strong impact to the bounds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*In which sense are the results in the paper not a simple corollary of those in Liang's LNs?*\n\nWe thank the reviewer for this deep question, and for giving us the opportunity to be clearer about this topic. \n\nTheorem 4.1 is an immediate extension (not a corollary though) to the credal case of Liang’s LNs, as shown in Corollary 4.3. Corollaries 4.2 and 4.4, instead, cannot be immediately traced back to Liang. The first because it shows that the diameter of the credal set plays a role in deriving a bound to the expected risk of the ERM computed according to any possible distribution within the credal set itself (not necessarily the oracle one). Corollary 4.4, instead, inspects the possibility of distribution drift in the newly observed dataset, something Liang does not take into consideration. Similar considerations hold for the results in Sections 4.2 and 4.3. \nLet us add two considerations. Theorem 4.5 utilizes a different proof technique than the analogous, non-credal counterpart in Liang’s work, and Theorem 4.9 requires the definition of the extremal Rademacher complexity $R_{n,P^{ex}}(\\mathcal{A})$, which, to the best of our knowledge, has never been introduced before. \n\n*Is it possible to characterize the impact of the realizability assumption on the bounds? What about considering this case even in the experiments?*\n\nWe thank the reviewer for this deep question. Yes, it is possible. By looking at the proofs of Theorem 4.1 (in particular the last equation of page 13) and Theorem 4.5 (in particular Equation (9)) in Appendix A, we see how foregoing realizability implies a slightly looser bound. We did not include this consideration in the original version of the manuscript because of page limitations, but we will do so in the updated version. A newly added synthetic experiment  shows this behavior. In the experiment we computed the theoretical bound $\\epsilon^{\\star\\star}(\\delta)$ and verify (i) that it is slightly looser (e.g. with 200 samples, $\\epsilon^{\\star}(\\delta) = 0.03785$ while $\\epsilon^{\\star\\star}(\\delta) = 0.03800$) and (ii) whether the difference $L_P(\\hat{h}) - L_P(h^\\star)$ is within this bound. The results showed that the empirical risk of the empirical risk minimizer $\\hat{h}$ is within the bound $\\epsilon^{\\star\\star}(\\delta)$ of the best theoretical model $h^\\star$. The difference $L_P(\\hat{h}) - L_P(h^\\star)$ also satisfies the condition \n$$L_P(\\hat{h}) - L_P(h^\\star) \\leq \\epsilon^{\\star\\star}(\\delta).$$ \nThe experiment validates empirically (in a synthetic environment) Theorem 4.5, showing that even under no realizability, the empirical risk minimizer's performance is close to the theoretical best within a computable bound. The experimental results are presented in revised manuscript’s Table B.4, and can be seen here: https://shorturl.at/x6JFq\n\n*Is it possible to advocate the multi-dataset setup considered by the authors better? Are there examples of real tasks that cope with such a situation?* \n\nOur approach is closely linked with continual learning applications, which emphasize the need to handle diverse and sequential datasets to achieve robust and generalizable models. Recent works in continual learning have demonstrated the practical applications and benefits of using a multi-dataset setup. For instance, [1] introduces a novel method for domain incremental learning, leveraging multiple datasets to adapt seamlessly across different tasks. Another example is [2], which proposes a parameter-efficient continual learning framework that dynamically expands a pre-trained CLIP model through Mixture-of-Experts (MoE) adapters in response to new tasks. [3] addresses the challenges of multi-modal medical data representation learning through a continual self-supervised learning approach. These examples from recent studies demonstrate the practical applications and benefits of using a multi-dataset setup in a continual learning framework. Furthermore, some techniques use a multi-dataset setup in continual learning without relying on a specific temporal order. For example, [4] presents a replay mechanism based on single frames, arguing that video diversity is more crucial than temporal information under extreme memory constraints. By storing individual frames rather than contiguous sequences, they can maintain higher diversity in the replay memory, which leads to better performance in continual learning scenarios. \n\nBy including citations to these works and discussing their relevance, we will strengthen the advocacy for our approach in the updated version of our manuscript.\n\nReferences:\n\n[1] Jeeveswaran, K., et al. Gradual Divergence for Seamless Adaptation: A Novel Domain Incremental Learning Method. ICML 2024.\n\n[2] Yu, Jiazuo, et al. \"Boosting continual learning of vision-language models via mixture-of-experts adapters.\" Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024.\n\n[3] Ye, Y., et al. Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning. \" Proc. IEEE/CVF CVPR. 2024.\n\n[4] Alssum, Lama, et al. \"Just a glimpse: Rethinking temporal information for video continual learning.\" Proc. IEEE/CVF CVPR. 2023.\n\n*I found the experiments very good for the paper, while the discussion about the credal set learning was not so crucial. What about having the latter in the appendix and the former in the main body of the paper?*\n\nWe thank the reviewer for the suggestion, and we will try to move the experiments in the main body of the paper. The current structure is the result of feedback we received in the importance of eliciting actual credal sets from the available data. Because deriving credal sets directly from data is currently an open question in Imprecise Probabilistic Machine Learning, in the new version we plan to keep a sketch of both how to derive credal sets, and of the key experimental results, in the main part of the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GD52UjLshT", "reviewer": "Reviewer_SbDN", "summary": "This paper extended traditional statistical learning theory which usually considers a fixed underlying data-generating distribution to a case where the underlying distribution is assumed to be from a convex set of distributions. Several excess risk bounds (finite realizable, finite unrealizable, and infinite hypothesis space) were derived.", "questions": "Minor issues:\n- ERM, DA, DG: please provide references.\n- the position of Figure 1.\n- l. 74: the citing style \"Zhou et al. [63]\" seems strange (to me)", "limitations": "The author discussed the limitations briefly in the conclusion section.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper aims to analyze the uncertainty and divination of training/test data distributions in a principled manner, which is very appreciated.\n- Using convex sets of distributions is a reasonable approach.\n- This paper considered different assumptions on the hypotheses.", "weaknesses": "- The writing can be improved. For example, the first paragraph is basically the notation and a review of ERM. The second paragraph mentioned DA and DG, but it is still unclear what problems existing theories and problem settings have and why it is important to solve them. Then, the author implied that the existing assumptions are unrealistic and too strong so the resulting theories are not generalizable. However, these statements are too vague so it is difficult to verify or falsify them.\n- Partly due to the writing, I cannot fully understand the proposed theorems, let alone their proofs. For example, I do not understand what exactly a _well-defined_ quantity $\\epsilon^\\star(\\delta)$ in Theorem 4.1 (and the following theorems) is. Therefore, the theoretical implications are very unclear to me, and I cannot accurately assess the significance and novelty of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*The writing can be improved. For example, the first paragraph is basically the notation and a review of ERM.* \n\nWe thank the reviewer for their input. In the updated version, we will strive to further improve our writing. We believe, however, that the first paragraph is needed to set up the notations, so that every reader can be put in a position to follow our arguments, regardless of their background knowledge of the matter. \n\n*The second paragraph mentioned DA and DG, but it is still unclear what problems existing theories and problem settings have and why it is important to solve them. Then, the author implied that the existing assumptions are unrealistic and too strong so the resulting theories are not generalizable. However, these statements are too vague so it is difficult to verify or falsify them.*\n\nRegarding DA and DG, we agree with the reviewer, and we will improve clarity regarding existing theories and their shortcomings. At a very high level, no existing technique takes into account possible distribution misspecification and drift like we do, thanks to our credal set approach.\n\nAdditional clarifications statements have been added in the Related Work section. The following changes have been made addressing the reviewer’s comments:\n\n\nIn line 78, remove from \", or are reliant on strong assumptions (e.g.,\" until line 80 up to \"DG approaches.\" Include: \n\"Regarding Kernel-based methods, assumptions related to boundedness of Kernels and continuity of feature maps and loss function render the approaches not directly applicable to broader scenarios (Deshmukh et al., 2019; Hu et al., 2020; Ye et al., 2021)\"\n\n\n\n\nIn line 82, after the sentence ending \"H-divergence [1 ].\", remove text up to line 84 ending in \"robust generalization [49 ].\" Include:\nResearchers have also focused on adaptation to new domains over time, treating DG as an online game and the model as a player minimizing the risk associated with introducing new distributions by an adversary at each step (Rosenfeld et al. , 2022).\nHowever, in scenarios where the training distribution is significantly outside the convex hull of training distributions (Albuquerque et al. 2019), or because of unmet strong convexity loss function  assumption (Rosenfeld et al., 2022), they fall short from achieving robust generalization.\n\n\nRemove text from line 90 up to line 92 ending in \"an adversary at each step [49 ].\"\n\n\nAfter line 96 ending in \"range of models [ 36 ]. \", include: \nThough this simplification has a number of practical benefits, models trained under covariate shift assumptions might suffer in terms of robustness to other distribution shift types.\n\n\n*Partly due to the writing, I cannot fully understand the proposed theorems, let alone their proofs. For example, I do not understand what exactly a well-defined quantity $\\epsilon^\\star(\\delta)$ in Theorem 4.1 (and the following theorems) is. Therefore, the theoretical implications are very unclear to me, and I cannot accurately assess the significance and novelty of this work.*\n\nWe thank the reviewer for pointing this out. Because of the 9 page limitation, we were not able to deliver the proofs to our results in the main part of the paper. They are postponed to Appendix A. We will likely not be able to move them to the main body of the paper, as there is a limitation in the rebuttal phase as well. The interested reader, though, can find there proofs to all the statements in the appendix mentioned. \n\nWe note in passing that we use the term \"well defined\" in the usual mathematical sense of \"expression whose definition assigns it a unique interpretation or value\" [1]. In particular, given any value of $\\delta$, the quantity $\\epsilon^\\star(\\delta)$, depending on $\\delta$, is indeed well-defined. To see this, we refer the reviewer to the last equation on page 13.\n\n[1] Joseph A. Gallian, Contemporary Abstract Algebra, Houghton Mifflin, 2006.\n\n*ERM, DA, DG: please provide references*\n\nWe thank the reviewer for pointing this out. Regarding ERM, we refer to Liang’s lecture notes and [38] extensively in our manuscript. In the new version of the manuscript, we will mention them already in the first page. Regarding DG, we explicitly mention [47] in the last line of page 1. We will add a reference for DA in the new version (https://link.springer.com/article/10.1007/s10994-009-5152-4).\n\n*the position of Figure 1.*\n\nWe thank the reviewer for their suggestion, and we will move Figure 1 right before the \"Paper outline\" subsection of the paper.\n\n*l. 74: the citing style \"Zhou et al. [63]\" seems strange (to me)*\n\nWe do not know how to handle this suggestion, we must confess. It is the visualization of the \\citet command that we use extensively throughout the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**We first answer to Reviewer sJhN's last question**

*Are there any analyses of Computational Complexity that are specific to the use of credal sets as an approach to infer models? Can such an approach be implemented for limited but large datasets?*

We thank the reviewer for their question, the answer to which we will try to add to the new version of our manuscript. Yes, there are analyses in the literature of computational complexity specific to the use of credal sets, particularly in the context of graphical models and probabilistic inference. Such approaches can be implemented for large datasets, but they often require approximation techniques to be computationally feasible [1, 2, 3 ].
Despite this, credal set approaches can be implemented for large datasets using techniques like parallel processing, distributed computing, and efficient data structures. Similar to Deep Learning-based approaches, utilization of high-performance computing resources, algorithm optimization, and domain-specific adaptations, the computational challenges can be effectively managed. Recent advancements demonstrate the practicality of these approaches. For instance, "Credal-Set Interval Neural Networks" (CreINNs) have shown significant improvements in inference time over variational Bayesian neural networks [4]. Thus, while the computational demands are comparable to those of deep learning-based methods, the robustness and flexibility of credal sets, as demonstrated in recent research, make them a practical and valuable approach [5, 6].

References:

[1] Mauá, Denis Deratani, and Fabio Gagliardi Cozman. "Thirty years of credal networks: Specification, algorithms and complexity." International Journal of Approximate Reasoning 126 (2020): 133-157.

[2] Lienen, Julian, and Eyke Hüllermeier. "Credal self-supervised learning." Advances in Neural Information Processing Systems 34 (2021): 14370-14382.

[3] Mauá, Denis D., et al. "On the complexity of strong and epistemic credal networks." arXiv preprint arXiv:1309.6845 (2013).

[4] Wang, Kaizheng, et al. "CreINNs: Credal-Set Interval Neural Networks for Uncertainty Estimation in Classification Tasks." arXiv preprint arXiv:2401.05043 (2024).

[5] Marinescu, Radu, et al. "Credal marginal map." Advances in Neural Information Processing Systems 36 (2024).

[6] Wang, Kaizheng, et al. "Credal Wrapper of Model Averaging for Uncertainty Estimation on Out-Of-Distribution Detection." arXiv preprint arXiv:2405.15047 (2024).

---

**We also attach a pdf with the Table pertaining to the new experiment we discussed in our answer to reviewer 2ZTx's second question.**

Meta-review: This paper develops statistical learning bounds (e.g. PAC-style) based on the assumption that the user can identify a convex credal set that includes the true (test) distribution, and use this information to provide generalization guarantees. This allows the authors to derive bounds on the risk of models derived from multiple data sets, for example. Contributions are mainly theoretical, with some reviewers concerned about the density of notation and results.  I agree with one reviewer that the experiments (in the supplement) are helpful, but clearly the elicitation of the credal set is also critical, so there is some tension in how to allocate the available space.  Some reviewers were concerned about the practical implications, since the results depend on the correctness of the identified credal set, for example, but I do not consider this a significant flaw.
Decision: Accept (poster)