Conformalized Credal Regions for Classification with Ambiguous Ground Truth
Authors: Michele Caprio
David Stutz
Shuo Li
Arnaud Doucet
Venue: Accepted by TMLR ()

Abstract: An open question in Imprecise Probabilistic Machine Learning is how to empirically derive a credal region (i.e., a closed and convex family of probabilities on the output space) from the available data, without any prior knowledge or assumption. In classification problems, credal regions are a tool that is able to provide provable guarantees under realistic assumptions by characterizing the uncertainty about the distribution of the labels. Building on previous work, we show that credal regions can be directly constructed using conformal methods. This allows us to provide a novel extension of classical conformal prediction to problems with ambiguous ground truth, that is, when the exact labels for given inputs are not exactly known. The resulting construction enjoys desirable practical and theoretical properties: (i) conformal coverage guarantees, (ii) smaller prediction sets (compared to classical conformal prediction regions) and (iii) disentanglement of uncertainty sources (epistemic, aleatoric). We empirically verify our findings on both synthetic and real datasets.

Published in Transactions on Machine Learning Research (10/2024)
Conformalized Credal Regions for Classification with Ambiguous Ground Truth
Michele Caprio
michele.ogroo@mmnnchsteer.cccc
Deportment of Computer Science University of Monchester
David Stutz
dstutz Selnutt@ddepmmindd.o com
DeepMond
Shuo Li
lishuol I Bseas. @seas.upenn. edu
Department of Computer and Information Science University of Pennsyloania
Arnaud Doucct
araaddioce
DeepMind
Department of Statistics, University of Orford
Reviewed OpenReview: https: //
https://openrevieu openreview. net/ ptr/formm? id= L7sQ8C2FF
Abstract
An open question in Imprecise Probabilistic Machine Learning is how to empirically derive credal region (i.e., closed and convex family of probability measures on the output space) from the available data, without any prior knowledge assumption. In classification prob- lems, credal regions are a tool that is able to provide provable guarantees under realistic assumptions by characterizing the uncertainty about the distribution of the labels. Building on previous work, we show that credal regions can be directly constructed using confor- mal methods. This allows to provide novel extension of classical conformal prediction to problems with ambiguous ground truth, that is, when the exact labels for given inputs are not exactly known. The resulting construction enjoys desirable practical and theoret- ical properties: (i) conformal coverage guarantees, (ii) smaller prediction sets (compared to classical conformal prediction regions) and (iii) disentanglement of uncertainty sources (epistemic, alcatoric). We empirically verify our findings on both synthotic and real datasets.
1 Introduction
In most real-world applications of machine learning, especially in the context of safety-critical applications such as healthcare, researchers have found it difficult to reason with precise probabilities. Instead, practition- ers have become comfortable reasoning in terms of familics of probabilities, often in the form of closed and convex sets called credal regions. Imprecise probabilistic machine learning (TPML.) (Denoeux
2000 Zaffalon
2002
al. 2024 Caprio 2024; Caprio et al.) 2024a d Sale et al. 2024) aims to develop machine learning theory and
Destercke al. 2008 Caprio Gong 2023 Caprio Mukherjee 2023a by Dutta et al. 2023
methods that work with such imprecise probabilities. Such tools allow to better quantify and disentangle different types of uncertainty, e.g., epistemic (model) and alcatoric (data) uncertainties, which play key roles in any machine learning system. However, other sources of uncertainty are also highly relevant; for example, uncertainty originating from the annotation process used to derive ground truth labels (Stutz et al. 2023a
A crucial line of research IPML that of empirically deriving credal regions without any prior knowledge or assumption. The first steps in this direction were made recently by Cella. Martin (2022a) b). They discovered that, subject to a so-called consonance assumption (see Section 3.2 and given exchangeable

---

Published in Transactions on Machine Learning Research (10/2024)
calibration data, the conformal transducer assigning a p-value to each possible label uniquely identifies credal region. This is a very promising result, since this only relies on having to select non-conformity measure, and the obtained coverage guarantee is valid irrespective of this choice. Unfortunately, [Cella & Martin 2022a b) do not provide an implementation of their results on credal regions in real-world, complex datasets. Moreover, they ignore the particularly interesting case of ambiguous ground trulh (Stutz et al.
2023a b) where labels are not crisp, but subject to uncertainty due to rater disagreement (Yan et al. 2014 Zheng et al., 2017) and imperfect labeling tools. In fact, calibration data are often not exactly classified: we can see this in machine learning (Dawid & Skene 1979:
Smyth et al., 1994), but especially in medicine
(Feinstein & Cicchetti, 1990; McHugh, 2012; Raghu et al. 2019; |Schakkermann 2020 natural language
processing (Pavlick & Kwiatkowski 2019
Nie et al. 2020
Baan et al. 2022 Plank 2022). and computer
vision (Peterson el al. 2019a).
Contributions. In this paper, we follow an alternative way of constructing credal regions in a conformal
way, inspired by recent work (Javanmardi et al., 2023 2024 Lienen et al. 2023 Martin Bordini et al., 2023). Specifically, we directly construct conformalized families of probabilities and show that these are in fact credal regions, i.e. convex and closed. Furthermore, we extend this approach to classification problems with ambignous ground truth, first studied in the context of conformal prediction by Stutz et al. (2023b c) These problems are extremely relevant from an applied point of view. Instead of a "precise" calibration set (x".y") =
}n C X' x y we consider a calibration set D. = {(xx,>;) that uncompasse, the ground truth ambiguity, where the X; 's are probability vectors, and the k-th entry X1, is the probability of being the correct label for xı
i To see that this is a generalization of the usual classification problems, notice that can write pair (Iaa: y/) as a pair (I;;A,) where X is a one-hot encoding vector with entry on the correct label k Y/- By dircctly conformalizing in the probability space, we obtain appcaling calibration property for the constructed credal region: the true data generating process belongs to the credal region with high probability 1 - a, where a is chosen by the user. Due to this property, our credal regions can be used to disentangle and quantify alcatoric and epistemic uncertainties associated with the analysis at hand (Hüllrmeier & Waegeman 2021 Sale et al. 2023). On both synthetic and real datasets, we verify both the coverage of the constructed regions and the true label coverage guarantee of the derived predictive sets. In contrast to Cella & Martin (2022 b), we do not require any assumption beyond exchangeability. Using derive predictive sets of labels that are more efficient, i..,, smaller on average, compared to those by Stulz et al. 2023ct.
imprecise highest density sets,
Section 2 provides the required background. First, in Section 2.1 we recall the plausibility regions and the conformal predictive sets derived in [Sutz et al. (023c and define the corresponding conformal coverage guarantee which we built on in the following. Then, in Scction 2.2 we revisit the necessary imprecise probabilistic concepts relevant for this paper, inchuding
Outline. This paper is structured as follows:
imprecise highest density sets (IHDSs). Part of our main results, Section 3 shows that the plausibility
region in Stutz et al. (2023c) is equivalent to credal region, which also satisfies a desirable calibration
property. Then, Sections 3.1 and 3.2 discuss the utility of our credal region the field of Imprecise Probabilistic Machine Learning in general and the relation and improvements over the work of Colla & Martin (2022a by in particular. Finally, Section presents our findings on obtaining more efficient prediction sets from our credal regions compared to (Stutz et al. 2023ch. We present our experimental evidence in Section and conclude in Scction 0
'Reference Stutz et al. (2023b) refers to the published version, while Stutz et al. (2023c to the first version of the paper. Since many of the concept needed in the present paper were not included in the final, published version, we differentiate between the two for case of reference.
2We use lower case letters for realizations, while capital letters for random variables. In this case, calibration data (1s, 91) (a., Bh) realizations that we observe, while the test example X is a random variable that has not yet been realized. 3 A pair (at, Ax) in the calibration set the realization of random pair (X,,
2

---

Published in Transactions on Machine Learning Research (10/2024)
2 Background
2.1 Conformal Plausibility Regions
In Stutz et al. (2023b the authors study conformal prediction for K-class classification in the context of ambiguous ground truth. Standard split conformal prediction typically assumes instead a calibration set of examples with "crisp" ground truth labels {(x,, Y:)) (Shafer & Vovk 2008p. Realizing that such crisp ground truth labels might not be available in many practical scttings, Stutz et al. (2023b/c) assume calibration set of examples and so-called plousibililies, {(X,, A=) where plausibilities X (i.e., the re- This allows to represent ambignous examples where the corresponding distribution P[ is not one-hot and might have high en- tropy. In the full-information setting, these plausibility vectors may represent the true distribution P[ directly; in many practical scenarios, however, we can only approximate the true distribution. For example, disagreement among annotators frequently indicates ambiguity and deterministic or probabilistic aggrega- tion of multiple annotators can be used as plausibilitics (Stutz et al.] 2023a). Then, we assume the true dist ion can be obtained in the limit of infinite " "faithful" annotators.
alizations of A,) represent categorical distributions over the K possible labels !
For new test example Xnt1 and significance level a E [0, so-called plausibility region c(Xn+1) is derived as
(((r;)))) Xati, a) ~s c(xntt)== (M K-1 : e(Xut1, Ant1 = 2) 2
(1)
where A K-1 is the unit simplex in RK and e(Xn+1. = X) = Ex Ax k) is score derived from a so-called conformity score F((x+1,,) k) based on the probabilistic model prediction pa (x Such a model (e.g. the softmax output for class & of a trained neural network) approximates the postcrior class probabilities, (x R IP(Y++1 = Xa+1, D), where denotes the training set, and it is assumed to be available. Conformity score E is built so that a high score is more unlikely, hence we can interpret E function assigning a score to the assertion "K is the correct label for the realization Ta+1 of test example Xn+1", which is lower the more the pair k)) "lacks conformity" with the training dala. It is worth noting that alternative definitions of have also recently been explored by Javanmardi et al. (2024) The threshold r is then chosen using a simple quantile computation on the calibration set,
T T (fe (X,, A.) [a(n + 1))/n))
Under the assumption that alure (Stutz et al. 2023c Equation (13)), the plausibility region c(X,+1) provides a coverage guaranlee stating that P P|A,+1 E c(Xx+1)) 1- F example. If the A in the calibration set correspond to the true distributions IP = z:], this provides coverage with respect to the true distribution. As (Stutz et al., 2023c Section 3.1) details, however, in many practical settings, the plausibilities are obtained from expert annotations. With finite annotators, the authors are able to give a coverage guarantee, [Ang| E c(Xn+1)] 2 1 1- a, stated in terms of the distribution Pagg that explicitly captures how annotations are aggregated into plausibilities. The underlying assumption is that \n+1, = P Pagg (Y411 = k|[x,+1] where, ideally, Pagg s P, and the annotators are the same for the calibration set and the test example. In any case, annotators can "agree to disagrec" such that An+1 has high entropy in which case In+1 is called ambiguous. For simplicity, we ignore this caveat for the presentation of this paper and write P henceforth. The interested reader can find some extra details on the difference between Fagg and P in Appendix A
1 are exchangeable, following standard conformal prediction liter-
where An+1 is the unobserved true plausibility vector of the test
The plausibility region c(Xx+1) is then used in (Stutz et al. 2023c Equation (41)) to derive "plausibility- reduced" predictive sets (PRPS) for some user-chosen E
[0. 1].
(c((Xn-1)) = y e 3X
B c(X ((x+1))) LE), S.L. Am. 2
1 8 and 3i < s.l. 02 =
(2)
"They may be, for example, realizations from random variables A; distributed according Dirichlet distribution. Thii probability is also the calibration set, since the latter is used derive c(Xx+1).
3

---

Published in Transactions on Machine Learning Research (10/2024)
Here X" (DoL..... )ow) " corresponds to vector À sorted in descending order. Equation tells us that V (cc(x++1) is the set of all labels y that are assigned probability non-smaller than (of being the correct one for the realization of Xn+1), by at least one probability vector À that belongs to the plausibility region c(Xa+1).
In Stutz al. 2023cp; this also contrasted with regular conformal predictive sets (CPS) that are obtained by calibrating threshold a on the per-label scores and constructing the CPS C(X++1) as {k y k)) 2 K). There, the latter is further adapted to allow for ambiguous examples with plausibilities X available for calibration, see (Stutz et al., 2023c Algorithm 1). These adapted CPS C(Xn+1) are shown to be generally more efficient, i.c., smaller, compared to V (c((X++1), but are unable to capture or even disentangle different sources of uncertainty. We make a step towards addressing this gap by deriving narrower predictive sets that allow for uncertainty quantification and disaggregation, using methods from imprecise probability.
2.2 Imprecise Probability
Following Cella & Martin 2022a bj and additional previous work on IPML (Augustin et al. 2014 Caprio
& Scidenfeld 2023 Coolen 1992 we briefly introduce the notions of lower and upper probabilitics, and also the concept of Imprecise Highest Density Sets (IHDS). These will play pivotal role in the paper - and especially Section [ as way derive predictive sets of labels from our credal regions, that are shown to be more efficient than the predictive sets (c((x++1)).
We begin by recalling that credal region P is convex and closed family of probabilities. Its lower envelope = inf pep P is called lower probabilily, while its upper envelope P = supper P is called upper probability. Considering the assertion that the true label Yn+1 of test example Xn+1 is included in set A C D, and using P(A) = p(Y E A), we see that the upper probability is conjugate to the lower probability, i.e. for all _ y, P(A) = 1 - P(A"), where A ===Ji > A. Hence, studying one sufficient to then retrieve the other. Morcover, in the present paper y is finite, meaning
Σ P((AA)
P(A) inf P(A) inf
PEP
PEP
LEEA
As we can see, P(AA) can calculated in polynomial time. This is important, since using the lower probability we can derive predictive sets labels.
Definition 1 (Imprecise Highest Density Set, Coolen (9922 ). Let ô E [0, 1] be any significance level. Then,
the (1 Fimprecce Highest Density Set (THDS) ISp,s associated with P the subset of y that satisfies the following two conditions,
(1) P(ISp,s) 2 5-
(it |18p,s) is a minimum of all sets for which (t) holds.
In order to compute IHDSs in practice, we need another important result - proved in (Augustin et al. 2014 Section 4.4) and de Campos et al. (1994h. Proposition 2 (Computing P(A)). If P avoids sure loss, i.e. if EkEy P(())) $15 Exey P((k)), then
P(A) max
Σ P((K)), Σ (((A))
(3)
REA
REA"
Σ AEA
P((A)), VACV.
Since P is the lower envelope of credal region, the sure loss avoidance condition is always met, as proven in Walley 1991). We make this explicit in the following Lemun.
Lemma 3 (Lower Envelopes Avoid Sure Loss). The lower probabilily P. associated with P avoids sure loss.
4

---

Published in Transactions on Machine Learning Research (10/2024)
Definition 1 can be used to derive a parallel between (generic) Conformal Prediction Sets (CPSs) and IHDSs. Condition (i) tells us that Ya+1 belongs to ISp,8 with P-probability of at least: 1 6, for all distributions P in the credal region P. Condition (ii) tells us that 15p,s is "efficient". That is, it is the narrowest subset of the label set y that is able to ensure the probabílistic guarantee of condition (i). Compared to CPSs, this is a weaker guarantee: First and foremost, the conformal guaranlee is uniform, holding for all possible exchangeable distributions on y. Second, without assumptions on the credal region P, condition (i) ignores whether the true distribution belongs to the credal region P.
3 Conformal Plausibility Regions as Credal Regions
A key contribution of this paper is relating the plausibility regions c(Xn+1) in equation 1 to the imprecise probabilistic notion of credal regions. As we will show, this leads to a remarkable synergy that allows us construct credal regions for X in a conformal way, and subsequently sets of labels. Compared to Cella & Martin (2022a b). our credal regions provide coverage, only requiring the exchangeability of {(X,, A. In addition, we improve over Stutz et al. (2023c): the predictive sets that we derive are more efficient, i.c., always non-broader and sometimes strictly narrower. For a start, we show that the plausibility regions c(Xn+1) are convex and closed, and thus proper credal regions.
use IHDSs to construct predictive
Proposition 4 (Properties of the Plausibility Region). The plausibilily region el c(Xu+1) derived in equalion is conver and closed
Proof. We first show that c(Xx+1) is convex. Pick 10), E c(Xa+1), 1 # A(2), and 8 [0, 1]. Then,
K
e(Xn+1, + (1 (1- ()) = Σ
(BX
+(1 -1--10 -1) (((Xn+1,
k=1
K
8
(1)
E X n+1 k)) (1 (1-11 B)
Σ x
(2)
E(X,+1,h)
k=1
k=1
2 Br + B) =
Hence, BX(1 + +(1 B)A(9) E c(X.-1), which proves convexity.
Let us then turn our attention to closure. We begin by showing that e(Xx+1,)) is a continnous function. Pick a converging sequence (A(m) S c(Xn+1). This means that there a probability vector X such that (mm) + >'. That is, for all 0, there exists M E N such that 14- ((m) v, for all m 2 M and all & e (1,...,K] KK. Now, to show continuity, we prove that, for all e 0, there exists & = e E E(X,+1, Indeed,
1 k) >
such that [Ag 2 ( < & implies that le(Xnt1, A²) e(Xn+1, A(m))
K Σ (A X (m)) ((m) E(X+111
X(* e(X e(Xn+1,, (m))
=
k=1
x
(m) (
E(Xn-1,k)
k= 1
8
k()
ke-1 B(X
This proves that .) is continuous. Then, after noting that e(Xn-1, A) 2T, for all \ E c(xn+)) 1+++)= here the weak inequality plays a crucial role in showing closure we can conclude that e(Xn+1, X+) 2 T, and so X E c(X,-1), proving that c(Xx41) is indeed closed.
6Te topology in which of ((Nn+)) closed specified the proof statement.
5

---

Published in Transactions on Machine Learning Research (10/2024)
(0.0. P((3))
P((1))
P(()))
(1,0,0)
(0,1,0)
Figure 1: Suppose we are in a 3-class classification setting, so y (1,2,3). Then, any probability measure P y can be seen as probability vector. For example, suppose P((1)) 0.6, P([2)) 0.3, and P([3)) 0.1. We have that P (0.6, 0.3, 0.1)T. Since its elements vector P belongs to the unit simplex A in RR, the purple (2D) triangle in the figure. Then, plausibility region c(Xa+1) is a closed and convex body in A?, such as the depicted orange (2D) heptagon (we depict it with a solid black border to highlight the fact that is indeed closed).
positive and sum up to probability
As can see, then, every À E e( c(Xn+1) uniquely identifies a Categorical distribution Cat(A) parameterized by À itself. A visual representation for c(Xn=1) is given in Figure | A consequence of Proposition is that for all E y, we can find Ag, Ag E [0, 1], A3 S Xx, such that X E Aul, for all E c(X,+1) The following result relates c(Xn+1) to a credal region, that is, a closed and convex family of probabilities Levi 1980). It an immediate consoquence of Proposition and of the definition of c(Xx+1) in equation
Corollary 4.1 (Plausibility Region as Credal Region). The plausibilityy region c(Xn+1) derived in equa- tion is equivalent to the credal region P == {Cat(A) X E c(Xn+1)).
Notice how P is predictive credal region. This is because its elements can be seen as predictive distributions resulting from a Dirichlet (conjugate) prior and a Categorical likelihood. In addition, as we shall see in Section P can be used to derive a set of labels that contain the true one for Xn+1 with high probability.
Recall that Ant1 denotes the "truc" probability vector over the K labels, once we observe the new test exam- ple Xn+I, and consider the Categorical distribution Cat(An+1) parameterized by such vector. Cat(An+1) captures the intrinsic difficulty of labeling X,+1- For an casy-to-categorize instance, we will have a low- entropy Categorical, and vice-versa. for a highly ambiguous input X,+1- We can also give subjectivist interpretation to the credal region P: we can think of An+1 as the vector subsuming the opinions of all the experts, and that cannot be "further refined" given the available information. The sharper the disagreement between the expert around the right label for Xn+1 is, the wider P will be. We have the following important result.
Proposition (Probabilistic Correctness). Let a be the same significance level selected in equation Them,
P(Cat(An+1) E PI
where P depends on the statistical model relating expert opinions to plausibililies as outlined in Section 2. and Stutz et al. 2023c. Equation (6)).
Proof. Immediate from Corollary
4.1 and Stutz et al. 2023c
Equation (13)).
7 As consequence, greedy algorithm approximate the values of de. Ag, for all & Y, is easy design, e.g. based random samples from the uniform Unif(c(X,+1)) the plansibility region c(Xu+1).

---

Published in Transactions on Machine Learning Research (10/2024)
3.1 Significance for IPML
In this section, we argue that Proposition 5 is extremely important for the field of Imprecise Probabilistic Machine Learning (TPML) (Caprio et al. 2024a Hüllermeier Wacgeman 2021 Zaffalon 2002). We will further show (in the next section) that making this connection to imprecise probability allows us to derive more efficient predictive sets from these credal regions, compared to (cc(Xn++1) in equation 2
In IPML, scholars tend to adopt either of the following two approaches. One approach is the robust statisti- cian one, also called Huberian (Huber & Ronchetti 2009). This is more frequentist in nature, hence the true data generating process - Cat(()n1) in our case - is a well-defined concept. Those who take this approach proceed in two different ways: either they assumee that Cat((Nn1)) E P (in which case a result like Proposition 5 is very interesting, as it allows to forego such requirement). or they verify that calibration à la. Proposition
holds for the methodology they propose (Acharya et al. 2015 Gao et. al. 2018 Mortier et al. 2023 Liu
& Briol 2024 Chau et al. 2024). In particular, in the case of transductive conformal prediction, Martin (003)) shows that the upper envelope II of the credal region M(III induced by conformal transducer as we discussed in Section ! is the minimal outer consonant approximation of the true data generating process. This mcans that IT is the narrowest upper bound for the true distribution, that also satisfics the consonance assumption. We can then conclude although it is not explicitly shown in their work that credal region M(II) contains the true data generating process. This is important in Cella and Martin's framework because it is linked to the concept of type-2 validity: since the upper probability II provides control on erroneous predictions uniformly (Cella & Martin 2022b Definition 3), does the true probability if the credal region is well-calibrated. Of course, to crisp labels. In a sense, then, in Proposition 5 we prove (a probabilistic version of) this property for the ambiguous ground truth case in classification problems.
cannol directly use their result in our context because we do not have access
Another approach the deFineltian or Walleyan one (de Finetti 1974
1975 Walley 1991), in which it is
posited that "probability does not exist", and what we call probability mcasure just a way of capturing the subjective assessments of the likelihood of events to obtain, according to the agent. In light of this, the concept of "true data generating process" is absent, and so it does not make sense to look for probabilistic guarantees à la Proposition 5
3.2 Relation to Cella & Martin (2022a b)
Given calibration data with crisp labels in D' as outlined in the beginning of Section 2.1 Cella & Martin (2022a] b) consider the conformal transducer и : y + [0, I], which assigns p-value to each possible label in y (Cella & Martin 2022a Definition 2). It uniquely identifies a credal region, assuming consonance. Here, I is typically interpreted as telling us how "in line" the pair (In-1, 9) is with the previously observed data (x".y") .)). A value closer to means that seeing (Xx+1, Yu+1) = (In+1, 9) would align with the preceding observations, and vice versa for values closer to 0. Once is obtained, consonance posits that there exists at least one label ÿ such that (ū)) 1. Loosely, this means that the label ÿ that makes pair (I W) "the most conformal to" the observations (x", y)), is required to have the highest possible value 1. This can be obtained artificially by setting и for E arg max, TVy) to 1 (Cella & Martin 2022al Section 7). Then, an upper probability (that is, the upper envelope of a credal region) II derived for ' by letting I[(A) suPye I (t), for all subsets A of the label space y. In turn, credal region is defined as M(II) = {P' P(A) < I[(A)) for all A C y. That is, M(II) contains all the probabilities on y that are set-wise dominated by II where P(A) = P(Yn+1 E A).
In comparison Cella & Martin (2022a] bb, we avoid the consonance assumption. In Cella & Martin (2022a b), this assumptions is primarily required to bridge the so-called possibilistic approach to imprecise probabilities (Augustin et al.) 2014 Chapter 4) with conformal prediction. Moreover, we do not explicitly construct the conformal transducer, also due to the fact that we work with probability vectors instcad of "precise" (that is, one-hot) labels. In this sense, our work subsumes and extends Cella & Martin (2022a) b). Furthermore, our credal region P establishes a coverage guarantee on the true data generating process being in P, and it also enjoys (a version of) type-2 validity, a notion important for uncertainty quantification, introduced for the first time in (Cella & Martin 2022b Definition 2). Let us be more formal about it.
7

---

Published in Transactions on Machine Learning Research (10/2024)
Proposition 6 (A Version of Type-2 Validity for Our Credal Region). The credal region P in Corollary 4.1 is (5/(1 a1)p-1pe-22 valid, where o is the quanlity chosen in equation 7
That is,
8 P(P(A) 6, Y, E 1 A 1-a²
for all ô E [0, 1, all n E N, and all A A y. Here as in Proposition IP depends on the statistical model relating expert opinions plausibilities.
Proof. Pick any 8 E [0, 1), any n n E,,, and any A C y. Notice that P(P(A) S 6, Y E A] S P n+1 E A]. Now, if 11) E P, then P[Yn+1 E A] S P(A). In addition, given our assumption on 6, we know that P(A) I 5. Hence, we can conclude that if Cat(An+1) E P. then P(P(A) < 8, Y, A| 6.
But Cat(An+1) P happens with P- F-probability (1 a) shown in Proposition 5 and we have that ((A((A) S 5, Yat1 E A] S 8/(1-0) a].
In words, this means that the (true/agrregated) probability of a set A having small upper probability, and yet containing the true output Yn+1 for a given new input Xn+1, is controllably small, and it depends on the parameters a and 6 chosen by the user. When giving the bound, we need to divide by 1- a because we need to take into account. what is the P-probability that, the true distribution (a+(()+1) belongs to the credal region P. Notice that in equation 4 we find slightly looser bound than the one we would derive if we were not dealing with ambiguous ground truth, i.e. if we knew that P(Cat((n+1) E P] 1. To see that this is indeed the case, notice that if we pick 8
0.05, then the bound in equation 4 is sz 0.526, slightly larger than the one we would have (0.5) if we were not facing ambignity.
4 Improving over Conformal Prediction Sets
In this section, we show how prediction set T (c((nn+1) can be improved by using the IHDS of the lower probability of credal region P. In addition, we show how P can be used to quantify and disentangle aleatoric and epistemic uncertainties. Specifically, let us denote by À the vector whose k-th entry is Au = P((k)), and by Ā the vector whose k-th entry is d = P({k))) Then, we can rewrite equation as
P(A) max Σ
TE
AACA.
(5)
REA"
This gives us an easy way to compute P(A) in practice, to then derive IDHSs. We can use equation 5 and Definition (i) to derive Algorithm D greedy algorithm to build 18p,s- In turn, IHDS ISp,s is used to derive a predictive set narrower than the conformal one Vc((xn+1) from equation 2 and that retains the same probabilistic guarantees. Recall that Definition | (i) gives us a probabilistic guarantee that holds for all distributions in the credal region P that we built in Corollary 4.1 This is a slightly weaker guarantee than that of classical conformal prediction, whose guarantee instead holds for all possible exchangeable distributions P on y. Thanks to Proposition 5 though, a consequence of the following Proposition is that the loss of coverage for the IHDS with respect to classical CPS is negligible.
Proposition (Improving on Conformal via IPs). Let a be the same significance level selected in equation 1 Pick any 8 E [0, 1]. Then, 18p,8 (c((x++))), and the inclusion is strict for some value of 6. In addition, P[Yx41 18p sl 2 (1 5)(1 a) where Yn+1 denotes the correct label for input Xu+1.
Proof. First, let us introduce the concept of (1 6)-(Pre))se) Highest Density Set HDSp,5, for some Pe P Hyndman (1996),
HDSp == ty E ; P((n)) 2 pd),
where p6 is the largest constant such that
P|Y E HDSPsp 2 1-6.
8

---

Published in Transactions on Machine Learning Research (10/2024)
By (Coolen 1992
Page 3), we know that
18p,5 _ U HDSSpp
PEP
{y y EP E P, P(l)) 2 pd),
for all 5 E [0, I], and the inclusion is strict for some value of ò. The first part of the proof is concluded by v(c(Xn+1)), The probabilistic guarantee according to IP is a consequence of Proposition 5 and the fact that p(1Sp,s) 2 1 11-d = p( 2 1 - 6, for all P, by the definition of lower probability.
noting that UPgpHDSS
In standard conformal prediction that is, when we calibrate on crisp-labeled data. the probabilistic guarantee that we derive is of the form P|Y++1 CPS] 2 1 6, where we denote by CPS the conformal prediction set, and by 5 the same threshold as in Proposition [7] In the present work, we need to take into account the imprecision coming from the ambiguous labeling, which appears in the form of 1 probabilistic guarantee of Proposition 7 In our imprecise probabilistie framework, 1 - a is the guarantee that we have on the true distribution Cat(An+1) belonging to the credal region P (see Proposition 5) But 1-c a is chosen by the user, so letting e.g. 1- a = 0.95 or 1 - a = 0.99 will yield a negligible coverage loss with respect to the classical conformal prediction setting. Tt. also worth noting that the guarantee in Proposition is the same derived for "standard" conformal prediction in the case of ambiguous labels in Stulz et al. 2023c Equation (43)).
in the
Algorithm Computing Imprecise Highest Density Set ISp,6
Input: Vector
Parameter: Significance level [0,
Output: THDS TSp,8
2 possible A's for | = K labels. Step 2 Sort the A's in ascending order of their lower probability P(A); if two or more sets have the same lower probability, put the one with lower cardinality first. Denote the sorted order as {A {Ap A Apps) )
Step For all A Ç Y, compute P(A) using equation
Step
for E f1,...,,21 ,2....,, 2k} do
if P(Apt) ) 2 1-1 then
18p,8 Ap break
end if end for return ISp,6
Finally, let us point out that we can use credal regions to quantify and disentangle between aleatoric and epistemic uncertainties (AT and EU, respectively) in the analysis at hand (Caprio et al.] 2024b). AU refers to the uncertainty that is inherent to the data generating process; as such, it is irreducible. EU, instead, refers to the lack of knowledge about the data generating process; as such, it is reducible. It can be lessened on the basis of additional data, e.g. by retraining the model using an augmented training set (Lin et al, 2024). On the other hand, since AU is irreducible, there is incrcasing need for ML techniques that are able to detect and flag excess of AU, so that the user can "proceed with caution".
Recall that, for a single Categorical distribution P Cat(A) on Y, the (Shannon) entropy is defined as
K H(P) = b Logg(As)
(6)
The credal versions of the Shannon entropy as proposed by Abellán et al. (2006): Hillermeier Waegeman p are H(P) = supper ((P) and H(P) = Pep H(P), called the upper and lower (Shannon) entropy, respectively. The upper entropy is measure of total uncertainty (TU), since it represents the minimum level of predictability associated with the elements of P. Tn Abellán et. al. (2006): Hüllermeier & Waegeman
9

---

Published in Transactions on Machine Learning Research (10/2024)
2021), ; the authors postulate that TU can be decomposed additively a sum of alcatoric and epistemic uncerta mis,ss and that the latter can be specified as the difference between upper and lower entropy, thus obtaining
H(P)
I(P)
+
H(P) - (PP)
(7)
total uncertainty TU(P) alcatoric uncertainty AU(P) epistemic uncertainty BU(P)
Other measures based on credal regions are also available (sec Bronevich & Rozenberg (022] Hofman et al. (2024p; Hüllermeier & Waegeman (2021) or Appendix B for a few examples) and they can be used in place of upper and lower entropy to quantify EU and AU within our credal region P, as long as the measure chosen for the total uncertainty is bounded. We also note passing that the decomposition in equation 7 is extremely important for the field of conformal prediction, since, as pointed out in (Hüllermeier & Waegeman 2021, Section 5); the role of aleatoric and epistemic uncertainties in (classical) conformal prediction is in general not immediately clear. Our approach allows to overcome this shortcoming.
That being said, there is an active ongoing debate around whether TU actually decomposes additively into AU and EU (Baan et al.] 2023: Gruber et al.] 2023 Kirchhof 2024 Ulmer 2024 Wimmer et al. 2023). Furthermore, Mucsányi et al. (2024) convincingly proved that uncertainty measures are very context- and task-spocific, and so even under the assumption of an additive dccomposition the choice of upper and lower entropy in equation is subject to the specific analysis that the user is carrying out. Because of this, we defer to future work looking for the "correct" measures for AU and EU, and how such choice depends the problem at hand.
Let us add a remark. In many modern-day ML and Al methodologies that allow to disentangle and quantify different types of uncertainties, the uncertainty quantification (UQ) part is not an intrinsic feature of the model, but rather something that is put "on top of" the main procedure. In contrast, uncertainty inherent to the method we propose, via the plausibility region c(Xn+1) (and so the credal region P). We are then able to quantify the amount of AU, and discern its types. They may even be used to build an abstaining option: if TU is "too high", the IHDS should not be returned, and instead the excess of which between AU and EU is responsible for the ambiguily should be reported.
It is worth noting that our method is not the only one exhibiting intrinsic UQ capabilities. Another notable class of such models are evidential deep learning ones (see c.g. Gao et al. (2024) for a comprchensive survey), in which uncertainty is modeled via second-order distributions, that is, using distributions over distributions. Second-order methods, though, have been recently shown to suffer from major pitfalls when used to quantify predictive EU due to their sensitivity to regularization parameters, and to underestimate predictive AU (Bengs et al., 2022 Pandey & Yu! 2023 Jürgens et al.) 2024). In addition, they are not immediately comparable to credal methods because the imprecise probabilistic apparatuses that underpin them are inherently different. Indeed, contrary to second-order methods, credal regions are (convex and closed) collections of first-order probabilities.
Let us also add another comment. It is true that the results in Cella & Martin (202aa b) could in principle be related to second-order methods. This because through the consonance assumption maxyey TTy) = they effectively make the upper probability II of the credal region ((II) associated to conformal prediction consonant plausibility function. In turn, the latter corresponds to a possibility function [Zadeh] [986). which is a sccond-order distribution. In this paper, instead, we do not need consonance to derive our credal region, and so the connection with second-order approaches, if it exists, is more subtle. We defer looking for it to future work.
5 Experiments
We verify the proposed algorithm on three datascts with ambiguous ground truth, including the toy and Dermatology DDx datasets from Stutz el al. 20023b cb ("Toy" and "Derm", respectively), and CIFAR-10H [Peterson et al. 2019b) ("Cifar10h"). For Derm, whose details are discussed in Appendix we use risk labels as classos, classifying cases into low, modium and high risk. For CIFAR- for computational convenience, we only consider data points whose annotated ground truth label in the original CIFAR-10 dataset is within the three classes (airplane, automobile, and bird). The toy dataset contains 1,000 data points, Dermatology
10

---

Published in Transactions on Machine Learning Research (10/2024)
DDx contains 1,947 data points, and CIFAR-IOH contains 3,000 data points. We split each dataset into random calibration and testing sets in a 50%-50% ratio. Furthermore, we use estimated classifier confidence scores as the conformity functions. Specifically, for the Toy dataset, we train single-layer MLP with 100 hidden neurons, achieving an accuracy of 77.5%. For CIF,R-10H, we employ a ResNet50 (He et al., 2013) model trained on the original CIFAR-10 training set, obtaining an accuracy of 93.6%. For the Derm datasel, we rely on the confidence scores collected from the testing set reported by (Stutz et al. 2023c).
Before going on, let pause here to add remark CIFAR-1OH. The latter was built upon the test set of the original CIFAR-10 dataset. To reflect human perceptual uncertainty, each image in CIFAR-1OH was annotated by 50 people. In our pre-processing, we first filtered out data points whose annotated labels in the original CIFAR-10 dataset were not included in (airplane, automobile, bird]. Then, for the remaining data points, we extracted the annotated probabilities of the three classes, and normalized them to sum up to one. For instance, let the original annotated probabilities be Pairplane: Pautomobile and Poird- Then, the normalized In addition to computational convenience, we focused on three classes only to be able depict our results in the unit simplex in R As a sanity check, we also computed the runtime for each testing point across different coverage levels using 5 random seeds. As can be seen from the plot in Appendix [ the average runtime for a 3-class problem is about 1.5 ms and 3.5 for a 5-class problem, which is reasonable incrcase.
annotated probabilitics for airplane is Paisplane Paatonotblll
Patrplane
We consider different miscoverage levels e, including 0.05, 0.1, 0.15, 0.2, 0.25 and 0.3. We denote the coverage level as 1 in our plots. For simplicity, we put the significance levels a and 8 to 1. We compare our method (denoted by "Ours (IS)" for "Improcisc Sct") Plausibility Reduced Predictive Set ((c(Xn-1)) proposod in (Stutz al. 2023c. Equation (41)) (denoted by "PRPS"). For Algorithm [ we construct the credal region c(Xu+1)) from equation by discretizing the simplex and computing the convex hull. We report three measures, including empirical distribution coverage, empirical label coverage, and average inefficiency. In particular, let {Ai...., } be the annotated label distributions on the N testing datapoints, {PI...., P be the constructed credal regions, [THDS],.... IHDSw} be the constructed Imprecise Highest Density Sets. The empirical distribution coverage is defined as
MEIA IPA E PNl,
(8)
where I[:] denotes the indicator function. The empirical label coverage is defined as
N K
Iik e IHDSy] x x
(9)
N
where Aly is the annotated probability for k-th class. The average inefficiency is defined as
N
Σ
(10)
N
n=1
where [IHDS,| is the cardinality of the n-th IHDS. To account for randomness, each experiment with 20 random seeds.
Empirical distribution and label coverage. First, we report the empirical distribution coverage levels on different datasets in Figure 2 Since our method and the conformal-based method utilize the same algorithm to compute distribution prodiction sets, we only report the true distribution coverage levels using our method. As shown in the figure, the empirical distribution coverage levels are equal to a, which is
consistent with Proposition 5
As shown in the figure, the true label coverage levels of both methods are above 1 -e,, which is consistent with the coverage guarantee in (Stutz et al. 2023c Equation (43) and Proposition 7 Both our method and the baseline served to obtain the coverage guarantee. Furthermore, the empirical coverage is higher than the expected level because, even if the true distribution is not fully contained within the specified credal region, the resulting high density set might still include the true label. This mechanism elevates the empirical coverage above (1 - a1 8).
Sccondly, we report the end-to-end empirical label coverage levels in Figure 3.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Distribution coverage
Toy
D stribution coverage
D Distribution coverage
Cifar10h
0.9
0.9
0.8
0.8
0.8
0.8
0.825
0.85
0.875
0.9
0.925
0.95
0.975
0.8
0.825
0.5
0.975
0.9
0.925
0.95
0.975
0.8
0.825
0.85
0.875
0.9
0.925
0.98
0.975
Coverage Level
Coverage Level
Coverage Level
Figure 2: Empirical distribution coverage levels. The left plot is on toy dataset; the middle plot is on Dermatology DDx dataset; the right plot is on CIFAR-1OH.
Inefficiency reduction. We report the inefficiencies using our method and the baseline in Figure 4 In our experiments, the inefficiency is defined as the average size of the resulting label prediction sets, with larger sizes indicating higher inelficiency. As shown in the plots, in most cases, our method can obtain lower inefficiencies than the baseline method. These results are consistent with Proposition 7 This implies that while both methods provide the true coverage guarantee, our method can construct smaller prediction sets that provide more information to the users. The outlier cases, where the baseline achieves lower inefficiency (e.g., 1- 0.8 on the Toy dataset), arise from discretization in the search for plausible distributions in P, potentially introducing additional conservativeness (indeed, calibration for 1 - slightly larger or lower than 0.8 removes this outlier in our experiments). Furthermore, the observed differences in inefficiency across datasets can be attributed to the variation and inherent ambiguity present within these datasets. For instance, the average inefficiency on CIFAR-10H lower due to the concentration of probability mass on the correct label for most data points, resulting in low inherent ambiguily. In contrast, when the probability mass is more evenly distributed across labels, it induces higher inherent ambiguity.
Let us add a remark on the differences between the datasets. They stem from the underlying ambiguity of the datasets. Compared to CIFAR-10H, the Dermatology DDX dataset is significantly more challenging by having more ambiguity in the ground truth labels. This produces sets that are more conservativ ive overall, leading to higher coverage at lower variance.
Toy
Derm
Cifar10h
1.00
1.00
1.00
Label coverage
Label coverage
Label coverage
Method
Ourstis)
0.95
0.99
0.95
PAPS
Method Ours((S) PAPS
Method PAPS
0.90
0.95
0.6
0.9
0.95
Coverage Level 1
Coverage Level 1
Coverage Level 1-e
Figure 3: True label coverage levels. The left plot is on the toy dataset; the middle plot is on Dermatology DDx; the third is on CIFAR-10H.
Effect of a and 8. According to Proposition 7 for specified coverage level 1 1-c, the probability P|Y.++ E 18p,s] holds for any combination of and 8 as long as they satisfy (1 - 8) - a) > - However, different combinations of and 5 can impact the inefficiency of the resulting prediction set. To study this effect, we consider E (0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4), select a on a grid in (0,e) with intervals of e/10, and compute the corresponding levels. Figure 5 shows the average inefficiency across various a and 8 combinations. The plot indicates that a has more significant effect on inefficiency; allowing higher a typically results in lower inefficiency (i.e., smaller prediction sets). Furthermore, given a (1-ar (1-a)(1 8) confidence level, variations in have more pronounced impact on the resulting inefficiency. Therefore, we
12

---

Published in Transactions on Machine Learning Research (10/2024)
Toy
Derm
Cifar10h
3.0
3.0
3.0
2.8
Prediction Set
Prediction get
2.5
Prediction
2.6
2.9
2.0
2.4
1.5
OursoS]
2.8
Ours(()
Ours(())
PRPS
1.0
0.6
0.65
0.95
0.65
0.95
0.6
0.65
0.1
0.95
Coverage 1evell-e
Coverage Level
Coverage Level 1
Figure 4: Average inelficiencies. The left plot is on the toy dataset; the middle plot is on Dermatology DDx; the third on CIFAR-10H.
recommend allocating more error tolerance to a while maintaining a reasonable 8. Developing a technique identify the optimal combination of a and 5 levels remains direction for future work.
0.40
0.35
0.30
0.25
10 0.20
0.15
0.10
0.05
0.00
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
2.0
2.2
2.4
2.6
2.8
3.0
Inefficiency
Figure 5: Inefficiency using different combinations of a and 6.
Qualitative analysis. We conduct a qualitative analysis of the constructed credal regions using two examples from the DDx dataset, with results summarized Table [ First, as shown the middle and right columns, the constructed eredal regions (right) encompass the ground truth label distributions (middle), which were annotated by domain experts. Second, as higher level demands greater coverage, the resulting credal regions become larger.
In addition, we also compute lower and upper entropy for both examples. For the first one, we have that AU(P) = H(P) = 0 and TU(P) = (P)) = 1.091, so that EU(P) = HPP) H(P) 1.091. For the second one, we have that AU(P) H(P) = 0 and TU(P) = H(P) = 0.991, so that EU = H(P) (((P)-H(P)>))) = 0.991. As we can see, in both cases the entire uncertainty faced by the user is of epistemic nature. The reason why this happens is that in both cases at least onc-hot encoding probability vector belongs to the credal region P. In particular, in both cases the credal regions contain the one-hot encoding vector for the label "Medium", X = (0, T, Then, given the current state of epistemic knowledge, we cannot exclude that if we were to collect enough extra data, we would end up finding a singleton credal region P = {P* = Cat()*)). In that case, we would have zcro alcatoric uncortainty, since we would be sure (with P* P'-probabilityy 1) that the label "Medium" the correct one for the new input. In turn, since for the time being we cannot exclude this best-case scenario, all the uncertainty encoded in the credal regions of reducible nature (EU).
6 Conclusion
In this paper, we address an important problem imprecise probabilistic machine learning, namely how empirically derive credal regions in a data-driven and efficient way, without any prior assumptions. To this end, build previous work by Stutz al. (202bb cb and apply conformal prediction in the probability
13

---

Limitations. Eliciting P requires exchangeability as consequence of using conformal approach: also requires the availability of calibration set. Tn addition, some subjectivity enters the analysis via the choice of the non-conformity measure for the conformal prediction methodology. Moreover, proven e.g. in Lei & Wassermani (2013), with finile samples, cannol give conditional version of the probabilistic guaranlee Published in Transactions on Machine Learning Research (10/2024)
Image
Ground Truth
Credal region (c 0.1)
Credal region (c 0.2)
High
0.2
0.0
Low
Medium
Medium
High
High
0.4
Labels
0: LoW
Medium
Low
1: Medium
Table 1: Qualitative Examples from the DDx Dataset: The left column presents the original medical images, the middle column shows the label distributions annotated by experts, and the right column displays the credal regions constructed using our method.
space. Given calibration set of examples with associated categorical distributions over classes, we explicitly construct credal regions that enjoy calibration property guarantecing the true data generating process being included with high probability. Besides being efficient, this approach also allows to deal with ambiguous ground truth, examples where the ground truth label is uncertain. By constructing imprecise highest density sets (Coolen 1992), we can derive predictive sets labels that are narrower, i.e., more efficient than those from Stutz we obtain credal regions without explicitly constructing the conformal transducer (i.e., computing p-values for all classes of the test example) and avoid the consonance assumption (stating that the p-value for the class ÿ that makes (#n+1, Ø) the "most conformal" to what we see in the calibration set has to be 1). Morcover, due to our calibration property, we do not need to assume that the true data generating process is included in the credal region. We also show that our credal region is [5/(1 a)]-type-2 valid, a version of a notion introduced in Cella & Martin
(2023c). Compared the seminal work of Cella Martin (2022a
2022b Definition 2).
in Proposition 7 Furthermore, the discretization involved in constructing credal regions can lead to high computational complexity, particularly when the label space is large. Finally, throughout the paper we (lacitly) assumed that the plausibility vectors X in the calibration set are available and accurate; this may not be the case in some applications (Stutz et al. 2023a).
References
Joaquín Abellán, George Jifí Klir, and Serafin Moral. Disaggregated total uncertainty measure for credal sets. International Journal of General Systems, 1(35):29 44, 2006.
Jayadev Acharya, Constantinos Daskalakis, and Gautam Kamath. Optimal Testing for Properties of Dis- tributions. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL httpss//pprocediings fwwes/ http:///.....aaaaooooo//ooo/ooaa
Paper .df
Thomas Augustin, Frank P. A. Coolen, Gert de Cooman, and Matthias C. M. Troffacs. Introduction to imprecise probabililies. John Wiley & Sons,, West Sussex, England, 2014.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. Stop measuring calibration when humans disagree. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1892 1915, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 10.18665//1122222.mmm 2022..mmllmmmainnmn 124. URL https://aclanthhooog. org 2022.
124
Joris Baan, Nico Dahcim, Evgenia Mia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. Uncertainty in natural language generation: From theory to applications, 2023. URL lhttps://arat //arxiv. www.aaiwww/aaaaaaeeeeeeee (org/abs/2300 15703
Viktor Bengs, Eyke Hillerreeer, and Willem Waegeman. Pitfalls of epistemic uncertainty quantification through loss minimisation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL https://openreviee: wttpppprrrrreee.ett/oruuummmmmia net
epjxT ARZW5]
Andrey G. Bronevich and Igor N. Rozenberg. Generalized Hartley measures on credal scts. In Andrés Cano, Jasper De Bock, Enrique Miranda, and Serafín Moral (eds.), Proceedings of the Twelvelh International Symposium on Imprecise Probability: Theories and Applications, volume 147 of Proceedings of Machine Learning Research, pp. 32-41. PMLR, 2021.
Michele Caprio. Imprecise Markov Semigroups and their Ergodicity.... arv preprint Xiv:220 00081 2024.
Michele Caprio and Ruobin Gong. Dynamic precise and imprecise probability kinematics. In Enrique Miranda, Ignacio Montes, Frik Quaeghebeur, and Barbara Vantaggi (eds.), Proceedings of the Therteenth International Symposium Imprecise Probability: Theories and Applications, volume 215 of Proceedings of Machine Learning Research, pp. 72-83. PMLR, 11-14 Jul 2023.
Michele Caprio and Sayan Mukherjee. Ergodic theorems for dynamic imprecise probability kinematics. International Journal of Approzimate Reasoning, 152:325-343, 2023a.
Michele Caprio and Sayan Mukherjee. Extended probubilities and their application to statistical inference, chapter 2. World Scientific, 2023b.
Michele Caprio and Toddy Seidenfeld. Constriction for sets of probabilities. In Enrique Miranda, Ignacio Montes, Erik Quaeghebeur, and Barbara Vantaggi (eds.), Proceediings of the Thirleenth International Symposium on Imprecise Probability: Theories and Applications, volume 215 of Proceedings of Machine Learning Research, pp. 84-95. PMLR, 11-14 Jul 2023. URL https://proceeedigs. mmr...gss........mmmmmmm.....m.. waa...presssv1...
caprio23b. html
Michele Caprio, Souradeep Dutta, Kuk Jang, Vivian Lin, Radoslav Ivanov, Oleg Sokolsky, and Insup Lee. Credal Bayesian Deep Learning. Transactions of Machine Learning Research, 2024a.
Michele Caprio, Kuk Jin Jang, Souradeep Dutta, Shireen K. Manchingal, Fabio Cuzzolin, Oleg Sokolsky, and Insup Lee. Credal and Interval Deep Evidential Classifications. Presented at WUML 2024, 2024b.
Michele Caprio, Yusuf Sale, Eyke Hüllermeier, and Insup Lee. A Novel Bayes' Theorem for Upper Probabil- ities. In Fabio Cuzzolin and Maryam Sultana (eds.), Epistemic Uncertainty in Artificial Intelligence, pp. 1-12, Cham, 2024c. Springer Nature Switzerland.
Michele Caprio, Maryam Sultana, Eleni Elia, and Fabio Cuzzolin. Credal Learning Theory. NewrIPS 2024, 2024d.
Leonardo Cella and Ryan Martin. Valid inferential models for prediction in supervised learning prob- lems. International Journal of Approzimate Reasoning, 150:1-18, 2022a. ISSN 0888-613X. doi:
http://///0..0111111110//// wtw.://........2...0... 2022.08.00.. URL http:://ww..ccieeddrrrc...c http://ddd..cccccccc/c////eetaaa piJ/808888613XX220113xx
Loonardo Cella and Ryan Martin. Validity, consonant plansibility measures, and conformal prodiction. Inter- national Journal of Approzimate Reasoning, 141:110-130, 2022b. Probability and Statisties: Foundations and History. In honor of Glenn Shafer.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Siu Lun Chau, Antonin Schrab, Arthur Gretton, Dino Sejdinovic, and Krikamol Muandet. Credal two-sample tests of epistemic ignorance. Xiv preprint orXin:24 10. 12921, 2024.
Frank P. A. Coolen. Imprecise highest density regions related to intervals of measures. Memorandum COSOR, 9254, 1992.
A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Stalistionl Society (JRSS), 28(1):20-22) 1979.
Luis M. de Campos, Juan F. Huete, and Serafin Moral. Probability intervals: tool for uncertain reasoning, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 2(2):167-196, 1994.
Bruno de Finetti. Theory of Probability, volume 1. New York Wiley, 1974.
Bruno de Finetti. Theory of Probability, volume 2. New York Wiley, 1975.
Thierry Denoeux. A neural network classifier based on Dempster-Shafer theory. IEEE Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans, 30(2):131-150, 2000.
S. Destercke, D. Dubois, and E. Chojnacki. Unifying practical uncertainty representations i: Generalized p-boxes. International Journal of Approcimate Reasoning, 49(3):649-663 2008. ISSN 0888-613X. doi: hutp:///boi.rrg/10001////11 2008.07.00. URL p11/80888613X08011447
Souradeep Dutta, Michele Caprio, Vivian Lin, Matthew Cleaveland, Kuk Jin Jang, Ivan Ruchkin, Oleg Sokolsky, and Insup Lee. Distributionally Robust Statistical Verification with Imprecise Neural Networks. arXiv preprinl ar Xio:2300 14815 2023.
Alvan R. Feinstein and Dominic V. Cicchetti. High agreement but low kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43 6:543-9, 9, 1990.
Junyu Gao, Mengyuan Chen, Liangyu Xiang, and Changsheng Xu. A comprehensive survey on evidential deep learning and its applications, 2024. URL httts:://arriiii http://aaxxvvv.///////// wwwgoogg///////e.eee/..ooeeeeeee 047200
Rui Gao, Liyan Xie, Yao Xie, and Huan Xu. Robust hypothesis testing using wasserstein uncortainty sets. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL http::///pppeeeeee..ee.........s httapper...eeeppppp//////////// 44652222888778533544777677781183
Cornelia Gruber, Patrick Oliver Schenk, Malte Schierholz, Frauke Kreuter, and Göran Kauermann, Sources of uncertainty in machine learning statisticians' view, 2023. URL hhttts://aariiiii wwt.:/rrg////////eeeeee
abs 2atp./235..111 16703.
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL
03385
Paul Hofman, Yusuf Sale, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty: A credal approach. In ICML 2024 Workshop Structured Probabilistic Inference (166 Generative Modeling, 2024.
Peter J. Huber and Elvezio M. Ronchetti. Robust statistics. Wilcy Scries Probability and Statistics. Hoboken, New Jersey Wiley, 2nd edition, 2009.
Rob J. Hyndman. Computing and graphing highest density regions. 120-126, 1996.
The American Statistician, 50(2):
Eyke Hillermeier and Willem Wacgeman. Alcatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Machine Learning, 3(110):457-50 -066, 2021.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Alireza Javanmardi, Yusuf Sale, Paul Hofman, and Eyke Hiilermeier. Conformal prediction with partially labeled data. In Harris Papadopoulos, Khuong An Nguyen, Henrik Boström, and Lars Carksson (eds.); Proceedings of the Twelfth Symposium on Conformal and Probabilistic Prediction with Applications, volume 204 of Proceedings of Machine Learning Research, pp. 251-266. PMLR, 13-15 Sep 2023. URL https: 7/proceedings. www. pt....rree///20// 2 javanmardi23a.... html
Alireza Javanmardi, David Stutz, and Eyke Hüllermeier. Conformalized credal set predictors. In Proceedings of the 98th International Conference on Neural Information Processing Systems. NeurIPS, 2024.
Mira Jürgens, Nis Meinert, Viktor Bengs, Eyke Hüillermeier, and Willem Waegeman. Is epistemic uncertainty laithfully represented by evidential deep learning methods? Available al arXiv:2402.0900 09006 2024.
Michael Kirchhof. Uncertainties of latent representations in computer vision, 2024.
URL. https://
publikationen.
Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Stalistical Sociely Series B: Stalistical Methodology, 76(1):71-900 1996 07 2013.
Isaac Levi. The Enterprise of Knowledge. London, UK MIT Press, 1980-
Julian Lienen, Caglar Demir, and Eyke Hüllermeier. Conformal credal self-supervised learning. Available at or ar0iv:2205. 15239999995299
2023.
Vivian Lin, Kuk Jin Jang, Souradeep Dutta, Michele Caprio, Oleg Sokolsky, and Insup Lee. DCAL: Dis- tribution shift recovery via data-driven control for deep learning models. In Alessandro Abate, Mark Cannon, Kostas Margellos, and Antonis Papachrisstodoulou (eds.), Proceedings of the 6th Annual Learning for Dynamics 6amp; Control Conference, volume 242 of Proceedings of Machine Learning Research, pp. 1526 1538, PMLR, 15-17 Jul 2024. URL https: wttp...p.....//////////2222. html
Xing Liu and François- Prançois-Xavier Briol. On the Robustness of Kernel Goodness-of-Fit Tests. Available at arXiv:2408.0580 2024.
Yuan Liu, Ayush Jain, Clara Eng, David H. Way, Kang Lee, Peggy Bui, Kimberly Kanada, Guilherme de Oliveira Marinho, Jessica Gallegos, Sara Gabriele, Vishakha Gupta, Nalini Singh, Vivek Natarajan, Rainer Hofmann- Hofmann-Wellenhof,, Gregory S. Corrado, Lily H. Peng, Dale R. Webster, Dennis Ai, Susan Huang, Yun Liu, R. Carter Dunn, and David Coz. A deep learning system for differential diagnosis of skin diseases. Nature Medicine, 26:900-900 10000090,,000 2020.
Pengyuan Lu, Michele Caprio, Eric Eaton, and Insup Lee. IBCL: Zero-shot Model Generation for Task Tradc-offs Continual Learning. arXiv preprint arXiw:2305.14788 2024.
Ryan Martin. Valid and efficient imprecise-rrobablistae inference with partial priors, II. general framework. Available at arXic:2211.14507 2023.
Vitor Martin Bordini, Sébastien Destercke, and Benjamin Quost. Learning calibrated belief functions from conformal predictions. In Enrique Miranda, Ignacio Montes, Erik Quacghebeur, and Barbara Vantaggi (eds.), Proceedings of the Thirleenth International Symposium on Imprecise Probobility: Theories and Applications, volume 215 of Proceedings of Machine Learning Research, pp. 311-320 PMLR, 11-14 14 Jul 2023. URL https::/ mlr. wtwpae/s.wrrrr.eree/em titesss/211/maavee http1//maaainnoooriii222222. www.booida.dd...onooeooooo html.
Mary L. McHugh. Interrater reliability: the kappa statistic. Biochemia Medica, 22:276 282, 2012.
Thomas Mortier, Viktor Bengs, Eyke Hüllermeier, Stijn Luca, and Willem Waegeman. On the calibration probabilistic classifier sets. In Francisco Ruiz, Jennifer Dy, and Jan Willem van de Meent (eds.).
Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206
of Proceedings of Machine Learning Research, pp. 8857-8870. PMLR, 25-27 Apr 2023. URL https: 7/proceedings.. mlr. wtt.e/aiing...m....reeee//////ee wttppp/ss///////rreieeeeeee
17

---

Published in Transactions on Machine Learning Research (10/2024)
Bálint Muesányi, Michael Kirchhof, and Scong Joon Oh. Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. In The Thirly-eight. Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreevier. www.:evviiiiieeeennunuuuum.u
x8RgF2xQT]
Yixin Nie, Xiang Zhou, and Mohit Bansal. What can we learn from collective human opinions on natural language inference data? In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9131 9143, Online, November 2020. Association for Computational Linguistics. doi: 10. http:////200../mlppp/ 734. URL https: https/////claahhoog/o// .rg.eoogggoo. 2020. emnlp-main. 734
Deep Shankar Pandey and Qi Yu. Learn to accumulate evidence from all training samples: Theory and practice. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 26963 26989. PMI.R, 23 29 Jul 2023. URL
https://proceedinggs
www.ing..mmm........m.......m mlr, wwwppppwwe.ee/pepp 2rress/v222/paade223aa.. html
Ellic Pavlick and Tom Kwiatkowski. Inherent disagroements in human textual inferences. Transactions of the Associalion for Compulational Linguislics (TACL), 7:677 694, 2019.
Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Grilfiths, and Olga Russakovsky. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019a.
Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. Human uncertainty makes classification more robust, 2019b.
Barbara Plank. The "problem" " of human label variation: On ground truth in data, modeling and evaluation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 10671 10682, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/vl, 10022.mmmlp-mastt main.7n...... URL org/2022
main. 731
Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Bobby Kleinberg, Sendhil Mullainathan, and Jon Kleinberg. Direct uncertainly prediction for medical second opinions. In Proc. of the International Conference on Machine Learning (ICML), 2019.
Yusuf Sale, Michele Caprio, and Eyke Hüllermeier. Is the volume of a credal set good measure for epistemic uncertainty? In Robin J. Evans and llya Shpitser (eds.), Proceedings of the Thhrty--Nntth Conference Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 1795- 1804. PMLR, 31 Jul-Oe 04 Aug 2023.
Yusuf Sale, Viktor Bengs, Michele Caprio, and Eyke Hüllermeier. Second-order uncertainty quantification: A distance-based approach. In Ruslan Salakhutdinov,, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 43060-43076. PML., 21-27 Jul 2024. URL htttps://proceeding:: mlr mataas/v335//aaaa
html
Mike Schaekermann. Human-Al Interaction in the Presence of Ambiguity; From Deliberation-basee Labeling to Ambiguity-awar AI. Ph.d. thesis, University of Waterloo, 2020.
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9:371 9:371-421, 2008
Padhraic Smyth, Usama M. Fayyad, Michael C. Burl, Pictro Perona, and Pierre Baldi. Inferring ground truth from subjective labelling of venus images. In Advances in Neural Information Processing Systems (NeurIPS), 1994.
18

---

Published in Transactions on Machine Learning Research (10/2024)
David Stutz, Ali Taylan Cemgil, Abhijit Guha Roy, Tatiana Matejovicova, Melih Barsbey, Patricia MacWilliams, Mike Schackermann, Jan Freyberg, Rajeev Rikhye, Beverly Freeman, Javier Perez Matos, Umesh Telang, Dale Webster, Yuan Liu, Greg Corrado, Yossi Matias, Pushmeet Kohli, Yun Liu, Arnand Doucet, and Alan Karthikesalingam. Evaluating ai systems under uncertain ground truth: a case study dermatology. or Xiv.oorg abs/ 2200/3377, 2023a.
David Stutz, Abhijit Guha Roy, Tatiana Matejovicova, Patricia Strachan, Ali Taylan Cemgil, and Arnaud Doucet. Conformal prediction under ambiguous ground truth. Transactions on Machine Learning Re- search, 2023b. ISSN 2835-8856. URL |atzps://openrevien
David Stutz, Abhijit Guha Roy, Tatiana Matejovicova, Patricia Strachan, Ali Taylan Cemgil, and Ar- naud Doucet. Conformal prediction under ambiguous ground truth (version 1). arXiv preprint 0rXiv:2307.093020 2023c.
Dennis Ulmer. On uncertainty in natural language processing, 2024. URL http:///axxiv..r////////// 03446
Peter Walley. Statistical Reasoning with Imprecise Probabililies, volume 42 of Monographs on Statislies and Applied Probability. London Chapman and Hall, 1991.
Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures? In Robin J. Evans and Πya Shpitser (eds.), Proceedings of the Thirly-Ninll, Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pp. 2282- 2292. PMLR, 31 Jul 04 Aug 2023. URL htttps://procceednggs. mmr. mlr, wwwpppesss/////ee http://222/wwmmrm2222.22 html wtt.eeeddinnaa.....rress//////
Yan Yan, Rómer Rosales, Glenn Fung, Subramanian Ramanathan, and Jennifer G. Dy. Learning from multiple annotators with varying expertise. Machine Learning, 95(3):291 327, 2014.
Lotfi A. Zadeh. A Simple View of the Dempster-Shafer Theory of Evidence and its Implication for the Rule of Combination. AT Magazine, 7(2):85 90, 1986. doi: hups://doi.org/ 10/gg/10.000 10..169//imag.v/./ v722.542. URL wwt.::.wwwww.............mm wwt./////////////////ooooooooaat 10. 1609/ 1609//imag..a. v712.54)
Marco Zaffalon. The naive credal classifier. Journal of Statistical Planning and Inference, 105(1):5 22, 2002. Imprecisc Probability Models and their Applications.
Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, and Reynold Cheng, Truth inference in crowdsoure- ing: Is the problem solved? Proc. of the VLDB Endowment, 10(5):541 552, 2017.
A Differences Between P Pagg and P
Following the argumentation Stutz et al. 2023b Section 3.1), the first step to realize that reality, we always deal with Pagg rather than P. This is bocause most cases, our labels are derived from annotations rather than from some objective process. Even if ground truth is obtained from a measurement not involving human annotation, there might be noise and errors in these measurements. With this realization, the best we can do in terms of coverage guarantee is to guarantee coverage with respect to Pagg (since we do not know and will likely never know the true underlying P). In other words, performing well against IP Pags means performing well against expert annotators. In light of being unavailable, performing as good as the expert annotations the best we can ever hope for.
Mathematically, we can model Pags using an annotation process as described in detail in the last two paragraphs of Stutz et al. (2023b) Section 3.1).
19

---

Published in Transactions on Machine Learning Research (10/2024)
B
Uncertainty Quantification
Other measures for AU and EU are also available in the context of credal regions (Bronevich & Rozenberg, 2021 Hofman et al. 2024 Hüllermeier & Waogeman 2021), and they can be used in place of upper and lower entropy lo quantity EU and AU within our credal region P, as long as the measure chosen for the total uncertainty is bounded. We chose upper and lower entropy because of their ease of computation. In fact, let us give a case in which the quantities in equation 7 can be easily calculated bounded. Let exP denote the extreme elements of P, that is, those elements that cannot be written convex combination of one another. We have that p = Cony(e and Conv(.) denotes the convex hull operator. Then, the following was proven in (Caprio et al. 2024b Theorem 8).
Proposition 8. Suppose that |exP] S < 00. Let
H(P = min H(P
PrE
and
H(P
max H(Pm)).
Eeep
Let
[TU(P)] = max
sup
Σ s=1
B. (( HH(pe))
SEA
then
TU(P) (TUUP),
Σ
B. (P.t) + 1og2(()
sup
BEA s=1
AU(P) H(P
EU(P)
0,[TU(P) - H(Pen)
B, (P.(P.T)) + logg(() HPe
sup -EAS-1
Calculating [[(Pex) and H(Px) is immediate from equation
On the other hand, the supremum
sup3eas-1 Σ.= (P.( can be computed in polynomial time
C Dermatology Dataset Details
In Section by studying the Dermatology DDX dataset, we follow (Liu et al. 2020 Stutz et al. 2023a and consider very ambignous as well as safety-critical application in dermatology: skin condition classifi- cation from multiple images. Such dataset from Liu el al. (2020). and consists 1949 test examples and 419 classes with up to 6 color images resized 448 448 pixels. The classes, i.e., conditions, were annotated by various dermatologists who provide partial rankings. These rankings are aggregated deterministically obtain the plausibilities À using the inverse rank normalization procedure of (Liu et al. 2020) described in [tutz et al. 2023b Section 3.1).
D Runtime of CIFAR-10H with More Classes
As a sanity check, for CIFAR-10H we computed the runtime for each testing point across different. coverage levels using random seeds. As can be seen from the plot below, the average runtime for 3-class problem is about 1.5 ms and 3.5 for a 5-class problem, which we doem to be a reasonable incrcase.
We also point out how bounds for TUPP), AU(P), and EU(P) in terms of the entropy of the lower probability and of the entropy of the upper probabikiy, H(P) and HP)). respectively and (P),, respectively were given CCaprio al. 2024a Theorem 13).
be confused with the lower and upper entropy H(P)
20

---

Published in Transactions on Machine Learning Research (10/2024)
12
10
Time (ms)
8
6
4
2
3-class
5-class
Number of Classes
21


Reviews:
[{"review_id": "ymi2Pm2UWj", "reviewer": "Reviewer_6HLc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-----------\n\n* The paper is mostly well-written, includes formal proofs and gives intution about some of the theoretical concepts, and is thus a very educational resource on the topic.\n* The contribution is an important step towards credal sets with statistical guarantees.\n\nWeaknesses\n--------------\n\n* The paper is missing a more explicit review of related work in credal sets, conformal prediction, and potentially also evidential deep learning.\n* The authors reiterate a postulated decomposition into aleatoric and epistemic uncertainty amnd claim that their method is well-suited for this decomposition, but it is never tested in experiments in the paper.\n* Some of the experimental results are not discussed in a lot of detail.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors provide a nove method to create conformalized credal regions, i.e. convex sets of distributions on the probability simplex that contain the true (ambiguous) distribution for an input in expectation. To this end, they build on former work on Conformal plausibility regions, which combined with concepts from imprecise probabilities yields conformalized credal regions. They provide formal motivations and proofs for their idea and validate it on three datasets, including a toy and a Dermatology dataset and CIFAR-10H.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* I would very much like to see a more comprehensive related work section on similar works in conformal prediction and credal sets, with potential connections to evidential deep learning as another family of second-order methods, if applicable.\n* Test the disentanglement of uncertainty into aleatoric and epistemic using equation 6 empirically. This could for instance be tested using misclassification or OOD detection as a proxy task. The authors claim their method is well-suited for this disentanglement (see contributions paragraph in section 1), but no empirical evidence is given to support this claim.\n   * This part might also warrant a slight critique of the aleatoric / epistemic decomposition, the limits of which have been argued from a multitude of perspective, see for instance the works by [1-5]\n   * I don't fully agree with the remark at the end of section 5; while I (think i) understand its intention, I think this remark is a bit too general. You could argue that methods like the aforementioned evidential deep learning also come with uncertainty \"built in\", and thus I would request the authors to sharpen this statement and provide more details / references about the types of methods they criticize here.\n* Explain some of the empirical results in more detail. \n  * As far as I can see, it was not reported which underlying models and conformity function were used in the experiments in section 6.\n  * I would also like to understand better why there seems to be a gap between desired and actual coverage in the plots in Figure 3, the potential reasons for the wide distribution of inefficiency values in the third plot in Figure 4 for 0.7 desired coverage.\n  * The small number of datapoints used for experiments (<= 3k) and the reduction of the number classes in CiFAR10H to three for \"computational convenience\" makes me suspect that the expontential number of searches in algorithm 1 is computationally prohibitive, which should be reported in the limitations section.\n* Some additional information / intuition on how to choose $\\alpha$ and $\\delta$ in practice beyond the analysis in section 6.\n* Some minor typos / issues: \n    * Uppercase \"equation\" at the end of the first paragraph in 4.1 and another spots in the paper.\n    * In the proof of proposition 4, I would suggest to replace the notation $\\lambda^1$ with $\\lambda^{(1)}$ to avoid confusion. Also, I believe that in the same proof the quantity $\\lambda^*$ is not properly introduced or explained.\n    * Some of the citations including page or section numbers feel a bit wonky, take for instance page 8 \"By (Coolen, 1992, Page 3)\" when it feels more natural to have \"By Coolen (1992), p. 3\" or similar. This occurs multiple times throughout the paper.\n    * In the equation under this exact citation, I believe a \"P({y})\" was erroneously denoted as \"P(y)\".\n    * The last paragraph in section 6 seems a bit to superficial to be called qualitative analysis, although I really like the examples shown.\n* I am not sure if this technically violates the requirements of TMLR, but the appendix before the bibliography feels a bit unnatural to me.\n* The authors cite Stutz (2023a;b;c) as work learning from ambiguous ground truth. However, ambiguous ground truth has also been studied in many works in Natural Language Processing already [6,7], with a more comprehensive account in [8], and examples in computer vision as well [9].\n\n[1] Baan, Joris, Nico Daheim, Evgenia Ilia, Dennis Ulmer, Haau-Sing Li, Raquel Fernández, Barbara Plank, Rico Sennrich, Chrysoula Zerva, and Wilker Aziz. \"Uncertainty in natural language generation: From theory to applications.\" arXiv preprint arXiv:2307.15703 (2023).\n\n[2] Gruber, Cornelia, Patrick Oliver Schenk, Malte Schierholz, Frauke Kreuter, and Göran Kauermann. \"Sources of Uncertainty in Machine Learning--A Statisticians' View.\" arXiv preprint arXiv:2305.16703 (2023).\n\n[3] Kirchhof, Michael. \"Uncertainties of latent representations in computer vision.\" arXiv preprint arXiv:2408.14281 (2024).\n\n[4] Ulmer, Dennis. \"On Uncertainty In Natural Language Processing.\" arXiv preprint arXiv:2410.03446 (2024).\n\n[5] Wimmer, Lisa, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. \"Quantifying aleatoric and epistemic uncertainty in machine learning: Are conditional entropy and mutual information appropriate measures?.\" In Uncertainty in Artificial Intelligence, pp. 2282-2292. PMLR, 2023.\n\n[6] Nie, Yixin, Xiang Zhou, and Mohit Bansal. \"What can we learn from collective human opinions on natural language inference data?.\" arXiv preprint arXiv:2010.03532 (2020).\n\n[7] Baan, Joris, Wilker Aziz, Barbara Plank, and Raquel Fernandez. \"Stop measuring calibration when humans disagree.\" arXiv preprint arXiv:2210.16133 (2022).\n\n[8] Plank, Barbara. \"The'Problem'of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation.\" arXiv preprint arXiv:2211.02570 (2022).\n[9] Peterson, Joshua C., Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. \"Human uncertainty makes classification more robust.\" In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9617-9626. 2019.", "broader_impact_concerns": "I foresee no direct ethical implication with the reviewed work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "P0vBoJsLIe", "reviewer": "Reviewer_9sQe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strong points**\n\nThe submission connects the fields of conformal prediction and imprecise probability. This is very good.\n\nThey show a way to deal with label noise in conformal prediction. This is interesting.\n\nThe conclusions section 7 honestly reports the merits of the paper. Overall the paper seems fair and well representing the contents of it.\n\nThe experimental section corroborates the claims of the paper.\n\n**Weak points**\n\nThe presentation is a little convoluted to me, (coming from a background in conformal prediction, not Imprecise Probability). It seems like the authors introduce a lot of theory and concepts that are not really needed to show the main result of Proposition 7.\n\nThe algoroithm to compute the prediction sets are exponential in the number of labels. In numerical experiments, only 3 labels are considered. This is a very severe limitation, since the application of machine learning to healthcare (mentioned in the introduction) may well use more than hundreds of labels when predicting diagnoses. A discussion on the practical usefulness of the algoriothm would strengthen the work, although arguably falling outside the claims of the article. \n\nThe discussion about aleatoric and epistemic uncertanity is quite weak. It discusses this only in the narrow sense of using upper and lower shannon entropy to quanify these uncertanities. It is not investigated in numerical experiments. I think the paper would improve in focus and not lose much by dropping that section. Still - I think that the paper clearly delineates this discussion from the rest, and that the authors do not oversell their claims. Thus, I don't see a big problem with keeping it either.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the task of producing prediction sets for classification tasks with K classes, and the training set has label noise as quantified by plausibility vectors that are assumed accurate and available.\n\nA direct application of conformal prediction produces prediction sets over label distributions with coverage level $1-  \\alpha$. Previous works has reduced this into a prediction set over labels by taking the union over Highest Density Sets (equation 2 and proof for Proposition 7). The authors instead use of Imprecise Highest Density Sets of level $1-\\delta$, which gives potentially improved tightness, at the cost of  controllable miscoverage. \n\nThe derivation is connected tightly to the litterature of Imprecise Probability, and framed in terms of Credal Regions. The authors show that the prediction set produced by the conformal prediction step is a credal region with typ 2 validity. Being a credal set, it unlocks theory for analyzing aleatoric vs epistemic uncertanity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Crititcal changes**\n\nThe phrasing in the abstract \"a closed and convex family of probabilities on the output space\" threw me off initially. I think many other readers will be similarly startled. More precisely would be stating \"a closed and convex family of probability measures on the output space\", or similar.\n\nRewrite the proof of proposition 6. It was hard to read due to its succintness, and I believe there is a minor mistake in it due to a factor $1-\\alpha$ being dropped.\n\nI believe that the definitions of upper and lower shannon entropy should have symbols $\\bar H(\\mathcal P)$ not $\\bar H(P)$ due to being functions on sets of measures.\n\n**Changes that would strengthen the work**\n\n\"Inefficiency\" in the experiment seems like a misnomer. A better measure of inefficiency would be the number of excess labels in the prediciton set, compared with an oracle. This is doable for the toy data set. I recommend you to do this. Keep the prediction set size that you have now computed, but rename it to \"Prediction set size\" or similar.", "broader_impact_concerns": "I don't think one is needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vK53vvMJRr", "reviewer": "Reviewer_DMPv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Overall, the paper is well-written.\n* Good description of relevant background on Imprecise Probability.\n* The discussion of how conformalized credal regions can be used to disentangle aleatoric and epistemic uncertainty is very interesting. \n* I appreciate the specificity when referring to related work (the section and equation numbers are helpful).\n\nWeaknesses: The experiments section could be improved. See Requested Changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper shows that if you run conformal prediction on “plausibilities” (which can be thought of representing P(Y|X_i)), then the resulting sets of distributions are credal regions, meaning that they are convex and closed. Taking the Imprecise Highest Density Set of such a credal region can yield smaller prediction sets compared to the method of Stutz et al. 2023c (shown empirically) while not compromising coverage significantly (shown theoretically in Prop. 7 and also empirically).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Experiments section: \n* I’d like to see a more detailed description of how the plausibilities for each dataset are generated (could go in the Appendix). Were they expert annotators? How many annotators? For CIFAR-10, how did you deal with probability mass assigned to classes you did not include? \n* You say that “for computational convenience” you only run the experiments using 3 classes — can your algorithm not be easily run with more than 3 classes? Can you give a sense of run time (minutes? hours?) of your current experiments? I’d like to see experiments using more classes if it is not prohibitively hard. If it is too hard, then you should acknowledge the computational burden of your method in your Limitations paragraph at the end of the paper. \n* Can you write out mathematical definitions of the metrics you use? I presume that “empirical distribution coverage” refers to the fraction of times the credal region contains $\\lambda_i$ and “empirical label coverage” refers to the fraction of times the IHDS derived from the credal region contains $Y_i$. \n* I’d like to see the coverage plot with a dotted y=x line to make it easier to compare the desired coverage level to the realized coverage level. \n* Do you have explanations for the cause of differences between the different datasets? For example, why does the Derm dataset have such high label coverage and such small variance in set sizes as we vary the desired coverage level? \n\nSmall changes: \n* I don’t believe “this allows to represent ” is grammatically correct (bottom of p. 2). Perhaps “this allows us to represent” or “this allows the representation of.” The same applies to the first paragraph of Sec 4.1\n* Eq (2) - has $\\delta$ been defined by this point? You can just add “for some user-chosen $\\delta \\in [0,1]$.\" \n* Eq (2) requires quite a bit of effort to parse, so it would be good to describe it in words or provide an example of how a PRPS is constructed given a plausibility region. \n* In the Experiments section, why is PRPS called “[49] PRPS”?\n\nSome light suggestions. (Feel free to follow them if you think they make sense, but they are not required changes.) \n* As mentioned above, the discussion of using the credal regions to identify aleatoric vs. epistemic uncertainty was great. If there are any compelling case studies/examples that can be extracted from the experiments you ran, that would be interesting to see. Perhaps this can be applied with the qualitative analysis in Table 1. \n* You could consider creating combining Sec 2 and Sec 3 into one section called “Background.” That would make it a bit easier to distinguish between existing work and your contributions.  \n* I would appreciate some more discussion of how to interpret the probabilistic guarantees if $P_{\\mathrm{agg}}$ is significantly different from the true data-generating $P$ and whether this knowledge should affect how we choose to “split” $\\epsilon$ between $\\delta$ and $\\alpha$. \n* Are there any non-conformal methods for generating credal regions that are relevant to compare against in your experiments?", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers agree that the paper provides sufficient support for its claims
and is of high enough quality to be relevant to the TMLR readership. 
I share their opinion and therefore recommend acceptance.
Decision: Accept (poster)