Credal Prediction based on Relative Likelihood
Authors: Timo Löhr
Paul Hofman
Felix Mohr
Eyke Hüllermeier
Venue: NeurIPS 2025 spotlight ()
Keywords: Uncertainty Representation
Credal Set
Relative Likelihood
Primary area: probabilistic_methods
TL;DR: This paper proposes a principled method to construct credal sets using relative likelihood and ensemble learning, achieving improved uncertainty representation without sacrificing predictive accuracy.

Abstract: Predictions in the form of sets of probability distributions, so-called credal sets, provide a suitable means to represent a learner's epistemic uncertainty. In this paper, we propose a theoretically grounded approach to credal prediction based on the statistical notion of relative likelihood: The target of prediction is the set of all (conditional) probability distributions produced by the collection of plausible models, namely those models whose relative likelihood exceeds a specified threshold. This threshold has an intuitive interpretation and allows for controlling the trade-off between correctness and precision of credal predictions. We tackle the problem of approximating credal sets defined in this way by means of suitably modified ensemble learning techniques. 
To validate our approach, we illustrate its effectiveness by experiments on benchmark datasets demonstrating superior uncertainty representation without compromising predictive performance. We also compare our method against several state-of-the-art baselines in credal prediction.

Credal Prediction based on Relative Likelihood
Timo Löhr* LMU Munich, MCML. timo. www.oooeerrfffff.moom 1mu. de
Paul Hofman"
LMU Munich, MCMI.
paul hof.am@iffi..... 1mu de
Felix Mohr
Eyke Hüllermeier LMU Munich, MCML, DFKI eykečlmu.de de
Universidad de La Sabana felix, mohhr@nisabann edu.
Abstract
Predictions in the form of of probability distributions, so-called credal sets, provide a suitable means to represent learner's 's epistemic uncertainty. In this paper, we propose theoretically grounded approach to credal prediction based on the statistical notion of relative likelihood: The target of prediction is the set all (conditional) probability distributions produced by the collection of plausi- ble models. namely those models whose relative likelihood exceeds specified threshold. This threshold has an intuitive interpretation and allows for controlling the trade-off between correctness and precision of credal predictions. We tackle the problem approximating credal sets defined in this way by means of suitably modified ensemble learning techniques. To validate our approach, we illustrate its effectiveness by experiments on benchmark datasets demonstrating superior uncertainty representation without compromising predictive performance. We also compare our method against several state-of-the-an baselines credal prediction.
Introduction
The distinction between two types of uncertainty, referred to as aleatoric and epistemic, is receiving increasing interest in machine learning [Hüllermeier and Waegeman, 2021]. Roughly speaking, the aleatoric uncertainty of predictive model is caused by the inherent randomness of the data-genee generating process, whereas epistemic uncertainty is caused by the learner's lack of knowledge about the true (or best) predictive model. While aleatoric uncertainty is irreducible, epistemic uncertainty can be reduced on the basis of additional information, e.g., by collecting more training data.
Alcatoric uncertainty can be captured adequately in terms of (conditional) probability distributions, whereas the representation of epistemic uncertainty requires "second-order" formalisms more general than (single) probability distributions. In the Bayesian approach to machine learning, second-order probability distributions are used for this purpose [Depeweg et al., 2018, Kendall and Gal, 2017]:
The learner maintains a probability distribution over the model space, which, if each model is probabilistic predictor, results predictions in the form of probability distributions of probability distributions (on outcomes). The Bayesian approach is theoretically appealing but computationally demanding, and commonly criticized for the need specify a prior distribution (strongly influencing prediction and uncertainty quantification) [Gawlikowski et al., 2023].
An alternative second-order formalism offered by credal sets [Walley, 1991], i.c., (instead of distributions) of probability distributions that are commonly assumed to be closed and convex [Cozman, 2000J. Although a appears provide weaker information than distribution, based representation also has advantages. In particular, it can be argued that sets are more apt at
equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Input
Credal Relative Likelihood (CreRI.))
Credal Prediction
Entailed
Entalled
Initiaiize envemble members
GecunS-ruli
Criiiix
CreZlaco
CHaENN
Creliei
Creillax
Premise: "thooe boys wearing an Innercube in the water.
9.4
Hypothesia "threw boya awiaming.
Neutral
Controdicted
Neatral
Controdicted
Figure 1: Overview of our method. Language inference example from the ChaosNLI dataset, showing a hypothsss pair with three possible labels: Entailed, Neutral and Contradicted. Middle: Illustration of the learning mechanism of our Credal Relative Likelihood (CreRL) framework. showing ensemble member training up to different a thresholds. Right: Unlike baselines, which are centered around the MLE and remain fixed, our method allows to adapt credal set size to improve coverage of the ground-truth distribution.
representing ignorance in the sense of a lack of knowledge [Dubois et al., 1996]. because distributions always involve additional assumptions beyond the mere distinction between plausible and implausible candidate models. Decision- making on the basis of credal sets has been proposed as a reasonable alternative to Bayesian decision-maker making [Levi, 1978, Girón and Ríos, 1980]. The learning of credal predictors (i.e., models making predictions in the form of credal sets) has been considered in machine learning in the past [Zaffalon, 2001, Corani and Zaffalon, 2008] and, in light of the quest for proper representation of epistemic uncertainty, received renewed interest more recently [Wang al., 2024a, Nguyen et al., 2025, Cella and Martin, 2024].
periiiionmmaiingg
A major question to be addressed in this regard concerns the distinction between plausible and implausible models. In this paper, we adopt a theoretically grounded approach for constructing credal sets based on the notion of relative likelihood. Our approach explicitly defines the prediction target as the set of all probability distributions generated by plausible learners, namely those whose relative likelihood exceeds specified threshold (cf. Section 3). That said, realizing this in complex machine learning settings comes with several practical challenges, notably the questions of representation, approximation, and inference: How to formally represent subsets of a nigh-dimensional model space, how to approximate such sets algorithmically, and how to infer credal predictions from them? We propose methods to address these questions based suitably modified ensemble techniques (cf. Section 4). We highlight the effectiveness of our approach on real-world datasets, evaluating its performance in terms of coverage and efficiency as key criteria of credal prediction. Also, we analyze its performance on downstream tasks such as Dut-of-Distribuus
(OoD) detection (cf. Section 5).
Contributions. In summary our contributions are as follows:
1. We develop theoretically sound approach learning credal predictors, which grounded in the statistical notion of relative likelihood.
2. We cast the learning task as optimizing multi-objective generalization performance, namely finding compromise between coverage and efficiency of credal predictions.
3. We propose an adaptive and conceptually intuitive ensemble-based method for approximat- ing sets of plausible models and the induced credal predictions.
4. We empirically compare our method to representative baselines and, for the first time, compare state-of-the-ar credal predictors by coverage and efficiency. Our approach achieves superior uncertainty representation and strong OoD performance.
Related Work. In machine learning, uncertainty is often represented by (approximate) Bayesian methods [Mackay, 1992, Blundell et al., 2015, Lakshminarayanan et al., 2017, Gal and Ghahramani, 2016, Daxberger et al., 2021]. An important characteristic of such representations, especially for uncertainty tasks, is diversity [D' Angelo and Fortuin, 2021, Wood et al., 2023], which can be enforced by regularization [de Mathelin et al., 2023), varying hyper-paramette [Wenzel et al., 2020], or increasing diversity in representations [Lopes et al., 2022]. Alternatively, credal sets have been

---

used in the fields of imprecise probability and machine learning to represent model uncertainty [Zaffalon, 2001, Corami and Zaffalon, 2008, Corani and Mignatti, 2015]. Such sets can be generated based on the relative likelihood, also referred to as normalized likelihood [Antonucci et al., 2012a and have been used in machine learning with simple model classes [Senge et al., 2014, Cella and Martin, 2024]. In this work, we build upon these approaches and address the challenges that emerge when adapting the relative likelihood to a setting with complex predictors. Recently credal sets have also been applied in the context of deep learning. Some approaches use ensemble learning to derive class-wise lower and upper probabilities [Wang et al., 2024b, Nguyen et al., 2025]. Others train models to directly predict probability intervals [Wang et al., 2024a]. Hybrid methods, combining multiple uncertainty frameworks, have also been proposed. Caprio et al. [2023] combine Bayesian deep learning and credal sets by considering sets of priors. Another approach leverages conformal prediction to construct credal sets with validity guarantees [Javanmardi et al., 2024]. A more detailed discussion of the related work is provided in Appendix A.
2 Problem Statement
We consider classification in a supervised learning setting. We have access to training data D = {(,, y: y:) C X 3/ where X' the instance space and y = {yI..... yK) } is the label space with K E I classes. The training data are realizations of random variables that are independently and identically distributed according to probability measure on X' y. We consider a hypothesis space H with probabilistic predictors h : X' + Ak. where Ax denotes the (K - ))-simplex (that we will also refer to as the probability simplex). Given an instance æ, predictor h assigns a probability distribution p(- æ. h) = h(x), which is an estimate of the ground- truth probability p(1 æx, h²) generated by the moound-truth model h²"
Predictive models are often evaluated in terms of their likelihood L(h) I1 plys x,, h). Then, adopting the established principle of maximum likelihood inference, the model of choice is the maximum likelihood estimator (MLE) A4,,, i.e., the model whose likelihood is highest. By predicting probability distributions h 1 on V' this model is able to represent aleatoric uncertainty. However, it cannot capture information about the uncertainty of a itself, i.e., about how much possibly deviates from the ground-truth model h".
In order to capture this epistemic uncertainty, in the form of credal sets: Instead of relying on single predictor h, the uncertainty about the true underlying model h* is represented by the set of plausible predictors C S H. For a given query instance æ E X, this set induces prediction in the form of a credal set of distributions:
we consider a second-order uncertainty representation
e {p(- xr, h): : h E C} C AK.
(1)
We define the problem of learning credal predictor of supervised learning as introduced above: Given training data D, the task is to induce a model H X' + 2 that delivers predictions in the form of credal sets Qz H(x) C Ax. In our setup. (xx) represents the mapping of an instance & E X' through a set of plausible models C to the corresponding credal set Qu In general, H could also be realized differently. Inspired by other set-valued prediction methods such as conformal prediction [Vovk et al., 2005], we evaluate such a predictor in terms of its coverage and efficiency. Coverage (of the ground-truth distribution by the cedal set) is defined as follows:
generalization of the standard setting
C(H) E ([p(-11,M, H())]].
(2)
where I:I denotes the indicator function and the expectation is taken with regard to the marginal distribution of x on X. Moreover, efficiency captures the idea that "small" (more informative) credal sets are preferred over "large" (less informative) ones. It can be measured in different ways, for example as follows:
1 K
E(H)= = E
plyk 1x) p(ux (x)
where P(ys .>) = sup(p(u> x) : H()) (and p(vx ()) is defined analogously). We assume efficiency to be positively oriented, meaning that higher efficiency corresponds to smaller credal sets. As opposed to the volume of a credal set, which, besides being challenging to compute, not intuitive in high dimensions, this measure of efficiency is particularly interpretable. Specifically, it describes
3

---

the (complement of the) average interval length for each class. Importantly, we use efficiency strictly as an indicator of set size, not as a measure of epistemic uncertainty (cf. Section 2).
The empirical coverage and efficiency is determined in terms of the corresponding averages on a finite set of (test) data æ1.-...: NN- Note that ground-truth (first-order) distributions p(- æi h²) are typically unavailable during both training and testing, which often necessitates approximating coverage through alternative means. In this work, however, we make use of data that provides access to ground-truth distributions.
Evaluating credal predictor in terms of its coverage and efficiency means that two predictors H and H' are not necessarily comparable in the sense that one of them is "better" than the other one. Instead, such predictors are only comparable in Pareto sense: H is better than H' if C(H) > C(H") and F(H) 2 F(H') (and one of the inequalities is strict). The task can then be specified as learning Pareto-optimal credal predictors.
3 Relative Likelihood- Credal Sets
To learn a credal predictor H X - 2 from training data, we adopt the representation given in (1), defining C as set of plausible (first-order) predictors h X' - Ax drawn from an underlying hypothesis space H. To characterize plausibility, we employ the concept of relative likelihood:
L(h) 7h) LL
L(h)
sup L(h')
WEH
Here, M& represents the model in H with the highest likelihood given the training data D. This notion, also called extended or normalized likelihood, has been proposed by Birnbaum [1962] and used for statistical inference [Wasserman, 1990, Walley and Moral, 1999]. It offers an attractive alternative Bayesian or frequentist reasoning eliminating the need to specify priors or perform resampling [Giang and Shenoy, 2002]. In machine learning this is especially useful as specifying a meaningful prior on H is non-trivial problem and resampling from the data distribution is usually not possible either, as we only have access to the given samples D. An axiomatic justification of the relative likelihood (in the context of evidence theory) has been given by Denoeux [2014], who derives it from (i) the likelihood principle, (ii) compatibility with the Bayes rule, and (iii) the minimal commitment principle.
On the basis of the relative likelihood, a set of models can be constructed, analogous to confidence region [Aitkin, 1982], by including models that are plausible in the sense of surpassing threshold a E [0, 1]. This set of models is also referred to as an a-cut [Antonucci et al., 2012a]:
Gromd MLE
O 0.9
0.6
C. = (h: (h) a} C H.
(4)
According to this definition, a model is considered im- plausible (and hence ignored) if its likelihood is too small compared to the likelihood of the (presumably) best model, namely if its likelihood is less than a times the likelihood of the best model. Thus, the parameter a has a quite intuitive meaning. The concept of a-cuts illustrated in Figure 2.
0.0
Figure 2: Relative Likelihood curve. Based on 10 samples from a Bernoulli distribution with parameter 0.5. The credal set of a = 0.9 around the MLE at 0.6 does not cover the ground-truth. Lowering a increases the coverage at the cost of efficiency.
Given an instance x E X, the set of predictors maps to a set of (conditional) distributions:
Qx.a = (p(-1 x h) : h E C.) CAK-
This credal set can then be used for predictive tasks and quantify uncertainty. In essence, it can be used to express the amount of uncertainty one has about the best predictor that can be obtained from the given data sample.
Considering that we aim to simultaneously optimize coverage and efficiency, is important to highlight the role of the parameter a as a means to move along the Pareto front and trade coverage against efficiency: The lower o, the higher coverage tends to be. At the same time, however, efficiency will deteriorate (cf. again Figure 2). As increases, the intervals for given data points shrink since

---

only models with high relative likelihood for the given data sample are included. Conversely, lowering allows for the inclusion of models with lower likelihood, resulting larger credal intervals. This controllability induced by emphasizes the appeal of the relative likelihood as a tool for (plausible) model selection and, thereby, credal set construction.
4
Approximating Relative Likelihood-Baset Credal Sets
Building on the theoretical framework for constructing credal sets using relative likelihood, it becomes evident that, in practice, we must approximate the set models C., and consequently the resulting credal set Qz.a. In this section, we examine how the formal notion of relative likelihood can be applied to construct credal sets in nigh-ddmensiooal machine learning settings. Up to this point, we made no assumptions about the specific model class. Obviously, to approximate the a-cut (4), the model class must allow to define and optimize toward a target relative likelihood during training. As concrete and widely used example of complex models in machine learning, focus neural networks throughout the remainder of this work.
Estimating the Maximum Likelihood Estimator: Several strategies can be employed to obtain MML, including training model with parameters known perform well, utilizing pre-trained model, or leveraging AutoML techniques to determine optimal parameter settings during training. Throughout this work, we adopt the first approach and train the maximum likelihood estimator A using parameter configurations that have been identified in the literature as effective. Further details and configurations of experiments are provided in Appendix B.
Given an estimated maximum likelihood model, the estimated relative likelihood is obtained as L(h) ((b)= T(Kar))
Approximating the a-cut. Having found suitable candidate model to com- pute the maximum likelihood, the next step is to construct the a-cut. A straight- forward approach to train an ensem- ble of hypotheses and construct the a-cut based on these hypotheses. proaches based on regular ensemble train- ing [Wang et al., 2024b, Nguyen et al., 2025] form an example of this strategy. However, an issue that all hypotheses h typically tend to cluster around h M hence the obtained hypotheses do not ac- curately approximate the a-cut (unless a R 1, see Figure 1).
Algorithm 1 Train Credal Relative Likelihood ensemble.
Require: a.M .M
1: Step 1: Approximate maximum likelihood model
2: AL +- arg min, C(h)
3: Step 2: Train ensemble members
essence, ap-
4: Ar 1-o
5: for i E M 1 do
6:
т; = = - Ar
7:
h; 4 ToBias initialization
8:
Train h, such that (h,,) =
9: end for
To obtain better coverage of Ca. propose to train M hypotheses using early stopping strategy, namely until a specific relative likelihood value T; is reached. Given a, we consider thresholds { v; a + i Ar). where Ar I-a M- and i E {0...., M 2}. This guarantees broad range of hypotheses in terms of relative likelihoods. The influence of the ensemble size M is analyzed an ablation study in Appendix C.2.
Although this should cover the a-cut well, there might still be a lack of diversity in the predictions of the resulting hypotheses. In the literature, it is known that training ensembles without explicitly encouraging diversity actually minimizes diversity [Abe et al., 2022]. To encourage diversity, we introduce novel initialization strategy called ToBias. The idea is make sure that the initial state of the ensemble, i.e., prior to training, represents state of full uncertainty (or no knowledge). Hence, the resulting credal prediction should entail the entire probability simplex. As the predictors the ensemble are trained, the amount of knowledge increases, and the predicted credal set should shrink. To enforce this finite predictor scenario, we ensure that the initial predictions of the learners correspond to degenerate probability distributions at vertices of the (K 1)-simllex effectively covering the entire simplex in the initially predicted credal set. Figure/ illustrates the mechanism of ToBias initialization and the overall learning process.
5

---

Technically, the initialization is performed by assigning large constant to one of the biases in the final layer of each predictor. For example, given the biases b, = , bs x] of the last layer of predictor hi, we set bi, mod = 3, where 3 is large constant, following the regular initialization. In the remainder of this work, we use 100. An ablation study examining the influence of ToBias initialization is presented in Appendix C.2.
We illustrate the difference between regular ini-
a) Standard Initialization
b) ToBias Initialization
tialization and ToBias initialization in Figure 3.
Ground-truta
Observe that without ToBias initialization, the credal set initially represents state of low epis- temic uncertainty, all predictions are concen- trated around the barycenter of the probability simplex, and increases its uncertainty as train- ing progresses. The opposite can be observed with ToBias initialization. The credal set starts representing state of full uncertainty and de- creases its uncertainty as the learners acquire more knowledge.
Initial
Final
Figure 3: Effect of ToBias. Example from ChaosNLII a) Standard initialization initially pre- dicts close to uniform distributions and converges to small credal set. b) ToBias initializes at the vertices resulting in diverse predictions.
Having trained M models until the respective thresholds, the approximate a-cut Ca can be constructed as follows:
(h; (hhh) ss T 1-1 C H.
where (h;)) x t; enforced by the training process described above. The full algorithm to train Credal Relative Likelihood (CreRL) ensembles is presented in Algorithm 1.
Credal Set Predictions. Given query instance X, the set C₂ induces finite approximation of the credal set:
Q (p(-1x,hh) h C.) Ak.
There are several principled ways of transforming this finite set into credal set. Two common approaches are to derive (i) the convex hull of the set Qu,a and (ii) the intervals (lower and upper probabilities) for individual classes obtained from this set. Note that the convex hull is contained in the "box" credal set induced by the intervals. In the remainder of this work, we opt for the latter approach and define the approximation of Qa.a as follows [de Campos et al., 1994]:
Qe {p( xx) plyx xx) [p(8 ply x)), Vyu }}
Empirical Results
To empirically evaluate our method, we measure its performance in terms of coverage and efficiency, as defined in Section 2, along with the corresponding Pareto criterion. Additionally, we evaluate approach the downstream task of OoD detection, which is often used assess the quality of (epistemic) uncertainty representations. We compare our approach (CreRL) to suitable baselines by implementing the following approaches: Credal Wrapper (CreWra) [Wang al., 2024b], Credal Ensembling (CreEns) INguyen et al., 2025], Credal Bayesian Deep Learning (CreBNN) [Caprio al., 2023], and Credal Deep Ensembles (CreNet) [Wang al., 2024a]. These methods represent the current state-of-ar or-art for credal prediction and to the best of our knowledge we are the first to do systematic comparison of these approaches in a unified benchmark. Since coverage and efficiency are not well-defined for Bayesian methods, a direct comparison to such methods is not possible.
The code for all implementations and experiments is published in Github repository! Further exper- imental details our method and the implementation of the baselines are provided in Appendix B.
5.1 Predictive Performance
We method train ensemble of fully-connected neural networks the embeddings of the ChaosNLI dataset [Nie et al., 2020]. This dataset features premise-sypothesis pairs in textual form
-ikellivood

---

ChaosNLI
CIFAR10
QualityMRI
1.0
0.2
3.0
0.8
Efficiency
+
0.6
0.4
0.2
0.2
0.2
0.0
0.0
0.2
0.4
066
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Coverage
Coverage
Coverage
CreWra
CreEns
CreRL (Ours)
O'
CreBNN
CreNet
Figure 4: Pareto front between coverage and efficiency for ChaosNLI, CIFAR-10, and QualityMRI. CreRL (Ours) and CreEns allow trading off efficiency and coverage by varying a. The other baselines do not allow direct adaptation of the credal set and hence result in single point.
with the goal to classify each instance as one of three classes: entailed, neutral, and contradicted. Each instance in ChaosNLI has 100 annotations, which can be seen ground-truth distribution. We train an ensemble of ResNet18 models [He et al., 2016] based on our method on the CIFAR-10 10 dataset [Krizhevsky et al., 2009]. This dataset consists of 10 classes of images, commonly used as benchmark for image classification tasks. We use the human annotations provided by the CIFAR- 10H dataset [Peterson et al., 2019] as ground-truth distributions. In addition, we train an ensemble of ResNet18 8 models on the QualityMRI dataset from Schmarje et al. [2022]. This dataset comprises MRI images with multiple annotations from radiologists, each providing quality assessment of the image [Obuchowicz et al., 2020]. These multi-class annotations were merged into annotations for two classes [Schmarje et al., 2022]. The uncertainty in these labels, represented by the high average entropy reported in Appendix C, confirms the importance of uncertainty quantification in medical settings [Löhr al., 2024].
To train the neural networks, the annotations are converted into a probability distribution, which we consider to be the ground-truth. The neural networks then trained targets that sampled from the ground-truth probability distribution. Thus, our method does not require ground-truth probability distributions, soft labels, for training.
We evaluate our method using the coverage and efficiency as defined in (2) and (3), where the expectation is approximated by averaging over the instances in the test set of the respective dataset. The results can be found in Figure 4. Appendix C, we report the accuracy of the ensemble members for each implementation.
Trade-off between Efficiency and Coverage. In general, both our method (CreRLa) and the CreEns, approach are flexible and allow to trade off efficiency and coverage of the credal predictor by varying a. However, CreEns can only attain credal sets in the low coverage region, whereas our approach can predict credal sets with wide range of coverage values, including high coverage. The other methods only deliver fixed credal sets, meaning there is no direct way to to increase (or decrease) the coverage and efficiency. The Cre approach and CreEnsg (with the same efficiency, but a different coverage, due to the different credal set constructions (interval vs. convex hull), The CreNet converges to a point prediction, because its loss function is constructed such that the lower probability converges to the upper probability, and hence is generally in the top left region of the Pareto figure. We refer to Appendix A for more details about these methods.
0) will generally have
For the ChaosNLI dataset, our approach dominates the CreBNN approach and the CreWra resembles point on the Pareto front obtained by method. As mentioned, the CreEns, method can only reach the low coverage region and the CreNet is in the top left.
On 10, our method generates Pareto front in a region of high efficiency and high coverage. The Cre baseline is close to our Pareto front. Since all ensemble members have high performance, illustrated by their accuracies in Appendix C, and there is no explicit diversity promotion in
7

---

Table 1: Out-of-Distriuution detection based epistemic uncertainty. CIFAR-I0 is used the in- in-Distribution dataset. The mean and standard deviation over 3 runs are reported. Best performance is in bold.
Method
SVHN
Places
CIFAR-100
FMNIST
ImageNet
CreWra
0.957=0.000
0.006
0.916+0.000
0.952
0.890
CreEnsp.o
0.955=+0.000
0.000
0.914=0.0001
0.949
0.888=0.000
CreBNN
0.907
0.885:10.002
0.880110.002
0.935
0.859
CreNet
0.943
0.918
0.91211.000
0.951 100002
0.884_0.000
CreRL1.0
0.94810.003
0.918 1918810.000 0.000
0.91610.001
0.957 1.0.002
0.88910.000
CreRLo.95
1.0.71
1..1010.001
0.901 1.0.000
0.945110.004
0.87811.002
CreRLo.9
0.91810.011
0.90710.001
0.896110.001
0.944
0.87410.000
CreRLo.s
0.906=+0.00s 0008
0.894+0.001
0.884
0.936+#00009
0.865 002
CreRLo.6
0.862:+0.08
0.874+0.003
0.852+0.000
0.893+0.000
0.837 003
CreRLo.4
0.739+0.029
0..01
0.796=#0.007
0.815+0.020
0.788=00.010
CreRLo.2
0.582+0.041
0.736=000010
0.700=t0.013
0.676
0.698 013
CreEns.: it produces small sets for every CreNet, again, produces a (close to) point prediction with low coverage. We refer to Appendix for more details.
For QualityMRI, the story is largely consistent with that of the ChaosNLI dataset. The difference is that our method is only positioned the high coverage half of the Pareto figure.
The ability of our method to target the high coverage area can be explained by the fact that we promote diversity, by means of our ToBias initialization, in order promote high coverage, at the cost of efficiency. In addition, one may argue that the low coverage region represents credal sets that of limited benefit for practitioners, they are unlikely demand coverage of, say, 0.3.
Furthermore, Figure 4 shows that the shape and position of the Pareto front varies across datasets suggesting that the soverage-efficieng trade-off is inherently data-dependent. In particular, the better the maximum likelihood model, the more the Pareto front should move towards the top right (high coverage, high efficiency), which we also observe in the case of CIFAR-10.
To further study the behavior of our method, we present ablation experiments in Appendix C, varying the number of members in the ensemble and the ToBias initialization constant.
5.2
Duttor-Distribution Detection
Due to a lack of ground-truth uncertainties for most datasets, uncertainty methods are usually evaluated downstream tasks such as OoD detection. We perform the OoD task to assess the (epistemic) uncertainty representation of our method. The predictor is trained on a dataset called the in- nn-Distribution (iD) dataset and test time introduced to instances from both the iD dataset and another OoD dataset that it has not seen before. An effective epistemic uncertainty representation should assign higher epistemic uncertainty to OoD instances than to iD instances. In the literature, this is commonly evaluated by computing epistemic uncertainty and using it to distinguish between iD and Ool instances. We adopt the epistemic uncertainty measure based on the additive decomposition proposed (and axiomatically justified) by Abellán et al. [2006]:
(U,( =((....) S((n.a):
where 5(Qn.a) denotes the maximum Shannon entropy of all distributions in 2n.a and. likewise, S( ((Qm.a) the minimum entropy. 2 The performance is then measured using the AUROC. Appendix B.4 provides details about the implementation of the optimization.
We utilize the same CIFAR-10 10 ensembles of ResNet18 8 models and compare to the same baselines as in Section 5. We introduce SVHN Netzer et al. [2011], Places365 Zhou et [2018], CIFAR-100 Krizhevsky al. [2009], FashionMNIST Xiao et al. [2017], and ImageNet Deng et al. [2009] as out-of-distribution datasets, and compute epistemic uncertainty as (5). Our method (CreRL,) and
'The bounds of Shannon entropy are computed numerically using SciPy [Virtanen et al., 2020J.

---

the Credal Ensembling (CreEns,)) approach make use of a hyper- papaem-rramtter a The results with the best performing are presented here. The results, summarized in Table 1, present the OoD detection performance. We provide further results for both methods with different ar values in Appendix C.
Our method (CreRLa)) with a performs either the best or is par with the best methods in Ool> detection, except for the SVHN dataset, where the CreWra approach has a small advantage. This shows the good (epistemic) uncertainty representation of method.
Influence of a. Furthermore, the value of the parameter of our method has significant impact on the behavior of credal sets for both iD and OoD data. We expect greater epistemic uncertainty for instances from distribution that differs from the training data, predictions may diverge for these instances. When a is close to 1, the resulting set consists of models with high relative likelihood, leading to smaller credal sets and thus lower epistemic uncertainty for iD data. Conversely, smaller a includes low relative likelihood models, resulting in higher epistemic uncertainty. Given that strongly influences epistemic uncertainty for iD instances, we expect a larger discrepancy in uncertainty between iD and OoD instances. As shown in Table 1, across all OoD datasets, larger indeed leads to higher AUROC scores, confirming that larger values lead to better separation of iD) and OoD instances.
increase the
6
Conclusion
We proposed new approach to generating credal predictors in a machine learning setting with complex learners on the basis of relative likelihood, which is used to identify set of "plausible" models. The relative likelihood is an intuitive notion of model plausibility, grounded in classical statistics. Given a query instance for which a prediction is sought, a credal set obtained as the collection of probability distributions predicted by all plausible models. Specifically, we introduced novel method for training neural networks to approximate the credal set generated by an a-cut of models. In order to obtain good approximation, the a-cut needs to be covered well, i.c., the included hypotheses should be sufficiently spread over the hypothesis space. To this end, we proposed ToBias initialization as simple but effective diversification strategy. Furthermore, the parameter a, which specifies the likelihood ratio between the maximum likelihood model and the "weakest" model that is still included in the a-cut, be adjusted depending on the task hand and used to control the trade-off between the coverage and efficiency of credal predictions.
Experimentally, we have shown that this workflow provides strong coverage of the ground-truth con- ditional distributions, while maintaining competitive efficiency, as well as performing competitively in downstream tasks such as OoD detection. The relation between and the performance on different tasks sheds further light on the trade-off between diversity and prediction performance.
Limitations and Future Work. Our method has so far been evaluated exclusively on neural networks. However, the general concept of relative likelihood is applicable to any model class. Certain aspects of our method suchh as ToBias initialization are specifically tailored to neural networks and would have to be adapted to other learners. A promising direction for future work would be to extend this framework to other model classes.
When generating a convex credal set from our finite approximation, we include all probability distributions such that the respective class probabilities are bounded by their lower and upper probabilities. This makes the optimization problem of quantifying uncertainty easier, but it also generates sets that are larger than necessary. An alternative would be to take the convex hull of the probability distributions as the credal set, which would generate more efficient sets. Future work may explore the effect of this on metrics such as coverage or OoD performance.
Another interesting question that arises relates to the approximation quality of our approach. Having specified a threshold for the relative likelihood, there exists some ground-truth credal set for an instance. It could be interesting to explore how many probability distributions, and thus predictors, required to accurately approximate this credal set.
Broader Impacts. This work contributes to the development of reliable machine learning models by improving uncertainty quantification. We do not foresee any direct negative broader impacts arising from this.

---

Acknowledgments and Disclosure of Funding
We gratefully thank the anonymous reviewers for their valuable feedback, which helped improve this work. This work has received funding from the European Union's Horizon Europe research and innovation programme under the Marie Sklodowska-Cure grant agreement No 101073307. Felix Mohr participated through the project ING-312-2023 at Universidad de La Sabana. We acknowledge the Munich Center for Machine Learning (MCML) for their support of this work.
References
T. Abe, E. K. Buchanan, G. Pleiss, and J. P. Cunningham. The best deep ensembles sacrifice predictive diversity. In I Can't Believe It's Not Better Workshop: Understanding Deep Learning Through Empirical Falsification, 2022.
Abellán and S. Moral. A non-specificity measure for convex sets probability distributions. Int. J.
Uncertain. Fuzziness Knowl. Based Syst., 8(3):357-368, 2000.
J. Abellán and S. Moral. Maximum of entropy for credal sets. Int. J. Uncertain. Fuzziness Knowl. Based Syst., 115):587-558 2003.
J. Abellán, J. Klir, and S. Moral. Disaggregated total uncertainty measure for credal sets. Int. J. Gen. Syst. 35(12999444, 2006.
M. Aitkin. Direct likelihood inference. In GLIM 82: Proceedings of the International Conference on Generalised Linear Models, pages 76-86. Springer, 1982.
A. Antonucci, M. E. G. V. Cattaneo, and G. Corani. Likelihood- bayesian networks. In S. Greco, B. Bouchon-Meunier, G. Coletti, M. Fedrizzi, B. Matarazzo, and R. R. Yager, editors, Advances in Computational Intelligence 14th International Conference on Information Processing and Management of Uncertainty in Knowledge-Bart 2012, Catania, Italy, July 9-13, 2012, Proceedings, Part III, volume 299 of Communications in Computer and Information Science, pages 491-59115 190-500. Springer, 2012a.
robust classification with
Systems, IPMU
A. Antonucci, G. Corani, and S. Bernaschina. Active learning by the naive credal classifier. 2012b.
A. Birnbaum. On the foundations of statistical inference. Journal of the American Statistical Association, 57(298):26-335 1962.
C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pages 1613-1622. PMLR, 2015.
M. Caprio, S. Dutta, K. J. Jang, V. Lin, R. Ivanov, O. Sokolsky, and I. Lee. Imprecise bayesian neural networks. CoRR, 0bs/2300.099566 2023.
L. Cella and R. Martin. Variational approximations of possibilistic inferential models. In Y. Bi, A. Jousselme, and T. Denoeux, editors, Belief Functions: Theory and Applications - 8tth Interna- tional Conference, BELIEF 2024, Belfast, UK. September 2-4, 2024, Proceedings, volume 14909 of Lecture Notes in Computer Science, pages 121-130. Springer, 2024.
G. Corani and A. Mignatti. Credal model averaging for classification: representing prior ignorance and expert opinions. Inf. J. Approx. Reason.. 56:264-277, 2015.
G. Corani and M. Zaffalon. Learning reliable classifiers from small or incomplete data sets: The naive credal classifier J. Mach. Learn. Res., 9:581-621, 2008.
F. G. Cozman. Credal networks. Artificial intelligence, 1trieeeee..0000 120(2):199-233 2000.
D' Angelo and V. Fortuin. Repulsive deep ensembles are bayesian. Advances in Neural Information Processing Systems, 34:3451-3466 2021.
E. A. Daxberger, A. Kristiadi, A. Immer, R. Eschenhagen, M. Bauer, and P. Hennig. Laplace redux effortless bayesian deep learning. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neurul Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virtual, pages
2021.
10

---

L. M. de Campos, J. F. Huete, and S. Moral. Probability intervals: a tool for uncertain reasoning, Int. Uncertain. Fuzziness Knowl. Based Syst. 2(2):167-196, 1994.
A. de Mathelin, F, Deheeger, M. Mougeot, and N. Vayatis. Deep anti-regularized ensembles provide reliable out-of-distribution uncertainty quantification. CoRR, Abs/2304.04042, 2023.
J. Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
T. Denoeux. Likelihood-based belief function: justification and some extensions low-quality data. International Journal of Approximate Reasoning, 55(7): 2014.
S. Depeweg, J. M. Hernández-Lobato,, F. Doshi-Velez, and S. Udluft. Decomposition of uncertainty in bayesian deep learning for efficient and risk-sensitive learning. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1 1192-1201. PMLR, 2018.
J. Donnelly, Z. Guo, A. J. Barnett, H. McTavish, C. Chen, and C. Rudin. Rashomon sets for prototypical-pan networks: Editing interpretable models in real- real-time. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 4528 4538,,53,4-- 2025.
D. Dubois, H. Prade, and P. Smets. Representing partial ignorance. IEEE Transactions Systems, Man and Cybernetics, Series A, 26(3):361-377, 1996.
Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In M. Balcan and K. Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning. ICML 2016, New York City, NY. USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050- 1059. JMLR.org, 2016.
J. Gawlikowski, C. R. N. Tassi, M. Ali, Lee, M. Humt, J. Feng, A. M. Kruspe, R. Triebel, P. Jung, R. Roscher, M. Shahzad, W. Yang. R. Bamler, and X. Zhu. A survey of uncertainty in deep neural networks. Artif. Intell. Rev, 56($1):1513-1588, 2023.
H. Giang and P. P. Shenoy. Statistical decisions using likelihood information without prior probabilities. In A. Darwiche and N. Friedman, editors, UAI '02, Proceedings of the 18th Conference in Uncertainty in Artificial Intelligence, University of Alberta, Edmonton, Alberta, Canada, August 144, 2002, pages 170-178. Morgan Kaufmann, 2002.
F. Girón and S. Ríos. Quasi-bayesian behaviour: A more realistic approach to decision making? Trabajos de estadística y de investigación operativa, 31:17-38, 1980.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
P. Hofman, Y. Sale, and E. Hüllermeier. Quantifying aleatoric and epistemic uncertainty: A credal approach. In ICML 2024 Workshop on Structured Probabilistic Inference & Generative Modeling, 2024.
E. Hüllermeier and W. Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine learning. 110(3):557-506, 2021.
E. Hüllermeier, S. Destercke, and M. H. Shaker. Quantification of credal uncertainty in machine learning: A critical analysis and empirical comparison. In Cussens and K. Zhang, editors, Uncertainty in Artificial Intelligence, Proceedings of the Thirty-Eighth Conference on Uncertainty in Artificial Intelligence, UAI 2022, 1-5 August 2022, Eindhoven, The Netherlands, volume 180 of Proceedings of Machine Learning Research, pages
PMLR, 2022.
A. Javanmardi, D. Stutz, and E. Hüllermeier. Conformalized credal set predictors. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
A. Kendall and Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.

---

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
A. Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6402-644 20136413, 2017.
Y. Le and X. S. Yang. Tiny imagenet visual recognition challenge. 2015.
I. Levi. On indeterminate probabilities. In Foundations and Applications of Decision Theory: Volume
I Theoretical Foundations, pages 233-261. Springer, 1978.
T. Löhr, M. Ingrisch, and E. Hüllermeier. Towards aleatoric and epistemic uncertainty in medical image classification. In International Conference on Artificial Intelligence in Medicine, pages 145-155. Springer, 2024.
R. G. Lopes, Y. N. Dauphin, and E. D. Cubuk. No one representation to rule them all: Overlapping features of training methods. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
I. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
J. C. Mackay. Bayesian methods for adaptive models. California Institute of Technology, 1992.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 20/1, 2011.
V. Nguyen, H. Zhang, and S. Destercke. Credal ensembling in multi-class classification. Mach. Learn., 114(1):19, 2025.
Y. Nie, X. Zhou, and M. Bansal. What can we learn from collective human opinions on natural language inference data? In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 9131-9143. Association for Computational Linguistics, 2020.
R. Obuchowicz, M. Oszust, and A. Piórkowski. Interobserver variability in quality assessment of magnetic resonance images. BMC Medical Imaging, 20(1):109, 2020.
J. C. Peterson, R. M. Battleday, T. L. Griffiths, and O. Russakovsky. Human uncertainty makes classification more robust. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 -Noemmerr 2019, pages 9616-9633 9625. IEEE, 2019.
L. Schmarje, V. Grossmann, C. Zelenka, S. Dippel, R. Kiko, M. Oszust, M. Pastell, J. Stracke, A. Valros, N. Volkmann, and R. Koch. Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
L. Semenova, C. Rudin, and R. Parr. On the existence of simpler machine learning models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1858-1858, 2022.
R. Senge, S. Bösner, K. Dembezynski, J. Haasenritter, O. Hirsch, N. Donner-Banzhoff and E. Hüller- meier. Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncer- tainty. Inf. Sci., 255:16-29 2014.
12

---

P. Virtanen, R. Gommers, I. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weekesser, J. Bright, S. van der Walt. M. Brett, Wilson, K. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, i. Polat, Y. Feng. E. W. Moore, J. VanderPlas, D. Laxalde, Perktold, R. Cimrman, I. Henriksen,
A. Quintero, Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors.
SciPy 1.0; Fundamental Algorithms for Scientific Computing Python. Nature Methods, 17:
261-272, 2020.
V. Vovk, A. Gammerman, and G. Shafer. Algorithmic learning in random world, volume 29. Springer, 2005.
P. Walley, Statistical Reasoning with Imprecise Probubilities. Chapman & Hall, 1991.
P. Walley and S. Moral. Upper probabilities based only on the likelihood function. Journal of the Royal Statistical Society: Series (Statistical Methodology), 61(4):831-88) 1999.
K. Wang. Cuzzolin,
Manchingal, Shariatmadar, Moens, and H. Hallez. Credal deep ensembles for uncertainty quantification. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15. 2024, 2024a.
K. Wang, F. Cuzzolin, K. Shariatmadar, D. Moens, and H. Hallez. Credal wrapper of model averaging for uncertainty estimation nut-of-distributinn detection. CoRR, abs/2405. 15047, 2024b.
L. A. Wasserman. Belief functions and statistical inference. Canadian Journal of Statistics, 18(3): 183-196, 1990.
F. Wenzel, J. Snoek, D. Tran, and R. Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification.
Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
D. Wood, T. Mu, A. M. Webb, H. W. J. Reeve, M. Luján, and G. Brown. A unified theory diversity in ensemble learning. J. Mach. Learn. Res., 24:359:1-35::9, 2023.
H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. CoRR, 01117708077777 2017.
M. Zaffalon. Statistical inference of the naive credal classifier. In G. D. Cooman, T. Fine, and T. Sei- denfeld, editors, ISIPTA '01, Proceedings of the Second International Symposium on Imprecise Probabilities and Their Applications, Ithaca, NY, USA, pages 384-393. Shaker, 2001.
B. Zhou, A. Lapedriza, A. Khosla, Oliva, and Torralba. Places: 10 million image database for scene recognition. IEEE Trans. Pattern Anal. Mach. Intell., 40(6):1452 1452-1464,, 2018.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our claims are supported by Sections 3 to 5,
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
We do not provide theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The full experimental details are discussed in Section and Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: Code provided in Section with run instructions the respective repository. The data used is publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: /CodeSubmis //nips.coom wttpccccppplc/ppceeeeeeecc (Cooduuuuud ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The full details are described in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide information about the error bars for every figure and table, see Section and Appendices and C.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the compute resources in Appendix B.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https pattpa//nnerripee hwtpp//ccc...ttcieceieiieeeoeeee
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and confirm that our work complies with
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: provide a broader impacts statements Section 6.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not contribute any such models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We provide references to the models used and additionally include licenses for the datasets in Appendix B.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The method proposed in this work does involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: /tap:::///eurpssss http://rrp....ccooooeeeeeeee//// cct.//oonfeeeeccee222/////// for what should or should not be described.
20

---

A Extended Related Work
Uncertainty Representation. In machine learning, uncertainty often represented by Bayesian methods [Mackay. 1992, Blundell et al., 2015]. Frequently, the Bayesian posterior is approximated by ensembles [Lakkhminarayanan et al., 2017] or methods such as Dropout [Gal and Ghahramani, 2016] or Laplace approximation [Daxberger et al., 2021]. An important characteristic of such representations, especially for uncertainty tasks, is diversity [D' Angelo and Fortuin, 2021, Wood et al., 2023]. Some works enforce this by means of regularization [de Mathelin et al., 2023], whereas others vary hyperparameters across models within the ensemble [Wenzel et al., 2020] or enforce diversity in the representations [Lopes et al., 2022].
Alternatively, credal sets have been used in the fields of imprecise probability and machine learning to represent model uncertainty [Zaffalon, 2001, Corani and Zaffalon, 2008, Corani and Mignatti, 2015]. Antonucci et al. [2012a] proposed to generate such sets based on relative likelihoods. Relative likelihoods, also referred to as normalized likelihoods, have also been used in machine learning with simple model classes such as logistic regression [Senge et al., 2014, Cella and Martin, 2024]. In this work, we build upon these approaches and address the challenges that emerge when adapting the relative likelihood to a setting with complex predictors.
Recently, credal sets have been applied in the context of machine learning. Wang et al. [2024b] take multiple samples from a Bayesian posterior or ensembles and derive class-wise lower and upper probabilities. Based on these samples, a credal set is constructed by including all probability distributions such that the individual predicted class probabilities are in the respective lower to upper class probability interval. Nguyen et al. [2025] also construct credal sets from ensemble predictions, but with the additional option of discarding potential outliers to prevent the set from becoming too large. This is done by comparing all ensemble predictions to representative prediction, e.g. the mean prediction, using some distance between distributions, and keeping only (1 a-a) 100% closest predictions. The credal set is then constructed by taking the convex hull of the remaining probability distributions. Other methods directly train neural networks to predict intervals by explicitly predicting a lower and upper probability for every class [Wang et al., 2024a]. In combination with a custom loss function, consisting of regular cross-entropy for the upper probabilitics and cross-entropy computed on the highest loss subset of batch for the lower probabilities, the claim is that this approach encourages both "optimistic" and "pessimistic" predictions. Credal sets are constructed by taking the same interval-basee -based approach [Wang et al., 2024b]. Besides this, hybrid methods, combining multiple uncertainty frameworks, have also been proposed. Caprio et al. [2023] combine Bayesian deep learning and credal sets by considering sets of priors over weights of neural networks. Training these neural network by variational inference then results in a set of posteriors. Based on this, a credal set is constructed by sampling from the Bayesian neural networks and taking the convex hull of the sampled probability distributions. Another approach leverages conformal prediction to construct credal sets with validity guarantees [Javanmardi et al., 2024). However, this work uses distributions over classes to perform the conformal calibration step. Since our method does not require such data, we exclude this method as a baseline. Our work distinguishes itself from aforementioned works by using relative likelihood cuts which allow for an intuitive and adaptable construction of the credal set.
Moreover, our approach is conceptually related to Rashomon sets [Semenova et al., 2022]. Both characterize a collection of models whose performance exceeds a given threshold, thereby facing the same challenge in approximating this set of plausible models [Donnelly et al., 2025]. However, the objectives differ: Rashomon sets are primarily concerned with interpretability and (syntactic) model diversity, whereas focus is on uncertainty quantification and predictive diversity.
Uncertainty Quantification. Given a credal representation of uncertainty, there are many ways to quantify uncertainty. Some measures consider only the epistemic uncertainty [Abellán and Moral, 2000], whereas others use entropy to reason about the total uncertainty [Abellán and Moral, 2003]. In this line of work, Abellán et al. [2006] also proposed measures based entropy that decompose the total uncertainty into an aleatoric and epistemic component. Conversely, Antonucci et al. [2012b] quantify uncertainty by measuring the lack of dominance of one class over others in the predictive distributions represented by the credal set. Hüllermeier et al. [2022] offer a critical analysis of these measures and propose alternative for the dominance-batt based measure. Recently, Hofman al. [2024] proposed to quantify credal uncertainty based on a decomposition of scoring rules. In the following, we consider the uncertainty measures by Abellán et al. [2006] in order to ensure fair comparison with previous works.
21

---

B Experimental Details
B.1 Datasets
The ChaosNLI dataset, introduced by Nie al. [2020], is a large-scale dataset de- signed to study human disagreement in natural language inference (NLI) tasks. It comprises 100 human annotations per example for 3,115 examples from the SNLI and MNLI datasets, and 1,532 examples from the aNLI dataset, totaling approximately 464,500 annotations. In line with Ja- vanmardi et al. [2024], we use only the SNLI and MNLI subsets of the ChaosNL. dataset, but for simplicity, we will refer to this as the ChaosNLI dataset. Each example includes metadata such as the unique identifier, counts of each label assigned by annotators, the majority label, la- bel distribution, entropy of the label distribution, the original example text, and the original label from the source dataset. This dataset enables a detailed analysis of the distribution of human opinions in NLI tasks, highlighting instances of high disagreement and questioning the valid- ity of using majority labels as the sole ground truth. ChaosNLI is publicly available under the 4.0 International (CC BY-NC 4.0) license. We
ChaosNLI
Creative Commons train our models on the 768-dimensional embeddings of the ChaosNLI dataset retrieved from https:/ / /gi thub. commalireza- aawarrea-avvvaaarreeeeeaaaaaaaee jaaan-aaaaaaaaaaaaaaa womaa-cremaaall------aa mardi et al., 2024] for more details on the generation of the embeddings.
We refer to [Javan-
The CIFAR-10 dataset is a widely used benchmark in machine learning and computer vision, introduced by Krizhevsky et al. [2009], and Geoffrey Hinton in 2009. It comprises 60,000 color images at a resolution of 32x32 pixels, evenly distributed across 10 distinct classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. The dataset is partitioned into 50,000 training images and 10,000 test images, organized into five training batches and one test batch, each containing 10,000 images. The dataset is publicly available and has been utilized extensively for developing and benchmarking machine learning models. While the original dataset does not specify a license, various distributions, such as those provided by TensorFlow Datasets, are released under the Creative Commons Attribution 4.0 License
CIFAR-10
CIFAR-10H The CIFAR- 10H dataset provides human-derived soft labels for the 10,000 images in the CIFAR-10 10 test set, capturing the variability in human annotation during image classification tasks. Developed by Peterson et al. [2019], the dataset comprises 511,400 annotations collected from 2,571 Amazon Mechanical Turk workers, with each image receiving approximately 51 labels. Annotators classified images into one of the ten CIFAR-10 categories, enabling the construction of probability distributions over labels for each image. CIFAR- 10H is publicly available under the Creative Commons BY-NC-SA 4.0 license.
CIFAR-100 The CIFAR-100 dataset, introduced by Krizhevsky et al. [2009], comprises 60,000 color images at 32x32 resolution, divided into 100 classes with 600 images each. Each image has "fine" label (specific class) and "coarse" label (superclass), with the 100 classes grouped into 20 superclasses. The dataset is split into 50,000 training and 10,000 test images. It a subset of the Tiny Images dataset and is commonly used for evaluating image classification algorithms. While the original dataset does not specify a license, various distributions, such as those provided by TensorFlow Datasets, are released under the Creative Commons Attribution 4.0 License.
QualityMRI The QualityMRI dataset, introduced by Obuchowicz et al. [2020], is part of the Data-Centric Image Classification (DCIC) Benchmark, which aims to evaluate the impact of dataset curation on model performance. The dataset contains 310 magnetic resonance (MRI) images spanning various quality levels and is designed to assess the MRI image quality. The dataset is publicly available under the Creative Commons BYSSA 4.0 license.
The SVHN dataset, introduced by Netzer et al. [2011], consists of over 600,000 32x32 RGB images of digits (0-9) obtained from real-world house number images in Google Street View. It includes three subsets: 73.257 training images, 26,032 test images, and 531,131 additional images for extra training. The dataset is designed for digit recognition tasks with minimal preprocessing. While the original dataset does not specify a license, various distributions, such as those provided by TensorFlow Datasets, are released under the Creative Commons Attribution 4.0 License.
SVHN
22

---

Places365 Places365, introduced by Zhou et al. [2018], is large-saale scene recognition dataset containing 1.8 million training images across 365 scene categories. The validation set includes 50 images per category, and the test set has 900 images per category. An extended version, Places365- Challenge-2016, adds 6.2 million images and 69 new scene classes, totaling 8 million images over 434 categories. While the original dataset does not specify license, various distributions, such as those provided by TensorFlow Datasets, are released under the Creative Commons Attribution 4.0 License.
FMNIST Fashion-MNIST (FMNIST), introduced by Xiao et al. [2017]. is dataset of Zalando's article images, comprising 70,000 28x28 grayscale images labeled across 10 classes, such as T- shirt/top, Trouser, and Sneaker. It includes 60,000 training and 10,000 test images and serves direct replacement for the original MNIST dataset for benchmarking machine learning algorithms. FMNIST is publicly available under the MIT License.
ImageNet ImageNet, introduced by Deng et [2009], is a large-scale image database organized according to the WordNet hierarchy, containing over 14 million images across more than 20,000 categories. The ILRC subset (ImageNet-1KK includes 1,281,167 training images, 50,000 validation images, and 100,000 test images across 1,000 classes. The dataset is available for free to researchers for non-commercial use.
B.2 Models
Fully-Con mally-Connected Network We train fully-connected neural networks on the ChaosNLI dataset.
The network consists of 4 linear layers with [768 256 - 64 16 - 3] units with ReLU activations, except for the last layer, which has Softmax transformation transform the logits into probabilities. We use the hyperparameters (cf. Table 2) similar to the optimal parameters found Javanmardi et al. [2024].
ResNet18 For experiments on the CIFAR-10 dataset, we use the Py PyTorch ResNet-18 implementation and hyperparameters provided by https: //github// cww/coo///oo//. specifically optimized for CIFAR- and is trained from scratch, without any pretraining on ImageNet. For experiments on the QualityMRI dataset, we use the ResNet18 implementation from the PyTorch torchvision package with random initialization, i.e. no pretrained weights.
cifar. This model
Hyperparameters Each dataset trained using dedicated set of hyperparameters as presented in Table 2. We evaluated multiple configurations and selected the best-performing ones for each dataset. To ensure fair and consistent comparisons, all models trained on a given dataset, both our approach and the baselines, requires a KL-divergence penalty of le 7 and zero weight decay when using the Adam optimizer [Kingma and Ba, 2015]. When apply the SGD optimizer with learning rate scheduler, namely Cosine Annealing [Loshchilov and Hutter, 2017], CreBNN requires additionally a momentum of 0.9 enable effective learning.
the same hyperparameter settings. The only exception is the CreBNN, which
Table 2: Hyperparameters used for each dataset.
Hyperparameter
ChaosNLI
CIFAR-10
QualityMRI
Model
FCNet
ResNet18
ResNet18
Epochs
300
200
200
Learning rate
0.01
0.1
0.01
Weight decay
0.0
0.0005
0.0005
Optimizer
Adam
SGD
SGD
Ensemble members
20
20
LR scheduler
CosineAnnealing 100
CosineAnnealing
Tobias value
100
100
23

---

B.3 Out-of-Distribattio Detection
We use the SVHN [Netzer et al., 2011], Places365 [Zhou et al., 2018], CIFAR-101 [Krizhevsky et al., 2009], Fashion-MNIST [Xiao et al., 2017], and ImageNet [Le and Yang, 2015] datasets for Dut-of-Distribution detection.
The Out-of-Distribution task is treated as a binary classification task where the epistemic uncertainty used the classification criterion. In order balance the data, we sample 10000 instances from the test set of the respective datasets. On both the in- in-Distribution data (CIFAR-10) and the Out-of- Distribution data, the same transforms normalization and resizing 32 by 32 pixels are applied. After computing the epistemic uncertainty the area under the receiver operating characteristics curve (AUROC) is computed and used as the comparison metric.
B.4 Computing Uncertainty
Computing the total and aleatoric uncertainty involves solving a per instance optimization problem, after which the epistemic uncertainty is obtained their difference (see Equation (5)). This op- timization is performed using SciPy's minimize function with the SLSQP solver and the default parameters.
Interval-Based For the interval- based credal sets, the optimization is initialized with the mean of the predicted distributions, bounded between the lower and upper probabilities for each class, and constrained to ensure that the solution forms valid probability distribution, namely, the class probabilities must sum to 1.
Convex Hull For the credal sets based on the convex hull, the optimization is done on the weights of convex combination, instead of the probability distribution. Uniform weights are used as the initial value, the weights are bounded between 0 and 1, and constrained to sum to 1.
Estimated Computing Time Here, we provide an estimated upper bound of the computation time for computing the lower and upper entropy of a credal set based on the example of the largest possible credal set, namely the full probability simplex for a 10-class problem, such as CIFAR-10. For this largest set, computing the upper entropy using interval-based credal set takes on average 0.03 seconds, while computing the lower entropy takes about 0.02 seconds. For the convex hull-based credal set of the same size, the average computation time is 0.07 seconds for upper entropy and 0.03 seconds for lower entropy. To simplify our estimation, we average these times and assume that for single instance it takes 0.03 seconds to optimize the lower entropy and 0.04 seconds to optimize upper entropy, summing up to 0.07 seconds of computing time per instance.
In the cutt-f--Disribbutiin (OoD) detection experiments, we need to compute both lower and upper entropy for each instance in both the in-Distribution (iD) and OoD datasets. Each dataset contains 10, 000 instances, resulting in roughly 23 minutes of computation time per model to obtain the epistemic uncertainty across all instances. Since we run three seeds per model and evaluate 12 different models, the total runtime for single OoD experiment on one dataset is approximately 14 hours. As we evaluate OoD detection across five datasets, the total computing time amounts to around 70 hours excluding any time needed to train models beforehand.
B.5 Computing Coverage
Coverage is evaluated by checking whether the ground-truth distribution lies within the predicted credal set. For the interval-based approach, this involves verifying that each class probability of the ground-truth distribution falls between the corresponding lower and upper bounds of the credal set. For the convex hull approach, assess whether the ground-truth distribution can be expressed as convex combination of the extreme points defining the credal set. This optimization is done using the SciPy linprog function.
B.6 Baselines
We list all details regarding the implementations of the baselines that were used in the paper. In general, all baselines were implemented in own code base.

---

Credal Wrapper (CreWra) The Credal Wrapper was initially implemented in TensorFlow, but we reimplemented it in PyTorch to ensure compatibility with our framework. It follows a standard ensemble learning approach, training multiple models independently. Like our method, the Credal Wrapper constructs credal sets using class-wise upper and lower probability bounds, making well-aligned with our implementation. Overall, we closely follow both the original paper and their available implementation [Wang et al., 2024b].
Credal Ensembling (CreEns..) Since no official code was available for neural network implemen- tations of Credal Ensembling. we reimplemented the method ourselves. Our implementation closely follows all details provided in Nguyen et al. [2025]. The approach builds on standard ensemble training procedures, with inference adapted according to their proposed method of sorting predictions based on a distance measure and selecting only a% closest predictions construct credal sets. In our experiments, we use the Euclidean distance measure and evaluate several values of a.
Credal Deep Ensembles (CreNet) As the official implementation of Credal Deep Ensembles is only available in TensorFlow, we reimplemented the method in PyTorch to ensure compatibility with codebase. Our implementation closely mirrors the original TensorFlow code, particularly in adapting the model architecture and loss function. Specifically, we replace each model's 's final linear layer with head comprising linear layer outputring 2 classes (representing upper and lower probability bounds), followed by a batch normalization layer and the custom IntSoftmax layer. We also reimplemented the proposed loss function, which computes cross-entropy loss for the upper bounds and selectively backpropagates the lower-bound loss only for the 5% of samples with the highest loss values, described Wang al. [2024a]. In our experiments, we use å 0.5, as suggested in Wang et al. [2024a].
Credal Bayesian Deep Learning (CreBNN) No code or implementation details for Credal Bayesian Deep Learning (CreBNN) were made publicly available, and despite multiple attempts to contact the authors, we received no response or further clarification. As a result, reimplemented the method ourselves based solely on the high-level description provided in the paper. In our im- plementation, each ensemble member is a Bayesian neural network (BNN) trained with variational inference using different priors, with prior means sampled from [-1, 1] and standard deviations o from [0.1, 2] form diverse prior set. During inference, we draw one sample from each BNN to obtain a finite set of probability distributions, and construct the credal set as the convex hull of these predictions.
B.7 Computing Resources
To run the experiments presented in this work, we utilized the computing resources detailed in Table 3. The total estimated GPU usage amounts to approximately 750 hours.
Table 3: Specifications of Computing Resources.
Component
Specification
CPU
AMD EPYC MILAN 7413 Processor, 24C/48T 2.65GHz 128MB L3 Cache 2 x NVIDIA A40 (48 GB GDDR each)
GPU
RAM
128 GB (4x 32GB) DDR4-3200Hz ECC DIMM
Storage
2 480GB Samsung Datacenter SSD PM893
25

---

C
Additional Experiments
This section presents additional results, including ablation studies and alternative hyperparameter configurations, that complement the findings reported the main paper.
C.1 Out-of-Distribution detection
In addition to the OoD experiments in Section 5, we evaluate broad range of values for the CreEns approach.
Table 4:
detection based on epistemic uncertainty. CIFAR-10 is used as the Di-Distribution dataset. The mean and standard deviation over 3 runs are reported. Best performance is in bold.
Method
SVHN
Places
CIFAR-100
FMNIST
ImageNet
CreWra
0.957
0.91610.001
0.916::.0000
0.95211.0008
0.001
CreEnso.95
0.500=10.000
0.0000
0.500
0.500:10.000
0.500:10.000
CreEnsp.9
0.921
0.879
0.883+0.000
0.915
0.857:100002
CreEnSp.s
0.937=0.001
0.00
0.900+#.001
0.929440.001
0.875++0.001
CreEnsg.6
0.944-10.002
0.902=0.001
0.906+0.000
0.93541..00
0.88111..00
CreEnso.
0.947=0.001
0.906=1.000
0.908110.000
0.940+4..00
0.883110.001
CreEnso.2
0.950=0.001
0.909
0.91
0.946+00.0m 001
0.885++0.00
CreEnSo.0
0.955=0.001
0.913=0.000
0.91 0.000
0.949+0.000 0.001
0.888+00.00 000
CreNet
0.943=0.008
0.918=0.000
0.91 1.912+0.000
0.951+0 002
0.884400000
CreBNN
0.907
0.885 0.002
1.880110.022
0.935 1..022
0.859:10.002
CreRL1.0
0.948
0.918
0.916:10.001
0.957 000002
0.889
CreRLo.se 0.95
0.917=0.013
0.910=#.001
0.901+:.000
0.945=1:.004
0.878:10.005
CreRI .0.9
0.918=0.011
0.907
0.896:+0.001
0.944+0.000
0.874+0.000
CreRL
0.906=0.008
00.00
0.80440.003
0.936:10.009
0.865:10.002
CreRI-o.6
0.862
0.004
0.852+1.000
0.893110.00
0.837
CreRLoo.4
1..09
0.821 10.007
0.796110.007
0.815.10.029
0.788110:110
CreRLo.2
0.582=0.011
0.736=0.010
0.700++0001 000
0.676++0.04 0446
0.698++0.0 013
C.2 Ablations
In the following ablation study, we evaluate two factors: the impact varying the initialization constant in our proposed ToBias initialization, and the effect of changing the number of ensemble members. The study is conducted on the ChaosNLI dataset.
C.2.1 ToBias Initialization
We study the impact of the ToBias initialization constant on the coverage and efficiency of the resulting credal predictor on the ChaosNLI dataset. To do we vary the constant, taking values (5, 10, 20, 30, 50, 80, 100, 200, 500). The ensemble is then trained as proposed in Algorithm 1. The coverage and efficiency Pareto front is shown in Figure 5 with the mean and standard deviation over three runs. We split the results into three Figures for better readability.
For low values of 3, the coverage of the ensembles is rather low and as 3 increases, the coverage also increases. For large values B, c.g. 3 200 and 500, the ensembles with large values no longer able to reach into the low coverage, high efficiency region. In particular, 8 = 500 has higher coverage and lower efficiency for larger values of than for smaller values of a. This may caused by converge problems due to having very large bias for particular class. This causes the individual ensemble members not always converge to their desired threshold, hence resulting in large credal sets (with high coverage and low efficiency), because the border of the credal set is not reached. In essence, the 3 value provides the ability to slightly shift the Pareto front to reach the desired coverage, efficiency region (in addition to a).
26

---

PC 05, 10, 50, 20.
BC (80, 500, 2003
#e (200000 5. 200, 10, 80, 50, 20, 500. 30)
Eitelency
65-
64
G4-
64.
3.4
24
62-
02
92-
93
RA Coverage
Covorage
Coverage
B-5
B-10
B-20
B-30
p-50
B-80
1-100
C- 200
P- 8-000
Figure 5: Pareto front between coverage and efficiency for different values of ToBias constant B. Left: low values of B3, middle: high values of f3, right: all values of [3.
C.2.2 Number of Ensemble Members
We study the impact of the numbers of ensemble members M on the coverage and efficiency of the resulting credal predictor on the ChaosNLI dataset. To do so, we vary the constant, taking values M E {1, 2, 3, 4, 5, 10, 20, 30, 50). The ensemble is then trained as proposed in Algorithm 1. The coverage and efficiency Pareto front shown in Figure with the mean and standard deviation over three runs. We split the results into three Figures for better readability.
M (1. 4,5,
Mc (10, 20, 50, 303
M 2. 4, 5, 50, 20, 303
Fiteiency
68-
04
2
64-
64-
3.4
24
62
02
02
03
X2
8.4
KA Coverage
0.8
Coverage
Coverage
M-1
M-2
M-3
Mm4
M-5
M- 10
M-20 20
M-30
M-50
Figure 6: Pareto front between coverage and efficiency for different number of ensemble members M. Left: low values of M, middle: high values of M, right: all values of M.
Naturally, when M = 1, the coverage is O and efficiency 1, because the credal set reduces to a point prediction. With M 2 and M = 3. the
ensemble does not have coverage and efficiency O, because there are not enough ensemble members, and hence probability distribution, to span the whole probability simplex for this class problem. As M increases, the Pareto front starts span more of the Pareto figure, until stabilizing around M = 20. After this, the number of ensemble members does not have a significant impact on the resulting Pareto front anymore.
C.3 Data Uncertainty
Table 5: Average entropy of ground- fruth distributions in the test sets of ChaosNLI, CIFAR-10 10 and QualityMRI.
The datasets used in our experiments, namely ChaosNLI, 10, and QualityMRI, differ in their inherent levels of (aleatoric) uncertainty. This is reflected, for example, in the degree of disagreement among the collected an- notations, which serve as the ground-truth distributions in our evaluations. To provide further insight, we report the average entropy of these ground-trutt distributions in Table 5.
Dataset
Avg. Entropy
ChaosNLI CIFAR-10
0.932
0.223
QualityMRI
0.782
27

---

C.4 Performance of Ensemble Members
We visualized the trade-off between coverage and efficiency in Figure 4, and provide the correspond- ing numerical values in Table 6. We also report the accuracies of the individual predictors within each ensemble for both the baselines and our proposed method provide insight into their standalone performance (cf. Tables and 8). For CreNet, whose ensemble members directly predict probability intervals, we use their intersection probability technique to derive pointwise predictions [Wang et al., 2024a].
Table 6: Coverage and efficiency for different methods across datasets.
ChaosNLI
CIFAR-10
QualityMRI
Approach
Coverage
Efficiency
Coverage
Efficiency
Coverage
Efficiency
CreRLo.o
1.000:0.0000
0.00010.000
1.000=0.000
0.00011.0000
1.000
0.000
CreRLo.2
0.851:1.0m
0.312:1.025
0.754:10.00s
0. 066:10.003
0.796=10.66
0.261=t0.053
CreRLo.4
0.707=10.07
0.358+0.013
0.680
0.8244
0.747
0.2031+0.010
Lo.6
0.559410.034
0.488110.012
0.634
0.862440.004
0.677 0..03
0.360+000116
CreRL no.8
0.410:10.021
0.631 10.022
0.581 0.0005
0.89441..000
0.613
0.417:40025
CreRLo.9
0.294=000007
0.716=t0.010
0.556= 001
0.911+0.000 0.00
0.548=00.01
0,475++0.05
CreRLo.9t 0.95
0.260=0.022
0.743+0.00 012
0.543 002
0.920+0 001
0.511=0.016
0.508+0.037
CreRL1.0
0.246=0.011
0.757=0.000
0.498=0.000 001
0.950+0.000 .000
0,526+0000 00000
Cre
0.453+0.050
0.607+0.08 037
0.450=0.00f
0.963+00 000
0.355+0.057
0.608+0.021
CreNet
0.001
0.97810..00s
0.004
0.999110.000
0.188 1..020
O. 0.702
CreBNN
0.105
0.649
0.331100005
0.871110.00 0.003
0.898 10313
0.090::0.124
CreEnso.o
0. 0.2
0.607
0.000=0.0000
0.963:t0.000 0.000
0.329
0.611#0.uzz
CreEnsg.2
0.7007
0.769
0.000=0.000
0.983++0.000 0.000
0. 10.60
0.787
CreEnso.4
0.096:10.008
0..06
0.000=0.000
0.986+10.000
0. 1.118
0.84410.002
CreEnsg.6
0.043
0.88610.000
0.000
0.988110.000
0.081 +0.057
0.899-10.012
CreEnSo.s
0.012=0.002
0.941 .002
0.000=0.00m
0.993++00000
0.054=0.033
0.950+0.008
CreEnSo.9
0.000=0.000
0.976 0011
0.000=0.00m 0.000
0.997++.000
0.016=0.023
0.983
CreEnso.05
0.000=0.000
1.000=0.000
0.000=0.000
1.000+0.000
0.000
1.000+0.000
28

---

Table 7: Accuracy per ensemble member of CreRL for varying a values on ChaosNLI, CIFAR- 10, and QualityMRI.
ChaosNLI
0.0
0.2
0.4
0.6
0.8
0.9
0.95
1.0
0.678=20.017
0.678=0.017 0.678=0.011
0.678=c.017
0.678+0.017
0.678
0.678=0.017
0.678+00.017
2
0.370=20.02
0.526
0.512=0.063
0.577=0.011
0.608
0.66140.025
0. 1.68220.016
0.678+0.019
3
0.5499 a.453
ous
0.407
0.604.0..45
0.634:10.016
0.66820.003
0. 1..63
0.657 a:0 uzu
0.178:00.01
0.468 n22
0.483
0.483
0.651
0.660
0.661 100010
6.667 4:0 006
0.568 0.370
0.425200.05
0.56020.012
0.624
0.656220006
0.677:0.017
0.68611.00
G
00.03000.010
0.505
0.530=0.023
0.585=0.025
0.647
0.603+0.012
0.676 0.674=0.007
0.1 0
0.453
0.506g0.001
0.59420.014
0.637 020
0.647.20.00
0.664:20.005
1.675:t0.000
0.370==0.012
0,585
0.1944=..19
0 0.585=0.030
0.646
0.673+0.022
0.677
0.052+0.014
Member T78
1..53-1.000
0.595
0.604.0.029
0,663 ouT
0. 0.66..0.002
0,661 10.023
0.670
10
0,178 78
0.4
0.514=0.050
0.552=0.054
0.650 004
0.61840..00
O. 1.661an.013
0.675=+0.008
11
0.370
0.533 u61
0.584
0.587
0.676 ou8
0.684.2.00
0.683 0.684200012
12
0.153
0.6:14...011
0.638
0. 0.650=0.011
0,659
0.670
0.672:0.005
0. 0.665
13
0.1 78220.01
0.510 005
0.584g0.0r9
0.6 0.633220.2
0.652 009
0.664220006
1.6682000000
0.670
14
0.310
0.060=0.040
0.603=0.015
0. 0.632=0.081
0. 0.608++0.00 020
0.675
0. 0.684=0.014
0.676
15
0.453:20.010
0.644 019
0.643=0.013
0.662-0.007
0.679 014
0.685
0.670 0.677=00.07
16 00.0040
0.630
0.637=0.001
0.647=0.013
0,651#0.005
0.608
O. 1.658
0.671 #0.002
17 0.370
0.623
0.6442=00017
0. 1.65520.010
0.68411.0014
0.688
0. n..80
0. 6..87 10.010
18
0.652 00.05
000
0.655=0.009
0, 0.6044.0.014
0.68440.000
0.609
1..63
1..63330000
19 0.117800.11
0.650
0.659
0. 0.6
0.67310.0x
0.67620.005
0. 6..07.00001
0.667 10.010
20
0.370=0.012
0.68920.017
0.677=0.014
0.679=0.024
0.684 020
0.68820.024
0.671 #0.026
0.673
CIFAR-10
0.0
0.2
0.4
0.6
0.88
0.9
0.95
1.0
1
0.935=s0.00s
0.935 ons
0.935=0.00s
0.935=0.00s
0.935 00s
0.935=a.008
0.943 0.935=*0.003
0. 0.100:20.000
0.490220022
0.711=0.090
0.792=0.014
0.860 000
0.88920.008
0.898220005
0.9344
00.000000000
0.582
0.725=0.027
0.809=0.004
0.873
0.891#0.002
0.898
0.934
4
0, 1..0000..00
0.59110.um 009
0.722=0.00s
0.816=00000
0,871
0,80110.00
0.902
0.936:1..00
5
0.090
0.6-15 0.6
0.754==.002
0.807=0.0011
0.874 002
0 0.8862..004
0. n.o0a
0. 0.935=#0.002
0.098.1..0 000
0.704 010
0.775
0, 0.843320.002
0.87U 006
0.891 003
0.903 100000
0.934 DoM
0, 10.000000000
0.697
012
0.782=0.004
0,851
0,881
0.893.00000
0,902
0.936
8
0..00020.000
0.748 015
0.801 21:01
0.85020.06
0.880 006
0.897.0.000
0.906220.04
0.935-1:0.00
Member m72
0.101 =0.000
0.765 025
0.813=00020
0.848=0.004
0.890 001
0.89620.008
0.908=00001
0.936 0002
10
10.100000000
0.779 007
0.830=00007
0.862=0.011
0.886 012
0.895
0.904200008
0.936 001
11
O. 1000=000000
0.790
0.831
0.865=0.005
0,880
0.90
0.908
0. 0.936=#0.001
12
0. 1000000000
0.821 1: 009
0 0.85420.013
0. 0..000
0. 0.80010.000 002
O 0.902200004
0. 0.907
0. 0.934 0.0.00
13
0. 00.00=00.000
0.82
001
0.850=0.002
0.875=0.008
0,884+0.000 0000
0.904
+0.002
0. 0.937 +:0 40.002
14
00.10010.000
0.840 002
0.86020.00
4.
0.891 002
0.90620.006
0.91 2200004
0.934
15
0.099=00.000
0.849 007
0.868=0.006
0.879=0.004
0.898 003
0.905=20.001
0.91 1:0000
0..33 0.00
16
0.098.20.00
0.868 000
0.87320.00
0.88720.004
0.904 0ot
0.97020.008
0.91 1200002
0.9366 002
17
0.100=0..00
0.863 008
0.879=0.00s
0.888=0.004
0.901 006
0.908
0.909
0.935
18
0.100==0.00
0.879
0.892=0.005
00.08
0.902 000
0.9132.0.06
1.916=0.001
0.9311:0.001
19
0. 1.101=0.00m
0.887 =0.70 007
0.897=0.004
0.902=0.001
0.909 002
0.915=a.002
0.91 1.9.00
0.936=00.001
20
0. 0.100
0.903 005
0.90420.003
0..08
0.910 004
0.920
0.921 10.001
0.935
QualityMRI
0.0
0.2
0.4
0.6
0.8
0.9
0.95
1.0
0.624 euu
0.602 uz0
0.6022..020
0.602.1.000
0.602.0.00 o20
0.602a..00 020
0.532.10.00 0.624
0.376=20.0us
0.511so.0p
0.532=0.023
0.570=0.o4s
0.575
0.597
0.58620.055
0.4955::0.11
0.62423.00s 3
0.608 015
0.67220.027
0.62920.02
0.61810..n.
0.6241..005
0.655220.022
0.570...076
0.376=20.008
0.555
0.548 1os
0.543=0.020
0.581+0.015
0.618*0.027
0.61 1.61.820.063
0.51 1:0.112
0.624=0.008
0.608 n27
0.602:20.r22 0.645
0.597
0.62
0.651 20.008
0.570 101
0.376=20.00s
0.5-13=0.020
0.581=0.025 0.586=0.040
0.591
0.570
0.532 a0.00
7 0.621
0.618 042
0.624
0. 00.00
0.61820.000 030
0.6402o.ooz
0. a..20
0.51 1:0124
0.376=
0.597 090
0.575=0.046 0.586=00002
0.597 000
0.501
0, 0.602=0.020
0. 0.500
Member 171
0.624
0.6-10 0go
0.651
0 0863320.0
0.613
0. J.629.00013 03645.2.003
0.522.10.07
10
00.76:20.00s
0.565 0065520.000 035
0.60220.008
0.548=0.085
0.602 059
0.591 +
0.570=0.008
01505 0.95
11
U. 10.22
0.629 026
0.62422.020
0. 007
0.597 2.0 oxs
0.67320.0ee 066
0.581 200013
0.565 DO8
12
0.376=20.00s
0.538 015
0.570=0.055
0. 0.640=0.008
0.591 008
0.602
0.570=0.080
0.538
13
0.621=0.00s
0.688
0.613=0.013
0.677=0.02
0.656 00s
0.602
0.618 G1882.0422
0.559
14
0.376==0.00s
0.532 035
0.58 100.055
0.570=c.046
0.58G 020
0.591+0.020
0. 0.554=n.003
0. 0.538 0.117
15
0.624
0.6341 040
0.651=2..00
0. 0.000
0.651
ose
0.620 0203
O. 110.00
0.559
16
0.376620.0os
0.565
0.61820.000
0 0.58
0.57U
0.597
0.504220.008
U.51 6:166.00000 092
17
10.22
0.624 040
0.67720.035
0. 007
0.618 027
0.50110.003
0. 1..40000000
0.55-4 066
18
0.376=20.006
01575 055
0. 0.597=2.018 0.58620.008
0.591 +:0 000
0.570z0.006
O. 0.600200.027
0.522
19 0.624 ous
0.608 050
0.65620:045
0. 0.634220005
0.629 040
0.624-20.oom
0.61 820.000
0.527 1:0 077
20
0.376
0.608 027
0.570z0.065
0.608=0.020
0.597 026
0.60820.020
0.554400020
0..15
29

---

Table 8: Accuracy per ensemble member of baselines on ChaosNLI, CIFAR-10, and QualityMRI.
ChaosNLI
CreWra
CreEnsa
CreBNN
CreNet
0.658=0.013
#0.01
0.525+0.051 0.680+0.013
0.64820.012
0.676:20015
0.67420.031
0.488220 008
0..21
0.676=20.013
0.517
0.542+00.0m
0.636=0.012
0.606=0.008
0.683+0.030
0.485+0.000
0.651
10.011
0.533
0.534410.00
0.660=0.016
0.674
0.667
0..17:
7
0.637=0.014
0.677=0.009
0.675+0.012
0.518 0.00
8
0.667
0.670
0.67510.029
U.446.10.000
Member m
0.584
0.610=0.104
0.4622+0.1
0.66310.895
0.674110002
0.66810.017
0.531
0.64810.0117
0.653-10.015
0.52910.096
0.48411.00
12
0.660=0.816
0.662
0.68L+0.019
0.5499:0.071
13
0.66410.014
0.6 00601110.017
0.52610.105
0.487.1.000
14
0.65010.011
0.654110.006
L6.000
0.505.10.00
0.63520.098
0.65 20.018
0.673
0.458
0.6522..002
0.654410.007
0.6161:0 000.00000
0.486.10
0.62810.019
0.603.10.015
0.67510.030
0.518.10.0
18
0.652=0.009
0.605
0.598+0.109
0.519
19
0.66020.012
0.581 150057
0.6
0.487:10.000
20
0.651
0. 0.4804
0.536+0.116
0.560
CIEAR-10
CreWra
CreEns,
CreBNN
CreNet
0.942
0.955=20.000
0.875
0.941
0.944
0.954.10.000
0.87710.000
0.94310
0.004
0.05210.001
0.867.10.08
0.944
0.943=0.0022
0.952=00.00
0.88 +0.003
0.942:20.0m 001
0.943
0.952:10.001
0.87220.004
0.942:10 001
0.94310.000
0.952.10.
0.86010.006
0.942.10 002
00.00
0.9522+0.00
0.877
0.943
8
0.94220.001
0.95 1:0.001
0.88020.006
0.943
0.943=0.001
0.95 +0.000
0.879
0.944
Member
0.943=0.000 10
0.953=+0.000
0.873+0.007
0.943
11 0.942
0.953
00.02
U.9433:0.000
0.903=000000
0.953++0.00
0.879
0.9422=0.00m
0.9.330.0000
0.953++0.001
0.874+0.000
0.943+t0
0.945
0.952
0.872
U.941
15
0.942=0.n02
0.948
0.856=0.019
0.942
16
0.94210.000
0.941 1111110.00
0.80010.002
0.942.10.00 0000
17
0.9422..001
0.930
0.85411.0035
0.942.10.00
0.944=0.001
0.921 +00.000
0.872+0.007
0.9433=0.00m 0002
0.94310.001
0.906
0.87310.001
0.942.10.
20
0.9421..001
0.87210.002
0.86010.012
0.942.10.0
Quality
CreWra
CreEns
CreBNN
CreNet
00.57
0. 1.
0.5488
0.581
0.452=0.159
0. 0.409=0.11
0.484+0.115
0.559
0.4125.0.124
0.41910.3
0.46810.126
0.570...111
0.419=0.116
0.4466=0.11
0.54840.115
0.575=a0.002
0.39810.135
0.387
0.46810.126
0.548.0.1
6
1..031
0.398 158
0,55411.118
0.554.10.084
7
0.414
0.446
0.554:2.095
0.430...119
0.441 110. 182
0.55410.118
0.559 153
0.39210.137
0.419
0.47310.110
0.586.1.111
Member
0.446
0.430 140
0.462+0.130
0.570
11
0.425
0.414.1..
0.55410.118
0.602.1. 107
12
0.398
0.430
0.55410.118
0.565
0.398=0.150 13
01430==0. 140
0.575 0.446=+.107
0.40920.006
0.430 142
0.4
0.581 070
0.4
0.41440.00
0.468+0.126
0.624
0.414=0.107
0.425 167
0.554+0.118
0.608
17 0.41920.139
0.41442..1 119
0.55420.118
0.58 091
18
0.403
0.414#0.115
0.468+0.126
0.581+0.117
19
0.425
0.414+0.073
0.554+0.118
0.565
20
0.44
0.446620.068
0.5488
0.58110.100
30


Reviews:
[{"review_id": "2JmPMBd5Fi", "reviewer": "Reviewer_vaSE", "summary": "This paper presents a method to obtain a credal set prediction and introduces $\\alpha$, a threshold on relative likelihood to balance between coverage and efficiency. The credal set prediction is based on ensemble of model predictions. The approach is empirically evaluated on three datasets for AUC curve on efficiency vs. coverage while also including a study of OOD detection", "questions": "Why is there a need to do thresholding on relative likelihood? Let’s say, we train 10 models on CIFAR-10 and choose $\\alpha$ to be 0.9, how many models will be dropped?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n-\tThe method is relatively simple and presented very clearly.\n-\tThe use of ToBias approach to get a wide credal set at the start of training is commendable.\n\nWeaknesses:\n-\tThe experimental analysis lacks in depth. More datasets could be explored.\n-\tIt is claimed that the proposed outperforms existing methods but it is not evident in the results. The proposed approach clearly lacks in coverage (and by extension accuracy). The OOD results are comparable to the existing methods and do not outperform.\n-\tA clear study of the accuracy metric is missing.\n-\tThe only novel thing, as per my understanding, is the use of thresholding on relative likelihood to select the best models. While interesting, it does not make a case for NeurIPS level publication.", "was_revised": true, "final_justification": "I appreciate the author's response. It has cleared my concerns and confusions. Therefore, I raise my score.", "rebuttal": "We gratefully thank you for reading our manuscript and providing a valuable review.\n\n> **W1:** The experimental analysis lacks in depth.\n\nWe would like to clarify that our experimental analysis focuses on evaluating the quality of the uncertainty representation through the trade-off between coverage and efficiency. The objective is to construct credal sets that contain the ground-truth distribution (coverage) while being as informative (small) as possible (efficiency). Ideally, both metrics would be close to 1, corresponding to the top-right corner in the Pareto plot (Figure 4). However, this ideal is rarely reachable in practice, as improving efficiency often comes at the cost of reduced coverage, and vice versa. \n\nTo illustrate this trade-off, we provide experiments on three diverse datasets and compare our method to four baselines. As also positively noted by Reviewers gS33 and BkoJ, we believe this provides a well-balanced and comprehensive evaluation of our approach, and in fact goes beyond the experimental scope of comparable credal prediction papers. Additionally, we include an out-of-distribution (OoD) detection task as a sanity check to demonstrate that our method captures meaningful uncertainty, even though improving downstream task performance is not the primary aim of our work.\n\nWe hope this clarifies our experimental design and would appreciate your feedback on which aspects are still lacking.\n\n> **W2:** More datasets could be explored.\n\nAs noted in the paper, our evaluation relies on datasets with first-order annotations in the form (ground-truth) probability distributions over classes, which are essential for computing the coverage metric. Such datasets are difficult to find, but we identified and used three that support meaningful evaluation, demonstrating the effectiveness and adaptability of our approach. We gladly welcome suggestions for additional datasets to expand our empirical analysis in future work.\n\n> **W3:** It is claimed that the proposed outperforms existing methods but it is not evident in the results. The proposed approach clearly lacks in coverage (and by extension accuracy). - A clear study of the accuracy metric is missing.\n\nCoverage: Based on the results in Figure 4 (and Table 6 in the appendix), we do not understand the concern regarding lack of coverage in our approach. Quite the opposite, one of the key strengths of our method is the ability to control the trade-off between coverage and efficiency by adjusting the threshold $\\alpha$. This allows us to move along the Pareto front, outperforming all baseline methods across a range of settings. In fact, ours is the only method that reaches high coverage regions by varying $\\alpha$, while others remain stuck in low coverage, often failing to include the ground truth.\n\nAccuracy: We are not entirely sure what you mean by accuracy. Typically, accuracy refers to the fraction of correct predictions in a setting where both prediction and ground truth are class labels associated with an instance x. In our setting, however, the ground truth is a probability distribution (associated with x), and the prediction a set of such distributions (a credal set). In this setting, it is not at all obvious how accuracy should be defined. \n\nThat said, please note that we do report standard accuracy scores in Tables 7 and 8 in the appendix. We did this as a kind of sanity check, to make sure that the individual models used in our ensembles are well trained. As these are still probabilistic models, we considered a prediction correct if the class with the highest predicted probability coincides with the class with the highest true probability. \n\nWhen considering these tables, note that the individual ensemble members for lower $\\alpha$ values may also have lower accuracy values. This is expected, and even desirable, as we wish to include models with a lower (relative) likelihood, which will in turn have an accuracy that is lower than the MLE (the first row of the tables). This is in line with our method design.\n\n> **W4:** The OOD results are comparable to the existing methods and do not outperform.\n\nIndeed, the OoD results are comparable to the baselines as also claimed in the paper. We would like to emphasize again that our goal is not to show superior performance on this downstream task, but to demonstrate that a competitive OoD detection is possible with our method. In particular, a major benefit of our approach is that it offers the user the flexibility to choose a particular relative likelihood threshold. Thus, the user could opt for a large $\\alpha$, thereby increasing efficiency and OoD performance, or a smaller $\\alpha$, thereby increasing coverage but sacrificing efficiency and OoD performance (see Figure 4 in the paper).\n\nTo further highlight the flexibility of our approach and the comparable results to credal wrapper, we analyze the mean epistemic uncertainty (EU) values on the in-distribution (iD) data (CIFAR-10) and one OoD dataset (ImageNet) in the table below.\n\n| $\\alpha$ |   0.2  |  0.4  | 0.6   |  0.8  |  0.9  | 0.95 | 1.0  | CreWra |\n|---------|:--------:|:-------:|:--------:|:--------:|:-------:|:--------:|:-------:|:------------:|\n| iD       | 2.29 | 1.74 | 1.34 | 0.95 | 0.79 | 0.70 | 0.45 |    0.38   |\n| OoD   | 2.78 | 2.60 | 2.47 | 2.30 | 2.20 | 2.14 | 1.96 |    1.83   |\n\nNotably, increasing $\\alpha$ shrinks the credal sets for iD data more than it does for OoD data, hence the gap is highest for $\\alpha = 1$, as evidenced by the mean EU values. This results in an increased ability to discriminate between iD and OoD data. Broadly speaking, reducing $\\alpha$ (purposely) increases the uncertainty for iD data and makes it less distinguishable from OoD data (note that $\\alpha = 0$ would produce the same vacuous credal sets for both iD and OoD data). Intuitively, increasing uncertainty and including predictions with lower likelihood indeed means less exploitation of the iD data. This explains why $\\alpha = 1$ is best for discriminating between iD and OoD data -- which, of course, does not mean that it also yields the best or most faithful representation of uncertainty. Note that with $\\alpha=1$, only models with a high (relative) likelihood are included, similar to the “regular” ensemble learning used in the credal wrapper method, which may explain their similar performance in general. \n\n\n> **Q1:** The only novel thing, as per my understanding, is the use of thresholding on relative likelihood to select the best models.\n> Why is there a need to do thresholding on relative likelihood? Let’s say, we train 10 models on CIFAR-10 and choose  to be 0.9, how many models will be dropped?\n\nThis can’t be said in general. Particularly, our algorithm does not involve a post-hoc selection step, hence trained models will not be dropped. In fact, the threshold $\\alpha$ (e.g. 0.9) is used to stop the training process once the model reaches the specified threshold. Consequently, all models have a relative likelihood greater or equal to 0.9, so that none of them is dropped (see Algorithm 1 in Section 4). Recall that the ensemble is conceived as a discrete approximation of the true $\\alpha$-cut of the relative likelihood function. The more ensemble members we find inside that cut, the better the approximation will be.\n\nLet us reiterate the main purpose of our method, namely to approximate the set of all plausible models, and hence the set of all plausible (probability) predictions, where “plausible” is operationalized in terms of the $\\alpha$-constraint on relative likelihood: a model is deemed plausible if its relative likelihood exceeds $\\alpha$. \n\nIn addition to this conceptual contribution, we provide a practical and scalable framework for training credal predictors in complex machine learning settings. The novel thing here is clearly not the thresholding, but rather the training procedure for constructing ensembles. It is designed in such a way that an accurate approximation of the $\\alpha$-cut can be obtained from the ensemble members in an efficient and effective way (which, as already said, implies that as few models are discarded as possible). Standard ensemble learning is unsuitable for that purpose.\n\nWe hope to have adequately addressed your questions and remarks. If you have any further questions, we would be happy to discuss them. Otherwise, we would appreciate it if you would reconsider your score."}, {"review_id": "q29c5XkTVc", "reviewer": "Reviewer_F5Am", "summary": "This work deals with the task of learning neural network ensembles capable of cautious decision-making. The authors propose CreRL, which constructs this ensemble by learning models satisfying the relative likelihood constraint: each model’s (conditional) likelihood cannot be less than a given percent (\\alpha) of that of the maximum likelihood solution. To ensure diversity the authors (1) use early stopping, that is, stopping training once the neural network satisfies the relative likelihood constraint; (2) they propose an initialization strategy call ToBias, which sets the one of final layer biases to a high value, e.g., 100; models with this an initialization cover the probability simplex better, approximating prior ignorance. Inference is performed by computing the predictive probabilities under all the models and taking the minimum and maximum values to construct a closed convex credal set. These credal sets are evaluated on their efficiency and their coverage; the former measuring the smallness of the credal set, while the latter measuring whether the ground truth probability lies in the credal set. The authors perform this evaluation on the ChaosNLI, CIFAR-10, and QualityMRI datasets. The proposed method CreRL achieves efficiency and coverage. Moreover, varying the hyperparameter \\alpha allows it to tradeoff one for the other. The authors also evaluate out of distribution detection, using CIFAR-10 as the in-distribution data set, and SVHN, Places, CIFAR-100, FMNIST, and ImageNet as out of distribution data sets. They find that CreRL with \\alpha = 1 performs on par or better than the best performing baselines.", "questions": "The relative likelihood credal sets seem to be closely related to Rashomon sets in Rudin et al. (2024). Does CreRL approximate the Rashomon set of neural predictors?\n\nRudin, Cynthia, et al. \"Position: Amazing Things Come From Having Many Good Models.\" ICML 2024", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Cautious decision making is important in high-stakes domains like healthcare. This method allows highly expressive neural network models to be used with a principled, cautious decision-making strategy.\n2. The proposed method CreRL is simple and easy to implement\n3. Evaluation of credal set-based methods using coverage and efficiency metrics from conformal prediction literature.\n\nWeaknesses\n1. Section 5 Empirical results says that: “Our method (CreRL) with \\alpha 1 performs either the best or is on par with the best methods in OoD detection, except for the SVHN dataset, where the CreWra approach has a small advantage.” But, it is unclear why CreWra has an advantage in the SVHN data set.\n2. In the OOD evaluation, the best performing model has \\alpha = 1. This seems to be the same as an ensemble of fully trained maximum likelihood models, indicating that credal sets based on the relative likelihood constraint are not useful in OoD detection", "was_revised": true, "final_justification": "The authors have addressed all my concerns. So, I have updated my score to 5 (accept)", "rebuttal": "Thank you very much for your careful reading of our manuscript and for your valuable feedback. We appreciate your recognition of the relevance and usefulness of our approach for decision-making. \n\n> **W1, W2:** Section 5 Empirical results says that: “Our method (CreRL) with \\alpha 1 performs either the best or is on par with the best methods in OoD detection, except for the SVHN dataset, where the CreWra approach has a small advantage.” But, it is unclear why CreWra has an advantage in the SVHN data set. \nIn the OOD evaluation, the best performing model has \\alpha = 1. This seems to be the same as an ensemble of fully trained maximum likelihood models, indicating that credal sets based on the relative likelihood constraint are not useful in OoD detection\n\nIndeed, our method performs better for some datasets, e.g. FMNIST, and worse for SVHN, as you rightfully pointed out. Please note, however, that the difference on SVHN is rather small (0.009) and not statistically significant (in light of the standard deviation). This is consistent with our claim that our method performs comparably on the OoD task to the evaluated baselines. To analyze the performance on the SVHN dataset further, we present the mean epistemic uncertainty (EU) values on the in-distribution (iD) data (CIFAR-10) and out-of-distribution (OoD) data (SVHN) in the table below.\n\n| $\\alpha$ |   0.2  |  0.4  | 0.6   |  0.8  |  0.9  | 0.95 | 1.0  | CreWra |\n|---------|--------|-------|--------|--------|-------|--------|-------|------------|\n| iD       | 2.29 | 1.74 | 1.34 | 0.95 | 0.79 | 0.70 | 0.45 |    0.38   |\n| OoD   | 2.60 | 2.49 | 2.53 | 2.42 | 2.34 | 2.25 | 2.23 |    2.22   |\n\nNotably, we observe that increasing $\\alpha$ shrinks the credal sets for iD data more than it does for OoD data, hence the gap is highest for $\\alpha = 1$, as evidenced by the mean EU values. This results in an increased ability to discriminate between iD and OoD data. The pattern above can also be observed for other data sets. Broadly speaking, reducing $\\alpha$ (purposely) increases the uncertainty for iD data and makes it less distinguishable from OoD data (note that $\\alpha = 0$ would produce the same vacuous credal sets for both iD and OoD data). This explains why $\\alpha = 1$ is best for discriminating between iD and OoD data — which, of course, does not mean that it also yields the best or most faithful representation of uncertainty. Note that with $\\alpha=1$, only models with a high (relative) likelihood are included, similar to the “regular” ensemble learning used in the credal wrapper method, which may explain their similar performance in general. \n\nAs for the claim that credal sets based on the relative likelihood are not useful in OoD detection, we kindly disagree. A major benefit of our approach is that it offers the user the flexibility to choose a particular relative likelihood threshold. Thus, the user could opt for a large $\\alpha$, thereby increasing efficiency and OoD performance, or a smaller $\\alpha$, increasing coverage, but sacrificing efficiency and OoD performance. While our aim is not to obtain OoD performance, our method still performs competitively with smaller $\\alpha$ values such as 0.8.\n\n> **Q1:** The relative likelihood credal sets seem to be closely related to Rashomon sets in Rudin et al. (2024). Does CreRL approximate the Rashomon set of neural predictors?\n\nThis is an interesting question! Indeed, there seems to be a strong connection to Rashomon sets. Both Rashomon sets and relative likelihood credal sets describe the set of all models with a performance above some threshold [1]. The purpose is quite different though. While we are interested in uncertainty quantification and diversity of predictions, Rashomon sets focus on interpretability and (syntactic) model diversity (because interpretability is compromised if there are many different models that describe the data equally well).\n\nDespite the different objectives and motivations, they both have to address the same problem: the approximation of the set of plausible models [2]. In the approach proposed by us, ToBias initialization is used as a way to promote diverse outputs, whereas in the Rashomon set literature other techniques are used. An interesting avenue for future research could explore whether such techniques would also be beneficial for credal set generation. Thank you for bringing these works to our attention. We will make sure to highlight this connection in our paper.\n\n[1] On the Existence of Simpler Machine Learning Models. Lesia Semenova, Cynthia Rudin, and Ronald Parr. 2022.\n\n[2] Rashomon Sets for Prototypical-Part Networks: Editing Interpretable Models in\nReal-Time. Donnelly et al. 2025."}, {"review_id": "FwtWTEeceI", "reviewer": "Reviewer_gS33", "summary": "The paper proposes a method for generating credal predictors within a machine learning framework, with a focus on neural networks. The approach selects plausible models based on relative likelihood. To promote diversity among predictors, the authors introduce a novel initialisation method called ToBias. This strategy ensures that the ensemble begins in a state of full uncertainty, represented by initial predictions located at the vertices of the probability simplex. As training proceeds, the ensemble’s knowledge increases, leading to a progressive shrinking of the credal set.", "questions": "None", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The paper presents a structured approach to generating credal predictors using neural networks, with model selection based on relative likelihood. The method is technically sound and supported by experimental results. A key component is the ToBias initialisation strategy, which assigns initial predictions at the vertices of the probability simplex to promote diversity. While this is not a major conceptual innovation, it is a well-considered design choice that supports the overall framework. The authors provide a balanced evaluation, carefully stating the scope and limitations of their contribution.\n\nThe writing is clear and the organisation is logical. The paper presents its arguments methodically, with sufficient detail to support reproducibility. The clarity of the exposition makes the technical material accessible to readers familiar with the field.\n\nThe work is relevant to ongoing research on uncertainty quantification in neural networks. The proposed method is practical and may be of interest to those working with ensemble learning or imprecise probabilities. It addresses a persistent challenge in the area and offers a structured solution that can inform future research.\n\nAlthough the paper does not introduce an entirely new method, it coherently combines known techniques and applies them to a problem of practical importance. In line with the NeurIPS 2025 Reviewer Guidelines, the work provides a meaningful contribution through thoughtful adaptation and analysis of existing methods.", "was_revised": true, "final_justification": "I was already inclined to accept this paper, and the authors’ responses to the other reviewers’ critiques seem to have had a positive effect.", "rebuttal": "We thank you for the thoughtful and encouraging feedback. We appreciate the recognition of our structured approach to credal prediction, the practicality of the ToBias initialization strategy, and the clarity of our exposition.\nWe’re especially glad that you found our evaluation balanced and reproducible, and that our approach was seen as a meaningful contribution in line with the NeurIPS guidelines. Our aim was indeed to address a persistent challenge in uncertainty quantification with a principled and implementable solution."}, {"review_id": "MXUyGrBEYv", "reviewer": "Reviewer_BkoJ", "summary": "The paper proposes a method to learn credal set-valued predictors for classification problems. To train a credal set-valued predictors, standard approaches often use ensemble models, but those methods lack the notion of \"learning\" a credal set in the sense that there is no selection criteria going on. This is what makes this work shines. \n\nThe authors also proposed a smart initialisation approach to promote exploration during the training called ToBias.", "questions": "1. (about choosing alpha) Is there a way to propagate my epistemic uncertainty about the threshould to your framework? or does it even make sense to use some withheld dataset to choose some Pareto-ly undominated alphas? What is your opinion on this?", "limitations": "Limitations are clearly discussed.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n- the paper is well written. Concepts are easy to understand. The methods are principled.\n- I particularly appreciate the authors not using predictive accuracy of any sort (collapsing your credal set e.g.) as a metric, instead use coverage and diversity/efficiency as the comparison metric. \n\nWeakness:\n- As with most credal set-valued or IP incorporated predictors, there aren't a lot of discussion to regression problems. It is not a big deal but seems like your method should also work for regression problem too? (not the ToBias I suppose but the relative likelihood business)", "was_revised": true, "final_justification": "I remain very positive about this paper.", "rebuttal": "We gratefully thank you for carefully reading our manuscript and providing a valuable review. We’re glad to hear that you appreciated our choice to evaluate using coverage and efficiency rather than collapsing the credal set to a single prediction. We believe these metrics better measure and reflect the goals of imprecise prediction in representing uncertainty.\n\n> **W1:** As with most credal set-valued or IP incorporated predictors, there aren't a lot of discussion to regression problems. It is not a big deal but seems like your method should also work for regression problem too? (not the ToBias I suppose but the relative likelihood business)\n\nWe agree that the core idea, particularly the relative likelihood-based construction, should, in principle, extend to regression settings as well. We are currently exploring this direction in follow-up work. That said, applying our framework to regression requires specifying the form of the predictive distributions (e.g., Gaussian, etc.) and how to define credal sets over them. Additionally, metrics such as coverage and efficiency are less straightforward in regression and would need to be adapted or newly defined to reflect meaningful uncertainty representation.\n\nWe appreciate the suggestion and inspiration and we will report on it in follow-up research.\n\n> **Q1:** (about choosing alpha) Is there a way to propagate my epistemic uncertainty about the threshould to your framework? or does it even make sense to use some withheld dataset to choose some Pareto-ly undominated alphas? What is your opinion on this?\n\nWe are not entirely sure what you mean by (epistemic) uncertainty about $\\alpha$, because there is no “true” $\\alpha$ that we seek to estimate. Instead, $\\alpha$ is used as a control parameter to find a good compromise between coverage and efficiency (very much like a p-value in statistics, which controls the compromise between type-1 and type-2 error in hypothesis testing). What is indeed uncertain is the (expected) coverage and efficiency for a given $\\alpha$, and using extra data for estimating these quantities certainly makes sense (we are currently computing them on test data). \n\nIf we compute the entire Pareto-curve, as we do in the paper, then the user can choose the most appropriate point on this curve (and hence implicitly the corresponding $\\alpha$). Thus, there is no need to specify an $\\alpha$ beforehand. \n\nHowever, computing the entire Pareto-curve might be considered too costly. Then, the best point on the curve should be found in a more targeted manner. One could imagine, for example, that this is done in an interactive process with the human in the loop. The system could make a suggestion (e.g., a specific point on the curve for an initial $\\alpha$), the user may provide feedback, and the system can adapt the $\\alpha$ based on that feedback. This looks like a very interesting direction for future work, thanks for raising this topic."}]

Author rebuttal:

Meta-review: This paper proposes a principled framework for credal prediction based on relative likelihood, introducing novel methods to construct diverse ensembles that generate set-valued probability predictions with controllable trade-offs between coverage and efficiency. The reviewers converged on the view that the contribution is technically sound, novel in its use of relative likelihood for defining plausible model sets, and clearly written, with empirical validation across multiple datasets. While one reviewer initially raised concerns about the experimental evaluation, these were convincingly addressed in the rebuttal, and the reviewer increased their score accordingly. Other reviewers emphasized the clarity, structured design, and potential impact of the work in advancing uncertainty quantification, though they noted that connections to related areas (such as Rashomon sets) and extensions to regression could be explored further in future research. Overall, the rebuttal strengthened the consensus around the paper’s novelty, soundness, and relevance, and I recommend to accept this paper as a poster.
Decision: Accept (spotlight)