Credal Deep Ensembles for Uncertainty Quantification

Authors: kaizheng  wang
fabio  cuzzolin
shireen  kudukkil manchingal
keivan  shariatmadar
david  moens
hans  hallez
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 17:11:27

Abstract: This paper introduces an innovative approach to classification called Credal Deep Ensembles (CreDEs), namely, ensembles of novel Credal-Set Neural Networks (CreNets). CreNets are trained to predict a lower and an upper probability bound for each class, which, in turn, determine a convex set of probabilities (credal set) on the class set. The training employs a loss inspired by distributionally robust optimization which simulates the potential divergence of the test distribution from the training distribution, in such a way that the width of the predicted probability interval reflects the epistemic uncertainty about the future data distribution. Ensembles can be constructed by training multiple CreNets, each associated with a different random seed, and averaging the outputted intervals. Extensive experiments are conducted on various out-of-distributions (OOD) detection benchmarks (CIFAR10/100 vs SVHN/Tiny-ImageNet, CIFAR10 vs CIFAR10-C, ImageNet vs ImageNet-O) and using different network architectures (ResNet50, VGG16, and ViT Base). Compared to Deep Ensemble baselines, CreDEs demonstrate higher test accuracy, lower expected calibration error, and significantly improved epistemic uncertainty estimation.

Main paper: 
 introduction: 
 The quantification of the uncertainty associated with neural network predictions has recently attracted increasing attention, to enhance the reliability and robustness of neural networks. Researchers agree to distinguish aleatory uncertainty (AU) from epistemic uncertainty (EU): the former arises from the inherent randomness, e.g., data noise, and is irreducible. The latter is caused by a lack of knowledge about the process which generates the data, due to the limited availability of training data, and is reducible [1, 36] . Effective EU quantification is beneficial for out-of-distribution (OOD) detection [30, 54] and can contribute to a variety of safety-critical applications, including autonomous driving [21] , medical diagnosis [44] , flood uncertainty estimation [10] , structural health monitoring [72] .In classification, standard neural networks (SNNs) whose predictions amount to single probability distributions are unable to account for epistemic uncertainty, because a single distribution assumes precise knowledge about the dependency between inputs and outputs. To properly capture the EU, the network's outcome needs to express the uncertainty about a prediction's uncertainty itself [35, 62] .The most well-known approach to EU quantification in deep learning leverages Bayesian neural networks (BNNs) [7, 22, 38] . BNNs model network parameters as distributions and thus predict a 'second-order' distribution (i.e., a distribution of distributions) [36] , although in practice predictions are often generated by running the network on sample parameters extracted from a posterior. While efficient training techniques (such as sampling [33, 57] and variational inference [7, 22] ) have been developed to mitigate their complexity, practical challenges persist for BNNs, including the scaling to large datasets, handling complex network architectures, and real-time applicability [54] .An alternative approach, called Deep Ensembles (DEs), handles uncertainty quantification by aggregating multiple individually-trained SNNs [43] , so that predictions amount to finite sets of probability distributions over the classes. DEs, often viewed as an approximation of Bayesian model averaging (BMA) [38] , have become a powerful baseline for uncertainty estimation [2, 25, 53, 54, 60] . However, empirical evidence [2] suggests that DEs could yield relatively low-quality estimates of epistemic uncertainty. Further, DEs lack a sound theoretical justification [27, 47] .Credal inference [13, 36, 62] (which predicts convex sets of probability distributions or credal sets [46] on the target space) can provide an alternative way of quantifying epistemic uncertainty. Credal representations have been extensively studied within the broader field of machine learning, including, e.g., the naive credal classifier [14] , the notion of credal network [13] or credal random forest classification [65] . 'Imprecise' BNNs have been recently introduced which model both network weights and predictions as credal sets [8] . While these models demonstrate robustness in Bayesian sensitivity analysis, their computational complexity is on a par with that of ensembles of BNNs, significantly limiting their practical applicability.Novelty and Main Contributions. This paper presents an innovative approach to classification tasks called Credal Deep Ensembles (CreDEs), ensembles of novel Credal-Set Neural Networks (CreNets), aiming to improve EU quantification in the framework of credal inference. At the cost of merely doubling the number of output nodes compared to classical SNNs, CreNets are trained to predict a lower and an upper probability bound for each class rather than a single probability value. Such probability intervals over classes thus efficiently determine a prediction in the form of a credal set. The training strategy is inspired by Distributionally Robust Optimization [42, 55, 61] , which simulates the potential divergence of the test distribution from the training distribution. As a result, the width of the predicted probability interval reflects the 'epistemic' uncertainty about the future data distribution. Adopting an ensemble strategy, CreDEs derive the final prediction by averaging the probability intervals outputted by the members of the ensemble. A conceptual comparison between CreDEs and DEs is illustrated in Figure 1 .Extensive experimental validation is conducted on several OOD detection benchmarks, including CI-FAR10/100 (ID) vs SVHN/Tiny-ImageNet (OOD), CIFAR10 (ID) vs CIFAR10-C (OOD), ImageNet (ID) vs ImageNet-O (OOD), and across different network architectures: ResNet50, VGG16 and Visual Transformer Base (ViT Base). Compared to traditional Deep Ensembles, our CreDEs achieve higher test accuracy and lower expected calibration error (ECE) on ID samples, and significantly improve the quality of EU estimation.! ! ! " ! # SNN-1Target Space 𝕐 Target Space 𝕐 The former aggregate a collection of credal set predictions from CreNets as the final (credal) prediction, whereas the latter average a set of single probability distributions from standard SNNs as the outcome. E.g., in the probability simplex [16] associated with the target space Y = {A, B, D} (the triangle in the figure), a probability vector (q A , q B , q D ) is represented as a single point. For each CreNet, the predicted lower and upper probabilities of each class act as constraints (parallel lines) which determine a credal prediction (in gray). Single credal predictions are aggregated as in Sec. 2.4.Average AggregateBesides BNNs, DEs, and credal inference, other 'second-order' uncertainty estimation approaches exist, such as Dirichlet-based methods [9, 48, 49, 50, 63] , in which predictions are represented as Dirichlet distributions. One significant challenge for the latter is the absence of ground truth labels. Although various loss functions have been proposed, these models' performance often diverges from theoretical EU assumptions [71] . Another rationale for the exclusion of Dirichlet-based approaches as baselines for our CreDE work is that such models often necessitate the inclusion of OOD data during training [48, 49, 56] . This challenges their practical adaptability, as it cannot guarantee their robustness against other forms of 'unseen' OOD data [71] . Moreover, a recent study [39] has shown that these methods often fail to capture the EU properly, making the resulting measures difficult to interpret quantitatively.Paper Outline The remainder of this paper is structured as follows. Sec.  
credal-set neural networks: 
 Architecturally, our CreNet design focuses only on the final classification layers, and can therefore be applied on top of any representation layers of neural network models.The final layers of a CreNet (Figure 2 ) first output a deterministic interval for each class, using for each class an output node associated with the interval midpoint m and one associated with its half-length h, respectively (a total of 2C nodes for C classes).𝑚 ! 𝑚 " ℎ ! 𝑚 # ℎ " ℎ # 𝑎 $! 𝑎 $" 𝑎 %! 𝑎 $# 𝑎 %" 𝑎 %# 𝑞 $! 𝑞 $" 𝑞 %! 𝑞 $# 𝑞 %" 𝑞 %# + - + + - -Credal Set on Probability Simplex Let z be the input vector to the final layer. CreNets compute m and h (the vectors collecting interval midpoints and half-lengths for all classes) as:EQUATIONwhere The deterministic intervals associated with all classes, denoted as[a L , a U ] := {[a Li , a Ui ]} C i=1 , can then be obtained as [a L , a U ] = [m-h, m+h]. A proper mapping from such deterministic intervals [a L , a U ] to a collection of probability intervals [q L , q U ] := {[q Li , q Ui ]} Ci=1 for each class needs to ensure that [q L , q U ] satisfies the conditions:EQUATIONThe former condition guarantees a proper [q Li , q Ui ] for each class. The latter enables the resulting collection of probability intervals to determine a non-empty credal set, Q, as follows [52] :EQUATIONThe probability vectors in Q meet the normalization condition, and their probability value per class is constrained by the probability intervals (Eq. ( 2)).Traditional SoftMax activation cannot ensure that the convexity conditions in Eq. ( 2) are met when computing [q L , q U ] using q L = SoftMax(a L ) and q U = SoftMax(a U ), respectively. A toy example is given in Appendix §A. Therefore, we employ Interval SoftMax activation as proposed in [75] to compute [q L , q U ] from [a L , a U ], as follows:EQUATIONwhere q Li and q Ui are the lower and upper probability bound for the i th class, respectively. As proven in Appendix §A, the probability intervals generated by the Interval SoftMax duly satisfy Eq. (2).The rationale for the training of a CreNet is for the predicted lower and upper bounds (Eq. ( 4)) to the probability of the classes, q L and q U , to express the epistemic uncertainty (induced by the limited size and variability of the training set) about how different the distribution of future test data may be from that of the training data.To this extent, we designed a composite loss function with two components: one, which applies classical cross entropy to the upper probability vector, encourages the latter to optimistically assume that test data distribution will be similar. The other, inspired by Distributionally Robust Optimization (DRO) [42, 55, 61] , pushes the lower probability to reflect a 'pessimistic' stance on future distributional divergence. The width of the resulting interval will thus reflect the epistemic uncertainty associated with the prediction.We first contrast the classical training strategy with that of Distributionally Robust Optimization in Sec. 2.2.1. We then delve into the design and implementation of our CreNet loss in Sec. 2.2.2.Vanilla Strategy Given a training set D = {x n , t n } N n=1 , the conventional neural network training process aims to solve the following optimization problemEQUATIONwhere θ denotes the model's trainable parameters in the space Θ and L denotes an arbitrary loss function. The underlying assumption is that the training and test distributions are identical. As a result, the trained network serves as an empirical risk minimizer [36] . However, this ideal assumption often results in over-optimistic predictions because the test observations may, in practice, significantly differ from the training data [34] .In contrast to the vanilla strategy, the objective of DRO [6, 20] is to minimize the worst-case expected risk (R(θ)) over an uncertain set of distributions U, as follows:EQUATIONin which E is the expectation operation. In practice, a group DRO setting [59] is adopted in which the training distribution P is assumed to be a mixture of m groups P g , indexed by g ∈ G = {1, 2, ..., m}.Because the optimum of a linear program is attained at a vertex, the worst-case risk in Eq. ( 6) is equivalent to a maximum over the expected loss of each group, as follows:EQUATIONIn practice, the group DRO model minimizes the empirical worst-group risk R(θ), namely:EQUATION)where Pg is the empirical distribution of the g-th group of training points. Therefore, group DRO learns models with good worst-group training loss across groups [59] . One special form of group DRO is adversarially reweighted learning [42] , which consists of a minimax game between a learner and adversary. The learner optimizes for the main classification task and aims to learn the best parameters θ that minimize the expected loss. In contrast, the adversary maximizes the expected loss by making an adversarial assignment of weights w n , collected in a vector w. Consequently, the training optimization proplem assumes the formEQUATIONwhere the set S of weight vectors varies across different implementations [42, 55, 61] .Design As anticipated, the CreNet training process applies the vanilla training strategy to the upper probability prediction vector q U (Eq. ( 5)), and the DRO strategy to the lower probability prediction q L (Eq. ( 9)). The resulting overall loss function has a composite structureEQUATIONwhere CE denotes the classical cross-entropy loss function used in classification. Given a predicted discrete probability vector q and the ground-truth label t, CE is defined as: CE := -C k t k • log 2 q k .The vanilla component is applied to the upper probability vector q U because such a loss takes the training distribution at face value and is thus more likely to encourage 'optimistic' (overconfident) or 'upper bound' predictions for the class scores. The DRO component is computed on the lower probability vectors q L , as it weighs training outliers to simulate future differences in data distribution at test time, encouraging 'pessimistic' or 'lower bound' class score predictions. Thus, the width of the resulting probability interval will reflect the uncertainty associated with the model's ignorance of how much the future test distribution will differ from the train distribution, using the boundary/outlier cases observed at training time to guess what the uncertainty on future test cases will be.Cross-Entropy of Lower/Upper Probability Vectors Please note that in Eq. ( 10), the CE is applied to lower/upper probability vectors, which are not (normalized) probability vectors. However, as the ground truth (label) vector t equals 1 for the true class j * and 0 for all the other elements, calculating CE(q, t) for any predicted probability vector q reduces to -log 2 q(j * ). Consequently, all probability vectors with the same component for the true class will generate the same CE for that sample.The consequence for CreNet training is that feeding a lower (upper) probability vector q L (q U ) to Eq. ( 10) is equivalent to computing the CE with any one of the probability vectors in the credal prediction (Eq. ( 3)) whose probability for the true class j * equals the lower (upper) probability value there. It can be shown that these form one of the 'faces' of the boundary of the credal set.Importantly, because of the functional structure of Interval SoftMax activation (Eq. ( 4)), upper and lower probability vectors are not computed independently, but are correlated. Thus, they are minimized together via the total loss (Eq. ( 10)), with the DRO component also influencing the upper probability q U , driving the solution away from the trivial one (all-ones upper probability vectors).Implementation As the S of weight vectors in Eq. ( 10) varies across different implementations [42, 55, 61] and estimating w in Eq. ( 10) is not straightforward when using batch-wise training [34] , we resort to a simpler heuristic proposed by [34] . For each training batch, only the δ ∈ [0.5, 1) portion of samples with the highest cross-entropy with the lower probability vector (CE(q Ln , t n )) are selected to compute the DRO component of the loss. As a result, w n > 1 is implicitly set in Eq. (10) for selected samples while w n = 0 for deselected samples.The underlying rationale is the following. Within a batch of samples, those instances that demonstrate high losses are identified as 'hard-to-learn' samples, essentially representing the 'minority group' within a training dataset [34] . Setting a value δ thus identifies what fraction of the training points is chosen to represent potential future domain shifts at test time. A smaller δ signifies a more cautious approach, in which even a few training outliers can indicate future challenges.The lower bound to the design range for δ is 0.5 because we empirically observed that values of δ < 0.5 may destabilize the training process, as a too-large averaged loss is returned for backpropagation. When δ approaches 1, the data distribution of the samples considered by the vanilla and the DRO components of the loss becomes similar, implicitly assuming a less pronounced divergence between train and test distributions. The corresponding predicted probability intervals become narrower. If δ were theoretically set to 1, all samples would be selected for backpropagation, implying that w n = 1 for any n in Eq. (10) . Consequently, the loss in Eq. ( 10) would be the sum of the vanilla component on q U and the vanilla component on q L . Empirically, we observed that this leads to a collapse of the upper and lower probability bounds to single values. , t n ) 3. Define η δ = ⌊δη⌋ 4. Minimize L CreNet = 1 η η n=1 CE(q Un , t n )+ 1 η δ η δj=1 CE(q Lm j , t mj ) end whileClass Prediction For the class prediction we employ the 'maximin' and 'maximax' criteria [66] :EQUATIONwhich output (respectively) the class indices with the highest lower and upper reachable probability (q * Li and q * Ui ) within the same credal set induced by the predicted lower and upper probabilities q Li , q Ui . Figure 3 illustrates how the lower and upper probabilities q Li , q Ui that determine the credal set Q may differ from the probabilities q * Li and q * Ui actually reachable for each class within Q. The reachable lower and upper probabilities for class i can be easily obtained as follows [17] :EQUATIONUncertainty Quantification Given a credal set prediction, upper and lower entropies generalizing Shannon's entropy, denoted as H(Q) and H(Q), can be defined which may serve as measures for TU and AU, respectively [3, 36] .Computing H(Q) boils down to solving the following optimization problem:EQUATIONThis seeks the highest entropy value of a probability distribution within the predicted credal set Q. H(Q), for which maximize is replaced by minimize, searches for the minimal such entropy. Such optimization problems can be addressed using a standard solver, e.g., the SciPy optimization package [73] . Epistemic uncertainty can then be quantified as H(Q)-H(Q) [36] .Computational Complexity Reduction To reduce the computational complexity of Eq. ( 13) for a large value of C (e.g., C = 1000), we propose an original approach called Probability Interval Dimension Reduction (PIDR) in Algorithm 2. This method first identifies the K -1 classes with the highest lower probability values, then merges the remaining elements into a single class with the associated upper and lower probability calculated using Eq. ( 12). Consequently, the dimension of the probability interval is reduced from C to K.Inspired by conventional DEs [43] , the final step of our approach is to introduce Credal Deep Ensembles (CreDEs). CreDEs aggregate M individually trained CreNets and predict the aggregated probability intervals, denoted as [ q * L , q * U ], as follows:EQUATIONwhere [q * Lm , q * Um ] is the set of reachable probability intervals predicted by the m-th CreNet. Eq. ( 20) in Appendix D proves that [ q * L , q * U ] satisfies the convexity condition in Eq. ( 2) for constructing aInput: [q * L , q * U ]; Chosen number of classes K Output: Reduced-dimensional probability intervals [r L , r U ] 1. Index vector of q * L in descending order: l ← argsort(q * L ) 2. Define the upper and lower probability per selected class:r Lj ← q * L l j , r Uj ← q * U l jfor j = 1, ..., K -1 3. Define upper and lower probability for deselected classes:r L K ← max(1- l C i=l K q * Ui , K-1 j=1 r Lj ); r U K ← min(1- l C i=l K q * Li , K-1 j=1 r Uj ) 𝑞 !! 𝑞 !! * Figure 3:If intervals are redundant, some of the (e.g.) upper probabilities q U A may not be actually reachable in the credal set that results from the intersection of all interval constraints. non-empty credal set. Therefore, class prediction and uncertainty estimation as described in Sec.2.3 apply to CreDEs. We discuss the rationale for averaging strategy and the alternative ensemble approaches for CreDEs in Appendix §D. 
experimental validation: 
 Setup We assessed CreDEs through OOD detection benchmarks across various dataset pairings (ID vs OOD samples), including CIFAR10 [41] /CIFAR100 [40] vs SVHN [31] /Tiny-ImageNet [45] , CIFAR10 vs CIFAR10-C [29] , ImageNet [18] vs ImageNet-O [31] . We trained 15 CreNets (using δ = 0.5) and SNNs on the ResNet50 architecture [28] starting from different random seeds, using the training set as per the ID dataset in the pair. Following this, we constructed 15 different CreDEs and DEs, respectively, by randomly selecting five members from the pool of 15 trained models. The same ensemble member lists are used for both DEs and CreDEs, with each ensemble strictly guaranteed to be distinct. More details are given in Appendix §C. Codes are available at https://gitlab.kuleuven.be/m-group-campus-brugge/distrinet_public/ credal-deep-ensembles.git.Uncertainty Quantification in DEs Total uncertainty (TU) can be quantified in DEs via the Shannon entropy (H) of the averaged predicted distribution. The AU, on the other hand, can be obtained by averaging the entropies of the predictions of each ensemble member [2, 36] . Namely,EQUATION)where M is the number of networks, q and q m denote the average probability vector and the single probability vector of the m-th SNN model, respectively. The level of epistemic uncertainty, representing an approximation of mutual information [36] , can be obtained as EU := H( q)-H(q).We evaluated the test accuracy and expected calibration error (ECE) [24, 58] of CreDEs-5 and DEs-5 on the test set of each ID dataset. A lower ECE value signifies a closer alignment between the model's confidence scores and the true probabilities of the events. Since ECE is designed for a singular probability vector, we implemented a compromise calculation as follows. Suppose our model predicts the class indices k and j when using the îmin and îmax criteria, respectively, ECE values are then computed based on the associated lower q * L k and upper q * Uj reachable probabilities in the credal set. Table 1 reports the test accuracy and ECE for DEs-5 and CreDEs-5 on the various datasets, indicating that our CreDEs-5 achieved higher test accuracy and lower ECE on ID samples. Note that employing the îmin prediction showed higher ECE on the challenging ImageNet dataset. This is likely because the strategy, selecting the class with the highest lower reachable probability, is a conservative one. EU Quantification for OOD Detection It is our hypothesis that OOD data express a higher EU. Hence, we can use a better EU quantification as the means to improve the OOD detection [54] . Thus, superior OOD detection performance provides compelling evidence of enhanced uncertainty estimation quality. For the OOD detection performance assessment, we 4 presents averaged AUROC and AUPRC scores for OOD detection on CIFAR10 vs CIFAR10-C across types of corruption, against the intensity of corruption. Table 2 and Figure 4 confirm CreDEs-5's superior OOD detection performance over DEs-5. This indicates the effectiveness of CreDEs in improving the EU quantification quality, using H(Q)-H(Q) as the uncertainty measures.Qualitative Evaluation Due to the high dimensionality, visualizing or directly computing the size of the credal set becomes challenging as C increases. Consequently, we indirectly evaluate whether our CreDEs consistently generate nearly Dirac credal sets as predictions through the maximum attainable upper bound probability of the prediction. The closer this probability is to 1, the more it approximates a Dirac credal set. Figure 5 shows the results of ResNet50-based CreDEs-5 for the CIFAR10, SVHN, and Tiny-ImageNet datasets. It verifies that our method does not consistently generate nearly Dirac credal sets, especially for OOD samples. For CIFAR10, a substantial proportion of (but not all) the credal sets are quasi-Dirac. This observation is reasonable as it is consistent with the high test accuracy of CreDEs and the low ECE reported in Table 1 .CIFAR10 SVHN Tiny-ImageNetFigure 5 : Maximum reachable upper probability max (q * U1 , ..., q * U C ) per sample from 15 runs. Additionally, Figure 6 shows the reliability diagram [24] of the ResNet50-based DEs-5 and CreDEs-5 on the CIFAR10 dataset, demonstrating better calibration performance of our CreDEs. Figure 7 showcases the EU estimation plots for these models. Although the EU estimates for DEs-5 and CreDEs-5 are not directly comparable due to differing representations, CreDEs-5 demonstrates significantly higher EU estimates for OOD samples, as observed qualitatively.We also performed an ablation study on network backbones different from ResNet50, including VGG16 [67] and Vision Transformer Base (ViT Base) 3 reports the test accuracy and ECE of CreDEs-5 and DEs-5 on the CIFAR10 test set (representing ID samples) and the OOD detection performance on CIFAR10 vs SVHN/Tiny-ImageNet. Figure 8 compares OOD detection performance in the CIFAR10 vs CIFAR10-C setting against the intensity of corruption, using both AUPRC and AUROC as metrics. The results consistently demonstrate that CreDEs achieve higher test accuracy, lower ECE, and significantly improved epistemic uncertainty estimation, leading to enhanced OOD detection performance. ablation findings verify the robustness of Cre-DEs across hyperparameter setups and indicate the δ = 0.5 might be too pessimistic a choice in CIFAR10 settings. Performance peaks at δ = 0.875 in most cases, implying that δ = 0.875 may provide the 'optimal' estimate of how test and train sets diverge for CIFAR10. One possible way to find the 'best' δ in practice is to conduct standard cross-validation on specific test scenarios. However, the method is not particularly sensitive to this hyperparameter. Perspectively, an interesting option, in the presence of multiple datasets (e.g.acquired over time in a continual learning setting), could be applying the DRO loss component to different components of the training set, and assessing the results to robustly select δ.We also report the average EU estimation values of CreDEs-5 for each dataset in Table 5 . Increasing the value of δ (i.e., giving less importance to the divergence between test and training distributions) leads to a decreasing trend in the average EU estimates per dataset (particularly for ID CIFAR10 samples). This aligns with the intuition that, if the model is more uncertain about the divergence of the distributions (smaller δ), it should express a larger EU. Despite smaller uncertainty values at high δ's, the difference between ID and OOD samples remains noticeable. This explains why a δ closer to 1 does not always lead to low-performance OOD detection and why our model's OOD detection performance is robust against the choice of δ.Model Inference Complexity Table 6 reports the parameter count and inference cost on one NVIDIA A100-SXM4-40GB GPU for a single SNN and CreNet on ImageNet. CreNets show a marginal increase in complexity due to its minor architectural modifications. More discussions on the inference and training complexity are presented in Appendix §C.Additional Experiments Appendix §B.1 discusses the implementation and performance of EU quantification in CreDEs when using a different uncertainty measure for credal sets (the generalized Hartley measure [4, 36] ). The results demonstrate that our CreDEs consistently enhance the quality of EU quantification, exhibiting robustness against different measures. Appendix §B.2 reports an ablation study for the hyperparameter K of our PIDR Algorithm 2, which shows the effect of K on CreDEs's uncertainty quantification and time cost. Appendix §B.3 assesses the ability of CreDEs to evaluate total uncertainty (TU) (as opposed to EU) in OOD detection, suggesting that our CreDEs also achieve an improved TU estimation, compared to DEs. Appendices §B.4, §B.5, and §B.6 compare the uncertainty quantification abilities of CreDEs versus those of traditional DEs that also apply the DRO strategy, DEs that apply the 'product of experts' [32] ensemble setting, several BNN baselines, respectively. CreDEs continue to demonstrate superior performance in uncertainty estimation. Appendix §B.7 assess CreDEs in a case study involving active learning [23, 54] . All these additional experiments demonstrate that our CreDEs deliver improved uncertainty quantification. 
conclusion: 
 Conclusion In this paper, we introduced a novel Credal-Set Neural Network (CreNet) for classification tasks. Given any given input instance, CreNet is designed to predict a lower and an upper probability for each class, rather than a single probability value, thus providing an efficient and effective implementation of credal inference. We also proposed Credal Deep Ensembles (CreDEs), ensembles of CreNets, which extend the traditional deep ensemble idea to the credal domain. Extensive experimental validation was conducted on several OOD detection benchmarks, and across different network architectures and uncertainty measures. Compared to traditional Deep Ensembles, our CreDEs achieve higher test accuracy and lower ECE on ID samples, while significantly improving the quality of EU and TU estimation, leading in turn to strongly enhanced OOD detection performance. Hence, we believe our work can potentially improve neural network safety and reliability, and have wide applicability to real-world scenarios such as medical image analysis.Limitation Despite the superior performance of CreDEs, neither traditional DEs nor CreDEs may be desirable when memory usage is stringent and computational resources are limited.Future Work Three essential objectives of our future research include elaborating on statistical coverage guarantees of our CreDEs (outlined in Appendix §E.2), extending our framework to regression tasks (a roadmap is provided in Appendix §E.3), and assessing our CreDEs alongside other uncertainty-aware models in real-world applications comprehensively, like medical image analysis. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide all information about the computational resources needed in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts of our work in the Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper focuses on uncertainty quantification in classification tasks and is evaluated on existing benchmarks. The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We use open-source datasets with references for the experimental evaluation.We have cited all the datasets and models used in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Toy Problem for Unavailability of Traditional SoftMax The traditional SoftMax activation function cannot be used to define the credal set, as it cannot ensure the condition in Eq. ( 2) when computing [q L , q U ] as q L = SoftMax(a L ) and q U = SoftMax(a U ), respectively.For example, assuming that we have a L := (-1, 0, 1) and a U := (0, 1, 3) from CreNet, the q L and q U computed using SoftMax are: q L = SoftMax(a L ) = (0.0900, 0.2447, 0.6653), q U = SoftMax(a U ) = (0.0420, 0.1142, 0.8438).The resulting 'probability intervals' are not properly defined and appear unreasonable, as some lower bounds are considerably higher than the upper bounds.Mathematical Proof for Interval SoftMax The proof that Interval SoftMax in Eq. ( 4) does satisfy the conditions in Eq. ( 2) is straightforward:C i=1 q Li = C i=1 exp(a Li ) C k̸ =i exp( a U k +a L k 2 ) + exp(a Li ) ≤ C i=1 exp( a U i +a L i 2 ) C k̸ =i exp( a U k +a L k 2) + exp(EQUATIONIn this section, Appendix §B.1 discusses the implementation and performance of EU quantification of CreDEs using another uncertainty measure for credal sets: the generalized Hartley measure. Appendix §B.2 performs an ablation study on the value of the hyperparameter K of the PIDR Algorithm 2. Appendix §B.3 assessed CreDEs' performance in the OOD detection task when quantifying uncertainty using TU instead of EU. Appendices §B.4, §B.5, and §B.6 compare CreDEs versus traditional DEs that also apply the DRO strategy, DEs that apply the 'product of experts' ensemble setting [32] , and several BNN baselines, respectively. Appendix §B.7 assessed CreDEs in an active learning case study.Uncertainty quantification in credal sets merits further investigation. For instance, recent research [35] has explored, e.g., the use of probability interval length as a measure of epistemic uncertainty, in the special case of binary classification. However, these measures cannot be readily extended to multi-class cases. Recently, the most established methods for decomposing the total uncertainty of credal sets are generalized Entropy [3, 36] and the generalized Hartley Measure [4, 36] .Definition The generalized Hartley measure [4] , GH(Q), measures the non-specificity across the distributions in the credal set, and can be seen as a proxy for its volume [36] . Mathematically, GH(Q) calculates the expectation of the Hartley measure [26] over all possible subsets B on the target space Y,2 as follows [4] :EQUATIONin which m Q denotes the mass assignment function associated to Q and |B| indicates the cardinality of B. m Q (B) can be computed using the Möbius inverse of the capacity function ν Q [36] , as follows:EQUATIONwhere B\A = {y|y ∈ B and y / ∈ A} and ν Q describes the lower probability of all possible subsets A ⊆ B. Efficient Implementation in CreDEs One of the reasons that hinder the application of the Generalized Hartley Measure is its computational complexity. In our work, we proposed an efficient implementation and an approximate approach (the PIDR Algorithm 2) for computing it.In our case, the lower probability ν Q (A) associated with the predicted credal set can be readily computed as follows:EQUATIONwhere q * L and q * U are the reachable lower and upper probability values per class in the defined credal set. They can be easily obtained from Eq. (12) . Figure 3 illustrates how the lower and upper probabilities q Li , q Ui that determine the credal set Q may differ from the probabilities q * Li and q * Ui actually reachable for each class within Q.The full GH(Q) calculation process is presented in Algorithm 3.Input: 17)) end for Although the use of probability intervals simplifies the calculation of GH(Q) in general, a significant challenge arises for large values of C (e.g. C = 100) due to the complexity of involving subsets of C.EQUATION)) end for GH(Q) = GH(Q)+m Q •log 2 (|B|) (Eq. (However, when applying our proposed PIDR Algorithm 2, the dimension of the probability interval is reduced from C to K; therefore, calculating GH(Q) requires only 2 K subsets.The OOD detection results for CreDEs-5 using GH(Q) are shown in Tables 7, 8 and in Figure 9 . Probability Interval Dimension Reduction (PIDR) (Algorithm 2) is utilized with settings K = 4 and K = 10 when computing GH(Q) for dataset pairs containing CIFAR10 and CIFAR100/ImageNet, respectively. The results verify that • Our CreDEs consistently enhance the quality of EU quantification, exhibiting robustness across different uncertainty measures, i.e., the generalized Shannon entropy and the generalized Hartley measure. This improved EU quantification leads to better OOD detection performance compared to Deep Ensemble baselines.• The proposed PIDR algorithm ensures an efficient implementation of the generalized Hartley measure in our framework. An ablation study on PIDR's hyperparameter is conducted in Appendix §B.2.Note, however, that applying K = 10 for the setting ImageNet vs ImageNet-O does not yield a better result, due to the coarseness of approximating 1000 classes using only 10. This suggests that computing GH(Q) is still challenging for tasks involving 1000 or more classes.Effect on GH(Q) Quantification Figure 10 illustrates the influence of various settings of K on GH(Q) quantification. The average value GH(Q) suggests that the use of the PIDR Algorithm 2) results in an underestimated GH(Q) value, compared to the result without using PIDR (K = 10). Consequently, increasing the value of K enhances OOD detection performance. However, as K grows execution time increases exponentially, due to the iterative calculations of m Q (B) and ν Q (A) in Algorithm 3 across 2 K subsets. The time cost is measured on a single Intel Xeon Gold 8358 CPU@2.6 GHz. While higher than the time cost for EU calculation of DEs (4.1e -4 ms), this figure shows that calculating GH for 10 classes (K = 10 for 17 ms) remains practical without actual computational constraints. Besides, the numbers reported are for GH calculation without any optimization: a more Effect on TU Quantification In this experiment, we examine the effect of K on TU estimation. Figure 11 shows the average values of TU estimates (H(Q)) per sample, together with the AUROC and AUPRC scores for CIFAR100 vs. SVHN/Tiny-ImageNet. The results indicate that applying PIDR (Algorithm 2) tends to underestimate TU values. Consequently, increasing the value of K improves the OOD detection performance, but it also leads to an increase in execution time. This is because solving the constrained optimization problem in Eq. ( 13) involves more variables and constraints.The reported time cost is measured on a single Intel Xeon Gold 8358 CPU@2.6 GHz, without optimization in the calculation process. We believe a more efficient code implementation could significantly mitigate this. In further additional experiments, we also assess the quality of the total uncertainty (TU) estimates produced by CreDEs-5 on the various OOD detection benchmarks [43, 54] . The results in Tables 9 and 10 consistently demonstrate CreDEs' improved OOD detection performance using TU as a metric.In this experiment, we additionally train 15 ResNet50-based SNNs on the CIFAR10 and CIFAR100 datasets using the DRO loss component and the same training strategy as in Algorithm 1, respectively. For a fair comparison, we set δ = 0.5 (just as for CreNets), using the same random seeds and training epochs. Other training configurations are described in Appendix §C. We named the resulting deep ensembles with 5 ensemble members as DEs*-5. We compare test accuracy and ECE for DEs*-5 and CreDEs-5 in Table 11 , and their OOD detection performance on the CIFAR10/100 (ID) vs SVHN/Tiny-ImageNet (OOD) benchmark in Table 12 . The reported results demonstrate that CreDEs-5 outperforms DEs*-5 ensembles by achieving higher test accuracy and lower ECE values. Concerning OOD detection tasks, it can be found that CreDEs in general improve the AUPRC and AUROC scores using either the TU or the EU metric, pretty much across the board. These results suggest that CreDEs provide higher-quality EU and TU estimation.In Table 12 , a 0.1% drop in AUPRC using the TU metric can be observed. However, remember that CreDEs calculate TU (using the upper entropy) by solving a constrained optimization problem in Eq. ( 13) using a numerical solver from SciPy. The slight performance decrease is likely due to numerical errors during the optimization process.In this experiment, we conduct a comparison between our CreDEs and DEs with the 'product of experts' (POE) [32] ensemble setting, as opposed to the more commonly employed 'mixture of experts' approach in our primary analysis. Here, DEs p -5 denotes the deep ensembles that process the final predictions from the ensemble members using the POE strategy. The experimental setup mirrors that used by OOD detection benchmarks, involving data pairs CIFAR10 (ID) vs SVHN/Tiny-ImageNet (OOD).Table 13 shows that DEs p -5 could improve test accuracy but significantly reduce the calibration performance of DEs-5 (larger ECE values). Among these comparisons, CreDEs-5 emerged as the most superior method. Furthermore, we evaluate the uncertainty estimation through the OOD detection benchmark. Specifically, =the entropy of the final prediction of DEs p -5 is calculated to quantify the total uncertainty. For CreDEs-5 and DEs-5, we use the upper entropy, H(Q), and H( q), respectively. The results in Table 14 consistently demonstrate the superior performance of our method.Although the POE strategy improves the test accuracy of classical DEs, it significantly degrades calibration performance and leads to inferior OOD detection performance. As discussed in the main body, the main reason for excluding Bayesian neural network (BNN) approaches in our main evaluation is that they generally have difficulty scaling to large datasets and complex network architectures [54] . In this section, we conducted an additional comparison between CreDEs and DEs, MCDropout [22] , and two TensorFlow-standardized BNNs (BNN-R [51] and BNN-F [76] ). All the models are trained on the ResNet50 for the CIFAR10 dataset from scratch. The input data shape is (32, 32, 3) . The Adam optimizer is applied with a learning rate scheduler, initialized at 0.001. The learning rate is subject to a reduction of 0.1 at epochs 80 and 120. For BNNs, 10 forward passes are used for uncertainty estimation.The uncertainty evaluation via OOD detection on the CIFAR10 vs SVHN/Tiny-ImageNet dataset is reported in Table 15 . The results consistently demonstrate the significant improvements of our CreDEs.Active learning (AL) aims to efficiently train models with minimal data by acquiring additional samples from a vast pool of unlabeled data, which are then labeled by experts [11] . After each acquisition step, the model is retrained using the expanded training set. The iterative process continues until either the desired accuracy or the maximum allowable acquired samples are reached. Efficient data acquisition can be a reliable estimate of the uncertainty of models [23, 54] .Setup We deploy CreDEs-5 (δ = 0.5) and DEs-5 (baseline) using the ResNet18 architecture and utilizing clean MNIST samples in the pool set. TU and EU estimations from each approach for the acquisition functions are utilized. We begin with an initial training set of 20 randomly selected MNIST points. In each iteration, we acquire the 5 samples with the highest reported uncertainty estimates (EU or TU per model). After each step, we train models using the Adam optimizer for 20 epochs and select the one with the best accuracy from the validation set. AL process stops when the training set size reaches 150.Results Figure 12 shows the result comparison between CreDEs-5 and DEs-5 using TU and EU estimates as the acquisition functions per model. In the evaluation using MNIST, aiming for a 90% accuracy or a maximum sample count of 150, CreDEs-5 employing acquisition functions TU (H(Q)) and EU (GH(Q)), demonstrates superior performance compared to DEs-5 using TU (H( q)). In addition, CreDEs-5 with EU (H(Q) -H(Q)) outperforms DEs-5 with EU (H( q)-H(q)). The additional evidence verifies the improved quality of EU and TU estimation of CreDEs, compared to DEs. In future work, we aim to explore the potential integration of our methods into other active learning benchmarks [70] and real-world applications or further improve on them.For the main experiments on the ResNet50 backbone, we used two Tesla P100-SXM2-16GB GPUs as devices to independently train 15 SNNs and CreNets using CIFAR10 and CIFAR100 datasets. The input shape of both networks was (224, 224, 3). We employed the Adam optimizer, with a learning rate scheduler set at 0.001 and reduced to 0.0001 during the last five training epochs.Figure 13 shows the averaged training and validation accuracy for training process monitoring.In the ImageNet experiments, we employed three NVIDIA A100-SXM4-80GB GPUs. To create deep ensembles, we independently retrained 15 deep SNNs based on a pre-trained ResNet50 model for 3 epochs, using the Adam optimizer with an initialized learning rate of 1e -6 . For CreDEs, we initialized CreNet weights using a pre-trained ResNet50 model and independently retrained 15 CreNet models for 5 epochs, using the Adam optimizer with an initialized learning rate of 1e -5 . The choice of a larger learning rate value and epoch count for CreNets is a consequence of their modified final later compared to SNNs. For the ablation study on various network architectures, we again utilized two Tesla P100-SXM2-16GB GPUs and one NVIDIA A100-SXM4-80GB GPU as devices to independently train 15 SNNs and CreNets, based on VGG16 and ViT Base architectures, respectively, and using the CIFAR10 dataset. VGG16-based SNNs and CreNets were trained for 20 epochs. SNNs and CreNets using the ViT Base backbone were trained for 25 and 40 epochs, respectively. The input Regarding the uncertainty estimation cost, we report the cost of calculating the Generalized Hartley (GH) measure and the upper entropy in Figures 10 and 11, respectively. For example, the time cost for GH calculation for CIFAR10 without approximation is 17 ms (0.02 ms in the reduced case considering 4 out of 10 classes) while calculating the EU in deep ensembles for CIFAR10 takes 1×10 -4 ms, measured on the same single CPU. Though higher CreDEs remain practical without actual computational constraints. In addition, the reported numbers are without code efficiency optimization: a more efficient code implementation could significantly reduce the cost.The practical takeaway here is that, as demonstrated by extensive experimental variation, our CreDEs exhibit strong potential to enhance the uncertainty quantification performance of DEs in real-world applications, with only a modest increase in computational complexity. However, if DEs are already deemed impractical due to computational limitations, our CreDEs would not be a suitable alternative.The randomness of parameter initialization in neural networks is one of the reasons leading to (epistemic) uncertainty about the 'ground-truth' model. As we gather more information, both epistemic and total uncertainty should decrease. For example, if we assume that we can train an infinite number of standard neural networks, then the Deep Ensembles would eliminate the source of ignorance caused by the randomness of parameter initialization.Our proposed averaging approach to creating an ensemble of CreNets follows a similar rationale. Specifically, if we aggregate an infinite number of ensemble members, the uncertainty caused by the randomness would vanish. The outputted probability interval of CreDEs, primarily acknowledges the lack of precise insights into the divergence between the training and test distributions.CreDEs aggregate predictions from multiple individually trained CreNets, producing credal sets based on probability intervals. In addition to averaging, two alternative approaches, namely union (disjunctive combination) and intersection (conjunctive combination) [17] , can be envisaged.These alternative methods are illustrated in Figure 14 . Averaging CreDEs average the upper and lower probabilities per class from M individually-trained CreNets and predict the aggregated probability intervals, denoted as [ q * L , q * U ], as discussed in Eq. ( 14). It can be proved that [ q * L , q * U ] is guaranteed to generate a non-empty credal set, as follows:EQUATIONThe semantic behind averaging is that we equally trust all pieces of information (individual credal sets) without judging the authenticity of the information. Similar to traditional deep ensembles (DEs), the averaging ensemble approach can alleviate the influence of training process randomness.Union Given a collection of convex probability intervals, denoted as {[q * Lm , q * Um ]} M m=1 , De Campos et al. have proposed the computationally efficient way to calculating the union of credal sets [17] , as follows:EQUATIONThe union ensemble method implies that at least one piece of information is considered to be true. The union operation in Eq. ( 21) has a significant limitation, as it results in an expanded credal set and introduces an overestimation effect on the precise union of credal sets, as shown in Figure 14 .Intersection A collection of convex probability intervals {[q * Lm , q * Um ]} M m=1 can formulate an intersection asEQUATIONHowever, the obtained [ q * L , q * U ] does not inherently satisfy the condition outlined in Eq. ( 2) for constructing a credal set [17] . Therefore, the intersection approach is not applicable in CreDEs.In this experiment, we mainly evaluate the impact of averaging and union ensemble approaches on the EU estimation (GH(Q)) of CreDEs. Utilizing 15 individually trained ResNet50-based CreNets on CIFAR10 dataset, we formulate 15 CreDEs-M by varying the ensemble number M from 2 to 10 through averaging and union ensemble methodologies. Each kind of CreDEs-M is assessed for the averaged GH(Q) concerning samples and the quantity of CreDEs-M, and the averaged standard deviation (STD) of GH(Q) related to samples and the quantity of CreDEs-M. The results are plotted in Figure 15 (b) and (a), respectively. Besides, we also present the AUPRC and AUROC scores of OOD detection using GH(Q) as the uncertainty metric in Figure 15 (c) and (d) , accordingly. While the average EU estimates for ID samples are overall lower than those for OOD instances, the overestimation may lead to OOD mis-detection in some specific samples. This explains the "fluctuations" in the associated AUPRC and AUROC score curves in Figure 15 .E Further Discussion on Future work t(j) log t(j) q(j) between a predicted probability vector q and the general ground truth vector t, to lower/upper probabilities is still an open research subject [68, 69] .In our case, the credal set Q is defined by the outputted proper probability intervals [q L , q U ]. Therefore, the KL divergence for a lower probability inducing a credal set may be calculated by:• Finding the probability vector that best approximates it. For probability intervals, there are two established such ways: normalizing either the lower or the upper probability (see [74] ) or computing the so-called 'intersection probability' (see [15] ). • Computing the KL divergence between the ground truth vector and the approximation obtained.In future work, we aim to investigate the approach and compare those other well-founded methods for calculating the cross-entropy loss with the one used in the paper.In the current stage, our CreNets do not provide coverage guarantees, e.g., on how likely it is for the divergence of future data distributions to be within the modeled bounds. Nevertheless, various approaches to incorporating statistical guarantees in our framework can be envisaged.In particular, a CreDE, being a classifier, can be employed as the 'underlying model' in an inductive conformal learning framework [64] , which builds an empirical cumulative distribution of the 'nonconformity' scores of a set of calibration samples and at test time outputs the set of labels whose empirical CDF is above a desired significance level ϵ (e.g., 90%).Namely, given a test input x and the associated predictive system of probability intervals [q Lc , q Uc ], c = 1, ..., C (the output of CreDE), a sensible choice, for instance, is to set as nonconformity score of a pair (test input, class), (x, c), the complement of the upper probability for that class, given input x: s(x, c) . = 1 -q Uc (i.e., a label c would be considered 'non-conformal' if its predicted upper probability, for that input x, is low), and compute predictive regions as standard in conformal learning:Γ(x) = c ∈ C : p c > ϵ,wherep c = |(x j , c j ) : s(x j , c j ) > s(x, c)| q + 1 + u • |(x j , c j ) : s(x j , c j ) = s(x, c)| q + 1 , (x j , c j )is the j-th calibration point, q is the number of calibration points, and u ∼ U(0, 1) (the uniform distribution on the interval (0, 1)).We plan to explore this integration as the next step of our future work.The vast majority of papers using credal sets in machine learning focus on classification [12, 5, 78] , or, more recently, on self-supervised learning (but still in a classification setting [37] ). Nevertheless, a recent study [19] has shown that the formalism of belief functions (a special class of credal sets) can be extended to regression, by leveraging random fuzzy sets. It might thus be possible to explore such connections between probability intervals and random fuzzy sets and devise a suitable regression framework based on CreDEs.The following section outlines a more direct possible extension of CreDEs to regression problems as a future research direction.Remember that a CreNet outputs a credal set on the simplex of probability distributions over the classes. Each vertex of this credal set is therefore a probability distribution over the target space (the set of classes C for classification).On the other hand, a Bayesian regressor network (trained to learn a distribution of its weights) would output a (continuous) probability density over the target space (for the sake of simplicity, assume Y = R).One could then train an ensemble of Bayesian regressor networks to predict a credal set with a fixed number of vertices (one network outputting one vertex probability) so that the final predicted credal set is the convex closure of those. Figure 16 illustrates the concept briefly. The main objective of this paper is to advance the field of Machine Learning by improving the quality of uncertainty quantification. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. The proposed method demonstrates superior performance in detecting out-of-distribution (OOD) samples. Such capability can potentially safeguard end users from misguided decisions that stem from the incorrect predictions of neural networks on OOD instances. Therefore, our approach can potentially improve the safety, reliability, and trustworthiness of machine learning systems for classification tasks and be applied in missioncritical domains, such as autonomous driving and medical sciences.• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Justification: The research conforms with the Code Of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).In classification, target space Y comprises a finite set of class labels, namely Y = {y1, ..., yC }. 


Decision: Dear reviewers, thank you for all the hard work you put in your reviews and in engaging with the authors and their responses.
To the Authors, thank you for giving such a detailed rebuttal.

After reading all of the reviews and comments, I am very happy to recommend this paper for acceptance. Especially after the authors in depth rebuttal.  

Best,

    AC

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose a novel method for uncertainty estimation in deep networks called Credal Deep Ensembles, which combines credal inference and ensembling approaches. During inference, the model predicts intervals (lower and upper probability values) for each class, resulting in a final output that is a simplex subset defined by these intervals. The ensembling technique predicts N such intervals and averages them out. Based on this final prediction, the authors derive several uncertainty measures (both aleatoric and epistemic). The authors validate their method through extensive experiments on several image classification tasks (CIFARs and ImageNets) and various model architectures (ResNets, VGGs, ViTs), demonstrating significant improvements over ensembles in terms of model accuracy, quality of uncertainty estimations, OOD detection, and model calibration.
strength_weakness: strength_weakness: * The approach involves too many parameters and procedures for training and inference, making it more complex compared to other methods such as ensembles. This added complexity could limit its practicality (and applicability) and ease of implementation.

* The paper does not provide a comparison with second-order methods such as evidential learning, which leaves a gap in the evaluation of the proposed approach. Including such comparisons could be important since the proposed credal set approach is deeply connected to second-order distribution prediction methods. Including such comparisons would strengthen the validation of the proposed approach and offer a more comprehensive assessment of its performance against established techniques.

* The output size is doubled, which, for tasks with a high number of classes, could significantly increase the number of weights. This could result in higher computational costs and memory requirements, potentially limiting the scalability and efficiency of the approach.

* It is not entirely clear where the improvements over ensembles are coming from or why credal inference brings additional benefits. More explanation is needed to understand the source of these enhancements.

* From the perspective of the experimental evaluation, I would be curious to see evidence that the behavior demonstrated in the paper would hold in other domains, such as texts, graphs, more complicated vision tasks (e.g. segmentation), not limiting to image classification task.

I tend to assess the paper positively and would be happy to increase my score after the discussion.
questions: questions: * Given the increased complexity and number of parameters in the proposed approach compared to other methods like ensembles, how do the authors justify its practicality and ease of implementation? Are there any strategies to mitigate these complexities (for example, how to chose $\delta$ parameters for training or $K$ parameter for the optimized inference)?

* I believe that comparisons with and discussions of second-order methods such as evidential learning in the context of credal inference are important. How does the proposed approach perform relative to these techniques?

* The paper mentions significant improvements over ensembles but does not clearly explain why credal inference brings additional benefits. Can the authors provide a more detailed explanation of the source of these improvements?
limitations: limitations: The authors provide an adequate discussion of limitations in the last section.
paper_summary: paper_summary: This paper introduces Credal Deep Ensembles, which are ensembles of Credal-Set Neural Networks designed to predict lower and upper probability bounds for each class, representing epistemic uncertainty.
strength_weakness: strength_weakness: Empirical results are not very convincing, since the improvements (if there are any) seem to be of marginal nature.

While the paper introduces novel methods and validates them empirically, it could benefit from a deeper theoretical analysis of why the proposed methods improve uncertainty quantification. E.g. the paper introduces novel components like the Interval SoftMax function and the application of DRO, but it lacks a detailed theoretical analysis that justifies their effectiveness.

Lack of baseline comparisons (some model classes were excluded).
questions: questions: Why were Dirichlet-based approaches and other recent uncertainty quantification methods excluded from the comparison? 
(See also comment on baseline comparisons).

Please elaborate on the loss function (10). Why is this a sensible choice? It is not immediately clear to me from the corresponding explanations in the paper. 

How does the Interval SoftMax function handle cases where the intervals for different classes overlap significantly? What impact does this have on the final predictions and uncertainty estimates?

Instead of learning upper/lower probabilities, what is the difference to get the upper/lower probabilities e.g. from an ensemble of NNs? 

How do you guarantee that the computation of the lower and upper probabilities within the credal set is scalable for high-dimensional classification tasks?
limitations: limitations: There is a very short discussion of potential limitations at the end of the paper.
paper_summary: paper_summary: The paper proposes credal neural networks, which output probability intervals for each class as opposed to a single probability estimate. They also propose to use ensembles of these models, averaging the outputs of the members. Their ensembles of credal neural networks show a higher performance than traditional ensembles in various experiments, from epistemic uncertainty (OOD detection), to the normal accuracy on ImageNet.
strength_weakness: strength_weakness: In all experiments, there is only one baseline, namely traditional deep ensembles. I would encourage to pull results for the standard ensemble with DRO loss (Called DEs*-5) from the appendix to the main paper and to also implement other methods, like Mahalanobis (https://arxiv.org/abs/1807.03888), which recently showed SOTA performance on EU estimation (https://arxiv.org/pdf/2402.19460)

The resampling setup is internally correlated as the authors construct their 15 ensembles by drawing 5 members out of 15 trained models repeatedly. The 15 ensembles thus overlap which will likely lead to underestimating the error bars in the results
questions: questions: To calculate accuracy, did you use i_max or i_min?

Do you have some insights into how big the credal intervals are usually, etc? I’m asking because I want to rule out the alternate hypothesis that your method is producing almost-Dirac credal sets.

I understand your motivation of why you apply the vanilla cross entropy loss to the upper bound estimate of the credal set $q_U$ as opposed to the midpoint $m$, but do you have ablations quantitatively supporting this choice?

Do the computational complexity studies in Table 6 include the constrained optimization search for maximum / minimum entropy? In Appendix C it seems like this could be a burden, where training a single ensemble credal ensemble member takes five times as long as training a standard cross-entropy loss member.

There’s a small typo in line 284, “AUORC”
limitations: limitations: The authors acknowledge that the runtime and RAM costs for ensembles are a hindering aspect of their work.
paper_summary: paper_summary: This paper presents Credal Deep Ensembles (CreDEs), an ensemble framework of Credal-Set Neural Networks (CreNets) to produce a high-quality epistemic uncertainty by using a credal set with the Distributionally Robust Optimization (DRO) technique. 
Contributions are:
- A rigorous CreNet final layer in Section 2.1, where the number of nodes is doubled to present the confidence intervals with lower and upper probability bounds for each class. The final set of probability intervals is shown to be proper by satisfying convexity conditions.
- A training procedure with CreNet loss in Section 2.2, where the standard cross-entropy loss is trained with the upper-class probability bounds, and adversarially reweighted DRO is trained with the lower ones.
- Experimental results show CreDEs are better than Deep Ensembles (DEs) by having a higher test accuracy, lower ECE in IID classification, AUROC/AUPRC in OOD detection tasks.
strength_weakness: strength_weakness: 1. The theoretical contributions of this paper are weak. There are no theories to show the proposed method and the loss function can improve the uncertainty or robustness quality. The only theoretically sound result is showing the results in Eq. 4 satisfy convexity conditions in Eq. 2, but this can be derived straightforwardly by using the sum of standard exponential functions for the output vector.
2. Regarding the practical novelty, I also raise a concern about the computational limitation of the proposed method. Because DEs has been raised as computationally inefficient in estimating the NN uncertainty field [1, 3, 4, 5], CreDEs is even less efficient than DEs. In particular, for each ensemble member, in training time, the complexity of Alg. 1 with sorting and the training objective function is more complex than Standard Neural Network (SNN). In test time, the model also requires doubling the number of output nodes than SNNs and needs to solve two optimization problems in Eq. 13, causing a higher latency.
3. The proposed method only works for classification tasks. Additionally, it is also a heuristic and depends on hyperparameter tuning. In particular, choosing the adversarial weights $\displaystyle w$ for DRO is non-trivial and can significantly change the training objective function. Furthermore, choosing the number of class $K$ in Alg. 2 is also heuristic, causing a trade-off between uncertainty estimation quality and computational efficiency. 
4. Lack of empirical comparison with many related works in uncertainty baselines [1]. I would be happy to raise my initial score if important baselines will be added, including MC Dropout [2], BatchEnsemble [3], Rank-1 BNN [4], MIMO [5], SNGP [6], and NatPN [7].
5. Lack of uncertainty evaluations and comparisons (see Questions). Better performance in OOD detection or active learning tasks is only a necessary condition, it is not a sufficient condition for a high-quality epistemic uncertainty algorithm.
6. The training complexity of CreNet is considerably higher than SNN in L-712, causing CreDEs's complexity to be even much higher than DEs by linearly increasing the number of ensemble sizes. The model inference complexity in Tab. 6 is not evaluated in different model architectures across different hardware settings, e.g., GPU, CPU, etc.
questions: questions: 1. Can you qualitatively evaluate the uncertainty quality by using the reliability diagrams [8] and the PDF plot of predictive entropy $\mathrm{H}(p(y|x))$ like Fig. 3 in [9]?
2. Regarding the quantitative evaluation of the epistemic uncertainty in Table 5: How about the performance of DEs? Can you add an evaluation by calculating $Var_i[\mu_i(x)]$ in [10]?
3. Regarding the robustness performance, can you compare your method with DEs regarding the Negative log-likelihood, test accuracy, and ECE on OOD data (e.g., train on CIFAR-10, test on CIFAR-10-C)?
4. Regarding Table 1: What is the number of $K$ used for this setting? Could you please specify why there are significant differences in your reported ECE with [1] on ImageNet about DEs?
5. I am also curious about the results of different settings of DEs. For instance, the DEs used in your paper is the mixture of experts (or operation), how about the product of experts (and operation)?

References:

[1] Nado et al., Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learning, arXiv preprint arXiv:2106.04015, 2021.

[2] Gal et al., Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, ICML, 2016.

[3] Wen et al., J. Batchensemble: an alternative approach to efficient ensemble and lifelong learning, ICLR, 2020.

[4] Dusenberry et al., Efficient and scalable Bayesian neural nets with rank-1 factors, ICML, 2020.

[5] Havasi et al., Training independent subnetworks for robust prediction, ICLR, 2021.

[6] Liu et al., Simple and principled uncertainty estimation with deterministic deep learning via distance awareness, NeurIPS, 2020.

[7] Charpentier et al., Natural posterior network: Deep bayesian predictive uncertainty for exponential family distributions, ICLR, 2022.

[8] Guo et al., On Calibration of Modern Neural Networks, ICML, 2017.

[9] Lakshminarayanan et al., Simple and scalable predictive uncertainty estimation using deep ensembles, NeurIPS, 2017.

[10] Valdenegro-Toro et al., A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement, CVPR Workshop, 2022.

---

**After rebuttal.** The authors provide detailed additional results, and I believe that they are valuable to support the claim of the main paper. So, I increased my initial rating for this paper.
limitations: limitations: Please see my comments on the Weaknesses.

Comments: title: Further responses for remaining questions
Further responses for remaining questions
Further response to the remaining questions
Further response to the remaining questions
Response to authors
Further discussion on Question 5
Further discussion on Question 5
Clarification for Question 5
comment: Dear reviewer, 

Thank you for your positive feedback. We appreciate the time and effort you have dedicated to reviewing our paper and the discussions!
Thank you for your explanation. I also thank the authors for clarifying the ablation study of hyperparams $\delta$ regarding test accuracy. Most of my concerns are addressed. Because of theoretical contribution and computational limitations,  I updated my final score to 5. Good luck!
Dear Reviewer,

We are grateful for your efforts and valuable comments and suggestions in this review process. We promise to include the additional results (e.g. Figure 1 of the PDF) as suggested by you in the revised paper.  

Once again, we thank you for recognizing the quality of our work and supporting the acceptance of our paper.
Dear Reviewer,

Thank you for recognizing the value of our work and supporting the publication of our paper. We are glad that your concerns are fully addressed. Once again, we appreciate your time and effort in reviewing our work and providing valuable comments.
**hyperparameter $\delta$**

Throughout the main experimental evaluations (except the ablation study on $\delta$), we set the default $\delta=5$, to reflect a balanced assessment of the train-test divergence and show how such a value allows our model to outperform the baselines. Upon this setting of $\delta$, the extensive comparisons consistently show our CreDEs outperform DEs.

The ablation study on Hyperparameter $\delta$ (lines 287-312) verifies the robustness of CreDEs across hyperparameter setups in terms of test accuracy, ECE, and OOD detection performance, consistently better than the performance of DEs. For a visible sense, we copied the results of Table 4 of the paper as follows in Table R2. 

Table R2: Performance comparison of CreDEs trained on the CIFAR10 dataset under different settings of $\delta$.

|                     |                 |   0.5  |  0.625 |  0.75  |  0.875 | 0.9375 | 0.96875 |
|:-------------------:|:---------------:|:------:|:------:|:------:|:------:|:------:|:-------:|
|     Test ACC (%)    | $\hat{i}_{max}$ |  93.74 |  94.54 |  94.47 |  94.57 |  93.88 |  93.99  |
|                     | $\hat{i}_{min}$ |  93.75 |  94.55 |  94.47 |  94.56 |  93.87 |  93.99  |
|      ECE Values     | $\hat{i}_{max}$ | 0.0092 | 0.0059 | 0.0053 | 0.0056 | 0.0087 |  0.0086 |
|                     | $\hat{i}_{min}$ | 0.0109 | 0.0087 | 0.0089 | 0.0093 | 0.0087 |  0.0087 |
|      SVHN (OOD)     |      AUROC      |  97.44 |  97.44 |  97.92 |  97.95 |  97.42 |  97.51  |
|                     |      AUPRC      |  93.07 |  96.34 |  97.00 |  96.92 |  98.79 |  98.82  |
| Tiny-ImageNet (OOD) |      AUROC      |  88.28 |  89.01 |  89.10 |  89.18 |  89.95 |  89.24  |
|                     |      AUPRC      |  88.13 |  89.81 |  89.76 |  89.72 |  89.18 |  89.26  |


Table 5 of the paper reports the averaged EU estimate under different settings of $\delta$. You are correct, we could see significant differences between $\delta=0.5$ and $\delta=96875$ (The EU estimates using $\delta=96875$ are much smaller than those using $\delta=0.5$). 

From Table 5, we could also observe that increasing the value of $\delta$ (i.e., giving less importance to the divergence between test and training distributions) leads to a decreasing trend in the average EU estimates per dataset (particularly for ID CIFAR10 samples). This aligns with the intuition that, if the model is more uncertain about the divergence of the distributions (smaller $\delta$), it should express a larger EU. Despite smaller uncertainty values at high $\delta$’s, the difference between In-domain (ID) and OOD samples remains noticeable. This explains why a $\delta$ closer to 1 does not always lead to low-performance OOD detection and why our model’s OOD detection performance is robust against the choice of $\delta$.

The ablation study also shows that we further improve the performance of CreDEs by finding the 'best' $\delta$. One possible way is to conduct standard cross-validation on specific test scenarios. Considering the performance of CreDEs robust against $\delta$, an alternative interesting option, in the presence of multiple datasets (e.g., acquired over time in a continual learning setting), could be applying the DRO loss component to different components of the training set, and assessing the results to robustly select $\delta$. (lines 294-300).

**Regarding the trade-off parameter $K$**

The parameter $K$ represents a trade-off between uncertainty estimation and computational efficiency. As demonstrated in the ablation study in Appendix B.2,  $K\leq10$ is a practical range for measuring epistemic uncertainty using the Generalized Hartley (GH) measure. As shown in Tables 7 and 8 and Figure 7 of Appendix B.2, the application of $K=4$ for OOD detection involving the CIFAR10 dataset (reducing the dimension from 10 to 4 classes and costing 0.02 ms per input sample) and the application of $K=10$ for the OOD detection involving the CIFAR100 datasets (reducing the dimension from 100 to 10 classes and costing 17 ms per input sample) can guarantee the outperformance of CreDEs over the DEs.

Choosing $K$ for computing the upper and lower entropy measures is more flexible. For instance, in Tables 2 and 7 and Figure 8 of the paper, $K=20$ for the ImageNet experiment dataset (reducing the original 1k to 20 classes and costing 6.1ms per sample) shows an improved OOD detection performance compared to DEs.

We appreciate your efforts to review our work and your active engagement in this discussion. We benefit from the process. We hope that the increased clarity of our responses has addressed your concerns.
Dear Reviewer,

We express our sincere appreciation for your recognition of our additional results and your effort in reviewing our paper. Please allow us to address your remaining concerns.

**Lacking of a generalization bound or robustness certificate from a theoretical perspective**

Thank you for clarifying your question. This is true, CreDEs currently do not provide coverage guarantees/robustness certificates. However, the extensive experiments across different datasets, architectures, and tasks do empirically show that our CreDEs generally achieve consistent improvements over DEs. We have acknowledged lacking theoretical guarantees of our CreDEs in our conclusion (lines 349-350) and detailed our future research plan in Appendix E.2. Incorporating your comments, we will expand our emphasis on this point in the main body of the revised paper.

**Computationally inefficient than other baselines**

*Regarding more hardware settings to evaluate the inference cost*

Following your comments, we report the inference cost measured on a single AMD EPYC 7643 48-core CPU in Table R1. From the result in Table R1, we did not observe any significant overhead. In addition, we can also observe that using VGG16 (a lighter model compared to ResNet50) can visibly reduce the inference cost of DEs and CreDEs.

Table R1: Inference cost comparison between SNN and CreNet per single CIFAR10 input of different architectures.

|        | VGG16 (ms) | ResNet50 (ms) |
|:------:|:----------:|:-------------:|
|   SNN  | 19.2 ± 3.8 |  148.2 ± 49.0 |
| CreNet | 23.1 ± 5.2 |  163.3 ± 39.4 |

Since our main evaluation scope focuses on image classification of different deep neural network architectures across various tasks, good-performance GPUs are a favorable choice for implementation. 

We appreciate your consideration of the complexity of applying our models on smaller hardware such as mobile and IoT devices. To our knowledge, in the context of smaller hardware, lightweight neural networks are generally used, and there are also techniques such as quantization to further improve computational efficiency.  We believe that these factors, as well as optimized code implementation of our CreDEs, could help mitigate the added complexity compared to DEs. *If practitioners are already using or planning to use DEs in their applications, we believe it is worth considering our CreDEs to improve performance with an acceptable increase in complexity.*

*Regarding further comparisons.*

We fully agree that there are uncertainty-aware models that are more computationally efficient than DEs and CreDEs. As you rightly pointed out, achieving comparable or slightly lower performance of DEs while being computationally cheaper is the main objective of this line of research. However, since DEs are active strong baselines [1, 2] and are actively applied in practice, such as [3] [4] [5], we aim to improve the performance of DEs (e.g., the quality of uncertainty estimation) to benefit safety-critical applications in particular.

Following your comments, we tried to implement the Rank-1 BNNs in our environment to fairly compare the performance between Rank-1 BNNs and CreDEs. Implementing Rank-1 BNNs requires installing a package of Edward2's Bayesian Layers from their official GitHub repository. Unfortunately, although we followed the instructions, we could not import the edward2 package correctly due to dependency conflicts, software version incompatibility, and limited time. There are the same unresolved issues posted by others on the official GitHub repository. 

Nevertheless, from the main results of Tables 1-4 in the original paper on Rank-1 BNNs, we could observe that Rank-1 BNNs achieve comparable or slightly lower performance compared to DE baselines while being computationally efficient. Combined with our observations in our paper that CreDEs generally outperform DEs, we believe, there is a high probability that our CreDEs could achieve better performance than Rank-1 BNNs.

The practical message we could deliver to the community is that our CreDEs have a high potential to improve the performance of DEs in real-world applications with an acceptable increase in complexity. If DEs are already out of consideration in practice due to computational constraints, our CreDEs would indeed not be an ideal option.
As promised, we will extend our original discussion on the computational limitations of CreDEs (lines 347-348) in the revised paper.

[1] Benchmarking uncertainty disentanglement: Specialized uncertainties for specialized tasks. 2024.

[2] Deep deterministic uncertainty: A new simple baseline. CVPR 2023.

[3] De-tgn: Uncertainty-aware human motion forecasting using deep ensembles. IEEE Robotics and Automation Letters 2024

[4] Flood uncertainty estimation using deep ensembles. Water, 2022.

[5] Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise. Medical image analysis, 2023.
Thank you for your responses, especially Figure 1 of the rebuttal PDF. Given that this will be included in the revised paper and that the runtime analysis may get an improvement (like testing the traditional training in your custom training loop, for a fair comparison), I remain with my score of 7.
Thank you for thoroughly addressing my concerns in your rebuttal. I find the proposed method both novel and compelling, especially with the detailed explanations you provided regarding the source of the improvements and the strategies to manage the complexity of the approach. Additionally, your acknowledgment of the potential for extending this work to other domains strengthens the paper's broader applicability. Given these considerations, I find the contributions of this paper to be very interesting and impactful, and I would like to upgrade my score by 1 point.
**Ablation study on comparing our CreDEs and the post-processing approach**

Thank you for your enlightenment. Inspired by your comments, we did an ablation study to compare the methods.
Regarding the post-processing approach, we compute the upper and the lower bound per class $k$ from the five single predictions from DEs-5, as follows:
$$
q_{U_k} =  \max_{n=1,..,5}{q_{n,k}};  p_{L_k} = \min_{n=1,..,5}{q_{n,k}}.
$$

Then, we formulated a credal set by post-processing for uncertainty representation. Table 2 reports the performance comparison on CIFAR10 vs SVHN and Tiny-ImageNet benchmarks.

Table 2: OOD performance comparison on CIFAR10 vs SVHN and Tiny-ImageNet benchmarks.

|                           | SVHN: AUROC | SVHN: AUPRC | Tiny-ImageNet: AUROC | Tiny-ImageNet: AUPRC |
|:-------------------------:|:-----------:|:-----------:|:--------------------:|:--------------------:|
|           DEs-5           |    89.58    |    92.29    |         86.87        |         83.02        |
|  DEs-5 (post processing)  |    93.77    |    96.06    |         88.78        |         86.83        |
|  CreDEs-5 ($\delta=0.5$)  |  **96.55**  |  **98.17**  |         88.10        |         **87.85**        |
| CreDEs-5 ($\delta=0.875$) |  **97.95**  |  **96.92**  |       **89.18**      |       **89.72**      |

*From Table 2, we can observe that the post-processing approach can improve the performance of classical DEs.  Our method still performs better than others. In addition, due to the post-processing nature (representing the DEs predictions in credal sets), our CreDEs also achieve better performance in test accuracy and ECE values. For instance, test accuracy: **93.75%** ($\delta=0.5$), **93.87%** ($\delta=0.875$) vs 93.32% (DEs-5) and ECE: **0.0092** ($\delta=0.5$), **0.0093** ($\delta=0.875$) vs 0.0131.*

We deeply thank your comments and inspiration, we believe incorporating more extensive comparisons, especially in real-world applications such as medical image analysis could strengthen our future work.

**Following questions about the loss Eq. (10)**

For a training batch, our CreNet first forward propagates via the Interval SoftMax activation (Eq. (4) of the paper) to generate meaningful upper and lower probability vectors to define the credal sets (lines 105-166). Because of the functional structure of Interval SoftMax activation, the upper and lower probability vectors are not computed independently but are correlated. 

In backward propagation, we apply the upper and lower probability vectors to the vanilla and DRO components, respectively. Following the rationale of the design choices (as we explained in lines 157-166 of the paper and the previous responses), the learned prediction interval can represent the boundary cases of the credal set by considering the optimistic scenario (Vanilla component) and the pessimistic case (DRO component), respectively. Namely, assuming that the true class of the label is index $j^*$, the Vanilla and DRO components in Eq. (10) minimize the Cross-Entropy loss $-log_2q_U(j*)$ and $-log_2q_L(j^*)$, respectively, in a mini-batch optimization process (lines 167-171). Thanks to Interval SoftMax, the trained probability intervals are valid for generating credal sets.

In addition, because of the correlated computation between the upper and lower probability vectors in Interval SoftMax, minimizing the DRO component also affects the upper probability, driving the solution away from the all-one upper probability vectors.

Once again, Thank you very much for your effort and your engaging in this discussion! We hope the increased clarity in our responses has addressed your concerns.
Dear reviewer,

We deeply appreciate your recognition of our response and your engagement in the discussion with us. In the following, we address your remaining questions.

**Interpretation of the epistemic uncertainty**

In supervised learning, we assume an underlying data-generating process in the form of an unknown probability distribution $P$ on the input space $\mathbb{X}$ and the target space $\mathbb{Y}$. The lack of knowledge regarding the underlying process is the source of the epistemic uncertainty [A, B, C]. In an ideal scenario, if the model could learn the exact distribution $P$ from the training data, there would be no epistemic uncertainty about a prediction given a new input. Nevertheless, in practice, given that the training dataset represents only a subset of the full space and that the test data may differ from the training data, it is not possible to guarantee that the distribution $\hat{P}$ learned from the training data will be identical to the true distribution $P$. Therefore, the epistemic uncertainty is induced. In our paper, we proposed CreDEs that output lower and upper probability vectors for each input to express this epistemic uncertainty in predictions.

OOD data are significantly different from the training data, but not limited to the OOD data. For example, active learning aims to efficiently train models using minimal data by selectively acquiring additional samples, which are then labeled by experts. In this context, a larger EU indicates that such samples are significantly different from those used in the previous active learning iteration. Using the effective epistemic uncertainty (EU) measure as the acquisition function to obtain samples with a higher EU allows for an efficient active learning procedure [D].

The motivation of our paper is to improve the uncertainty estimation using Credal Deep Ensembles, not limited to the OOD case. We chose OOD detection benchmarks as the main evaluation because they are widely used to assess the quality of epistemic uncertainty estimation. We also performed an active learning case study in Appendix B.5 of the paper, which also shows that our CreDEs provide improved uncertainty quantification.

If we consider only the in-domain samples, effective uncertainty estimation can also improve classification performance through a rejection operation [C, E]. For example, when processing a batch of input data, those with higher epistemic uncertainty are first rejected, and then the accuracy of the remaining test samples is calculated. If the estimation of the prediction uncertainty is valid, the accuracy shows a monotonic increase. To illustrate, we show the results of the CreDEs and DEs on the CIFAR100 dataset in Table 1. 

*The result shows that our CreDEs provide valid EU estimates and better accuracy than DEs.*

Table 1. Test accuracy on the CIFAR100 dataset using EU estimates as rejection metrics under different rejection rates.
|       Rejection Rate       |         0        |        0.1       |        0.2       |        0.3       |        0.4       |        0.5       |        0.6       |        0.7       |        0.8       |        0.9       |      1.0      |
|:--------------------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:----------------:|:-------------:|
|            DEs-5           |   75.80 ± 0.28   |   80.48 ± 0.29   |   84.80 ± 0.29   |   88.37 ± 0.16   |   92.07 ± 0.15   |   95.28 ± 0.16   |   97.44 ± 0.23   |   98.73 ± 0.15   |   99.49 ± 0.10   |   99.87 ± 0.06   | 100.00 ± 0.00 |
| CreDEs-5 ($\hat{i}_{min}$) | **79.54 ± 0.21** | **84.26 ± 0.23** | **87.84 ± 0.18** | **90.90 ± 0.22** | **93.80 ± 0.13** | **96.63 ± 0.15** | **98.32 ± 0.13** | **99.10 ± 0.10** | **99.62 ± 0.15** | **99.90 ± 0.09** | 100.00 ± 0.00 |
| CreDEs-5 ($\hat{i}_{max}$) | **79.65 ± 0.19** | **84.31 ± 0.22** | **87.86 ± 0.18** | **90.90 ± 0.22** | **93.80 ± 0.13** | **96.63 ± 0.15** | **98.32 ± 0.13** | **99.10 ± 0.10** | **99.62 ± 0.15** | **99.90 ± 0.09** | 100.00 ± 0.00 |

[A]  A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion, 2021

[B] Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning, 2021

[C] Quantification of credal uncertainty in machine learning: A critical analysis and empirical comparison, UAI, 2022.

[D] Deep deterministic uncertainty: A new simple baseline. CVPR 2023.

[E] Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise." Medical image analysis, 2023.
I thank the authors for their effort responding to my questions. 

> Epistemic uncertainty in predictions is induced by the probability distribution(s) that generated the training data may be very different from the distribution that generates a test point.   

Can you please elaborate on this comment? If the test data is generated from a distribution different from the training data, the model may be uncertain because it hasn't been trained on similar data. While this is true in the OoD case, it's not the only factor contributing to epistemic uncertainty. If I am not wrong, the motivation for this paper is not only the OoD case, correct? 

> Minimizing the two components together, as in Eq. (10), thus encourages the predicted credal set to extend from the best-case to the worst-case prediction.

This does not fully address my question. For clarification, let me ask a follow-up question, which will help me understand why Eq. (10) is a sensible choice:

Why does Eq. (10) give incentive to learn suitable interval predictions? Are we not minimizing the first term in the sum in (10) by setting all upper probabilities to 1?

> Response to Q4: The main difference is that getting upper/lower probabilities e.g. from an ensemble of NNs would be a post-processing procedure only in the prediction space, no training process is involved. In our CreDEs, the model learns the probability interval from data and could encourage the predicted credal set to extend from the best-case to the worst-case prediction.

I apologize if my question was ambiguously stated. I see your point, that your approach involves a learning process instead of “post-processing” of the ensemble predictions. The question would be, can’t we achieve similar good results with this naïve post-processing approach? At least it would be interesting to look at. 

I remain skeptical about the loss itself, but given the results in the global rebuttal and taking into account the responses to my questions, I increased my initial score.
I thank the authors for the detailed response. I appreciate additional results from the authors, and as promised, I have raised my initial rating. However, there are still concerns, including:

**1. From the theory point of view, the contribution is still weak.**

The intuition answer in the rebuttal for W1 doesn’t’ address my concern. If your model outperforms DEs in practice, e.g., in terms of robustness, I expect the authors to show a generalization bound or robustness certificate. This will show whether your model is generally better than DEs, or only in some specific cases.

**2. CreDEs is even more computationally inefficient than other baselines.**

Regarding the computational limitation in W2-4-6, 
 - Firstly, in Tab.1-PDF, as mentioned in W6, I think more hardware settings should be considered when evaluating computational efficiency. For instance, in powerful resources, CreDEs can be slightly slower than SNNs. But for smaller hardware (lower GPU memory, smaller CPUs), CreDEs can be significantly slower. This evaluation will reflect where CreDEs can be deployed in the real world (e.g., servers, mobiles, IoT devices, etc.).
- Secondly, comparisons with BNN-R, and BNN-F are not convincing to me. They are quite outdated methods. For instance, in the literature on BNNs, please check Rank-1 BNN (others in my citations in W4). I understand that all of them might not be as good as DEs, but they are more computationally efficient than DEs and CreDEs. Adding them to the comparison, I believe will help the community know which model they should use in specific cases.

**3. CreDEs is heuristic and depend on hyperparameter tuning.**

Regarding the heuristic limitations in W3, I am still concerned about the additional hyperparameter selection. I think that they are not trivial in practice. Specifically,
- In Tab.5, I still see significant differences between $\delta=0.5$ and $\delta=96875$. Also, this table only shows EU estimates, but I believe it also changes the accuracy by $\delta$ related to the DRO objective function.
- Regarding the trade-off limitation of the selection of $K$, the rebuttal “increasing the value of improves performance and increases cost” is vague. Increasing the cost is clearly not good in practice.
Table 5: OOD detection performance comparison (%) on VGG16 architecture ViT Base.

|             | CIFAR10 vs. SVHN | CIFAR10 vs. SVHN | CIFAR10 vs.Tiny-ImageNet | CIFAR10 vs.Tiny-ImageNet |
|:-----------:|:----------------:|:----------------:|:------------------------:|:------------------------:|
|             |      AUROC       |      AUPRC       |          AUROC           |           AUPRC          |
|   CreDEs-5  |  **87.30±1.77**  |  **92.24±1.15**  |      **88.17±0.44**      |      **86.94±0.60**      |
|    DEs-5    |    79.80±1.75    |    87.97±1.17    |        83.81±0.81        |        81.67±0.89        |
| DEs-5 (POE) |    82.41±1.56    |    88.51±0.95    |        83.21±1.02        |        78.24±1.17        |

*The ablation study shows again the outperformance of our proposed CreDEs-5. The product of expert strategy improves the accuracy of classical DEs, but significantly damages the calibration performance and overall leads to worse OOD detection performance.*

We deeply thank you again and hope the increased clarity in our responses has addressed your questions.
Dear Reviewer,
 
Thank you for your engagement in the discussion. 

Following your enlightenment, we conducted the ablation study over deep ensembles using the product of expert strategy, denoted as DEs-5 (POE). 

Table 1 reports the test accuracy (ACC) and ECE values on the CIFAR10 dataset of different models across ResNet50, VGG16, and ViT Base architectures.

Table 1. Test accuracy (ACC) (%)and ECE comparison.
|                            |  ResNet50: ACC |   ResNet50: ECE   |   VGG16: ACC   |     VGG16: ECE    |  ViT Base: ACC |   ViT Base: ECE   |
|:--------------------------:|:--------------:|:-----------------:|:--------------:|:-----------------:|:--------------:|:-----------------:|
| CreDEs-5 ($\hat{i}_{max}$) | **93.74±0.11** | **0.0109±0.0017** | **87.92±0.11** | **0.0611±0.0012** | **93.59±0.39** | **0.0104±0.0012** |
| CreDEs-5 ($\hat{i}_{min}$) | **93.75±0.11** | **0.0092±0.0016** | **87.94±0.11** | **0.0203±0.0014** | **93.60±0.40** | **0.0107±0.0014** |
|            DEs-5           |   93.32±0.13   |   0.0131±0.0010   |   85.53±0.10   |   0.0815±0.0011   |   90.43±0.97   |   0.0181±0.0019   |
|         DEs-5 (POE)        |   93.47±0.11   |   0.0610±0.0011   |   85.55±0.08   |   0.1368±0.0008   |   90.56±0.90   |   0.0894±0.0087   |

*Table 1 shows that DEs-5 (POE) could improve test accuracy but significantly reduce the calibration performance of DEs-5 (larger ECE values). In these comparisons, our CreDEs-5 is the most superior method.*

In addition, we also present the Reliability diagram in the table format in Table 2. Table 2 can equally reflect the results of CreDEs-5 and DEs-5 in Figure 3 of the rebuttal file.

Table 2. Test accuracy (ACC) of the ResNet50-based models in each confidence bin of the reliability diagram. 

|      Confidence Bin      | [0.0, 0.1] | [0.1, 0.2] | [0.2, 0.3] | [0.3, 0.4] | [0.4, 0.5] | [0.5, 0.6] | [0.6, 0.7] | [0.7, 0.8] | [0.8, 0.9] | [0.9, 1.0] |
|:--------------------------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|
| CreDEs-5 ($\hat{i}_{max}$) |     0.0    |     0.0    |   0.3333   |   0.1951   |   0.4206   |   0.5492   |   0.6361   |   0.7532   |   0.8849   |   0.9863   |
| CreDEs-5 ($\hat{i}_{min}$) |     0.0    |     0.0    |   0.3333   |   0.2759   |   0.4688   |   0.5701   |   0.6444   |   0.7673   |   0.8899   |   0.9872   |
|            DEs-5           |     0.0    |     0.0    |     0.5    |   0.2857   |   0.4031   |   0.5035   |   0.6569   |   0.7092   |   0.8583   |   0.9839   |
|         DEs-5 (POE)        |     0.0    |     0.0    |     0.0    |     0.0    |     0.0    |     0.3    |   0.3947   |     0.5    |   0.3529   |   0.9396   |

*Table 2 demonstrates the worse performance of DEs-5 (POE) from the reliability perspective. Our model performs best out of the three.*

Furthermore, we also study the uncertainty estimation performance via OOD detection. Here, we computed the entropy of the final prediction of DEs-5 (POE) as the total uncertainty. In the cases of CreDEs-5 and DEs-5, we use the upper entropy $\overline{H}(\mathbb{Q})$ and $H(\tilde{\boldsymbol{q}})$ as in our paper, respectively. *The results in Tables 3, 4, and 5, consistently show the outperformance of our method.*

Table 3: OOD detection performance comparison (%) on ResNet50 architecture.

|             | CIFAR10 vs. SVHN | CIFAR10 vs. SVHN | CIFAR10 vs.Tiny-ImageNet | CIFAR10 vs.Tiny-ImageNet |
|:-----------:|:----------------:|:----------------:|:------------------------:|:------------------------:|
|             |      AUROC       |      AUPRC       |          AUROC           |           AUPRC          |
|   CreDEs-5  |    **95.71±0.42**    |    **97.73±0.27**    |        **89.02±0.10**        |        **88.02±0.15**        |
|    DEs-5    |    94.80±0.43    |    97.26±0.29    |        88.80±0.19        |        87.21±0.29        |
| DEs-5 (POE) |    93.90±0.24    |    96.10±0.21    |        88.03±0.20        |        84.11±0.32        |

Table 4: OOD detection performance comparison (%) on VGG16 architecture.

|             | CIFAR10 vs. SVHN | CIFAR10 vs. SVHN | CIFAR10 vs.Tiny-ImageNet | CIFAR10 vs.Tiny-ImageNet |
|:-----------:|:----------------:|:----------------:|:------------------------:|:------------------------:|
|             |      AUROC       |      AUPRC       |          AUROC           |           AUPRC          |
|   CreDEs-5  |    **87.05±0.80**    |    **93.36±0.42**    |        **82.14±0.14**        |        **80.81±0.16**        |
|    DEs-5    |    84.50±0.49    |    90.78±0.35    |        79.40±0.10        |        75.91±0.14        |
| DEs-5 (POE) |    84.10±0.22    |    89.83±0.16    |        78.11±0.08        |        72.23±0.16        |
Thanks for your response. Regarding Q5, I am curious about the ensemble setting with the "product of experts" [1], not "mixture of experts" setting that is currently used in your paper.

References:

[1] Hinton, G.E, Products of experts, 9th International Conference on Artificial Neural Networks, 1999.