Title: Credal Prediction based on Relative Likelihood
Venue: NeurIPS 2025
Accepted: yes
Keywords: Uncertainty Representation

Credal Set

Relative Likelihood
Abstract: Predictions in the form of sets of probability distributions, so-called credal sets, provide a suitable means to represent a learner's epistemic uncertainty. In this paper, we propose a theoretically grounded approach to credal prediction based on the statistical notion of relative likelihood: The target of prediction is the set of all (conditional) probability distributions produced by the collection of plausible models, namely those models whose relative likelihood exceeds a specified threshold. This threshold has an intuitive interpretation and allows for controlling the trade-off between correctness and precision of credal predictions. We tackle the problem of approximating credal sets defined in this way by means of suitably modified ensemble learning techniques. 
To validate our approach, we illustrate its effectiveness by experiments on benchmark datasets demonstrating superior uncertainty representation without compromising predictive performance. We also compare our method against several state-of-the-art baselines in credal prediction.


Peer reviews:
comments: SUMMARY:
The paper proposes a method to learn credal set-valued predictors for classification problems. To train a credal set-valued predictors, standard approaches often use ensemble models, but those methods lack the notion of "learning" a credal set in the sense that there is no selection criteria going on. This is what makes this work shines. 

The authors also proposed a smart initialisation approach to promote exploration during the training called ToBias.

STRENGTHS AND WEAKNESSES:
Strength:
- the paper is well written. Concepts are easy to understand. The methods are principled.
- I particularly appreciate the authors not using predictive accuracy of any sort (collapsing your credal set e.g.) as a metric, instead use coverage and diversity/efficiency as the comparison metric. 

Weakness:
- As with most credal set-valued or IP incorporated predictors, there aren't a lot of discussion to regression problems. It is not a big deal but seems like your method should also work for regression problem too? (not the ToBias I suppose but the relative likelihood business)

QUESTIONS:
1. (about choosing alpha) Is there a way to propagate my epistemic uncertainty about the threshould to your framework? or does it even make sense to use some withheld dataset to choose some Pareto-ly undominated alphas? What is your opinion on this?

LIMITATIONS:
Limitations are clearly discussed.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-23

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for generating credal predictors within a machine learning framework, with a focus on neural networks. The approach selects plausible models based on relative likelihood. To promote diversity among predictors, the authors introduce a novel initialisation method called ToBias. This strategy ensures that the ensemble begins in a state of full uncertainty, represented by initial predictions located at the vertices of the probability simplex. As training proceeds, the ensemble’s knowledge increases, leading to a progressive shrinking of the credal set.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a structured approach to generating credal predictors using neural networks, with model selection based on relative likelihood. The method is technically sound and supported by experimental results. A key component is the ToBias initialisation strategy, which assigns initial predictions at the vertices of the probability simplex to promote diversity. While this is not a major conceptual innovation, it is a well-considered design choice that supports the overall framework. The authors provide a balanced evaluation, carefully stating the scope and limitations of their contribution.\n\nThe writing is clear and the organisation is logical. The paper presents its arguments methodically, with sufficient detail to support reproducibility. The clarity of the exposition makes the technical material accessible to readers familiar with the field.\n\nThe work is relevant to ongoing research on uncertainty quantification in neural networks. The proposed method is practical and may be of interest to those working with ensemble learning or imprecise probabilities. It addresses a persistent challenge in the area and offers a structured solution that can inform future research.\n\nAlthough the paper does not introduce an entirely new method, it coherently combines known techniques and applies them to a problem of practical importance. In line with the NeurIPS 2025 Reviewer Guidelines, the work provides a meaningful contribution through thoughtful adaptation and analysis of existing methods.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work deals with the task of learning neural network ensembles capable of cautious decision-making. The authors propose CreRL, which constructs this ensemble by learning models satisfying the relative likelihood constraint: each model’s (conditional) likelihood cannot be less than a given percent (\\alpha) of that of the maximum likelihood solution. To ensure diversity the authors (1) use early stopping, that is, stopping training once the neural network satisfies the relative likelihood constraint; (2) they propose an initialization strategy call ToBias, which sets the one of final layer biases to a high value, e.g., 100; models with this an initialization cover the probability simplex better, approximating prior ignorance. Inference is performed by computing the predictive probabilities under all the models and taking the minimum and maximum values to construct a closed convex credal set. These credal sets are evaluated on their efficiency and their coverage; the former measuring the smallness of the credal set, while the latter measuring whether the ground truth probability lies in the credal set. The authors perform this evaluation on the ChaosNLI, CIFAR-10, and QualityMRI datasets. The proposed method CreRL achieves efficiency and coverage. Moreover, varying the hyperparameter \\alpha allows it to tradeoff one for the other. The authors also evaluate out of distribution detection, using CIFAR-10 as the in-distribution data set, and SVHN, Places, CIFAR-100, FMNIST, and ImageNet as out of distribution data sets. They find that CreRL with \\alpha = 1 performs on par or better than the best performing baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Cautious decision making is important in high-stakes domains like healthcare. This method allows highly expressive neural network models to be used with a principled, cautious decision-making strategy.\n2. The proposed method CreRL is simple and easy to implement\n3. Evaluation of credal set-based methods using coverage and efficiency metrics from conformal prediction literature.\n\nWeaknesses\n1. Section 5 Empirical results says that: “Our method (CreRL) with \\alpha 1 performs either the best or is on par with the best methods in OoD detection, except for the SVHN dataset, where the CreWra approach has a small advantage.” But, it is unclear why CreWra has an advantage in the SVHN data set.\n2. In the OOD evaluation, the best performing model has \\alpha = 1. This seems to be the same as an ensemble of fully trained maximum likelihood models, indicating that credal sets based on the relative likelihood constraint are not useful in OoD detection\n\nQUESTIONS:\nThe relative likelihood credal sets seem to be closely related to Rashomon sets in Rudin et al. (2024). Does CreRL approximate the Rashomon set of neural predictors?\n\nRudin, Cynthia, et al. \"Position: Amazing Things Come From Having Many Good Models.\" ICML 2024\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper presents a method to obtain a credal set prediction and introduces $\alpha$, a threshold on relative likelihood to balance between coverage and efficiency. The credal set prediction is based on ensemble of model predictions. The approach is empirically evaluated on three datasets for AUC curve on efficiency vs. coverage while also including a study of OOD detection

STRENGTHS AND WEAKNESSES:
Strengths:
-	The method is relatively simple and presented very clearly.
-	The use of ToBias approach to get a wide credal set at the start of training is commendable.

Weaknesses:
-	The experimental analysis lacks in depth. More datasets could be explored.
-	It is claimed that the proposed outperforms existing methods but it is not evident in the results. The proposed approach clearly lacks in coverage (and by extension accuracy). The OOD results are comparable to the existing methods and do not outperform.
-	A clear study of the accuracy metric is missing.
-	The only novel thing, as per my understanding, is the use of thresholding on relative likelihood to select the best models. While interesting, it does not make a case for NeurIPS level publication.

QUESTIONS:
Why is there a need to do thresholding on relative likelihood? Let’s say, we train 10 models on CIFAR-10 and choose $\alpha$ to be 0.9, how many models will be dropped?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03