Epistemic Uncertainty Quantification To Improve Decisions From Black-Box Models
Authors: Sébastien Melo
Gaël Varoquaux
Marine Le Morvan
Venue: ICLR 2026 Poster (2026)
Keywords: Epistemic uncertainty
Excess risk
Uncertainty quantification
Aleatoric
LLM
Deferral
Calibration
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We introduce a novel estimator of epistemic uncertainty in confidence scores that reveals local miscalibration and supports improved decision-making and more efficient LLM cascades.

Abstract: Distinguishing epistemic uncertainty (model ignorance) from aleatoric uncertainty (task randomness) is critical for reliable AI systems, yet standard confidence evaluation metrics capture different and incomplete aspects of uncertainty. While AUC and accuracy measure predictive signal, proper scoring rules assess overall uncertainty, and calibration metrics isolate part of the epistemic uncertainty but ignore within-bin heterogeneity of errors, known as grouping loss. We bridge this evaluation gap by introducing asymptotically consistent and sample-efficient estimators of the grouping loss and excess decision risk, providing a fine-grained assessment of epistemic uncertainty that complements existing calibration metrics. Applied to LLM question-answering with inherent aleatoric noise, our estimators reveal substantial grouping loss which decreases with model scale and instruction tuning. Their local nature enables automatic identification of subgroups with systematic over- or under-confidence, supporting interpretable confidence audits. Finally, we leverage these estimates to design LLM cascades that defer high excess decision risk predictions to stronger models, achieving higher accuracy at lower cost than competing approaches.

Published as a conference paper at ICLR 2026
EPISTEMIC UNCERTAINTY QUANTIFICATION TO IM- PROVE DECISIONS FROM -OX MODELS
Sébastien Melo', Gaël Varoquaux 1.2 & Marine le Morvan'
1 Soda team, Inria Saclay, Palaiseau, France
2 2:probabl., Paris, France
(sebast ien. melo, gael. varoquaux, marine www.arreeeeeeeorrroo.....e -emrmnee.e-morraeeeenrrrr
ABSTRACT
Distinguishing epistemic uncertainty (model ignorance) from alcatoric uncertainty (task randomness) is critical for reliable Al systems, yet standard confidence evaluation metries capture different and incomplete aspects of uncertainty. While AUC and accuracy measure predictive signal, proper scoring rules assess overall uncertainty, and calibration metries isolate part of the epistemic uncertainty but ignore within- bin heterogeneity of errors, known as grouping loss. We bridge this evaluation gap by introducing asymptotically consistent and sample-efficintt estimators of the grouping loss and excess decision risk. providing fine une-grained assessment of epistemic uncertainty that complements existing calibration metrics. Applied to I.LM question-answering. with inherent aleatoric noise, our estimators reveal substantial grouping loss which decreases with model scale and instruction tuning. Their local nature enables automatic identification of subgroups with systematic over- or under-confidence, supporting interpretable confidence audits. Finally, we leverage these estimates to design LLM cascades that defer high excess decision risk predictions to stronger models, achieving higher accuracy at lower cost than competing approaches.
INTRODUCTION: A NEED TO QUALIFY PREDICTION UNCERTAINTY
The recent European Artificial Intelligence (AI) Act requires careful oversight in some Al applications (EUC, 2024). In high-risk settings, predictive uncertainty behind model's classification choice is central to proper risk assessment and flexible deployment (e.g. adapting to changing regulation on when the risk considered sufficiently low). For example, some hospitals use models to estimate a patient's ss likelihood of surgical success based on their medical records (Mahajan et al., 2023), but these use cases call for thorough evaluation (Senge et al., 2014).
In many high-stakes settings, the best answer is not clear-cut. For instance, in the of surgery prognosis, many unforeseen events can alter the outcome. question such as "Will this patient relapse within the next 6 months?" must be answered with confidence score that accounts for the amount of information available on the patient, unlike a question such as "Was Einstein born before 1800?", which admits deterministic answer. The corresponding challenge of aleatoric uncertainty is often overlooked when evaluating LLMs confidence scores (Xia et al., 2025), which is at odds with their usage, e.g. in health settings (Jiang al., 2023; Singhal al., 2023; Omar et al., 2025; Katz et al., 2024).
Confidence scores may arise naturally from the continuous function that powers many classifiers such as the simple logistic regression, or neural network with softmax output activation. But other settings require confidence score elicitation, as with L.MMs. Indeed, L.MMs output natural language answers which require dedicated techniques to be associated to a confidence (Shorinwa et al., 2025; Vashurin et al., 2025). Our work does not focus on this confidence scores elicitation, but rather on their evaluation. Evaluation and elicitation are often conflated, as evaluation gives signal that can often be used to improve confidence scores in return (e.g. measuring calibration error an evaluation, but recalibration is an improvement technique). And yet, to decide whether to trust Al system's answer, or whether to deploy it, in particular in a high-stake settings, the question of how good its confidence score is crucial.

---

Published as a conference paper at ICLR 2026
But how to measure the "optimality" of confidence scores? Confidence scores should capture the probability that the predicted outcome does occur. Standard metrics such as accuracy or AUC fail to capture this probabilistic aspect, they cannot detect over- under-confidence. Calibration (Gneiting et al., 2007) addresses this by comparing confidence scores to the rate of observed outcome. However, is control on average: classifier may be well-calibrated overall yet overconfident on some subgroups and underconfident on others, with the two effects canceling out (Perez-Lebel et al., 2023). An optimal confidence score should reflect the true probability of the outcome depending on characteristics of the input X, not just match observed frequencies on average. For an input with insufficient information (e.g. not enough details in an LLM query), even the best predictor comes with lot of uncertainty, called "aleatoric uncertainty". It is then a virtue of Al system to report (Kendall Gal, 2017), as predictions are bound to come with errors and reporting this uncertainty can lead to better decisions. On the other hand, the errors of an Al system may arise from imperfections in this system, not using well the available information. Such uncertainty is known as epistemic uncertainty. A well-designed confidence score conveys both: expressing uncertainty when information insufficient and indicating where the model could, in principle, improve.
Measuring the epistemic uncertainty, i.e. whether model is knowledgeable enough, is highly desirable because it indicates where a model can be improved, either globally or locally (Wimmer al., 2023). Global improvements might involve fine-tuning with additional data, while local improvements could include deferring decisions to a human expert or a more specialized, but costlier, model. Such deferral inefficient if the uncertainty aleatoric, since in that case the prediction is already optimal given the available input. Teasing out epistemic from alcatoric uncertainty is very difficult task and arguably the holy grail of learning theory. Some metrics -'proper scoring rules" (Gneiting & Raftery, 2007), such as Brier score or log loss- can select models with smallest epistemic error; but they do not quantify the remaining epistemic error. For a calibrated model, residual epistemic error arises from heterogeneity in the error: the model may be overconfident on some inputs and underconfident on others (see Figure I), a phenomena captured by the grouping loss (Kull & Flach, 2015; Perez-Lebel et al., 2023).
We tackle epistemic error quantification, to characterize how suboptimal a model is. We contribute:
Sample-efficient and local epistemic uncertainty estimators: We introduce asymptotically consistent estimators for grouping loss and excess decision cost that bypass the limitations of traditional confidence binning.
Auditing LLM reliability: We demonstrate that LLMs suffer from significant grouping loss that decreases with model scale and instruction tuning. By providing local estimates, our approach also enables fine-grained and interpretable confidence audits, identifying specific data subgroups where models exhibit systematic over- or under-confidence (see Figure 1).
LLM decision deferral: We further show that our per-sample excess decision cost estimates enable superior LLM cascades, outperforming standard predictive routing by triggering deferral only when the epistemic risk high.
1.00
Calibration curve
Figure 1: Over and under-confident groups for Llama 3.3 Instruction-tuned 70B on the ACSIncome dataset. Groups were obtained by applying our estimator to the confidence scores provided by the L.M. Exact group def- initions obtained from the tree cuts are: Group 37 (green): Master's degree and higher, Age S 24, Female.
Fraction of individuals eaming >50ks/year
0.75
1105 Group 62,
Experienced Graduate women
Group 37, n = 53 Young Graduate
0.50
0.25
Group 62 (yellow): Professional degree be- yond bachelor's o degree, Age 2 37, Female.
0.00
0.25
0.50
0.75
1.00
Predicted probability eaming >50k$/year
2

---

Published as a conference paper at ICLR 2026
2
BACKGROUND AND RELATED WORK
We consider binary classification task where (X, E xx (0, 1) are jointly distributed random variables representing the input space and class labels respectively. We write (*) der P[Y = | the unknown posterior class probabilities, and f X' + [0, 1] the confidence scores.
Notations
2.1 CALIBRATION
Calibration (Lichtenstein al., 1977) measures whether the confidence scores produced by a model f: + [0, match, average, the observed frequency of positive outcomes.
Definition (Calibration). A probabilistic classifier f is calibrated if
Vp E [0,
E E ((X) = p)
We note e co f(X def E|y f(X)] the calibrated score (c is recalibration function). The most widely used calibration metric the Expected Calibration Error (ECE) (Naeini et al., 2015), which measures the difference between predicted confidence and observed frequency by averaging absolute differences across probability bins. Cruz et al. (2024) is among the few studies to quantify LLM uncertainty tasks with inherent aleatoric uncertainty using the ECE. They found that base LLMs tend to be underconfident, while instruction-tune models often become overconfident.
2.2 PROPER SCORING RULES AND GROUPING LOSS
Proper scoring rules evaluate a predicted probability distribution p against an observed outcome Y (Dawid, 1986; Gneiting& & Raftery, 2007). They can be decomposed to separate epistemic uncertainty (due to the model's lack knowledge) and alcatoric uncertainty (due to inherent data randomness), while also isolating calibration component.
Theorem 1 (Proper scoring rule decomposition (Kull & Flach, 2015)). Let & be proper scoring rule and de its divergence. Then the expected loss writes as:
Epistemic loss (EL)
((d,( (ox, Eldalc ((x, + (x),(
(I)
Expected Loss
Calibration Loss (CL)
Grouping Loss (GL)
Aleatoric Loss
The above shows that the epistemic loss can be further decomposed into calibration and grouping components, highlighting that calibration only provides a partial view of epistemic error. When o is the Brier score, the grouping loss corresponds to variance term.
Definition 2 (Grouping Loss (Lemma 4.1 Perez-Lebel al., 2023)). If the proper scoring rule is the Brier score, the grouping loss is given by:
GL t
(V
(2)
The grouping loss measures the variance of true probabilities within a level set of the model f. reflecting that the model groups together inputs with different true probabilities. Modern neural networks were shown to exhibit grouping loss (Perez-Lebel al., 2023; Chen al., 2024), especially under distribution shift. In the remainder of this paper, we use the Brier score as proper scoring rule.
2.3 EXCESS DECISION RISK
key distinction lies between assessing probabilistic suboptimality- the discrepancy between scores and true probabilities and decision risk, which captures the operational suboptimality of the resulting actions. The former is typically measured by proper scoring rules, while the latter evaluates the decision rule itself.
Confidence scores are often used to make decisions, i.c., mapping sample X with confidence f(X) output Y (0. 1}. Given cost matrix A E 1x2x2 where Aiji is the cost of predicting class
3

---

Published as a conference paper at ICLR 2026
when the true class is Y decision theory (Elkan, 2001) gives the optimal decision rule from f*:
+ (1))2 with
def
A1.0 A
Au.o Aa
def
A1,0 o Au,1 -Aoo A1,1.
1" is the optimal threshold that depends on misclassification costs. Given a model f and selected decision threshold I, we consider the decision rule 65,1 def x + 1 f(x)2 and the associated expected
cost of the decision E def Eyjx ( (X),y X z]. Suboptimal confidence scores lead to suboptimal decisions, resulting in excess decision risk. In particular, Perez-Lebel et al. (2025) showed that the excess decision cost can be decomposed into two components: one arising from calibration errors in the confidence scores, and one due to grouping errors.
Proposition 1 (Excess decision risk decomposition (Perez-Lebel et al., 2025)). Ler E R be a cost matrix. The excess decision risk R can be decomposed
E|EC(,,, x) R -
(C( + E|EC(Oes,, ( RT
where R2L and R.L. are the excess costs stemming from miscalibration and grouping loss.
As for the evaluation of confidence scores, calibration alone does not fully capture the suboptimality of an individual decision. Perez-Lebel et al. (2025, prop 3.2) proposed calibration risk and group- ing risk estimates averaged per level ser. Specifically, the bin-wise calibration risk is defined as B C (X))((X) = [(x) and admits closed-formen expression recalled in Appendix B.
2.4 RELATED WORK
Epistemic Uncertainty Estimation Bayesian-inspired methods like MC-Dropout and Deep En- sembles are common techniques for this purpose (Gal Ghahramani, 2016; Lakshminarayanan al., 2017). However, key criticism is that they primarily capture approximation uncertainty (due to finite dataset) instead of model uncertainty (stemming from a misspecified model class), even though both contribute to epistemic uncertainty (Hüllermeier & Waegeman, 2021; Lahlou al., 2023). Bickford Smith et al. (2025) further criticizes the many existing definitions of epistemic uncertainty that are derived solely from the learned distribution (e.g., MC-Dropout and Deep Ensembles) and advocates for the use of externally grounded epistemic evaluation (e.g. using ground truth realizations of the output). In this setting, Mimori et (2021) proposes an epistemic loss estimator that leverages multiple labels per sample, while Johnson et al. (2024) introduces a training paradigm where model learns to predict pairs of outcomes for given sample to estimate epistemic uncertainty. However, multiple annotations are not always available. Finally, Perez-Lebel et al. (2023) propose lower-bound estimation of the grouping loss using local averages within bins of equal predicted probability.
Calibration for Decision Making and Net Benefit In the medical decision melssion-making literature, expected decision risk minimisation is known under the name of Net Benefit maximisation, with the optimization performed over the decision threshold (Rousson & Zumbrunn, 2011) 1). Interestingly, selecting the threshold that minimizes the expected decision risk is equivalent to calibrating the candidate model for use with the optimal threshold t" (Proposition 3.1 in Perez-Lebel et al., 2025; Van Calster & Vickers, 2014). Pfohl et al. (2022) introduced algorithmic fairness objectives in the Net Benefit framework, showing that different subgroups may require different decision thresholds. Indeed, optimizing decisions on average from calibrated scores co (XX (as with net benefit) does not account for group- or individual disparities.
Decision Deferral Decision deferral studies strategies by which a model abstains when uncertain, improving reliability in high-stakes domains (Hendrickx et al., 2024). Simple approaches defer based on a confidence or uncertainty threshold, but these assume that the model's confidence scores are reliable (Wang et al., 2022; Jitkrittum et al., 2023). More sophisticated methods learn deferral policy by training a dedicated rejection function alongside the classifier (Mozannar & Sontag, 2021; Mozannar et al., 2023), or perform model routing, selecting among set of models the one best suited for a given input (Shnitzer et al., 2023). the LLM setting, several works propose cost-based cascading systems but often lack principled decision rule to guide deferral (Hu et al., 2024).
4

---

Published as a conference paper at ICLR 2026
3 PARTITION-BASE BASED ESTIMATION OF EPISTEMIC UNCERTAINTY
Estimating the grouping loss and epistemic decision risk requires knowledge of the true conditional distribution (X) PIY
X]. In practice, however, only observe discrete outcomes Y. Without assumptions on the underlying distributions, a natural approach is to aggregate outcomes to form local average estimates. We adopt this strategy, using a carefully chosen partition build our estimators. Sections 3.1 and 3.2 assume a fixed partition and introduce estimators for the grouping loss and epistemic decision risk, respectively. Section 3.3 discusses the construction the partition.
Notation Let [(n) def { :j [1, [(")] be partition of the input space. For each j. let my (resp.
F3 ( be the number (resp. fraction) of samples belonging to region j. and let PA def PPX E Gy] Definition 3 (Partitioning estimates). For (X, Rt) joint random variables in X x I, the
population level partitioning estimate is defined as:
j(m)
200 (X) = Σ rj x
where =|R X 1.n1
(4)
Let (X,, R.) be n sample points in x IR. The sample level partitioning estimate is defined as:
f(m]
p(m) (x)
xecf")
where T
Σ
R
(5)
TJj
X,EC/M)
The superscript (n) indicates dependence on a dataset of size n. To keep notations lightweight, it will be omitted whenever appropriate.
3.1 GROUPING LOSS ESTIMATOR
The grouping loss corresponds to the variance of probabilities (X) within bins of same probability (eq. (2)). Perez-Lebel et al. (2023)'s grouping loss estimator directly mirrors this definition by binning confidence scores and computing per-bin variances. However, binning comes with several drawbacks. Bin boundaries prevent information from being shared across bins, and level sets corresponding to a bin may form disconnected subsets in feature space X. Both effects may hurt sample efficiency. Deriving binning-free estimators requires departing from the grouping loss expression given in eq. (2), which we achieve in proposition 2.
Proposition 2 (Grouping loss lower bound, A.I). Let E a partition of the input space and r"c the associated population level partitioning estimate applied to R = Y (X). Then the grouping loss is lower-bounded bounded as:
GL (xx)
2 Σ poys de GLi(C)
j=1
GLi(() represents the fraction of the grouping loss that is captured by the partition L. It is defined as the expected squared difference between the outcome and the calibrated score within each region Cy. The tightness of this lower bound depends on how homogeneous the regions are: the smaller the variability of r rXX) = (X) - c cf (X) within a region, the tighter the lower bound, with equality when f - fo is constant over each region. In practice, good partition should group together points with similar under- or over-confidence. Note that such regions may span multiple confidence-score bins, similar over/undeer-confidnnee levels
at different confidence levels p.
Proposition 3 (Debiased grouping loss estimator, A.2). Given partition ((n) and dataset D(a) = {(X,, Y=) let i, (resp. úz) be the sample mean (resp. sample variance) of (X) in region A debiased estimalor of GLy (L())) is:
GL
(n) de
Σ
pj
1
(6)
j=1
5

---

Published as a conference paper at ICLR 2026
The terms involving f3 are plugin estimators for GL)(CC, while the terms involving 8n provide bias correction. Importantly, given a partition, the quantities F3. 0j and pj can easily be computed to obtain an estimator of GL While the proposed estimator is a lower bound of the total grouping loss for given partition, asymptotically converges the true total grouping loss GL for well-suited sequence of partitions:
Proposition 4 (Consistency of the grouping loss estimator, A.3). Let f(n) be partitioning estimate applied to R = Y (X). Assume that f(n) is weakly universally consistent in L2 towards :: + B caf(x) X æ), and that the sequence of partitions L(n) satisfy J(m) /Vn - 0.
(n)
Then the sequence of estimators GL weakly universally consistent in L1, i.e.
[GL (GL-GLØ GL (
0.
Figure experimentally confirms that our estimator convergences to the true value from below as the sample size increases. This consistency result crucially relies the consistency assumption of p(a) towards which according Györfi et al. (2002)'s Theorem 4.1 and 4.2 is ensured given two conditions: (i) the maximum diameter the partition regions vanishes, and (ii) the number of samples per region grows with the total sample size. These assumptions are reasonable for tree-based partition.
Note that our grouping loss estimator, summed with calibration error estimator, yields lower bound of the epistemic loss (Theorem 1).
3.2 POINT-WISE EXCESS RISK ESTIMATOR
The epistemic decision risk, defined as Rx(xx) EC(6),,,)) E ()) (eq. (3)), quantifies the excess classification cost of a candidate model f relative to the optimal (but unknown) model f*. For pointwise estimation, we focus directly on the total epistemic risk rather than its decomposition into calibration and grouping components. Calibration may improve or degrade individual predictions and only guarantees risk reduction expectation. Consequently, while binwise or global calibration risks are non-negative, the pointwise calibration risk R RC*(x) isn't always positive. We therefore directly target the pointwise epistemic decision risk and provide below an analytical expression for R (z).
Proposition 5 (Oracle epistemic decision Risk, A.4). For any X E X, the epistemic risk writes:
Rx(X)) = A (x)
- (x) if f # 1xx)22 else.
The epistemic risk is therefore zero whenever f and f agree, and otherwise proportional to the distance between (X) and the decision threshold t* This distance is represented by the red gradient regions in fig. Intuitively, the further away (X) is from the threshold t*, the costlier approximate the target quantity
is to mispredict. As in grouping loss estimation, f is unknown (x) via residual partitioning estimate.
Proposition 6 (A Bartition-Based Pointwise estimation of the Epistemic decision Risk, A.5). Let ((X) be partitioning estimate (corresponding to partition C) applied to R = Y cx)f(X) and let t" and As as defined in Section 2.3. We define our estimator:
{
AA co 1(x)+ +(x) (( # 1(8)2 else.
(x)
(8)
Then, for any X belonging to region j [1,J]:
- - Rx)((() s |Aal [Aa] rxx) - -
(x))(X) represents the individual epistemic decision risk of candidate classifier f captured by the partitioning estimate f. beyond the calibration risk. Proposition provides a bound on the approximation error, showing that the absolute difference between the oracle and estimated epistemic

---

Published as a conference paper at ICLR 2026
C+ fj
Frequence E[Y f(x)l
0.5X
1F -. 6- =1
Epistemic Risk R
Figure 2: Epistemic decision risk estimation. For model f, we plot the calibration curve c (blue) and the region-wise corrections c + fj (grey). The epistemic decision risk is non-zero in the quadrants where the decisions of / and f disagree (red quadrants), and grows linearly with the distance to the decision threshold (red gradient). Only points lying in the orange part of the region- wise curves have (x, > 0.
6" z Dr,e
0.4
0.3
0.2
6
6 1
0.1
0
0.0
0 6, 1
Predicted probability f(X)
decision risk at X is controlled by the distance between the residual r(X) -co) and its partitioning estimate fy. In practice, tree partitions are constructed to minimize the intra-leaf variance of the residuals, thereby tightening this control. Figure 2 illustrates our epistemic risk estimator: each region fy provides a localized correction to calibrated scores, enabling a refined mapping of individual points to their corresponding decision epistemic risk estimates.
The pointwise nature of R2,c(X) X) enables estimating the potential accuracy loss for each individual input, rather than on average over level sets or the entire data distribution. This property crucial for decision making, as in our LLM cascading experiments (section 4.3), with per-sample risk estimation to guide model selection. As for the grouping loss, this estimator converges to the true value of the epistemic risk if computed from sequence of consistent partitioning estimates.
Proposition 7 (Consistency of the epistemic decision risk estimator, A.6). Let p(m) be a partitioning estimate applied to Y- -co((X) (X). Assume that f(n) is weakly universally consistent in Lz. Then the sequence of estimators R (n)
f.C(n) is weakly universally consistent in L2, ie.,
Epin)
Ex
R
(n) (n)
5. (X) (R)(X)
P]] 11
O.
3.3 ESTIMATORS IN PRACTICE
Partitioning estimate Our estimators GL and RX) both rely on partitioning estimate A. To obtain i we use honest regression trees (Wager Athey, 2017). They mitigate bias in leaf estimates by ensuring that each sample is used either to fit the tree structure to compute the leaf-level estimates, but never both. Concretely, we fit regression tree on the model residuals after calibration, ie.. Y f(X,, using one subset of the data. The resulting leaves define the partition. The estimates within each leaf are then computed on a disjoint subset. Algorithm details this procedure. built to minimize the intra-leaf variance of
Consistent with Proposition 6, the tree partitions the residuals. Under some assumptions (Appendix A.7), honest trees are consistent partitioning estimate. Finally, while boosting models typically outperform single trees predictive accuracy, simple honest trees provide a lower-variance and more interpretable partitioning, which is preferable for evaluation purposes under limited sample sizes. Results with more sophisticated residual models are presented in Appendix C.8.
Implementation aspects For calibration, we use Platt scaling (Platt, 1999). The partitioning estimate uses the same hyperparameters across all experiments to ensure robustness: decision trees with unlimited depth and minimum of 15 samples per leaf at training time. Appendix C.6 experimentally investigates the influence of tree depth. Datasets are divided into a calibration set (10%), fitting set (40%), and an evaluation set (50%) to estimate GL or fit the epistemic decision risk estimator. The choice of input space for the tree is determined by the data type. For standard tabular data, the original feature space is used. In the case of neural networks handling unstructured data like images or text, the partition is constructed from the model's internal representations, such the activations from the last hidden layer (Perez-Lebel et al., 2023) or learned wiw-dimensiooal UMAP embedding (Detommaso et al., 2024). Appendix B.3 further details our proposed implementation.
7

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
4.1 VALIDATION OF THE GROUPING LOSS ESTIMATOR ON SEMI-SYNTHETIC DATA
We evaluate the grouping loss estimator (eq. (6)) on semi-synthetic dataset with ground-truth
0.005
knowledge of fx().. We use the Weather dataset from TabReD (Rubachev et al., 2025),
Grouping Loss
0.004
0.003
which contains more than 16 million samples and exhibits temporal drift-a favorable setting to study grouping loss and consistency. Ap- pendix B.4 details data construction.
0.002
Porez et
0.001
Ground truth GL (0.0041)
0.000
10a
Osamples available the estimator
Confidence scores f are obtained from His- togram Gradient Boosted tree trained on the training split. These scores are then evaluated using our estimator GLa as well the method proposed in Perez-Lebel et al. (2023). Figure 3 confirms that GLa indeed provides a lower bound on the ground-truth grouping loss in expectation. Furthermore, we observe its asymptotic consistency: as the number of samples increases, the estimator converges to the true grouping loss. Finally, our estimator demonstrates greater sample efficiency compared to Perez-Lebel et al. (2023), capturing more grouping loss for a given sample size. This highlights that a binning- free estimator can leverage available data more effectively to capture the remaining heterogeneity in confidence scores.
Figure 3: Sample efficiency and asymptotic con-
sistency of ug semi-synthetic experiment.
Quantifying ying suboptimality The excess decision risk introduced in section 2.3 provides principled way to quantify overall suboptimality. Following the experimental setup of Perez-Lebel et al. (2025), show in Appendix C.1 that decision risk estimator correlates more strongly with the average cost reductions achieved through various fine-tuning strategies than theirs.
4.2 LLMS" GROUPING LOSS: OVER- AND UNDER-CONFIDENCE ACROSS GROUPS
To evaluate LLM confidence scores, we use Folktexts (Cruz et al., 2024), benchmark designed evaluate large language models (LLMs) question-answering tasks with aleatoric uncertainty. Folktexts is constructed by converting tabular data from Folktables (Ding et al., 2021) into natural language questions with binary-choice answers. Each prompt includes general context, a description of an individual row (formatted as "The (column name) is [value)" for each column), and the question of interest. We use all available tasks from Folktexts (ACSIncome, ACSEmployment, ACSMobility, ACSTravelTime, and ACSPublicCoverage) each involving binary choice setting based on socio-cconomic information. These tasks inherently involve aleatoric uncertainty, as the outcome cannot be determined with certainty from the available information. We use 10% of the data for all datasets except ACSEmployment, for which we use 5%, Confidence scores are derived from next-token probabilities of candidate answers. If the possible outcomes are yes and no, the confidence score is defined as P(yes)
P(yo))+
We evaluate the zero-shot confidence scores of 27 open-source LLMs (listed in Appendix B.5), whose grouping loss are summarized in fig. 4. Open-source models required to access next-token probabilities used to compute confidence scores. The selection covers model sizes from IB to 70B parameters, sampled uniformly possible in this range, and includes families with multiple scales such as Llama (IB, 3B, 8B, 70B). Models are evaluated using our estimator GL
Figure 4 reveals a clear size effect: grouping loss decreases with model size, suggesting that larger models provide more reliable confidence estimates. Another consistent trend emerges: across all sizes, instruction-tuned models exhibit lower (or equal) grouping loss compared to their base counterparts, indicating that instruction-tuning may reduce subgroup-specifee biases. This observation complements the calibration analysis of Cruz et al. (2024), which shows that instruction-tunin polarizes answer distributions and induces over- confideniie.cc. Appendix C.2 provides the non- non-nozal grouping loss estimates per dataset, highlighting that LLMs display substantial confidence biases regardless of model size. A closer inspection of the learned groups for Llama 3.3 70B Instruction tuneed (fig. I) reveals systematic biases for certain demographic subgroups, such underestimating the probability
8

---

Published as a conference paper at ICLR 2026
Normalized Grouping Loss
Normalized Grouping Loss vs Model Size
1.00
Base Trend
Mixtral
Instruct Trend
Orca
0.75
DeepSeek R1
Gemma
Llama
Phi
0.50
Liama Mistral
Base
Instruct
0.25
0.00
10a
10
Model Size (Number of parameters)
Figure 4: Normalized Grouping loss function of LLM size, for base and instruct models. For each (LLM, dataset) pair spanning the 27 LLMs and 5 ACS datasets, GL estimates were averaged over 5 random seeds governing the calibration, fitting, and evaluation splits. The resuting GL. estimation was min-max normalized across models for each dataset, and we report the mean and standard error per model, across datasets. The base and instruct trends were computed using a LOWESS regression.
of high income for experienced graduate women, even after calibration. These results echo the findings of Cruz et al. (2024) highlighting biases for Black and White subpopulations.
4.3 ESTIMATING EPISTEMIC RISK IMPROVES LLM CASCADES
A key strength of our risk estimator is that it captures the quality of individual confidence scores. We illustrate this by using our estimator for LLM routing, i.c. selecting from pool of expert models the one expected to perform best at the lowest cost (Hu et al., 2024).
Baseline: Predictive router Predictive routing selects an LLM from a pool without first generating outputs for the incoming query (Shnitzer et al., 2023). For- mally, the accuracy of model m on query denoted Acem (zr), estimated by training on past queries la- beled by whether the model's answer was correct. Hu al. (2024) implements this with kNN and MIP routers trained on query embeddings. Since estimators rely on decision trees applicd to structured query features, trained decision-tree router in the same feature space, using identical hyperparameters and the same total number of training samples as the risk estimator (see section 3.3) to ensure a fair comparison (see Ap- pendix C.4 for complementary results using query em- beddings). At inference time, given a new query x, the router selects the model that best balances expected accu- racy and cost: arg maX pool X Acem (z) - cost
Llama 1B Liama 3B Llama 8B Llama 70B
0.72
Accuracy
0.70
20-20
0.68
Llama 708 *
km1
0.66
R (ours)
0.64
Llamm 18
Lama 66
Predictive router
Llama 38
0.62
0.0
0.2
0.4
0.6
0.8
Average Cost per Sample ($)
Cost-accuracy tradeoffs the ACSIncome dataset when varying the threshold of risk and willingness pay for the Llama instruction-ttune cascade. Here, encodes the user's willingness to pay: larger values favor more accurate, potentially more expensive models.
Figure 5:
Cascades: Cascades provide an alternative routing strategy to predictive routing. A conventional confidence-based cascade sequentially queries a list of models (here, LLMs). At each step, the current model is queried to get a confidence score: if exceeds user-defined threshold 5 for one class, the model is selected; otherwise, the next model is evaluated (Jitkrittum et al., 2023). Compared to predictive routing, costs accumulate each time a new model is queried since a forward pass is required get confidence score, whereas predictive routing queries only a single model per input. However, this strategy proved ineffective in our experiments, as LLMs are poorly calibrated (see Appendix C.5)
Epistemic decision risk cascade (ours): Instead of directly relying confidence scores threshold- ing, we employ our epistemic risk estimate, RLLM,,, (x). rule for deferral. This replaces the oracle quality scores used by Hu et al. (2024)-addee functions- and serves the metric for the cascade process. Appendix B.6 illustrates our proposal.
the difficulty of designing effective scoring

---

Published as a conference paper at ICLR 2026
Ours vs biggest model
Ours vs RC only
Ours vs predictive router
0.30
0.05
ACSiss ACS
Gain in accuracy
0.25
0.125
0.04
ACSEmaloyment
0.20
0.100
0.03
ACSMobilly by
0.15
0.10
0.075
0.02
0.05
0.050
0.01
0.00
0.025
0.00
-0.05
0.000
0.0
0.5
1.0
1.5
1.0
1.5
2.0
2.5
3.0
Fraction of cost
Fraction of cost
Fraction of cost
Figure 6: Cascades: our risk estimator gives better accuracies at lower cost. Each point corre- sponds to a cascade (an LLM sequence). The y-axis is Acc(ours) - Acc(baseline). Values are positive when our cascade improves accuracy over the baseline. The x-axis is Cost(ours)/Cost(baslln)) For values below 1, our model is less expensive. Our cascade based on R (r 0) is compared to Left: the biggest LLM in the cascade, Middle: A cascade based on RC only (T = 0). Right: the predictive router router baseline (A = 100).
If no model satisfies the threshold, the model with the lowest risk is chosen. We compare cascades built using our epistemic risk estimator R a baseline cascade that relies on the bin-wise calibration risk Rd (Appendix B.1) for individual deferral decisions. This baseline highlights the difference between our approach and a strategy based solely on calibration to guide deferral.
Figure 5 shows the cascades and predictive router performances for the LLama 3 instruction-tunee tuned pool of models, on the ACSIncome dataset. For high willingness to pay X (equivalently, low threshold r), our cascade outperforms the largest model in the pool +4% in accuracy compared to Llama 70B - at about 30% of its cost. This indicates that smaller models were appropriately retained for some inputs. Moreover, our cascade based on R.L.M.. (x) (shortened as R. in the legend) systematically boosts accuracy by up to 2% compared to the two competing approaches at any given cost.
Figure summarizes performances across the 5 Folktexts datasets and 120 different cascade pools built from the Llama instruction tuned family (IB, 3B, 8B, 70B), Phi4 (14B), Gemma 2 (27B) and Mixtral8x7B (ordered by size). Performances are reported for T = 0. As shown in fig. 6 (left), our cascade consistently improves upon the accuracy of the largest model average, a 6% accuracy gain at 46% of the cost). Figure 6 (middle) shows that while the calibration risk ReL provides useful score (as seen in fig. 5), taking into account the full epistemic risk allows for high accuracy gains. This is particularly pronounced when models exhibit high grouping loss, such as on the ACSEmployment dataset (see Appendix C.2 for grouping loss values) where accounting for the full epistemic risk leads to accuracy gains of up to 15%. Finally, fig, 6 (right) shows that our cascades systematically outperform the predictive router baseline and achieve strict Pareto optimality (higher accuracy at lower cost, i.e. left upper quadrant) in 60% of cases.
fraction of its cost (on
Effect of the number of samples These results are robust to training set size (Appendix C.7).
With more than 6,000 samples to train the risk estimator (either R or RCL), accounting for full epistemic risk consistently improves cascade accuracy or cost over calibration-only risk across all datasets. Below 6,000 samples, performance gains depend on the grouping loss magnitude. For high-grouping-osss datasets (c.g.. ACSEmployment), full epistemic risk remains beneficial even in low-data regimes, yielding up to + +12% accuracy with only 2,000 samples.
5 CONCLUSION: BETTER EPISTEMIC EVALUATION GIVES BETTER DECISIONS
Additional data can be used either for fine-tuning or for evaluation, analogous to recalibration versus estimating calibration error. While the trade-offs depend on the downstream tasks, evaluation is indispensable. We introduced estimators targeting the total epistemic loss and associated excess decision risk. They improve sample efficiency while revealing model biases thanks to an interpretable learned partition. Importantly, the excess risk estimator operates at the sample level, enabling selective deferral of specific cases to a stronger model or a human at a controlled cost. This property is particularly valuable when deploying L.LSs in complex high-risk environments.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work received funding from Inria's Action Exploratoire AuditAl and was performed using HPC resources from GENCI- -DRIS (Grant 2025-AD0110164118) The authors thank Tristan Haugomat for helping refine the final manuscript.
REPRODUCIBILITY STATEMENT
The source code is available on GitHub'. All datasets and open-source models are publicly available with references in Appendix B. For the semi-synthetic experiment of section 4.1, dataset modifications details are given in Appendix B.4. Finally, the proofs of all theoretical results are listed in Appendix A.
REFERENCES
Freddie Bickford Smith, Jannik Kossen, Eleanor Trollope, Mark Van Der Wilk, Adam Foster, and Tom Rainforth. Rethinking aleatoric and epistemic uncertainty. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference Machine Learning. volume 267 of Proceedings of Machine Learning Research, pp. 4355-4359. PMLR, 13-19 Jul 2025.
Leo Breiman. Random Forests. Machine Learning, 45(1):5-32, October 2001. ISSN 1573-0565.
Lihu Chen, Alexandre Perez-Lebel, Fabian Suchanek, and Gaël Varoquaux. Reconfidencing LLM Uncertainty from the Grouping Loss Perspective. EMNLP 2024 Conference on Empirical Methods in Natural Language Processing, Miami, United States, November 2024. arXiv.
André Cruz, Moritz Hardt, and Celestine Mendler-Dünner, Evaluating language models as risk scores. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), December 2024.
A. Dawid, Probability forecasting, In Encyclopedia of Statistical Sciences. John Wiley Sons, Ltd, 1986. ISBN 978-0-471-66619--3,
Gianluca Detommaso, Martin Andres Bertran, Riccardo Fogliato, and Aaron Roth. Multicalibration for confidence scoring in 1ls.. In International Conference on Machine Learning, pp. 10624- 10641. PMLR, 2024.
Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt, Retiring Adult: New Datasets for Fair Machine Learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. Curran Associates, Inc., 2021.
Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17, pp- 973-978. Lawrence Erlbaum Associates Ltd, 2001.
EUC. Al Act Regulation (EU) 2024/1689 Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). Publications Office of the European Union, 2024.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR
hhttp:://gtt http://gtttttttteee thub. com/ soda- inri /epp stemi c-ug-deci en
11

---

Published as a conference paper at ICLR 2026
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. 102(477):359 378,, 2007. ISSN 0162- 1459, 1537-274X.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(2): 243-268, 2007.
László Györfi, Michael Kohler, Adam Krzyžak, and Harro Walk. Partitioning Estimates. In A
Distribution Free Theory of Nonparametric Regression, pp. 52 69. Springer, New York, NY, 2002. ISBN 978-0-387-22442--.
Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with reject option: A survey, 2024.
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-lIm routing system, 2024.
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. 110(3):457-500 2021. ISSN 1573-0565.
Lavender Yao Jiang, Xujin Chris Liu, Nima Pour Nejatian, Mustafa Nasir-Moin, Duo Wang, Anas Abidin, Kevin Eaton, Howard Antony Riina, Ilya Laufer, Paawan Punjabi, al. Health system-scale language models are all-purpose prediction engines. Nature, 619(7969):57-3622 2023.
Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. When does confidence-based cascade deferral suffice? In Thirly-sevnnth Conference on Neural Information Processing Systems, 2023.
Daniel D Johnson, Daniel Tarlow, David Duvenaud, and Chris J Maddison. Experts don't cheat: Learning what you don't know by predicting pairs. arXiv preprint 20X2v22402.08733, 2024.
Uriel Katz, Eran Cohen, Eliya Shachar, Jonathan Somer, Adam Fink, Eli Morse, Beki Shreiber, and Ido Wolf. Gpt versus resident physicians benchmark based on official board scores. NEJM AI, I(5::Adbb23001192, 2024.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Meelis Kull and Peter Flach. Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration. In Annalisa Appice, Pedro Pereira Rodrigues. Vítor Santos Costa, Carlos Soares, João Gama, and Alípio Jorge (eds.), Machine Learning and Knowledge Discovery in Databases, volume 9284, pp. 68-85. Springer International Publishing. Cham, 2015. ISBN 978-3-319-235277 978-3-319-23528-1
Salem Lahlou, Moksh Jain, Hadi Nekoci, Victor I Butoi, Paul Bertin, Jarrid Rector-Brooke Maksym Korablyov, and Yoshua Bengio. DEUP: Direct epistemic uncertainty prediction. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. Expert Certification.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.
Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Sarah Lichtenstein, Baruch Fischhoff, and Lawrence D. Phillips. Calibration of probabilities: The state of the art. In Helmut Jungermann and Gerard De Zeeuw (eds.), Decision Making and Change in Human Affairs: Proceedings of the Fifth Research Conference Subjective Probability, Utility, and Decision Making, Darmstadt, 1-4 September. 1975, pp. 275-3244 Springer Netherlands, 1977.
ISBN 978-94 94-0-0--2777--8
Aman Mahajan, Stephen Esper, Thien Htay Oo, Jeffery McKibben, Michael Garver, Jamie Artman, Cynthia Klahre, John Ryan, Senthilkumar Sadhasivam, Jennifer Holder-Murray. and Oscar C. Marroquin. Development and Validation of Machine Learning Model to Identify Patients Before Surgery at High Risk for Postoperative Adverse Events. JAMA Network Open, 6(7)xe2322285 July 2023. ISSN 2574-3805.
12

---

Published as a conference paper at ICLR 2026
Nicolai Meinshausen. Quantile regression forests. J. Mach. Learn. Res., 11983-9999 2006.
Takahiro Mimori, Keiko Sasada, Hirotaka Matsui, and Issei Sato. Diagnostic uncertainty calibration: Towards reliable machine predictions in medical domain. In International Conference on Artificial Intelligence and Statistics, pp. 3664-3672. PMLR, 2021.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert, 2021.
Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning to defer to humans, 2023.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
M Omar, R Aghareia, B Glicksherg. G Nadkarni, and E Klang. Benchmarking the confidence of large language models in answering clinical questions: Cross-sectional evaluation study. JMIR Med Inform, 2025.
Alexandre Perez-Lebel, Marine Le Morvan, and Gaël Varoquaux. Beyond calibration: estimating the grouping loss of modern neural networks. In ICLR, 2023.
Alexandre Perez-Lebel, Gael Varoquaux, Sanmi Koyejo, Matthieu Doutreligne, and Marine Le Morvan. Decision from suboptimal classifiers: Excess risk pre- and post-calibration. In Yingzhen Li, Stephan Mandt, Shipra Agrawal, and Emtiyaz Khan (eds.), Proceedings of The 28th Interna- tional Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pp.
PMLR, 03-05 May 2025.
Stephen R. Pfohl, Yizhe Xu, Agata Foryciarz, Nikolaos Ignatiadis, Julian Genkins, and Nigam H. Shah. Net benefit, calibration, threshold selection, and training objectives for algorithmic fairness in healthcare, 2022.
John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pp. 61-74. MIT Press, 1999.
Valentin Rousson and Thomas Zumbrunn. Decision curve analysis revisited: overall net benefit, rela- tionships to roc curve analysis, and application to case-control studies. BMC Medical Informatics and Decision Making, 11:45 45, 2011.
Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy. and Artem Babenko. Tabred: Analyzing pitfalls and filling the gaps in tabular deep learning benchmarks. In The Thirteenth International Conference on Learning Representations, 2025.
Robin Senge, Stefan Bösner, Krzysztof Dembezyński, Jörg Haasenritter, Oliver Hirsch, Norbert Danner-Banzhofff, and Eyke Hüllermeier. Reliable classification: Learning classifiers that distin- guish aleatoric and epistemic uncertainty. 255:16-29 2014. ISSN 0020-0255.
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuckai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets, 2023.
Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Ren, and Anirudha Majumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Computing Surveys, 2025.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 172-7972 2023.
Ben Van Calster and Andrew Vickers. Calibration of risk prediction models. Medical decision making an international of the Society for Medical Decision Making, 35, 08 2014, doi: 10.1 10.1177/022298XX1444722
13

---

Published as a conference paper at ICLR 2026
Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, AbdeIrahman Sadallah, Kirill Grishchenkov, et Benchmark- ing uncertainty quantification methods for large language models with Im-polygraph. Transactions of the Association for Computational Linguistics, 13:220-248, 2025.
Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests, 2017.
Xiaofang Wang. Dan Kondratyuk, Eric Christiansen, Kris M. Kitani, Yair Alon, and Elad Eban. Wisdom of committees: An overlooked approach to faster and more accurate models, 2022.
Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: are conditional entropy and mutual information appropriate measures? In Proceedings of the Thirty-Ninth Conference Uncertainty in Artificial Intelligence, UAI '23. JMLR.org. 2023.
Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, and Hang Liu. A survey of uncertainty estimation methods on large language models. arXiv preprint (xXiv:5533.0012,, 2025.
14

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
A Proofs
16
A.1 Proof of Proposition 2: Grouping loss lower-bound
16
A.2 Proof of Proposition 3: Debiasing the grouping loss estimator
16
A.3 Proof of Proposition 4: Consistency of the grouping loss estimator
18
A.4 Proof of Proposition 5: Oracle epistemic risk expression...
21
A.5 Proof Proposition 6: Control the epistemic risk estimator
22
A.6 Proof Proposition Consistency of epistemic risk estimator estimator.......................
26
Honest trees consistency
26
B Additional experimental details
27
B.1 Recall of the bin-wise calibration risk
27
B.2 Honest tree construction algorithm
28
B.3 Discussion on hyperparameters and implementation choices
28
B.4 Semi-synthetic data generation.
28
List LLMs used section 4.2 and
29
B.6 Diagram of cascading method
29
C Additional experiments
30
C.I Quantifying suboptimality...
30
C.2 Grouping loss results on every individual datasct from folktexts
33
C.3 LLM accuracy on each ACS dataset..
34
Computing risks on an emhedding the queries
34
C.5 Comparing to cascades deferring based on confidence scores.
35
C.6 Tree depth ahlation study
36
C.7 Training size evaluation
38
C.8 Using stronger models to predict individual residuals
40
15

---

Published as a conference paper at ICLR 2026
A PROOFS
A.1 PROOF OF PROPOSITION 2: GROUPING LOSS LOWER-BOOE BOUUDD
Proposition 2 (Grouping loss lower bound, A.1). Let L be partition of the input space and r's the associated population level partitioning estimate applied to R= Y fx)( Then the grouping loss is lower-bounded as:
J
GL -L=B -
2
Σ porj dnl GLi((E)
==1
Proof. We recall that:
= cof( 1.yu
and Pj = P(X Cg]-
We have:
GI. (
(9)
E,[E[[((() - cx]())) X 1
(10)
(x) x 1)
(11)
-co X E 1)
(12)
Σ porj²
(13)
(10) Total Expectation (C is partition of X)
(11) Jensen's inequality to the function x + x
(12) Y is binary and Y ~ (X))
(13) By definition of r and p5
Which concludes the proof.
A.2 PROOF OF PROPOSITION 3: DEBIASING THE GROUPING LOSS ESTIMATOR
Proposition (Debiased grouping loss estimator, A.2). Given a partition ((a) and a dataset D(a) {(x,, Y.), let (resp. úg) be the sample mean (resp. sample variance) of - ccof (xf in region j. A debiased estimator of GLis is:
GL
(n)
de
P5
Új
(6)
T
The proof is inspired from the proof of proposition 4.3 in Perez- et al. (2023).
Proof. We note
GLplugin
Σ
p
j=1
the plugin version of the estimator from the definition of GL1b in Proposition We have
GT plreginn
=
Σ
j=1
16

---

Published as a conference paper at ICLR 2026
In the case where region j is empty (i.e. n2 0), P, 0 and fy is undefined. We therefore set Py = in this case. Hence, for j E [1, J] fixed, we have:
[[ $,20 py]]] E p, [
total expectation
with:
[ py] Eiry 1 VDA 1
Furthermore
Bif, |,,1 = E
Σ
Y, co-cf((X)) lis
(14)
XiEC,
E
Σ X;ELi
Y cof(o((X))
(15)
Nj
B|Y; cof(Xi) | Xr E cy]
(16)
E|Y - cof(x) x E c,l
(17)
(14): by definition.
(16): linearity of expectation.
(17): Y, c(x() are identically distributed Lj-
And
ViA py]
V 1
Σ n2 X,EL;
Y, cof 18)
(18)
-/
Σ
Y, co-cof(X))
(19)
11j
X,ELS
Σ
VIY coo((x)) X E,l
(20)
-
1 1=1
vly - cof( voyecx(()|XXEE)
(21)
(18): by definition.
(20): (Y: - cco((X)) are independent.
(21): Y, - coo(()) are identically distributed Lj-
Which gives:
17

---

Published as a conference paper at ICLR 2026
E[p,r}]
E p5 1 By 20 Bly [j
B|y -co((X)) X E c,1²+ _vy - cof((x) x Lil)
E p 20
=
=P cof(X) X E 2,1² 2
+E
V - cof(()) x 2,110/20
n
-, cof(x) x E +,}²+ -1 (1-1;, -co x E Gyl
1 (1 (1- -il-,)VV--ooff x E EG)
B| - cof((o x xx E 4,}
Pj
nj
plugin torm
bias term
With Vj (1 Pj)". the probability that samples falls in region Lj. We make the approximation that My R O, which correct when n and pj are sufficiently large. Therefore, we debias the plugin estimate leading estimator:
GLi((D) Σ
Pj
0j
nlj
f=1
with the unbiased sample variance (i.e. with Bessel correction) estimate:
8 det
Σ (Y: co-
1 Σ Y - -oxx)))())
f iEC,
SEC
A.3 PROOF OF PROPOSITION 4: CONSISTENCY THE GROUPING LOSS ESTIMATOR
Recall that r is defined as r: + E|Y - cof(X) X = z), while f(X) is a region-wise wise constant approximation, as defined in eq. (5).
Recall that by definition:
GL = Exl(f co f(x))²
- co ((x((()()))))))))))))))) x]3]
Ex(r(X)2
And defined:
GT.p
(n)
def
Σ
p3
13
1
ij
nz
j=1
in eq. (6).
We introduce the true expectation of the sample level partitioning estimate as an intermediate value:
GL.(C(")) ((())) Ex(p²(X)]
(22)
E
(23)
(24)
With p, P[x c.")). For the proof of Proposition 4, we require two lemmas: lemma showing (n)
the consistency of (Lin(C))), D(n)) towards GL, and lemma for the consistency of GL towards
18

---

Published as a conference paper at ICLR 2026
Lemma 1. Let (L())) be sequence of partitions such that the associated partitioning estimate p(m) is weakly universally consistent in L2 towards r TTen GLo((((), (D()) is weakly universally consistent L1 towards GL:
Epon) [| [IGT. (GI.-Glan - D(m)) ||
0
Proof. The partitioning estimate p(n) is consistent towards
r: r:Z E|y cof(X) Xx= x])
that from Györf al. (2002)'s definition 1.1. page 13:
D(w)
Ex ((Pt)) - -xx))]] (x)-)())))]
and from (24):
GL (D()) Ex (x)2)
We compute:
GL - (L)
Ex Ex|r(x) (x) 1
(25)
= Ex((r((X (xx) 2
(26)
Exll r(x) - (x) || rxx) [1
(27)
4Ex r(X)- (x) [2]
(28)
(27): Cauchy Schwarz.
(X) (28): r(X) + p(*) (X) |< 2 by definition.
So taking the expectation over (()), we have:
Epon
GL GLo,(C("), D(m)) |2] < 41E [x([ - (X( |2]
4-+0000
by consistency hypothesis. Finally, we use Cauchy Schwarz to obtain a result in L1 norm:
ED(()) [] GT. - GT.
-TTa ((()),(()) 11 Bpmn GT. (Tm)) Dim)) [2]
n-++00
Lemma 2. Let (L(")) be sequence of partitions such that the associated partitioning esti- mate p(n) is weakty universally consistent in L2 towards with sequence of partitions ((n) s.t.
J(a) /v 1/-+00 (C)(((m), D(n)) is weakly universally consistent towards GLi((C()), D(n)) in L
Epo:; ( ((()) (Ln((C(m)
0
+00+000
"weakly universally consistent according Györfi et al. (2002)'s definition 1.1. page
19

---

Published as a conference paper at ICLR 2026
Proof.
Jn.
(T)(((())) - GT..1 ((()) ((())) 1= =
Σ By (p
)² 1 (mm
Σ
ps (f)
(29)
j=1
Jn
Jn
x
Py py((fin) )² - -py 1
+
(())
(30)
Jn
βi Pj 1 (1/ +
(31)
2=1
pj - pı 1 (1))))2 Jn
<
Σ 3=1
(32)
(29): By definition of both terms.
(30): Triangle inequality.
(31): Triangle inequality.
(32): Y - cx IS 1 so oft) s 1.
Taking the expectation over D(n):
Jn
D [| GLo(( GLib (Li)( ((())) || s ED(n)
Σ
Do Pj (1/
+
In
(33)
n
E EDI [1 [5 - Pj (1())))]
In
(34)
n
Jn
Jn
EDOM) [110]] Pj +
(35)
Jn
Jn
E [1 [3j - Ps 121 +
(36)
Jn
V Din [é, +
(37)
p1(1 (1 pU))
J
(38)
n
n
/=1
Jn
Jn
5
+
(39)
2/n
n2
3J
<
(40)
2/n
n-++00
(34): Iu doesn't depend on ((n).
(35): |Y co 1 so v(m) < 1.
(36): Cauchy Schwarz.
(37): (pi [p,]] Pj-
(38): Variance of a Bernoulli randome variable.
(39): (1 p3) & 1/ for Pj E [0, 1].
20

---

O. Published as a conference paper at ICLR 2026
(40): J/m///// /vñ -
We now prove the final result.
Proposition 4 (Consistency of the grouping loss estimator, A.3). Let pin) be a partitioning estimate applied to R = Y - c co f(). Assume that f(n) is weakly universally consistent in L2 towards r: + E cof(X) X = 2). and that the sequence of partitions ((v) satisfy J(m) /vin -> O.
(n)
Then the sequence of estimators GLub is weakly universally consistent in L1, i.e.
D())
[L
GL
|]
Proof. Combining Lemma and Lemma 2, and using the triangle inequality, we have the result.
A.4 PROOF OF PROPOSITION 5: ORACLE EPISTEMIC RISK EXPRESSION.
We use Perez-Lebel et al. (2025)'s corollary B.4 recalled below to derive an expression of (X):): Corollary (Difference in expected utilities, (Perez- et al., 2025)), Let 61, E X + {0. 13 and x E X. Then the difference in expected costs (defined as the opposite of the expected utilities in Perez-Lebel et al. (2025)) of 6, and 82 conditional to x is:
EC x) EC((1, x) AA(6)1 (f(z))(f(())
Proposition 5 (Oracle epistemic decision Risk, A.4), For any X E X, the epistemic risk writes:
Rx(X) - AA (X)) - fl # 1(x)22
else.
Proof. We set X X. Using Perez-Lebel et al. (2025)'s corollary B.4, we have
R2 = EC X) EC( -EC(
(41)
- (X) t*)
(42)
We proceed by case disjunction:
Case 88- (X) and (X) =0
R2 As(ō -x,,,))))),,,,)))()(()) (x)( -t)
(43)
= =Aalf' )²)
(44)
= =Alf - -t"l
(45)
Because (x) - -1 0 since 50-,1- (X) 1.
Case 8-- (X) and (X)
=
(46)
As(tt (X))
(47)
= =Asff((X) -t
(48)
Because (X) - since 8,-,0- = =0.
Case 3: 8-- (X) 6),(X)
21

---

Published as a conference paper at ICLR 2026
Ry(X) = As (x)
(49)
0.
(50)
This gives
(X) = { Aa (((() - 1
if (x)2 # 18022
(51)
else.
And concludes the proof.
A.5 PROOF OF PROPOSITION 6: CONTROL OF THE EPISTEMIC RISK ESTIMATOR
We now prove lemma 3, which will be useful prove propositions and
For any partitioning estimate g : X + Σj Ej=19;1xec; define:
Rs (X) del Ao Aalcoo(X))+ + -
if (o((())))())) 1x0)2
(52)
else.
Moreover, let:
AR X) del R2(()- R)/(
Lemma 3. For any partitioning estimate 9 X + E/=1,, gx1xe have:
Vj E [1,]],XE Lj, AR S (Mallr(XX) 93
Or equivalently,
E
(a)l)(()))))))-)----))) -
Proof. Let E [1 and X Cj. We proceed by case disjunction:
Case 1: ((X) <1
Case 1.1: f(X: (t) 2t
2
We have:
=
= Ao (x -
and
Rx) Ao (x) + -1
Therefore,
(R, = As (X) -1 11 (x))) + - 1
(53)
A x [If((x) - -1" 1e + +1
(54)
| fxX) t- - (cof(x) 03 -t))
(55)
-|Aa| ((() (co f(x) +
(56)
|AA ((X) g3
(57)
22

---

Published as a conference paper at ICLR 2026
(53) By definition
(55) Because (X) 2 t" and co f(X) + 9 2
(57) By definition of rX))
Case 1.2: f(X) <t, (,,, <t"
We have:
R2(X) Ao (X) 1*
and
Rx) = 0.
Therefore,
(g) AA (X) t"
(58)
(X) - -01
(59)
I (x)( 111 + ( 1()
(60)
(X) t* +t (cof(X)+
(61)
$ [Aal(f"(X) (cof(X)+
(62)
$ | r(X) 93
(63)
(60) [eo + - -
(61) (' 1" and c (() + gj <t
(63) By definition of r(X)
* Case 1.3: f(X) < <t, <x,,
We have:
Ry(X)=0 =0
and
Rx) = Ao cco((X) + - -
Ag X)
Aa (x)) + + --1 (Allcof(M)) +9, -t'
(64)
(65)
(A)( + +9ù 1" + (x) -x))
(66)
(x)(cof(X)+ + 9; f" +t -f(())
(67)
4 Aal(cof(X) + 9 - (X))
(68)
< AA r(X) - 9/
(69)
(66) (X) 1- 0
(67) (X) t" and c o f(X) + 9; 2
(69) By definition of rXX)
< Case 1.4: f(X) < <X)
+ <t
23

---

Published as a conference paper at ICLR 2026
We have:
Rx(X)= = 0
and
(x =
Therefore.
AR2(X) =0
(70)
$ (|Aa|1(()) - 03
(71)
Case 2: f(X) 2t
Case 2.1: 2() <15 <0
We have:
Rx(X)= = Ao 1 (X) -t"
and
(x) = - (alcco(X)+ppp
+ + -1
Therefore,
X9, = Aa (XX - 1 co)(()) - 1
(72)
A [I((X) -t' 1c1- + - t"
(73)
= |Aa/ (x() - (t" - ((x) + 9,)))
(74)
= |A cxs(x) + 9j -x)
(75)
= |Aa/ r(X) 9.
(76)
(74) (XX) < t" and fxx + 9j <t"
. Case 2.2: f(X) 2 1, (X) < 2 co)))(())- + +g 2
We have:
Ry(X)= = Ao A (X) 1")
and
Rx) =0
Therefore,
(g, =
A (xI5 -
(77)
(x) -t
(78)
$ (*1 + - -1")
-
(79)
x AAal( -f'())+ + (cof(() + 9j t*)
(80)
- f
(81)
< A r(X) -93
(82)
(X) < 1" and co (X) gj 2
24

---

Published as a conference paper at ICLR 2026
Case 2.3: f(X) 2t, 2 (t,f(()) <t'
We have:
Ry(X)= =0
and
R1 As coo((X + +g -t
(g)
AA I cof(() t - t*
(83)
Malloo f(X) + 9 1*]
(84)
A + 9, f"l + (x)
(85)
(AX) 9₁ t' - (x()
(86)
$ (o))cof((X+ 9j f((X]
(87)
|Aa| r(X)- -
(88)
(86) (X) < and co f(X) + 9; 2
Case 2.4: f(X) 21, f" 1", co, + <p
We have:
Rx(X)
and
Ri O.
Therefore,
A) =0
(89)
<|Aa||(X)) 9j
(90)
This ends case disjunction and concludes the proof of lemma
Proposition (A Partition-Based Pointwise estimation of the Epistemic decision Risk, A.S). Ler F(X) be partitioning estimate (corresponding partition L. applied to Y cco((X) (X), and let 1" and Na as defined in Section 2.3. We define our estimator:
(x) der {
Aa co f(x) ((x)) -
( # 1(x12*
(8)
else.
Then, for any X belonging to region j [1,
(x) - Rx)( [4x]]((X) -(ax))--))) -
Proof. Let L be partition of the input space, and X E X' belonging to region Lj.
From lemma 3, by using g we have by definition R RG, and:
AR, (F, |Aa r(X) F3
which ends the proof of proposition 6.
While proposition 6 provides pointwise control, it is easy to obtain a control on average over each region of the partition.
25

---

Published as a conference paper at ICLR 2026
From lemma 3, by using 9 r2 we have
(R,( s |دد (XX) -r
Taking the expectation conditional to by:
(()R((X) X E 4,] S E AA r ((X) - r3 x cs)
(91)
< | B[((()) - -33) Cgl
(92)
- | V(rxx) | 12)
(93)
(92) Cauchy Schwarz
(93) rj E c cof(X) 4,) co 4,] [[][)rr( [,]
A.6 PROOF OF PROPOSITION 7: CONSISTENCY OF THE EPISTEMIC RISK ESTIMATOR
Proposition (Consistency of the epistemic decision risk estimator, A.6). Let p(m) partitioning estimate applied to Y -cof(() Assume that f(m) is weakly universally consistent in Lz. Then the sequence of estimators R (n) j,c(m> is weakly universally consistent in L2, i.e.,
DDD
[x
R(n) f.cmm (X) - R2((x) P]] m-++00 0.
Proof. We have:
Eps:
Ex Rx(X) - R(n)
5.c(m) (X) [3]] < ED(n) E (r(x) - p(m) (X))²
(94)
A Epti Ex (r(X) (x))
(95)
(96)
カ-+00
(95) Lemma 3 applied to g = p(n)
(96) Consistency of p(n)
Which concludes the proof.
A.7 HONEST TREES CONSISTENCY
We recall that honest trees proceed in two stages:
Learn the tree partition (i.e., the splits) using one subset of the data.
2. Estimate the leaf values using a disjoint subset.
Because disjoint data subsets are used for partitioning and leaf value estimation, the analysis is greatly simplified. In particular, the partition can be treated as fixed when analyzing the leaf value estimates, since it does not depend on the data used for estimation. It allows to directly apply Theorem 4.2 from Györfi et al. (2002).
We consider that the number of samples used for learning the partition (training set) and the number of samples used for leaf values estimation (evaluation set) both increase with the total number of samples available, so that both tend to infinity when the total number of samples tend to infinity.
Based on previous results (Meinshausen, 2006; Györfi et al., 2002), we can show that honest trees satisfying the following assumptions consistent.
26

---

Published as a conference paper at ICLR 2026
Assumption 1: X is random variable with compactly supported distribution.
Note: Meinshausen (2006), is assumed that features are bounded in [0, 1]d for notational purposes, but the distribution of X only needs to be of compact support.
Assumption 2: for large n, i.e. ja(, R2//n growing for large n, that is 1/ min ny n-+++000
The proportion observations in leaf j relative to all observations is vanishing
O. The minimal number of observations in leaf is
0.
Assumption The probability that variable m ..., d is chosen construct split point is bounded from below for every leaf by a positive constant. After a split, child leaves contain at least proportion 7 of the parent leaf observations, for some 0 < S 0.5.
Meinshausen (2006)'s 's Lemma 2 shows that for trees following the above construction assumptions, the leaves' sizes converge to 0 as the number of samples in the training set grows (equivalently as the number of total samples grows), within the limits of the support of X. Formally, this means that
max [e]l.Jo0),,,ynnnppp Xx)0
diam((;) +
As we show below, the second part of assumption 2 implies that the total number of leaves relative to the number of samples available in the evaluation set is vanishing (i.e. ((m) within the support of X.
n 0), for leaves
Let be the total number samples of the evaluation set, and let n, denote the number of samples leaf j. l...., J(), Then
j(m)
Σ
n
(m)
min ? 15i5J(n)
n
min ny
J(n)
15j5
Equivalently,
(*)
5
(igs()
Assumption 2 (part 2) is verified, it follows that
((n
0 as - 00.
Therefore, we can apply Györfi et al. (2002)'s Theorem 4.2.
Theorem (Theorem 4.2 of Györfi et al. (2002)). ffr each sphere S centered at the origin diam(L;)
max 1,,
n+++
and
:5:4, 2g 01
1++20
then the partitioning regression function estimate is weakly universally consistent in T-g-
Hence, honest trees constructed based on Meinshausen (2006)'s 's assumptions lead to consistent partitioning estimates.
B ADDITIONAL EXPERIMENTAL DETAILS
B.1 RECALL OF THE BIN-WISE CALIBR ATION RISK
Proposition (Calibration decision risk, Perez-Lebel et al. (2025)). For any confidence score f(z) E 10, the binwise calibration risk writes:
AA | co fxx -
if flof())z # 15222
(f(((())) = 0
(97)
else.
27

---

Published as a conference paper at ICLR 2026
B.2 HONEST TREE CONSTRUCTION ALGORITHM
We describe here the algorithm used to compute our partitioning estimate from an honest tree.
Algorithm 1 Honest tree F
Input:
Probabilistic classifier f, (Xeat, Yeal), (X Ysa), (Xevats Yevat) Fit of using Platt scaling with (X Yeat)
Fit the tree on fit, Yju - coo for L. E L do
We now have the partition L from
Compute f, = Exerval Ex,ec (Y(-eof(Xi))
Eienual Lx;ec
end for
B.3 DISCUSSION ON HYPERPARAMETERS AND IMPLEMENTATION CHOICES
Calibration We use Platt scaling (Platt, 1999) obtain cof. The quality of calibration matters: the underlying model is poorly calibrated, our approach will partly capture miscalibration error. For Figure 3, we allocate at least 4000 samples to the calibration set, or follow the proportions described in 3.3 to limit the variance at low sample sizes due bad calibration c.
Hyperparameters: We adopt conservative strategy in our experiments: we use single fixed set of hyperparameters for the partitioning estimate across all tasks (unlimited depth with a minimum of 15 samples per leaf at training time). Although task-specific tuning could yield stronger results, this setup illustrates the robustness of our chosen hyperparameters. We split the data into calibration (10%), fitting (40%) and evaluation (50%) sets.
Choice of input space The input space which the tree is fitted vary. In standard tabular problems, the natural choice is the original feature space. In contrast, neural networks often operate on less structured data (e.g., images, text) and produce richer internal representations. In such cases, the partition can be built on an embedded representation of the data-fot example, the activations of learned low-dimensional
the last hidden layer before the softmax (Perez-Lebel et al., 2023) embedding such as UMAP (Detommaso et al., 2024).
B.4 SEMI-SYNTHETIC DATA GENERATION.
For fig. 3, we use the Weather dataset from TabReD (Rubachev et al., 2025). We retain the two most important features, identified by permutation importance (Breiman, 2001). To obtain semi-synthetic version (i.e., with access to f*), we fit ground-truth model (X) on half of the test set split. Synthetic class labels then generated for the remaining half by resampling from (X) according to Bernoulli distribution, Y B(ff((X) This yields a semi-synthetic test dataset where X consists of the original features and Y is sampled from f*(X).
28

---

Published as a conference paper at ICLR 2026
Train data
Test data
(X,Y) Fit V
(X,Y) x)
(x, Y)
f*
(X,B(f(())
Semi-synthetic data with known f
f
Evaluate
Figure Diagram of the semi-synthetic dataset construction
B.5
LIST OF LLMS USED IN SECTION 4.2 AND 4.3
All models were gathered from the HuggingFace API.
Table 1: List of models used. Models in bold were used to create the cascades.
Model Name
Size
Company
Type
DeepSeek-PR BoaaSeek-RI-Distl---ae -illaa
8B, 70B
DeepSeek
Non Instruct Instruct & Non Instruct
Gemma 2
9B, 27B
Google
Llama-2
13B, 70B
Meta
Instruct & Non Instruct
Llama-3
8B, 70B
Meta
Instruct & Non Instruct
Ilama-3.1
8B
Meta
Instruct & Non Instruct
Ilama-3.2
1B, 3B
Meta
Instruct & Non Instruct
Llama-3.3
70B
Meta
Instruct
Mistral
7B (v0.3)
Mistral AI
Instruct & Non Instruct
Mixtral
8x7B (v0.1)
Mistral AI
Instruct & Non Instruct
Orea-2
13B
Microsoft
Non Instruct
phi-4
14B
Microsoft
Non Instruct
B.6 DIAGRAM OF OUR CASCADING METHOD
We give a visualization of our cascading method. We consider set of LLMs, ordered from smallest to biggest in size. The cascade starts with the smallest LLM and proceeds follows:
1. Send the question to the LLM and recover its confidence f(X).
2. Verify if the estimated risk R, (X) is smaller than the fixed threshold r.
3. If yes, stop the cascade and use the answer of the model.
4. If not, take the next model in the cascade and go back to step
no model in the cascade has risk estimate below т, we return the answer of the model with the lowest estimated risk.
29

---

Published as a conference paper at ICLR 2026
Query
LLM 1
(X)
(x) +?
-Yes
Out
f_1(x)
No
Out f_2(x)
Query
LLM
fx(X)
Rx(X) S
-Yes
No
Query-
LLM 3
Figure 8: Diagram of our cascading method
C ADDITIONAL EXPERIMENTS
C.1 QUANTIFYING SUBOPTIMALITY
The expected excess decision risk provides a principled criterion for evaluating whether model would benefit from fine-tuning or other forms of post-training.
We replicate the hate speech detection tasks experiments of Perez-Lebel et al. (2025), which aim to evaluate the correlation between estimated excess decision cost and actual decision cost reduction after post-training.
C.1.1 HATE SPEECH EXPERIMENTAL SETUP
The experimental setup is briefly recalled here, see Perez-Lebel et (2025) for details.
Hate speech detection consists in a binary classification with Y=1 for hate speech and = otherwise. The study is conducted across 14 datasets (listed in section C.1.3) and pretrained language models (listed in section C.1.2) which provide confidence scores to be evaluated.
The experiments use the feature embeddings from each model's penultimate layer the input for all post-training methods.
Experimental Procedure:
1. The text from all 14 datasets is processed by each of the pre-trained models. For every sample, the following triplet recorded: (i) the model's predicted probability f(X); (ii) the model's internal feature embedding X; (iii) the ground-truth label Y.
2. The collected data triplets are partitioned into a train set and test set.
3. The post-training method are fitted on the train set.
4. Using only the test set data (prior to any corrections), the excess decision risk estimators are calculated.
5. The post-training corrections learned in the previous step applied to the test set. The actual gain in expected utility is measured by comparing the performance the corrected model against the original, uncorrected model, between different post-training methods.
30

---

Published as a conference paper at ICLR 2026
C.1.2
MODELS USED
Table 2: Detailed description of all the models used for the quantifying suboptimality experiment.
Model
HuggingFace
Latent Layer
CNERG Hatexplain
classifier
CNERG en MURIL
hwweesppeeccCNEEee//aaiiiisiaaaa MuRIL
classifier
CNERG en mono
classifier
CNERG portuguese
classifier
CNERG tamil
classifier
FB Roberta
classifier.out__pn
C.1.3 DATASETS USED
Table 3: Detailed description of all the datasets used for the quantifying suboptimality experiment.
Dataset
Hugging Face or CSV
Split
Input
Target
Pos. class
Tweets
tweets_hate _ates
train
tweet
label
Speech18
hate sseespceeh11
train
text
label
Offensive
hate_speech see__seeech_ffeeiiieee offensive
train tweet
class
0
Davidson
krishan-CSHV...
train+test
text labels
0
Gender
ctoraman/...
train+testt
Text Label
2
FRENK
classla/FRENK -hate-en
train+val+ttst
text
label
limjiayi/hatef..
train+val+ttst
text
label
Check Tweets 2
Paul/hatecheck
test test case
label_ gold
hateful
thefrankhsu/...
train+test tweet label
Open
parnoux/hate_...
test tweet class
0
UCB
ucberkeley-dl...
train text hate hate_speec...
y> 0.5
Merged
tweets_hat. hat..
train tweet label
hate_speech18
train text label
hate_speech_
L_sppeech_offensiee
train
tweet class
0
krishan-CSE/...
train+test
text labels
0
ctoraman/gende...
train+test
Text Label
2
Merged 2
JlassaaFRENNKhhttee
train+val++tst
text
label
limjiayi/hatef...
train+val+test
text
label
Paul/hatecheck
test
test case
label gold
hateful
thefrankhsu/hat...
train+test
tweet
label
parnoux/hat...
test tweet
class
0
DynGen
CSV: bvidgen/Dynamiic..
text label
hate
Merged3
tweets:.
train tweet
label
mate_speech18
train text label
hate sicesspecch___ offensive
train tweet class
0
krishan-CSE/Da...
train+test train+ttest
text labels
O
ctoraman/ge...
Text Label
2
Maasla/FRENK-hatee
train+val+ttst
text label
limjiayi/hateful...
train+val+test
text label
Paul/hatecheck
test test case
label _oodd
hateful
thefrankhsuuhate....
train+test test
tweet label
parnoux/hate..
tweet class
0
C.1.4 RESULTS
Perez-Lebel et (2025) first compares their grouping decision risk estimator to the cost reduction brought by fine-tuning, after recalibration. Based excess decision risk formulation, our RGL estimator writes follows:
31

---

Published as a conference paper at ICLR 2026
Definition 4 (Grouping decision risk estimator). For X X the pointwise grouping risk estimator
writes:
{ RJ (X)(X) - 0
As |cof(X) 1* (X) # (oo(X))2
(98)
else.
We now replicate Perez-Lebel et al. (2025)'s experiments with our estimator instead of theirs. Figure shows the correlation between grouping decision risk estimator R and the observed fine-tuning gain after calibration. We observe a notable increase in squared Spearman rank correlation, from 2 = 0.45 with Perez-Lebel et al. (2025, fig. 5)'s estimator to r 0.61 (fig. 9) with ours. This demonstrates that our estimator predicts better the benefit post-training outperforming the previous estimator, which itself outperformed standard metrics such as accuracy, Brier score, AUC.
Cost reduction (FT vs recal.)
0.00
00.0
-0.0
-0.03
r2=0661
-0.04
00.02
0.00
XAS
RGL
Figure 9: Correlation between fine-tuning gain after re-calibration, and grouping decision risk estimation. The y-axis shows the average cost reduction achieved by fine-tuning recalibrated probabilities. Each point corresponds to a (model, dataset, cost matrix A) triplet. The dashed line corresponds to σ.
Furthermore, fig. 10 correlates the total risk with the cost reduction achieved by fine-tuning the base model. We again observe an increase in squared Spearman rank correlation, from y.2 0.83 in Perez-Lebel et al. (2025)'s figure 6.a to 2 0.88 here.
x
Fine-tuning gain
1.00
0.0
x2
0.88
0.75
-0.1
0.50
0.25
-0.2
0.00
-0.2
0.1
0.0XA
CL f.r
+ R
Figure 10: Correlation between fine-tuning gain and excess decision risk estimation. The y-axis shows the average cost reduction achieved by fine-tuning the base model. Each point corresponds to (model, dataset, cost matrix A) triplet. The dashed line corresponds
32

---

Published as a conference paper at ICLR 2026
C.2 GROUPING LOSS RESULTS ON EVERY INDIVIDUAL DATASET FROM FOLKTEXTS
We give here the grouping loss evaluations for every ACS dataset separately (figs. 11 to 15). We observe that the grouping loss is consistently non-negligible and generally decreases as model size increases, although its magnitude and rate of decay vary across datasets.
For each LLM, GL estimates were averaged over 5 random seeds governing the calibration, fitting, and evaluation splits. We report the mean and standard deviation per model over these 5 splits. The base and instruct trends were computed using LOWES regression.
With Brier score- based estimator, grouping loss of 0.01 corresponds to squared probability error, i.e., an average deviation of 0.1 or less in individual confidence scores (see e.g. group deviations in fig. 1).
Grouping Loss vs Model Size ACSIncome
Base Trend
Mixtral
Grouping Loss
0.15
Instruct Trend
Orca
DeepSeek R1
Gemma
Liama
Liama
Base
Mistral
*
Instruct
0.05
10a
10a
Model Size (B parameters)
Figure 11: Grouping loss function of LLM size, for base and instruct models, ACSIncome.
Grouping Loss vs Model Size ACSPublicCoverage
Grouping Loss
Base Trend
Mixtral
Instruct Trend
Drca
0.10
DeepSeek
Llama
Phi
Llama
Base
0.08
Mistral
Instruct
0.06
100
10a
Model Size (B parameters)
Figure 12: Grouping loss as function of LLM size, for base and instruct models, ACSPublicCover.
age.
Grouping Loss vs Model Size ACSTravelTime
Grouping Loss
*
0.06
Instruct Trend
Orca
DeepSeck R1
Gemma
0.04
Llama Mistral
Base
Instruct
0.02
10a
10
Model Size (B parameters)
Figure 13: Grouping loss function of L.M size, for base and instruct models, ACSTravelTme.
33

---

Published as a conference paper at ICLR 2026
Grouping Loss vs Model Size ACSEmployment
0.25
Base Trend
Mixtral
Grouping Loss
Instruct Trend
Orca
0.20
DeepSeek
Gemma
Llama
Phi
0.15
Liama
Base
Mistral
Instruct
0.10
0.05
109
10a
Model Size (B parameters)
Figure 14: Grouping loss function of LLM size, for base and instruct models, ACSEmployment.
Grouping Loss vs Model Size ACSMobility
Grouping Loss
Base Trend Instruct Trand
Mixtral
Orca
0.050
DeepSeek R1
Gemma
Uama Mistral
Base
0.045
Instruct
0.040
10
10a
Model Size (B parameters)
Figure 15: Grouping loss as function of LLM size, for base and instruct models, ACSMobility,
C.3 LLM ACCURACY ON BACH ACS DATASET
We report in table 4 LLMs' test accuracies (confidence scores thresholded at 0.5 as in the cascading experiment) across ACS datasets. Standard deviations are computed across train/test splits. We note that on ACS TravelTime, all models perform close to chance level (the best constant predictor achieves 56.3% accuracy on this dataset) (Ding et al., 2021)).
Table 4: Model Accuracies across ACS Datasets
Model
ACS Income
Employment
ACS Mobility
ACS PublieCoverage
ACS Travel Time
Llama: 3.2 1B Instruct
0.6307 1 0.0024
0.4566 1. 0.0033
0.3426 1 0.0043
0.5038 1. 0.0062
0.4415 1. 0.0070
IJama-3,2. 3.2ma-3.2.3B--m... Innssett
0.3920
0.3787
Llama 3.1 1-8BB-BBB88 Instruct phi-4
0.6399 1 0.0013
0.7184 1 0.0021
0.7311
1 0.0034
0.7015 1 0.0049
0.5337 1 0.0056
gemmna-22 2-27b-
0.7378 1 0.0023
0.7392
1 0.0017
0.7311
0.0034
0.6262
1 0.0033
0.5295
1 0.0059
Mixtral-8x 1x.B--nsttntttVOO.
Llama-3. 3,3-70B Instruct
0.6756 1. 0.0017
0.7562 1 0.0015
0.6834 1 0.0040
0.7051
1. 0.0032
0.5187
.. 0.0062
C.4 COMPUTING THE RISKS ON AN EMBEDDING OF THE QUERIES
In this section, we present the results the cascade when inputs are treated as free text rather than tabular data. For the baseline, use Hu et al. (2024)'s kNN with the best hyperparameter they found (k: = 40 with cosine similarity and embedding the queries using the all-MiniLM-LL2-1 embedding model). We do apply any dimension reduction and keep the embeddings of dimension 384. The embedded queries serve as input space for our partitioning estimate.
Results are shown in fig. 16. For ACSTravelTmme, all models perform close to random accuracy in the cascade (see table 4). As a result, models are arbitrarily passed through the cascade, which artificially inflates the costs.
34

---

Published as a conference paper at ICLR 2026
Ours vs biggest model
Ours vs Rd only
Ours vs predictive router
0.30
ACChnnome
Gain in accuracy
0.25
0.20
0.10
ACShublsSweergg ACSMob byy
ACSE mp eyment
0.15
0.10
0.05
0.00
-0.05
0.05
Fraction of cost
Fraction of cost
Fraction of cost
Figure 16: Cascades: our risk estimator gives better accuracies at a lower cost. The router and risk estimators are fitted on the embeddings of the prompts given to the LLMs. Each point corresponds cascade (an LLM sequence). The y-axis Acc(ours) Acc(baseline). Values are positive when our cascade improves accuracy over the baseline. The x-axis is Cost(
For values below our model is less expensive. Our cascade based R (r 0) is compared to Left: the biggest LLM in the cascade, Middle: A cascade based RCL only (T 0). Right: the predictive router router baseline (A = 100).
C.5 COMPARING TO CASCADES DEFERRING BASED ON CONFIDENCE SCORES.
We evaluate the classical confidence-bard -basd cascades, mentioned in section 4.3. We start with the smallest LLM,, order them by increasing size, and apply decision threshold 1/2. Based Algorithm in Jitkrittum et al. (2023), the confidence-based cascade proceeds as follows:
1. Send the question to the LLM and recover its confidence fXX).
2. Verify if either f(X) > or 1 - f(X) > c (i.c. if either the confidence for class 0 or 1 exceeds c).
3. If yes, stop the cascade and use the anwser of the model.
4. If not, take the next model in the cascade and go back to step
Figures 17 and show that confidence-basse cascades perform poorly in our experiments. These poor results most probably stem from the high miscalibration and grouping loss of LLMs. In contrast, our method aims at assessing the quality of confidence scores and accepts models when they incur low excess decision error, not when they are confident enough. In fig. 18, our method achieves +7% increase in accuracy on average while using only 52% of the cost on average, and is strictly Pareto-optimal in 90% of the cases
35

---

Published as a conference paper at ICLR 2026
Liama (-0:05
Liama + Liama 8B - Llama 70B
Liama 1B Liama 8B Liama 70B
Tw0.85
9.72
0.70
Llama 708.
85
0.65
fyna
0.70
X=20
Accuracy
0.60
0.68
Liama 708
C-0 tt00.9
0.55
C-0.6
Am1
0.66
(ours)
0.45
0.64 03
12
Liama 80
Predictive Confidence-based
0.40
*
Liama
Lama
0.00
0.25
0.50
0.75
1.00
0.0
1.0 Average Cost per Sample ($)
0.2
0.4
0.6
0.8
Average Cost per Sample ($)
Cost-accuracy tradeoff's when varying the threshold risk r and willingness to pay À and confidence threshold two Llama instruction-tungr family cascades. Adding the confidence- based traditional approach for cascades.
Figure 17:
Ours vs Confidence (c = 0.80)
0.4
Gain in accuracy
0.3
ACSincome
ACS ACSTravelTime
0.2
ACSEmployment
ACSPublicCoverage
ACSMobility
0.1
0.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Fraction of cost
Figure 18: Accuracy gain versus relative cost for our risk-based cascade compared to the traditional confidence-based cascade. Confidence threshold 0.8 and risk threshold set O.
C.6 TREE DEPTH ABLATION STUDY
We investigate the effect of tree depth on the cascade results described in section 4.3. The experimental setup remains identical, with the only change being the max__epppp max__eephhhhhh parameter of the honest tree for the partitioning estimate, which we vary over [3, 6, 9, 12). Overall, our results are very robust to the max depth value (see figs. 19 to 23). The results of the cascade experiments hold up down to max _ee___epthhh = 3, though the accuracy gain becomes more limited. This is expected as very coarse trees reduce the capacity of R. to improve RLL alone.
36

---

Published as a conference paper at ICLR 2026
Llama Llama 3B Llama 8B Llama 70B
Llama 1B
Llama LJama 8B Llama 70B
740
0.72
0.72
Dw0
0m
Accuracy
0.70
X-20
Accuracy
0.70
A=20
0.68
0.68
Llama 708
Liama 708 *
Am1
Am1
0.66
0.66
+ (our)
+ (ours)
Ra
0.64
Lamm 88
0.64
* Lamm 8B
Predictive router
Predictive router
Laaa
Llama
0.62
0.62
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Average Cost per Sample (S)
Average Cost per Sample ($)
(a) max_mmap_pppell 12
(h) max 9
Llama 1B Llama 38 Liama 8B + Llama 70B
Liama 1B - Llama 3B Liama 8B Llama 70B
0.72
0.72
OwO
0,70
0.70
X-
Accuracy
Accuracy
Liama 708 *
Llama 708 *
A=1
0.66
È (ours)
X (ours)
0.64
Laama Liama 18
ga
0.64
*
Liama aB
R
Predictive router
Uaama 18
Predictive router
Llama 3B
Liama 3B
0.62
0.62
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Average Cost per Sample (S)
Average Cost per Sample ($)
(c) max_ 6
(d) max_ max__epph==3
Figure 19: Cost-accuracy tradeoffs when varying the threshold of risk f and willingness to pay À on the Llama 3 instruction-tun tuned family cascade. Changing the max_ _eeph parameter of the honest tree.
Ours vs biggest model
Ours vs Na only
Ours vs predictive router
0.30
0.150
0.05
Gain in accuracy
0.25
0.125
0.04
syment
0.20
0.100
ACSMib ity
0.15
0.075
0.10
0.01
0.05
0.025
0.00
0.000
0.0
Fraction of cost
Fraction of cost
Fraction of cost
Figure 20: Same experimental setup section 4.3, max_ max__epph=111 12 for the honest tree.
37

---

Published as a conference paper at ICLR 2026
Ours vs biggest model
Ours vs Ra only
Ours vs predictive router
0.30
0.150
0.05
ACCrnnome
Gain in accuracy
0.25
ACSToveTTE
0.125
0.04
ACSE mp eyment
0.20
AcShublsGwerrgg
0.100
0.03
ACSMobi by
0.15
0.075
0.02
0.10
0.050
0.01
0.05
0.00
0.00
-0.01
0.05
Fraction of cost
Fraction of cost
Fraction of cost
Figure 21: Same experimental setup as section 4.3, max depth=9 for the honest tree.
Ours vs biggest model
Ours vs a only
Ours vs predictive router
0.30
0.05
0.150
ACSincaee
Gain in accuracy
0.25
0.04
0.125
0.20
0.03
0.100
0.15
0.02
0.10
0.01
0.050
0.05
0.00
0.025
0.00
0.000
-0.01
-0.05
0.02
0.0
0.5
1.0
1.5
1
Fraction of cost
Fraction of cost
Fraction of cost
Figure 22: Same experimental setup as section 4.3, max_ mea__deph==6 for the honest tree.
Ours vs biggest model
Ours vs R only
Ours predictive router
0.30
0.150
0.04
Gain in accuracy
0.25
0.20
0.02
0.15
0.075
0.00
0.10
0.050
ACSincome
0.05
0.025
ACSTovelTime
ACSEmp myment
0.00
0.000
ACSPubiisowerage ACSMo0 Ry
-0.05
-0.04
0.0
0.5
1.0
1.5
Fraction of cost
Fraction of cost
Fraction cost
Figure 23: Same experimental setup as section 4.3, maapmmaa_pppaaae max__epth=3 for the honest tree.
C.7 TRAINING SET SIZE EVALUATION
We vary the number of samples available to train the router and risk estimators (see figs. 24 to 27). For the router, it corresponds to the number of queries labeled with whether or not the LLM was correct. For the calibration risk CL. it corresponds to the number of samples used to calibrate confidence scores. For our epistemic risk estimator R,, it corresponds to the number of samples used to calibrate the confidence scores, create the partition from decision tree and estimate leaf values.
For the ACSTravelTime dataset, all models show around chance accuracy (see table 4).
38

---

Published as a conference paper at ICLR 2026
Ours vs biggest model
Ours vs R. only
Ours vs predictive router
0.30
0.05
0.15
Gain in accuracy
0.25
0.04
ACSEmaloymett
0.20
0.10
0.03
ACSMobilly by
0.15
0.10
0.05
0.02
0.05
0.01
0.00
0.00
0.00
-0.05
-0.05
0.0
0.5
1.0
1.5
1.0
1.5
2.0
2.5
3.0
Fraction of cost
Fraction of cost
Fraction of cost
Figure 24: Varying the number of samples used to train the estimators: Atraii 50.000.
Ours vs biggest model
Ours vs RC only
Ours vs predictive router
0.30
0.05
0.15
ACSiscome
Gain in accuracy
0.25
0.04
ACSEmaloymet
0.20
0.10
0.03
ACSMobil
0.15
0.10
0.05
0.02
0.05
0.01
0.00
0.00
0.00
-0.05
2.0
Fraction of cost
Fraction of cost
Fraction of cost
Figure 25: Varying the number of samples used to train the estimators: Alrriin 10.000.
Ours vs biggest model
Ours vs RCL only
Ours vs predictive router
0.30
0.05
0.15
AC3iscoms
Gain in accuracy
0.25
0.04
ACSEmaloymeti
0.20
0.10
0.03
ACSMobil
0.15
0.10
0.05
0.02
0.05
0.01
0.00
0.00
0.00
-0.05
-0.05
0.0
0.5
1.0
1.5
1.0
1.5
2.0
2.5
3.0
Fraction of cost
Fraction of cost
Fraction of cost
Figure 26: Varying the number of samples used train the estimators: Pttrinn 6.000.
39

---

Published as a conference paper at ICLR 2026
Ours vs biggest model
Ours vs R. only
Ours vs predictive router
0.30
0.85
0.15
Gain in accuracy
0.25
0.04
ACSEmaloymeet
0.20
0.03
AcSMobilly
0.15
0.10
0.05
0.02
0.05
0.01
0.00
0.00
0.00
-0.05
0.0
0.5
1.0
1.5
1.0
15
2.0
2.5
3.0
Fraction of cost
Fraction of cost
Fraction of cost
Figure 27: Varying the number of samples used train the estimators: Berain 2.000.
C.8 USING STRONGER MODELS TO PREDICT INDIVIDUAL RESIDUALS
We present in this section the results of our cascade experiment using Histogram Gradient Boosting Regressor for our residual estimate f. As expected, the accuracy of our approach increases with this method.
Ours vs biggest model
Ours vs º only
Ours vs predictive router
0.08
Gain accuracy
0.25
0.20
0.06
0.15
0.10
0.050
0.05
0.025
0.00
0.000
0.00
-0.05
0.0
0.5
1.0
1.5
Fraction of cost
Fraction of cost
Fraction of cost
Figure 28: Using stronger model for the residual estimate é yields higher accuracy gains. We reproduce the experimental setting of 4.3 using HiscGradienBoostingggggss estimate the residuals f(X).
40


Reviews:
[{"review_id": "Po5cBVsL1U", "reviewer": "Reviewer_xMCH", "summary": "The authors propose estimators of “grouping loss” and “excess risk”, and apply them to three decision problems: predicting the benefit of LLM finetuning, assessing LLM confidence, and routing queries between multiple LLMs.", "questions": "Can you provide a technical argument for the insufficiency of proper scoring rules? Note that any appeal to the idea of epistemic/reducible uncertainty would have to be rigorously derived, not just stated as important.\n\nSection 4.2:\n\n- Can you clarify the cost matrices you used?\n- Can you explain the axis magnitudes in Figure 3?\n- Why not just evaluate the “cost” on the training set, if this is what we care about?\n\nSection 4.3: what is the concrete task or notion of success you have in mind?\n\nSection 4.4: can you extend the lines in Figure 5 to include $t=0$?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "Originality: the proposed estimators appear to be novel.\n\nQuality: Figures 2, 4 and 6 communicate the authors’ results well.\n\nClarity: a good amount of the low-level text is easy to follow.\n\nSignificance: query routing is a potentially impactful application.", "weaknesses": "I believe the technical motivation of this work is unsound.\n\n- First, describing an aleatoric-epistemic uncertainty decomposition as “crucial for reliable AI” and using that as the premise of the rest of the paper is not appropriate. The aleatoric-epistemic view has been shown to be incoherent, and subjective model-based uncertainties alone are not a legitimate basis for assessing model reliability (Bickford Smith et al, 2025).\n- Second, dismissing standard evaluation metrics—particularly proper scoring rules, which are very well established and have strong theoretical foundations (Savage, 1971)—in favour of new quantities ought to be extremely well argued from a technical perspective, yet it isn’t. Instead the authors offer an assortment of statements about various metrics, then quickly land on “grouping loss” and “excess risk” as the quantities of apparent interest.\n\nEven if we were to set this aside, the practical contribution is hard to get behind.\n\n- In Section 4.2 we see results for predicting the benefit of LLM finetuning. The practical significance of these is unclear. We see a scatter plot of the “excess risk” estimator against “cost reduction”, but the magnitudes of these quantities are hard to interpret from the plot. This is a nonstandard setup that needs explaining better or supplementing with a more practically relatable setup. It also appears a basic baseline method is missing: why not just evaluate the “cost” on the training set, if this is what we care about?\n- In Section 4.3 there isn’t a well-defined task or notion of success. Instead the authors just report confidence results (confidence is not a quantity they are proposing) in Figure 1 and “grouping loss” results in Figure 4. What are we to do with these?\n- In Section 4.4 the focus is on routing a query to an appropriate LLM, balancing prediction costs against predictive accuracy. The results in Figure 5 appear to be positive, although the choice to stop at $t=0.001$ gives a false impression of how expensive the proposed method can be. This high cost is clear from Figure 6 (right): even with the baseline method in a high-cost configuration ($\\lambda=100$), the proposed method is up to ~60 percent more expensive, and the payoff in terms of accuracy is small, at less than ~3 percentage points. At a higher level it seems the extent to which the results are positive here is highly contingent on the dataset having many examples that can be accurately classified by the smaller models.\n\n---\n\nBickford Smith et al (2025). Rethinking aleatoric and epistemic uncertainty. ICML.\n\nSavage (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W1bMEeDAKm", "reviewer": "Reviewer_6jMk", "summary": "The paper propose lower-bound estimators for grouping loss and excess risk, designed to address deficiencies in standard metrics (AUC, accuracy, calibration error) that conflate model confidence without properly distinguishing epistemic error or bias. Experiments cover semi-synthetic, tabular, and LLM question answering tasks, including cost-sensitive settings and cascades.", "questions": "- What are the practical limits for inference-time overhead with honest tree partitioning?\n\n- How does the proposed approach adapt to free text tasks?\n\n- How to select tree hyperparameter for new tasks or domains?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper is generally well written. Extensive notational and methodological setup supports reproducibility and rigour.\n\n- It advances epistemic uncertainty estimation using consistent, efficient lower-bound estimators,\n\n- It addresses a critical gap for high-stakes AI applications.", "weaknesses": "- It does have dense mathematical sections which might make it harder to digest. Intuitive diagrams could help.\n\n- Computational considerations for large-scale or real-time applications are missing. Scalability and latency constraints need more discussion.\n\n- An ablation on tree depth is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ugKq4tOdZt", "reviewer": "Reviewer_SEdh", "summary": "This paper introduces asymptotically consistent and sample-efficient lower-bound estimators for the grouping loss and excess risk, i.e., suboptimality of a prediction, serving as a complement to existing calibration metrics. The authors leverage the proposed estimator to design efficient LLM cascades that defer to stronger models, achieving higher accuracy at a lower cost than competing approaches.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work paper is well structured, balancing the theoretical analysis and experimental validations.\n\n2. Proofs for the related propositions are given in the appendix.\n\n3. The extensive evaluations demonstrated the performance improvements convincingly.", "weaknesses": "1. Would the author be more explicit in indicating what the resources of the epistemic uncertainty are considered in the proposed method?\n\n2. Minor: There is a lack of a REPRODUCIBILITY STATEMENT in the main body.\n\n3. Minor: In the appendix, there is an unnecessary comment in green remaining. line 699.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u5iXHMAJte", "reviewer": "Reviewer_wB4E", "summary": "This paper considers the excess risk, focussing on the grouping loss, of predictors. In particular, it proposes an estimator for these quantities in the context of proper scoring rules. The estimators are shown to have desirable properties such as consistency and unbiasedness, and instantiations with the 0-1 loss are considered. The method is evaluated on a semi-synthetic setting with known ground-truth probabilities and on a range of experiments involving LLMs and real data.", "questions": "-", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper covers a valuable topic and does a good job of explaining exactly why considering the epistemic uncertainty quantification and, specifically, grouping loss, is important by including illustrative examples.\n- The provided solution is reasonable, satisfies desirable properties, and performs well empirically.\n- The paper reads well, and has clear notation and exposition for the formal results.\n- There is a good range of empirical settings.", "weaknesses": "- The paper focuses on the binary setting, which is, of course, fairly limited.\n\nThere seems to be a leftover comment in the Appendix p. 13, l. 697.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)