# Epistemic Uncertainty Quantification to Improve Decisions From Black-Box Models

Sébastien Melo, Gaël Varoquaux, Marine Le Morvan

<figure><img src="image_1.png" /></figure>

To cite this version:

Sébastien Melo, Gaël Varoquaux, Marine Le Morvan. Epistemic Uncertainty Quantification to Improve Decisions From Black-Box Models. 2025. hal-05393027

HAL Id: hal-05393027

https://hal.science/hal-05393027v1

Preprint submitted on 2 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# EPISTEMIC UNCERTAINTY QUANTIFICATION TO IMPROVE DECISIONS FROM BLACK-BOX MODELS

**Sébastien Melo**
Inria, Soda

**Gaël Varoquaux**
Inria, Soda

**Marine Le Morvan**
Inria, Soda

## ABSTRACT

Distinguishing a model's lack of knowledge (epistemic uncertainty) from inherent task randomness (aleatoric uncertainty) is crucial for reliable AI. However, standard evaluation metrics of confidence scores target different aspects. AUC and accuracy capture predictive signal, proper scoring rules capture overall uncertainty, and calibration metrics isolate part of the epistemic uncertainty but ignore heterogeneity of the errors within bins, known as grouping loss. We close this evaluation gap by introducing asymptotically consistent and sample-efficient lower-bound estimators for the grouping loss and excess risk, i.e. suboptimality of a prediction. Our estimators complement existing calibration metrics to provide a more complete, fine-grained assessment of epistemic uncertainty. Applied to LLM question-answering with inherent aleatoric noise, our estimator reveals substantial grouping loss which decreases with model scale but is amplified by instruction tuning. The local nature of our estimators provides actionable insights: they automatically identify subgroups with systematic over- or under-confidence for interpretable audits. We also show that it reveals better the need of post-training. Finally, we leverage our estimator for efficient LLM cascades that defer to stronger models, achieving higher accuracy at a lower cost than competing approaches.

## 1 INTRODUCTION

The recent European Artificial Intelligence (AI) Act requires careful oversight in some AI applications (EUC, 2025). In high-risk settings, predictive uncertainty behind a model's classification choice is central to proper risk assessment and flexible deployment (e.g. adapting to changing regulation on when the risk is considered sufficiently low). For example, some hospitals use models to estimate a patient's likelihood of surgical success based on their medical records (Mahajan et al., 2023), but these use cases call for thorough evaluation (Senge et al., 2014).

In many high-stakes settings, the best answer is not clear-cut. For instance, in the case of surgery prognosis, many unforeseen events can alter the outcome. A question such as “Will this patient relapse within the next 6 months?” must be answered with a confidence score that accounts for the amount of information available on the patient, unlike a question such as “Was Einstein born before 1800?”, which admits a deterministic answer. The corresponding challenge of aleatoric uncertainty is often overlooked when evaluating LLMs confidence scores (Xia et al., 2025), which is at odds with their usage, e.g. in health settings (Jiang et al., 2023; Singhal et al., 2023; Omar et al., 2025; Katz et al., 2024).

Confidence scores may arise naturally from the continuous function that powers many classifiers, such as the simple logistic regression, or neural network with softmax output activation. But other settings require confidence score elicitation, as with LLMs. Indeed, LLMs output natural language answers which require dedicated techniques to be associated to a confidence (Shorinwa et al., 2025; Vashurin et al., 2025). Our work does not focus on this confidence scores elicitation, but rather on their evaluation. Evaluation and elicitation are often conflated, as an evaluation gives a signal that can often be used to improve confidence scores in return (e.g. measuring calibration error is an evaluation, but recalibration is an improvement technique). And yet, to decide whether to trust an AI system's answer, or whether to deploy it, in particular in a high-stake settings, the question of how good is its confidence score is crucial.
But how to measure the “optimality” of confidence scores? Confidence scores should capture the probability that the predicted outcome does occur. Standard metrics such as accuracy or AUC fail to capture this probabilistic aspect, as they cannot detect over- or under-confidence. Calibration (Gneiting et al., 2007) addresses this by comparing confidence scores to the rate of observed outcome. However, it is a control on average: a classifier may be well-calibrated overall yet overconfident on some subgroups and underconfident on others, with the two effects canceling out (Perez-Lebel et al., 2023). An optimal confidence score should reflect the true probability of the outcome depending on characteristics of the input X, not just match observed frequencies on average. For an input with insufficient information (e.g. not enough details in an LLM query), even the best predictor comes with a lot of uncertainty, called “aleatoric uncertainty”. It is then a virtue of an AI system to report it (Kendall & Gal, 2017), as predictions are bound to come with errors and reporting this uncertainty can lead to better decisions. On the other hand, the errors of an AI system may arise from imperfections in this system, not using well the available information. Such uncertainty is known as epistemic uncertainty. A well-designed confidence score conveys both: expressing uncertainty when information is insufficient and indicating where the model could, in principle, improve.

Measuring the epistemic uncertainty, i.e. whether a model is knowledgeable enough, is highly desirable because it indicates where a model can be improved, either globally or locally (Wimmer et al., 2023). Global improvements might involve fine-tuning with additional data, while local improvements could include deferring decisions to a human expert or a more specialized, but costlier, model. Such deferral is inefficient if the uncertainty is aleatoric, since in that case the prediction is already optimal given the available input. Teasing out epistemic from aleatoric uncertainty is a very difficult task and arguably the holy grail of learning theory. Some metrics –“proper scoring rules” (Gneiting & Raftery, 2007), such as Brier score or log loss– can select models with smallest epistemic error; but they do not quantify the remaining epistemic error. For a calibrated model, residual epistemic error arises from heterogeneity in the error: the model may be overconfident on some inputs and underconfident on others (see Figure 1), a phenomena captured by the grouping loss (Kull & Flach, 2015; Perez-Lebel et al., 2023).

We tackle epistemic error quantification, to characterize how suboptimal a model is. We contribute:

* **Sample-Efficient Estimators for Epistemic Uncertainty:** We introduce asymptotically consistent estimators for grouping loss and excess risk that offer significant gains in sample efficiency by bypassing the per-bin computations common in prior work. This improved efficiency yields tighter and more reliable lower-bound estimates, which, when combined with calibration error, provide a more precise measure of a model's epistemic uncertainty.

* **Actionable and Interpretable Insights:** Our approach provides local, per-sample uncertainty estimates. This facilitates model auditing by identifying specific regions of under- or over-confidence (see Figure 1).

* **Uncovering Bias in LLMs:** We demonstrate that LLMs exhibit significant grouping loss on question-answering tasks, revealing a source of systematic bias in their predictions.

* **Practical Applications for Model Improvement:** We showcase the utility of our enhanced uncertainty evaluation through two key applications: 1) more accurately determining if a model will benefit from post-training, and 2) designing more cost-effective and accurate cascading systems of LLMs for decision deferral.

<figure><img src="image_3.png" /><figcaption>Figure 1: **Over and under-confident groups for Llama 3.3 Instruction-tuned 70B** on the folktexts dataset. Groups were obtained by applying our estimator to the confidence scores provided by the LLM. Exact group definitions obtained from the tree cuts are: Group 37 (green): School level: Master's degree and higher, Age ≤ 24, Sex = Female. Group 62 (yellow): School level: Professional degree beyond a bachelor's degree and higher, Age ≥ 37, Sex = Female.</figcaption></figure>
## 2 BACKGROUND AND RELATED WORK

**Notations** We consider a binary classification task where $(X, Y) \in \mathcal{X} \times \{0, 1\}$ are jointly distributed random variables representing the input space and class labels respectively. We write $f^*(X) \stackrel{\text{def}}{=} \mathbb{P}[Y = 1 | X]$ the unknown posterior class probabilities, and $f: \mathcal{X} \to [0, 1]$ the confidence scores.

### 2.1 CALIBRATION

Calibration (Lichtenstein et al., 1977) measures whether the confidence scores produced by a model $f: \mathcal{X} \to [0, 1]$ match, on average, the observed frequency of positive outcomes.

**Definition 1 (Calibration).** A probabilistic classifier $f$ is calibrated if

$$ \forall p \in [0, 1], \quad \mathbb{E}[Y | f(X) = p] = p. $$

We note $c \circ f(X) \stackrel{\text{def}}{=} \mathbb{E}[Y | f(X)]$ the calibrated score (c is a recalibration function). The most widely used calibration metric is the Expected Calibration Error (ECE) (Naeini et al., 2015), which measures the difference between predicted confidence and observed frequency by averaging absolute differences across probability bins. Cruz et al. (2024) is among the few studies to quantify LLM uncertainty on tasks with inherent aleatoric uncertainty using the ECE. They found that base LLMs tend to be underconfident, while instruction-tuned models often become overconfident.

### 2.2 PROPER SCORING RULES AND GROUPING LOSS

Proper scoring rules evaluate a predicted probability distribution $p$ against an observed outcome $Y$ (Dawid, 1986; Gneiting & Raftery, 2007). They can be decomposed to separate epistemic uncertainty (due to the model's lack of knowledge) and aleatoric uncertainty (due to inherent data randomness), while also isolating a calibration component.

**Theorem 1 (Kull & Flach (2015) Proper scoring rule decomposition).** Let $\phi$ be a proper scoring rule and $d_\phi$ its divergence. Then the expected loss writes as:

$$ \underbrace{\mathbb{E}[d_{\phi}(f(X), Y)]}_{\text{Expected Loss}} = \underbrace{\mathbb{E}[d_{\phi}(f(X), c \circ f(X))]}_{\text{Calibration Loss (CL)}} + \underbrace{\mathbb{E}[d_{\phi}(c \circ f(X), f^*(X))]}_{\text{Grouping Loss (GL)}} + \underbrace{\mathbb{E}[d_{\phi}(f^*(X), Y)]}_{\text{Aleatoric Loss}}. \quad (1) $$

The above shows that the epistemic loss can be further decomposed into calibration and grouping components. When $\phi$ is the Brier score, the grouping loss corresponds to a variance term.

**Definition 2 (Grouping Loss (Lemma 4.1 Perez-Lebel et al. (2023)))**. If the proper scoring rule $\phi$ is the Brier score, the grouping loss is given by:

$$ \mathrm{GL} \stackrel{\text{def}}{=} \mathbb{E}[(f^*(X) - c \circ f(X))^2] = \mathbb{E}_p[\mathbb{V}[f^*(X) | f(X) = p]]. \quad (2) $$

The grouping loss measures the variance of true probabilities within a level set of the model $f$, reflecting that the model groups together inputs with different true probabilities. Modern neural networks were shown to exhibit grouping loss (Perez-Lebel et al., 2023; Chen et al., 2024), especially under distribution shift. In the remainder of this paper, we use the Brier score as proper scoring rule.

### 2.3 EXCESS 0-1 RISK

A key distinction lies between assessing the discrepancy between predicted confidence scores and the true conditional probabilities, and evaluating the decision risk induced by applying a rule derived from those scores. The former reflects the suboptimality of the probabilistic classifier, as measured by the expected loss under a proper scoring rule, whereas the latter measures the operational suboptimality of the decisions it produces. Confidence scores are often used to make decisions, i.e., mapping a sample $X$ with confidence $f(X)$ to an output $Y \in \{0, 1\}$. From a cost matrix $\Lambda \in \mathbb{R}^{2 \times 2}$ where $\Lambda_{i,j}$
is the cost of predicting class i when the true class is $Y = j$, decision theory (Elkan, 2001) gives the optimal decision rule from $f^*$:

$$ \delta^* \stackrel{\text{def}}{=} x \mapsto \mathbf{1}_{f^*(x) \ge t^*} \quad \text{with} \quad t^* \stackrel{\text{def}}{=} \frac{\Lambda_{1,0} - \Lambda_{0,0}}{\Lambda_{\Delta}}; \quad \Lambda_{\Delta} \stackrel{\text{def}}{=} \Lambda_{1,0} + \Lambda_{0,1} - \Lambda_{0,0} - \Lambda_{1,1}. $$

$t^*$ is the optimal threshold that depends on misclassification costs. Given a model $f$ and a selected decision threshold $t$, we consider the decision rule $\delta_{f,t} \stackrel{\text{def}}{=} (x \mapsto \mathbf{1}_{f(x) \ge t})$ and the associated expected cost of the decision $EC(\delta_{f,t}, x) \stackrel{\text{def}}{=} E[\Lambda_{\delta_{f,t}(X),Y} | X = x]$. Suboptimal decision scores lead to suboptimal decisions, resulting in excess risk. In particular, Perez-Lebel et al. (2025) showed that the excess cost can be decomposed into two components: one arising from calibration errors in the confidence scores, and one due to grouping errors.

**Proposition 1** (Excess 0-1 risk decomposition (Perez-Lebel et al., 2025)). Let $\Lambda \in \mathbb{R}^{2 \times 2}$ be a cost matrix. The excess 0-1 risk $\mathcal{R}_f(X)$ can be decomposed as:

$$ \mathcal{R}_f(x) = \underbrace{EC(\delta_{f,t}, x) - EC(\delta_{\text{cof},t^*}, x)}_{\mathcal{R}_f^{CL}(x)} + \underbrace{EC(\delta_{\text{cof},t^*}, x) - EC(\delta_{f^*,t^*}, x)}_{\mathcal{R}_f^{GL}(x)} \quad (3) $$

where $\mathcal{R}_f^{CL}(x)$ and $\mathcal{R}_f^{GL}(x)$ are the excess individual errors from miscalibration and grouping loss respectively.

Perez-Lebel et al. (2025, prop 3.2) proposed a calibration risk estimate $\mathcal{R}_f^{CL}(X)$, but gave the grouping risk per level set, rather than given an input $X$. Furthermore, this grouping risk estimate was built by averaging lower and upper bounds of $\mathcal{R}_f^{GL}$ per level set.

## 2.4 RELATED WORK ON QUANTIFYING BLACK-BOX EPISTEMIC ERROR

**Grouping Loss Estimation** Other works tackle grouping-loss estimation: Johnson et al. (2024) introduces a training paradigm where a model learns to predict pairs of outputs, using a "cheating" mechanism to directly estimate its epistemic uncertainty. However, this approach requires access to two outcomes for the same input. Mimori et al. (2021) propose an epistemic loss estimator based on a local average of $f^*$, which can be biased, but also assumes multiple labels per instance. Finally, Perez-Lebel et al. (2023) proposes a lower-bound estimation of the grouping loss using local averages within bins to estimate $f^*$ and its variance.

**Direct Epistemic Uncertainty Estimation** An alternative to breaking down uncertainty into calibration and grouping loss is to estimate it directly. Bayesian-inspired methods like MC-Dropout and Deep Ensembles are common techniques for this purpose (Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017). However, a key criticism is that they primarily capture approximation uncertainty (due to a finite dataset) while often failing to account for model uncertainty (stemming from a misspecified model class). Both are critical components of total epistemic uncertainty (Hüllermeier & Waegeman, 2021; Lahlou et al., 2023). Bickford Smith et al. (2025) criticizes the many existing definitions of epistemic uncertainty which are based on the learned distribution $f$ and advocates for the use of externally grounded epistemic evaluation (i.e. with respect to an external reference evaluation function or ground truth realizations of the output).

**Decision Deferral** Decision deferral studies how strategies for a model to abstain when uncertain, improving reliability in high-stakes domains (Hendrickx et al., 2024). Simple approaches defer based on a confidence threshold, but these assume the model's confidence scores are reliable (Wang et al., 2022; Jitkrittum et al., 2023). More sophisticated methods "learn to defer" by training a dedicated rejection function alongside the classifier (Mozannar & Sontag, 2021; Mozannar et al., 2023), or "route", choosing among a set of models, the one best suited to answer a specific question (Shnitzer et al., 2023). Within the LLM space, some works propose cascading models based on cost but often lack a principled function to guide the deferral decision (Hu et al., 2024).

## 3 LOWER-BOUNDED EPISTEMIC UNCERTAINTY

Estimating the grouping loss and grouping 0-1 risk in theory requires knowledge of the true conditional distribution $f^*(X) = P[Y = 1 | X]$. In practice, however, we only observe discrete outcomes $Y$.
Without assumptions on the underlying distributions, a natural approach is to aggregate outcomes to form local average estimates. We adopt this strategy, using a carefully chosen partition to build our estimators. Sections 3.1 and 3.2 assume a fixed partition and introduce estimators for the grouping loss and grouping 0-1 risk, respectively. Section 3.3 discusses practical construction of the partition.

**Notation** Let $\mathcal{L}^{(n)} \stackrel{\text{def}}{=} \{\mathcal{L}_j^{(n)} : j \in [1, J^{(n)}]\}$ be a partition of the input space. For each $j$, let $n_j$ (resp. $\hat{p}_j$) be the number (resp. fraction) of samples belonging to region $j$. For any $X$, we denote by $\mathcal{L}^{(n)}(X)$ the region that contains $X$.

**Definition 3 (Partitioning estimate).** Let $(X_i, R_i)_{i=1}^n$ be $n$ sample points in $\mathcal{X} \times \mathbb{R}$. The partitioning estimate is defined as:

$$ \hat{r}^{(n)}(X) = \sum_{j=1}^{J^{(n)}} \hat{r}_{j}^{(n)} \mathbf{1}_{X \in \mathcal{L}_{j}^{(n)}} \quad \text{where} \quad \hat{r}_{j}^{(n)} = \frac{1}{n_j} \sum_{i \in \mathcal{L}_{j}^{(n)}} R_i \quad (4) $$

The superscript (n) indicates dependence on a dataset of size n. To keep notations lightweight, it will be omitted whenever appropriate.

## 3.1 GROUPING LOSS ESTIMATOR

According to eq. (2), the grouping loss corresponds to the variance of probabilities $f^*(X)$ within bins of same probability $p$. The grouping loss estimator of Perez-Lebel et al. (2023) directly mirrors this definition by binning confidence scores and computing per-bin variances. However, binning comes with several drawbacks. Bin boundaries prevent information from being shared across bins, and level sets corresponding to a bin may form disconnected subsets in feature space $\mathcal{X}$. Both effects may hurt sample efficiency. Deriving binning-free estimators requires departing from the grouping loss expression given in eq. (2). For this purpose, we use the following lemma:

**Lemma 1 (Grouping loss lower bound).** Given a partition $\mathcal{L}$, the grouping loss is lower-bounded as:

$$ \mathrm{GL} = \mathbb{E}[(f^*(X) - c \circ f(X)^2)] \geq \mathrm{E}_j[\mathrm{E}_{(X,Y)}[Y - c \circ f(X) | X \in \mathcal{L}_j]^2] \stackrel{\text{def}}{=} \mathrm{GL}_{lb}(\mathcal{L}) $$

*Proof.* The result is a direct application of Jensen's inequality. □

$\mathrm{GL}_{lb}(\mathcal{L})$ represents the fraction of the grouping loss that is captured by the partition $\mathcal{L}$. It is defined as the expected squared difference between the outcome and the calibrated score within each region $\mathcal{L}_j$. The tightness of this lower bound depends on how homogeneous the regions are: the smaller the variability of $f^*(X) - c \circ f(X)$ within a region, the tighter the lower bound, with equality when $f^* - c \circ f$ is constant over each region. In practice, a good partition should group together points with similar under- or over-confidence. Note that such regions may span multiple confidence-score bins, as similar over/under-confidence levels can occur at different confidence levels $p$.

**Proposition 2 (Debiased grouping loss estimator).** Given a partition $\mathcal{L}^{(n)}$ and a dataset $\mathcal{D}^{(n)} = \{(X_i, Y_i)\}_{i=1..n}$, let $\hat{r}_j$ (resp. $\hat{v}_j$) be the sample mean (resp. sample variance) of $Y - c \circ f(X)$ in region $j$. A debiased estimator of $\mathrm{GL}_{lb}(\mathcal{L}^{(n)})$ is:

$$ \widehat{\mathrm{GL}}_{lb}^{(n)} \stackrel{\text{def}}{=} \sum_{j=1}^{J} \hat{p}_j \left( \hat{r}_j^2 - \frac{1}{n_j} \hat{v}_j \right) \quad (5) $$

The proof can be found in Appendix C.1. The terms involving $\hat{r}_j^2$ are plugin estimators for $\mathrm{GL}_{lb}(\mathcal{L})$, while the terms involving $\hat{v}_j$ provide a bias correction. Importantly, given a partition, the quantities $\hat{r}_j^2$, $\hat{v}_j$ and $\hat{p}_j$ can easily be computed to obtain an estimator of $\mathrm{GL}_{lb}$.

For a given partition, the proposed estimator is a lower bound of the total grouping loss, but for a well-suited sequence of partitions, it asymptotically converges to the true total grouping loss GL:
**Proposition 3 (Consistency of the grouping loss estimator).** Let $\hat{r}^{(n)}$ be a partitioning estimate applied to $R = Y - c \circ f(X)$. Assume that $\hat{r}^{(n)}$ is consistent. Then the sequence of estimators $\hat{\text{GL}}_{lb}^{(n)}$ is weakly universally consistent, i.e.,

$$ \mathbb{E}_{D^{(n)}} \left[ \left| GL - \hat{\text{GL}}_{lb}^{(n)} \right| \right] \xrightarrow{n \to +\infty} 0. $$

Our estimator is consistent, though with a downward bias: Appendix C.2 shows it converges in probability to the true value from below as the sample size increases (see Figure 2). This result relies on the theory of partitioning-based estimation. Györfi et al. (2002)'s Theorem 4.1 and 4.2 ensure consistency given two conditions: (i) the maximum diameter of the partition regions vanishes, and (ii) the number of samples per region grows with the total sample size. Therefore, summed with the calibration error, our grouping loss estimator yields a lower bound of the epistemic loss (Theorem 1).

## 3.2 EXCESS RISK LOWER BOUND ESTIMATOR

The grouping 0-1 risk, denoted $\mathcal{R}_f^{GL}$ in eq. (3) measures the loss in accuracy of model $f$ after calibration relative to the optimal (but unknown) model $f^*$. As with GL estimation, a partitioning of the residuals can be used to obtain a lower bound on $\mathcal{R}_f^{GL}$.

**Definition 4 (Grouping 0-1 risk estimator).** Let $\hat{r}(X)$ be a partitioning estimate applied to $R = Y - c \circ f(X)$, and let $t^*$ and $\Lambda_\Delta$ as defined in Section 2.3. Define:

$$ \hat{\mathcal{R}}_f^{\text{GL}}(X) \stackrel{\text{def}}{=} \begin{cases} \Lambda_\Delta | c \circ f(X) + \hat{r}(X) - t^* | & \text{if } \mathbf{1}_{c \circ f(X)} + \hat{r}(X) \ge t^* \ne \mathbf{1}_{c \circ f(X)} \ge t^* \\ 0 & \text{else.} \end{cases} \quad (6) $$

Figure 7 gives a visual diagram of our grouping risk estimator. $\hat{r}$ acts as the local correction between the calibrated scores and the true conditional distribution captured by the partition. If this adjustment between $c \circ f$ and $f^*$ is true on average over each region $\mathcal{L}_j$, our estimator matches the excess 0-1 risk of the calibrated classifier captured by the partition and is a lower bound of the true grouping risk.

**Proposition 4.** If the true distribution of $f^*(X) | \mathcal{L}_j, f(X)$ is of mean $c \circ f(X) + \hat{r}_j$, then $\hat{\mathcal{R}}_f^{\text{GL}}(X)$ represents the excess 0-1 risk of the calibrated classifier $c \circ f$ captured by the partition, i.e., for $x \in \mathcal{L}_j$,

$$ \hat{\mathcal{R}}_f^{\text{GL}}(x) = \mathbb{E}[EC(\delta_{c\circ f, t^*}, X) - EC(\delta_{c\circ f+\hat{r}, t^*}, X) | X \in \mathcal{L}_j, f(X) = f(x)]. $$

Moreover, under the above assumptions, $\hat{\mathcal{R}}_f^{\text{GL}}(X)$ is a lower-bound on the true grouping risk, i.e.,

$$ \hat{\mathcal{R}}_f^{\text{GL}}(X) \le \mathcal{R}_f^{\text{GL}}(X). $$

The proof can be found in Appendix C.3. The result in expectation over $X$ directly follows (Appendix B). The use of partitioning estimates thus gives a guarantee on the risk estimation that we evaluate. As in the previous section, this lower bound estimator converges to the true value of the regret if it is computed from a sequence of consistent partitioning estimates.

<figure><img src="image_4.png" /><figcaption>Figure 7: Summary diagram of our grouping risk estimator. For a model $f$, we plot the calibrated scores $c$ and the grouping loss of the true probabilities around the calibration scores captured by $\hat{r}$. The excess 0-1 grouping risk stems from the disagreement in the decision from $c$ and the one from $c + \hat{r}$ (in orange). The further away the disagreement is from the correct decision quadrant (in white) the higher the grouping risk $\hat{\mathcal{R}}_f^{\text{GL}}$ is (as denoted by the red gradient).</figcaption></figure>
**Proposition 5** (Consistency of the grouping regret estimator). Let $\hat{r}^{(n)}$ be a partitioning estimate applied to $R = Y - c \circ f(X)$. Assume that $\hat{r}^{(n)}$ is consistent. Then the sequence of estimators $\hat{R}_{f}^{\text{GL}(n)}$ is weakly universally consistent, i.e.,

$$ \mathbb{E}_{D^{(n)}} [ | \hat{R}_{f}^{\text{GL}(n)} - \mathcal{R}_{f}^{\text{GL}} | ] \xrightarrow{n \to +\infty} 0. $$

The proof is in Appendix C.4.

## 3.3 ESTIMATORS IN PRACTICE

**Partitioning estimate** - Our estimators $\hat{GL}_{lb}$ and $\hat{R}_{f}^{\text{GL}}$ both rely on a partitioning estimate $\hat{r}$. To obtain $\hat{r}$ we use honest regression trees (Wager & Athey, 2017). They mitigate bias in leaf estimates by ensuring that each sample is used either to fit the tree structure or to compute the leaf-level estimates, but never both. Concretely, we fit a regression tree on the model residuals after calibration, i.e., $Y - c \circ f(X)$, using one subset of the data. The resulting leaves define the partition. The estimates within each leaf are then computed on a disjoint subset. Algorithm 1 details this procedure. Under some assumptions (Appendix C.5), we show that honest trees are a consistent partitioning estimate.

**Implementation aspects** - For probability calibration, we use Platt scaling (Platt, 1999). The regressor uses the same hyperparameters across all experiments to ensure robustness: decision trees with unlimited depth and a minimum of 15 samples per leaf at training time. Appendix F.4 further develops an ablation study of the influence of the tree's depth. Datasets are divided into a calibration set (10%), a fitting set (40%), and an evaluation set (50%). The choice of input space for the tree is determined by the data type. For standard tabular data, the original feature space is used. In the case of neural networks handling unstructured data like images or text, the partition is constructed from the model's internal representations, such as the activations from the last hidden layer (Perez-Lebel et al., 2023) or a learned low-dimensional UMAP embedding (Detommaso et al., 2024). Appendix A.2 further details our proposed implementation.

# 4 EXPERIMENTS

## 4.1 GROUPING LOSS ESTIMATOR VALIDATION ON SEMI-SYNTHETIC DATA.

We first evaluate our lower bound on the grouping loss on a semi-synthetic dataset with ground-truth knowledge of $f^*(X)$. We use the Weather dataset from TabReD (Rubachev et al., 2025), which contains more than 16 million samples and exhibits temporal drift—a favorable setting to study grouping loss and consistency. Appendix D.1 details data construction.

Confidence scores $f$ are obtained from a Histogram Gradient Boosted tree trained on the training split. These scores are then evaluated using our estimator $\hat{GL}_{lb}$ as well as the method

<figure><img src="image_5.png" /><figcaption>Figure 2: Sample efficiency and asymptotic consistency of $\hat{GL}_{lb}^{(n)}$ on a semi-synthetic experiment.</figcaption></figure>

proposed in Perez-Lebel et al. (2023). The results (fig. 2) confirm that $\hat{GL}_{lb}$ indeed provides a lower bound on the ground-truth grouping loss in expectation. Furthermore, we observe its asymptotic consistency: as the number of samples increases, the estimator converges to the true grouping loss. Finally, our estimator demonstrates greater sample efficiency compared to Perez-Lebel et al. (2023), capturing more grouping loss for a given sample size. This highlights that a binning-free estimator can leverage available data more effectively to capture the remaining heterogeneity in confidence scores.
<figure><img src="image_6.png" /><figcaption>Figure 4: **Grouping loss as a function of LLM scale, for base and instruct models.** Each GL estimation was computed on 5 seeds for the data splits described in section 3.3.</figcaption></figure>

## 4.2 QUANTIFYING SUBOPTIMALITY

The excess 0-1 risk provides a principled criterion on whether a model requires fine-tuning or other forms of post-training, for example when the data distribution shifts over time. Our estimator $\hat{\mathcal{R}}^{GL}$ can be used to detect such situations and anticipate the benefit of updating the model.

We replicate experiments of Perez-Lebel et al. (2025), on hate speech detection tasks (binary classification with $Y=1$ for hate speech and $Y=0$ otherwise) across 14 datasets and 6 pretrained language models (see Appendix D.2 for details). The evaluation targets the confidence scores produced by the pretrained language models. As a post-training procedure, we fine-tune only the last layer of the language model, and compare the resulting reduction in expected cost against recalibration alone. Since the inputs are natural language, we apply our estimator to embeddings extracted from the penultimate layer of the model.

Figure 3 shows the correlation between our excess 0-1 risk estimator and the observed fine-tuning gain. We observe a notable increase in squared Spearman rank correlation, from $r^2 = 0.45$ in Perez-Lebel et al. (2025, fig. 5) to $r^2 = 0.61$ (fig. 3). This demonstrates that our estimator predicts better the benefit of post-training, outperforming the previous estimator, which itself outperformed standard metrics such as accuracy, Brier score, or AUC.

<figure><img src="image_7.png" /><figcaption>Figure 3: **Correlation between fine-tuning gain and excess 0-1 risk estimation.** The y-axis shows the expected cost reduction of fine-tuning compared to isotonic recalibration. Each point corresponds to a (model, dataset, cost matrix $\Lambda$) triplet. The dashed line corresponds to $y = x$.</figcaption></figure>

## 4.3 LLMs' GROUPING LOSS: OVER- AND UNDER-CONFIDENCE ACROSS GROUPS

To evaluate LLM confidence scores, we use Folktexts (Cruz et al., 2024), a benchmark designed to test Large Language Models (LLMs) on question answering tasks involving aleatoric uncertainty. Folktexts is built by turning tabular data from folktables (Ding et al., 2021) into natural language multiple-choice questions. Each prompt includes general context, a phrase representing the individual's row (using "The {column name} is {column value}" for each column), and the question of interest. From Folktexts, we use the ACS Income task (166,450 samples), predicting whether an individual's annual income exceeds $50,000 based on demographic and education features. This task inherently involves aleatoric uncertainty, since the outcome cannot be determined with certainty from the available information. Confidence scores are derived from the next-token probabilities of candidate answers. If the possible outcomes are yes and no, the confidence score is $\frac{P(\text{yes})}{P(\text{yes})+P(\text{no})}$.

**Results on LLMs' group-level confidence biases** We evaluate the zero-shot confidence scores of 27 open-source LLMs. Open-source models are required to access next-token probabilities for confidence scores. The selection covers model sizes from 1B to 70B parameters, sampled as uniformly as possible in this range, and includes families with multiple scales such as Llama 3 (1B, 3B, 8B, 70B) –full list of models in Appendix E.1. Models are evaluated using our estimator $\hat{\text{GL}}_{lb}$.
As previewed on Figure 1, LLMs are over and under-confident on demographics subgroups, echoing the miscalibration results of Cruz et al. (2024). Figure 4 displays, across all LLMs studied, the grouping error, that summarizes this behavior. With our Brier score-based estimator, a grouping loss of 0.01 corresponds to a squared probability error, i.e., an average deviation of 0.1 or less in individual confidence scores (see e.g. group deviations in fig. 1). This highlights that LLMs can display substantial confidence biases regardless of model size. Interestingly, fig. 4 also reveals a clear size effect: grouping loss decreases with size, suggesting that larger models provide more reliable confidence estimates. Yet a counter-trend emerges: across all sizes, instruction-tuned models consistently show higher grouping loss than their base counterparts. This indicates that instruction-tuning may amplify subgroup-specific biases. This phenomenon reflects Cruz et al. (2024)'s observation that instruction-tuning polarizes answer distributions and induces over-confidence.

## 4.4 ESTIMATING EPISTEMIC RISK IMPROVES LLM CASCADES

A key strength of our estimators is that they capture individual confidence scores. We illustrate this by applying our risk scores to LLM routing that selects from a pool of expert models the one expected to deliver the best task performance (Hu et al., 2024). Here, we use the Llama 3 instruction-tuned family (1B, 3B, 8B, 70B) and complement it with Phi-4 (14B), Gemma 2 (27B), and Mixtral-8x7B, to cover a wide range of model sizes, varying costs. We again use the Folktext ACS Income data.

A conventional confidence-based cascade sequentially queries a list of models (here, LLMs). At each step, the cascade stops if the confidence score for a given class exceeds a user-defined threshold; otherwise, the next model is queried. In some variants, the stopping criterion uses the average confidence across all models queried so far. However, this strategy proved ineffective in our experiments, as LLMs are poorly calibrated (see Appendix F.3 for results with this baseline) (Jitkrittum et al., 2023). Instead, we adopt the predictive router baseline introduced by Shnitzer et al. (2023); Hu et al. (2024).

**Predictive router baseline:** Predictive routing selects an LLM from a pool without first generating outputs for the incoming query. Formally, the accuracy of each model $m$ on a query $x$, denoted $Acc_m(x)$, is estimated by training on past queries labeled with whether the LLM's answer was correct. Hu et al. (2024) implements this with kNN and MLP routers trained on query embeddings. Since our partitioning estimators operate on structured query features, we additionally fit a decision tree in this space for a fair comparison. Appendix F.2 gives results comparing embedding-based models. At inference time, given a new query $x$, the router selects the model that best balances expected accuracy and cost: $\arg\max_{m \in \text{LLM pool}} \lambda Acc_m(x) - \text{cost}_m$. Here, the parameter $\lambda$ encodes the user's willingness to pay: higher values favor the most accurate model even if it is more expensive.

**Ours:** We propose to use the cascade strategy of Hu et al. (2024) together with our risk estimates. For each input, the procedure begins with the least expensive model and assigns it a quality score. If the score falls below a user-defined threshold $t$, that model is selected. Otherwise, the decision is deferred to the next, more costly model, and the process repeats. If no model satisfies the threshold, the model with the lowest overall risk is chosen. Whereas Hu et al. (2024) rely on an oracle score—reflecting the difficulty of designing effective scoring functions—we replace it with our risk estimates, $\hat{\mathcal{R}}_{LLM,m}^{CL}(x) + \hat{\mathcal{R}}_{LLM,m}^{GL}(x)$, which act as quality scores within their framework. Theoretically, if the true risk $\mathcal{R}_f$ were available, this policy with $t=0$ would yield optimal individual predictions that minimize total risk. A key trade-off, however, is that the cost of the cascade is the cumulative cost of all LLMs queried, as estimating a risk requires a forward pass to obtain confidence scores.

<figure><img src="image_8.png" /><figcaption>Figure 5: Cost-accuracy tradeoffs when varying the threshold of risk $t$ and willingness to pay $\lambda$ on the Llama 3 instruction-tuned family cascade.</figcaption></figure>

Figure 5 shows the performance of our cascade based on $\hat{\mathcal{R}}^{GL} + \hat{\mathcal{R}}^{CL}$, compared with a cascade based on $\hat{\mathcal{R}}^{CL}$ only and with the predictive router baseline. For high willingness to pay $\lambda$ (equivalently, low threshold $t$), our method outperforms the largest model in the cascade – +3% in accuracy compared
<figure><img src="image_9.png" /><figcaption>Figure 6: Cascades: our risk estimator gives better accuracies at a lower cost. Each point corresponds to a cascade (an LLM sequence). The y-axis is Acc(ours) - Acc(baseline). Values are positive when our cascade improves accuracy over the baseline. The x-axis is Cost(ours)/Cost(baseline). For values below 1, our model is less expensive. Our cascade based on $\hat{\mathbb{R}}^{CL} + \hat{\mathbb{R}}^{GL}$ ($t=0$) is compared to Left: the best LLM in the cascade, Middle: A cascade based on $\hat{\mathbb{R}}^{CL}$ only ($t=0$). Right: the predictive router baseline ($\lambda=100$). The red point denotes the cascade in fig. 5.</figcaption></figure>

to Llama 70B – at about 40% of its cost. This is possible because smaller models are retained when their estimated risk is low. Results at $t=0$, aggregated across 120 cascades built from the Llama 3 instruction tuned family, Phi4, Gemma 2 27B and Mixtral8x7B (ordered by size), are summarized in fig. 6. As shown in fig. 6 (left) our cascade consistently improves upon the accuracy of the largest model (up to 8%), at a fraction of its cost (25% - 50%). $\hat{\mathbb{R}}_f^{CL}$ alone already provides a useful score (fig. 5), consistent with prior reports that LLMs are heavily miscalibrated (Cruz et al., 2024). Adding $\hat{\mathbb{R}}_f^{GL}$ however yields further gains of up to 3% accuracy (fig. 6 middle) at only a small cost increase, showing that targeting the full epistemic risk enables better routing decisions. Finally, fig. 6 (right) shows that our cascades systematically outperform the predictive router baseline, and is strictly Pareto optimal (better accuracy at a smaller cost) in 40% of the cases.

# 5 CONCLUSION: BETTER EPISTEMIC EVALUATION GIVES BETTER DECISIONS

We introduced consistent estimators targeting the total epistemic loss and associated excess risk in decision settings. They improve sample efficiency while being interpretable, as the input-space partition reveals model biases. Crucially, they apply at the individual-sample level. We show that tighter control of epistemic error—even for black-box models—enables better decisions, whether by improving the model or deferring specific cases to a stronger model or a human, a key property as LLMs are deployed in complex or high-risk systems.

# REFERENCES

* Freddie Bickford Smith, Jannik Kossen, Eleanor Trollope, Mark Van Der Wilk, Adam Foster, and Tom Rainforth. Rethinking aleatoric and epistemic uncertainty. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 4345–4359. PMLR, 13–19 Jul 2025.

* Leo Breiman. Random Forests. Machine Learning, 45(1):5–32, October 2001. ISSN 1573-0565.

* Lihu Chen, Alexandre Perez-Lebel, Fabian Suchanek, and Gaël Varoquaux. Re-confidencing LLM Uncertainty from the Grouping Loss Perspective. In EMNLP 2024 - Conference on Empirical Methods in Natural Language Processing, Miami, United States, November 2024. arXiv.

* André F Cruz, Moritz Hardt, and Celestine Mendler-Dünner. Evaluating language models as risk scores. 2024.
540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

A. P. Dawid. Probability forecasting. In *Encyclopedia of Statistical Sciences*. John Wiley & Sons, Ltd, 1986. ISBN 978-0-471-66719-3.

Gianluca Detommaso, Martin Andres Bertran, Riccardo Fogliato, and Aaron Roth. Multicalibration for confidence scoring in llms. In *International Conference on Machine Learning*, pp. 10624–10641. PMLR, 2024.

Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. Retiring Adult: New Datasets for Fair Machine Learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan (eds.), *Advances in Neural Information Processing Systems*, volume 34, pp. 6478–6490. Curran Associates, Inc., 2021.

Charles Elkan. The foundations of cost-sensitive learning. In *International joint conference on artificial intelligence*, volume 17, pp. 973–978. Lawrence Erlbaum Associates Ltd, 2001.

EUC. AI Act Regulation (EU) 2024/1689 – Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). Publications Office of the European Union, 2025.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Maria Florina Balcan and Kilian Q. Weinberger (eds.), *Proceedings of The 33rd International Conference on Machine Learning*, volume 48 of *Proceedings of Machine Learning Research*, pp. 1050–1059, New York, New York, USA, 20–22 Jun 2016. PMLR.

Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. 102(477):359–378, 2007. ISSN 0162-1459, 1537-274X.

Tilmann Gneiting, Fadoua Balabdaoui, and Adrian E Raftery. Probabilistic forecasts, calibration and sharpness. *Journal of the Royal Statistical Society Series B: Statistical Methodology*, 69(2): 243–268, 2007.

László Győrfi, Michael Kohler, Adam Krzyżak, and Harro Walk. Partitioning Estimates. In *A Distribution-Free Theory of Nonparametric Regression*, pp. 52–69. Springer, New York, NY, 2002. ISBN 978-0-387-22442-8.

Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with a reject option: A survey, 2024.

Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi-llm routing system, 2024.

Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. 110(3):457–506, 2021. ISSN 1573-0565.

Lavender Yao Jiang, Xujin Chris Liu, Nima Pour Nejatian, Mustafa Nasir-Moin, Duo Wang, Anas Abidin, Kevin Eaton, Howard Antony Riina, Ilya Laufer, Paawan Punjabi, et al. Health system-scale language models are all-purpose prediction engines. *Nature*, 619(7969):357–362, 2023.

Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. When does confidence-based cascade deferral suffice? In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.

Daniel D Johnson, Daniel Tarlow, David Duvenaud, and Chris J Maddison. Experts don't cheat: Learning what you don't know by predicting pairs. arXiv preprint arXiv:2402.08733, 2024.

Uriel Katz, Eran Cohen, Eliya Shachar, Jonathan Somer, Adam Fink, Eli Morse, Beki Shreiber, and Ido Wolf. Gpt versus resident physicians — a benchmark based on official board scores. NEJM AI, 1(5):A1dbp2300192, 2024.
594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? *Advances in neural information processing systems*, 30, 2017.

Meelis Kull and Peter Flach. Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration. In *Annalisa Appice, Pedro Pereira Rodrigues, Vítor Santos Costa, Carlos Soares, João Gama, and Alípio Jorge (eds.), Machine Learning and Knowledge Discovery in Databases*, volume 9284, pp. 68–85. Springer International Publishing, Cham, 2015. ISBN 978-3-319-23527-1 978-3-319-23528-8.

Salem Lahlou, Moksh Jain, Hadi Nekoei, Victor I Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. DEUP: Direct epistemic uncertainty prediction. *Transactions on Machine Learning Research*, 2023. ISSN 2835-8856. Expert Certification.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In *Advances in Neural Information Processing Systems*, volume 30. Curran Associates, Inc., 2017.

Sarah Lichtenstein, Baruch Fischhoff, and Lawrence D. Phillips. Calibration of probabilities: The state of the art. In *Helmut Jungermann and Gerard De Zeeuw (eds.), Decision Making and Change in Human Affairs: Proceedings of the Fifth Research Conference on Subjective Probability, Utility, and Decision Making*, Darmstadt, 1–4 September, 1975, pp. 275–324. Springer Netherlands, 1977. ISBN 978-94-010-1276-8.

Aman Mahajan, Stephen Esper, Thien Htay Oo, Jeffery McKibben, Michael Garver, Jamie Artman, Cynthia Klahre, John Ryan, Senthilkumar Sadhasivam, Jennifer Holder-Murray, and Oscar C. Marroquin. Development and Validation of a Machine Learning Model to Identify Patients Before Surgery at High Risk for Postoperative Adverse Events. *JAMA Network Open*, 6(7):e2322285, July 2023. ISSN 2574-3805.

Nicolai Meinshausen. Quantile regression forests. *J. Mach. Learn. Res.*, 7:983–999, 2006.

Takahiro Mimori, Keiko Sasada, Hirotaka Matsui, and Issei Sato. Diagnostic uncertainty calibration: Towards reliable machine predictions in medical domain. In *International Conference on Artificial Intelligence and Statistics*, pp. 3664–3672. PMLR, 2021.

Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert, 2021.

Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning to defer to humans, 2023.

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In *Proceedings of the AAAI conference on artificial intelligence*, volume 29, 2015.

M Omar, R Agbareia, B Glicksberg, G Nadkarni, and E Klang. Benchmarking the confidence of large language models in answering clinical questions: Cross-sectional evaluation study. *JMIR Med Inform*, 2025.

Alexandre Perez-Lebel, Marine Le Morvan, and Gaël Varoquaux. Beyond calibration: estimating the grouping loss of modern neural networks. In *ICLR*, 2023.

Alexandre Perez-Lebel, Gael Varoquaux, Sanmi Koyejo, Matthieu Doutreligne, and Marine Le Morvan. Decision from suboptimal classifiers: Excess risk pre- and post-calibration. In Yingzhen Li, Stephan Mandt, Shipra Agrawal, and Emtiyaz Khan (eds.), *Proceedings of The 28th International Conference on Artificial Intelligence and Statistics*, volume 258 of *Proceedings of Machine Learning Research*, pp. 2395–2403. PMLR, 03–05 May 2025.

John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In *Advances in Large Margin Classifiers*, pp. 61–74. MIT Press, 1999.

Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, and Artem Babenko. Tabred: Analyzing pitfalls and filling the gaps in tabular deep learning benchmarks. In *The Thirteenth International Conference on Learning Representations*, 2025.
648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

Robin Senge, Stefan Bösner, Krzysztof Dembczyński, Jörg Haasenritter, Oliver Hirsch, Norbert Donner-Banzhoff, and Eyke Hüllermeier. Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty. 255:16–29, 2014. ISSN 0020-0255.

Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets, 2023.

Ola Shorinwa, Zhiting Mei, Justin Lidard, Allen Z Ren, and Anirudha Majumdar. A survey on uncertainty quantification of large language models: Taxonomy, open research challenges, and future directions. ACM Computing Surveys, 2025.

Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023.

Roman Vashurin, Ekaterina Fadeeva, Artem Vazhentsev, Lyudmila Rvanova, Daniil Vasilev, Akim Tsvigun, Sergey Petrakov, Rui Xing, Abdelrahman Sadallah, Kirill Grishchenkov, et al. Benchmarking uncertainty quantification methods for large language models with Im-polygraph. Transactions of the Association for Computational Linguistics, 13:220–248, 2025.

Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests, 2017.

Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M. Kitani, Yair Alon, and Elad Eban. Wisdom of committees: An overlooked approach to faster and more accurate models, 2022.

Lisa Wimmer, Yusuf Sale, Paul Hofman, Bernd Bischl, and Eyke Hüllermeier. Quantifying aleatoric and epistemic uncertainty in machine learning: are conditional entropy and mutual information appropriate measures? In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, UAI ’23. JMLR.org, 2023.

Zhiqiu Xia, Jinxuan Xu, Yuqian Zhang, and Hang Liu. A survey of uncertainty estimation methods on large language models. arXiv preprint arXiv:2503.00172, 2025.
# A FURTHER DETAILS ON THE ESTIMATORS IN PRACTICE

## A.1 ALGORITHM

We describe here the algorithm that we use to compute our partitioning estimate with an honest tree from Section 3.3.

**Algorithm 1** Honest tree $\hat{r}$

Input: $(X_{cal}, Y_{cal}), (X_{fit}, Y_{fit}), (X_{eval}, Y_{eval})$

Fit $c \circ f$ using Platt scaling with $(X_{cal}, Y_{cal})$

Fit the tree $\hat{r}$ with $(X_{fit}, Y_{fit} - c \circ f(X_{fit}))$

for $\mathcal{L}_j \in \mathcal{L}$ do

Compute $\hat{r}_j = \frac{\sum_{i \in \mathrm{eval}} \mathbf{1}(X_i \in \mathcal{L}_j)(Y_i - c \circ f(X_i))}{\sum_{i \in \mathrm{eval}} \mathbf{1}(X_i \in \mathcal{L}_j)}$ with $(X_{eval}, Y_{eval} - c \circ f(X_{eval}))$

end for

## A.2 DISCUSSION ON THE PARAMETERS CHOSEN

**Calibration** - We use Platt scaling (Platt, 1999) to obtain $c \circ f$. The quality of the calibration matters: if the underlying model is poorly calibrated, our approach will partly capture miscalibration error, but at the expense of grouping error, and less effectively than a dedicated calibration method. For Figure 2, to limit the variance at low sample size due to a bad calibration $c$, we allocate at least 4000 samples to the calibration set, or follow the proportions described in 3.3.

**Hyperparameters** - We adopt a conservative strategy in our experiments: a single fixed set of hyperparameters for $\hat{r}$ across all tasks (unlimited depth with a minimum of 15 samples per leaf at training time). Although task-specific tuning could yield stronger results, this setup illustrates the robustness of our chosen hyperparameters. We split the data into calibration (10%), fitting (40%) and evaluation (50%) sets.

**Choice of input space** - The input space on which the tree is fitted can vary. In standard tabular problems, the natural choice is the original feature space. In contrast, neural networks often operate on less structured data (e.g., images, text) and produce richer internal representations. In such cases, the partition can be built on an embedded representation of the data—for example, the activations of the last hidden layer before the softmax (Perez-Lebel et al., 2023)—or on a learned low-dimensional embedding such as UMAP (Detommaso et al., 2024).

# B ADDITIONAL THEORETICAL RESULT

Following proposition 4, we have the result on the entire distribution.

**Corollary 1.** If the true distribution of $f^*(X) | \mathcal{L}_j, f(X)$ is of mean $\hat{r}(X) + c \circ f(X)$, then

$$ \hat{\mathcal{R}}_{f}^{\mathrm{GL}} \leq \mathcal{R}_{f}^{\mathrm{GL}}. $$

# C PROOFS

We derive in this part the proofs of the results described in the main paper.

## C.1 PROOF OF PROPOSITION 2: DEBIASING OF THE GROUPING LOSS ESTIMATOR

**Proposition 6 (Debiased grouping loss estimator).** Given a partition $\mathcal{L}^{(n)}$ and a dataset $\mathcal{D}^{(n)} = \{(X_i, Y_i)\}_{i=1..n}$, let $\hat{r}_j$ (resp. $\hat{v}_j$) be the sample mean (resp. sample variance) of $Y - c \circ f(X)$ in region $j$. A debiased estimator of $\mathrm{GL}_{\mathrm{lb}}(\mathcal{L}^{(n)})$ is:

$$ \widehat{\mathrm{GL}}_{\mathrm{lb}}^{(n)} \stackrel{\mathrm{def}}{=} \sum_{j=1}^{J} \hat{p}_{j} \left( \hat{r}_{j}^{2} - \frac{1}{n_{j}} \hat{v}_{j} \right) \qquad (7) $$
The proof is heavily inspired by the proof of proposition 4.3 from Perez-Lebel et al. (2023).

*Proof.* We note

$$ \widehat{GL}_{plugin} = \sum_{j=1}^{J} \hat{p}_j \hat{r}_j^2. $$

The plugin version of the estimator from the definition of GL<sub>lb</sub> in Lemma 1.

We have

$$ \mathbb{E}[\widehat{GL}_{plugin}] = \sum_{j=1}^{J} \mathbb{E}[\hat{p}_j \hat{r}_j^2] $$

For $j \in [1, J]$ fixed, we have:

$$ \begin{aligned} \mathbb{E}[\hat{p}_j \hat{r}_j^2] &= \mathbb{E} [\mathbb{E}[\hat{p}_j \hat{r}_j^2 \mathbf{1}_{\hat{p}_j \ge 0} | \hat{p}_j]] & \text{total expectation} \\ &= \mathbb{E} [\hat{p}_j \mathbf{1}_{\hat{p}_j \ge 0} \mathbb{E}[\hat{r}_j^2 | \hat{p}_j]] \end{aligned} $$

We have:

$$ \mathbb{E}[\hat{r}_j^2 | \hat{p}_j] = \mathbb{E}[\hat{r}_j]^2 + \mathbb{V}[\hat{r}_j] $$

Furthermore

$$ \mathbb{E}[\hat{r}_j] = \mathbb{E} \left[ \frac{1}{n_j} \sum_{X_i \in \mathcal{L}_j} Y_i - c \circ f(X_i) \mid X_i \in \mathcal{L}_j \right] \quad (8) $$

$$ = \frac{1}{n_j} \sum_{X_i \in \mathcal{L}_j} \mathbb{E}[Y_i - c \circ f(X_i) \mid X_i \in \mathcal{L}_j] \quad (9) $$

$$ = \mathbb{E}[Y - c \circ f(X) \mid X \in \mathcal{L}_j] \quad (10) $$

(8): by definition.

(9): linearity of expectation.

(10): $Y_i - c \circ f(X_i)$ are identically distributed $| \mathcal{L}_j$.

And

$$ \mathbb{V}[\hat{r}_j] = \mathbb{V} \left[ \frac{1}{n_j} \sum_{i \in \mathcal{L}_j} Y_i - c \circ f(X_i) \right] \quad (11) $$

$$ = \frac{1}{n_j^2} \mathbb{V} \left[ \sum_{i \in \mathcal{L}_j} Y_i - c \circ f(X_i) \right] \quad (12) $$

$$ = \frac{1}{n_j^2} \sum_{i \in \mathcal{L}_j} \mathbb{V}[Y_i - c \circ f(X_i) \mid \mathcal{L}_j] \quad (13) $$

$$ = \frac{1}{n_j} \mathbb{V}[Y - c \circ f(X) \mid \mathcal{L}_j] \quad (14) $$

(11): by definition.

(13): $(Y_i - c \circ f(X_i))$ are independent.

(14): $Y_i - c \circ f(X_i)$ are identically distributed $| \mathcal{L}_j$.
Which gives:

$$
\begin{align*}
\mathbb{E}[\hat{p}_j \hat{r}_j^2] &= \mathbb{E}[\hat{p}_j \mathbf{1}_{\hat{p}_j \ge 0} \mathbb{E}[\hat{r}_j^2 | \hat{p}_j]] \\
&= \mathbb{E} \left[ \hat{p}_j \mathbf{1}_{\hat{p}_j \ge 0} \left( \mathbb{E}[Y - c \circ f(X) | X \in \mathcal{L}_j]^2 + \frac{1}{n_j} \mathbb{V}[Y - c \circ f(X) | X \in \mathcal{L}_j] \right) \right] \\
&= p_j \mathbb{E}[Y - c \circ f(X) | X \in \mathcal{L}_j]^2 + \mathbb{E} \left[ \frac{1}{n} \mathbb{V}[Y - c \circ f(X) | X \in \mathcal{L}_j] \mathbf{1}_{\hat{p}_j \ge 0} \right] \\
&= \underbrace{p_j \mathbb{E}[Y - c \circ f(X) | X \in \mathcal{L}_j]^2}_{p_j \mathrm{GL}_{\mathrm{lb}}^{(j)}} + \underbrace{\frac{1}{n} (1 - \nu_j) \mathbb{V}[Y - c \circ f(X) | X \in \mathcal{L}_j]}_{\text{bias term}}
\end{align*}
$$

With $\nu_j \stackrel{\text{def}}{=} (1 - p_j)^n$, the probability that no samples falls in region $\mathcal{L}_j$. We make the approximation that $\nu_j \approx 0$, which is correct when $n$ and $p_j$ are sufficiently large. Therefore, we debias the plugin estimate with the unbiased sample variance (with Bessel correction) leading to our estimator:

$$
\widehat{\mathrm{GL}}_{\mathrm{lb}}(\mathcal{L}) = \sum_{j=1}^{J} \hat{p}_j \left( \hat{r}_j^2 - \frac{1}{n_j} \hat{v}_j \right)
$$

C.2 PROOF OF PROPOSITION 3: CONSISTENCY OF THE GROUPING LOSS ESTIMATOR

We first define $\mathrm{GL}_{lb}(\mathcal{L}, \mathcal{D}) \stackrel{\mathrm{def}}{=} \mathbb{E}_X[\hat{r}(X)^2]$ the expectation of the partitioning estimate obtained from $(\mathcal{L}, \mathcal{D})$. This is to be distinguished from $\mathrm{GL}_{lb}(\mathcal{L})$ which considers the expectation of the true values of $\mathbb{E}[Y - c \circ f(X) | X \in \mathcal{L}_j]$ for each region $j$. ($\mathrm{GL}_{lb}(\mathcal{L}) = \mathbb{E}_j[\mathbb{E}[Y - c \circ f(X) | X \in \mathcal{L}_j]]$).

For the proof of Proposition 3, we will require two lemmas

**Lemma 2.** Let $(\mathcal{L}^{(n)})_{n \in \mathbb{N}}$ be a sequence of partitions such that the associated partitioning estimate $\hat{r}^{(n)}$ is consistent. $\mathrm{GL}_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)})$ converges in $L_1$ to $\mathrm{GL}$:

$$
\mathbb{E}_{\mathcal{D}^{(n)}} \left[ | \mathrm{GL} - \mathrm{GL}_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) | \right] \xrightarrow{n \to +\infty} 0
$$

Proof. The partitioning estimate $\hat{r}^{(n)}$ is weakly universally consistent towards

$$
r : x \mapsto \mathbb{E}[Y - c \circ f(X) | X = x],
$$

that is:

$$
\mathbb{E}_{\mathcal{D}^{(n)}} \left[ \mathbb{E}_X \left[ (\hat{r}^{(n)}(X) - r(X))^2 \right] \right] \xrightarrow{n \to +\infty} 0
$$

And by definition,

$$
\begin{align*}
\mathrm{GL} &= \mathbb{E}_X [\mathbb{E}[Y - c \circ f(X) | X]^2] \\
&= \mathbb{E}_X [r(X)^2]
\end{align*}
$$

and

$$
\mathrm{GL}_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) = \mathbb{E}_X[\hat{r}^{(n)}(X)^2]
$$

We compute:

$$
| \mathrm{GL} - \mathrm{GL}_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) |^2 = | \mathbb{E}_X [r(X)^2 - \hat{r}^{(n)}(X)^2] |^2 \quad (15)
$$

$$
= | \mathbb{E}_X [ (r(X) - \hat{r}^{(n)}(X))(r(X) + \hat{r}^{(n)}(X)) ] |^2 \quad (16)
$$

$$
\leq \mathbb{E}_X [ |r(X) - \hat{r}^{(n)}(X)|^2 ] \mathbb{E}_X [ |r(X) + \hat{r}^{(n)}(X)|^2 ] \quad (17)
$$

$$
\leq 4 \mathbb{E}_X [ |r(X) - \hat{r}^{(n)}(X)|^2 ] \qquad (18)
$$
(17): Cauchy Schwarz.

(18): $|r(X) + \hat{r}^{(n)}(X)| \le 2$ by definition.

So taking the expectation over $\mathcal{D}^{(n)}$, we have

$$ \mathbb{E}_{\mathcal{D}^{(n)}} [ | GL - GL_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) |^2 ] \le 4 \mathbb{E}_{\mathcal{D}^{(n)}} [ \mathbb{E}_X [ | r(X) - \hat{r}^{(n)}(X) |^2 ] \xrightarrow{n \to +\infty} 0 ] $$

by consistency hypothesis.

Which, by Cauchy Schwarz, gives

$$ \mathbb{E}_{\mathcal{D}^{(n)}} [ | GL - GL_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) | ] \le \sqrt{\mathbb{E}_{\mathcal{D}^{(n)}} [ | GL - GL_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) |^2 ]} \xrightarrow{n \to +\infty} 0 $$

**Lemma 3.** Let $(\mathcal{L}^{(n)})_{n \in \mathbb{N}}$ be a sequence of partitions such that the associated partitioning estimate $\hat{r}^{(n)}$ is consistent. We have:

$$ \mathbb{E}_{\mathcal{D}^{(n)}} [ | GL_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) - \widehat{GL}_{lb}(\mathcal{L}^{(n)}) | ] \xrightarrow{n \to +\infty} 0 $$

*Proof.* From the proof of Theorem 2 in Section C.1, we have that:

We remind that

$$ GL_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) = \mathbb{E}_X [\hat{r}^{(n)}(X)^2] $$

Which gives,

$$ | \widehat{GL}_{lb}(\mathcal{L}^{(n)}) - GL_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) | = | \sum_{j=1}^{J_n} \hat{p}_j(\hat{r}_j^{(n)})^2 - \frac{1}{n}\hat{v}_j^{(n)} - \sum_{j=1}^{J_n} p_j(\hat{r}_j^{(n)})^2 | \quad (19) $$

$$ \le | \sum_{j=1}^{J_n} \hat{p}_j(\hat{r}_j^{(n)})^2 - p_j(\hat{r}_j^{(n)})^2 | + | \sum_{j=1}^{J_n} \frac{1}{n}\hat{v}_j^{(n)} | \quad (20) $$

$$ \le \sum_{j=1}^{J_n} | \hat{p}_j - p_j | (\hat{r}_j^{(n)})^2 + | \sum_{j=1}^{J_n} \frac{1}{n}\hat{v}_j^{(n)} | \quad (21) $$

$$ \le \sum_{j=1}^{J_n} | \hat{p}_j - p_j | (\hat{r}_j^{(n)})^2 + \frac{4J_n}{n} \quad (22) $$

(19): By definition of both terms.

(20): Triangle inequality.

(21): Triangle inequality.

(22): $|Y - c \circ f(X)| \le 2$ so $\hat{v}_j^{(n)} \le 4$.
Taking the expectation over $\mathcal{D}^{(n)}$:

$$ \mathbb{E}_{\mathcal{D}^{(n)}} [ | \widehat{GL}_{lb}(\mathcal{L}^{(n)}) - GL_{lb}(\mathcal{L}^{(n)}, \mathcal{D}^{(n)}) | ] \le \mathbb{E}_{\mathcal{D}^{(n)}} \left[ \sum_{j=1}^{J_n} | \hat{p}_j - p_j | (\hat{r}_j^{(n)})^2 + \frac{4J_n}{n} \right] \quad (23) $$

$$ \le \sum_{j=1}^{J_n} \mathbb{E}_{\mathcal{D}^{(n)}} [ | \hat{p}_j - p_j | (\hat{r}_j^{(n)})^2 ] + \frac{4J_n}{n} \quad (24) $$

$$ \le \sum_{j=1}^{J_n} 4 \mathbb{E}_{\mathcal{D}^{(n)}} [ | \hat{p}_j - p_j | ] + \frac{4J_n}{n} \quad (25) $$

$$ \le 4 \sum_{j=1}^{J_n} \sqrt{\mathbb{E}_{\mathcal{D}^{(n)}} [ | \hat{p}_j - p_j |^2 ]} + \frac{4J_n}{n} \quad (26) $$

$$ \le 4 \sum_{j=1}^{J_n} \sqrt{V_{\mathcal{D}^{(n)}}[\hat{p}_j]} + \frac{4J_n}{n} \quad (27) $$

$$ \le 4 \sum_{j=1}^{J_n} \sqrt{\frac{p_j(1-p_j)}{n}} + \frac{4J_n}{n} \quad (28) $$

$$ \le \frac{2J_n}{n} + \frac{4J_n}{n} \quad (29) $$

$$ \le \frac{6J_n}{n} \xrightarrow{n \to +\infty} 0 \quad (30) $$

(24): $\frac{4J_n}{n}$ doesn't depend on $\mathcal{D}^{(n)}$.

(25): $|Y - c \circ f(X)| \le 2$ so $\hat{r}_j^{(n)} \le 2$.

(26): Cauchy Schwarz.

(27): $\mathbb{E}_{\mathcal{D}^{(n)}}[\hat{p}_j] = p_j$.

(29): $p_j(1 - p_j) \le \frac{1}{4}$ for $p_j \in [0, 1]$.

(30): By hypothesis. $\square$

We now prove the final result.

**Proposition 7 (Consistency of the grouping loss estimator).** Let $\hat{r}^{(n)}$ be a partitioning estimate applied to $R = Y - c \circ f(X)$. Assume that $\hat{r}^{(n)}$ is consistent. Then the sequence of estimators $\widehat{GL}_{lb}^{(n)}$ is weakly universally consistent, i.e.,

$$ \mathbb{E}_{\mathcal{D}^{(n)}} \left[ | GL - \widehat{GL}_{lb}^{(n)} | \right] \xrightarrow{n \to +\infty} 0. $$

*Proof.* Combining Lemma 2 and Lemma 3, and using the triangle inequality, we have the result. $\square$

### C.3 PROOF OF PROPOSITION 4: LOWER BOUND OF THE GROUPING RISK ESTIMATOR

**Proposition 8.** If the true distribution of $f^*(X) | \mathcal{L}_j, f(X)$ is of mean $c \circ f(X) + \hat{r}_j$, then $\hat{\mathbb{R}}_f^{\text{GL}}(X)$ represents the excess decision risk of the calibrated classifier captured by the partition, i.e. for $x \in \mathcal{L}_j$,

$$ \hat{\mathbb{R}}_f^{\text{GL}}(x) = \mathbb{E}[EC(\delta_{\text{cof},t^*}, X) - EC(\delta_{\text{cof},\hat{r},t^*}, X) | X \in \mathcal{L}_j, f(X) = f(x)]. $$

Moreover, under the above assumptions, $\hat{\mathbb{R}}_f^{\text{GL}}(X)$ is a lower-bound on the true grouping risk, i.e.,

$$ \hat{\mathbb{R}}_f^{\text{GL}}(X) \le \mathbb{R}_f^{\text{GL}}(X). $$
**Proof.** From Perez-Lebel et al. (2025)'s corollary B.4, we have, for $x \in \mathcal{L}_j$ that

$$ 
\begin{aligned}
\mathbb{E}[\Lambda_{\delta_{cof,t^*}(X),Y} - \Lambda_{\delta_{cof+\hat{r},t^*}(X),Y} | X \in \mathcal{L}_j, f(X) = f(x)] &= \\
&\quad \mathbb{E}[\Lambda_{\Delta}(\delta_{cof,t^*}(X) - \delta_{cof+\hat{r},t^*}(X))(f^*(X) - t^*) | X \in \mathcal{L}_j, f(X) = f(x)] \\
&= \Lambda_{\Delta}(\mathbf{1}_{cof(X) \ge t^*} - \mathbf{1}_{cof(X)+\hat{r}(X) \ge t^*})(\mathbb{E}[f^*(X) | X \in \mathcal{L}_j, f(X) = f(x)] - t^*) \\
&= \Lambda_{\Delta}(\mathbf{1}_{cof(X) \ge t^*} - \mathbf{1}_{cof(X)+\hat{r}(X) \ge t^*})(c \circ f(x) + \hat{r}_j - t^*)
\end{aligned}
$$

The last formula gives $\begin{cases} \Lambda_{\Delta} | c \circ f(X) + \hat{r}(X) - t^* | & \text{if } \mathbf{1}_{cof(X)} + \hat{r}(X) \ge t^* \neq \mathbf{1}_{cof(X) \ge t^*} \\ 0 & \text{else.} \end{cases}$

Therefore, writing

$$ 
\begin{aligned}
\hat{\mathcal{R}}_{f}^{\text{GL}}(x) &= \mathbb{E}[\text{EC}(\delta_{\text{cof},t^*}, X) - \text{EC}(\delta_{\text{cof}+\hat{r},t^*}, X) | X \in \mathcal{L}_j, f(X) = f(x)] \\
&= \mathbb{E}[\Lambda_{\delta_{\text{cof},t^*}(X),Y} - \Lambda_{\delta_{\text{cof}+\hat{r},t^*}(X),Y} | X \in \mathcal{L}_j, f(X) = f(x)]
\end{aligned}
$$

And using the fact that $\forall (g, p) \in [0, 1]^X \times [0, 1]$, $\mathbb{E}[\Lambda_{\delta^*(X),Y} | X] \ge \mathbb{E}[\Lambda_{\delta_{g,p}^*(X),Y} | X]$, we directly have that

$$ 
\begin{aligned}
\mathcal{R}_{f}^{\text{GL}} - \hat{\mathcal{R}}_{f}^{\text{GL}} &= \mathbb{E}[\Lambda_{\delta^*(X),Y} - \Lambda_{\delta_{\text{cof},t^*}(X),Y} - (\Lambda_{\delta_{\hat{r}+\text{cof},t^*}(X),Y} - \Lambda_{\delta_{\text{cof},t^*}(X),Y}) | X] \\
&= \mathbb{E}[\Lambda_{\delta^*(X),Y} - \Lambda_{\delta_{\hat{r}+\text{cof},t^*}(X),Y} | X] \\
&\ge 0
\end{aligned}
$$

Where the last inequality is obtained because $\delta^*$ is optimal

## C.4 PROOF OF PROPOSITION 5: CONSISTENCY OF THE GROUPING REGRET ESTIMATOR

**Proposition 9 (Consistency of the grouping regret estimator).** Let $\hat{r}^{(n)}$ be a partitioning estimate applied to $R = Y - c \circ f(X)$. Assume that $\hat{r}^{(n)}$ is consistent. Then the sequence of estimators $\hat{\mathcal{R}}_{f}^{\text{GL}(n)}$ is weakly universally consistent, i.e.,

$$ \mathbb{E}_{\mathcal{D}^{(n)}} [ | \hat{\mathcal{R}}_{f}^{\text{GL}(n)} - \mathcal{R}_{f}^{\text{GL}} | ] \xrightarrow{n \to +\infty} 0. $$

**Proof.** We have:

$$ 
\begin{aligned}
\mathbb{E}_{\mathcal{D}^{(n)}} [ | R_f - \hat{R}_f^{(n)} | ] &= \mathbb{E}_{\mathcal{D}^{(n)}} [ | \mathbb{E}_X[R_f(X)] - \mathbb{E}[\hat{R}_f^{(n)}(X)] | ] \\
&= \mathbb{E}_{\mathcal{D}^{(n)}} [ | \mathbb{E}_X[R_f(X) - \hat{R}_f^{(n)}(X)] | ] && \text{Linearity of expectation} \\
&\le \mathbb{E}_{\mathcal{D}^{(n)}} [ | \mathbb{E}_X[| R_f(X) - \hat{R}_f^{(n)}(X) |] | ] && \text{Triangle inequality}
\end{aligned}
$$

Let's get interested in $| R_f(X) - \hat{R}_f^{(n)}(X) |$. We will directly decompose into the cases that can happen due to the indicator function.

**Case 1: $\mathbf{1}_{\text{cof}(X)} + \hat{r}(X) \ge t^* \neq \mathbf{1}_{\text{cof}(X) \ge t^*}$ and $\mathbf{1}_{f^*(X) \ge t^*} \neq \mathbf{1}_{\text{cof}(X) \ge t^*}$** Then

$$ | R_f(X) - \hat{R}_f^{(n)}(X) | = \| f^*(X) - t^* \| - \| \hat{r}^{(n)}(X) + c \circ f(X) - t^* \| $$

By hypothesis, either both

$$ f^*(X) \ge t^* \quad \text{and} \quad \hat{r}^{(n)}(X) + c \circ f(X) \ge t^* $$

or

$$ f^*(X) < t^* \quad \text{and} \quad \hat{r}^{(n)}(X) + c \circ f(X) < t^* $$

Which gives

$$ | R_f(X) - \hat{R}_f^{(n)}(X) | = | \hat{r}^{(n)}(X) + c \circ f(X) - f^*(X) | $$
**Case 2:** $1_{cof(X)}+\hat{r}(X)\ge t^* \ne 1_{cof(X)}\ge t^*$ and $1_{f^*(X)}\ge t^* = 1_{cof(X)}\ge t^*$ Then

$$|R_f(X) - \hat{R}_f^{(n)}(X)| = |\hat{r}^{(n)}(X) + c \circ f(X) - t^*|$$

**Case 3:** $1_{cof(X)}+\hat{r}(X)\ge t^* = 1_{cof(X)}\ge t^*$ and $1_{f^*(X)}\ge t^* \ne 1_{cof(X)}\ge t^*$ Then

$$|R_f(X) - \hat{R}_f^{(n)}(X)| = |f^*(X) - t^*|$$

**Case 4** $1_{cof(X)}+\hat{r}(X)\ge t^* = 1_{cof(X)}\ge t^*$ and $1_{f^*(X)}\ge t^* = 1_{cof(X)}\ge t^*$ Then

$$|R_f(X) - \hat{R}_f^{(n)}(X)| = 0$$

Case 1 and 4 are easily dealt with with the consistency of the estimator $\hat{r}^{(n)}$. We can however control cases 2 and 3 thanks to the consistency hypothesis. Case 2 and 3 correspond to when $f^*(X)$ and $\hat{r}(X) + c \circ f(X)$ aren't on the same side of the threshold $t^*$. All the 4 cases described correspond to all the universe of possibilities.

We compute:

$$\mathbb{E}[\mathbf{1}_{Case1} | \hat{r}^{(n)}(X) + c \circ f(X) - f^*(X) |] \le \sqrt{\mathbb{E}[\mathbf{1}_{Case1} | \hat{r}^{(n)}(X) + c \circ f(X) - f^*(X) |^2]} \quad (31)$$

$$\le \sqrt{\mathbb{E}[|\hat{r}^{(n)}(X) + c \circ f(X) - f^*(X)|^2]} \quad (32)$$

(31): Cauchy Schwarz.

So taking the expectation over $\mathcal{D}^{(n)}$ gives

$$\begin{aligned} \mathbb{E}_{\mathcal{D}^{(n)}} [\mathbb{E}[\mathbf{1}_{Case1} | \hat{r}^{(n)}(X) + c \circ f(X) - f^*(X) |]] \\ \le \mathbb{E}_{\mathcal{D}^{(n)}} \left[ \sqrt{\mathbb{E}[|\hat{r}^{(n)}(X) + c \circ f(X) - f^*(X)|^2]} \right] \\ \le \sqrt{\mathbb{E}_{\mathcal{D}^{(n)}} [\mathbb{E}[|\hat{r}^{(n)}(X) + c \circ f(X) - f^*(X)|^2]]} \\ \xrightarrow{n \to +\infty} 0 \end{aligned}$$

by Jensen's inequality and consistency.

We compute:

$$\mathbb{E}[\mathbf{1}_{Case2} | \hat{r}^{(n)}(X) + c \circ f(X) - t^* |] \le \sqrt{\mathbb{E}[\mathbf{1}_{Case2} | \hat{r}^{(n)}(X) + c \circ f(X) - t^* |^2]}$$

Furthermore,

$$\mathbb{E}_X [(\hat{r}^{(n)}(X) + c \circ f(X) - f^*(X))^2] = \sum_{i=1}^{4} \mathbb{E}_X [\mathbf{1}_{Casei} (\hat{r}^{(n)}(X) + c \circ f(X) - f^*(X))^2]$$

And

$$\mathbb{E}_X [\mathbf{1}_{Case2} (\hat{r}^{(n)}(X) + c \circ f(X) - f^*(X))^2] \ge \mathbb{E}_X [\mathbf{1}_{Case2} (\hat{r}^{(n)}(X) + c \circ f(X) - t^*)^2]$$
by definition of case 2.

Similarly,

$$ \mathbb{E}_X[\mathbf{1}_{\text{Case3}}(\hat{r}^{(n)}(X) + c \circ f(X) - f^*(X))^2] \geq \mathbb{E}_X[\mathbf{1}_{\text{Case3}}(f^*(X) - t^*)^2] $$

by definition of case 3.

Indeed, looking at the definitions of cases 2 and 3, we see that on these events, $f^*(X)$ and $\hat{r}^{(n)}(X) + c \circ f(X)$ aren't on the same side of the threshold $t^*$. Therefore, $t^*$ in an intermediate point between the two quantities, thus giving the inequality.

So at the same time, both

$$ \mathbb{E}_{D^{(n)}}[\mathbb{E}_X[\mathbf{1}_{\text{Case2}}(\hat{r}^{(n)}(X) + c \circ f(X) - t^*)^2]] \xrightarrow{n \to +\infty} 0 $$

and

$$ \mathbb{E}_{D^{(n)}}[\mathbb{E}_X[\mathbf{1}_{\text{Case3}}(f^*(X) - t^*)^2]] \xrightarrow{n \to +\infty} 0 $$

We have

$$ \mathbb{E}_X[\| R_f(X) - \hat{R}_f^{(n)}(X) \|] = \mathbb{E}[\mathbf{1}_{\text{Case1}} | \hat{r}^{(n)}(X) + c \circ f(X) - f^*(X) | + \quad (33) $$

$$ \mathbf{1}_{\text{Case2}} | \hat{r}^{(n)}(X) + c \circ f(X) - t^* | + \quad (34) $$

$$ \mathbf{1}_{\text{Case3}} | f^*(X) - t^* |] \quad (35) $$

$$ = \mathbb{E}[\mathbf{1}_{\text{Case1}} | \hat{r}^{(n)}(X) + c \circ f(X) - f^*(X) |] + \quad (36) $$

$$ \mathbb{E}[\mathbf{1}_{\text{Case2}} | \hat{r}^{(n)}(X) + c \circ f(X) - t^* |] + \quad (37) $$

$$ \mathbb{E}[\mathbf{1}_{\text{Case3}} | f^*(X) - t^* |] \quad (38) $$

$$ \leq \sqrt{\mathbb{E}[\mathbf{1}_{\text{Case1}} | \hat{r}^{(n)}(X) + c \circ f(X) - f^*(X) |^2] + \quad (39) $$

$$ \sqrt{\mathbb{E}[\mathbf{1}_{\text{Case2}} | \hat{r}^{(n)}(X) + c \circ f(X) - t^* |^2]} \quad (40) $$

$$ \sqrt{\mathbb{E}[\mathbf{1}_{\text{Case3}} | f^*(X) - t^* |^2]} \quad (41) $$

(39-41): by Cauchy Schwarz.

We take the expectation over $D^{(n)}$.

$$ \mathbb{E}_{D^{(n)}}[\mathbb{E}_X[\| R_f(X) - \hat{R}_f^{(n)}(X) \|]] \leq \mathbb{E}_{D^{(n)}}\left[\sqrt{\mathbb{E}[\mathbf{1}_{\text{Case1}} | \hat{r}^{(n)}(X) + c \circ f(X) - f^*(X) |^2]} + \quad (42) \right] $$

$$ \mathbb{E}_{D^{(n)}}\left[\sqrt{\mathbb{E}[\mathbf{1}_{\text{Case2}} | \hat{r}^{(n)}(X) + c \circ f(X) - t^* |^2]}\right] \quad (43) $$

$$ \mathbb{E}_{D^{(n)}}\left[\sqrt{\mathbb{E}[\mathbf{1}_{\text{Case3}} | f^*(X) - t^* |^2]}\right] \quad (44) $$

$$ \leq \sqrt{\mathbb{E}_{D^{(n)}}[\mathbb{E}[\mathbf{1}_{\text{Case1}} | \hat{r}^{(n)}(X) + c \circ f(X) - f^*(X) |^2]]} + \quad (45) $$

$$ \sqrt{\mathbb{E}_{D^{(n)}}[\mathbb{E}[\mathbf{1}_{\text{Case2}} | \hat{r}^{(n)}(X) + c \circ f(X) - t^* |^2]]} \quad (46) $$

$$ \sqrt{\mathbb{E}_{D^{(n)}}[\mathbb{E}[\mathbf{1}_{\text{Case3}} | f^*(X) - t^* |^2]]} \quad (47) $$

$$ \xrightarrow{n \to +\infty} 0 \quad (48) $$

(45-47): Jensen inequality.

(48): From the results above.

Which concludes the proof.

□
## C.5 HONEST TREE PARTITIONS

We show here that in particular cases of construction, honest trees are consistent estimators. We build on the results from Györfi et al. (2002); Meinshausen (2006); Wager & Athey (2017), which for the last two rely on the following assumptions (the assumption having the features bounded in $[0, 1]^d$ is said to be for notational purposes in Meinshausen (2006), the distribution of $X$ only need to be of compact support).

We define $d$ the dimension of the input space. The leaves are the regions of the partition so as to keep continuity with the notations, we will call them $\mathcal{L}_j$.

**Assumption 1:** $X$ is of compact support.

**Assumption 2:** The proportion of observations in a node, relative to all observations, is vanishing for large $n$, $\max_j n_j/n \xrightarrow{n \to +\infty}$ 0. The minimal number of observations in a node is growing for large $n$, that is $1/\min_j n_j = \xrightarrow{n \to +\infty}$ 0.

**Assumption 3:** When finding a variable for a splitpoint, the probability that variable $m = 1, \dots, d$ is chosen for the split point is bounded from below for every node by a positive constant. If a node is split, the split is chosen so that each of the resulting sub-nodes contains at least a proportion $\gamma$ of the observations in the original node, for some $0 < \gamma \le 0.5$.

Meinshausen (2006)'s Lemma 2 shows that the size of the leaves converge to 0 as $n$ grows, solely from the above assumptions, while Wager & Athey (2017) gives a rate of convergence in probability with its Lemma 2.

Due to the last part of assumption 1, we have that the total number of leaves relative to the number of all samples available is vanishing $\frac{J^{(n)}}{n} \xrightarrow{n \to +\infty}$ 0.

Therefore, we can apply Györfi et al. (2002)'s Theorem 4.2, which gives the consistency of the above partitioning estimate built from the honest tree.

# D DATASET CONSTRUCTION

## D.1 SEMI-SYNTHETIC DATA

We use the Weather dataset from TabReD (Rubachev et al., 2025) to build a rich dataset with knowledge of the ground-truth grouping loss. We retain the two most important features, as identified by permutation importance (Breiman, 2001). To obtain a semi-synthetic version (i.e., with access to $f^*$), we first fit a ground-truth model $f^*(X)$ for both half the training and test sets. Synthetic class labels are then generated for the remaining halves by resampling from $f^*(X)$ according to a Bernoulli distribution, $Y \sim B(f^*(X))$. This yields a semi-synthetic dataset where $X$ consists of the original features and $Y$ is sampled from $f^*(X)$.
<figure><img src="image_10.png" /><figcaption>Figure 8: Diagram of the semi-synthetic dataset construction</figcaption></figure>

## D.2 HATE SPEECH DATA

We replicate the expirmental setup from Perez-Lebel et al. (2025). The experiments use the feature embeddings from each model's penultimate layer as the input for all post-training methods. To simulate a variety of real-world conditions, we evaluate performance across 11 distinct cost-sensitive scenarios, each defined by a unique optimal decision threshold ($t^*$) ranging from 0.01 to 0.99. In each scenario, the final classification is determined by applying this threshold to the model's estimated probabilities. We define cost matrices $\Lambda \in \mathbb{R}^{2 \times 2}$ of the shape

$$ \Lambda \stackrel{\text{def}}{=} \begin{bmatrix} -1 & 0 \\ 0 & \Lambda_{11} \end{bmatrix} $$

with $\Lambda_{11} \in \mathbb{R}^-$. The values of $\Lambda_{11}$ are selected so that the optimal threshold $t^*$ takes values [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.975, 0.99].

**Experimental Procedure** The experiment is the following:

1. The text from all 14 datasets is processed by each of the 6 pre-trained models. For every sample, a triplet of data is recorded:

(i) the model's predicted probability, $f(X)$;

(ii) the model's internal feature embedding, $X$;

(iii) the ground-truth label, $Y$.

2. The collected data triplets are partitioned into a *train set* and a *test set*.

3. The post-training method are fitted on the *train set*.

4. Using only the *test set* data (prior to any corrections), the regret estimators are calculated.

5. The post-training corrections learned in the previous step are applied to the *test set*. The actual gain in expected utility is measured by comparing the performance of the corrected model against the original, uncorrected model, or between different post-training methods.
## D.2.1 MODELS USED

<table><caption>Table 1: Detailed description of all the models used for the quantifying suboptimality experiment.</caption><thead><tr><th>Model</th><th>HuggingFace</th><th>Latent Layer</th></tr></thead><tbody><tr><td>CNERG Hatexplain</td><td>Hate-speech-CNERG/bert-base-uncased-hate...</td><td>classifier</td></tr><tr><td>CNERG en MURIL</td><td>Hate-speech-CNERG/english-abusive-MuRIL</td><td>classifier</td></tr><tr><td>CNERG en mono</td><td>Hate-speech-CNERG/dehatebert-mono-englis...</td><td>classifier</td></tr><tr><td>CNERG portuguese</td><td>Hate-speech-CNERG/dehatebert-mono-portug...</td><td>classifier</td></tr><tr><td>CNERG tamil</td><td>Hate-speech-CNERG/tamil-codemixed-abusiv...</td><td>classifier</td></tr><tr><td>FB Roberta</td><td>facebook/roberta-hate-speech-dynabench-r...</td><td>classifier.out Project</td></tr></tbody></table>

## D.2.2 DATASETS USED

<table><caption>Table 2: Detailed description of all the datasets used for the quantifying suboptimality experiment.</caption><thead><tr><th>Dataset</th><th>HuggingFace or CSV</th><th>Split</th><th>Input</th><th>Target</th><th>Pos. class</th></tr></thead><tbody><tr><td>Tweets</td><td>tweets_hate..</td><td>train</td><td>tweet</td><td>label</td><td></td></tr><tr><td>Speech18</td><td>hate_speech18</td><td>train</td><td>text</td><td>label</td><td></td></tr><tr><td>Offensive</td><td>hate_speech_offensive</td><td>train</td><td>tweet</td><td>class</td><td>0</td></tr><tr><td>Davidson</td><td>krishan-CSE/...</td><td>train+test</td><td>text</td><td>labels</td><td>0</td></tr><tr><td>Gender</td><td>ctoraman/...</td><td>train+test</td><td>Text</td><td>Label</td><td>2</td></tr><tr><td rowspan="2">FRENK</td><td>classla/FRENK-hate-en</td><td>train+val+test</td><td>text</td><td>label</td><td></td></tr><tr><td>limjiayi/hatef..</td><td>train+val+test</td><td>text</td><td>label</td><td></td></tr><tr><td>Check</td><td>Paul/hatecheck</td><td>test</td><td>test_case</td><td>labelgold</td><td>hateful</td></tr><tr><td>Tweets 2</td><td>thefrankhsu/...</td><td>train+test</td><td>tweet</td><td>label</td><td></td></tr><tr><td>Open</td><td>parnoux/hate...</td><td>test</td><td>tweet</td><td>class</td><td>0</td></tr><tr><td>UCB</td><td>ucberkeley-dl...</td><td>train</td><td>text</td><td>hate_spec...</td><td>y > 0.5</td></tr><tr><td rowspan="5">Merged</td><td>tweetsHat..</td><td>train</td><td>tweet</td><td>label</td><td></td></tr><tr><td>hate_speech18</td><td>train</td><td>text</td><td>label</td><td></td></tr><tr><td>hate_speech_offensive</td><td>train</td><td>tweet</td><td>class</td><td>0</td></tr><tr><td>krishan-CSE/...</td><td>train+test</td><td>text</td><td>labels</td><td>0</td></tr><tr><td>ctoraman/gende...</td><td>train+test</td><td>Text</td><td>Label</td><td>2</td></tr><tr><td rowspan="5">Merged 2</td><td>classla/FRENK-hate-en</td><td>train+val+test</td><td>text</td><td>label</td><td></td></tr><tr><td>limjiayi/hatef...</td><td>train+val+test</td><td>text</td><td>label</td><td></td></tr><tr><td>Paul/hatecheck</td><td>test</td><td>test_case</td><td>labelgold</td><td>hateful</td></tr><tr><td>thefrankhsu/hat...</td><td>train+test</td><td>tweet</td><td>label</td><td></td></tr><tr><td>parnoux/hat...</td><td>test</td><td>tweet</td><td>class</td><td>0</td></tr><tr><td>DynGen</td><td>CSV: bvidgen/Dynamic...</td><td></td><td>text</td><td>label</td><td>hate</td></tr><tr><td rowspan="10">Merged3</td><td>tweets..</td><td>train</td><td>tweet</td><td>label</td><td></td></tr><tr><td>hate_speech18</td><td>train</td><td>text</td><td>label</td><td></td></tr><tr><td>hate_speech_offensive</td><td>train</td><td>tweet</td><td>class</td><td>0</td></tr><tr><td>krishan-CSE/Da...</td><td>train+test</td><td>text</td><td>labels</td><td>0</td></tr><tr><td>ctoraman/ge...</td><td>train+test</td><td>Text</td><td>Label</td><td>2</td></tr><tr><td>classla/FRENK-hate-en</td><td>train+val+test</td><td>text</td><td>label</td><td></td></tr><tr><td>limjiayi/hateful...</td><td>train+val+test</td><td>text</td><td>label</td><td></td></tr><tr><td>Paul/hatecheck</td><td>test</td><td>test_case</td><td>labelgold</td><td>hateful</td></tr><tr><td>thefrankhsu/hate...</td><td>train+test</td><td>tweet</td><td>label</td><td></td></tr><tr><td>parnoux/hate..</td><td>test</td><td>tweet</td><td>class</td><td>0</td></tr></tbody></table>
# E LLM EXPERIMENTS

## E.1 LIST OF LLMs USED

All the models were used with through the HuggingFace API.

<table><caption>Table 3: Summary of models used</caption><thead><tr><th>Model Name</th><th>Size</th><th>Company</th><th>Type</th></tr></thead><tbody><tr><td>DeepSeek-R1-Distill-Llama</td><td>8B, 70B</td><td>DeepSeek AI</td><td>Non Instruct</td></tr><tr><td>Gemma 2</td><td>9B, 27B</td><td>Google</td><td>Instruct &amp; Non Instruct</td></tr><tr><td>Llama-2</td><td>13B, 70B</td><td>Meta</td><td>Instruct &amp; Non Instruct</td></tr><tr><td>Llama-3</td><td>8B, 70B</td><td>Meta</td><td>Instruct &amp; Non Instruct</td></tr><tr><td>Llama-3.1</td><td>8B</td><td>Meta</td><td>Instruct &amp; Non Instruct</td></tr><tr><td>Llama-3.2</td><td>1B, 3B</td><td>Meta</td><td>Instruct &amp; Non Instruct</td></tr><tr><td>Llama-3.3</td><td>70B</td><td>Meta</td><td>Instruct</td></tr><tr><td>Mistral</td><td>7B (v0.3)</td><td>Mistral AI</td><td>Instruct &amp; Non Instruct</td></tr><tr><td>Mixtral</td><td>8x7B (v0.1)</td><td>Mistral AI</td><td>Instruct &amp; Non Instruct</td></tr><tr><td>Orca-2</td><td>13B</td><td>Microsoft</td><td>Non Instruct</td></tr><tr><td>phi-4</td><td>14B</td><td>Microsoft</td><td>Non Instruct</td></tr></tbody></table>

# F ADDITIONAL EXPERIMENTS

## F.1 CORRELATION OF THE TOTAL RISK WITH RETRAINING

<figure><img src="image_11.png" /><figcaption>Figure 9: Correlation between fine-tuning gain and excess 0-1 risk estimation. The y-axis shows the expected cost reduction of fine-tuning compared to the base model. Each point corresponds to a (model, dataset, cost matrix Λ) triplet. The dashed line corresponds to y = x.</figcaption></figure>

## F.2 COMPUTING THE RISKS ON AN EMBEDDING OF THE QUERIES

For the comparison with a router where we fit the models on an embedding of the prompt, we use Hu et al. (2024)'s kNN baseline with the best hyperparameter they found ($k = 40$ and the prompts are embedded by the all-MiniLM-L12-v2 embedding model). Results are shown on Figure 10.
<figure><img src="image_12.png" /><figcaption>Figure 10: **Cascades: our risk estimator gives better accuracies at a lower cost.** Routers fitted on the embeddings of the prompts given to the LLMs. Each point corresponds to a cascade (an LLM sequence). The y-axis is Acc(ours) - Acc(baseline). Values are positive when our cascade improves accuracy over the baseline. The x-axis is Cost(ours)/Cost(baseline). For values below 1, our model is less expensive. Our cascade based on $\hat{R}^{CL} + \hat{R}^{GL}$ ($t=0$) is compared to **Left:** the best LLM in the cascade, **Middle:** A cascade based on $\hat{R}^{CL}$ only ($t=0$). **Right:** the predictive router baseline ($\lambda=100$).</figcaption></figure>

## F.3 CASCADE WITH THE CONFIDENCE SCORE BASELINE ON THE OUTPUTS OF THE LLM

We also studied the traditional approach of confidence-based cascades, described in section 4.4. This approach fails compared to our methods because the models are miscalibrated, making the convergence towards a definite choice much harder for the confidence-based cascade.

<figure><img src="image_13.png" /></figure>

(a) **Cost-accuracy tradeoffs** when varying the threshold of risk $t$ and willingness to pay $\lambda$ on the Llama 3 instruction-tuned family cascade. Adding the confidence-based traditional approach for cascades.

<figure><img src="image_14.png" /></figure>

(b) **Cost-accuracy tradeoffs** when varying the threshold of risk $t$ and willingness to pay $\lambda$ on the Llama 3 instruction-tuned family (without Llama 3 3B) cascade. Adding the confidence-based traditional approach for cascades.

<caption>Figure 11: Using the traditional confidence-based approach for cascades doesn't work in our setting.</caption>
<figure><img src="image_15.png" /><figcaption>Figure 12: Gain in accuracy vs relative cost of the cascade method between our proposed risk-based cascade and the traditional confidence-based cascade. Confidence threshold set at 0.7 and risk threshold set at 0.</figcaption></figure>

## F.4 TREE DEPTH ABLATION STUDY

In this section, we study the influence of the depth of the tree in the cascading results of section 4.4. We use the exact same experimental setup, only changing the `min Samples Leaf` parameter of the honest tree for the partitioning estimate between {5, 100, 1000, 10000}. Overall, from figs. 13 to 17 our results are very robust to the `min Samples Leaf` value. The results of the cascading experiments hold up to `min Samples Leaf = 1,000` (which correspond to trees of depth around 6) and degrade at `min Samples Leaf = 10,000` (trees of depth 3 approximately). This is expected as very coarse trees reduce the capacity of $\mathcal{R}^{GL}$ to improve over $\mathcal{R}^{CL}$ alone.
1458

1459

1460

1461

1462

1463

1464

1465

1466

1467

1468

1469

1470

<figure><img src="image_16.png" /></figure>

(a) minSamples_Leaf=5

<figure><img src="image_17.png" /></figure>

(b) minSamples_Leaf=100

1471

1472

1473

1474

1475

1476

1477

1478

1479

1480

1481

1482

1483

<figure><img src="image_18.png" /></figure>

(c) minSamples_Leaf=1000

<figure><img src="image_19.png" /></figure>

(d) minSamples_Leaf=10000

1484

1485

1486

1487

1488

1489

1490

1491

1492

1493

1494

1495

1496

1497

1498

1499

1500

1501

1502

1503

1504

1505

1506

1507

1508

1509

1510

1511

<caption>Figure 13: **Cost-accuracy tradeoffs** when varying the threshold of risk *t* and willingness to pay *λ* on the Llama 3 instruction-tuned family cascade. Changing the *min_sample Leaf* parameter of the honest tree.</caption>
1512

1513

1514

1515

1516

1517

1518

1519

1520

1521

1522

1523

1524

1525

1526

1527

1528

1529

1530

1531

1532

1533

1534

1535

1536

1537

1538

1539

1540

1541

1542

1543

1544

1545

1546

1547

1548

1549

1550

1551

1552

1553

1554

1555

1556

1557

1558

1559

1560

1561

1562

1563

1564

1565

<figure><img src="image_20.png" /><figcaption>Figure 14: Cascades: our risk estimator gives better accuracies at a lower cost. Same experimental setup as section 4.4, minSamples_Leaf=5 for the honest tree.</figcaption></figure>

<figure><img src="image_21.png" /><figcaption>Figure 15: Cascades: our risk estimator gives better accuracies at a lower cost. Same experimental setup as section 4.4, minSamples_Leaf=100 for the honest tree.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>Figure 16: Cascades: our risk estimator gives better accuracies at a lower cost. Same experimental setup as section 4.4, minSamples_Leaf=1000 for the honest tree.</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>Figure 17: Cascades: our risk estimator gives better accuracies at a lower cost. Same experimental setup as section 4.4, minSamples_Leaf=10000 for the honest tree.</figcaption></figure>