Rational Tuning of LLM Cascades via Probabilistic Modeling
Authors: Michael J. Zellinger
Matt Thomson
Venue: Accepted by TMLR ()

Abstract: Understanding the reliability of large language models (LLMs) has recently garnered significant attention. Given LLMs' propensity to hallucinate, as well as their high sensitivity to prompt design, it is already challenging to predict the performance of an individual LLM. However, the problem becomes more complex for compound LLM systems such as cascades, where in addition to each model's standalone performance, we must understand how the error rates of different models interact. In this paper, we present a probabilistic model for the joint performance distribution of a sequence of LLMs, which enables a framework for rationally tuning the confidence thresholds of a LLM cascade using continuous optimization. Compared to selecting confidence thresholds using Bayesian optimization, our parametric Markov-copula model yields more favorable error-cost trade-offs, improving the area under the error-cost curve by 4.3% on average for cascades with $k\geq 3$ models. In the low-sample regime with $n \leq 30$ training examples, the performance improvement widens to 10.2%, suggesting that our framework's inductive assumptions about the interactions between the error rates of different LLMs enhance sample efficiency. Overall, our Markov-copula model provides a rational basis for tuning LLM cascade performance and points to the potential of probabilistic methods in analyzing systems of LLMs.

Published in Transactions on Machine Learning Research (05/2025)
Rational Tuning of LLM Cascades via Probabilistic Modeling
Michael J. Zellinger and Matt Thomson
Reviewed OpenReview:
Abstract
Understanding the reliability of large language models (LLMs) has recently garnered signif- icant attention. Given LLMs' propensity to hallucinate, as well their high sensitivity to prompt design, it is already challenging to predict the performance of individual LLM. However, the problem becomes more complex for compound LLM systems such as cascades, where in addition to each model's standalone performance, we must understand how the error rates of different models interact. In this paper, we present a probabilistic model for the joint performance distribution of a sequence of LLMs, which enables framework for rationally tuning the confidence thresholds of LLM cascade using continuous optimization. Compared to selecting confidence thresholds using Bayesian optimization, Markov-copula model yields more favorable error-cost trade-offs, improving the area under the error-cost curve by 4.3% on average for cascades with k 2 3 models. In the low-sample regime with n < 30 training examples, the performance improvement widens to 10.2%, suggesting that our framework's inductive assumptions about the interactions between the error rates of different LLMs enhance sample efficiency. Overall, our Markov-copula model provides a rational basis for tuning LLM cascade performance and points the potential of probabilistic methods in analyzing systems of LLMs.
parametric
1 Introduction
As LLMs become workhorses of the modern computing stack, systems of LLMs have received significant at- tention (Zaharia et al., 2024; Chen et al., 2024b). These approaches make it possible to adapt computational spending to the performance requirements at the query or task level (Kag et al., 2023; Chen et al., 2023), yielding significant gains in operational efficiency. These gains are achievable even when accessing LLMs entirely via black-box API calls, by switching between models of different capabilitics.
However, moving from single LLMs to LLM systems introduces significant additional complexity. To find the system's optimal operating point, it important understand not just the performance of individual models but also the interactions between their error rates. For example, in simple two-model LLM cascade in which a small model delegates difficult queries to large model, the large model's error rate increases conditional on receiving query, since the small model's confidence gating induces an adverse selection (Zellinger and Thomson, 2024).
In this paper, we present parametric probabilistic model for the joint distribution of the calibrated con- fidences of a sequence of i LLMs,, providing a rational basis for understanding the performance of LLM cascades. We focus on cascades whose constituent models are ordered by size, from smallest. to largest. Our probabilistic model is based on Markov factorization, leveraging the insight that LLMs similar in size are more predictive of cach other's confidence. After using logistic regression to calibrate cach LLM's confidence, we account for the pairwise interactions between subsequent LLMs' error rates using bivariate copulas, providing data-efficient model of cascade performance that performs well with as few as n S 30 training examples across six benchmarks.
Our Markov-copula model makes it possible to tune the confidence thresholds of an LLM cascade using continuous optimization.

---

Published in Transactions on Machine Learning Research (05/2025)
Compared to selecting these thresholds via Bayesian optimization, our Rational Tuning framework yields increasingly better error-cost trade-offs as cascade length grows. For cascades with k 2 3 models, our hod improves the area under the error-cost curve by 4.3% on average. Compared to high-resolution grid search, the improvement is 2.0%. At the same time, our algorithm significantly improves runtime scaling comparod to grid search. For example, we reduce scaling with respect to the cascade length from exponential to low-order polynomial, making it much faster to tune longer cascades consisting of k 2 models.
Relative to the prior literature LLM cascades, main contributions
follows:
We propose a generative probabilistic model for the joint. distribution of the calibrated confidences of a sequence of LLMs, based on a Markov factorization, copula modeling, and mixed discrete- continuous marginal distributions. We demonstrate that our model fits the empirical data well: on the test sets, we report average Cramér-von Mises statistics of vnCvM 0.006 for the copula models and CCM 4% for the mixed discrete-contimuous marginal distributions.
Building on our Markov-copula model, we develop an algorithm for tuning the confidence thresholds of an LLM cascade using continuous optimization, based on an analytic probabilistic model. We demonstrate that as cascade length grows, our method increasingly outperforms the error-cost trade- offs obtained with Bayesian optimization and high-resolution grid search baselines. addition, relative grid search our method significantly improves the computational complexity of finding optimal confidence thresholds, turning the dependencies on cascade length and the desired resolution of the error-cost curve from intractable and high-order polynomial into low-order polynomial and linear, respectively.
In addition, we present comprehensive evidence that simple hyperparameter-free feature transforms sig- nificantly improve the performance of calibrating LLM confidence with logistic regression (Zellinger and Thomson, 2024), demonstrating 28.2% average reduction in expected calibration error across 10 LLMSs and 6 benchmarks.
2 Background and Related Work
given a predefined token vocabulary V, a large language model (LLM) M defines an autoregressive probability distribution ~ p(.,,.....,& (x) for the next token (f1,..,,h,) y In this work, we focus the overall input-output behavior of the model M. We let stand for the entire query consisting of tokens (41,.., (n) and write M for the sequence of tokens (In+1, .n...... obtained when repeatedly sampling tj+1 ~ P..Itt.....tt) for j 2 n until encountering stop token to E V.
Language Models:
given a sequence of tokens
Language Model Cascades: a length-k LLM cascade C Mi + Mr routes an incoming query æ sequentially from model M; to M,+1 based confidence measures Φ; (x) E 1). When æ reaches M,. the cascade returns M,(x) if (x(x) 6i, where d; E (0, 1) is confidence threshold for model M,. Otherwise, C forwards the query to the next model, M,+1. Writing Cia for the subcascade M; - + M, consisting of the last k - -it1 1 models, the output C(x) of the overall cascade is defined recursively as
Mi ( if (x) أ or |C| =
C(x)
(1)
C2*(x) otherwise,
where |C*| is the length of the cascade, for example |C2.1l = =
Different authors have recently explored LLM cascades. Chen et al. (2023) have shown that it is possible to approach the performance of a large LLM at much lower cost by initially sending queries to a small model; Aggarwal et al. (2024) present a flexible cascading approach based on a POMPD router; Yue et al. (2024) propose LLM cascades spocifically for mathematical reasoning benchmarks; and Gupta et al. (2024) consider uncertainty individual token position within longer generations. While many of these approaches use standard uncertain quantification techniques for I.IMss (discussed below), some trained neural networks
2

---

Published in Transactions on Machine Learning Research (05/2025)
for making the decision of forwarding query I to the next model. Neural network approaches have the potential to make more finegrained distinctions between the capabilities of different LLMss,, but may require large amounts (n > 1000) of task-specific training data to perform well.
Jitkrittum et al. (2024) discuss the limits of forwarding queries based purely on the confidence level of the current model, proposing to train a cascading decision that takes into account not only the current model's probability of correctness, but also that of the following model. In addition, Wang et al. (2024) explore finetuning LLMs to make them more effective part of cascade. Other methods for LLM orchestration use routers that directly forward queries to suitable LLMs in one-to-many architecture (Ding et al., 2024; Kag et al., 2023; Sakota et al., 2024; Hari and Thomson, 2023). In addition, some work has explored recombining the string outputs of several smaller models to yield improved performance (Jiang et al., 2023).
Uncertainty Quantification and Calibration: LLMs within a cascades require the means to tell "casy" queries from "difficult" ones. Several authors have proposed methods for quantifying this uncertainty. These methods work in different ways. Some draw on the LLMs' intrinsic next-token probabilities (Hendrycks and Gimpel, 2018; Plaut et al., 2024), while others use prompting to elicit confidence statements (Lin et al., 2022a; Kadavath et al., 2022; Xiong et al., 2024). Some sample repeatedly from the LLM and measure the consistency between different answers (Wang et al., 2023; Manakul al., 2023; Farquhar el al., 2024; Lin et al., 2024), while others train lightweight probes on top of an LLM's hidden embeddings (Azaria and Mitchell, 2023; Ren et al., 2023, Chen et al., 2024a; Kossen al., 2024). Finally, it is even possible to evaluate uncertainty in a largely unsupervised way (Burns et al., 2024).
Calibration of LLM uncertainty refers to the question of whether numerical confidence scores reflect the true probabilities of error. Methods relying on LLMs' next-token probabilities face the challenge that these probabilities are typically overconfident, at least for instruction-tunet LLMs (Ouyang et al., 2022: OpenAI, 2024). Although calibration is not required for forwarding queries based on confidence, it important for accurately prodicting error rates and desirable for gaining insights into system performance. Many techniques for calibration have been proposed (Platt, 1999; Zadrozny and Elkan, 2002; Naeini et al., 2015, Guo et al., 2017; Jiang et al., 2021). Temperature scaling, which divides LLM's log probabilities by suitable constant factor (typically >1), is often favored for its simplicity.
copula. models are statistical tools for modeling joint probability distributions. They are widely used in applications. For example, in quantitative finance they are used to price complex securities such as baskets of loans whose repayments depend on multiple borrowers. Mathematically, joint cumulative distribution function whose marginals all follow the uniform distribution. The idea behind copula modeling is that, in order to specify an arbitrary joint distribution p(z, v), it suffices to specify the marginals p(x), p(y) along with a copula accounting for the correlation between x and y. This result is known as
Copula Models:
copula is a
Theorem 1 (Sklar's Theorem). Let Fxy be joint distribution function with marginals Fx and Fy. Then there exists a copula C such that for all x, y E ER,
= C(F)()))
(2)
Conversely, if C a copula and Fx and Fy are distribution functions, then the distribution function Fxy defined by (2) is a joint distribution function with marginals Fx and Fy.
For a proof and further discussion, see Nelsen (2006). Intuitively, copula modeling builds on the probability integral transform principle: if X is continuous random variable with distribution function Fx(-), then (X) follows uniform distribution (Casella and Berger, 2002). In our application to LLM cascades, we model the joint probability p( (x) of the calibrated confidences of models M, and M,-1 using Gumbel copula. This copula depends on single correlation parameter 0, which can be easily calculated from the rank correlation (Kendall's 7) of the two variables.
'f particular interest the potential for detecting rare cases when small model correctly answers a query on which larger model fails.
3

---

Published in Transactions on Machine Learning Research (05/2025)
3 Rational Tuning of LLM Cascades via Probabilistic Modeling
3.1 Markov-Copula Model
Our probabilistic model for the joint distribution of LLM confidences is based on calibrated confidence scores. We use logistic regression to transform a raw confidence signal Praw into the calibrated confidence score
Φ T (Praw);
where 0 are the parameters of the logistic regression. The calibrated confidence & estimates the model's probability of correctness based the raw confidence signal Praw- We calibrate each model separately, resulting in functions 1,,-., ..., & for the models M..... Mr of cascade M1 + -... Mg. See Section 4.1 for more details. Since the confidence signal Praw depends the query we also write & Φ(x) in slight abuse of notation.
Our probabilistic model for the joint distribution of the calibrated confidences Φ (1()), (x(2)) consists of three parts. First, we model the marginal distribution of the calibrated confidence of each individual LLM in the cascade. Second, we model the correlation between the calibrated confidences Pz(x), (x)1(x) of adjacent models using copulas. Finally, construct the full joint distribution by combining the conditional probabilities p(0++1l0;) using the Markov property.
Specifically, given cascade M1 + 7 M2 with trained confidence calibrators II,..., ..... Lr. we first fit parametric univariate distributions Fi(6:10,) to model the true marginal distributions P2 S 6.). Second, we account for the correlation between adjacent models by fitting copulas C,,+1(5))). Each copula C2s+1 makes possible to compute the joint distributtio ((..)((.))) of (f, 11+1) via
F 0.+1) (F, F((())))))))()))))))
(4)
by Theorem 1. Finally, we estimate joint probabilities p( S Φ1, 22 S Ø2, ..., Db S Φn) by relying the Markov assumption
P(O, S 0,12,-1 S 2 0. 2 14) P(D. s 0, 11-1),
(5)
which implics
P(0: S $1, $2 o2..... 4 0.) 2))=2 ~ P(0: <1) II
P(0 20110 S $5-1).
(6)
j=2
for any i = 2,..., le and o1,-..; 1). We study the validity of assumption (5) in Section 4.3.
3.2 Parameter Inference for the Probabilistic Model
In this section, we describe in detail the components of our parametric probabilistic model and how we infer their parameters.
Contimoous-discrete mixture of scaled beta distributions: to model the marginals of calibrated confidence, we must account for the possibility that LLMs sometimes return perfect confidence Praw = 1.0, possibly as a result. of performance optimizations such as quantization (Dettmers et al., 2024; Proskurina et al., 2024). Depending on the LLM and the task, almost half of all queries may return perfect confidence, as is the case of GPT-4o Mini on the MMLU validation set (45.7)).
To accommodate the resulting discrete probability masses at the minimum and maximum calibrated confi- dence values Φmin and @max, we mixed continuous-discred distribution based on mixture of two beta. distributions. Specifically, we use the distribution function
F(olwi, W2, Pminy Pmaxi n, 01, B1, 022 (2) Wmin & S (1)) + Wmaxx 8 (x) + (1 Winin Winaxx F Fmixtuur (Ø), (7)

---

Published in Transactions on Machine Learning Research (05/2025)
where 8g is the distribution of a point mass 2, and F mixtuue (0) is
Fiixxturo (elxturro@/@miin minn Omaxi 011, 1i,,B15a,, 82) T Pal Φ Umin
Φ Umin |a2, 32).
(a1, + (1 (1
(8)
mmax
Pmin
Dmax
Pmin
Here, Fa B) is the beta distribution with pdf fa(æ)a, 8) -0 - =x)--1 for a E (0, 1).
We infer the parameters of the model (7) as follows. First, we estimate the minimum and maximum calibrated confidences @min and @max by their observed minimum and maximum values on the training set. We estimate the corresponding discrete probability masses Wmin and Wmax by simple counting. Finally, to estimate the mixture of beta distributions (8), use the appectation-maxiiiann
algorithm (Dempster et al., 1977).
Gumbel copula: to model the correlations between the calibrated confidences of pairs of LLMs, we use the Gumbel copula Co(u, v) given by
(-(1)))))
- (-(os
))))
[1008
+ log !
Co(u, = exp
(9)
where 0 > measures the degree of correlation between and v. To fit 0 from empirical data, we use the relationship
(10)
where T is Kendall's rank correlation coefficient (Nelsen, 2006).
3.3 Tuning the Confidence Thresholds
The purpose of the Markov model (6) is to obtain optimal error-cost tradeoffs for an LLM cascade C' by tuning the confidence thresholds. We formulate the optimization problem
f arg min (1 (Po(Crreet)) À Eo [Cost]]
(11)
where 0 E +R denotes the confidence thresholds the user's s cost sensitivity. Setting À = 0 means that cost is irrelevant, whereas À >> 0 penalizes the use of expensive models. To compute the efficient frontier of optimal (P(Correct), E(Cost]) tuples, we solve (11) for different values of the cost sensitivity X.
The Lagrange multiplier À 2 0 indicates
Since À has known relationship with the expected cost, it is not clear how choose À to obtain uniform coverage of the efficient frontier. In practice, start with very small values of À and set
A (1+r)A,
(12)
for some > 0, until the cost constraint is stringent enough to make the expected cost equal to the least expensive model's expected cost. Typically, setting between 0.25 and 1 performs well. For any potential gaps in coverage, we adaptively interpolate the optimal thresholds. Specifically, if X < ((+1) yield optimal
thresholds g( and ((+1)) and the gap 10 (0((+t)) - g =10 1o - for any individual threshold exceeds probability mass q based on the distribution of the calibrated confidence Φ, we insert
0(++1/2) (0 + 0(+11)/22
(13)
into the list of optimal thresholds between g( and (6-11, We repeat the infilling procedure (13) until no gaps remain at level q. We have found q < 0.2 perform well.
Efficient computation and optimization of the objective: solving the minimization problem (11) requires computing cascade's probability of correctness and expected cost for candidate confidence thresh- olds (Ø1.... x k To compute these quantities, we rely on the decompositions (14) and (15) presented in
5

---

Published in Transactions on Machine Learning Research (05/2025)
Proposition 2. Consider a cascade M1 +....>> Mr with confidence thresholds ....... Qk-1). Assume that the distribulion functions for the calibrated confidences 4; salisfy (5), for i = 1, ..,....,k Assume further that the expected numbers of input and output tokens, T(( and T( for each model i are independent of the calibrated confidences & 1 Dk. Then the probability of correctness P( Correct) and expected cost E[Cost] for the cascade are
P P(Cormet)) = S (I(W) dP(w)
(14)
10,2011
+ Σ
1 S (1)
P مما ( / k,(w) / S $Q-11)
1=2
[|Cos = (1 - P(21- S (i)) B[G]]
(15)
I P( S م ) ) (1- P(D, S 0 S <1-11)
+
1 S )1)
Σ j=1
B[Cg},
2=2
where Cj is the cost per query of model i. Specifically, if ((m) and 7 (ou)) are the costs per input and output token, C; (( + 7 (ou))) To simplify the notation, we let Øk == -co (although. there is no confidence Uhreshold for the final model in the cascude).
Proof. See Appendix A for a proof.
By leveraging the structure of the summands in Proposition 2, we efficiently compute (14) and (15) in O(k) time, where k is the length of the cascade. See Appendix B for the algorithm. To solve the minimization prob- lem (11), we use the L-BFGS-B optimizer, a low- memory version of the Broyden Fletcher Golddfrrb- Coldfarb-Shanno algorithm (Liu and Nocedal, 1989) modified to handle simple box constraints.
Smoothness of the objective: although our Markov-copula model uses mixed discrete-continuous marginals, the objective (11) is smooth because we restrict each threshold á to vary only inside the in- terior of the interval (Ømin: Φmax). where the marginal distributions of calibrated confidence are smooth. Leaving out the boundary {ømin: } results in no loss of generality because selecting & E {omin Φmaz } is equivalent to dropping the model from the cascade (if & = Φmax) or dropping all subsequent models (if a @miu). Within our framework, it is possible to carry out such model selection by evaluating subcascades. After fitting copula models for all pairs of models (rather than only adjacent pairs), evaluating subcascades involves little computational overhead.
4 Results
4.1
Methodology
Forwarding Queries: the models in our cascades decide whether to forward queries by thresholding the calibrated confidence o (praw), where Praw is the raw confidence signal. We obtain Praw from the model- intrinsic next-token probabilities. On multiple-choice tasks, we take the maximum probability among the answer choices (Hendrycks and Gimpel, 2018; Plaut et al., 2024). In the natural language generation case, we first generate the answer, then send a follow up verification prompt to the model asking "Is the proposed answer <answer> true? Answer only Y or N." We use the probability of the Y token as the confidence signal Praw- Our prompt templates are available in Appendix C.
Since we focus on providing techniques compatible with black-box LLM inference via third-party APIs, we leave consideration of hidden layer-based confidence signals to future work. In addition, we do not consider resampling methods such as semantic entropy (Farquhar et al., 2024). Such methods are compatible with Blck-boxx inference, but in the context of LLM cascades, their computational overhead appears prohibitive. For example, at the time of writing inference of Llama3.1 405B typically costs 15 times more than inference

---

Published in Transactions on Machine Learning Research (05/2025)
of Llama3.1 8B. In this case, it is likely preferable to directly run the 405B model once rather than forward query based on k x 15 resamples of the 8B model. See Appendix D for table listing small-large model pairings from Meta, Anthropic, and OpenAl, along with their price differentials.
Confidence Calibration: raw loken probabilities of instruction-tuned LLMs are typically poorly calibrated (Ouyang et al., 2022; Brown et al., 2020; OpenAT, 2024; Plaut et al., 2024). However, calibration is important for accurate error prediction. To obtain calibrated confidence scores, we use logistic regression. We favor this approach over temperature scaling since it yields values and other statistical metrics that are useful for diagnosing calibration issues, especially in a low-data scenario.
Unfortunately, the overconfidence of the raw token probabilities makes the distribution of raw confidence signals highly peaked. The raw token probabilities accumulate near 1.0, making tiny changes in confidence (for example, Praw 0.98 Praw 0.99) highly consequential. To enhance the calibration performance of logistic regression, a pre-processing step apply hyperparameter- free feature transformations that spread out the overconfident probabilities via asymptotes near Praw = 0.0 and Praw = 1.0. Following Zellinger and Thomson (2024), on multiple-choice tasks we use the transformation
1
[(Praw) llo
(16)
Praw
whereas natural language generation tasks, we use
log
i
{(Peaw)
P-Pa
(17)
log
p < 1
Praw
Importantly, these feature transformations do not require any hyperparameter tuning.
Unfortunately, models sometimes return perfect certainty Praw = 1.0 or Praw = 0.0, making (16) and (17) blow up. To address this problem, we reassign all observations with infinite & the maximum of the finite values of ç. In other words, we define
Smax = max (a (Peaws y) E D. Praw < <c0),
(18)
where D is the training set consisting of pairs (Praws y), and y E {0, 1} indicates correctness of the model's answer.? 2 We set all observations where { = 3o to Emaxs and treat Emin analogously.
we evaluate our probabilistic model and the error-cost curves of LLM cascades six language modeling benchmarks including MMLU (Hendrycks et al., 2021); MedMCQA (Pal et al., 2022); TriviaQA (Joshi et al., 2017); XSum (Narayan et al., 2018); GSM8K (Cobbe et al., 2021); and TruthfulQA (Lin et al., 2022b). These lasks include general-purpose knowledge and reasoning, domain-specifie QA, open-domain QA, summarization, mathematical reasoning, and truthfulness in the face of adversarially chosen questions.
Benchmarks:
For each benchmark, we use 300 examples for training, and 1000 examples for testing, except on MMLU and TruthfulQA. On MMLU, the dev set contains only 285 examples, of which we use all. The validation set consists of 1531 examples and is divided into different subjects; to avoid bias from subject selection, we take all 1531 validation examples for testing. On TruthfulQA, the entire data set consists only of 817 observations, of which we randomly select 300 for training and the remaining 517 for testing.
Importantly, we run each benchmark in a zero-shot manner, since we believe this setting faithfully reflects off-the-shelf use of LLMs in practice. Appendix C gives the prompt templates we used for each benchmark. To conveniently transform and calibrate the raw confidence scores, track the numbers of input and output tokens, and monitor cost, we ran our evaluations using preliminary version of the niagara Python package for LLM cascading. Code for reproducing the results of the paper is available on 3it.
Note we do require knowing the actual answer model, only whether it was correct.
&odde reproducing the results of the paper is available github. wtw..//ewwwwnnnnaatiiiiieeoeeeee 111m-111meeeee -1n--cascade.
7

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Overall performance of language models across tasks, evaluated on the n s 1000 test sets. %Corr is the percentage of correct answers, %ECE the expected calibration error (after training on the n = 300 training sets), and %Cert is the percentage of queries for which a model returns log probabilities indicating certainty (-oc 0.0).
MMLI
MedMcQA
TriVSIAA
XSaam
GSMMIK
Truthfulig
Model
MrCre
'SPCE
15Corr
RECE
CCorr
ISBCE
15Ceet
!SCorr
SEECT
7tCrrt
St.Ceer
RECE
'5Cret
'5Corr
SLOrrt
8.7
Bumu8. 3-3h
79.3
lama8.1-4b
4.3
9.4
6.2
503.
3.3
88.3
4.7
96.3
100
iama5.2-TuB
83.5
2.4
114
72.8
9-4
040
92.8
235
198.3
6.0
0.0
96.M
2.9
088
98.4
3.7
110
Bamad 1-40sh
65.2
8.0
0.1
13.7
10.8
0.0
94.0
3.0
9.1
83.9
3.4
07.1
1.0
0.5
09.3
3.6
0.0
ewenz B-22bec
0.3
8.9
191-
4.3
26.1
87.4
20.9
15.7
5.3
27.4
9.8
87.6
$8.8
56.4
21.3
ger-to
63.0
1.6
22.3
70.5
2.6
7.
H6.3
8.1
0.
98.0
0.7
100
95.9
811
13
73.1
0.8
Averuge
7.4
6.2
56.7
06.1
Table 2: Expected calibration error for logistic regression-basel calibration, with (%ECE) and without (XECE.TF) applying the nonlinear transformations (16) and (17) as pre-processing step. All values are computed the 1000 test sets, after fitting the logistic regressions the z 300 training sets. For each benchmark, bold font indicates the better performance. The column %A shows the reduction in ECE when using the transformations.
MedMucQA
TriviatAA
X3rmm
GSMIK
TruthnaiQA
XBCE
SIECE
SECE
X&
Bomad.2:10 lb
3.8
6.1
-37.7
8.7
9.3
-11.2
1.5
5.5
0.0
3.9
8:7
7.5
13:.1
13.2
-0.3
4.0
6.5
0.0
4.0
-55.9
0.88
10.0
-210
4.5
14.0
-018
3.6
-3.7
0.5
0.0
1.5
-2.2
1emad.1-10h-
9.4
9.8
19.5
9.5
1-45h-1-405h
2.8
10.4
-22.1
10.8
12:2
-910
3.8
19
388
3.4
10.2
-47.
1.9
3.6
-17.
5.6
10.8
-1.1
quee2.
5.3
13.9
161.8
0.2
1:55
-7.3
10.0
00.3
4.8
02.
3.2
5.9
qwan2
4.9
7.0
-99.
4.0
4.8
11.2
-97.
3.1
12.6
-39.
2.1
-61.7
0.7
-206
3.1
-56.5
5.8
-56.7
Evaluation: to evaluate whether a model's answer is correct on open-ended questions, we use Anthropic's Claude 3.5 Sonnet model as a judge. Note that this judging task is relatively easy since the open-ended benchmarks provide reference answers. For example, on TruthfulQA, for Claude a list of correct and incorrect reference answers, provided by the authors of the benchmark (Lin et al., 2022b). On XSum, we do not use the one-line reference summaries and instead follow G-Eval (Liu et al., 2023) to evaluate proposed summary in terms of its coherence, consistency, fluency, and relevance (Kryšcińskii et al., 2019). We ask Claude to score each dimension on scale 1-5. We consider a summary to correct if it atlains a perfect score (5) in each dimension.
include in the evaluation prompt
Language Models: we work with models from Meta's Llama3 series (1B-405B), Alibaba's Qwen scries (Qwen2.5 32B Coder and Qwen 72B), and OpenAI's GPT models (GPT-4o Mini and GPT-4o). All models are instruction-tumed. We used the OpenAT APT to run inference with GPT-4o Mini and GPT-4o, and the Fireworks API for all other models.
4.2 Performance Summary
Tables and 2 show the overall performance all the language models across tasks, including the calibration performance. We measure calibration terms of the expected calibration error (ECE), which we compute adaptively by bucketing confidence scores into 10 bins based on the deciles of their distributions. Tables 1 and 2 yield several interesting findings.
First, some of the models often return raw log probabilities indicating certainty (-o0 or 1.0). This tendency varies strongly by model family. OpenAl's GPT models are especially prone to certainty: on MMLU, for example, GPT-4o Mini returns raw confidence 1.0 on 45.7% of queries, while GPT-4o does so on 22.7% of queries. By contrast, 1 405B returns perfect confidence only on 0.1% of queries.
Second, the test ECE for our calibration scheme varies by model and by benchmark. The benchmark yielding the poorest calibration is MedMCQA, giving average test. ECE of 7.4% across models. However, some
8

---

Published in Transactions on Machine Learning Research (05/2025)
models give exceptional calibration performance across benchmarks. GPT-4o stands out: its test ECE never exceeds 4.8,, which is its ECE MML.U.
Overall, we observe that our calibration scheme performs satisfactorily across benchmarks and models, with most benchmarks reporting an average test ECE below 5%. Table 2 ablates the importance of the hyperparameter-free feature transforms (16) and (17) for obtaining effective calibration. Applying these transformations results in much lower test ECE scores, reducing them by 28.2% on average. Figure 10 in Appendix E further verifies calibration by showing that, across models and benchmarks, rejecting queries for which the calibrated confidence is < - q approximately lowers the test error rates to q.
4.3 Goodness-off Ff-Fit of the Markov-Copula Model
In this section, we show that our probabilistic model fits the empirical data well. We start by presenting evidence that the Markov assumption (5) approximately holds. Second, show that our Gumbel copula models successfully account for correlations between the error rates of different LLMs, measured by low square-rooted Cramér-von Mises (CvM) statistics and low rejection rales of the null hypot hesis. Finally, we show that our mixed discrete-contimuour mixtures of beta distributions provide an adequate model for the marginal confidence distributions, as mcasured by low square-rooted CvM scores. However, the high rejection rates of the mull hypothesis suggest the potential for further improvements.
4.3.1 Verifying the Markov Assumption
To verify that (5) approximately holds, we first visualize the rank correlation between the calibrated con- fidences of different models. Figure 1 shows that the Kendall's rank correlation is higher for models of similar sizes. In addition, models sharing the same architectural family (Llama, GPT, or Qwen) are more highly correlated than models of different families.
Our findings suggest that a cascade composed only of Llama models (1B-405B) satisfies the Markov as- sumption more exactly. Consider Figure la as an example. For the Llama cascade, Kendall's is highest near the heatmap's 's diagonal, suggesting a Markov property. By contrast, the mixed cascade composed of Llama, GPT, and Qwen models shows more haphazard pattern. For example, the rank correlation between GPT-4o Mini and GPT-4o (r = 0.55) is higher than that between GPT-4o and Llama3 405B (7 = 0.54), even though the latter pair of models are more similar in size. Similarly, Llama3 405B is more strongly correlated with Llama3 70B (r = 0.58) than with Qwen2.5 72B (r = 0.46), even though the latter models are of near-identical size. These examples highlight that, in order for the Markov property to hold based on model size, seems important that models share the same architectural family,
In Appendix F, further verify the rank correlation patterns between different LLMs by recomputing the rank correlations only on those queries where both models answer correctly or incorrectly.
To probe the Markov property for the Llama cascade in different way, we train logistic regressions for predicting correctness of the 8B, 70B, and 405B models based on the calibrated confidences of two ancestor models in the cascade. Specifically, we consider the immediate predecessor model (the Markov predictor) paired with each available carlier ancestor. If the Markov property holds, the Markov prodictor should hold much greater significance than any other ancestor. Table lists the results, revealing diagonal pattern for each benchmark that confirms that the Markov predictor is usually much more significant. However, the carlier ancestor often shares statistical significance. To evaluate the significance of this finding, we also computed the mdg of the regression coefficients corresponding to Table 3. The coefficients follow a similar pattern, revealing that even if multiple predictors are significant, the Markov predictor usually carries the greatest weight.
In sum, our findings suggest that for cascades composed of models sharing the same architectural family, a Markov property holds approximately, though not exactly.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Llama Cascade
Mixed Cascade
Llama Cascade
Mixed Cascade
beal2
Tarat200
fend
Loudt 20
uwed
lea 343b
Lmaks
Keeat
Remal
Rleal
sewtt Reat
Bewes L-gi
(a) MMLU
(b) XSum
gund 77
Fardl 1.25
bad:: 68.
temsta
tamal
Bleat
Rleat
(c) MedMCQA
(d) GSMSK
Eawtt 1I
Iaral1-70
Bas 4400
Brnd 5-25
Real 1a
deatld
anr pute to
Ransta
nome
TriviaQA
(L) TruthfulQA
Figure 1: Evaluates the Markov property by showing the Kendall's rank correlation between the calibrated confidences of pairs of LLMs, as evaluated on the test set (n ~ 1000 examples). In a Markov pattern, the largest rank correlations occur near the diagonal, based on similarity in model size. For each benchmark, the figure compares the rank correlation structure of a cascade of Llama models to that of a mixed cascade consisting of models from the Llama, GPT, and Qwen families, suggesting that a cascade drawn from models of the same architectural family is more nearly Markovian.
4.3.2 Testing the Gumbel Copulas for Modeling LLM Correlations
To evaluate the goodness-of-fe fit of our Gumbel copula models, first visualize the correlation between the calibrated confidences of pairs of LLMs. Figures 2 and 3 show scatterplots for several pairs of Qwen, OpenAI, and Llama models. Each scatterplot shows the copula-transformed variables
Fn(
(19)
where & the calibrated confidence and F, its empirical distribution on the test set. The marginal distri- bution of each is uniform, since restrict our copula models to the region (ømin, (ømax) of calibrated confidence where the marginal confidence distribution is smooth. Note that Figure 3 highlights the Markov property by showing the increasing rank correlation between Llama models of similar sizes.
We formally lest the goodness-of-fit between the fitted Gumbel copulas and the test data by carrying out a Cramér-von Mises test using parametric bootstrapping, following the "Kendall's transform" approach described Genest et al. (2009). The test involves computing the univariate distribution of copula values Ciy (F) for p(x), using both the empirical copula and the fitted Gumbel copula. We evaluate the difference between these two distributions using the Cramér-von Mises (ViCvM) statistic and obtain
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table 3: Verifies the Markov property for the Llama cascade by showing the results using logistic regression to predict each model's correctness based on the calibratod confidences of ancestor models in the cascade: the immediate predecessor model (Markov predictor) and each available earlier ancestor. For the Markov predictors, the table displays the average p values across all these logistic regressions; for the carlier ancestors, the p value corresponds to single logistic regression. Underlined values indicate statistical significance (5% level); the lowest values in each row are bolded. The diagonal pattern in the table suggests the Markov property.
log10
Value of Markov Predictor vs Earlier Ancestor
Benchmark
Predicted
1B
3B
8B
70B
8B
-2.66
-26.86
MMLU
70B
-0.52
=3.48
-1.71
405B
-0.78
-2.41
-6.32
-25.78
8B
-1.85
-26.40
MedMCQA
70B
-0.26
-2.72
-4.35
405B
-0.23
-0.82
-2.45
-24.63
-0.14
-2.38
TriviaQA
70B
-0.58
-1.02
-6.42
405B
-0.26
-.88
-3.72
-11.45
8B
-0.72
-1.58
XSum
70B
-0.97
-0.61
-6.94
405B
-0.56
-0.50
-2.81
-1.62
8B
-2.85
-7.48
GSMSK
70B
-0.51
-0.17
-6.49
405B
-0.36
-0.13
-3.22
:2.76
8B
-1.77
-0.42
TruthfulQA
70B
-0.30
-0.44
-0.52
405B
-0.20
-0.67
-0.59
-1.55
qwen2.5 11725-3320 11..5--3bb-oo codeoeoddeeeee Sewtructtttt
0.3
0.8
0.8
upt- upt-4o-nini
apt 40 mini
0.4
0.4
0.4
0.0
020
0.0
022
0.0
0.0
0.2
0.4
0.8
0.5
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
10
qwen2.5:
gpt
qwen2.5
(a) GPT-4o Mini and Qwen2.5 32B Coder on GSM8K
(b) GPT-4o Mini and GPT-4o on MedMCQA
Qwen2.5 32B Coder and (c)
Qwen2.5 72B on TriviaQA
Figure 2: Correlations between the calibrated confidences of selected pairs of LLMSs on different benchmarks, showing a range of rank correlations between models. The Kendall's 7 rank correlation, shown in the bottom right corner, ranges from т 0.20 to = 0.44.
value by parametric bootstrapping with B = 1000 samples. In each case, we fit the Gumbel copula on the training data (n ss 300) and evaluate the value relative to the test data (n & 1000).
Table breaks down the results by benchmark for two groups of models (Llama models vs OpenAI & Qwen models). Each reported number based on considering all pairs of models within each group, regardless of similarities in size. There are 10 pairs of Llama models and 6 pairs of Qwen and OpenAl models. The results show that for the Llama models, the fitted Gumbel copulas closely match the empirical correlation
11

---

Published in Transactions on Machine Learning Research (05/2025)
0.3
0.8
0.5
Hamma3.211b
Nomm3. 8b
1oma3.117708 1-700
0.4
0.4
0.4
0.0
0.0
0.41
0.0
0.57
0.
1.0
Baama3.1 1-405b
Ilama3.1-400
(a) IB-105B
(b) 8B-405B
(c) 70B-105B
Figure 3: Correlations between the calibrated confidences of smaller Llama models (1B, 8B, 70B) (y axis) and the 405B model (Ir axis) on MMLU. The increasing rank correlation suggests Markov property based model size. The Kendall's rank correlation, shown in the bottom right corner, increases from 7 0.21 to 0.57.
Table 4: Shows the goodness-of-fit of our Gumbel copula models for modeling pairwise correlations between LLMs, based on a Cramér-von Mises (VпCvM) test using parametric bootstrapping. We report the vnCvM value, the number null hypothesis rejections (out of 10 and model pairs for the Llama and Qwen OpenAI groups, respectively), the percentage of rejections, as well the geometric and arithmetic mean of p values.
Liama Models
Qwen & OpenAl Models
Benchmark
VnCvM
#Rej-
% Rej-
Tp)
Σ>
VnCvM
#Rej.
% Rej.
{ p)
Σ p
MMLU
0.002
0.0
0.569
0.591
0.011
4
66.7
0.058
0.121
MedMCQA
0.004
10.0
0.394
0.560
0.004
0.397
0.444
TriviaQA
0.002
0.0
0.638
0.709
0.012
33.3
0.078
0.187
XSum
0.004
0.0
0.405
0.480
0.002
0.0
0.704
0.733
GSM8K
0.002
0.0
0.688
0.757
0.016
33.3
0.032
0.157
TruthfulQA
0.001
0.0
0.961
0.963
0.002
0.0
0,800
0.812
Average
0.003
1.7
0.609
0.677
0.008
22.2
0.345
0.409
structures between pairs of models on the test set, since the overall rejection rate the mull hypothesis is only 1.7% well below the 5% rejection rate expected by chance. In addition, the VTCVM statistic is only 0.003 average.
For the group of Qwen and OpenAl models, we observe higher rejection rates. The overall rejection rale of 22.2% suggests that the Gumbel copula model does not fit the data exactly. However, the average vnCvM value 0.008 suggests that the fit adcquate.
4.3.3 Testing the Discrete- Coirette-ooninuouss Marginal Confidence Distributions
First, visualize the agreement between the fitted contimuous-diserete mixtures of scaled beta distributions and the histograms of calibrated confidence values the test set. To construct these plots, first train the calibrators and marginal distributions on the training set (n R 300 examples). We then compute the calibrated confidence on the test set (n s 1000) using the trained calibrators. Figure 4 suggests that the fitted marginals align well with the calibrated confidence values on the test data.
Each histogram displays the discrete masses Φmin and émax of the fitted marginal distributions by shading corresponding areas on the first and last bars of each histogram. We observe in Figure 4a that the discrete probability massos are especially pronounced for GPT-4o Mini on TruthfulQA and GPT-4o on MedMCQA. The Irend that the OpenAl GPT models often report certainly also holds for other beuchmarks, as Table 1 shows.
We do not consider necessary train the calibrators and the marginal confidence distributions separate training data sets, since calibrators model p(y 2) and the marginal distributions ions model p(x).
12

---

Published in Transactions on Machine Learning Research (05/2025)
02
0.4
0.6
0.8
0.4
0.5
0..6
0.7
0.8
0.0
1.0
0.4
0.5
0.5
0.7
0.8
(a) GPT-4o Mini on TruthfulQA
(b) GPT-1o on MedMCQA
(c) Llama3.2 3B on MMLU
0.3
0.4
0.5
0.5
0.7
0.8
0.2
0144
0.6
0.8
0.0
0.2
0.4
0.6
0.8
(d) Llama3.1 8B on TruthfulQA
(e) Llama3.2 1B on GSMSK
(1) Qwen2.5 32B Coder on XSun
Figure 4: Sclection of trained marginal distributions (fitted on n = 300 training data), with histograms of the test data (n R 1000). Histogram areas shaded with hatch patterns (especially in (a) and (b) indicate the contributions of discrete probability masses in our models.
Table 5: Shows the goodness-of-fit of our discrete-contimuous mixtures of scaled beta distributions for mod- eling the marginal distributions of calibrated LLM confidence. We computed p values for the square-rooted Cramér-von Mises (VCvM) statistic using parametric bootstrapping with B 1000 samples. The vCvM statistic and its value were computed on the test set (n x 1000), whereas the marginal distributions were fitted on the training set (n = 300). We highlight p < 0.05 with an underline and < 0.001 with bold font. Additionally, we bold the largest VCvM value within each column. Highlighted values indicate the greatest discrepancies with our model.
MMLU
MedMCQA
TriviaQA
XSum
CSM8K
TruthfulQA
Model
CVM
VCVM
p
CVM
CCM
P
VCM
VCVM
llama3.22 2-1b
0.031
0.000
0.025
0.015
0.018
0.117
0.036
0.001
0.026
0.015
0.025
0.109
Ilama3.22 2-3b
0.014
0,144
0.115
0.000
0.043
0.000
0.020
0.076
0.020
0.071
0.030
0.053
Hlama3.1-8b
0.016
0.066
0.085
0.000
0.022
0.033
0.037
0.000
0.016
0.163
0.022
0.181
llama3.1-700
0.048
0.000
0.137
0.000
0.057
0.000
0.070
0.000
0.038
0.000
0.044
0.002
1lama3.1-405b
0.024
0.004
0.113
0.000
0.028
0.008
0.027
0.009
0.034
0.001
0.036
0.019
gpt-1o-mini
0.032
0.000
0.060
0.000
0.077
0.016
0.028
0.072
0.036
0.000
0.069
0.000
0.040
0.000
0.020
0.067
0.028
0.010
0.023
0.160
qwen2.5-72b
0.028
0.001
0.073
0.000
0.040
0.000
0.041
0.000
0.004
0.678
0.036
0.018
8pt-4o
0.029
0.000
0.100
0.000
0.046
0.000
0.026
0.013
0.036
0.001
0.065
0.000
Average
0.029
0.024
0.087
0.003
0.034
0.066
0.033
0.027
0.025
0.106
0.034
0.068
We formally test the goodness-offiit of the marginal distributions by computing the square-rooted Cramér- von Mises statistic
VCrM V (Freet()) [(
())² dF
(20)
where = = F(-10) is our marginal distribution model (7) with 0 = (ømins Omaxi Wmins Wmaxi IT, 01, 81, 02, B2). In Tables and 6, we report (20) both for i estimated from the training data (VCvM), and for O re-fitted the test
do(m,) is the empirical distribution of the calibrated confidence the test data, and
13

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Shows the goodness-of-fit of our discrete-contintous mixtures of scaled beta distributions for model- ing the marginal distributions of calibrated LLM confidence, after re-fitting the marginal distributions on the test set. We computed p values for the square-rooted Cramér-von Mises (VCvM,) statistic using parametric bootstrapping with B = 1000 samples. We highlight p < 0.05 with an underline and p 0.001 with bold font. Additionally, we bold the largest VCMM, value within each column. Highlighted values indicate the greatest discrepancies with our model.
MMLU
MedMCQA
TriviaQA
XSum
GSMSK
TruthfulQA
Model
VCM,
VVVI
p
VCVM,
VCVM,
VCVM,
P
VCV,
Ilama3. 2-11
0.018
0.046
0.020
0.085
0.005
0.986
0.006
0.935
0.018
0.126
0.008
0.970
llama3.2-3 2-3h
0.009
0.461
0.010
0.608
0.036
0.000
0.009
0.666
0.010
0.566
0.013
0.661
1lama3.1-88
0.012
0.248
0.010
0.574
0.011
0.492
0.009
0.688
0.006
0.947
0.010
0.846
1lama8,1-70b
0.016
0.072
0.018
0.121
0.024
0.029
0.017
0.141
0.023
0.037
0.015
0.460
1lama3.1-405b
0.015
0.133
0.498
0.017
0.130
0.006
0.933
0.031
0.002
0.019
0.290
gpt-4n-mini
0.004
0.928
0.007
0.853
0.003
0.913
0.011
0.393
0.009
0.549
0.014
0.548
qwen2.6-32b-c
0.011
0.282
0.012
0.355
0.020
0.070
0.022
0.047
0.015
0.217
0.013
0.600
qwen2.5-72b
0.018
0.039
0.016
0.157
0.028
0.005
0.014
0.301
0.002
0.966
0.008
0.970
gpt-4o
0.011
0.273
0.024
0.315
0.041
0.000
0.013
0.367
0.030
0.0015
0.021
0.739
Average
0.013
0.276
0.014
0.396
0.021
0.292
0.012
0.497
0.016
0.379
0.013
0.678
data (VOvM,). The reason we report VOMM, is to evaluate whether deficiencies in the fit arise from a bias problem, rather than a variance problem. To compute values for (20), we use parametric bootstrapping with = 1000 samples.
Table indicates a close fit between the trained marginal distributions and the empirical distributions of the calibrated confidences on the lest. data, with an average VCvM value of 4%. However, 74% of tests reject the null hypothesis at the p 0.05 level, suggesting that our model does not exactly match the data. When refitting the marginals on the test data, the average vCvM value falls to 1.5% and much lower 18.5% of tests reject the null hypothesis. Even on the refitted data, this overall rejection rate of 18.5% is significantly higher than the 5% we would expect by chance. We conclude that our marginal distribution model fits the empirical data well, as judged by a low VCvM value, but it clearly does not capture the true distribution of calibrated confidences exactly.
Notably, the results for the refitted marginals show that the quality of the fit strongly depends the benchmark. Specifically, TriviaQA displays much poorer fit than the other benchmarks. For many of the LLMs, TriviaQA's 'ss low difficulty (as judged by a 90%+ test accuracy for many models) explains the poor fit. The presence of a sharp peak of calibrated confidences near émax presumably raises the number of training samples required to preciscly estimate the shape of the distribution. In addition, the ability of the beta iistribution to fit sharply peaked unimodal distribuutions may be inherently limited. We hypothesize that these factors may explain the high p values despite rather low VCvM values.
4.4 Rational Tuning of Confidence Thresholds
In this section, examine the performance and runtime scaling of our continuous optimization-based algorithm (11) for selecting optimal confidence thresholds. We consider all 26 possible cascades of length k 2 2 composed of Meta's Llama models (1B, 3B, 8B, 70B, and 405B). We evaluate against. Bayesian optimization and high-resolution grid search baselines on six benchmarks (MMLU, MedMCQA, XSum, TriviaQA, GSMSK, TruthfulQA) spanning general-purpose knowledge and reasoning, domain-specifie QA, text summarization, open-ended QA, mathematical reasoning, and the ability to avoid hallucinations on adversarial questions.
Performance metrics: we evaluate the area under the error-cost curve (AUC) on the test set. Specifically, computing the AUC mcans plotting the test error (y axis) against the expected inference cost in dollars/query (z axis) and evaluating the integral of this curve. Figure 5a shows an example crror-cost curve and Figure 5b highlights the computation of AUC. We normalize the cost values to lie between and 1, resulting in
14

---

Published in Transactions on Machine Learning Research (05/2025)
Eroe Rote
Eiror Rate
bettes
Raticaal Tuiia (Ches)
AUC
Cost (8/Query)
Cost (8/ (S/Query)
(a) Error-cost curves
(b) Area under the error-cost curve
Figure Performance evaluation via the area under the error-cost curve (AUC). (a) Error-cost curves computed on the MedMCQA test for the Llama3 3B + 8B 70B 405B cascade. (b) Illustration of the area under the error-cost curve (AUC).
-0.5
% Reduction AUC
Reduction in AUC
-1.5
-20
%
-30
-8
-40
3
Cascade Length
Cascade Length
(a) Relative to Bayesian optimization
(b) Relative to grid search
Figure 6: Reduction in the area under the error-cost curve (AUC) on the test set when using our Rational Tuning framework to select confidence thresholds, as a function of cascade length. In (a), we compare against a Bayesian optimization baseline, while (b) we compare against high-resolution grid search. For longer cascades, our method outperforms both basclines by larger margins. Error bars show #la of the mean percentage change, and filled markers indicate statistical significance.
AUC scores between 0 and 1 (error rate x normalized cost). Broadly, a 1% reduction in AUC means that the error rate is 1% lower at the same inference cost (on average).
In addition, we measure how the runtime for finding optimal confidence thresholds scales with the length of the cascade and the desired resolution of the error-cost curve the axis, É.e., how densely sample the optimal thresholds. We have not overly optimized our code and mainly aim to contrast asymptotic scaling behavior.
Bayesian optimization baseline: this baseline runs Bayesian optimization with Gaussian process sur- rogate function, via the HEBO package (Cowen-Rivers et al., 2022, Shahriari et al., 2016). The Bayesian
15

---

Published in Transactions on Machine Learning Research (05/2025)
AUC 4
RT (Ours)
vs GS
RT (Ours) vs BO
Benchmark
RT (Ours)
GS
BO
%Acs
PGS
%ABO +
PBO
MMLU
0.288
0.288
0.294
0.02
2.8 10-1
-2.30
4.3 X 10 10-3
MedMCQA
0.381
0.384
0.389
-0.79
5.5 X 10
-2.14
3.0 X 10 10
TriviaQA
0.181
0.182
0.183
-0.74
6.0 x 10
-1.58
1.0 X 10 10
XSum
0.409
0.414
0.416
-1.48
2.0 x 10
-1.92
8.9 X 10 10
GSM8K
0.158
0.162
0.160
-2.68
1.0 10
-1.15
1.6 X 10
TruthfulQA 0.436
0.437
0.438
-0.26
3.8
10
-0.46
7.9 x 10
Average
0.309
0.311
0.313
-0.99
-1.60
Table 7: Area under the error-cost curve (AUC) on the test set, showing that our Rational Tuning ("RT") framework for selecting confidence thresholds consistently outperforms both Bayesian optimization base- line ("BO") and high-resolution grid search ("GS7). The mean percentage changes (%A) are statistically significant at the p 0.05 on almost all benchmarks, as measured by Wilcoxon rank-sum tests paired by cascade (highlighted in bold).
optimization minimizes (11), in an analogous manner to our Markov-copula ("Rational Tuning") approach. We run HEBO for as many iterations as needed until the change in loss between successive ilerations is below a numerical tolerance (e = 10 10-5). In practice, we found that the final change in loss typically 0.0. Following the practical guidance HEBO's authors", we use four parallel suggestions during each iteration. We adaptively interpolate the optimal thresholds computed by HEBO in the same way do for Rational Tuning (see Equation (13)).
High-resolution grid search baseline: this baseline selects optimal confidence thresholds by searching over adaptive grids computed from the model-specific quantiles of calibrated confidence. Specifically, each dimension the grid ranges from @min to Φmax increments of 2.5% probability mass. This results considering 40 candidate threshold combinations for cascades with k models, ranging from 40 candidates for two-model cascade to 40 2,560,000 candidates for five-model cascade. After scoring all candidate threshold combinations, we use the Skyline operator (implemented in the Python package paretoset") to filter the candidate threshold vectors down to the Pareto-optimal set (Börzsönyi et al., 2001). A candidate threshold vector 0 = ( 0x-1) is Parelo-oplimal if its performance metrics (Po(CCrrectt) [Coss]) are not dominated by any other candidate threshold vector the sense that Per (Correct) > Pe Po(Correct) and Egr [Cost] < Eg [Cost].
Figure 6 shows that our Rational Tuning framework for selecting confidence thresholds results in lower AUC on the test set compared to the basclincs. Each point on the plot shows the average percentage reduction AUC for all cascades of a given length k, averaged across all benchmarks. As cascade length i grows, our method outperforms the baselines by a larger margin. For example, the mean reduction AUG compared Bayesian optimization is 4.3% for & 2 3; 5.8% for k > 3; and 7.2% for k = 5. The corresponding performance gains relative to high-resolution grid search are 2.0%, 2.2%, and 2.7%. We computed statistical significance of these percentage differences using Wilcoxon rank-sum test paired by cascade.
We hypothesize that the performance gains of Rational Tuning relative to Bayesian optimization stem from the fact that our framework applies a (mostly correct) inductive assumption about the correlation structure of LLM cascades, whereas Bayesian optimization a general- algorithm for black-box optimization. Scction 4.4.1 corroborates this hypothesis by presenting more significant performance gains in the low-sample limit with < 30 training examples.
By contrast, it is not surprising that grid search performs as l increases, since searching for effective threshold combinations by trial and error suffers from the curse of dimensionality.
5 'github. www.w/www.eeeeeeeeeee http:///////eeeeee/eeeeeeeeeeeee Accessed April 6, 2025.
6 Ooen-source implementation available github.comm httm//ttoooo//pretteeee Accessed January 13, 2025.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Table 7 presents the results broken down by benchmark rather than cascade length. The table shows that Rational Tuning consistently out perforrrms Bayesian optimization and grid search across benchmarks, independent of cascade length. The only benchmark mark where we report a tie is MMLU. On almost all benchmarks, the reductions in AUC are statistically significant at the < 0.05 level.
4.4.1 Performance the Low-Sample Limit
Our Rational Tuning methodology relies small labeled training data set consisting of LLM confidence scores and corresponding binary correctness labels. Since labeled data is scarce in many applications, we supplement our main experiment (n xs 300 training examples) with study of the low-sample limit. Here, we re-run our experiment for n I 30 training examples. For cach benchmark, we ensure a balanced subsample with both correct and incorrect answers from each model by sampling training examples pairs (one correct, one incorrect for each model) for a fixed number of iterations, with target of n examples. Since collisions may occur, the final number of sampled training examples lies between 20 and 30, depending the benchmark.
Figure 7 displays the results, revealing that Rational Tuning significantly outperforms the Bayesian opti- mization and grid search baselines as cascade length grows. On cascades with k 2 3 models, the average performance gain is 10.2% relative to Bayesian optimization, and 5.6% relative to high-resolution grid search.
Table breaks down these results by benchmark. We see that Rational Tuning outperforms the basclines each benchmark except for TruthfulQA, where high-resolution grid search performs best. On the other benchmarks (MMLU, MedMCQA, TriviaQA, XSum, and GSM8K), the performance gains of our method are statistically significant at the p < 0.05 level, according to Wilcoxon rank-sum tests paired by cascade.
Our interpretation of these results is that our Rational Tuning framework benefits from making inductive as- sumptions about the interactions between the error rates of different LLMs. Crucially, fitting these inductive assumptions to empirical data requires only few observations: since each copula model depends on a single scalar correlation parameter A E R, our method requires only k 1 parameters to model the interactions between the error rates of k different LLMs.
-2.
% Reduction AUC
% Reduction in AUC
-75
-6
-12.5
-15.0
-17.5
-10
20.0
2
5
2
Cascade Length
Cascade Length
(a) Relative to Bayesian optimization
(b) Relative to grid search
Figure 7: Reduction in the area under the error-cost curve (AUC) cascade length grows, in the low-sample limit (n < 30 training examples), when using our Rational Tuning framework. In (a), we compare against a Bayesian optimization baseline, while in (b) we compare against high-resolut ion grid search. Our met hod increasingly outperforms the baselines as cascade length grows. Error bars show Lo of the mean percentage change, and filled markers indicate statistical significance.
17

---

Published in Transactions on Machine Learning Research (05/2025)
AUC 4
RT (Ours)
vs GS
RT (Ours)
vs BO
Benchmark
RT (Ours)
GS
BO
%Acs
PGS
%ABO +
РBO
MMLU
0.293
0.308
0.316
-5.51
8.0 10 10-6
-7.74
1.0 X 10 10-6 6
MedMCQA
0.399
0.416
0.419
-4.18
1.1 X 10
-4.72
5.0 X 10 10-6
TriviaQA
0.197
0.200
0.203
-2.11
1.5 x 10
-3.26
5.1 X 10 10
XSum
0.422
0.431
0.442
-2.42
2.6 x 10
-5.06
4.2 X 10
GSM8K
0.187
0.195
0.197
-3.41
2.5
10-2
-4.21
X 10
TruthfulQA 0.449
0.442
0.452
1.60
1.0 x 10
-0.73
1.2 x 10
Average 0.325
0.332
0.338
-2.67
-4.29
Table 8: Area under the error-cost curve (AUC) in the low-sample limit (n S training examples), showing that Rational Tuning (*RT") framework for selecting confidence thresholds consistently outperforms both Bayesian optimization baseline ("BO") and high-resolution grid search ("GS"). The mean percentage changes (%A) are statistically significant at the < 0.05 on almost all benchmarks, as mcasured by Wilcoxon rank-sum tests paired by cascade (highlighted in bold).
Performace (8)
Pesformacc Gan (95
Pecorrmsze Cata (5)
Pedormaae Ckin (8)
Rationsl Tunsag byitoe
"ag
Copyala
Moa Marginal Cr3d Scure
(a) Copula VRCMM
(b) Marginal VCvM
Figure 8: Sensitivity of Rational Tuming's performance gains to the Cramér-vor von Mises (CvM) test statistics (lower is better). Overall, performance appears to be more sensitive to mis-specification of the copula model.
4.4.2 Sensitivity to Statistical Assumptions
Our implementation of Rational Tuning models uses mixtures beta distributions to model the marginal distribution of confidence scores, and Gumbel copulas to model pairwise correlations between LLMs. In Sec- tion 4.3, we quantify the deviation between these modeling assumptions and the true empirical distributions via Cramér-von Mises (CvM) statistics.
Figure 8 shows the sensitivity of Rational Tuning's performance gains (relative to the high-resolution grid search baseline) to the mean CvM statistics for each cascade. For example, if M1 + + My has marginal VCvM scores 01, d and copula VnCvM scores 01,2, 02,3 0k.. - then the mean marginal and copula CvM scores marginal
Ei=1 i= 04, and copula Σ O1-1n respectively.
Figures 8a and 8b suggest that lower CvM divergences improve the relative performance of Rational Tuning. This effect is more pronounced for the copula statistios rather than the marginal statistics, highlighting the importance of correctly modeling the correlations between LLMs. In both plots, the light blue data points with little performance gain despite excellent CvM values are heavily enriched for two-model cascades. For cascades with & models, Rational Tuming generally performs on par with grid search or Bayesian optimization.
18

---

Published in Transactions on Machine Learning Research (05/2025)
10a
10
Roid scarch
104
grid search
10a
Runtime (s)
10
continiiote
Pontime
104
10a
10P
continuions
10-1
10-1
10-2
3
0.10
0.01
Cascade Length
Resolution of Cost-Error Curve
(a) The runtime of grid search grows exponentially in the cascade length, whereas the runtime of our method grows low-order polynomial (semilog-y plot).
(b) The runtime of our method always grows linearly in the desired resolution at which the error-cost curve is sampled along the cost axis, independent of the cascade length k. However, the runtime of grid search scales as h--l in h and (log-log plot).
Figure 9: Shows runtime scaling for computing the full crror-cost curve, comparing continuous- optimization based algorithm ("continuous", blue) to grid search ("grid search", gray). Our method scales much more favorably as the cascade length grows, and as the error-cost curve is sampled more densely along the cost axis. The shading shows #o of the observed data points.
The plots also suggest some robustness to deviations from the statistical assumptions. We are eager to explore Rational Tuning's robustness to model missspecification in greater detail in subsequent work.
4.4.3 Computational Scaling
Moving on comparison of computational complexity, Figure 9 shows that the runtime of our Rational Tuning framework for finding optimal confidence thresholds scales much more favorably compared to grid search, both in the length of the cascade & as well as the desired resolution h of the error-cost curve. Here, the resolution refers to the density at which we sample the optimal error-cost curve along the cost-axis. For grid search, is simply the reciprocal of the number of grid points in each dimension. For our method, h is the reciprocal of the number of times we solve the optimization problem (11). In other words, = 1/|A], where A the set of cost sensitivities we consider in (11).
We omit Bayesian optimization from Figure 9, since we observed longer runtimes (10-1000x longer) that exhibit less clear scaling with the length k of the cascade. Specifically, the average number of iterations until convergence increases from 2.9 for k = 2 to 5.3 for & = 4, then drops back to 4.0 for k = 5. Across the data, the minimum and maximum number of iterations required until convergence are 2 and 14.
4.4.4 Practical Guidelines
Our experiments suggest that Rational Tuning is the preferred methodology for tuning confidence thresholds for longer cascades (k: 2 3) with multiple deferral thresholds. However, for two-model cascades (k- = 2) with a single deferral threshold, the Markov assumption is void (as there are only two models). In this setting, the nonparametric nature of one-dimensional grid search should give the most reliable results.
When applying Rational Tuning for cascades with à 2 models, we recommend visually inspecting the match between the assumed probabilistic model and the empirical data. Specifically, we recommend the following visual diagnostics:
19

---

Published in Transactions on Machine Learning Research (05/2025)
Marginal distributions: compare histograms of the fitted and empirical distributions (as in Figure 4) and verify that the overall fit. is adequate.
Pairwise correlations: construct copula plots in Figures 2 and 3. Compare these plots to random samples from a Gumbel copula with correlation parameter 0 I-ft where f is the empirical rank correlation.
Markov assumption: construct rank correlation plots in Figure 1. Verify that rank correlations are strong near the diagonal.
In addition, it is important to assess the expected calibration error (ECE) of the confidence scores. We recommend computing the ECE using quantile binning with 10 or 20 bins; ideally, the ECE should not exceed 10%.
5 Conclusion
We have presented framework for rationally tuning the confidence thresholds of LLM cascades using continuous optimization. Our approach is based on a parametric probabilistic model for the calibrated confidences a sequence of LLMs. This probabilistic model is based on a Markov factorization, which accounts for pairwise correlations between the error rates of different LLMs using copulas, yielding a data- efficient approach. Goodness-offfit analyses spanning 10 LLMs and benchmarks have shown good agreement with the test data.
Importantly, our probabilistic model yields analytical expressions for a cascade's s error rate and expected in- ference cost. These expressions are differentiable with respect to the cascade's confidence thresholds, making continuous optimization possible. Compared to selecting confidence thresholds using Bayesian optimization and high-resolution grid search, our Rational Tuning framework yields more favorable error-cost trade-offs as cascade length grows, outperforming the basclines by up to 7.2% when using R 300 labeled training examples. In the low-sample limit (n < 30 training examples), the performance gains reach up to 16.5%, suggesting that our framework's inductive assumptions about the interactions between the error rates of different LLMs improve sample efficiency.
Building on these promising results, an interesting direction would be to apply our probabilistic modeling framework to LLM routing, in which a central routing model sends a query to the most suitable LLM in a single step, avoiding cumulative cost increases as the query propagates down a cascade. Since cumulative cost increases are especially severe for longer cascades (at which our methodology excels), the routing setting may more effectively leverage Rational Tuning's capacity for modeling dependencies between arbitrarily many distinct LLMs. For instance, suppose the routing decision depends noisy estimates @1, ..., on of the LLMs' true calibrated confidences. In this case, balancing the noisy observations Φ; against their probabilistic expectations E( & ...]) may lead to more effective routing decisions.
Ultimately, our results point to a larger vision for the future of deploying LLMs. Using probabilistic models, we will be able to adaptively seloct the most suitable model answer each query, improving both reliability and performance. Additionally, probabilistic modeling will enable us anticipate the performance of a system of LLMs under different conditions, making it possible to scamlessly adapt the system conditions shift. We are excited to further pursue this line of research in subsequent work.
References
Aggarwal, P., Madaan, A., Anand, A., Potharaju, S. P., Mishra, S., Zhou, P., Gupta, A., Rajagopal, D., Kappaganthu, K., Yang, Y., Upadhyay, S., Faruqui, M., and Mausam (2024). AutoMix: Automatically mixing language models.
Azaria, A. and Mitchell, T. (2023). The internal state of an LLM knows when it's lying.
Börzsönyi, S., Kossmann, D., and Stocker, K. (2001). The skyline operator. Proceedings 17th International Conference Data Engineering, pages 421-430.
20

---

Published in Transactions on Machine Learning Research (05/2025)
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodci, D. (2020). Language models are few-shot learners.
Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2024). Discovering latent knowledge in language models without supervision.
Casella, G. and Berger, R. (2002). Statistical Inference. Duxbury Press, Pacific Grove, 2 edition.
Chen, C., Liu, K., Chen, Z., Gu, Y., Wu, Y., Tao, M., Fu, Z., and Ye, J. (2024a). INSIDE: LLMs internal states relain the power of hallucination detection.
Chen, L., Davis, J. Q. Hanin, B., Bailis, P., Stoica, I., Zaharia, M., and Zou, J. (2024b). Are more LLM calls all you need? towards scaling laws of compound inference systems.
Chen, L., Zaharia, M., and Zou, J. (2023). Frugalgpt: How to use large language models while reducing cost and improving performance.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R.,, Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. arXiv preprint ar arX::2110014410
Coven-Rivers, A., Lyu, W., Tutunov, R., Wang, Z., Grosnit, A., Griffiths, R.-R., Maravel, A., Hao, J., Wang, J., Peters, J., and Bou Ammar, H. (2022). Hebo: Pushing the limits of sample-effiiien hyperparameter optimisation. Journal of Artificial Intelligence Research, 74.
Dempster, A. P., Laird, N. M., and Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Sociely: Series (Methodological); 39(1):1 22.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. (2024). LLM.int8(): 8-bit matrix multiplication for transformers at scale. In Proceedings of the 36th International Conference Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA. Curran Associates Inc.
Ding, D., Mallick, A., Wang, C., Sim, R., Mukherjee, S., Ruhle, V., Lakshmanan, L. V. S., and Awadallah, A. H. (2024). Hybrid LLM: Cost-efficient and quality-aware query routing.
Farquhar, S., Kossen, J., Kuhn, I.., and Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630(5017):625--00
Genest, C., Rémillard, B., and Beaudoin, D. (2009). Goodness-of-itt tests for copulas: A review and a power study. Insurance: Mathematics and Economics, 122)2119--
Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On calibration of modern neural networks.
Gupta, N., Narasimhan, H., Jitkrittum, W., Rawat, A. S., Menon, A. K., and Kumar, S. (2024). Language model cascades: Toven-level uncertainty and beyond.
Hari, N. and Thomson, M. (2023). Tryage: Real-time, intelligent routing of user prompts to large language models.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measur- ing massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
Hendrycks, D. and Gimpel, K. (2018). A baseline for detecting misclassified and out-of- examples in neural networks.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Jiang, D., Ren, X., and Lin, B. Y. (2023). LLM-blender: Ensembling large language models with pairwise ranking and generative fusion. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165-14178 Toronto, Canada. Association for Computational Linguistics.
Jiang, Z., Araki, J., Ding, H., and Neubig, G. (2021). How can we know when language models know? on the calibration of language models for question answering.
Jim, W., Gupta, N., Menon, A. K., Narasimhan, H., Rawat, A. S., and Kumar, S. (2024). When does confidence- based cascade deferral suffice?
Joshi, M., Choi, E.,, Weld, D., and Zettlemoyer L. (2017). TríviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Barzilay, R. and Kan, M.-Y,, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Kadavath, S., Conerly, T., Askell, A., Henighan, T., Drain, D., Perez, E., Schiefer, N., Hatfield Dodds, Z., DasSarma, N., Tran-Johnson, E., Johnston, S., El-Showk, S., Jones, A., Elhage, N., Hume, T., Chen, A., Bai, Y., Bowman, S., Fort, S., Ganguli, D.,, Hernandez, D., Jacobson, J., Kernion, J., Kravec, S., Lovitt, L.., Ndousse, K., Olsson, C., Ringer, S., Amodei, D., Brown, T., Clark, J., Joseph, N., Mann, B., McCandlish, S., Olah, C., and Kaplan, J. (2022). Language models (mostly) know what they know.
Kag, A., Fedorov, I., Gangrade, A., Whatmough, P., and Saligrama, V. (2023). Efficient edge inference by selective query. In The Eleventh International Conference on Learning Representations.
Kossen, J., Han, J., Razzak, M., Schut, L., Malik, S., and Gal, Y. (2024). Semantic entropy probes: Robust and cheap hallucination detection in LLMs.
Kryšcinskii, W., Keskar, N. S., McCann, B., Xiong, C., and Socher, R. (2019). Neural text summarization: A critical evaluation.
Lin, S., Hilton, J., and Evans, O. (2022a). Teaching models to express their uncertainty in words.
Lin, S., Hilton, J., and Evans, O. (2022b). TruthfulQA: Measuring how models mimic human falsehoods. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 3214-3252, Dublin, Ireland. Association for Computational Linguisties.
Lin, Z., Trivedi, S., and Sun, J. (2024). Generating with confidence: Uncertainty quantification for black-box large language models.
Liu, D. C. and Nocedal, J. (1989). On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45 45(:):533 528.
Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., and Zhu, C. (2023). G-Eval: NLG evaluation using GPT-4 with better human aligument.
Manakul, P., Liusie, A., and Gales, M. J. F. (2023). Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.
Naeini, M. P., Cooper, G. F., and Hauskrecht, M. (2015). Obtaining well calibrated probabilities us- ing bayesian binning. In Proceedings of the Twenty- Ninth AAAT Conference on Artificial Intelligence, AAAI'15, page 2901 2907. AAAI Press.
Narayan, S., Cohen, S. B., and Lapata, M. (2018). Don't give me the details, just the summary! Topic- aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Longuage Processing, Brussels, Belgium.
Nelsen, R. B. (2006)- An Introduction to Copulas. Springer Series in Statisties. Springer, 2 edition.
22

---

Published in Transactions on Machine Learning Research (05/2025)
OpenAI (2024). GPT-4 Technical Report.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. (2022). Training language models to follow instructions with human feodback.
Pal, A., Umapathi, L. K., and Sankarasubbu, M. (2022). MedMCQA: A large-scale multi-subject multi- choice dataset for medical domain question answering. Flores, G., Chen, G. H., Pollard, T., Ho, C., and Naumann, T., editors, Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Rescarch, pages 248-260. PMLR.
Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers, pages 61-74. MIT Press.
Plaut, B., Nguyen, K., and Trinh, T. (2024). Softmax probabilities (mostly) predict large language model correctness on multiple-choice q&a.
Proskurina, I., Brun, L., Metzler, G., and Velcin, J. (2024). When quantization affects confidence of large language models? In Duh, K., Gomez, H., and Bethard, S., editors, Pindings of the Associalion for Compu- tational Linguistics: NAACL 2024, pages 1918-19999 1928, Mexico City, Mexico. Association for Computational Linguistics.
Ren, J., Luo, J., Zhao, Y., Krishna, K., Saleh, M., Takshminarayanan,, B., and Liu, P. J. (2023). Out-of- distribution detection and selective generation for conditional language models.
Rudin, W. (1976). Principles of Mathematical Analysis. McGraw-Hiil New York, 3 edition.
Sakota, M., Peyrard, M., and West, R. (2024). Fly-swat or cannon? cost-effective language model choice via mede-mod In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, volume 35 of WSDM '44, page 606-615 ACM.
Shahriari, B., Swersky, K., Wang, Z., Adams, R. P., and de Freitas, N. (2016). Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEFE, 104(1):148-177 1755.
Wang, C., Augenstein, S., Rush, K., Jitkrittum, W., Narasimhan, H., Rawat, A. S., Menon, A. K., and Go, A. (2024). Cascade-aware training of language models.
Wang, X., Wei, J., Schuurmans, D., Le, Q, Chi, E.,, Narang, S., Chowdhery, A., and Zhou, D. (2023). Self-consistency improves chain of thought reasoning in language models.
Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., and Hooi, B. (2024). Can LLMs express their uncertainty? an empirical evaluation of confidence clicitation in LLMs.
Yue, M., Zhao, J. Zhang, M., Du, L., and Yao, Z. (2024). Large language model cascades with mixture of thoughts representations for cost-efficient reasoning.
Zadrozuy, B. and Elkan, C. (2002). Transforming classifier scores into accurate multiclass probability es- timates. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '02, page 694 699, New York, NY, USA. Association for Computing Machinery.
Zaharia, M., Khattab, O., Chen, L., Davis, J. Q., Miller, H., Potts, C., Zou, J., Carbin, M., Frankle, J., Rao, N., and Ghodsi, A. (2024). The shift from models to compound Al systems. https://bair btape://baar..bbkeeeyy. berkeley. edu/blog/ ocl/bloog2224402/111 cttp//mdmmomdeoe/e/eee ai-sastemsy//eeeeeeeeeeo Accessed: January 10, 2025.
Zellinger, M. J. and Thomson, M. (2024). Efficiently deploying LLMs with controlled risk.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Appendix A: Proof of Proposition 2
Proposition 2. Consider a cascade M1 + ... + My with confidence thresholds (Ø1, ...y Qx-1). Assume that the distribution functions for the calibrated confidences b, satisfy (5), for i = 1, ...,. Assume further that the expected numbers of input and output tokens, T!(i)) and T! for each model i are independent of the calibrated confidences 4 11,..., 4g. Then the probability of correclness P Correcl) and expected cost. [[[Cott]
for the cascade are
/
P(Correcl)
12(w) dP(w)
(21)
(81301)
+ Σ P( (b) i=2
S0] ( B.(w) dP(w) S 10-1)
E[Cost] = (1 - P( S <)))) B[G]]
(22)
II P(T, s 1 (00-1)) (1 PP S 1 sdi-1))
+
Σ 1=2
1 s (i)
Σ J=1
B(Cyb,
where Cs is the cost per query of model i. Specifically, if ((m)) and (out) are the costs per input and output token, C. 7/2 =: ((in) + 7 ((())) To simplify the notation, we let ΦE = -00 (although. there is no confidence threshold for the final model in the cascude).
Proof. We proceed by establishing the formula for the probability of correctness. Analogous reasoning then yields the formula for expected cost. Let E f1...., k} be the index of the model M, that returns the query, Specifically, {r = i} = { < ,1,,., . I Φ $ > o.)- We will decompose P(Correct) based on the value r. First, since the calibrated confidence 4; satisfies $; = E (M, correct) [z], we have
(ccoreect) = E[
4 0,1 = 1) >>E ET i}].
(23)
Hence, the problem reduces to computing (T =1] for each model i. This is the integral of 4; over the set {T = i}. For i 2 2, we have
f-1
( it} /
/ 1, p.I
II j=1
(o,5,,)) dp
(24)
IT (0,50,)
-P(Oi S $ o s s)-1) / 0:11002011 P(t $1, 1.11 S 1)--1)
dP
(25)
- 2 / (0.2000)
= Φ1)
II
P( S
b dP( nz { S b1}))
(26)
j=2
i-1
/
= P( s Φ1)
II j=2
P(
2 1 s 2 (0:20))) p, dP(-1 s $--)).
(27)
To obtain Equation (26), we applied the Markov assumption (5) and switched from the standard probability mcasure P(-) to the conditional probability mcasure P(. n A)/P(()), where A = { I $1,..., p I Φ1-1). To obtain the last. line, we applied the Markov assumption (5) again.
For i 1, we have that
f ((0.200) (1(w) dP(w).
E[0,1(7 1)}
(28)
24

---

Published in Transactions on Machine Learning Research (05/2025)
This concludes the proof of the formula for the probability of correctness. To obtain the formula for the expected cost, we reason analogously and note that the integral
S 10 Eg dP s Φ1-1)
(29)
simplifies the product P > de -- S Φi-1) Ej=1 B[C because we assume the model costs to be independent of the calibrated confidences.
Appendix B: Algorithm for Computing (((Corret)) and E [oott
Algorithm Computing P(Correct) and E[Cost]
Require: confidence thresholds $1,..., Φb-1 E R-1
1: cum_ cost + E[C]]
# cumulative expected cost # cumulative transition probobility # expected correctness due different models expected costs due different models
2: cum_ cum_transition_ppe f-
correctness_terve []
cost_terms e
db -00
htwpcctteessstttsssaaapppppppp (0,20) Φ1(w) aP(w))
cost cost_ttersssso.ooo cost_tterms.appen((( 1 Φ1)) cum_ com_c
cum won__tannitiion_peee
cum wom_trrnnsition_rro F(0) S Φ))
10:
11: for i= 2.. k do
12: cum cum_ cost + cos_ B[C]]
13: correctness, http://ppppppccmmtttttttttooooop
www.tttttttttoonnnnomoooom probb x / (0,6,,) );(w) S
14: cost_terms pest_teemm..ppend cum__ transitioon cimmstiin__pobbbbo x (1 P(Φ; oi -- (1-1)) cum_cost)
15: cum_ wwtataainiinnnoae + cum_ wew.transitionnn_ee x P(Φ; S o. 0. S (6111)
16: end for
17:
18: P Correct) + sumfcorrectness_terre
19: [[(stt)] +- sum(cost, terms)
20:
21: return (P(Correct), B[Cost]
Algorithm 1 provides an efficient way to compute the probability of correctness and expect cost in O(k) time, where & is the length of the cascade. We compute all probabilistic quantities using the fitted Markov-copula model. To compute the integrals
f - (.(w) dP(ww S 21-1)
(0)-1100)
(30)
of conditional correctness, we use numerical integration by treating (30) as Riemann-Stieltjes integral St & dF(() in the distribution function F(Ø) = P( - 0/ I Φi-1). See Rudin (1976). Before solving the minimization problem (11), pre-compute look-up tables for 4(0,1,, Φv) which can be re-used when solving (11) for different values of À and different subcascades.
Appendix C: Prompt Templates
Below, we provide the exact text of the prompts used in our experiments. (question}) replaced runtime with the relevant. content.
Placeholders (for example,
25

---

Published in Transactions on Machine Learning Research (05/2025)
5.1 MMLU
User Prompt (Zero-Shot)
Answer the multiple-choice question below by outputting A, B, C, or D. Don't say anything else.
Question:
(question}
Choices:
{choices)
Answer:
System Prompt
Correctly answer the given multiple-choice question by outputting "A", "B", "C", or "D". Output only "A", "B", "C", or "D", nothing else.
5.2 MedMCQA
User Prompt (Zero-Shot)
Below is multiple-choice question from medical school entrance exam. Output "A", "B", "C", or "D" to indicate the correct answer. Don't say anything else.
Question:
{question}
Choices:
{choices)
Answer:
System Prompt
Your job is to answer multiple-choice question from medical school entrance exam. Correctly answer the question by outputting "A", "B", "C" or "D". Output only "A", "B", "C", or "D", nothing else.
5.3 TriviaQA
User Prompt (Zero-Shot)
Correctly answer the question below. Give the answer directly, without writing a complete sentence.
Question: {question}
Answer:
System Prompt
Correctly answer the given question. Answer question directly without writing complete sentence. Output just the answer, nothing else.
26

---

Published in Transactions on Machine Learning Research (05/2025)
Evaluation User Prompt
to the following trivia question: {question). The proposed answer is poodelanswer), Decide if this answer correctly answers the question, from the standpoint of factuality. Output "Y" if the answer is factually correct, and "N" otherwise. Do not say anything else.
Consider proposed
Evaluation System Prompt
You are a helpful assistant who judges answers to trivia questions, Given a trivia question and a proposed answer, output "Y" if the proposed answer correctly answers the question. Otherwise, if the answer is not factually correct, output "N". Only output "Y" or "N" not say anything else.
5.4 XSum
User Prompt (Zero-Shot)
Summarize the given source document. Write concise summary that is coherent, consistent, fluent, and relevant, as judged by the following criteria:
Coherence collective quality all sentences
Consistency factual alignment between the summary and the source Fluency quality of individual sentences
Relevance selection of important content from the source
Source document: {source_docunl documentt
Summary:
System Prompt.
Summarize the given document. Output only the summary, and nothing else. not introduce the summary; start your answer directly with the first word of the summary.
Evaluation User Prompt
Consider a proposed summary the following source document: [source (isrrre__dcument.... Decide the following proposed summary is coherent, consistent, fluent, and relevant, as judged by the following criteria:
Coherence collective quality of all sentences Consistency - factual alignment between the summary and the source Fluency quality of individual sentences Relevance selection important content from source
Score each criterion (coherence, consistency, fluency, and relevance) scale from 1-5, where is best. Return JSON of the form ("coherence": a, "consistency": b, "fluency": c, "relevance": d}, where a, b, c, d are the scores for the criteria (1-5). Only return this JSON.
Proposed summary: (model_answed
JSON containing the scores for all criteria:
27

---

Published in Transactions on Machine Learning Research (05/2025)
Evaluation System Prompt
You are a helpful assistant who evaluates the quality of text summaries based on coherence, consistency, fluency, and relevance, as judged by the following criteria:
Coherence collective quality of all sentences Consistency - factual alignment between the summary and the source Fluency - quality of individual sentences
Relevance selection of important content from the source
Score each criterion on a scale from 1-5 (5 is best). Only respond with JSON. The JSDN should have keys "coherence", "consistency", "fluency", and "relevance", and the values should be the scores (integers from to 5).
5.5 GSM8K
User Prompt (Zero-Shot)
Reason step through the following math problem. End your answer by outputting the final numerical answer prefixed by *#*## '. For example, your final answer is x, end your answer on *#### x'.
Problem: {question)
Step-B Reasoning:
System Prompt
Consider given math problem, think step-by-step, then report the final numerical answer follows **** x'. Do not say anything after that.
Evaluation User Prompt
Consider a proposed answer to the following math problem: (question]. Decide if the following proposed answer correctly answers the question. Only evaluate the final answer; score answer as correct even the reasoning is faulty. For reference, the correct answer is provided below.
Respond with exactly " if the final answer is correct, or 1 if it is incorrect. Only output Y or N.
Proposed answer: ansser)
Correct answer:
answer)
Is the given final answer correct? Respond with exactly y or N:
Evaluation System Prompt
You helpful assistant who evaluates
Only respond with Y N.
5.6 TruthfulQA
User Prompt (Zero-Shot)
28

---

Published in Transactions on Machine Learning Research (05/2025)
Answer the question below truthfully and concisely. No need to write complete sentence, explain your answer, or say anything else afterwards.
Question:
{question)
Answer:
System Prompt
Your task is to answer the given question truthfully. It is okay to acknowledge do not know the answer. Give the correct answer if you can.
Evaluation User Prompt
Consider a proposed answer to the following question: {question). Decide if the following proposed answer truthfully answers the question. For reference, consult the lists of known correct and incorrect answers provided below. Respond with exactly 'Y' if the answer is truthful, or 'N' if it is not truthful. Only output Y or N.
Proposed answer: [model_answed
Known correct answers: (oree__bllettscooreett,
Known incorrect answers:
(inceerrtllttttnnnnreeeteeee (aancorrect_answerss)
the given answer truthful? Respond with exactly or N:
Evaluation System Prompt.
You are a helpful assistant who evaluates answers. Only respond with Y or N.
Appendix D: Price Differentials between Small and Large Models
Table lists the differentials between smaller and larger language models across varions providers.
Table 9: Price differentials between smaller and larger language models across various providers. Ratios indicate how many times more expensive the larger model compared to its smaller counterpart, in dollars per million tokens. Data as of December 20th, 2024.
A Intelligence
Provider
Smaller Model
Larger Model
Price Ratio
Meta
Hama3.1-70b claude-3.5-sonnet
llmm5311400BB
3,33x
Small Gap
Anthropic
claude-3-opus al
3.00x
OpenAI
gpt4o
6.00x
Meta
llama3.1-88 1-8h
llama3. llama3.1-405h 1-405b
15.0x
Medium Gap
OpenAT
gpt4o-mini
gpt4o
16.67x
Anthropic
1aude-3.5--haiee
claude-3-opus
18.75x
Meta
llams3.2-1b
1lam3311-401
30.0x
Large Gap
Anthropic
claudo-3-haiku
claude-3-opus
60.0x
OpenAl
gptio-mini
ol
100.0x 0x
29

---

Published in Transactions on Machine Learning Research (05/2025)
80
100
100
&oCrect
60
Cornett
Correct
40
60
00
20
-00
20
0
20
Calibrated Confidence Threshold
Calibrated Confidence Threshold
Calibrated Confidence Threshold
Llamaa32
(b) Llama3.2
(c) Llama331 8B
100
% Careect
50
% Corret
Correct
80
60
60
00
40
40
Calibrated Confidence Threshold
Calibrated Confidence Threshold
Calibrated Confidence Threshold
(d) GPT-4o Mini
(e) Qwen2.5 32B Coder
(f) Llama3.1 70B
100
56 Correct
50
% Conrect
80
Coreec
60
40
Calibrated Confidence Threshold
Callbrated Confidence Threshold
Calibrated Confidence Threshold
g Qwen2.5 72B
(h) Llama3.1 405B
(i) GPT4o
MMLU
MedMCQA
TriviaQA
XSum
GSMBK
TruthfulQA
Figure 10: Verifies that confidence thresholding works by showing that for most benchmarks and models, test accuracy increases to above when only accepting queries on which the calibrated confidence for the query exceeds q. Calibration was performed on the training set. The shading indicates #lo, computed by binomial model for the number of correct answers. Above the diagonal dashed line, the conditional accuracies exceed the confidence thresholds, as they should.
Appendix E: Verifying Confidence Thresholding on the Test Sets
We further verify calibration of LLM confidences by showing that confidence thresholding works: for most benchmarks and models, when only accepting queries for which the calibrated confidence exceeds g, the test error decreases to below < q.
Figure 10 plots the conditional accuracy with confidence thresholding on the test sets (n ~ 1000). In each case, the logistic regression calibrator was fitted on the training set (n ss 300). Each plot traces the empirical probability of correctness on the test set, (correcc) $ 2 o), for different values the calibrated confidence threshold The figure shows that, for the most part, the models' conditional accuracies increase as expected. This is indicated by the fact that the conditional accuracy curves mostly remain above the diagonal dashed lines, reflecting the theoretical expectation that Pres (correctt $ > Φ) o.
30

---

Published in Transactions on Machine Learning Research (05/2025)
E:33
645
0:04
0.0
2:30
136
832
234
Faralbbb
0.28
0.22
0.23
9.1
133
825
0.77
033
2.36
8.5
144
145
022
644
0.36
3.1
642
123
6G2
2.34
1M
Tamal 4.0
0.44
9.4
003
Besut Ade
Sansl Ai
Beatt Age
Reed AM
Beeu
(a) Incorrect Only
(b) Correct Only
(c) All
Figure 11: MMLU: Kendall's rank correlations of Llama3 models ordered by size.
E.3t
CU2
0:2
0.21
6:35
425
0.04
0:40
G43
4
0/4
655
C5
035
qM 5330
0.07
C:4E
0//
0.04
6:2
3.34
0.48
6:4
0.22
444
4
60
0.00
2:
6:7
3
0:23
3:35
qwil 5-703
C23
Q-4
66
0.15
4:47
3.20
G42
01L
0.02
1:25
6:24
3
3:31
C:44
0:42
054
0:53
0.22
2:49
3.4
C:2
46
09%
0.00
2:2
3.M
0.30
633
0.51
642
4044
0.18
E.5
0
003
Banut Ai
Rame Ag
(a) Incorrect Only
(b) Correct Only
(c) All
Figure 12: MMLU: Kendall's T rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size.
6 Appendix F: Recomputing Rank Correlations on Correct and Incorrect Answers
3.22
6:22
2.34
-14
E22
E:44
6:3
0.7
GUN
QAU
0.36
0.1
033
0.22
9.52
0.31
520
1:34
6:35
0.2
022
3.4
Bewid1 m0
0.7
0:3
0.0
3:11
015
-00
Ieed Ab
Ammak Sup
Seadh 400k
Read
(a) Incorrect Only
(b) Correct Only
(c) All
Figure 13: MedMCQA: Kendall's 7 rank correlations of Llama3 models ordered by size.
In this section, we verify the rank correlations between the confidence scores of different LLMs by recomputing them conditioned on both models answering correctly or incorrectly.
Figures 11-22 extend Figure 1 by separately re-computing the rank correlation patterns correctly and incorrectly answered queries. In addition, Table 10 below shows the average rank correlations computed separately on the correct, incorrect, and all answers, for each benchmark. We compute Time, Teorn Tall for each pair of models, as well as the rank correlations between these measurements across model pairs: Time, corr:
Tine, alls Toorr, all-
Note that since error rates are low for some models and benchmarks (see Table 1), conditioning on incorrectly answered queries leaves only few observations for some model pairs. In Figures 11-22, we print "?" for rank
31

---

Published in Transactions on Machine Learning Research (05/2025)
Serwad 2
E11
444
0/6
44
0.32
044
0:44
44
0.91
0.35
44
0.55
3:21
444
0.24
0.22
6444
0.35
3:33
3.04
0.0
2.01
0.88
6.0
0.27
6.12
0.08
0.00
033
600m
9:00
0.04
0.2
444
3:
9:11
C44
0.33
0A2
Apw Fuago
(a) Incorrect Only
(b) Correct Only
(c) All
Figure 14: MedMCQA: Kendall's T rank correlations of Llama3, GPT-40, and Qwen2.5 models ordered by size.
6:2
Sewt
E:30
641
1.86
+2:31
-0:31
-200
-000
0:19
211
-0
4:25
E47
-12
344
1000
642
6.11
-00
632
652
-01
1:4
3
NampP Ago
Eugte
(a) Incorrect Only
(b) Correct Only
(c) All
Figure 15: TriviaQA: Kendall's T rank correlations of Llama3 models ordered by size.
Exead2--1
2:21
4002
EA2
3:37
0A2
eved -3.04
5-386
-C40
larad
Rawak
3:01
4
awid
demul 4dio
(a) Incorrect Only
(b) Correct Only
(c) All
Figure 16: TriviaQA: Kendall's 7 rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size.
correlations with sample size less than 50; we use the n = 50 cut-off since it reduces the standard error for Kendall's 7 to around o, S 0.1, based on a normal approximation.
32

---

Published in Transactions on Machine Learning Research (05/2025)
FiFe
E:0
1:
2:11
0.12
-4:00
G04
Qudl
0:06
9.24
Sena
0.00
2:11
005
0.22
-065
6.9
600
1:22
E:7
Rawadt mM
006
0.3
830
085
006
1433
Bondl 100
Sanud A-
Hensh
(a) Incorrect Only
(b) Correct Only
(c) All
Figure 17: XSum: Kendall's T rank correlations of Llama3 models ordered by size.
Romul PL1x
022
4.11
0.25
0.33
Tanad 1400
6.1L
8:20
118
0/12
1
0.25
6m-4
Incorrect Only
(b) Correct Only
(c) All
Figure 18: XSum: Kendall's T rank correlations of Llama3, GPT-40, and Qwen2.5 models ordered by sizc.
4:2
4:85
-E.29
-3:3
111
-3:44
-0,2
-200
13
-02
6.14
623
830
3:39
3:2
-11
4.:3
E31
022
(3)
625
Lana 2-00
110
2.7
.005
Bent 1000
Foote
Rewakcup
Renuk 20p
(a) Incorrect Only
(b) Correct Only
(c)
All
Figure 19: GSM8K: Kendall's 7 rank correlations of Llama3 models ordered by size.
4:20
EdEL
Fanal2-ia
0:21
0/0
R-M
600
-0.04
5
0.00
E43
E:3
30
6
-002
-000
C33
0.30
2
2
C4
2-775
-0.0
225
0.27
6:77
G22
-005
0.00
422
2.2
6:27
2
B
6.00
4.82
0.2
E-r
21
Best
Banal
(a) Incorrect Only
(b) Correct Only
(c) All
Figure 20: GSM8K: Kendall's 7 rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size.
33

---

Published in Transactions on Machine Learning Research (05/2025)
045
fredbbi
640
0.00
622
G24
025
029
Srw
E-5
6:1
1.
155
0u24
0.19
8:0
027
8:3
0.34
Inab
Remal1/70
622
E44
E32
64)
0.35
3.8
Newe Aib
Send
(a) Incorrect Only
(b) Correct Only
(c)
All
Figure 21: TruthfulQA: Kendall's T rank correlations of Llama3 models ordered by size.
0:04
Exral 3-bb
E:22
0:0
SUN
0.0
A2
44
GAR
20
8:39
0.22
011
477
0.12
MM
X1
035
10.0
8:34
-2.44
6:35
005
LteeL apol
Bend Eedd
(a) Incorrect Only
(b) Correct Only
(c) All
Figure 22: TruthfulQA: Kendall's rank correlations of Llama3, GPT-4o, and Qwen2.5 models ordered by size.
Table 10: Rank correlations of calibrated confidences between Llama3 1B, 3B, 8B, 70B, and 405B models, computed separately for incorvectly answered queries ("ine"), correctly answered queries ("corr"), and all querics ("all"): F the average rank correlation across the 10 model pairs; Ta,b is the rank correlation between data subsets (inc, corr, all) across model pairs; and Pa,b the p value for Ta.b-
Benchmark
Tine
Tenrr
Tall
Tine.oorr
Pine,oner
Tinc,all
Pinc,all
Toorr,all
Poorr,all
MMLU
0.199
0.396
0.385
0.505
0.0001
0.717
0.0001
0.686
0.0001
MedMCQA
0.174
0.295
0.271
0.324
0.0055
0.486
< 0.0001
0.730
0.0001
TriviaQA
0.110
0.274
0.298
0.371
0.0014
0.413
0.0004
0.819
0.0001
XSum
0.138
0.049
0.065
0.180
0.1284
0.187
0.1148
0.721
< 0.0001
GSMSK
0.199
0.169
0.201
0.425
0.0003
0.467
< 0.0001
0.921
< 0.0001
TruthfulQA
0.222
0.196
0.178
0.667
< 0.0001
0.860
< 0.0001
0.756
0.0001
34


Reviews:
[{"review_id": "NfAhS3yBrQ", "reviewer": "Reviewer_jx1j", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Comprehensive probabilistic framework that handles multi-model error correlation using copulas.\n- Continuous-optimization approach provides better scalability than grid search, crucial for longer cascades.\n- Goodness-of-fit experiments on six benchmarks and multiple LLMs demonstrate the adequacy of the model.\n- Includes careful calibration and ablation to reduce overconfidence.\n- Empirical results show robust improvements in cost-error tradeoffs, highlighting practical value.\n\n**Weakness**\n- The paper’s approach relies on labeled data (300 training examples) to calibrate confidences; in settings with very limited data, performance might degrade.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel probabilistic framework for tuning the confidence thresholds of Large Language Model (LLM) cascades. By modeling the calibrated confidences of LLM with a Markov factorization and Gumbel copulas, the authors derive continuous, differentiable expressions for both the probability of correctness and expected cost. They then use these expressions in a continuous optimization procedure to select confidence thresholds that minimize error at a given cost. Empirical results across multiple benchmarks demonstrate its efficacy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nvGr6a5qEj", "reviewer": "Reviewer_Z6N8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The concept of LLM cascade is highly relevant in a time of LLM integration in various research areas and business applications.\n- The idea of modeling the joint calibrated confidence scores is novel, and the underlying motivation of tuning the confidence scores of the LLM cascade is relevant for the community.\n- Given the experiments, in selected benchmarks and cascade configurations (e.g. the cascade members belong to the same family of models), the assumptions underlying the proposed model are empirically validated on test-data.\n- The proposed confidence levels tuning algorithm benefits from linear (in the number of LLMs in a cascade) time to compute the objective function, therefore, scaling better than the grid search baseline.\n\n**Weaknesses:**\n\n- Relevance of statistical validity of the assumptions to the final performance of the cascade: The results section includes many experiments to test the statistical validity of the probabilistic model's assumptions, these experiments showcase statistical significance levels, number of rejections of the null hypothesis, p-values, and all the relevant statistical measurements. However, the authors do not discuss how this impacts the performance of the underlying LLM cascades. For instance, one may wonder how sensitive the solution of your tuning algorithm is to the rejection rate of the null hypothesis in the CvM test of the Gumbel copula model.\n- Potential confounding in the interpretation of the correlation experiments: In all of figures 1, 2, and 3 where the correlation scores between the calibrated confidence levels of pairwise LLMs are showcased, it seems that a strong correlation also correlates with similar performance (%Corr) in the respective benchmark. For instance, in Figure 1,c) the strongest correlation is between qwen2.5-72b and gpt-4o-mini who achieve respectively 69.1% and 66.% correctness in the corresponding benchmark. Similarly, in Figure 3,c) the strongly correlated llama3.1-405b and llama3.1-70b achieve respectively 81.5% and 85.2% in the corresponding benchmark. This suggests that the correlation observed in the confidence scores can be dominated by the correct answers that exhibit similar proportions whenever a strong correlation is observed. To make sure this is not the case, I suggest looking at the correlations between confidence scores grouped by correct and incorrect answers, to make sure both groups lead to similar conclusions.\n- Unclear trade-offs between AUC improvement and actual computational cost: What does a 2% improvement in AUC translate to in terms of computational efficiency? For a fixed performance level, how much faster is LLM inference using your method compared to grid search?\n- Possibly suboptimal grid-search baseline: Unless the grid search baseline is given enough budget to explore the full search space, other hyperparameters search engines might be able to find the optimal confidence thresholds in less time. Did the authors try that? This would also enable the search in the continuous space of thresholds (as opposed to the 2.5% resolution you currently use for grid search) as some engines parameterize distributions over continuous search spaces (e.g. HEBO https://arxiv.org/pdf/2012.03826).\n- Lack of practical use case guidance: summing up some of the already stated weaknesses, how can the person interested in applying an LLM cascade optimally configure the cascade in light of your contributions? For user-chosen LLMs, what are the most crucial assumptions to validate to use of the proposed probabilistic model and tuning algorithm? Starting from which level of statistical invalidity of the assumptions does the grid search baseline become a better alternative? Are there any benchmark-related or task-related (multiple-choice questions, text generation) conclusions?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a probabilistic model for the joint distribution of calibrated confidence scores of LLMs in an LLM cascade. The LLM cascade in this context, describes a process where LLMs with growing performance (i.e. size) are solicited in sequence if the calibrated confidence level of the previous cascade member fails to reach a given threshold. The probabilistic model relies on several assumptions:\n- 1. A parametric univariate distribution describing the marginal of each LLM \n- 2. A Gumbel copula model to capture successive pairwise correlations. \n- 3. The Markov property. \n\nThe probabilistic model is then used to tune the confidence thresholds by efficiently solving an optimization problem. The experiments include statistical tests to assess the correctness of the introduced assumptions, the individual performance and calibration of LLMs on standard benchmarks, and the comparison of the probabilistic model-enabled tuning of the cascade against a grid search baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here is a summary of requested changes, of which details are given in the Weaknesses section above:\n1. Analyzing the statistical validity of the assumptions in light of the final performance of the cascade.\n2. Repeat the correlation experiments for each group (correct and incorrect answers) to avoid potential misleading interpretations.\n3. Further quantify the LLM inference time gain from the improvement in the AUC.\n4. Try hyperparameters search engines instead of the grid-search baseline.", "broader_impact_concerns": "There is no *Broader Impact Statement* in the paper, and no particular ethical considerations to raise.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E0I9wWr1en", "reviewer": "Reviewer_6MPw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: Comprehensive experiments. Theoretical framework is logically sound for modeling cascade performance (up to Proposition 2).\n\nThe derivation of P(correct) includes a term P(Phi_1<=phi_1) even for i=1, which feels counterintuitive (why multiple  the first model’s low confidence probability when it is selected). The proof does not fully address the i=1 edge case either. Equation (26) appears to be wrong.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper looks at how to make systems of LLMs work better together with cascades. \n\nWhile confidence calibration is well-studied in ML, this work addresses the unique challenges of LLM cascades, where error rates interact dynamically across models.\n\nThe proposed model offers a fresh perspective by combining Markov assumptions (simplifying dependencies) with copulas (modeling pairwise correlations), enabling tractable optimization of thresholds.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The text claims Proposition 2’s proof is in Appendix B, but it appears in Appendix A.", "broader_impact_concerns": "No major ethical or societal risks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers were unanimously positive in their assessment of the paper. There were some suggestions regarding the experiments, such as considering smaller sample sizes, better discussing the relation of AUC gains to final cascade performance, and more careful analysis and discussion of the statistical significance. These were satisfactorily addressed by the authors in the response.

We have two minors suggestion that the authors may optionally consider:
- you may wish to defer some of the experimental results, and discussion thereof, to the Appendix; in the current form, the main body runs a little long.
- as noted in Section 4.4.4, the benefit of the framework could be in settings with multiple models. One challenge with cascades of several models is that the incurred cost could be non-trivial, as it is the cumulative cost of all models invoked. This is in contrast to simple query routing, which makes a one-shot model selection without invoking multiple models. This may be something worth discussing (for future work).

Overall, the paper makes an interesting new contribution to a problem of practical importance, with both mathematical and empirical analysis that could be built on by future work. We thus recommend acceptance for publication at TMLR.
Decision: Accept (poster)