Language Model Cascades: Token-Level Uncertainty And Beyond
Authors: Neha Gupta
Harikrishna Narasimhan
Wittawat Jitkrittum
Ankit Singh Rawat
Aditya Krishna Menon
Sanjiv Kumar
Venue: ICLR 2024 poster (2024)
Keywords: Cascades
Efficient Inference
Language Models
Primary area: generative models
TL;DR: Exploiting token-level uncertainty can significantly improve cost-quality tradeoff for language model cascades.

Abstract: Recent advances in language models (LMs) have led to significant improvements in quality on complex NLP tasks, but at the expense of increased inference costs. A simple strategy to achieve more favorable cost-quality tradeoffs is cascading: here, a small model is invoked for most “easy” instances, while a few “hard” instances are deferred to the large model. While the principles underpinning effective cascading are well-studied for classification tasks — with deferral based on predicted class uncertainty favored theoretically and practically — a similar understanding is lacking for generative LM tasks. In this work, we initiate a systematic study of deferral rules for LM cascades. We begin by examining the natural extension of predicted class uncertainty to generative LM tasks, namely, the predicted sequence uncertainty. We show that this measure suffers from the length bias problem, either over- or under-emphasizing outputs based on their lengths. This is because LMs produce a sequence of uncertainty values, one for each output token; and moreover, the number of output tokens is variable across different examples. To mitigate the length bias, we propose to exploit the richer token-level uncertainty information implicit in generative LMs. We argue that naive predicted sequence uncertainty corresponds to a simple aggregation of these uncertainties. By contrast, we show that incorporating token-level uncertainty through learned post-hoc deferral rules can significantly outperform such simple aggregation strategies, via experiments on a range of natural language benchmarks with FLAN-T5 models. We further show that incorporating embeddings from the smaller model and intermediate layers of the larger model can give an additional boost in the overall cost-quality tradeoff.

Published as a conference paper at ICLR 2024
LANGUAGE MODEL CASCADES:
TOKEN-LEVEL UNCERTAINTY AND BEYOND
Neha Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar Google Research, New York
(nehagup, hnarasimhan, wittawat, ankitsrawat, adityakmenon, hatpjvkk/ggogle.coo
ABSTRACT
Recent advances language models (LMs) have led to significant improvements in quality on complex NLP tasks, but at the expense of increased inference costs. Cascading offers a simple strategy achieve more favorable cost-quality trade- offs: here, small model is invoked for most "easy" instances, while few "hard" instances are deferred to the large model. While the principles underpinning cas- cading well-studied for classification tasks with deferral based predicted class uncertainty favored theoretically and practically - a similar understanding is lacking for generative LM tasks. In this work, we initiate systematic study of deferral rules for I.M cascades. We begin by examining the natural extension predicted class uncertainty to generative LM tasks, namely, the predicted sequence uncertainty. We show that this measure suffers from the length bias problem, ei- ther over- or under-empphaiiiing outputs based their lengths. This because LMs produce sequence of uncertainty values, one for each output token; and moreover, the number of output tokens is variable across examples. To mitigate this issue, we propose to exploit the richer token- uncertainty information im- plicit in generative LMs. We argue that naïve predicted sequence uncertainty cor- responds simple aggregation of these uncertainties. By contrast, we show that incorporating token-level uncertainty through learned post-hoc deferral rules can significantly outperform such simple aggregation strategies, via experiments range natural language benchmarks with FLAN-TS models. We further show that incorporating embeddings from the smaller model and intermediate layers of the larger model can give an additional boost in the overall cost-quality tradeoff.
INTRODUCTION
Recent advances in generative language modeling have yielded series of Trransformer-basee mod- els with remarkably improved quality on complex NLP tasks (Radford et al., 2018; Raffel et al., 2020; Brown al., 2020; Black al., 2022; Hoffmann al., 2022; Chowdhery et al., 2022; Wei et al., 2022; Chung et al., 2022; Tay et al., 2023; Anil et al., 2023; Touvron et al., 2023; Team al., 2023). Unfortunately, such models also involve significantly increased inference costs, which has motivated a series of efforts at reducing the same. These span careful infrastructure optimiza- tion (Chowdhery al., 2022; Pope al., 2022: Sheng al., 2023), rethinking the autoregressive decoding that underpin Transformers (Stern et al., 2018 Leviathan et al., 2023; Chen et al., 2023a; Sun al., 2023), modifications of the underlying model architecture (Dao et al., 2022), and model compression strategies (Frantar & Alistarh, 2023; Agarwal et al., 2023).
Cascading is one simple strategy to achieve more favorable cost-quality tradeoffs via adaptive in- ference. In two-model cascade, small model is invoked for most "easy" instances, while a few "hard" instances are deferred to a large model. Cascades have been widely explored in the vision domain (Viola & Jones, 2001; Trapeznikov & Saligrama, 2013; Bolukbasi et al., 2017; Huang al., 2018; Rawat et al., 2021; Kag et al., 2023; Jitkrittum et al., 2023), and have seen increasing adop- tion within NLP (Mamou et al., 2022; Varshney & Baral, 2022; Khalili et al., 2022; Dohan al., 2022: Chen et al., 2023b;a). Importantly, cascades can be implemented a black-box fashion over existing models, and do not necessitate any additional training.

---

Published as a conference paper at ICLR 2024
Confident: ho
PRODUCT MEAN measures affected by output longth
Fastume:
What capital of India?
PrODuCT
GeoMETTIc MENN
Labele mode
from model
MPP
Canbience Secre
Large
Den
0,6
nodel la weong
per-tokun uncortaiinty
Copturos nuanced
Confidant: You
Duggu
n0 2952
0.9xx0. 0.15
10.9 x0.21
& 0,11
TriviaQA
The capital DeN
Dais
Example 1: PRoDuCT ideniilee correct answer but MEAK does not.
Accuracy
0.10
What the capitss
Oupue:
0.4
0.4
0.08
The
2424 04 04 2406
(0.4 0.66
(0.4 0,6
0.06
Prdjuct
How to get single confidence score generative tasks?
Pae-token Confidence
The caple lnda
0,006
Example 2: MiAA identities cornect answer but PRODUCT does not.
deferred
Cascades Generative LMs
(b) Baselines
(c) Proposed Method
(a) In cascades, small models used for easy instances whereas hard instances routed to larger models. For generative LMs, the key challenge is to design a deferral rule based on uncertainties from multiple tokens. (b) Standard baselines which take the product and geometric mean of the probabilities are affected by the length of the output and perform sub-optimally. (c) Our proposed solution captures nuanced per-token uncertainty and outperforms both baselines.
Figure 1:
The key challenge in cascading is to design a deferral rule which decides whether to defer an input the larger model. The principles underpinning optimal deferral rules well known for the classification setting, where the standard recipe is to employ the small model's prediction confidence, canonically measured by its softmax probability output (Chow'y rule (Chow, 1970)). This simple deferral rule is remarkably hard to surpass in most natural settings (Jitkrittum et al., 2023).
However, the narrative is more complex for generative LMs. While one can naïvely translate Chow's rule for such models based on the softmax probability of the output sequence, this suffers from a length bias issue: one tends to defer longer predictions, regardless of quality. Further, simply normalizing the probability by sequence length tends to over-correct this bias, and defer shorter predictions. Intuitively, such naïve translations Chow's rule ignore a key distinction between the classification and generative LM setting: the former involves a single probability distribution over labels, while the latter involves a sequence of distributions over multiple tokens of the LM output: moreover, the number of output tokens differs across examples. This variability complicates summarizing the sequence of uncertainty (or confidence) values into a single deferral score.
To mitigate the length bias and capture fine-grained information from the uncertainty vector over tokens, we propose to use quantiles over the vector. Via experiments on a range of NLP benchmarks and FLAN-T5 models, show that these quantiles capture rich and complementary sources of uncertainty information from the uncertainty sequence vector and do better than the simple ag- gregation schemes like and average. However, observe that there is fixed quantile value which works across all datasets. This motivates us to learn deferral rule based on these quantile values as features, which can combine the strengths of these different quantile scores. We show that our trained deferral rule is the most consistently performant method compared to all the natural baseline aggregation strategies. We further show that using embeddings from the smaller model and intermediate embeddings from the larger model can give further performance improvement.
To summarize, our contributions are:
(i) We show that simple sequence-level LM confidence measures for deferral can yield strongly sub-optimal cost-quality tradeoffs, owing to length bias issue ($3.5).
(ii) We introduce token-level uncertainty in the form of distribution quantiles, and show that they can yield to consistently more effective cost-quality tradeoffs, owing to their finer- mer-grained measure of uncertainty. However, there is no fixed quantile which works well across all settings ($3.5).
(iii) We propose post-hoe deferral rule trained on quantile features, and show it can outperform all other strategies on range of NLP benchmarks for FLAN-TS models ($4.3). We further demon- strate that using the large model's intermediate embeddings can significantly boost performance.
2

---

Published as a conference paper at ICLR 2024
2 BACKGROUND AND PROBLEM SETUP
In this section, we discuss the relevant background and set up the problem of LM cascades.
Language models (LMs). Given finite vocabulary V (e.g.. tokens derived from Sentence- Piece (Kudo& & Richardson, 2018)), a language model (LM) defines a distribution p(- x) E A(V) over all possible tokens given any context æ (I1, (.1,...., ,.. bution over sequences y = (y),-... yn) E V for any n E N,, with p(y,,...,, Un x) = p(y
Vm This in turn defines a distri-
x)
n- p(Vi+1 æ. , U:) via the chain rule of probability.
i=
LMs based on Transformers (Vaswani et al., 2017) have proven increasingly popular in recent years. Such models are typically pre-truined on large corpora based on self-supervised objectives (Radford et al., 2018; Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020; Tay et al., 2022; Anil et al., 2023). These objectives involve different (input, output) pair constructions (æ, y) (e.g.. masking out the next token), upon which one minimizes the cross pross-entropy or log-loss, i.c. - log p(y x))
At inference time, given a trained LM and any input context æ, it is common to perform either clas- sification or generation. In the former, given a set of predefined choices e = [E (e.g.. yes, no }), one scores each p(c; x) and returns the highest scoring choice. In the latter, one performs sampling from p(- x:) to produce a suitable output string response, e.g., by temperature (Ficler & Goldberg, 2017), top-k (Fan et al., 2018), or nucleus sampling (Holtzman et al., 2020).
Model cascades. Cascades are simple, generic strategy to improve the inference cost-quality trade- off (Wang et al., 2022). Given a collection of models of varying inference cost, the key idea is to perform adaptive inference: "easy" samples afforded less computation compared to "hard" sam- ples. Concretely, for any test input, one first executes the lowest cost model, and uses a deferral rule to determine whether to terminate with its prediction, or to invoke the next cheapest model. Cascades can reduce the average inference cost if only a small fraction of inputs are deferred.
Cascades have long history of usage in vision (Viola & Jones, 2001; Huang al., 2018; Wang et al., 2018), where they are often applied for classification problems. Given an instance space X and label space y, the classification problem seeks a classifier h: X y with good average quality under some distribution IP, as measured by ((xx))=((((
h(x))] for some q(2, y, h(x)) E R.. In the simplest case, g(2,, y: 5) = (y = 5) measures the classifier accuracy.
h,(2) with inference costs (e.g.. latencies) c(L) < (22) Operationally, a cascade first invokes the "small" model (2) and then applies a deferral rule to de- cide whether to either defer prediction to the "large" model h22), or terminate with h(()ss prediction. More precisely, let r: X -> {0, }} denote the deferral rule, where r(x) = 1 denotes that we wish to defer to the large model. Then, the cascaded classifier (Kag et al., 2023; Jitkrittum et al., 2023):
Now suppose we have two classifiers ,
h cas (z) (x)(z) = 0) -h (=)=))(((()))
1(((z) = 1) h (x))
Given an input x, the corresponding cascade quality and cost are:
(z Clx, (x))))))) (x)) = ((((x) = 0) ( + (((x)) = 1) (6()) + c(2)).
((r(x) = 0) q(2, y. h (z)) + (((()) = 1) g(z, y. (x)
Ideally, one seeks to maximize quality given a budget on average inference cost:
max E Ee y, (x)(( Ex [Cx,, 0 (x)) S B. (0.11
(1)
We note that the average cost [x( cas (x)) is related to the deferral rate D(x) = P(r(x) = 1), via E (x, (=)) = (1) + D(z) e(2) In practice, one may set r(x) = (ss()) < t) for suitable
8: X + R and threshold E R. One may choose i to satisfy the inference cost constraint.
Now, we discuss cascades for generative LMs. This largely follows the setup described above, except that we
consider probabilistic models over sequences. Concretely, suppose have two language models p()),,(²)) with inference costs c()) c(?), Similarly, suppose q: Vm x pm" x A A(V)) - R4, is a measure of the quality of a given distribution over responses for a given prompt. A cascade cas of these models is parameterized by deferral rule r: 1m - {0, 1}, and is given by:
p cas x)) 1 (r((=) 0) TD) x²) + 1 (r((x) 1) ()) x))
3

---

Published as a conference paper at ICLR 2024
Given imput sequence and target sequence y. an LM cascade results in quality and cost
(x y. (- (-1 x)) = (((()) = 0) a y, p (: (:1 x)) + (((()) = =) q( y. (- x))
C( ( m)) = [r(a))))))))))) 0) et) 1 (r(()) 1) (c() c!
With these, we may construct a similar constrained objective as in Equation Similarly to the
classification case, we may parameterize the deferral rule as r(æ) ((s(z) < ()) for suitable deferral score function 8: V -> R (which may depend on the output of pl) æ)). We will investigate and analyze different types of deferral score functioos on different NLP tasks.
Recently, Chen al. (2023b) introduced the FrugalGPT system to achieve efficient inference via multiple strategies, including LM cascades. They also learn a deferral score to determine whether or not terminate prediction; however, this depends the input prompt and the generated output text, and does not consider the model's 's token-level uncertainty as we shall explore subsequently. few works have proposed to learn router which can decide which model to amongst a set of models depending upon the input prompt (Shnitzer et al., 2023; Hari & Thomson, 2023). However, their settings do necessarily consider models of increasing capacity and hence, their routers depend only on the input prompt not on the model confidence.
3 CONFIDENCE MEASURES FOR LANGUAGE MODEL CASCADES
A key question in the design of cascades is the choice of deferral rule. In this work, we seek to understand the behaviors of different types of deferral functions on NLP tasks. We start by discussing a few natural extensions of commonly used deferral rules for classification.
3.1 CHOW-SUN SUM AND CHOW-AVERAGE
Chow-Sum. We start with the multi-class classification setting where the output space [I....,}} and L E N4. In the simplest case, one may defer if the confidence in the prediction (x) of the small model is sufficiently low. There are several means of quantifying confidence in classification (Shafer& & Vovk, 2008; Guo et al., 2017; Kendall & Gal, 2017: Jiang al., 2018), but arguably the simplest is the predicted class probability (Huang et al., 2018; Wang et al., 2022; Jitkrit- tum al., 2023), which aligns with Chow's rule from the closely related problem (see Mozannar & Sontag (2020); Narasimhan et al. (2022)) of learning to reject (Chow, 1970):
s(x) - p (T) x),
(2)
where (.) x) denotes the predictive distribution over possible labels of the small model, and ò = arg max (y æ) denotes the predicted label.
To design deferral rule for LM cascading, a natural starting point is to mimic the predicted class probability (Equation 2): we may compute the (log) probability of the model generated sequence y,
Saum (x) = log p (1) x))
M
log (1) (1/41 x, Vy..... ".......,).. K),
(4)
1=0
We term this approach Chow-Sum, it involves the sum of per-token log probabilities. Analogous to the prediction rule for classification, may set ÿ = arg max (() æ), denoting by V the set of all sequences. This requires searching over an exponentially large set; however, efficient approximations via greedy or beam search are feasible.
Chow-Average- Chow Sum computes the aggregate sequence-level log- log-rrobability. A natural variant is the average of the per-token log-prbbabilities. This is equivalently the length normalized log- probability, or the log- log-peeplyxity (Chen et al., 1998):
Ø 1
Savg
xx:
log logp()) "(/41 x. ".....,,l ,,))-
(5)
İ=0
Note that ý may be computed as above, without incorporating any length-normalization.
4

---

Published as a conference paper at ICLR 2024
1.0
Model output tokens:
Token probability
0.8
0.6
The journey always leads to Es it i., where the singer visit their sponsor.
0.1
0.2
children..
0.0
Tokon index (sorted)
Figure 2: Example of tokenized FLAN-T5 Base model output on WMT FR - EN, Red tokens have a significantly higher uncertainty compared the others, as shown in the left plot. (For each red token, we note the rank of its uncertainty score in the right plot.) However, due to the large number of other more predictable tokens, Chow-Sum gives the output a relatively high score.
3.2 LIMITATIONS OF CHOW-SUM AND
Given that Chow tracks closely with the well-established Equation 2, is tempting to conclude that this emphatically solves the LM cascade problem. However, LMs be susceptible to the length bias problem (Murray Chiang, 2018; Adiwardana et al., 2020): shorter, lower quality responses may receive higher probability than longer, higher quality responses. This may be seen as a consequence of the fact that each p(- æ, .1...... y/) provides an imperfect estimate of a "true" probability p" (- æ, //1 .i...... V/).
.........
and that errors in these estimates compound with sequence length.
The length- issue naturally suggests using an average instead of sum of log- bogg-robbbblities. How- ever, Chow-Averr Average can over-correct for this length bias, and preferentially defer shorter predictions. We will see concrete examples of this behavior in $3.5.
More fundamentally, both approaches are inherently limited in the way they aggregate token-level uncertainty. In particular, computing the sum or average of per-token probabilities may mask set- tings where individual tokens are highly uncertain, even if the entire sequence has reasonably high probability. Such token-level uncertainty may be highly important in certain settings such as fact- answering: here, an LM may (correctly) highly confident on articles and other grammatical prim- itives, but these of less interest than confidence on tokens corresponding to entities (say). This observation has been previously noted and exploited to allow certain "easy" tokens to be quickly decoded (Schuster al., 2022). This observation has also been exploited in knowledge distillation by using different teaching modes for "easy" versus "hard" tokens (Zhong et al., 2024).
Figure presents example of this phenomenon the WMT FR EN dataset (details in $3.5): there can be cases where most tokens are highly predictable (i.e., p(v( x, V/....,,.--1 wh-1) ~ 1), but few tokens are less predictable (i.e., p(v' Ir, "/. v(-1) 0). In such cases, Chow-Sum can yield overly optimistic uncertainty estimates. This motivates to consider richer representations of uncertainty which can capture token-level uncertainty, instead of simply computing the sum or average over the sequence.
3.3 BEYOND CHOW-Stu SUM AND CHOW-AVERAGEE CHOW-QUANTILR
The discussion in $3.2 suggests there is value in considering the following generalization of the maximal sequence probability:
Squant (x, a) = quantill (p) (z) ()) xr ",,i).....
where y arg max yEV- (1)) (y) æ) is the most probable output sequence (or a suitable approxi- mation thereof) under UU) Here, quantile, computes the a-quantile of the set of per-token log probabilities. For instance, a would correspond to taking the minimum per-token log proba- bility the deferral score. One may regard quantiles as another way of converting the token-level uncertainty distribution into a single score, which are capable of capturing richer information from the token-level uncertainty distribution. For example, employing the maximal token uncertainty (i.e.. the minimum of the per-token probabilities (Stengel-Eskin & Van Durme, 2022)) can be useful in scenarios where most tokens predictable, but few important tokens are not (per Figure 2).
5

---

Published as a conference paper at ICLR 2024
MNU: Base Large
TrvbDDA Base Large
WMTT DE-+FR; Dase Large
0.41
Accutt
0.70
0..10
SLEUNE score
0.40
Chen-Sam
0.05
0.08
0.39
Chow Quantle
0.30
Chox-Quniledd
0.08
0.37
Chow Quantle-d.
20
40
00
80
1.00
20
40
60
30
100
20
40
80
100
15.00 of examples deferred
16 examples deferred
15 examples deterred
Figure 3: Deferral curves on MNLI, TriviaQA, and WMT DE - FR for a FLAN-T5 Base - Large cascade. Chow-Quant le consistently outperforms Chow- Sum and Chow- Average. This confirms there is value in going beyond naïve sequence probability as an uncertainty measure for cascading.
Next, we evaluate all the aforementioned approaches multiple NLP tasks. For that, we describe the experimental setup used for the evaluation.
3.4 EXPERIMENTAL SETUP
Models. We employ FLAN-T5 (Chung et al., 2022) models, which are TS models (Raffel et al., 2020) that have undergone instruction tuning (Wei al., 2022). This family offers range of models of different sizes, spanning Small (80M parameters) to XXL (11B parameters). and have demon- strated strong few-shot performance focus on a two-model cascade of FLAN-T5 Base and FLAN-T5 Large. Results for other models are included in the Appendix. We employ these models with few-shot prompting and greedy decoding.
range of NLP benchmarks. In the body, primarily
Evaluation. We summarize performance using the deferral curve. Consider a candidate deferral rule produced by thresholding s(x) E via r(x) ((s(x) < t). Let p denote the associated cascaded LM. For a fixed threshold f, we may compute the associated deferral rate P(r(x) = 1), and the associated cascade quality E[Q(x, y. y.paae pm (- x))]. The deferral curve is produced by plotting the trade-off off between deferral rate and cascade quality as is varied. As scalar summary, we report the area under the deferral curve (AUC-DF). For a given dataset, higher AUC-DF values indicate better deferral curves. Note that the range of AUC-DF values vary across datasets, however.
Datasets. In the body, we show deferral curves for three different NLP tasks: MNLI (Williams al., 2018), a multi-class classification problem; TriviaQA (Joshi et al., 201 7), a closed-book question an- swering problem; and WMT DE - FR, a translation problem. We report AUC-DF numbers for an expanded dataset pool. These span Classification (IMDb (Maas et al., 2011), SuperGLUE (Wang et al., 2019a), MNLI (Williams et al., 2018), ANLI (Nie et al., 2020)): Question answering (Triv- iaQA (Joshi et al., 2017), NaturalQA (Kwiatkowski et al., 2019). TyDiQA ID, SW, FI } (Clark et al., 2020)); Reading compreherssion (I Lambada (Paperno et al., 2016), 6), SQuAD (Rajpurkar et al., 2016)); Translation (WMT 14: EN FR (Bojar al., 2014), WMT 19: DE FR (Foundation), and WMT 14: FR + EN (Bojar et al., 2014)); and Common-sense reasoning (Winogrande (Sak- aguchi al., 2021)). Note that we treat all problems as finding text text mapping. So for classification tasks, we encode the classes as strings, For evaluation, we take the model's output text and perform a string comparison the label. See Table (Appendix) for more details.
3.5 EVALUATING CONFIDENCE MEASURES FOR CASCADES
We now empirically validate the critiques in $3.2, demonstrating that using the standard se- quence probability (Chow-Sum) to defer can result in overly penalizing longer sequences. More- over, Chow-Average flips this bias and overly defers shorter sequences. We then verify that the Chow-Quantile generalization proposed above capture richer token-level uncertainty.
Summary of results. Figure 3 plots the deferral curves for three datasets. We see that (for particular choice of quantile), Show-Quantile consistently outperforms standard Chow-Sum and Chow- Average AUC-DF values for all datasets are included in Table 4. Looking at the table, we see that while a particular choice of quantile is able to do well, there is single consistent choice that performs well across tasks. Next, we discuss insights into the results by using WMT FR-EN as an example dataset.

---

Published as a conference paper at ICLR 2024
# Outputt tolens Bage mode
WMT FR. EN: Base Large
Rase large
200
Taken grabab lity
200
BLEURT Score
0..6
150
(rea set
100
Chn-leaage
50
(Ten Quartle- Drace
0.4
examples deferred
100
index
30
(a)
(b)
(c)
Figure 4: (a) Relation between deferral rules and output length (number of tokens) for WMT FR - EN dataset and FLAN-T5 Base Model. Chow-Sum tends to defer longer prompts: the prompts with lowest scores have notably higher length than those with higher scores. Interestingly, Chow-Average over-corrects this bias: it tends overly defer prompts with lower length. Chow-Quantile- again defers longer outputs more whereas Show-Quantile eee initially focuses more on the shorter out- puts. Oracle refers to deferring using the difference of BLEURT scores of the two models. Oracle also tends to defer longer outputs, but the preference is moderate as compared to Chow-Sum. (b) Cor- responding deferral curves. (c) Analysis of token-level uncertainty on WMT FR + EN. For each token index i, the corresponding average prediction probability across all examples (with prediction length < i) for FLAN-T5 Base. We observe that later tokens tend to have higher probability, i.c., the model is generally the most uncertain for early tokens.
For the Australian Government, Keith Brown called Mr. Carmichael "unjustly" to support the inclusion of the Ecosse in the HS2. network.
announcement the release of new album by David Bowie has left everyone a little bit little a little little a little little bit little bit little bit little bit
?? ?? ?? ??
?? ??
The lyric Adéro, concert-recording held last August, added the repertoire, with the competition of the coloratura soprano Marie- ?? ve Munger. The lyric Adéro was added to the repertoire in
"The Emergency Room the Hotel-Dieu must be closed soon possible, and for us, it is November 4th," says Lo ?? с Capron, Chairperson of the APHP Medical Committee (MCM), who supports the direction.
August. The lyric Adéro added the repertoire in August.
Obama Lays out his reform plan
A memorial ceremony for the victims
Figure 5: FLAN-T5 Base predictions on WMT FR - EN. Top: Predictions with the longest lengths. These tend to have repetitions, indicating low quality output that could be resolved with larger model; length does have some signal in identification of good candidates for deferral. Middle: The predictions which Chow- Blow-quantileee tends to defer. This quantile tends to identify repetitions and "??" (unknown tokens) as these tokens tend to have lower probability. Bottom: The predictions which Chow-Quantile-o.. tends to defer. This quantile prioritizes deferring shorter inputs.
Why can Chow-Sum and Chow-Average be sub-optimal? To better understand the reason for the sub-optimality of Chow-Sum, Figure 4 studies the relation between the deferral rule and output length. Specifically, for each test prompt æ, let ÿ denote the result of decoding via the small model in the cascade. For each deferral rule, we compute the corresponding score s(x) and the length (ý)- For ease of comparison across different rules, we convert each score into the corresponding quantile.
Figure reveals that Chow-Sun Sum tends to defer prompts with larger output lengths: the prompts with lowest scores have notably higher output length than those with higher scores. This makes us ask if is all bad to defer prompts with longer outputs? We observe that for the Base model on the WMT datasets, even the BLEURT (Sellam et al., 2020) scores tend to have non-zero negative correlation with output lengths (Table 3, Appendix). A closer look at the model predictions shows that longer predictions tend to have repetitions, as shown in Figure 5 (Top) and hence, are good candidates for deferral. (The shown predictions are truncated for clarity.)
This shows that there is signal in output length overly biased towards deferring longer predictions and hence, can be sub-optimal. Interestingly, Chow-Aver Average over-corrects this bias: it tends overly defer prompts with lower output length.
deferral score. However, Chow-Sum is

---

Published as a conference paper at ICLR 2024
Why does Chow-Quantile help? As discussed above, Chow-Quantile is able to capture rich in- formation from the token-level uncertainty vector. We discuss below why Heov-Quantlle-ee and Chow-Quantile-o.. 8 work well with respect to the WMT FR-EN dataset.
Chow-Quantile-o: The main insight is that the minimum token probability is able to capture repe- titions and "??" (unknown tokens), they generally tend have lower probability values and more uncertain. This confirms our understanding that quantiles can capture richer token-level uncer- tainty. We show two examples with the minimum -oo--uuatile-0 value for the WMT FR-EN dataset and FLAN-TS Base in Figure 5 (Middle).
Chow-Quantill ieeeiilllgge 8: Interestingly, Chow-Quantile-e e.. 8 tends to defer shorter predictions. We show two examples with the minimum Chow-Quantile-o.1 value in Figure 5 (Bottom).
To understand this, Figure 4e shows the average token probability as a function of the token in- dex, for the WMT EN + FR dataset and FLAN-T5 Base model. As the token index increases, the average probability increases; i.e,, the model tends to become more confident. Hence, the Chow-Quantile-o. 8 is able to focus more on the shorter, uncertain outputs.
In summary, we have seen that Chow-Quantile- is able to focus more on identifying the presence of repetitions and unknown tokens "??" whereas Chow- arrow-Quantllee ae-anilleo..e is able to capture the uncertainty in shorter predictions better. Thus, we conclude that different quantiles are able to capture richer and complementary measures of uncertainty. Moreover, we have already seen that there is no one quantile which works well across all datasets. Given this, a natural option is to learn how to combine various quantiles for given dataset, which we consider next.
4 POST-HOC DEFERRAL RULES
We show that training post-hoc deferral rules based on probability quantiles, and (optionally) suit- able embeddings from the small and large model, can significantly improve the cost-quality tradeoff.
4.1 POST-IIOC DEFERRAL RULE TRAINING
The idea of learning when to defer in cascade follows recent line of work on classifica- tion (Narasimhan al., 2022; Kag et al., 2023; Jitkrittum al., 2023). In a nutshell, for suitable feature mapping ((x) E IRd, we seek to learn deferral score s(x) R via a standard model class (e.g., feedforward network). We then defer using r(ær = ((>(x) < t).
To construct the input features, we set (xx) to be fixed length vector comprising the per- token probability quantiles from the small model. Additionally, we add the aggregate scores from Chow-Sum and thow-Average (see Appendix F). To fit the deferral scorer a training set of input prompts } we minimize an empirical loss against a set of target labels {2;). For tasks based on accuracy, we set 2, 1 iff the large model is correct, and the small model is incorrect on the given example; i.e., it would benefit to defer to the larger model. We then fit the scorer with the binary logistic loss. For translation tasks, the target is the difference of BLEURT scores of the two models; we train with the square loss. We call this method Post post-Hoc-Quanteo Quantile ll (see Appendix D for details).
4.2 LEVERAGING INTERMEDIATE EMBEDDINGS
The above target labels exploit information from the large model during training. Importantly, we cannot directly use such information during inference, as it would require querying the large model (and thus defeat the point of cascading). Note, however, that in some settings it may be feasible to use intermediate information from the large model, e.g., token embeddings from an intermediate layer. Prior work has noted that such intermediate embeddings can often contain valuable information by themselves (Schuster et al., 2022).
Inspired by this, we thus study the viability using such intermediate embeddings for training post- hoc deferral rules. For encoder-decoder models such as T5, such embeddings can be from either the encoder, decoder, or both. We study two methods one which uses the final decoder embeddings of the smaller model averaged over all tokens. We call this method Post-Hoc- -mbe--- In the second method, we add the first token embedding from the first decoder layer of the large model as another input to the post-hoc rule. We call this method - Embed-22 1+2.
8

---

Published as a conference paper at ICLR 2024
Dataset
Choe-Sur
Choe-Average
Chow- Quantiicn
Chow-quantale-o.u
Choe-Quartile-c.5
Post-Hac-Qunntile
Nost-ioc-EwRed-1-
ANLURI
0.524 (+1.59)
(+3.9)
0.534 (+6.58)
0.315(+2279)
(+1.79))7))
0.534 (+6.5)
0.563 (+12.51)
ANLI-2
0.445 (411.67
(+0.67)
0.469 (+1.35)
0.441) (.067)
0.441 (-0.5)
0.442(-4.22)
0.462 (14.36
ANLI-R
(+3.5)))50)
0.423 (+5.76)
0.417 (4.55)
0.25 (+6.51)
0.424 (+6
0.425 (+6.1)
0.440 (+ (+10.45)
Bool
0,838 (+2.57)
0,838 (+2.57)
0338 (+2.5))
01538 (+2.57)
0.540 (+281)
0.854 (+4.64)
IMDh
0.964
0.964
0.964 (-1.15)
6.965
Lambuda
0.694
0.687
0.627 (-2.63)
0.642 (:0.31)
0.631
0.777 (25.12)
0.691 (+7.29)
(+30.44
0.722 (+12.34)
PIQA
0.712
0.708 (+0
0115(1141)
0.706 (4)
0.785 (-0.00)
(+1.70)
0.727 (+3.26)
SQiaD
(+2.24)
0.408 (-1
0.410(+2.30)
0.399 (..49)
0.39%(-074)
199
0.412
TRiviQ
0.073.-13300
0.087 (+3.57)
(-13.09)
0.88 (+4.76)
0.086 (-1.36)
0.997 (+15,47)
0.100 (+19. 56)
TyDiGA-F1
0.352 (+33100
0.522 (+00000
0.325 (+ 861
0.318 (-1.24)
(-312(-)111)
01331 (+2.79)
0.338 (+5.27)
TyD:QA-ID
0.247 (+7.39)
(+391
(.265 52)
0:330 (40.00))
0.230 (-0.00)
0 035 (+2.1 17)
0 0.242 (+5 35)
TyDsQA SW
0.170 (+9.67)
(:5.66)
(+7.09)
0,141 0.141 (9.05)
0.140
0.163 (+5.16)
0.162 (95)
Winggrude
0.648 (42.56)
0.648 (+2.56)
0.645 (42.56)
0.649 (+2.52)
0.648 (-2
0.648 (+2. 263
0.670 (45.877
WMT DE
0. 0,390 (+0.00)
0.392 (+0.51)
0,391 (+0255)
0.396 (+1.5)
0.401 (-2.82)
0.404 (+3.5K)
0.404 (+3.69)
WMI
(+0:92)
0.436 (11661
0.435 (+0.92)
0.439
(+2.32)
0.444
(42.8))
0.505 (-0.99)
(6618(42.82)
0.595 (40.9)
0,614
(-2.99)
Table 1: Table showing area under the deferral curve (AUC-DF). Numbers in brackets indicate % change over the random baseline. The post-hoc deferral rule approach is the most consis- tently performant method. We use bold & black to denote the best method amongst Chow-* and Post-Hoc-Quantia methods. We color Post-Hoc- Embed-1+ 1+2 with blue if is the best amongst all.
Lange
TrivieQit: Lerg
DE-FRI Lerge
0.75
0.12
0.42
0.71 Accuraty
0.10
0.40
0.65
0.09
BLEURT
Host-Hoc.Emmnng
100
deferred
exemmpapee deferred
deferred
Figure 6: Deferral curves MNLI, TriviaQA, and WMT DE - FR for FLAN-T5 Base Large cascade. The post-hoc deferral rule yields consistently on- mn-par or superior performance compared to other methods. Further exploiting intermediate embeddings of the large model yields large gains. See Table 4 for summary of deferral curves on multiple datasets.
We remark that previous work (Ren et al., 2023) has shown the value of final layer embeddings for selective generation, and the related problem of -ittr---iittiittiinn detection. We caution also that while not as expensive as querying the entire large model, even extracting intermediate decoder embeddings can involve non-trivial cost. Nonetheless, in settings where some increase in cost is acceptable, it is of interest whether these embeddings offer significantly valuable information.
4.3 How WELL DOES POST-HOC DEFERRAL WORK?
For the same experimental setup as the previous section, Table 4 summarizes the area under the deferral curve (AUC-DF) across various datasets. Numbers are averaged over 5 random runs. We see that the post-hoc deferral rule approach consistently performs the best; in scenarios where other methods are better, the difference is minimal. For more fine-grate -ne-grained analysis, Figure plots the full deferral curves on MNLI, TriviaQA, and WMT. In the plot, Chow-Quantil Sitantile-Bess Best chooses the best method out of Show--uantile--- based the validation data. We see that post-hoc deferral rules are generally on-par with the Chow- * family of (non-learned) deferral rules.
We that Post-Hoc- Emboc-11med1111111 1 is able to improve upon Post-Hoc fost-Hoc-Quantinn ll and Chow- Setow--uantlee--- methods slightly but is slightly inferior compared to the Post-Hoc-Embae 1+2 method. This intu- itively makes sense as this has more information compared the Nost-Hoc-Quantil method but still does not include any information about model 2.
Strikingly, further using the larger model's intermediate embeddings can lead to significant im- provements across all deferral rates, particularly for classification tasks. Intuitively, the first token's 's intermediate embedding could have lot of information for classification tasks, where the answers typically comprise of a single word and only couple of tokens. However, for generation and trans- lation tasks with longer outputs, the main token containing the answer could be present later in the sequence and thus, there may be limited use of using the first token embedding. One may wonder if we really need quantiles to train the post-hoc deferral rule. In Appendix E, we show that naïvely passing the probability vector with padded zeros performs poorly in many cases.

---

Published as a conference paper at ICLR 2024
REFERENCES
Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoe V. Le. Towards a human- like open-domain chatbot. CoRR, abs/2001.099777, 2020. URL https: https://arxiv.org httgg/ab//eee/1 09977.
Rishabh Agarwal, Nino Vieillard, Piotr Stanezyk, Sabela Ramos, Matthieu Geist, and Olivier Bachem. GKD: Generalized knowledge distillation for auto-regressive sequence models, 2023.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey. Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang. Gustavo Hermandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry. Christo- pher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa De- hghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hycontack Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Mar- cello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope. Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang. Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yun- han Xu, Linting Xue. Pengcheng Yin, Jiahui Yu, Qiao Zhang. Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. PaI.M technical report, 2023.
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He, Connor Leaby, Kyle McDonell, Jason Phang. Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. GPT- NeoX-20B: An open-source autoregressive language model. In Angela Fan, Suzana Ilie, Thomas Wolf, and Matthias Gallé (eds.), Proceedings of BigScience Episode #5 Workshop on Chal- lenges & Perspectives in Creating Large Language Models, pp. 95-136, Dublual+Dublin, May 2022. Association for Computational Linguistics. doi: 10.18653/ http:///222.iiiicceeeeee 1.9. URL https: /ttps://aallatthlllee
org/2o22.oe. 2022 bigsciincee-1..
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale Tamchyna. Findings of the 2014 workshop on statistical machine trans- lation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 12- 58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. URL http:/ http://wwwwcclccccccc org/ hwwph/lloyyyy/////////
3302.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for fast test-time prediction. In International Conference on Machine Learning, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Scan-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling, 2023a.
10

---

Published as a conference paper at ICLR 2024
Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How to use large language models while reducing cost and improving performance, 2023b.
Stanley F Chen, Douglas Beeferman, and Ronald Rosenfeld. Evaluation metrics for language mod- cls. In DARPA Broadcast News Transcription and Understanding Workshop, pp. 275-280, 1998.
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the relation between sensitivity and accuracy in in-context learning, 2023c.
C. Chow. On optimum recognition emoor and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46, 1970. doi: 10.1109/TT..190.100555
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Samjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. PaLM: Scaling language modeling with pathways, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction- finetuned language models, 2022. URL https://arxiv.. org. http://abs/221....// 2211.1141..
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A benchmark for information-seekinn question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 2020.
Tri Dao, Beidi Chen, Nimit S Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Re. Monarch: Expressive structured matrices for effi- cient and accurate training. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 4690-4721. PMLR, www.s//eessvveee http//ddd2aa.ammmmmm html.
17-23 Jul 2022. URL
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT:
Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 00.18653/1//119144555 URL https: l/aclanthology.com orgeeig-gg/iii-t
David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein, Kevin Murphy, and Charles Sutton. Language model cascades, 2022. URL https///::/arrii.rrrr http:://aaxxivooo//////////// 10342
Ekaterina Fadeeva, Roman Vashurin, Akim Tsvigun, Artem Vazhentsev, Sergey Petrakov, Kirill Fedyanin, Daniil Vasilev, Elizaveta Goncharova, Alexander Panchenko, Maxim Panov, et al. Lm- polygraph: Uncertainty estimation for language models. arXiv preprint arXiv:2311.07335. 2023.
11

---

Published as a conference paper at ICLR 2024
Angela Fan, Mike Lewis. and Yann Dauphin. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Pa- pers), pp. 889- 898, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v 1/P3 1 8- 1082. URL https: //ac lanthology wrg/o18-
Jessica Ficler and Yoav Goldberg. Controlling linguistic style aspects in neural language genera- tion. In Proceedings of the Workshop on Stylistic Variation, pp. 94-104, Copenhagen. Denmark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/v/W11-449122 URL https::/aclaattho logy- wwgg/177-g9/1111
Wikimedia Foundation. Acl 2019 fourth conference on machine translation (wmt19), shared task: task.httkk.tkk.t
Machine translation of news. URL http://www.stateee http:p//www..saatttttootttttttt/
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10323-10337. PMLR, 23-29 Jul 2023. URL
.ml.. wwwpppeessss.eeeeeeeepepeppp guees/v222/frantar333 wwt//arrantaaaaaaaaaaaaaaaaaaa html.
Adam Gleave and Geoffrey Irving. Uncertainty estimation for language reward models, 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML' 17, pp. 1321-1330. JMLR.org, 2017.
Surya Narayanan Hari and Matt Thomson. Tryage: Real-time, intelligent routing of user prompts to large language model. arXiv preprint arXiv:2308. / 16011 2023.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Mat- sushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint arXiv:2302.09210, 2023.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https: //openreview.c net/ httt/forummid=ryg@@yer
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form compe- tition: Why the highest probability answer isn't 't always right. In Proceedings of the 202/ Confer- ence on Empirical Methods in Natural Language Processing, pp. 7038-7051. Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: wttp://221....1mmmmeeee www.lemmain.nneeeeeemmm URL https:
10.18653/v1/2021
org/ 2021. emnlp-main.stt 564...
Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger. Multi-scale dense networks for resource efficient image classification. In International Confer- ence on Learning Representations, 2018
Heinrich Jiang, Been Kim, Melody Y. Guan, and Maya Gupta. To trust or not to trust a classifier. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NeurlPS' 18, pp. 5546, Red Hook, NY, USA, 2018. Curran Associates Inc
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 977, 2021. doi: 10.1162 URL.
https://aclanthhr wewgy..org/ooo/ 2egg tacl-1.57.
Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. When does confidence-basee cascade deferral suffice?, 2023.
12

---

Published as a conference paper at ICLR 2024
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th An- mual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/11/117-1147 URL
httprttoooooooooo/t---aaa-- org/P17-1177.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bow- man, Stanislav Fort, Deep Ganguli, Danny Hemandeez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Kamal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Ka- plan. Language models (mostly) know what they know, 2022.
Anil Kag, Igor Fedorov, Aditya Gangrade, Paul Whatmough, and Venkatesh Saligrama. Efficient edge inference by selective query. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL https://openreview werrtorumeeeeeeeeeeeeeeeeeeeeeee dd--mdddddddpddmmmmmm
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5574-5584, 2017. URL https://prrccedings.. neurips.cc/ paper/2e17/hasiee http://///////eeo666oooooooooeee -btp://aabaaaaaaaaaaaaaaaataaace Abstract. html.
Leila Khalili, Yao You, and John Bohannon. Babybear: Cheap inference triage for expensive lan- guage models, 2022. URL
Taku Kudo and John Richardson. SentencePiece: A simple and language independent sub- word tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 66-71, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DD
URL https://acc http://ccaannttoogyy.oooo wwwo/or//////oooe
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In The Eleventh International Conference on Learning Representations, 2023a. URL https://openneevew...
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Clam: Selective clarification for ambiguous questions with generative language models, 2023b.
Sawan Kumar. Answer-level calibration for free-form multiple choice question answering. In Pro- ceedings of the 60th Anmual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers). pp. 665-679, Dublin, Ireland, May 2022. Association for Computational Linguis- tics. doi: 10.18653/ lon... URL httt:::s//claanthol.gg... org/ 202. acl- acl-long.ee.....o Long.49.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019, doi: 10. 10.1162/tacL URL https://aclaattholl org/ (aagg/q1--000 1026.
Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via specula- tive decoding. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 19274-19286. www.eeeeiiin.......eeeeee httss//v22//evviiaeee /eeiataaiiaataaaaaaaaaaa html.
PMLR, 23-29 Jul 2023. URL
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Anmual Meeting of the Association for Computational Laciomaaliinngggn Linguistics: Human Language Technologies, pp. 142-150,
13

---

Published as a conference paper at ICLR 2024
Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
URL https:
Jonathan Mamou, Oren Pereg, Moshe Wasserblat, and Roy Schwartz. TangoBERT: Reducing infer- cost by using cascaded architecture, 2022. URL http:///aaxxv..ogabb///eaee 06271.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872, 2022. doi: a0.1162/aal....00994 URL https://
t.cl-1.50.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to expert. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 7076-7087.
PMLR, 13-18 Jul 2020.
Kenton Murray and David Chiang. Correcting length bias in neural machine translation. In Pro- ceedings of the Third Conference Machine Translation: Research Papers, pp. 212-223, Brus- sels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v1/ W18-6322. URL https::/aclantthllogg/oooo wtg//18- 2tt//W88-3322. 6322.
Harikrishna Narasimhan, Wittawat Jitkrittum, Aditya Krishna Menon, Ankit Singh Rawat, and San- jiv Kumar. Post-hoc estimators for learning to defer to an expert. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Sys- tems, 2022. URL http:////oppeeeeeeeeeeeeeeeeee
jg6Sf6tuF7.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: new benchmark for natural language understanding. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
URL https://api.
semanticscholar. samantiicccclaar..oooooooppsssso orgg/CooppsSID::555522855
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset:
Word prediction requiring broad discourse context. In Proceedings of the 54th An- mual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1525-15344 Berlin, Germany, August 2016. Association for Computational Linguisties. doi: 10.18653/v1/P161 144. URL httttss:/aalaaathool. org/P16 1144.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Lev- skaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling trans- former inference, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving lan- guage understanding by generative pre-training. https://cdn. www.ai.ennnaa.... www.//eeeaaccccoorrrr.... language httpppprrriiee//lnuuuaaae understanding paper. pdf, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 11:14001-114::: 2020. URL http://jmlr. org/ hrggpapers/vvz/ 20-074. 30-074.html html.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/vI/DI-- 1264. URL https::/aclantholll org/ D16-1264.
Ankit Singh Rawat, Manzil Zaheer, Aditya Krishna Menon, Amr Ahmed, and Sanjiv Kumar. When in doubt, summon the titans: Efficient inference with large models. arXiv preprint / 10. 10305, 2021.
14

---

Published as a conference paper at ICLR 2024
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan, and Peter Liu. Out-of- cuttof--ditributtinn detection and selective generation for conditional language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://apenrevieww net/ f -rrrummidd-UUSSDeeeee
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An ad- versarial winograd schema challenge at scale. Commun. ACM, 64(9)999-06, aug 2021. ISSN 0001-0782. doi: 10.1145/347438L. URL https://doi.org/// http://...14533444444 3474381.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. Confident adaptive language modeling. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
URL https://opentrviewwneee httpe/vwwwwwwweefreeeeeeeeeeee wttp://wwwwweeeeeeeeoeoooneeaaae www.://miiiyyyaaaa31aaa
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892 Online, July 2020. Association for Computational Linguisties. doi: URL https: http://aaccanntooooooo/////////a wttp//22.acllmama.. acl- 704.
10. 1/2020.acl-maaii.t
Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 912):371-421, 2008. URL http::/jmmrr..//////////// http:///jjir.oorrppppee///////// httppperr///////eeeeeeeeeeeeeeee wwtsssaafeerrro html.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher Re, Ion Stoica, and Ce Zhang. FlexGen: High-throughput generative inference of large language models with a single GPU. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 31094-31 116. PMLR, 23-29 Jul 2023. URL https::/procceddngg..tr.... v202/sheng23a. html.
Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun, Justin Solomon, Neil Thompson, and Mikhail Yurochkin. Large language model routing with benchmark datasets, arXiv preprint arXiv:2309.15789 2023.
Vaishnavi Shrivastava, Percy Liang, and Ananya Kumar. Llamas know what gpts don't show: Sur- rogate models for confidence estimation. arXiv preprint arXiv:2311.08877.0000 2023.
Elias Stengel-Eskin and Benjamin Van Durme. Calibrated interpretation: Confidence estimation in semantic parsing. arXiv preprint arXiv:2211.07444, 2022.
Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autore- gressive models. CoRR, abs/181 1.03115, 2018. URL http://arxiv. wwwgg//////......//eor/////
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, Felix Yu, Michael Riley, and Sanjiv Kumar. Spectr: Fast speculative decoding via optimal transport. In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023. URL https:
//openreview. net net/ wtt.foormmidoe www.runndddddnndnuuuu...e.o
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, 2022.
Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung. Dara Bahri, Tal Schuster, Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Met- zler. UL2: Unifying language learning paradigms. In The Eleventh International Conference Learning Representations, 2023. URL
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonogiou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford. Erica Moreira, Kareem Ay- oub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov,
15

---

Published as a conference paper at ICLR 2024
Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Laksh- man Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Prolcev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre. Marcello Maggioni, Fred Alcober. Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Martin Chadwick, Gaurav Singh Tomar, Xavier Gar- cia, Evan Senter, Emanuel Taropa, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomênech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanic Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Yujing Zhang, Ravi Addanki, Antoine Miech, Annie Louis, Laurent El Shafey, Denis Teplyashin, Geoff Brown, Elliot Catt, Nithya Attaluri, Jan Balaguer, Jackie Xiang, Pi- dong Wang. Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang. Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang. Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnll, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz. Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Do- minika Rogozińska, Vitaly Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwami, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Al- bin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Pa- tel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, Hanzhao Lin, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fer- nando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yinn Chang, Paul Komarek, Ross Mcllroy, Mario Lučic. Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yong Cheng, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butter- field, Justin Chung, Paul Kishan Rubenstein. Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko linuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Flena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Au- rko Roy, Zizhao Zhang, Ada Ma. Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Dou- glas, Rajkumar Samuel, Mingqiu Wang. Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely. Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víctor Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangx-
16

---

Published as a conference paper at ICLR 2024
1aoyu Feng. Wojeiech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Salch, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez. Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FizzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinherg, James Cobon-Kerr, Tara Sainath, Mari- beth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, YaGuang Li, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Pagannii, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Mísra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mital,, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Gamaleldin Flsayed, Ed Chi, Mahdis Mahdieh, Ian Tenney, Nan Hua, Ivan Petrychenko, Patrick Kane, Dylan Scandinaro, Rishub Jain, Jonathan Uesato, Romina Datta, Adam Sadovsky, Oskar Bunyan, Dominik Rabiej. Shimu Wu, John Zhang, Gautam Vasudevan, Edouard Leurent, Mah- moud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Betty Chan, Pam G Rabinovitch, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Sahitya Potluri, Jane Park, Elnaz Davoodi, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Zhe Chen. Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Chris Gorgolewski, Peter Grabowski, Yu Mao. Alberto Magni, Kaisheng Yao. Javier Snaider, Norman Casagrande, Paul Suganthan, Evan Palmer, Geoffrey Irving, Edward Loper, Manaal Faruqui, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Michael Fink, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikotaj Rybinski, Ashwin Sreevatsa, Jennifer Prendki, David So- ergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Di- ana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marin Georgiev, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Alena Repina, Xi- hui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Minnie Lui, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Lam Nguyen Thiet, Daniel Andor, Pedro Valenzuela, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kury- lowicz, Sarmishta Velury, Sebastian Krause, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Tejasi Latkar, Mingyang Zhang, Quoc Le, Blena Allica Abellan, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Sid Lall, Ken Franko, Egor Filonov, Anna Bulanova, Rémi Leblond, Vikas Yadav, Shirley Chung. Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Hao Zhou, Alck Dimitriev, Han- nah Forbes, Dylan Banarse, Zora Tung, Jeremiah Liu, Mark Omernick, Colton Bishop, Chintu Kumar, Rachel Sterneck, Ryan Foley, Rohan Jain, Swaroop Mishra, Jiawei Xia, Taylor Bos, Ge- offrey Cideron, Ehsan Amid, Francesco Piccinno, Xingyu Wang. Praseem Banzal, Petru Gurita, Hila Noga, Premal Shah, Daniel J. Mankowitz, Alex Polozov, Nate Kushman, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Anhad Mohananey, Matthieu Geist, Sidharth Mudgal, Sertan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp. Christopher Yew, Quan Yuan, Sumit Bagri, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Aliaksei Severyn, Jonathan Lai, Kathy Wu, Heng-Tre Cheng, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Mark Geller, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Neje Trdin, Andrei Sozan- schi, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Abhimanyu Goyal, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thortoo,, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Sabaer Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Tao Zhu, Ycongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang. Dustin Tran, Yeqing Li, Nir Levine, Ariel Stolovich, Norbert Kalb, Rebeca Cantamaria--Frranne Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Balaji Lak- shminarayanan, Charlie Deck, Shyam Upadhyay, Hyo Lee, Mike Dusenberry, Zonglin Li, Xuezhi
17

---

Published as a conference paper at ICLR 2024
Wang, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice Olivier Bachem, Summer Yue, Sho Arora, Eric Malmi, Daniil Mirylenka, Qijun Tan, Christy Koh, Soheil Hassas Yeganeh, Siim Pöder, Steven Zheng. Francesco Pongetti, Mukarram Tariq. Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Ragha Kotikalapudi, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszez, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Chenkai Kuang. Vinod Koverkathu, Christopher A. Choquette-Choo, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Pei Sun, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Ishita Dasgupta, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Yuan Liu, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Jimezak-Pluciise Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Ivo Penchev, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Vic- tor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Adam Kurzrok, Lynette Webb, Sahil Dua, Dong Li, Preethi Lahoti, Surya Bhupatiraju. Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mi- hir Sanjay Kale, Jinliang Wei, Ruoxin Sang. Brennan Saeta. Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma. Taylan Bilal, Evgenii Eltyshev, Daniel Balle, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang. Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Adams Yu, Christof Angermueller, Xiaowei Li, Weiren Wang, Julia Wiesinger, Em- manouil Koukoumidis, Yuan Tian, Anand Iyer. Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fer- nando, Kevin Brooks, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sehastian Ruder, Morgan Redshaw, Jinhyuk Lee, Komal Jalan, Dinghua Li, Ginger Perng Blake Hechtman, Parker Schuh, Milad Nasr, Mia Chen, Kieran Milan, Vladimir Mikulik, Trevor Strohman, Juliana Franco, Tim Green, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals. Gemini: A family of highly capable multimodal models, 2023.
David
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Kirill Trapeznikov and Venkatesh Saligrama. Supervised sequential classification under budget constraints. In Carlos M. Carvalho and Pradeep Ravikumar (eds.), Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, volume 31 of Proceedings of Machine Learning Rexearch, pp. 581-589, Scottsdale, Arizona, USA, 29 Apr-01 May 2013. PMLR.
Jordy Van Landeghem, Matthew Blaschko, Bertrand Anckaert, and Marie-Francine Moens. Bench- marking scalable predictive uncertainty in text classification. IEEE Access, 10:43703-43737, 2022. doi: 10.
Neeraj Varshney and Chitta Baral. Model cascading: Towards jointly improving efficiency and accuracy of nlp systems. arXiv preprint arXiv:2210.05552, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In L. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL https: https::/procceeding..ntt cwturpp..ccppppe.. wew..pps.cc/pppp"aiieee/ppeeee filles/papet//ozr/fiil. 3f5ee243547deee www.://deeeeeoooocccccccc... wwwe53c1cca8s5aaaa.a. Paper .df.
18

---

Published as a conference paper at ICLR 2024
P. Viola and M. Jones. Rapid object detection using boosted cascade of simple features. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume I, pp. I-1, 2001. doi: 10.1109/CVPR.2001999055
Marija Sakota, Maxime Peyrard, and Robert West. Fly-swat or cannon? cost-effective language model choice via meta-modeling, 2023.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: Stickier Benchmark for Seneral-Purposs Lan- guage Understanding Systems. Curran Associates Inc., Red Hook, NY, USA, 2019a.
Shuo Wang. Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun. Improving back Dack-translation with uncertainty-based confidence estimation. In Proceedings of the 2019 Conference on Empir- ical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 791-802, Hong Kong, China, Novem- ber 2019b. Association for Computational Linguistics. doi: 10.18653/w1/D19-1007. URL
1073.
Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M. Kitani, Yair Movshovizz-Attaa and Elad Eban. Wisdom of committees: An overlooked approach to faster and more accurate models. In International Conference Learning Representations, 2022. URL https://openreview.
het/forum? id=Nootteobso-
Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E. Gonzalez. IDK cascades: Fast deep learning by learning not to overthink. In Amir Globerson and Ricardo Silva (eds.), Proceedings of the Thirty-Four Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 580-590. AUAI Press, 2018
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL htw.:eeeeeetoooommdidsss-... http::////ooorrrreeeeerrr/rrrrrd
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. Interna- tional Conference Learning Representations, 2022. URL https::/openreviee.nn net.forrumee//om
id=gEZrGCozdqR.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 1112- 11221122. Association for Computational Linguistics, 2018. URL http:// http:://alwebbbor//annaat www.:oorg/aattoooooooooooooooooo /N-0 1101.
Tim Z. Xiao, Aidan Gomez, and Yarin Gal. Wat zei je? detecting -ittof-distribution translations with variational transformers, 2021. URL httpt: https://ppeerreviien www.:/eee///rrue iid=nuwy7R kemM.
Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thoughts representations for cost-efficient reasoning, 2023.
Yao Zhao, Mikhail Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter Liu. Calibrating sequence likelihood improves conditional language generation. In The Eleventh In- ternational Conference on Learning Representations, 2023. URL https://openreviev. net/ forum?id=oca o00mmdddaa.00
Zihao Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. Calibrate before use: Im- proving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Pro- ceedings of Machine Learning Research, pp. 12697-12706. PMLR, 18-24 Jul 2021. URL https://proceedingg mlr. pttp. 2ta.s//c...///o
Qihuang Zhong, Liang Ding, Li Shen, Juhua Liu, Bo Du, and Dacheng Tao. Revisiting knowledge distillation for autoregressive language models, arXiv preprint arXiv:2402. 1890, 2024.
19

---

Published as a conference paper at ICLR 2024
Language Model Cascades: Token-level uncertainty and beyond Appendix
A DISCUSSION AND FUTURE WORK
We have seen that there is value in going beyond simple sequence level uncertainty and considering finer measures of token level uncertainty as deferral rules for cascades. Moreover, we have seen that intermediate embeddings from the larger model can further boost performance.
This work raises a number of interesting directions for future work. First, we have used a 5-layer MLP to train post-hoc deferral rules using the quantiles of the probability distribution. It would be interesting to see how well a 1-layer Transformer works for this task, to potentially better exploit the sequential nature of the token probability vector. We have focused on FLAN-TS instruction tuned Encoder- -neoder-Decoder models in this work. We believe that the insights and methods should generalize Decoder-only architectures. would be interesting to evaluate the proposed approaches for such architectures. Moreover, it has been observed that models with RLHF finetuning become uncali- brated (Figure 8 in OpenAl (2023)). would be interesting to see how various finetuning steps affect the findings in this work and what consequences they have for designing efficient cascades.
Multiple works have considered alternative notions uncertainty using the generative abilities of LMs, for example, reprompting the model to ask how confident it is about the answer or output an additional confidence score as part of the outputs (Kadavath et al., 2022). It would be interesting to evaluate how well these measures work for cascades. See Appendix C for details.
B LIMITATIONS
One potential limitation of post-hoc deferral rule training is that it may not work well if there is some distribution shift between the probability values in the train and test sets. It also may not work in settings where we have limited training data, especially where we use high dimensional features like embeddings.
C UNCERTAINTY QUANTIFICATION FOR LMS
There has been large body of work uncertainty quantification for LMs. We discuss some of the approaches below. They can be broadly divided into the following categories.
Consensus-based One limitation of Equation 3 is that it considers single output sequence, e.g., the most likely one. However, as there are many sequences that have similar meaning, it is intuitively more reliable consider drawing multiple sequences from p(: .²). One may then assess the consensus in resulting predictions to measure confidence (Wang et al., 2023; Xiao et al., 2021; Chen al., 2023c); this can help distinguish between models that are locally diffused versus peaked around a candidate sequence. Recently, Yue et al. (2023) explored the use of answer consistency to construct effective cascades.
Deep ensembles and dropout. One approach to measure confidence is to create an ensemble of different models, and suitably aggregate their predictions (e.g., based on disagreement) (Van Lan- deghem et al., 2022; Wang et al., 2019b; Gleave & Irving, 2022). However, these uncertainty esti- mation procedures involve additional computation (e.g., multiple inferences with a single model in dropout- based approaches and single inference with multiple models in ensemble-based approaches) compared to simply using softmax probability outputs from a single network. Such approaches are less appealing for use in cascades, where the primary goal is to improve efficiency.
Post-hoc calibration/Answer- and length-bias calibration. For tasks involving question- answering with multiple choices (e.g., (A), (B), (C)), several works have demonstrated that LMs can have prior biases to certain answers, which can be identified and corrected (Zhao et al., 2021; Holtzman et al., 2021; Kumar, 2022; Murray & Chiang, 2018; Mielke et al., 2022; Jiang et al., 2021).
20

---

Published as a conference paper at ICLR 2024
Semantic entropy. Another key challenge in measuring the uncertainty for natural language out- puts is that there are lot of semantic equivalent sentences and hence, the probability can be divided among multiple outputs which mean the exact same thing. Kuhn et al. (2023a) proposes to mitigate this problem by sampling multiple outputs and then clustering semantically equivalent outputs to- gether and combining their probability together. It would be interesting understand how well this method can work for our setting.
Generative uncertainty. The above has largely focussed on generalizing the standard maximum predictive probability (Equation 2) from classification to the LM setting. While this by itself leads to rich array of possible confidence measures, LMs intriguingly offer wholly new possible means of assessing confidence: one may directly probe the model to obtain how confident it is on the proposed answer (Kadavath et al., 2022). Kadavath et al. (2022) discuss various ways of the input prompt format for this confidence probe. They also discuss the training of an additional head of the model to predict the model confidence but again, not clear how this compares with the standard probabil- ity output by the model without any additional finetuning. However, (Shrivastava et al., 2023) found that the confidence measures generated linguistically give worse estimates of uncertainty compared to the classical softmax- batsma--basdd measures even when these softmax-base probabilities come from different and weaker model. Moreover, they observed that two sources of uncertainty are comple- mentary and it can be beneficial combine them.
Other work. Zhao et al. (2023) proposed sequence-level calibration as a means to improve the gen- erative ability of LMs; such calibration could also be useful in improving methods such Chow-Sum. Kuhn et al. (2023b) proposed to ask the model to detect ambiguous questions which the model is likely get wrong and answer clarifying questions if the question is indeed ambiguous. Hendy al. (2023) proposed to use an exogeneous quality estimation model to decide how to route between two models. Šakota et al. (2023) similarly proposed to train a meta-model to pick an appropriate model from family. Fadeeva al. (2023) did a comprehensive experimental analysis of various uncer- tainty methods.
D EXPERIMENTAL DETAILS
We use a ing post-hoc deferral rules with Quantiles with batch normalization layers. We train for 200 epochs and early stop depending on the AUC of predicting the target label on the validation set for classi- fication tasks and validation regression loss for the regression tasks. Since the embeddings are high dimensional, we use MLP with 2 layers and 8 hidden units to prevent overfitting. We train for the same number of epochs with early stopping. We use ADAM optimizer with a learning rate of 10-5 We use batch size of 16 for training. We train over 5 random runs and show mean and standard deviations over the runs.
(MLP) with 5 layers and 32 hidden units per dimension for train-
We limit the number of examples per dataset per split to be 10k. Since many datasets do not have all the three splits available. We split the training set into train (0.8 fraction) and validation sets (0.2 fraction). And, use the validation or test split for reporting numbers. The details of all the datasets are included in Table
21

---

Published as a conference paper at ICLR 2024
Train
Test
target tokens
Dataset
Category
Shots
Split
Size
Split
Size
Mean
Std
BoolQ
Classification
Train
9417
Val
3270
IMDb
Classification
Train
9991
Test
9981
MNLI
Classification
Train
10000
Test
9832
1.9
1.4
ANLI-RI
Classification
Train
9999
Test
1000
1.9
1.3
ANLI-R2
Classification
Train
10000
Test
1000
1.9
1.4
ANLI-R3
Classification
Train
10000
Test
1200
1.9
1.3
PIQA
Classification
Train
10000
Val
1838
24.4
25.3
Winogrande
Classification
Train
9998
Val
1267
1.3
0.7
TriviaQA
Generation
Train
10000
Val
10000
4.0
3.3
NaturalQA
Generation
Train
10000
Val
3610
3.5
2.1
SQuaD
Generation
Train
10000
Val
9997
4.6
4.1
Lambada
Generation
Train
4857
Test
5152
2.0
1.1
TyDiQA-FI
Generation
2
Train
3659
Val
782
14.1
27.4
TyDiQA-SW
Generation
Train
3611
Val
499
12.2
24.3
TyDiQA-ID
Generation
Train
3655
Val
565
10.3
21.0
WMT FR EN
Translation
Train 9976
Test
3003
35.3
40.7
WMT EN-FR
Translation
Train 9974
Test
3003
54.3
48.5
WMT DE FRR
Translation
Train 9999
1497
33.6
23.2
Table 2: The table shows the number of shots used for each dataset, the splits used for training and testing, the number of examples each split and the average number of target tokens in the train split of each of the datasets.
22

---

Published as a conference paper at ICLR 2024
E ADDITIONAL EXPERIMENTAL RESULTS
E. ABLATIONS WITH PROBABILITY VECTORS AS FEATURES
In this section, we show that simply using the sequence of probability values with padded zeros to account for variable length sequences does not work as well as using quantiles. This intuitively makes sense, as the different token uncertainties are not aligned with each other which makes learn- ing hard for the MLP. We further show that using sorted probability values does not work well as using the quantiles: quantiles help better align the different uncertainty scores for variable length sequences.
DeatQ: Case - Large
IMD:: Base Large
MELLE Dase -4 Large
0..66
0.75
0.70
0.82
0.55
Phetilloc- Quanti
Partillos-Pot Prob
of examples defemed
of champies duforrod
of exammlls deforie
ANU R1: Base Large
ANL R2; Base Large
AMU R3: Base - Large
0.330
0.45
0.64
0.535
0.42
0.900
1.40
0.475
0.43
0.42
60
100
of examples detereed
examplee deferred
of examples deterred
PIGA: Base Large
Winggrender Saee - Lange
Lambbrder Bonu Large
0.750
0.70
0.72
0.735
Sccunccy
0.70
0.65
0.700
0..6
0.75
0.00
0.10
20
50
90
100
20
43
10
100
40
80
100
tosmmple defeneed
clamppes deferred
of exammple defenied
SQuaD: Large
Trvls04: Large
NaturalAA Basc Large
0.13
Accurasy
2.30
0.26
0.00
0.55
0.00
0.39
20
60
90
100
20
43
80
100
60
60
100
examples deterred
cxamples deferred
examples detereed
TADIGA-TI Large
Tyn104-sw Large
Tyniok--D:: Large
0.375
0.10
0.26
0.350
0..1
0.24
0.325
0.30
0.300
0.15
0.22
0.255
0.0
20
50
90
100
20
42
60
60
100
25
40
80
80
100
WMI DE-FRR Large
EN--FR
WMT FR-Gh) Base
BLEURT score
0.52
0.38
0.58
20
60
100
20
60
60
100
examples deteredd
Figure 7: Deferral curves different datasets for FLAN-T5 Base + Large cascade. Post-Hoc- Quantile denotes the post-hoc deferral rule which includes quantile features. Post-Hoc-Prok. denotes the post-hoe hoe deferral rule which includes the entire probability vector padded with zeros. Post-Hoc- SortedProb denotes the post-hoc deferral rule which includes the sorted probability vector padded with zeros.
23

---

Published as a conference paper at ICLR 2024
Dataset
Small
Base
Large
XI.
XXL
WMT DE FR
-0.35
-0.28
-0.22
-0.30
-0.24
WMT EN
FR
-0.26
-0.58
-0.26
-0.33
0.29
WMT
-0.16
-0.20
-0.11
-0.44
-.10
Table 3: Correlation of BLEURT scores with prediction lengths across FLAN-TS model sizes. We observe that there is a non-zero correlation between output lengths and BLEURT scores. Hence, there is some signal in using output length as a criterion for deferral.
WMT FR + EN: Base - Large
0.5
Correlation
0.61
AUC-DE
0.0
0.60
-0.5
Correlation
AUC-DF
0.59
0.00
0.25
0.50
0.75
1.00
Quantile value
Figure 8: Different quantile scores have different correlations with prediction lengths and different AUC-DF for the deferral curves. For this particular dataset, the best quantile values are 0.0 and 0.8.
E.2 OUTPUT LENGTH AND BLEURT CORRELATION
Table 3 shows there is significant (negative) correlation of BLEURT scores with prediction lengths across FLAN T5 model sizes. This verifies there is some signal in using output length as a criterion for deferral.
24

---

Published as a conference paper at ICLR 2024
E.3 ADDITIONAL RESULTS FOR VARIOUS MODELS AND DATASETS
25

---

Published as a conference paper at ICLR 2024
Booky: Small Base
IMDb: Small Base
MNLI Small Base
0.940
0.55
Accuracy
Chow Sum
Accuracy
0.935
Accuracy
0.50
0.70
Chow Averege
Quantile Dost-HocEEmbed-
0.930
0.45
Chaw-Quantlle
0.65
Post Hoc Embed
1+2
0.925
0.40
100
100
40
100
% examples deferred
% of examples deferred
% of examples deferred
ANLI-R1: Small Base
ANL--2: Small - Base
ANLI-R3: Small - Base
0.40
0.450
0.42
Accuracy
0.425
Accuracy
0.40
AccurBcy
0.38
0.400
0.38
0.36
0.36
0.350
0.34
20
40
60
8D
100
20
40
60
80
100
20
40
60
80
100
% examoles deferred
% of examples deferred
% of examples deferred
PIOA: Small Base
Winogrande: Small - Base
Lambada: Small Base
0.61
0.625
0.56
0.62
0.600
Arcuracy
Accuracy
Accurazy
0.60
0.575
0.54
0.58
0.550
0.56
0.525
0.54
100
100
%
deterred
examples deterred
examples deterred
SQuaD: Small Base
TriviaOA: Small Base
NaturalQA: Small Base
0.39
0.04
Accuracy
0.37
Accuracy
Recuracy
0.0.3
0.03
0.36
0.030
0.025
0.020
20
40
60
100
20
40
00
80
100
20
40
60
80
100
% examples deferred
96 excmpless deferred
%% of examplers deferred
TyDiQA [l: Smalll - Base
TyDiQA SW: Small Dave
TyDiQA ID: Small - Base
0.188
0,200
Accuracy
0.20
Accuracy
0.16
Accuracy
0.150
0.15
0.14
0.125
0.12
0.10
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
% examples deferred
% of examples deferred
% of examples deferred
WMI DD-+HH: Small -a Hase
WM1 EN-+ER: Small Hase
WMI HR-EN: Small - Base
0.35
0.57
BLEURT score
BLEUNI score
BLEURT score
0.375
0.56
0.30
0.350
0.55
0.25
0.300
0.54
0.20
100
%
examples deferred
% examples deferred
examples deferred
Figure 9: Cascade curves for all datasets and deferral methods for Small - Base model.
26

---

Published as a conference paper at ICLR 2024
BookQ: Base Large
IMDb: Base Large
MNLI: Base Large
0.86
0.965
0.75
0.84
0.960
Accuracy
Accuracy
Accuracy
0.70
Chow Sum
0.82
Chow Averege
0.955
Quantile Post-Hoc-Embedd
0.950
0.65
Chaw-Quantile
0.945
0.50
0.78
Post Hoc Embad
1+2
0.940
100
100
100
% examples deferred
% examples deferred
% of examples deferred
ANU-R1: Base Large
ANLI-R2: Base Large
ANU-R3: Base Large
0.60
0.48
0.46
0.44
Accuracy
0.55
Accuracy
03.46
AccurBCy
0.42
0.38
0.45
0.42
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
% of examples deferred
% of examples deferred
% of examples deferred
PIOA: Base Large
Winogrande: Base Large
Lambada: Base - Large
0.750
0.72
Accuracy
0.725
Accuracy
Accuracy
0.70
0.700
0.68
0.675
0.60
0.66
0.650
0.64
100
100
%
deterred Large
examples deterred
examples deterred
SQuaD:
TriviaQA: Base Large
NaturalQA: Base Large
0.12
Accuracy
0.41
Accuracy
0.10
Accuracy
0.06
0.08
0.05
0.06
0.39
0.04
20
40
60
100
20
40
00
83
100
20
40
60
80
100
% examples deferred
96 of examples deferred
% of exampler deferred
TyDiQA Fl: Base - Large
TyDiQA SW: Base - Large
TyDiQA ID: Dase - Large
0.28
0.18
0.375
0.17
0.26
Accuracy
0.350
Accuracy
Accuracy
0.325
0.10
0.300
0.15
0.22
0.275
0.14
0.20
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
% examples deferred
% of examples deferred
%
examples deferred
WM1 DD-+HH: Base Large
WM1 EN R: Base Large
WMI HR-EN: Hase Large
0.42
0.64
BLEURT score
BLEURT score
BLEURT score
0.62
0.40
0.44
0.60
0.43
0.42
0.58
100
% examples deferred
%
examples deferred
examples deferred
Figure 10: Cascade curves for all datasets and deferral methods for Base - Large model.
27

---

Published as a conference paper at ICLR 2024
Boalq: Large
IMDb: Large
MNLI: Large XL
0.90
0.967
0.85
Accuracy
0.89
Accuracy
0.996
Accuracy
0.88
Pust-Hoc-Eo Quantile
Chow-Average
0.965
0.80
Aust-Hoc-Emriee Emnbed-1
0.87
Chow Quantile
0.75
Hoc- Embed-1+2
0.964
0.86
%
examples
examples
examples deferred
ANU-R1: Large
ANLI-R2: Large
ANLI-R3: Large
0.52
0.50
Accuracy
Accuracy
Accuracy
0.65
0.48
0.50
0.60
0.46
0.55
0.46
0.44
examples
16 examples deferred
exampies
PIQA: Large
Winogrande: Large
Lambada: Large
0.82
0.80
Accurscy
Accuracy
Accuracy
0.76
0.80
0.76
0.74
0.72
% examples defemed
%6 examples deferred
% examples deferred
SQuaD: Large
TriviaQA: Large
NatursiQA: Large
0.12
0.8
Accuracy
Accuracy
0.2000
Accuracy
0.10
0.175
0.09
0.5
0.08
0.125
0.07
0.4
100
10
100
20
10
60
80
100
% examples deferred
% examples deferred
% examples deferred
TYDiQA-FI: Large
TyD QA-SW: Large XL
TYDIQA-ID: Large -
0.50
0.475
0.1.8
Accuracy
Accuracy
Accuracy
0.45
9.150
0.16
0.425
0.35
0.400
0.30
100
20
40
50
B0
100
20
40
00
80
100
% examples deferred
16
examples deferred
% of examples deferred
WMT DE-FR; Large XL
WMT EN-FR; Large - XL
WMT FR-EN: Large - XL
0.45
0.655
BLEURT Score
0.44
BLEURT score
0.650
0.47
0.13
BLEURT
0.645
0.46
0.610
0.45
100
20
40
30
100
20
40
60
80
100
% of examples defermed
56 of examples deferred
% of examples deferred
Figure 11: Cascade curves for all datasets and deferral methods for Large + XL model.
28

---

Published as a conference paper at ICLR 2024
E.4
DIFFERENT VERSION OF LENGTH PLOT
Frccionn points deferred
Fraction deferre
Fraction defered
1.0
1.0
1.0
Chow
Chow Sum
Chow-Sum
0.8
Oracle
0.6
0.6
0.4
0.4
0.2
0.2
50
100
150
200
250
50
100
150
200
250
50
100 150
200
250
Prediction Length
Prediction Length
Prediction Length
(a)
(b)
(c)
Figure 12: CDF plot for the WMT - EN dataset for the FLAN T5 Base Large model for different deferral rates. Each plot considers the points which are deferred by each method at the mentioned deferral rate. Each point (z, 3) represents that y fraction points at the desired deferral rate have prediction lengths less than z. We can that Oracle has larger fractions of points deferred with smaller prediction length. Hence, Chow-Sum prefers to defer longer predictions as compared to the Oracle method.
Pracien points deferred
Fracto points deferred
Deferral
deferred
Deferral
1.0
0.5
Phow-Sum
Chow-Sram
Chow-Sum
0.8
0.G
0.4
0.6
0.3
0.4
0.2
0.2
0.2
0.0
50
100
200
250
100 150
200
250
100
150
200
250
Prediction Length
Prediction Length
Prediction Length
(a)
(b)
(c)
Figure 13: CDF plot for the WMT FR - EN dataset for the FLAN T5 Base Large model. Each plot considers the points which are deferred by each method at the mentioned deferral rate. We can see that Oracle has larger fractions of points deferred with smaller prediction lengths. Hence, Chow-Sum prefers to defer longer predictions as compared to the Oracle method.
E.5
CASCADE CURVES WITH RELATIVE LATENCY
MNLL: Bose Lange
TrKOOA: Base Large
WNT DE =RR, B302 Lange
0.70
0.10
Rost Hoc Quantil
Chow-Sum
0.08
Chove-uuniike
Chow Average
0.60
0.08
Chow Quantile 0
Plow-Quintee 0.4
1.2
1.0
12
Helative Latency
Reletive Latency
Reletive Latency
Figure 14: Deferral curves on MNLI, TriviaQA, and WMT DE + FR for a FLAN-T5 Base + Large cascade.
29

---

Published as a conference paper at ICLR 2024
E.6 RESULTS WITH ENCODER AND DECODER EMBEDDINGS
Dost-Hoe-Embed-Dec the method Post-Hoc-Embed-I method from main draft which includes decoder embeddings from model I. Post-Hlo-EEmbed-Enn-DDD additionally includes encoder embeddings from model 1.
Similarly, Post-Hoc- Host-Ho--Embee- De the method Post-Hoc-Embed-I4t 1+2 method from main draft which includes decoder embeddings from model 1 and intermediate decoder embeddings from model I, Doss-Hoo-Emmbe-EEn--DDee- 2.
additionally includes encoder embeddings from model
We do observe any additional gains from including encoder embeddings in these methods.
MNLI: Bade
Large
TRvISQA: Base Large
WMT DE-FR; Base Large
0.75
EUEURT score
0.70
0.10
Poabi loe- Iibeee Dec------11
0.65
0.08
Poet-Hioe Embed Enc-DDe:
Poeb-lloe Embed Dec 1+2
Poat-llc--Embee Enc-----------ee Dec-1e2
0.60
0.06
100
1.00
96 of examples deferred
of examples dafered rod
of examples deferred
Figure 15: Deferral curves on MNLI, TriviaQA, and WMT DE + FR for a FLAN-T5 Base + Large cascade with additional encoder embeddings.
30

---

Published as a conference paper at ICLR 2024
E.7
RESULTS WITH BEAM SEARCH DECODING
TriviaQA: Base Large
WMT DE- Base - Large
0.14
0.35
BLEURT score
Accuracy
0.12
Chow-Sum
0.34
Chow-Average
Chow Quantile-O
Chow Quantile-0.4
0.08
0.32
Ohow-Quantile-0.8
20
40
60
80
100
20
60
100
examples deferred
examples deferred
Figure 16: Deferral curves TriviaQA, and WMT DE FR for FLAN-T5 Base - Large cascade with beam search decoding.
TriviaQA: Base Large
WMT DE-FR: Base Large
0.14
0.36
BLEURT score
Accuracy
0.12
Chow-Quantile-Bst
0.34
Post- Huc-Quantile
0.10
Chow-Sum
Chow-Average
0.32
0.06
20
60
100
20
100
% of examples deferred
% of examples deferred
Figure 17: Deferral curves on TriviaQA, and WMT DE + FR for FLAN-T5 Base + Large cascade with beam search decoding.
# Outpput tokents Base model
WMT FR-EN: Base Large
WMT FR-EN: Base Large
250
200
BLEURT score
0.64
Choe-Average
150
Chov-Aueragg Quantiic
0.62
Choe-Quante Quantile 0
100
Chowi- O. Oracle
Mowe-QuantieeBB. 0.8
0.60
Cracle
0.6
0.5
1.0
20
40
100
Score quantile
of examples deferred
(a)
(b)
Relation between deferral rules and output length (number of tokens) for WMT FR + EN dataset and FLAN-TS Base Model with decoding using beam search. Chow-Sum tends to defer longer prompts: the prompts with lowest scores have notably higher length than those with higher scores. Interestingly, Chow-Average over-corrects this bias: it tends to overly defer prompts with lower length. Oracle refers to deferring using the difference of BLEURT scores of two models. Oracle also tends to defer longer outputs, but the preference is moderate compared to Chow-Sum. Right: Corresponding deferral curves.
Figure 18: Left:
31

---

Published as a conference paper at ICLR 2024
WMT DE-FR: Base
WMT FR-EN: Base
0.4
Fraction of points
0.4
Fraction of points
0.2
0.2
Beam Search
Greedy
0.1
0.1
0.0
0.0
50
100
150
200
250
50
100
150
200
250
Output Lengths
Output Lengths
Figure 19: Comparison of Output lengths on WMT DE FR and WMT FR EN for FLAN-TS Base model with beam search decoding vs. greedy decoding. We observe that greedy decoding does seem have overall shorter output lengths as compared greedy decoding but still puts considerable weight on longer outputs.
32

---

Published as a conference paper at ICLR 2024
E.8 PERFORMANCE OF METHODS IN PREDICTING GOLDEN DEFERRAL LABEL
Dataset
Chow-Sur
Chow-Average
Chos-Quan L Pos (-too- Quannille
Post-Itoc- Embed-s
Dost-thoc-Enbed-122
MNLI
0.46
0.53
0.73
0.86
0.92
0,95
TriviaQA
WMT DE - FR
0.51
0.56
0.62
0.62
0.63
0.62
Table 4: The table shows the AUC-ROC score for predicting the golden binary deferring labels, which is I if model (FLAN-TS Large) is better than model (FLAN-T5 Base), and 0 otherwise. We see that Chow-Sum Sum and Chow-Average tend to be poorly predictive of this golden label. The recommended Post-Hoc deferral approaches yield significant gains in the AUC-ROC.
33

---

Published as a conference paper at ICLR 2024
F
HOw IS ( COMPUTED?
Here, we give an example of how ((x) is computed for each of the Post-Hoc methods.
For concreteness, suppose that the example outputs prediction which has 5 output tokens. Let the output log probability associated with each output token be p:( where i E {0, 1, 2, 3,
4]. Then, the Chow-Sum score 8sum (z) E, pi( and the Chow-Average score is Savg (æ)
> Pi (a:).
Let the various quantiles be Squant (x, a)) where E [o, 0.01, 0.02, 0..,, 0.3. 1.0]. For concreteness, take 20 quantiles.
Let the decoder embedding from model averaged over all output tokens be denoted as finaade,.fnalavg...l. (ax) R Let the decoder embedding from model 2's first layer for the first E 22
output token be denoted as Embaac,.in,,fstt.uu
Post-Hoc- Quantile
Hoa--uan-Quee For the method Post-Hoc- Post-tie-..a,tlle
c a:)
[Ssum (2)), Savg (x), Squant (x, 0;)]
where we concatenate the quantities, Chow-Sum and Chow- scores which makes this feature 22 dimensional.
Post-Hoc-Embed- For the method 1ost-Hoe-Embed-1,
+ (Φ)) [ssum (æ), (2),, Squant (x, ai), Embaae, avg.l (æ)]
where we concatenate the quantities, Chow-Sum, Chow-Average scores and model I's decoder em- beddings which makes this feature 22 + dimensional.
Dost-Hoc-Embed-e -1+2 For the method Dost-Hoc-Embed-lax
[sum (x). Savg( (z), Squant (x, a;), Baml.ae.. (x), Embdee int, fist,, (x)]
where concatenate the quantities, Chow-Sum, Chow-Average scores, model I's decoder beddings and model 2's intermediate decoder embeddings which makes this feature 22 + n + n2 dimensional.
34


Reviews:
[{"review_id": "FQLUZeQU8f", "reviewer": "Reviewer_7G2G", "summary": "This paper is about learning deferral rules for LM cascades -- essentially, how can you predict when to rely on a small model's output and when should you fall back to a large, more expensive model? They propose a connection to the classical problem of classification with rejection, in which a model can choose to reject classifying an instance. The optimal strategy in this case is to reject whenever the model's confidence fails to pass a threshold derived from the cost of rejection. It is easy to see how LM cascades fit into this framework: deferring to the larger model is equivalent to rejecting the output of the smaller model.\n\nHowever, it is not trivial to generalize rejection from classification to generation. In classification the model predicts only one label along with an easy-to-interpret probability score (e.g. from softmax), whereas generation entails producing variable-length *sequences* of tokens. Although each of these tokens has an associated per-token probability, aggregating them is a challenge: simply summing the logprobs causes short sequences to be rejected (this is mathematically necessary, as it is equivalent to multiplying numbers less than one), while averaging them causes *long* sequences to be rejected (this observation is interesting, and surprising to me).\n\nNoting the weaknesses of these two baselines, the authors' main contribution is to introduce alternative techniques to score the model output. The main technique, which they call Chow-Quantile, is based on sorting the per-token logprobs for an instance and then picking the value at some alpha-quantile of this, where alpha is a hyperparameter. For example, picking the 0-quantile is equivalent to scoring sequences based on the *lowest* per-token prob. They also propose various post-hoc techniques that allow classifiers to be trained on top of these quantiles.\n\nThey exhibit experiments applying their various deferral techniques to a variety of tasks, including MT and QA (and also surprisingly MNLI, which seems inapplicable because it classification, not generation). These results seem to show an advantage for using their techniques over the baselines across many deferral levels.", "questions": "--Which WMT set was used? It needs to be identified and cited. \n\n--I don't understand how the MNLI experiments work. It is noted in Section 3.4 that MNLI is a multi-class classification problem, but the techniques proposed in this paper are not for classification.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper presents a simple approach that seems to be very effective. The connection to rejection in classifiers is intuitive but had not occurred to. The paper is easy to follow. The clarity of presentation convinces me that it would be easy for me to try the approach myself, either for its own utility or as a replication study. I admire the accessibility of this work.", "weaknesses": "Although the paper as a whole is very clear, there are places where the experiments lack specifics (see the questions section). Additionally, there are places where the experimental set-up seems to be suboptimal: greedy decoding was used, but this is more prone to hallucination than beam search. Some of the conclusions of the experiments might be artifacts of these hallucinations. I can think specifically of these two: \n\n1) there was a negative correlation between translation quality and sequence length; was this because many of the long sequences were hallucinations? \n\n2) the Chow-average model favored longer sequences, which there is no intuitive reason for. Could it be because hallucinations often get trapped in loops where the same short phrases get repeated with high probability? This would drive up the average.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WkGKctOlae", "reviewer": "Reviewer_WdbA", "summary": "The authors propose a novel method for uncertainty estimation in large language models (LLM). The method is based on LLM cascades, where a large model predicts difficult instances and a second small model predicts easy instances. In addition, the model uses the Chow-sum and Chow-average as a rule for assigning confidence (uncertainty) to token-level outputs from the LLM cascade.  The main contributions are: i) method for token-level uncertainty estimates, and ii) application of different natural language processing (NLP) tasks. The method shows that the confidence estimates based on the output probabilities from LLMs are biased.", "questions": "Please address the following questions during the rebuttal:\n\n- Please elaborate on the relation/difference of the Chow estimates with proper scoring rules (e.g. NLL, Brier score).\n- Could the proposed estimates be directly compared/evaluated to estimates from deep ensembles instead of a cascade or even jointly? (Lakshminarayanan, Balaji et al. “Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles.” Neural Information Processing Systems (2016).)\n- For the machine translation evaluation: \n\n Is the output generated by beam search? Please speculate on the effect of the hyperparameters used on the generation, do they have an effect on the output length?\n\n Please speculate for the use of the proposed method for robustness to OOD in MT. different domains can be used to evaluate a change in distribution, is the uncertainty estimate robust to such change?\n\n- Please elaborate on the use of output probabilities from the LLMs as uncertainty estimates compared to other methods (e.g. deep enembles, MC dropout)? (e.g. Baan, Joris et al. “Uncertainty in Natural Language Generation: From Theory to Applications.” ArXiv abs/2307.15703 (2023): n. pag.)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- A principled method for uncertainty estimation in LLM (i.e. FLAN-T5).\n- Clear description of background knowledge and related work needed to understand the proposed method.  \n- The authors perform a  comprehensive comparison of the proposed method with different NLP tasks.", "weaknesses": "- Motivation for the lack of comparison with other uncertainty estimation methods.\n- A possible extra contribution can be the use or discussion of the method for NLP tasks under out-of-distribution (OOD) or domain adaptation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q1m9Bbj8Cs", "reviewer": "Reviewer_MA2R", "summary": "The paper focuses on the model cascade for generation tasks. It notices a crucial difference between cascading for classification tasks and cascading for generation task and points out that the natural extension of predicted class uncertainty to generative tasks, predicted sequence uncertainty, is biased by sequence length, leading to sub-optimal deferral decisions.\nIt then designs a deferral rule to obtain the score/confidence of the small LM and decides when to defer an input to the larger model.", "questions": "1.\tWhat is the performance of Post-Hoc-Embed-1?\n2.\tHow is the $\\Phi(x)$ computed in detail?\n3.\tWhat is the performance of the post-hoc-quantile when predicting the golden deferring label?\n4.\tIn figure2, figure3 and figure5, all figures use the deferring rate as the x-axis. I am curious about whether we could use the inference time cost as the x-axis.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tIt demonstrates that simple sequence-level LM confidence measures for deferral can lead to sub-optimal cost-quality tradeoffs due to length bias.\n2.\tThe paper proposes a simple yet effective method employing the quantile of the log-likelihood to design a deferral rule. \n3.\tThe Proposal of a post-hoc deferral rule trained on quantile features and the input embeddings of both the small LM and the large LM. The extensive experiments on FLAN-T5 verify the efficacy of the method.", "weaknesses": "1.\tCompared with simple averaging the log probability, the major advantage of quantile is that it reflects more about the overall log probability distribution of the sequence and is more robust to outliers. To highlight the motivation of the proposal, more evidence for the existence of the outlier is expected and the showcase in Figure 1 is not sufficient.\n2.\tIn Figure 2 and Figure 3, it seems that the best generation performance is obtained in the middle of the curve, other than the endpoint where all examples are deferred. Does this mean that sometimes smaller LM outperform larger ones?\n3.\tThe author claims that Chow-Sum is overly biased towards deferring longer predictions. However, from Figure3(a) we can obverse that when the output length (y-axis) is between 150 words to 250 words, the score of the oracle deferring strategy is smaller than the Chow-Sum, which says the opposite: the chow-sum isn’t biased towards deferring longer predictions when compared with the oracle.\n4.\tAs another line of work, speculative decoding also aims at the trade-off balance between efficiency and performance and I think the authors should discuss or compare the difference between these two lines of work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x4Ee7XnhLg", "reviewer": "Reviewer_Qrz5", "summary": "The authors aim to apply simple model cascades to structured output problems of varying length: try to first use a small model, then fall back to a larger model if it appears the small model is insufficiently confident. This task has been used to good effect on problems with simpler output spaces, such as multi-class prediction. In those settings, the log prob of the prediction can be used as a proxy for confidence; low probabilities from a small model trigger inference from a large model. However, in language generation tasks, the length can vary broadly, hence the log probabilities can vary broadly as well. The direct analog of log probabilities would be to sum the log probs of each prediction, but this has undesirable scale issues based on sequence length.\n\nThe authors propose both simple confidence estimate techniques (average, percentiles) and complex methods (learned functions of percentiles, embeddings) that lead to substantial improvements. In some settings the cascade performs better than either model alone, suggesting that the model is attaining some kind of ensemble effect.", "questions": "In the \"Intermediate embeddings\" approach, only decoder representations are used. However, wouldn't it potentially be useful to characterize aspects of the input? I could see that some inputs might be more reasonable for a smaller model; others might have complexities that are more suited to a larger model. Even input length could potentially be useful. Do you have empirical experimentation here?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors present an accessible introduction to cascades as well as the challenges of application to language generation tasks. The methods they propose are straightforward and easy to implement, and seem to work well.\n\nThe authors evaluate several different tasks, using both simple and complex models, and present reasonable gains.\n\nThe post-hoc methods provide some interesting insights.", "weaknesses": "The authors only work with a single base model: FLAN-T5. It's not clear how well these results generalize.\n\nThere are other methods of confidence estimation beyond logprobs (see, e.g. https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00598/117737/Calibrated-Interpretation-Confidence-Estimation-in) -- would like to see more analysis here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)