Confident Adaptive Language Modeling
Authors: Tal Schuster
Adam Fisch
Jai Gupta
Mostafa Dehghani
Dara Bahri
Vinh Q. Tran
Yi Tay
Donald Metzler
Venue: NeurIPS 2022 Accept (2022)
Keywords: adaptive compute
early exit
language model
transformer
nlp

Abstract: Recent advances in Transformer-based large language models (LLMs) have led to significant performance improvements across many tasks. These gains come with a drastic increase in the models' size, potentially leading to slow and costly use at inference time. In practice, however, the series of generations made by LLMs is composed of varying levels of difficulty. While certain predictions truly benefit from the models' full capacity, other continuations are more trivial and can be solved with reduced compute. In this work, we introduce Confident Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of compute per input and generation timestep. Early exit decoding involves several challenges that we address here, such as: (1) what confidence measure to use; (2) connecting sequence-level constraints to local per-token exit decisions; and (3) attending back to missing hidden representations due to early exits in previous tokens. Through theoretical analysis and empirical experiments on three diverse text generation tasks, we demonstrate the efficacy of our framework in reducing compute---potential speedup of up to $\times 3$---while provably maintaining high performance. 

Confident Adaptive Language Modeling
Tal Schuster
Adam Fisch
Jai Gupta'
Mostafa Dehghani'
Dara Bahri!
Vinh Q. Tran'
Yi Tay'
Donald Metzler'
Google Research
CSAIL, MIT
Abstract
Recent advances in Transformer-based large language models (LLMs) have led significant performance improvements across many tasks. These gains come with a drastic increase in the models' size, potentially leading to slow and costly use inference time. In practice, however, the series of generations made by LLMs is composed of varying levels of difficulty. While certain predictions truly benefit from the models' full capacity, other continuations are more trivial and can be solved with reduced compute. In this work, we introduce Confident Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of compute per input and generation timestep. Early exit decoding involves several challenges that we address here, such as: (1) what confidence measure to use; (2) connecting sequence-level constraints to local per-token exit decisions; and (3) attending back to missing hidden representations due to early exits in previous tokens. Through theoretical analysis and empirical experiments text generation tasks, we demonstrate the efficacy of our framework in reducing compute-speedup of up to x3-while provably maintaining high performance.
three diverse
1 Introduction
Recent advances in Large Language Models (LLMs) have led to breakthroughs in language under- standing and language generation across almost every widely-used Natural Language Processing (NLP) task considered in the field today [5: 15; 17; 20: 51; 52: 53; 75; 89; 73]. Autoregressive language modeling provides a flexible framework for solving complex tasks with a unified natural language input and output format, while also relaxing the need for large-scale task-specific data collection and training [67; 15; 17: 58: 80]. The large size of LLMs, however, results massive computational load that might be limiting for certain real-world applications (c.g., machine transla- tion) [9; 30; 42; 49; 59; 63; 71J. This is especially pronounced in the autoregressive decoding process where the full stack of Transformer layers is repeatedly computed for each output token [37; 40; 86).
While large models do better in general, the same amount of computation may not be required for every input to achieve similar performance (e.g., depending on if the input is easy or hard) [66]. Early exiting is a promising approach decreasing the computational cost of multilayered architectures such as those used in Transformer-based LLMs, where the number of layers used by the model is dynamically decided choose to generate a new token based off the representation at an intermediate layer instead of using the full model, and save computation a result. natural question that arises. however, is when is a good decision to exit carly, as opposed to wait? Naively choosing when to exit can be suboptimal in terms saving computation time, and also result in unpredictable degradations model performance, especially when predictions depend on each other, as in autoregressive language generation.
input-by-input basis [18; 23; 57; 60; 70]. In this setting, LLM can
Project leads. Correspondence to: talschuster@ google.com 'Code: https: httphtttttuhggggg/ ccm httpa//eeeeeeeaeeoco/eocc/eea/aa
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Guzrantend &.
hit
consistenzy
Early
Layer
Layer
Fxomple of "state propagation" (Tassed arrows show the hiddor- states prev steps used to compute layer at postton 6).
Layer
Yiu or gestt
User- defined & global constrants quide local extting docisions
Layer
Layer
Lawer
Sver
+
put
Tayer.cm Com
Laier 4
Lawer 2
Layer 3
Layer 3
Laper 3
Grant
Elliott
Laver
Laver Laver
&
2
Layer
Yearly
LeaSr
Layer
Laver
Layer
Layer
Figure 1: Illustration of CALM generation (see Figure 4 for the full example) with local per-token early exiting decisions that provably satisfy global user-defined constraints on the full sequence.
In this work, we analyze the early exiting paradigm for I. LLMs, and present principled method for increasing model efficiency while remaining confident in the quality of the resulting predictions. Specifically, we develop method for calibrating local, per-token, exit decisions such that global, sequence-level constraints- determined by lexical or semantic sequence-level metrics like ROUGE BLEURT score are provably maintained with arbitrarily high probability (e.g., 95%). This process, which we call Confident Adaptive Language Modeling (CALM), is illustrated in Figure 1.
Our approach leverages recent techniques in distribution risk control in order to create confident generations with strong statistical guarantees [2: 3; 10). Concretely, suppose we have been given calibration set S.al := {P:) E pp of independent and identically distributed (i.i.d)) prompts to our L.M (e-g., paragraphs to be summarized, sentences
translated, or questions to be answered via language modeling). Let Ptest be a new i..dd test prompt our LLM, where Yearly LLMearly (Pistt and Yiull := ELMMan(Ptes) are the adaptive and standard outputs of our LLM, respectively. In order to be satisfied with Yearly. might require it to be textually consistent with Yiul- Given any bounded text dissimilarity function D, we aim to calibrate the early-exiting LLM such that its predictions agree to tolerance 8 with the full model in expectation with high probability,
[D(Yaurly, Viuu)] s Seal 2 1 -
(1)
where the randomness is over draws of Seal, and E (0. 1). Eq. (1) has the significant advantage of being achievable using only unlabeled calibration data Seal (a quality that is critical for few- shot tasks, for example). Enforcing textual consistency with the original Yiun, however, may be unnecessarily strict for certain tasks, especially where multiple generations may be acceptable. As an alternative, given calibration set of prompts paired with a set of (potentially multiple) target references, Scal == {P,, Z=) E (P x 2"'y", and any bounded risk function R, we also consider an objective that enforces risk consistency by limiting the relative increase in risk of the predictions Yearly compared to Yuu., with respect to the set of test-time references Zuest i.e.,
R(Yearly, Zreat) R(Yruu, Zrest) S Scal 21-6.
Within the constraints of either Eq. (1) Eq. (2), the goal of our work is to find the most computation ally efficient Yearly, i.e., generations that exit as early as possible while still maintaining our desired performance guarantees. In order achieve this, it is necessary develop a reliable signal for how likely local, per-token early-exit decisions are to disrupt the global properties of the complete sequence. Here, first analyze how errors propagated in Transformer-basee LLMs, and then present an effective and efficient scoring mechanism for assigning "consistent early-exit" confidence scores after each layer used during the generation of new token. The decision to exit or based on these scores, and is carefully calibrated using Seal such that our performance bounds are provably satisfied.
Finally, empirically validate our method multiple, diverse NLP generation tasks, including text summarization, machine translation, and question answering. Our experiments demonstrate the potential of CALM in reducing the average complexity of the model and accelerating inference by about x3 while reliably controlling for high performance.
Contributions. In summary, our main contributions are as follows:
A framework (CALM) for reliably accelerating Transformer-basee LLM generations.
A systematic analysis of the token-wise early exit mechanism that motivates simple-but-effettvv class of confidence measures and threshold functions that are used as part of the CALM framework.
An empirical demonstration of CALM's efficiency gains on three diverse generation datasets.

---

2 Related Work
Improving inference-time efficiency of LLMSs has been an ongoing effort of the research community over the past several years [49; 72; 85), leveraging techniques such as knowledge distillation [6; 32; 36; 69; 69; 78; 56], floating point quantization [71: 65]. layer pruning [24], vector dropping [38], and others [41]. Another line of work involves conditional computation to train larger models that only use sparser subset of the full network during inference, for example by routing over mixture-of- experts [9; 22; 39; 91], recurring modules [18: 29; 35), or accessing external memory [82]. These models, however, still use the same amount of compute for all input examples.
Here, we focus on adaptive compute, a specific kind of conditional compute that aims to dynamically allocate different computational power per example, with the goal of reducing the overall complexity while maintaining high performance. This approach, often referred to as early-exiting [16; 25; 47; 74; 79; 87]. is complementary to many of the solutions above and can potentially be combined with them. Multiple early-exit techniques for encoder-only Transformers (e.g.. BERT [20]) have been recently proposed [8; 34; 43; 44; 45; 60; 68; 83; 90; 92]. Most of these methods rely on intrinsic confidence measures (e-g., based on the softmax distribution), while others try to predict the routing in advance [46; 70J. or train a small early-exit classifier [57; 84), as we also examine here. These measures can be calibrated to reliably guarantee consistency of the early prediction with the full model [57]. However, the techniques used for encoder-only classifiers are unsuitable for global consistency constraints with sequence of dependent predictions, which are inherent in the decoding process of autoregressive language models, which we address here.
Our work is also motivated by recent findings on the existence of saturation events in LMs, where the top-ranked prediction unchanged after some layer and is propagated upward. Geva et al. [28] exam- ined interactions of the hidden-state with feed-forward layers to predict these events. However, they only consider local single predictions and do not address the challenges involved with sequence gen- eration. Our early-exit LM architecture most closely relates to Elbayad et al. [23], who found token- level early-exit classifier to provide the best Setiiiincc--efformee Here, we introduce theoretically-greet grrounded calibration method for provably controlling the quality of the full sequence. By doing so, we provide reliable efficiency gains-deriving local early exiting decisions from the global desirable constraints. Moreover, we introduce several model improvements and empirical analyses, including (1) analyzing the primary sources of performance degradation, lead- ing to propose decaying threshold function for better tradeoff control without inflating the search space; (2) improving the early-exit classifier training; and (3) experimenting with two new tasks.
tradeoffs on machine translation.
Our calibration procedure for connecting global constraints to local decisions, relates to recent research around distribution-fres uncertainty quantification [1; 62; 77]. Several methods were developed in recent studies to expand and adjust the theoretical framework for obtaining practical efficiency gains on target applications [4; 7: 21: 26; 27; 48; 88). Here, we frame our consistency requirements around the Learn then Test (LTT) framework [3], and leverage the approximately monotonic behavior of our confidence measures and the nested structure of our problem, that by definition guarantees consistency with large enough threshold, to form tight and effective bounds.
3 Early Exiting for Adaptive Language Modeling
In the following, we describe and analyze the early-exiting Transformer LM. We begin with a brief recap of the Transformer architecture ($3.1) and early exiting ($3.2) for convenience, following previous work [23; 70; 76]. We then investigate the effects of early exiting on model performance, and identify primary sources of performance degradation and how to alleviate them ($3.3) which guide our architecture and training design ($3.4) and proposed per-token confidence measures ($3.5).
3.1 The Transformer architecture
We use the Transformer souuenceet---euencee model, based on the T5x implementation [55]. Here, we only review simplified details of the Transformer architecture relevant to early-exiting, and refer the reader to Vaswani et al. [76] for full details. At a high level, both encoder and decoder networks contain L stacked layers, where each layer is composed of multi-head self-attention sub-layer, followed by feedforward sub-layer, each with residual connections and layer normalization. The decoder network has additional multi-head attention sub-layer that attends to the encoder states.
3

---

Consider a prompt = (#1,...,, Zp), processed by the encoder to yield encoder states (ei....,
......,,pp.).... Ep),
and the current, partially generated response (y1,...,314 ye). When generating the next token 1t+ the decoder computes decoder state di for layer out of I. as:
h == Attention(d -d== dj == Attention e1:p); di (eedFoorarr)))))) (3)
Internal to each of the attention mechanisms, written as Attention Z1mm) for some imput æ and sequence of m states 21:ms I is first projected to query vector q := W Q E Rdim,, while is projected to a matrix of key-value vectors, K == W Z1m E Rmxddmm and V := Wy R.xddmm.. The output is then computed as o == softmax (gK' //dimz)) V.
Multi-head and normalization components omitted for brevity. Each layer uses different projec- tions W Wi and Wi (which are also unique for computing hi versus af).
Finally, after layer L, distribution over vocabulary tokens yt+1 E V is computed via softmax- normalized linear classifier W r. where p(y++1 df) = softmax( (W v.df).
3.2 Decoding with early exiting
Instead of always making a prediction based on the representation at the final layer, di, the key idea in early-exiting is to choose Yt+11 more quickly, if confident, by computing p( d;) softmax W;d)) for some intermediate layer i < L. Concretely, let cj E [0, 1J denote some local confidence score for layer while processing token t, where higher values indicate a higher propensity to exit early (we will propose effective instantiations of ct in $3.5). Let X E [0, 1 denote some local early-exiting threshold, where the model exits early if cf 2 X₁ or otherwise proceeds to compute the next representation. di+1. The (greedily chosen) prediction Mc+1 can then be written as:
arg p(ye+1 d)) arg ergmaxp(y--- d?)
if 2 X if 2 X
yt+1
(4)
arg max p(ye+1 | d()
otherwise.
Note that due to the self-attention mechanism the Transformer, computing the input hidden state hę for layer depends on di i.c., the output hidden states of the previous layer for all the tokens that have been generated so far. Therefore, if the model has early exited at some layer j < i 1 for token t, then d is not available. As an approximation, we set di de for all layers k >>> following Elbayad et al. [23], with the understanding that this will introduce some error. In the next section, in addition to other factors, we will analyze the impact of this copied state on performance.
3.3 The effects of early exiting on error propagation
We perform several controlled experiments to investigate the behavior and the potential of early- exiting during decoding. We use an 8-layer T5 encoder-decoder and the CNN/DM dataset for these experiments. See $5 for more details on this model and data.
3.3.1 State propagation
First, we control for the correctness of the predicted tokens to examine the effect of state copying ($3.2), and also measure an approximate upper bound for compute reduction. We use an oracle confidence measure that exits at the earliest layer that agrees with the top prediction (i.e., replacing the conditions in Eq. 4 with arg max p(y++1 | dd)) arg max p(y++11 | d.))). Hence, the only factor that can cause divergence in the generation is the state copying mechanism for skipped layers. The results of this experiment are highly encouraging. This oracle achieves an ROUGE-L score of 28,24, compared to 38.32 with the full model, while only using an average of 1.53 layers per token. We also try an oracle that always uses di:t-1 and it reaches 38.31 ROUGE-L. These results indicate that (1) the model is robust to state copying from lower layers, and (2) there is remarkable potential for saving compute- by up to x5.2 while preserving performance, given a good confidence measure.
We also experiment with copying the projected states KJ, Vá to skipped layers & > j. This version of the oracle results in significant drop in performance to 23.02 ROUGE-L. Overall, we conjecture
In autoregressive decoding, the ks, v. vectors cached to avoid repetitive compute for tokens /

---

06.0
272-
INTORISEAI brrperdanoo Dwe.aodtumhe
LIOWOSOEN PWTS
85.
3251
26
Docuding
(a) Sensitivity to local errors.
(b) Decaying threshold (A 0.9).
(c) Perf--efficiency tradcoffs.
Figure 2: Earlier noise in the decoding process has greater effect on the overall output (a), though in practice the affect of early exits is minor due to high performance of early layers. A decaying confidence threshold (b) allows finer control over the performance-efficiene tradeoff (c).
that the self-attention at layer for token f can safely use hidden-states d' for j < - 1 as key-values of tokens < t, long the projections W k/v of layer i are used. Notably, this projection can now be computed concurrently for all skipped layers as they all use the same d from the exited layer.
3.3.2 Sensitivity to local errors
Next, we examine the impact of local token modificationsswhich might occur due to early exits-on the whole generated sequence. We experiment with two kinds of perturbations: sampling-bat -based. where we select the 10th-ranked token according to layer L; and layer-based, where we select the the first layer's prediction timestep f. All other tokens are predicted greedily by layer L. As shown in Figure 2a, earlier perturbations result in lower sequence-level scores as there are more tokens that might suffer from the divergence. The degradation, though, is much smaller with layer- compared to campling-based perturbations since, in practice, the early exit predictions are mostly accurate.
Decaying threshold. Following the above observation, we introduce decaying early-exiting threshold that is more permissive towards exiting as the decoding process continues. Motivated by the logarithmic behavior in Figure 2a, we use an exponential function with user-defined temperature 7:
X(A, == clip
(10
(5)
10
where is the maximum output length. Figure 2b illustrates this function. Essentially. this function presents effective compromise between simply using the same threshold for all tokens, and searching over a huge space of per-position different thresholds. Practically, it supports finer and tradeoff compared
better control over the performance-efficien, presents the outcomes of a search over À with steps of 0.01 and softmax-based confidence ($3.5). With the single threshold variant (7 0), attempting improve the efficiency will lead drastic drop of more than 10 points the textual similarity against the full model's prediction. In contrast, the decaying thresholds reveal several intermediate points with desirable tradeoff's to consider.
single threshold. Figure 2c
3.4 Training early exit classifiers for local consistency
While our goal to preserve the quality of the complete output sequence, we note that this doesn't. t necessarily demand local token-level consistency. Consider the target sequence "the concert was wonderful and long." An output that switches the order of adjectives to "the concert was long and wonderfull would be called consistent by most semantic measures (and obtain 100 token-Fi score). Yet, the sentences diverge at the first adjective long which semantically different from wonderful.
Training for global consistency, however, could be challenging [81] as it depends possibly noisy signals that might affect the learning, and also breaks the efficient teacher- forchig training strategy of LMs that relies on docal-decisions On the other hand, perfect local consistency implies global consistency. Therefore, we opt to train for local consistency, which requires minimal changes to the training procedure, and relax the local requirement to global one during inference.
Specifically, similar to Elbayad al. [23], we average losses for each layer to obtain the objective
L
Σ wile, where Σ
1.
(6)
1=1
5

---

C is the negative log-likelihood loss. We set =// to favor higher layers, and find this objec- tive to mostly preserve the full model's performance compared to regular training. We note that there some misalignment between this training and inference behavior due the hidden states of skipped layers. However, as discussed in $3.3.1, the performance is not affected if the hidden-state is copied.
3.5 Local confidence measures
We experiment with three confidence measures for Eq. (4) that differ in their parameter and compute operation efficiencies. Our experiments ($6) will also show that they differ in their predictive power.
Softmax response. We take the difference between the top two values of Softmax Wid)). With a large output vocabulary, this results in many floating point operations (FLOPs)-thoul the next layer i + can start its computation in parallel, avoiding additional runtime.
Hidden-state saturation. As a simple parameter- free and fast to compute alternative, we take the cosine similarity sim d;= ( for > 1. By definition, the first possible exit is at the second layer (unless À= 0). This measure tries identify early saturation events of the hidden-state [28].
Early exit classifier. We train a dedicated linear classifier M to predict the likelihood of exiting with local consistency given the current hidden-state: cj MMdd).. This measure is very fast to compute at inference, and adds only d|] + new parameters. To avoid any impact on the core model's performance, train it as second step where use a per-layer independent cross-entropy loss against a consistency oracle 11 [arg max(p(y: V2++ (d)) = arg (pxxp((1+1 ). and average across the I. T- 1 layers. We also experimented with the geometric- like training of Elbayad et al. [23], but find it to be less effective here (see App. D). The two objectives closely related, but the geometric one ignores any signal from the states post the first oracle exit.
freeze all parameters other than M. We simply
Calibrating Local Early Exits from Global Constraints
We now describe our calibration procedure for finding a shared exit threshold E [0, 1] that can be used directly in Eq. (4), via Eq. (5), such that we provably satisfy our desired global constraints over the fully generated sequences. At high level, our approach uses the following basic recipe:
1. We specify a grid of possible values of A = (Ai,...,AA) that may result in acceptable generations;
2. We choose the lowest valid À E A that we can identify with rigorous statistical testing tools.
Let Puett be an i..dd prompt given the LLM at test time, and let Yiul LLM (Pest)) ) and Yearty := LLr (Preat: A) E y denote the full and adaptive responses, respectively, Optionally, let Zuest be set of gold references for our task, if assumed. Our goal, as introduced in $1, is to find valid À using Scal such that we satisfy either of two types of global "consistency"" constraints:
Definition (Textual consistency). An adaptive LIM is textually consistent if given any bounded text dissimilarity function, D: y y -> TR. and tolerance 5 TR. [[D(aaarlyy Yrati] S 8.
Definition 2 (Risk consistency). An adaptive L.M is risk consistent if given any bounded risk function, R: y x 2 + R. and tolerance 5 E R. (Y Zusst) S [R Zrest) +
Without loss of generality, we will assume that DD and R are always normalized to the unit interval [0, 1]. and therefore will only be considering tolerances 5 E (0. J. At a glance, to find a À that produces a consistent LLMearly we cast our problem as a multiple hypothesis testing problem over a large array of k candidate classifier exit thresholds, A = (Ai,..., As), and apply the Learn then Test (LTT) framework of Angelopoulos et al. [3] to identify a subset of statistically valid, constraint-satisfyiin thresholds Avalid A. Our final À is then chosen as À == min(Avali U (1)).
4.1 The Learn then Test calibration framework
Choosing a value of À that rigorously satisfies our consistency objectives is challenging, as the performance impact of increasing or decreasing not necessarily monotonic. Naively setting X, for example, based simply on average calibration set performance, can lead to statistically invalid results in our finite-sample, distribution-free setting. The LTT framework proposed by Angelopoulos et al. [3] solves this problem by reframing hyper- parameter selection as multiple testing problem.

---

Let A = (AI...., A&) be a finite grid of consistency. For example. when searching for value of [0, 1], we might consider the evenly spaced set ( i k.. LTT then identifies a subset of values, Avalid C A, where
values that may. or may not, obtain valid
p EE E Avalid: LLMearly (Prest. A) and LLM are not consistent S
Here, we are using consistency to refer to either textual consistency or risk consistency. Eq. (7) can be satisfied by applying standard multiple hypothesis testing techniques as long super-uniform p-values, Pj. are supplied for each value >3 E A that support the null hypothesis
Hy: LLMearly Peest 2ÿ) and LLM (Pest)) are not consistent.
Xj is placed in Ayali if H2 is rejected, and discarded otherwise. This yields a consistent LLMearly Proposition 1 (LTT for CALM). Suppose Pj is super-uniform for all under H₃ for some specified tolerance E (0, 1). Lel A be any family-wise error rate (FWER) controlling procedure at level E (0,1), where A(p1,-.-,PPh selecty H2 to reject. Choosing = min(Avalid U (13) then yields a consistent L.MMaarl with probability at least 1 6.
Note that a FWER -controlling procedure level is an algorithm that decides to accept or reject hypotheses ( {Ha) while ensuring that the probability of falsely rejecting any H, is less than e. The proof of Proposition 1, given in Appendix 1, follows directly from Theorem 1 of Angelopoulos et al. [3]. and the fact that LLMearly (Piest, 1) = LLM (Peeat) by construction per Eq. (4), so that we can always use À= = 1 as a valid fallback if we fail to identify non-empty Avalid. In the next sections, we describe how we calculate valid p-values using Seat, and our choice of FWERRcontrolling procedure.
4.2 Defining p-values for consistent early-exiting
LTT relies on valid p-values Pj. where Pj is a random variable satisfying P(pj I u) < u under H3 for all u E [0, 1]. For our purposes, we can obtain valid p-values from the empirical consistency of LLMearly (P,. A) measured over the random calibration sample, Seal. Since we have assumed w.l.o.g. that either of our bounded consistency functions D and from Defs. and 2 have been normalized to lie in [0, 1]. we can, for example, obtain valid p-value by simply inverting Hoeffding's 's inequality:
Hoefiding := Pj
where B(Ay) = Σ Lx(A,)
is the empirical average of random variable L,(,)) E [0, 1]. with
L,(,,) D(LLM (P,, A3), LLM (P.))
(10)
(, == max (0, R(LLMearly (P,, 1y), Z,) - R(LLM (P.)) (P.)..)).
(11)
for textual consistency versus risk consistency, respectively. Note that, as technicality of enforcing the r.v. (Aç)) to be within [0, 1J. Eq. (11) computes conservative estimate of the difference in the
empirical risk that doesn't reward instances in which the risk of the early-exit model is lower.
4.3 Efficient fixed sequence testing
The more values of À we test, the higher the chance that we might accidentally choose a that does not in fact result in consistent generations, despite whatever misleading performance we might have measured by chance on Seat. As part of LTT, we must select a multiple testing procedure that corrects for this (i.e., that controls the FWER at level e). Though the precise dependence between the early-exit LLM's performance and À is unknown, in practice we find that it tends to be fairly smooth and roughly monotonic. That is. nearby thresholds À X' tend perform similarly, whereas x > X' tends to result in relatively more consistent performance. Taking ad advantage of this structure, we choose to employ fixed sequence testing (FST) as our controlnn procedure [3; 11].
Here we define a sequence of descending thresholds A > X2 >..... d with a relatively coarse step size (e.g., increments of 0.05). For each >5 in order, compute p;. and reject H, if pj S c. The first time we fail to reject H, we immediately terminate our search, and return Aj-1 to use as our calibrated threshold (or if we fail to reject H1). An Algorithm of the full procedure is provided in Appendix E.
In practice, we the more powerful Hoeffding Bentkus bound [3: 10: 12; 33J.
7

---

113
123
273
225
BCC
docoder Ddlin
decader J0es
Reyers
CNN/DM
WMT
(C) SQUAD
Figure 3: Validation empirical performance-efficiee tradeoffs for different confidence measures, compared to static baselines and local oracle measure with state propagation for skipped layers.
5 Experimental Setting
We empirically evaluate our methods on three popular text gen- eration tasks that vary in their target generation length and ex- tractive degrees against the input. CNN/DM [31] is collection of news articles to be summarized in few sentences. WMT15
Table 1: Average number of to- kens in reference targets of eval- uation datasets (5/95th percentiles in parenthesis).
EN-FR [13] contains English sentences (one per example) be machine translated to French. Open-book SQUAD 1.1 [54] is a QA dataset with Wikipedia paragraphs paired with questions, where the target answer is text span from the input. Length statistics of the validation sets are summarized in Table
Dataset
Output length
CNN/DM
82 (42 141) 39 (10 82)
WMT EN-FR
SQUAD
5 13)
Model. We implement CALM on top of the T5 incoder-decode model that showed good performance on the tasks above [53], using the T5X framework [55]. We use the layers T5 1.1 model that doesn't share input and output embeddings. We share all output embeddings for the softmax predictions, and the early-exit classifier across all decoder layers. Based on validation results, we set the temperature of our decaying threshold to = 4 for the softmax and classifier measures of CNN/DM and WMT. In other settings, we use r = See App. C for more details, and App. B.3 for a 12 layers model.
Evaluation metrics. We use the standard metrics for each task: ROUGE-L for CNN/DM, BLEU [50] for WMT, and Token-F1 [54] for SQUAD. We rely the and textual distance, other than BLEU which is corpus-level metric that doesn't directly en- able expectation control. Instead,
metrics for computing the risk
use the BLEURT Iearned metric [61]. For a given metric m mVVearly Mfull or Stest) E [0, 1], we use 1 - m for distance or risk computation, respectively.
Our main efficiency metric is the average number of decoder layers used per output token, it directly measures complexity reduction without conflating with implementation or infrastructure spe- cific details [19]. For reference, we also report the average decoder FLOPs reduction per token [23]. Also, we compute an estimated speedup of the whole encoder-decoder model for generating the full sequence, based on TPUv3 benchmarking with 200 examples in Colab (see App. C for details).
Calibration experiments. For each task, we use the validation and test sets to evaluate our calibration method ($4) (for SQUAD only use the validation set the test answers hidden). We run 50 random trials per target tolerance å and consistency objective (textual or risk), where we partition the data to 80% calibration (Sca)) and 20% test (Prest). We set 0.05 for all experiments.
Baselines. We emphasize that the CALM framework is general for any autoregressive multi-layere LM with any confidence measure, allowing controlled consistency by Eq. (1) or Eq. (2). To empirically evaluate the efficiency gains enabled by our proposed confidence measures, we compare with static baselines that use the same number of layers for all tokens. We also compare our early-exit classifier training with the geometric method of [23] Appendix D. Also, we compute an oracle local measure ($3.3.1) as upper-bound estimate of the performance-eeffccinnyyy tradeoff.
Experimental Results
We first report the empirical performance-efficiengg tradeoff achieved with each confidence measure. For each task and measure, evaluate the full range of À on the validation set, with steps of 0.05.

---

Table Test efficiency gains per choice of 8, consistency objective, and confidence measure. is set to 0.05. For plots of the full range of ó with standard deviation, see Appendix B.
CNN/DM FLOPs
WMT
SQUAD FLOPs
6
Measure
layers
speedup
layers
FLOPs
speedup
layers
speedup
Textual consist.
softmax
5.73
x0.44
x1.41
3.35
x0.66
x2.01
1.65
x3.15
x1.63
07
state
8.00
x x1.00
x x1.00
7.68
x1.01
x1.00
2.00
x3.65
x1.68
classifier
7.16
x1.03
x x1.42
5.50
x1.06
x2.05
2.59
x2.37
x1.10
softmax
2.62
x0.49
x2.57
1.76
x0.91
x2.83
1.03
x5.68
x1.88
0.25
state
7.97
x1.00
x1.01
2.84
x1.93
x1.55
2,00
x3.65
x1.68
classifier
4.51
x1.15
x2.04
2.97
x1.22
x2.00
1.37
x6.09
x1.11
softmax
3.75
x0.47
1.966
3.19
x0.67
x2.10
1,65
x3.15
x1.63
Risk consist.
0.02
State
7.97
x 11.00
x x1.01
7.68
x1.01
x x1.00
3.13
x2.11
x1.68
classifier
6.49
x1.06
x1.71
5,05
x1.08
1..97
3.36
x1.55
x1.11
softmax
1.73
x0.50
x3.53
1.96
x0.85
x2.73
1.65
x3.15
x1.63
0.05
state
5.22
x 1.111
x1.64
2.72
x2.01
x1558
2.00
x3.65
x1.68
classifier
2.30
X 1.25
X2.09
3.08
x1.21
x11.8
2.59
X2.37
x1.10
The results, presented in Figure 3, show the power of the softmax response measure, allowing only minor performance loss while reducing more than half of the layers in all three tasks. The early-exit classifier, that is more FLOP-efficient, is also effective, mostly when targeting high performance (right hand side of plots). The simple and parameter- trrameter-free state saturation measure is competitive, but often falls bellow the static baseline, despite enabling per-token exit decisions.
The dynamic oracle obtains compelling efficiency gains, using only 1.5, 1.3, and 1.2 layers on average for summarization, WMT, and QA, respectively, without losing any performance. This illustrates the full potential of CALM and leaves further room for improvements with better confidence measures. It also shows the effectiveness of inference-time state propagation for skipped layers ($3.3.1).
6.1 Calibrated performance with guaranteed textual risk consistency
Next, we examine the outcomes of the calibration process. Since the obtained risk is guaranteed to he valid (i.e., S 5 at least 95% of the time), we focus here on efficiency gains per chosen 5. We refer the reader to Appendix B for empirical validation and for additional results and qualitative examples.
Table 2 presents the efficiency gains per choice of å for each consistency objective and confidence measure. We examine larger 8 values for textual consistency as this is generally a stricter requirement since the full model's error is not considered.
Across all, the softmax confidence measure leads the greatest decrease in number of decoder layers required. Accordingly, softmax mostly enables the highest speedup gains of to about three times faster than running through all the model's layers. The very lightweight early-exit classifier sometimes provides better gains than softmax, even if more decoding layers are used. Since the speedup computed over the full generated output, we see more gains on the longer outputs of summarization and translation where the decoding takes most of the time, compared to the short QA outputs where the whole decoding time is not much longer than the encoding time.
These encouraging efficiency gains are enabled even with the rigorous performance guarantees that are sometimes conservative (c.g., Eq. (11)). We note that relaxing these constraints, or tightening the confidence intervals (e.g., with larger calibration sets), can further improve the empirical gains.
The softmax operation potentially be paralleled), sometime leading to increased total FLOPss even with fewer used layers. The state-based and early-exit classifier measures require minimal FLOPs and provide good alternative with compelling efficiency gains, if total (parallelizable, or not) FLOPs is of concern.
the full output vocabulary is FLOPs heavy (though, this compute can
6.2 Example output: effectively distributing the model's capacity across timesteps
Figure 4 presents two CALM summary generations for article from the CNN/DM dataset,
pared to the output of the full model (See Figure B.5 in the Appendix for examples from the other
tasks). Yearly uses lower confidence threshold for early exiting compared to y-1)) eerly- The colors,
depicting the number of decoder layers used per output token, illustrate how CALM obtains the

---

Cup final Eliott's penultimete bell 7 Eliotl, 38, not played international crickel 14 monthe when picked. Win is vindkaaiio for the sllacking brand played under Brendon
South Alricabbom
maidh-inning
semi-linal. Caps reached
McCuluum New Zealand play the winer of the semi-fnal between Australla or India.
Yun
Grant Ellot hit six to put New Zoaland through the World Cup final. The 36-yeanold was born South Africa but naturalised Kini. EBlott will surely never another innings like his 84. New Zooland take eiher Australia or India the final Sunday.
Grant Elion hit 3
SX_ lo pul New Saceeenn_____puupeee 10 the_
_0______________________________ World_ _up__inal__________
Eliot was bom South Africa but.
pi)
naturalized
The__ 36-yeard _url______eell__________ neve___ee____________ _aathaaaaarr________ee____eeeeee nrings_________________________
unbeaten
New Zeo land will now lake on wither Austtarr__sssiaa..a or India (1_ 218_____________________________ linel Sunday, <EOS>
ytb
Grant Ellort hi B4 FL the Black Cape__ chase New Zealand reachedeeeeedd_eeeeeeee The Whor__ World_ Cup linel Elioil was born Jn in Afric a naturalized Kini Eliott will surely never play another innings like his unbeaten <E0S>>
casy
Oa Teal
Redty - Rus
Average lnyers
Spendup
Exit layer bolor magping: 12346678
exity
D and R are computed with ROUGE-L
0.33
-0.3
1.9
X38
Figure 4: CALM accelerates the generation by early exiting when possible, and selectively using the full decoder's capacity only for few tokens, demonstrated here CNN/DM example with softmax-based confidence measure. y'D and Early use different confidence thresholds for early exiting. Bellow the text, we report the measured textual and risk consistency of each of the two outputs, along with efficiency gains. The colors represent the number of decoding layers used for each token light green shades indicate less than half of the total layers.
efficiency gains. Only a few selected tokens use the full capacity of the model (colored in red), while for most tokens the model exits after one or few decoding layers (colored in green).
The example in Figure 4 also demonstrates one difference between the two types of consistency constraints, given a reference output Zrest. Textual consistency D(Yearly Yfiu)) generally (though, not always) degrades (i.e., increases) when decreasing the confidence threshold as the outputs tend to more significantly diverge from Yiull. The trend of risk consistency, however, depends also on the reference output Ziest- If Yiull ~ Zrest then the two constraints are nearly the same. In this example,
they are sufficiently different that y obtained better (lower) risk even though the textual distance from Yiull is higher. On the one hand, given the availability of reference outputs for calibration, this suggests that for an imperfect model, risk consistency could lead to more aggressive early-exiting while maintaining the quality of generations. On the other hand, since the Relu in Eq. (11) doesn't 't reward negative risk differences, the benefits might not fully materialize. Overall, the two constraints provide different alternatives for the user choose from depending on the availability of reference outputs, the performance of the full model, and the exact desired performance guarantees.
7 Conclusion
We present confident adaptive language modeling (CALM) for dynamically allocating different amounts of compute per generated token, following explicitly defined tolerance levels on the full gen- cration output. This paper covers both modeling solutions and analyses towards this goal, as well as a aneoretically-gronndee framework for provably controlling the quality of the full output meet the user-specified tolcrance levels. We investigate the effects of local early exiting during decoding on the final output, leading us to propose a decaying function over the initial threshold that enables finer con- trol over the performance-efficinng ent solutions for addressing missing computations of early-exited tokens that are dependent upon for future tokens. Overall, our complete adaptive compute framework for LMs requires minimal modifica- tions to the underlying model and enables efficiency gains while satisfying rigorous quality guarantees for the output. Also, our oracle experiments and runtime analysis demonstrates the full potential of this framework and leave room for future research to further improve the efficiency in a controllable way.
tradeoffs without inflating the search space. We also study differ-
Acknowledgements
We thank lonel Gog for significantly improving the implementation after submission. We also thank Anselm Levskaya, Hyung Won Chung, Seungyeon Kim, Tao Wang, Paul Barham, and Michael Isard for great discussions and code suggestions. We thank Orhan Firat, Carlos Riquelme, Aditya Menon, Zhifeng Chen, Sanjiv Kumar, and Jeff Dean for helpful discussions and feedback on the project.
10

---

References
[1] Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-ffree uncertainty quantification. 2021. doi: 1048555AARX1221000055111 URL
https: / org. http:/bs///// http://orgrbbb///////// 07511.
[2] Anastasios Nikolas Angelopoulos and Stephen Bates. A gentle introduction conformal prediction and distribution-free uncertainty quantification, 2021. URL https: //axxiv.. Stg/abs/2101 07511.
[3] Anastasios Nikolas Angelopoulos, Stephen Bates, Emmanuel J. Candès, Michael I. Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms achieve risk control. ArXiv preprint: 2110.01052, 2021.
[4] Anastasios Nikolas Angelopoulos, Amit Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distribution-free uncertainty quantification and applications imaging. ArXiv, abs/2202.05265, 2022.
[5] Vamsi Aribandi, Yi Tay. Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference Learning Representations, 2022. URL https: //wp///eeie.....e.e net/ / wwwrrummiii-iee Vahhiuiiiiiaaaiiii..
[6] Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit bert quantization. arXiv preprint arXiv:2012.15701 2020.
[7] Yu Bai, Song Mci, Haiquan Wang, Yingbo Zhou, and Caiming Xiong. Efficient and differen- tiable conformal prediction with general function classes. ArXiv, abs/2202. 1091, 2022.
[8] Aaron Baier-Reinio and Hans De Sterck. N-ode transformer: A depth-adaptive variant of the transformer using neural ordinary differential equations. ArXiv, abs/2010.113335 2020.
(9] Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Controlling computation versus quality for neural sequence models. 2020. doi: 1044850/ARXX122002.0711 URL httpp:://aaae org/ abs 2002. C7106.
[10] Stephen Bates, Anastasios Nikolas Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution free, risk controlling prediction sets. ArXiv preprint: 2101.02703, 2020.
[11] Peter Bauer. Multiple testing in clinical trials. Statistics in medicine, 10 6:871-89; discussion 889-90, 1991.
[12] Vidmantas Bentkus. On hoeffding's inequalities. Annals of Probability, 32: 32:1650-1673, 2004.
[13] Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp. Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. Findings of the 2015 workshop on statistical machine translation. Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 1-46, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/11W115-33 3001. URL https: / a aclanthology.......ooo.ooooolo.. org/ org/w15-3001
[14] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Miin Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http:// /ttp//gttttt...o/ wtt gowp//goolll/oooo 1 e/ jax.
[15] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh,

---

Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020. URL https: / newrips.ipp..... wttpiisss.ccappeeeeee cc/pappe/2220/ Paper. pdd.
[16] Berkant Barla Cambazoglu, Hugo Zaragoza, Olivier Chapelle, Jiang Chen, Ciya Liao, Zhaohui Zheng, and Jon Degenhardt. Early exit optimizations for additive machine learned ranking systems. In WSDM 10, 2010.
[17] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy MecerrHHellseen Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. 2022.
[18] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Uni- versal transformers. In International Conference on Learning Representations (ICLR), 2019.
[19] Mostafa Dehghani, Yi Tay, Anurag Arnb,, Lucas Beyer, and Ashish Vaswani. The efficiency misnomer. In International Conference on Learning Representations, 2022. URL https: /openrevieww net / perum?id=iulMMa YhluR.
[20]
Jacob Devlin, Ming-Wei: Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pages 4171-411 1171-4186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19..1 1423. URL https::tp///c.... / ac lacland aellanhhloogy wrw.:gg..rrr////.. 91123423911...
[21] Neil Dey, Jing Ding, Jack G. Ferrell, Carolina Kapper, Maxwell Lovig, Emiliano Planchon, and Jonathan P Williams. Conformal prediction for text infilling and part-of-speech prediction. ArXiv, abs/211 0ttp/2111.02555 1.02592, 2021.
[22] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathy Meier-Hellsterm, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture- of-experts. 2021.
[23] Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. Depth-adaptive transformer. In International Conference on Learning Representations, 2020. URL https: ht /ttpt////p/peeeeeeeeeeee fttt/forummid=sJppep
[24] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. arXiv preprint 1xXiv:1999.11556 2019.
[25] Michael Figurnov, Artem Sobolev, and Dmitry Vetrov. Probabilistic adaptive computation time. ArXiv, abs/1712.00388. 2017.
12

---

[26] Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay, Efficient conformal prediction via cascaded inference with expanded admission. In International Conference on Learning Representations (ICI.R), 2021.
[27] Adam Fisch, Tal Schuster, T. Jaakkola, and Regina Barzilay, Conformal prediction sets with limited false positives. ArXiv, abs/2202.07650, 2022.
[28] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. 2022.
[29] Alex Graves. Adaptive computation time for recurrent neural networks. 2016.
[30] Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, and Yoshua Bengio. On integrating language model into neural machine translation. Computer Speech & Language, 45:137- 201..01..14 URL https: 148, 2017. ISSN 0885-2308. doi: aaccinnceartiiiiiin /ttp://///p../////303333 6301395. fect.ccometttee
/ h/www. sci encedi sciencedi
[31] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and com- prehend. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gamett, ed- itors, Advances in Neural Information Processing Systems, volume 28, Curran Asso- ciates, Inc., 2015. URL https: / http:///pooeeeeeeeeeeceeo afdec 0eccc005ccee F1 40022440474 Pappe .pdf.
cccppaper/201///1ee
[32] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. ArXiv, Abs/1503.02231, 2015.
[33] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58:13-30, 1963.
[34] Lu Hou, Zhiqi Huang, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33: 9782-9793, 2020.
[35] Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recurrent neural networks. arXiv preprint arXiv: 16/ 180, 2016.
[36] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint urXiv:1909 10351, 2019.
[37] Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah Smith. Deep encoder, shallow decoder:
Reevaluating non-autoregressive machine translation. In International Con- ference on Learning Representations, 2021. URL https: //opnreeeeveeeeeee net/ forum? Id=KpfasTaLUjg,
[38] Gyuwan Kim and Kyunghyun Cho. drop, use anytime with search. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joins Conference on Nat- ural Language Processing (Volume 1: Long Pupers), pages 6501-6511, 1, Online, August 2021. Association for Computational Linguisties. doi: 10. N0.118653/112000 lall-long.500 URL https: ptttps://acllathoooog org 2021 acl 508.
transformer: Train once with length
[39] Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh- Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages
3577-3599, Punta Cana, Dominican Republic, November 2021, Association for Computational Linguistics. doi: 10. 10.18653/v1/2021, http5///w122222.fnnnnn---n-/ ainding--emnl.300 URL https:: / //all antho org/202/ findings emnlp. 304.
[40] Imad Lakim, Ebtesam Almazrouei, Ibrahim Abualhaol, Merouane Debbah, and Julien Launay. holistic assessment of the carbon footprint of noor, a very large Arabic language model. In Pro- ceedings of BigScience Epivode #5 Workshop on Challenges & Perspectives in Creating Large Language Models, pages 84-94, Dublua,,Dublin, May 2022. Association for Computational Linguistics. URL https: hitee://acaaaae /aclanthooogyy.c.ooo. org/ / 2022 .bigscience-1...
13

---

[41] Tao Lei. When attention meets fast recurrence: Training language models with reduced compute. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7633-7648, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 186138851122222 .emnlp-mainnn
URL
https: ac/ www.ocolnnntoooocooooooooooooooe org/ 2021 emnlp-ma i 602.
[42] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. {GS Jhard: Scaling giant models with conditional computation and automatic sharding. International Conference on Learning Representations, 2021. URL https: : www.reveeeeeeeeeeeeeeeee. net. /tt//ferurrr forum? d==qree 7XHTmYb.
[43] Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. Cascadebert: Accelerating inference of pre-trained language models via calibrated complete models cascade. arXiv preprint arXiv:2012. 14682, 2020.
[44] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. Fastbert: a self-distilling bert with adaptive inference time. arXiv preprint arXiv:2004.0 2220000442777 02008 2020.
[45] Xiangyang Liu, Tianxiang Sun, Junliang He, Lingling Wu, Xinyu Zhang, Hao Jiang, Zhao Cao, Xuanjing Huang, and Xipeng Qiu. Towards efficient nlp: A standard evaluation and strong baseline. arXiv preprint arXiv:2 10.07038, 2021.
[46] Yijin Liu, Fandong Meng, Jie Zhou, Yufeng Chen, and Jinan Xu. Faster depth-adaptive transformers. In AAAI, 2021.
[47] Zhuang Liu, Zhiqiu Xu, Hung-Ju Wang, Trevor Darrell, and Evan Shelhamer. Anytime dense prediction with confidence adaptivity. In International Conference on Learning Representations, 2022. URL https: //tp::////eeeeeeeeeeeeee
n fo rumm? (==NKFFX1ees
[48] Charles Lu and Jayasheree Kalpathy-Crames Distribution-fire federated learning with confor- mal predictions. ArXiv, abs/2110.0766.. 2021.
[49] Nafise Sadat Moosavi, Angela Fan, Vered Shwartz, Goran Glavaš, Shafiq Joty, Alex Wang, and Thomas Wolf, editors. Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, Online, November 2020. Association for Computational Linguistics. URL https::///// / //a///aaantttt aacaanthlllooyyocc org/ 2020. sustainlp-- 1.0.
[50] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3115//073083.1073333 URL https: 2atp:/22-11440
/ //aclanthologg
[51] Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/11/1888 1202. URL https: / /acl antho ogy- org//1 N1 -
[52] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
[53] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. 2019.
[54] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas, Novem- ber 2016. Association for Computational Linguistics. doi: 10.1 1653/v 1264. URI. https: / http///aaaaaaatl...ooo wwwp:yy.oorr////.. wrg//oggl/e........ 1264.
14

---

[55] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor. Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Mare van Zee, Jacob Austin, Sehastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepar Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with 5x and seqio. arXiv preprint arXiv:2203. 2022. URL https: / arxiv. org/ / abs / 2tt.//22...1111 17189.
[56] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 108, 2019.
[57] Tal Schuster, Adam Fisch, Tommi Jaakkola, and Regina Barzilay. Consistent accelerated infer- ence via confident adaptive transformers. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 4962-49799 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 16553/1/20211. emnlp- 406. URL https: / /ataaaaaaantttoooooeooa org/ 2021. emnlp main. 406.
[58) Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. Programming puzzles. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: openreview eww..w net/ fest/forree forum? fewddddcefhecfeeeeeeee
[59] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12))54-63,, Nov 2020. ISSN 1557-7317. doi: 10.1145/3381831 URL
http: http://x..diior//////////
doi. org. 114..111111133133111
[60] Roy Schwartz, Gabriel Stanovsky, Swabha Swayamdipta, Jesse Dodge, and Noah A. Smith. The right tool for the job: Matching model and instance complexities. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages July 2020. Association for Computational Linguistics. doi: 10.
Online,
URL https: / acc anthologyyy org/ 2020. acl-malll 593.
[61] Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 7881-7892, Online, July 2020. Association for Computational Linguis- tics. doi: 10. 10.18853//1120000 acl- main.704. URL https: :ttess://acllee /ttp//aaaaaanntaaayaaaaaaaaaaaaa org/2020. acl-ma 704.
[62] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research (JMLR), 9:371-427 June 2008.
[63] Or Sharir, Barak Peleg, and Yoav Shoham. The cost of training nlp models: A concise overview. arXiv preprint: arXiv 2006.0618, 2020.
[64] Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. 2018. doi: 10.48550/ARXIV.18000.0 04235.
URL https: /ttp///aww....bbbaaaaaaa/ 11804
[65] Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8815-8821, 2020.
[66] Antoine Simoulin and Benoit Crabbé. How many layers and why? An analysis of the model depth in transformers. In Proceedings of the 59th Annual Meeting of the Associ- ation for Computational Linguistics and the 11th International Joint Conference on Nat- ural Language Processing: Student Research Workshop, pages 221-228, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v 1 //222 acl-sr.. URL https: / aclalllaa //tp//caannttoogoooo wwwp:og...rgg2220// aww-sr..lssw........
15

---

[67] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao. Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, and Garrig--Alosoo al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. 2022. doi: 10.48550/ARXXIV2226..444
URL https: /aaiivivvv/////////v//////////aa vi..vvvvv .rgg abs/ 2200 04615. 5.
[68] Asa Cooper Stickland and Iain Murray. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. In International Conference on Machine Learning, pages 5986-5995. PMLR, 2019.
[69] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arviv: 199900009555 2019.
[70] Tianxiang Sun, Xiangyang Liu, Wei Zhu, Zhichao Geng, Lingling Wu, Yilong He, Yuan Ni, Guotong Xie, Xuanjing Huang, and Xipeng Qiu. A simple hash-based early exiting approach for language understanding and generation. In Findings of the Association for Computational Linguistics: 2022, pages 2409-2421, Dublin, Ireland, May 2022. Association for Computa- tional Linguistics. URL https: htttss://cclannetttttttt http://aacaantoooay..oo.o.ooooaa org. 2022. f findiin inndinngnacci 89
[71] Zhiqing Sun, Hongkun Yu, Xiaodan Song. Renjie Liu, Yiming Yang. and Denny Zhou. Mobilebert: compact task-agnostic bert for essuureelimited devices. arXiv preprint WXXi:2204.009988 2020.
[72] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Computing Surveys (CSUR), 2022.
[73] Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint wwxiv:2205.05133, 2022.
[74] Surat Teerapittayanon, Bradley McDanel, and H. T. Kung. Branchynet: Fast inference via early exiting from deep neural networks. 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2464-2469, 2016.
[75] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam M. Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos. Leslie Baker, Yu Du Yaguang Li, Hongrae Lee, Huaixiu Zheng, Amin Ghafouri. Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Chinn Chang, I. A. Krivokon, Willard James Rusch, Mare Pickett, Kathleen S. Meier-HHllstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Hartz Søraker, Ben Zevenbergen, Vinodkumar Prabbakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin HoffmannJohhn Josh Lee, Lora Aroyo, Ravindran Rajakumar, Alena Butryna, Matthew Lamm, V. O. Kuzmina, Joseph Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. Lamda: Language models for dialog applications. ArXiv, f/s/2201.082333 2022.
[76] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems (NeurIPS), 2017.
[77] Vladimir Vovk, Alex Gammerman, and Glenn Shafer, Algorithmic Learning Random World. Springer- Verlag, Berlin, Heidelberg, 2005.
[78] Wenhui Wang, Furu Wei, Li Dong, Hangho Bao, Nan Yang, and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020.
[79] Xin Wang. Fisher Yu, Zi-Yi Dou, and Joseph Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. ArXiv, abs/1711.09485, 2018.
[80] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. 2022.
16

---

[81] John Wieting, Taylor Berg- Kirkpatrrt Kevin Gimpel, and Graham Neubig, Beyond BLEUstraining neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4344 4355, Flo- rence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 118865//1//19// 14277 URL https: / ac annhhoooyy org/P19-1427.
[82] Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins. and Christian Szegedy. Memorizing transformers. In International Conference on Learning Representations, 2022. URL https:
//penrevieww net wtwpr//vwwe.eeeeoorrmtteeeee... forum?id-Tr
[83] Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. arXiv preprint arXiv:2004. 12993, 2020.
[84] Ji Xin, Raphael Tang. Yaoliang Yu, and Jimmy Lin. BERxiT: Early exiting for BERT with better fine-tuning and extension regression. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 91-104, Online, April 2021, Association for Computational Linguistics. doi: 10.18653/v1/2021 .eal222...al-mmi... URL htt / antto ogy. 2tgg/022: eacl -ma inmainn... 8.
[85] Jingjing Xu, Wangchunshu Zhou, Zhiyi Fu, Hao Zhou, and Lei Li. A survey green deep learning. ArXiv, abs/2111.05193, 2021.
[86] Linting Xue, Aditya Barua, Noah Constant, Rami AI-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. Byt5: Towards token-free future with pre-trained byte-to-byte models. 2021. doi: 1044550ARXXV.2100.335222 URL httpptt/t/eeeeee.tt /taxxaxxvvooroo.///aaa org /bs/ 2105.1362.
[87] Hongxu Yin, Arash Vahdat, José Manuel Álvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. Adavit: Adaptive tokens for efficient vision transformer. ArXiv, abs/2112.0766 2021.
[88] Margaux Zaffran, Aymeric Dieuleveut, Olivier F'ron, Yannig Goude, and Julie Josse. Adaptive conformal predictions for time series. ArXiv, abs/2202.07282, 2022.
[89] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. 2022.
[90] Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330-18341 2020.
[91] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoe Le, and James Laudon. Mixture-of-experts with expertchoice routing. 2022. doi: 10. 00.48500ARXX1.22222000
URL https: //apxi:////xxive htt.t:/:////aaiioooo//aaa//////s wrgp://///////////////////////// 00322.0222.0.....22.222222222221 09368.
[92] Wei Zhu. Leebert: Learned early exit for bert with cross-level optimization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 2968-2980, 2021.
17


Reviews:
[{"review_id": "KLs3yJ_vfAV", "reviewer": "Reviewer_FJ71", "summary": "UPDATE: The authors have answered my concerned so I increased my score to an 8. \n\nTransformer language models such as T5 are the most important tool in the current NLP world. Typically the decoder has a set number of layers and all of these layers are executed for each token. In the past few years we've seen many papers that do 'early exit' for tokens for which the output is easy to compute- during these timesteps, the model exists early and does not compute all the layers. \n\nThis paper first analyzes these types of model with an oracle to show what the best case speedups could be, and then proposes a few new ways to do early exit, for the T5 model, evaluated on a group of interesting downstream tasks such as summarization and translation. \n\nI felt like the paper wasn't very clear in comparing the final performance to the many prior works such as Elbayad et al or Schwartz et al. In fact Table 2 which is the main table that presents results only contains the 3 proposed models, it does not present the full model (no early exit) baseline and it does not present prior models such as Elbayad et al or Schwartz et al.\n\nIn addition, while these models have been proposed for the past couple of years, none of them have been adopted or used in production. I feel like this might be because of many issues with these methods that none of these papers really discuss. For example, you point out the \"potential FLOP speedup\", but what is the *actual* wall clock speedups of your methods? When doing batched inference, these methods can't really be used because every token in the batch might require a different amount of compute. It would be good to discuss these potential issues in the next version of the paper. ", "questions": "My questions are in the prior sections. The most important question is how would table 2 look if you added numbers from the full (no early-exit) model baseline and from the Elbayad and Schwartz papers. ", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The analysis in this paper is super interesting, the oracle analysis shows that these methods might have big potential.\n\nWeaknesses:\n1) No wallclock time. Sure- you might save some compute by not computing some layers. But you are also using more compute to decide when to stop running the layers. So what is the total running time? Providing wallclock times is the only way to show if these methods are actually efficient or not.\n2) Can this method be used in batched inference? Is there any downside to that?\n3) Table 2 should have comparisons with the full (no early-exit) model baseline and with models from previous papers. It's not clear where you stand in comparison to the prior literature. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5rsjfAuGY-y", "reviewer": "Reviewer_Jyd3", "summary": "**The research problem:** The authors study the problem of accelerating transformer-based autoregressive inference from the perspective of early-exit strategies - approaches that terminate the forward pass before the final layer is reached based on whether a confidence criterion is satisfied. The primary goal of the proposed framework, CALM, is to find a method that allows for exiting the forward passes as early as possible while guaranteeing performance bounds on the consistency between the full model and early-exit model. \n\n**Analysis and proposed method:** After analyzing the sources of error that arise when running early-exit models, the authors propose a decaying confidence threshold for early exit that allow for earlier exits closer to the end of sequences. \n\n**Theoretical guarantees on performance:** The authors use tools from multiple hypothesis testing literature to pick confidence thresholds that provably satisfy performance bounds. \n\n**Empirical Justification:** The proposed method is shown to (with some caveats) result in potential speedup of up to 3x. \n\n**POST REBUTTAL UPDATE:** I thank the authors for their responses. I've raised my score. \n", "questions": "* Could you elaborate on the meaning of the result reported on line 166? Am I right in thinking that if one uses only the first layer representations to attend to during decoding (as if all the previous tokens exited after the first layer), the performance basically doesn't take a hit? This seems highly counter-intuitive? \n* Does it ever happen that the early-exit prediction matches that of the full model at layer i < L, then starts disagreeing again at layer j where  i < j < L?\n* Line 201: \"Perfect local consistency implies global consistency\". Could you clarify what is meant here? \n\n\n\nNitpick: \n* $\\mathcal{Y}$ doesn't seem to be defined anywhere. \n* Line 194: \"Our goal\" instead of \"are goal\"", "limitations": "The authors are upfront about the computational cost of using the softmax-based confidence measure (the best performing metric explored in the paper), which could potentially surpass the gains of the proposed method.\n\n", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**STRENGTHS**\n* Provable performance bounds is appealing: The distribution free approach to proving performance bounds is appealing. The fact that this approach doesn't yield too-conservative bounds is promising. \n* Approach considers local and global performance metrics: The proposed method considers the deterioration of quality both locally and globally when one employs early-exit strategies. \n* Error analysis provides insight about transformer inner-workings: Experiments in Section 3.3 are quite interesting - that one can get away with using only 1.53 layers per forward pass on average in a 8-layer transformer while (almost) not incurring a significant performance deterioration is surprising. \n* No labelled data is needed: The proposed method, when using textual consistency, doesn't require any labelled data. \n* Comparison of different confidence measures provides insight into early-exit strategies: The comparisons between softmax, state and classifier based early-exit confidence measures is quite interesting \n* Clear writing: The paper is easy to follow and well written, with very few typos. \n\n\n**WEAKNESSES**\n* Comparison to baselines: While the contribution of this work is to facilitate early-exit while providing performance guarantees, it'd still be interesting to see comparisons with baselines to see how much (if any) cost one has to pay to obtain the provable guarantees. \n* Computational cost of some confidence measures: The best performing confidence measure (softmax-based one) is computationally heavy, potentially erasing the efficiency gains of early exit. (The authors do note that further parallelization can alleviate this issue, and other confidence measures also bring nontrivial improvements in efficiency)\n* Practical benefit of performance guarantees in early-exit strategies: I believe the authors could spend some time justifying why one would be interested in provable guarantees on early-exit performance deterioration. What are some applications where this type of guarantee is desired/required? We already don't expect to have performance guarantees on the full model - what's the marginal benefit of using an early exit strategy that have theoretical guarantees instead of simply doing empirical validation to pick $\\lambda$? This will probably be especially true in cases where full model (i.e. model without early-exit) performance is sub-optimal which will be the case for many difficult NLP tasks with even today's strongest models). (It's possible I'm missing something here). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1LvNcSCS_pl", "reviewer": "Reviewer_c1DX", "summary": "This paper introduces the \"Confident Adaptive Language Modeling\" (CALM) framework, which contains very detailed study about the source of errors of doing early exiting during the decoding process of Transformer based language model, and proposes a method to calibrate well about local and global predictions to overcome these errors. Several empirical results, on text summarization, machine translation and Squad demonstrated the usefulness of the proposed method.\n\n", "questions": "1. Would adding more softmax operators for each layer, for example listed in Equation (6), trigger more computational bottleneck? Understand that seemingly the trade-off here would be good since we can ignore a lot of full computations for the subsequent layers if we can early exit, but still Eqn (6) seems introducing the extra softmax operators for all the layers and all the timesteps, during training, and it would be good if there could be some ablation study here demonstrating the ROI;\n\n2. It is definitely technically sound that the LTT framework is used. But this seems to be a double-sided sword: to determine the \\lambda value in Eqn (5), is it really necessary to adopt such a complicated framework? Note supposedly doing complicated tech works might hurt the generalization of the proposed method in the real world application scenario. What if this problem is simply treated as a hyper-parameter selection one and use much simpler method to do so (e.g., some random search and early stopping methods used in AutoML)? ", "limitations": "See above. N/A for the \"negative societal impact of their work\" part.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper is technically very rigorous. It tackles the problem in a step-by-step manner with clear very sound logic chain. Also, it is not a common practice now to pay attention to the statistical testing criteria and this paper does a good job here, by adopting a principled way to address that.  The writing, including the mathematical notation system, is also very accurate. \n\n- The technical works, especially in section 3.3,  are very detailed and (hence) insightful. For example, the headroom analysis of state propagation and local errors are comprehensive and convincing. \n\nWeakness:\n\n- See the below question 2 in the \"Questions\" part: it seems good to discuss whether the technical proposal in section 4, to solve this problem, is really necessary to be so complicated. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "0TTJt5_zSeZ", "reviewer": "Reviewer_KxKz", "summary": "This paper proposes Confident Adaptive Language Modeling (CALM) to early-exit dynamically during the decoding process. The authors provide a theoretical guarantee that local confidence measures could lead to the confident generation of global outputs.", "questions": "It is surprising that the oracle early exiting only uses less than 2 layers on average. How is it possible? \n\nThere is still a large room between the oracle and the proposed methods. What are the potential directions to reduce this gap? Which assumptions can be the most easily relaxed for better efficiency?\n\nI wonder whether the observed patterns in Figure 2 will repeat on any tasks.\n\nCould you explain more about why the softmax-based measure gives the highest speedup compared to other measures?\n", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "CALM is practically useful to speed up the inference where autoregressive decoding with large language models is quite expensive. The method is principled based on the learn-then-test framework.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept