Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding
Authors: Sangmin Bae
Jongwoo Ko
Hwanjun Song
Se-Young Yun
Venue: EMNLP 2023 Main (2023)
Keywords: Efficient Decoding of Language Model
Early-Exiting Framework
Parallel Decoding
Adaptive Confidence Estimation

Abstract: To tackle the high inference latency exhibited by autoregressive language models, previous studies have proposed an early-exiting framework that allocates adaptive computation paths for each token based on the complexity of generating the subsequent token. However, we observed several shortcomings, including performance degradation caused by a state copying mechanism or numerous exit paths, and sensitivity to exit confidence thresholds. Consequently, we propose a Fast and Robust Early-Exiting (FREE) framework, which incorporates a shallow-deep module and a synchronized parallel decoding. Our framework enables faster inference by synchronizing the decoding process of the current token with previously stacked early-exited tokens. Furthermore, as parallel decoding allows us to observe predictions from both shallow and deep models, we present a novel adaptive threshold estimator that exploits a Beta mixture model to determine suitable confidence thresholds. We empirically demonstrated the superiority of our proposed framework on extensive generation tasks.

Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decoding
Sangmin Bae Jongwoo Ko** Hwanjun Song"1 2f
Se-Young Yun
KKIIST AI AAWS AI
(bsmn0223, jongwoo ko, yunseyoung kr
hwanjuns@amazon. com
https: https://github.coomm http:://ggttuu.cooo http:///t.r.....oooo////o/oaasss
Abstract
nificant computational burdens and hinders their real-time adaptability (Jiao al., 2020).
To tackle the high inference latency exhibited by autoregressive language models, previous studies have proposed early-exiting frame- work that allocates adaptive computation paths for each token based on the complexity gen- erating the subsequent token. However, observed several shortcomings, including per- formance degradation caused by state copy- ing mechanism or numerous exit paths, and sensitivity to exit confidence thresholds. Con- sequently, we propose a Fast and Robust Early- Exiting (FREE) framework, which incorpo- rates a shallow-deep module and a synchro- nized parallel decoding. Our framework ables faster inference by synchronizing the de- coding process the current token with previ- ously stacked early-exited tokens. Furthermore. as parallel decoding allows us to observe pre- dictions from both shallow and deep models, we present novel adaptive threshold estimator that exploits a Beta mixture model to determine suitable confidence thresholds. We empirically demonstrated the superiority of our proposed framework on extensive generation tasks.
In light of the necessity to expedite inference la- tency, the early-exiting framework (Elbayad et al.. 2020; Liu et al., 2021; Schuster et al., 2022) emerges as promising approach that dynamically allocates computation paths based on the complex- ity of generation for each token. As illustrated in Figure la, tokens that are relatively easy to predict the subsequent token yield consistent predictions with only few layer computations, while those with higher difficulty require computations across a larger number of layers to generate accurate predic- tions. In an ideal scenario, the early-exiting method empowers models to achieve notable acceleration in inference without compromising the generation quality when compared to that of a full model.
However, our extensive analysis identified four challenges in the carly-exiting framework. Firstly, despite the potential exit earlier layers, key and value states for remaining layers are still quired for processing subsequent tokens. While previous works have proposed the state copying mechanism (Elbayad et al., 2020; Schuster et al., 2022) to efficiently compute these states by reusing hidden states from the early-exited layer, our find- ings reveal that this method performs poorly with larger models and longer output sequences (see Sec- tion 4.1). Additionally, setting all layers as possible exit positions does not guarantee faster inference due to (1) the defective performance of earlier lay- ers that can generate abnormally long sequence outputs, and (2) the computational overhead from confidence measurement at every layer (see Sec- tion 4.2 and 4.3). Lastly, achieving the desired level of latency and accuracy with carly-exiting heavily depends on selecting the appropriate con- fidence threshold for the target task. This often entails significant efforts and additional computa- tional overhead (see Section 4.4). Hence, these challenges call for a new approach that consistently
1 Introduction
Recent advancements in autoregressive language models (Brown et al., 2020; Raffel et al., 2020; Hoffmann et al., 2022; Touvron et al., 2023) have revolutionized the quality of language generation in various generative tasks, including question answering (Rajpurkar& et al., 2016a), summariza- tion (Nallapati et al., 2016; Fabbri et al., 2019b), and machine translation (Cettolo al., 2017a). Nevertheless, these large transformer models have shown high inference latency due to the consider- able number of layers and the autoregressive de- coding step. As the multiple stacks of transformer layers have to be computed sequentially for each individual token, the inference process poses sig-
cqual contribution
corresponding authors

---

Earth
Callbration Set (D,)
Layer
Layer I. (deep)
Pred (deepo:
Farth
musnndl
hg
B
by
hy
(shallew):
Conf ootalle)
0.85
0.95
DLI
hi
hg
bj
hi
h2
Layer
Layer (shallon)
Deta Mixture Model
Layer
Layer
Earth
round
(a) Conventional Sarly-Exitinng
(b) Fast and Robust Early-Exiting (FREE)
Figure I: Overview our FREE framework compared to the conventional early-exiting framework. FREE exhibits three key differences: (1) FREE employs shallow-decy module that utilizes two exit points instead of employing all layers as exit points, (2) FREE replaces the state copying mechanism (yellow colored) with synchronized parallel decoding (red colored) to prevent performance degradation while accelerating inference speed, and (3) FREE utilizes adaptive threshold estimator to determine the appropriate threshold values for each dataset during inference.
demonstrates high performance and low latency across diverse language models and datasets.
Related Work
2.1 Early-exiting Framework
In this paper, we introduce a Fast and Robust Marly-Exiting (FREE) framework that incorpo- rates a shallow-deep module and synchronized par- allel decoding. Our framework not only offers con- sistent speedup and performance even for larger models and longer output sequences, but also elim- inates the need for the computationally expensive process of finding the appropriate exit threshold. Specifically, the shallow-dedt deep module bifurcates the computation paths into shallow model (with specified number of carly layers) and deep model (including all layers). Our synchronized parallel decoding accumulates consecutive early-exited to- kens that only pass through the shallow model until a non-cxiting token encountered. Thereby, we synchronize the decoding process of the current non-exiting token with the previously stacked to- kens, shown in the left of Figure 1b. This pre- vents performance degradation by utilizing actual attention computed key and value instead of approx- imated states through state copying, while it also achieves more efficient approach compared to de- coding each token autoregressively. Furthermore, we devise novel adaptive threshold estimator, as shown in the right of Figure 1,, by leveraging the fact that parallel decoding outputs predictions even for early-exited tokens from the deep model. This estimator uses Beta mixture model (BMM) cap- ture the correlation between confidence scores and prediction alignment of two models, determining the proper confidence threshold for each dataset. practice, we demonstrate the efficiency of our FREE framework on extensive generation tasks.
As the size of language models has significantly increased, there have been numerous efforts to de- velop efficient decoding methods that reduce the computational cost of language generation tasks. Motivated by prior literature (Téerapittayanon et al., 2016; Graves, 2016; Zhang et al., 201 20)9,), Elbayad et al. (2020) introduced an early-exiting framework for faster inference, which dynamically adjusts the depth of the decoder for each token generation by making predictions at an intermediate layer. To achieve the better trade-off between speed and ac- curacy, Schuster et al. (2022) recently explored confidence thresholding methodologies, including various confidence measures, decaying threshold function, and calibration method.
However, their experiments were primarily con- ducted on small-sized decoder models, necessitat- ing further validation on larger models. In addition, their approaches require additional training time for statistical tests on the extra calibration sets, which prevents them from real deployment scenarios.
2.2 Parallel Decoding
The non-autoregressive decoding, which gener- ates multiple output tokens in parallel, was ini- tially proposed by Gu et al. (2018). Several works (Ghazvininejad al., 2019; Gu and Kong, 2021; Savinov et al., 2022: Santilli al., 2023) have since focused enhancing generation quality in machine translation tasks. Subsequently, Leviathan et al. (2023) introduced speculative decoding for sequence generation tasks. this approach, an

---

approximation model (small size) predicts outputs autoregressively, while target model (large size) runs in parallel to verify the acceptance of predic- tions made by the approximation model. With only accepted tokens, they resample the next token from an adjusted distribution. Related approaches have been proposed by Chen et al. (2023) and Kim et al. (2023), where they also utilize two models of vary- ing depth and focus on refining the small model through speculative sampling or a rollback policy in a -oo-eestorerrsssvve manner.
Table I: Comparison of ROUGE-L scores between a full model, fine-tuned using all layer outputs, and oracle- exiting. We also measured cosine similarity between hidden states of the last layer and oracle-exited layer.
Dataset
Model
Full M.
Oracle
Sim.
T3-small
44.84
44.17 (-0.67)
0.913
SAMSum
T3-large
48.82
47.58 (-1.24)
0.809
T3-small
37.82
37.60 (-0.22)
0.902
CNN/DM
T5-large
41.15
40.15 (-1.00)
0.792
Multi-News
LongTS-buse
37.62
29.63 (-7.99)
0.724
BIGPATENT
LongTTS-hase
49.68
44,99 (-4.69)
0.686
Our approach notably distinct from the afore- mentioned works as we focus on early-exiting framework by introducing synchronized parallel de- coding within single network, which incorporates a shallow-deep module. While we also leverage the advantage of simultaneously obtaining predictions from models of different depth, we rather aim to de- velop a novel and effective estimation methodology to adaptively determine the optimal threshold for each dataset. is worth noting that their refining strategies may result in unbounded latency increase as they restart from incorrect predictions.
Afer the current token is carly-cxited at the gth layer, we need to calculate the key and value states for all deeper blocks in order to perform the self-attention for the subsequent tokens that pass through deeper blocks. For more efficient approach of caching key and value states, the early- exiting frameworks employ the state copying mech- anism. duplicates the hidden states of the early- exited layer (i.e., hi+1 = h Vi E [ 1, L)). allowing us to compute the approximate key and value states required for the self-attention of Trans- former networks. Schuster et al. (2022) have veri- fied that state copying from lower layers does not have detrimental effect on performance in the case of small-sized T5 models (Raffel et al., 2020).
3 Preliminary
A Transformer network (Vaswani et al., 2017) is composed of L layers, where each layer consists of two sublayers, a multi-head attention (MHA) layer and feed-forward network (FFN) layer. The computation for hidden states at time step L 4+1 1 via stacked Transformer blocks is as follows:
Re-evaluating Early-exit Framework
In this section, we present four new findings from our re-evaluation of the early-existing framework. We utilized different model sizes of T5 (Raf- fel et al., 2020) on SAMSum (Gliwa et al., 2019) and CNN/DailyMail (See et al., 2017), and LongT5-base (Guo et al., 2022) architectures on Multi-News (Fabbri et al., 2019a) and BIG- PATENT (Sharma et al., 2019).
b (hans (1), E [],
where hf41 is the embedding layer outputs of yt that represents the generated token at time step t. After th layer of the decoder network, the pre-
dicted token i1+1, is determined by the probability output from a softmax classifier Wt:
p(U--1 1 bf 1 softmax (W]. (.(..1)))
4.1
Lack of Robustness to Model Size and Output Sequence Length
However, unlike the standard LMs, the early- exiting framework enables the generation of a subsequent token in earlier layers by using P( h(11). .. If the confidence score of is larger than the predefined threshold, we can make a pre- diction at time step t + 1 as arg max p(v): 1 (b/+1). While classifiers can be parameterized indepen- dently or shared across the L layers, most early- exiting methods (Elbayad al., 2020; Liu al., 2021; Schuster et al., 2022) utilize the shared clas- sifier due to its large number of parameters caused by enormous vocabulary size.
We first re-evaluate the state copying mechanism which is an essential component of the early-exiting framework. Following Schuster et al. (2022), we use an oracle confidence measure that enables to- kens to exit at the earliest layer, such that their pre- dictions are identical to those of the final layer. No- tably, as observed in Table 1, the degradation of the generation quality with the state copying gets se- vere on larger models and datasets with the longer sequence (D> Obs. 1). For instance, when consider- ing the oracle-exiting results, the Tm-small model

---

demonstrates the degradation of only 0.67 on the SAMSum dataset, whereas the T5-large model ex- periences a much larger drop of 1.24. Similarly,
ROUGE-L G Gen, Length
50
100
ROUGE-LO
mntto 1009
on datasets such as Multi-News and BIGPATENT, which consist of relatively long output sequences, the oracle-exiting results exhibit decrease of 7.99 and 4.69, respectively.
50
20
25
10-
To strengthen the supporting evidence, we fur- ther discover the substantial variation in the distri- bution of hidden states across different layers. In Table 1, we also reported cosine similarity between the hidden states of the final layer and the oracle- exited layer. Even though the hidden states of the final and oracle-exited layers yield the same predic- tions, the cosine similarity between them decreases significantly the decoder network gets larger and the output sequences become longer.
6 18 24
Layer Index
Layer Index
Illustration of the ROUGE-L scores and gen- erated sequence length from the static-exiting approach in T5-small (left) and T5-large (right) the SAMSum dataset. The horizontal dashed line represents the aver- age sequence length of the ground truth.
Figure 2:
[SA] KV
[CA] KV
FFN
[SA] Alin
[CA] Aun
Conf
47.8
Normalized Latency
48
46.8
45.0
40.7
40.2
37.6
37.4
36.9
4.2 Performance Drop by Exit Position
1.0
39.
To facilitate carly-exiting for all decoder layers, the
training objectives need be a combination of the training objectives for each individual layer. We can present as follows:
SAMSum
CNN/DM
Multi-News
Σ agLi where Σa=
(1)
Figure 3: Component computational cost on three datasets. Four bars correspond to full model and early- exiting with thresholds of 0.9, 0.7, and 0.5. The hatched color denotes the elapsed time after the token exits, related to the state copying mechanism. The numbers above the bars represent the ROUGE-L scores. SA and CA denote self- and cross-attention, respectively.
C; and 0; is negative log- ligllikelhood loss function and weight coefficient for 1th layer, respectively. Es- pecially, previous work set as 1/L (unweighted average; Elbayad et al. 2020) or i/ >; i (weighted average; Schuster al. 2022). They demonstrated that these weighting rules effectively facilitate learning in earlier layers without compromising the overall performance the full model on small- sized decoder models.
4.3 Non-negligible Computational Cost
During our analysis, we observed that the conven- tional early-exiting framework not only presents performance disadvantages but also poses chal- lenges for inference latency. In Figure 3, we con- ducted breakdown of the computational costs associated with decoder model across three sum- marization datasets. Surprisingly, early-exiting has often shown an unexpected increase in total de- coding time when compared to the baseline model without using carly-cxiting.
However, as shown in Figure 2, we observed a notable decrease in the performance of static- exiting, which utilizes the same number of layers for all tokens, when utilizing only a small por- tion of the early layers from the T5-large model. (D- Obs. 2). For instance, if all tokens are exited in the first or second layers, the model achieved nearly zero ROUGE-L scores. Furthermore, when we apply the early-exiting framework to these mod- els during inference, we verified that the Ta-large model generates abnormally long sentences, actu- ally consuming more inference time. Based on these results, in the subsequent experiments, have excluded the first two or four layers from the candidates for early-exiting layers of base and large models, respectively.
This can be attributed to the non-negligible com- putational cost involved in measuring confidence at each layer, particularly due to the softmax oper- ations with the large vocabulary size. In addition, although the state copying method aims to reduce computation time in the MHA and FFN layers of the remaining layers, the computation of key and value states using duplicated hidden states incurs additional non non-negligible overhead (> Obs. 3).

---

Table 2: The optimal confidence threshold to achieve desired performance. We chose the best values among threshold value from to 1 with step size of 0.1. The numbers sequentially represent the selected threshold and corresponding performance (gray colored).
t=
deep model
Artificial
shallow model
(=2
Artificial
intelligence
=
<> Artificial intelligence
t=
Artificial intelligence the
t=5
Artificial intelligence the nne
Performance Drop
Artificial intelligence the new celetriciity t=6
Task
Dataset
11%
~5%
10%
t=7
Artificial
intelligence
the new electricity
SAMSum
1.0 (48.8)
0.7 (46.8)
0.5 (45.0)
Figure 4: Overview of synchronized parallel decoding. We colored the tokens used to generate the next token based on the model that they forward.
CNN/DM
1.0 (41.2)
0,5 (39.2)
0.3 (37.3)
SUM
Multi-News
0.8 (37.3)
0.5 (35.9)
0.4 (34.9)
BIGPATENT
1.0 (49.7)
0.8 (47.3)
0.6 (45.2)
QA
SQuAD
0.1 (90.1)
0.0 (88.3)
0.0 (88.3)
co-training numerous exiting layers in the conven- tional early-exiting framework.
MT
IWSLT
1.0 (39.4)
1.0 (39.4)
1.0 (39.4)
To enhance the performance of the shallow model, we exploit layerwise knowledge distilla- tion (KD) as an additive loss term to Eq. (1) with OL,, LTa((I L8) and az L/(L Lg):
4.4 Disparate Optimal Confidence Threshold
Determining the appropriate threshold for exit con- fidence is a crucial challenge in the early-exiting framework as it directly impacts the orf between performance and latency (Zhang et al., 2019b; Schuster et al., 2022). As summarized in Table 2, our observations indicate that the opti- mal confidence thresholds for achieving the low- est latency in the same performance significantly vary across datasets (> Obs. 4). For instance, SQuAD and CNNi datasets can maintain performance with relatively lower exit thresholds, whereas higher threshold values are required in the case of the IWSLT dataset. Previous work (Schus- ter al., 2022) has leveraged distribution-free risk control techniques for confident generations. How- ever, these methods require additional training time for statistical tests on the extra calibration set be- fore the deployment, where time can be also influ- enced by the size of the threshold candidate sets.
Lg
( H
LKD
L
where m(i) indicates the layer in the deep model that extracts knowledge into the corresponding layer of the shallow model. Hş and HD are hidden states from shallow and deep models.
We have experimented with the distillation from the last layer (KD-last; Wang et al. 2020; Ko et al. 2023), from fixed uniform mapped layers (KD-unif; Jiao al. 2020; Park et al. 2021), and from dynam- ically mapped layers (KD-dyna; Xia al. 2022). Especially, dynamic mapping function allows us to align each deep model layer with its closest coun- terpart in the shallow model:
m(i) = arg min MSE HD)
5 Novel Early-Exiting Framework: FREE
where j denotes the layer indices of the deep model selected by the total number of Ls. and the condi- tion of m(1) < Based on the consistently superior performance of
Building upon the discoveries in Section 4, we in- troduce Fast and Robust Early-Exiting framework named FREE, leveraging shallow-deep module and capitalizing on the structure of parallel decod- ing. Furthermore, we present confidence estima- tion algorithm designed to enhance the robustness of early-exiting within the FREE framework.
m(Ls) should be satisfied.
KD-dyna loss (see Appendix D.2), we utilized it for all experiments with the shallow-deep module.
5.2 Synchronized Parallel Decoding
We present synchronized parallel decoding as an al- ternative to the state copying mechanism, which is key component of the conventional carly-exiting framework but can lead to significant perfor- mance decrease, as demonstrated in Section 4.1. In contrast to traditional approaches that have multiple exit points, our method incorporates the shallow- deep module, enabling us to stack consecutive
5.1 Shallow-Deep Module
We present an effective shallow-deep module, which strategically assigns a predetermined num- ber of early layers (Isg) as shallow model, while all the layers as a deep model. This module tack- les the performance degradation associated with

---

early-exited tokens in the shallow model until non-exiting token encountered. When decoding the token with the deep model, we enhance eff- ciency and effectiveness through parallel decoding, synchronously computing the key and value states of previously stacked tokens. The example of the parallel decoding process is depicted in Figure 4.
Algorithm 1 Adaptive Threshold Estimation
Input: empty calibration dataset De, initial confi- dence threshold Xo, posterior condition Ç, update number T
Output: updated confidence threshold Xe
initialize <- 0, de - 20
2: while t - T do
The underlying principle of this approach is to leverage the enhanced parallelism offered by mod- ern hardware accelerators. This allows for efficient computations to be carried out simultancously on the large number of sequences. Thus, by employ- ing synchronized parallel decoding, we can directly compute multiple hidden states similar to a single token processing time. Besides, this can eliminate the potential performance degradation that may arise from inaccurate approximations of hidden states resulting from the state copying mechanism.
3: Generate tth sentence with N₃ tokens
4:
10 Update De %/
De - De u (cto 1(ot- (f)
5:
1e Find threshold with (2))2)
Ok. B& MLEBMM for (0,1)
Ae a min ( A
update <- t +
10: end while
otherwise. Similarly, aş is the standard deviation of confidence of related k.
5.3 Adaptive Threshold Estimation
Ck ΣN not E111 13 88
EE 7 (c -2,) 111 7i
(3)
We propose novel adaptive threshold estimation method that updates the threshold be retailed for different datasets. Unlike the previous methods that utilize extra calibration sets (Schuster al., 2022), we quickly adapt the threshold by using the information of early-stage instances, regardless of the initial threshold values. Especially, during parallel decoding, we collect samples to evaluate the correspondence between the confidence scores of the shallow model and the prediction alignment between shallow and deep models.
where 7/2 = 1( = ú denote whether the pre-
diction of two models are same.
After updating the BMM, we find an appropriate
threshold for future tokens by identifying the point at which the posterior probability, defined as below, reaches Ç:
p(k=1)p(ddo p(k = 1|Xc) Exc(o.1) pik paaaa)p(k--ppplplloop 3)p(Aelay,33)
(4)
Here, as we observe the severe imbalance between the case of of each class to 0.5 for the balance between two cases (i.e., p(k (k - j) 0.5 Vj). As this restriction makes us to use a smaller value of Ç, we naïvely set it as 0.4. A detailed algorithm can be found in Algorithm 1.
and 1. we restrict the prior value
As depicted in Figure 1b, we observe that when the predictions of the deep and shallow models are identical, the confidence tends to skew towards one, otherwise it skews towards zero. To model this skewed distribution over [0, 1]. we utilize a beta mixture model (BMM; Ma and Leijon 2011 due to its flexibility and the appropriate support set of the beta distribution. The probability density function of beta distribution over [0, 1] is defined as:
Experiments
6.1 Experimental Setup
We conducted experiments on various sequence modeling tasks, including question answering (SQuAD; Rajpurkar et al. 2016b), machine transla- tion (IWSLT 2017 En-De; Cettolo et al. 2017b), and text summarization tasks using SAMSum, CNN/DailyMail, Multi-News, and BIGPATENT datasets. The L5 base model was used for the Multi-News and BIGPATENT datasets, while the T5-large model was used for the other datasets. All implementations are based on PyTorch using Huggingface (Wolf et al., 2020; Lhoest et al., 2021). Further details can be found in Appendix B.
r(a + 3) a 1(1 =1-=)) p B) (3)
The parameters of the BMM are updated using the maximum likelihood estimator (MLE; Norden 1972) with observed data points.
0g = Cl [c((1 (0((1-)) -1) .3 (1-
(2)
2x
where Cp being average of the confidence
(cF H=1 for corresponding k. is set to the
predictions of the two models are identical, and 0

---

Static-oxit
CALM
FREE
FREE
50
40
50
ROUGE-L (T)
35
40-
40
30
0.2
0.4
0.6
1.0
0.2
0.4
0.66
0.8
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Normalized Latency
Normalized Latency
Normalized Latency
Normalized Latency
(a) SAMSum
(b) CNNiDailyMail
(c) Multi-News
(d) BIGPATENT
The trade-off between the generated output quality and normalized latency under different exit conditions. We varied the exit threshold values between 0 and 1 for both CALM and FREE and the number of exit layers for the static-exiting framework. We exclude the inner point of the Pareto curve, and the dashed line represents the ROUGE-L score the full model, which is the fine-tured shallow-deep module.
Figure 5:
Table 3: Comparison between carly-exiting frameworks various datasets. For CALM and FREET, reported the performance using the smallest threshold value that achieves 99% performance of the full model, fine-tuned by weighted average or KD-dyna losses, respectively. The parentheses denote relative speedup based the first row.
SUM
QA
MT
Method
SAMSum
CNN/DailyMail
Multi-News
BIGPATENT
SQuAD
IWSITT
Full Model
48.82 (x (.00)
41.15 1.00)
37.62 (x 1.00)
49.68 1.00)
90.63 (x 1.00)
39.19 (x 1.00)
CALM
48.37 (x0.72)
40.78 (x0.86)
37.27 (x0.85)
49.21 (x0.65)
90.09 (x2.03)
39.19 1.00)
Full Model
49.11 (x 1.00)
41.09 (x 1.00)
39.20 (x 1.00)
49.68 (x 1.00)
91.90 (x 1.00)
39.39 (x 1.00)
FREE
48.65 (x 1.50))
40.89 (x 1.80)
38.93 (x 1.07)
49.51 (x 1.62)
91.31 (x2.76)
39.04 (x 1.07)
FREE
48. (x4)
40.99 (x 1.65)
38.66 (x 1,23)
49.47 1,58)
91.82 (x2.16)
38. 38.17 1.18)
6.2 Experimental Results
Statie-exit
CALM
FREE
* FREE
In order to investigate the effect of the individual component of our proposed framework, we evalu- ate both FREE without and with an adaptive thresh- old estimator, denoted as FREEt and FREE.
ROUGE-L(()
45
36
34
Overall performance. In Figure 5, we present a comparison of the quality of generated output (ROUGE-L) and the inference latency between the FREE framework and baselines, including static- exiting and the conventional early-exiting method (CALM: Schuster et al. 2022). CALM method exhibited poorer performance compared sim- ple static-exiting approach on datasets, likely due to the state copying mechanism and the pres- ence of numerous exit positions, as observed in Section 4. In contrast, FREE/ demonstrated robust performance and the larger AUC (area under the curve) across datasets by adjusting exit thresholds.
40
32
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Normalized Latency
Normalized Latency
(a) SAMSum
(b) CNN//aillMaii
Figure 6: The trade-off between the generated output quality and normalized latency on T5-3B models.
and training time, our methodology effectively ad- dresses this challenge by leveraging the byprod- uet of parallel decoding. As summarized in Ta- ble 3, FREE with adaptive threshold estimation successfully achieved significant speedup, by up to 2.16,, when preserving the 99% of full model performance. Furthermore, Figure 5, the esti- mated threshold demonstrated nearly the maximum achievable speed improvement without sacrificing performance, represented by red stars.
Adaptive threshold evaluation. In the carly- exiting framework, choosing the appropriate con- fidence threshold is crucial for achieving the best trade-off between generation quality and latency. Unlike previous calibration methods (Schuster et al., 2022) that require an extra calibration set
Large language models. Recently, various stud- ies (Dettmers et al., 2022; Xiao et al., 2023; Leviathan et al., 2023; Liu et al., 2023b) have

---

Table 4: The comparison of ROUGE-L and speedup based on different numbers of layers for shallow model and confidence thresholds.
Table 5: The comparison between synchronized parallel decoding (SPC) and state copying (SC). The shallow- deep module is utilized in both decoding methods.
Threshold
Method
Threshold
Datasel
Ls
0.7
0.5
0.3
Dataset
SC
SPD
0.9
0.7
0.5
0.3
0.1
4
48.27 (x 1.04)
(x.95(
44.72 (x 1.15)
46.35
44.59
43.92
42.36
41.27
SAMSum
6
1x.89(xx1.32
48.65 (x (x1)50)
47.60 (x1)))
X
48.89
48,89
48.65
47.60
45.27
SAMSum
8
(x.11)
(x
47.09 (x 1.31)
40.92
40.92
40.71
39.99
38.17
CNN/DM
12
4x.97
(x.74
48.10 (x.37)
/
41.12
41.08
41.00
40.60
39,30
4
40,59 41.033(xx1445)
(68)
39.88 (x 1.86)
38.43
37.61
36.55
33.99
29.34
Newsi-News
41.08
(x2007) (x..9
39.16
39.06
38.78
37.87
33.98
CNN/DM
8
4x.15 (x.44)
(.64)
40.95 (x 1.69)
12
41.11 (x1.33)
(x
40.95 (x 1.55)
Table 6: The experimental results of FREE framework based on different sizes of the calibration set.
aimed at boosting the inference speed of large language models (LLMs). To validate the ap- plicability of the FREE framework on LLMs. we conducted experiments utilizing the T5-3B model (Raffel et al., 2020) on the SAMSum and CNN/DailyMail datasets. Due to substantial com- putational overhead, we utilized the LoRA adapter (Hu et al., 2022), targeting both self-attention and feed-forward layers with a rank of 64. Figure summarized a comprehensive comparison of early- exiting methods. Our method maintained superi- ority over the baselines in terms of latency and ROUGE-L scores, showing the consistent perfor- mance trend observed in the T5-large model. Thus, we are confident that our proposed framework would demonstrate consistent level of inference acceleration, even with larger language models.
10%
100%
Datasct
Thr.
Porf.
Speed
Thr.
Perf.
Speed
Thr.
SAMSum
0.51
48.66
x1.47
0.49
48.69
x x1.51
0.48
BIGPATENT
0.54
49.47
x1.58
0.54
49.39
x 1.63
0.54
module. Synchronized parallel decoding consis- tently outperformed state copying across all three datasets by much higher ROUGE-L metrics, as summarized in Table 5. This improvement can be attributed to the updated hidden states that are ob- tained through the accurate computation of Trans- former layers during parallel decoding. These find- ings suggest that our efficient decoding method for carly-cxited tokens can enhance the overall perfor- mance of the early-exiting framework as well.
Dependency on size of calibration set. By using the early-stage instances the calibration set, we iteratively update the adaptive confidence threshold to converge to the appropriate value. Here, we have observed the sample efficiency of the adaptive threshold estimator by varying the sizes of this calibration set. Interestingly, even with only 3% of the total samples, our estimator can approximate the threshold, measured by the full sample set, shown in Table 6. This ensures minimal additional computation time required for threshold estimation.
6.3 Ablation Study
Different depth of shallow model. In Table 4, we also ablate on the number of layers for the shallow model to observe the trade-offs. While our method demonstrated trend towards higher speedup gains the depth of the shallow model decreases, we experienced some decreases in per- formance and speed gain when the depth of the model is reduced too much (e.g., four layers). We assumed that this due to incorrect and redundant output sentences, similarly observed in the conven- tional early- framework. Consequently, with enough depth (e.g., six layers), FREE consistently showed robust performance and inference speedup.
Refining shallow model predictions. Prior works (Leviathan et al., 2023; Chen et al., 2023; Kim et al., 2023) have proposed refinement meth- ods to correct erroneous outputs from an approxi- mation model. Specifically,, when wrong token is detected in previous sequences, they remove all subsequently generated tokens and restart the gen- eration process from that point. In Table 7, we conducted experiments in order to evaluate the ef- fects of this refinement method (Kim et al., 2023)
Robustness of parallel decoding. In order to verify the robustness of our decoding mechanism, we conducted a comparative analysis between syn- chronized parallel decoding (SPD) and state copy- ing (SC), both implemented with the shallow-deep

---

Table 7: The evaluation of refinement methods the FREE framework. Refining thresholds control the level of acceptance for predictions from the shallow model.
Evaluate the quality of summaries written for news article. Rate each summary on four dimensions: (Dimension 1J, [Dimension 2), (Dimension 3), and (Dimension__ 4). You should rate on a scale from (worst) to 5 (best).
Thr: 0.7
Thr:
Dataset
Ref.
Thr.
Perf.
Speed
Perf.
Speed
Article: (Article)
48.89
x1.33
x1.80
Summary: [Summary]
SAMSum
/
1.0
49.08
x 1.26
48.31
x 1150
/
0.1
49.06
x1.17
48.27
x1.12
Figure 7: The template for Likert scale scoring. The four dimensions are relevance, informativeness, fluency,
41.08
11.3
40.60
x2.07
CNN/DM
1.0
40.86
x x1.11
40.78
1.67
and coherence.
0.1
40.85
x 1.35
40.75
x 11.21
Given a new article, which summary is better? Answer "Summary 0" or "Summary I", You do not need to explain the reason.
in our early-exiting framework. We observed that when the refinement threshold is set low, allowing for more correction by the deep model, the per- formance improvement is minimal compared to the increase in latency. Our findings suggest that these approaches that cannot guarantee upper bound on latency increase may not be well-suited for integration into the early-exiting framework.
Article: (Article)
Summary O: (Summary 0)
Summary 1: (Summary
Figure 8: The template for pairwise comparison. We measured twice by changing the order of summaries for a fair comparison.
Human-like summarization evaluation. Recent studies (Gao et al., 2023; Liu et al., 2023a; Zhang et al., 2023) have argued that existing summariza- tion evaluation metrics like ROUGE-L do not ac- curately represent the true summarization capabil- ities. Instead, they explored the human-like eval- uation using LLMs based on their strong corre- lation with human judgment. Thereby, we con- ducted two human-like evaluation methods, Likert scale scoring and pairwise comparison (Gao et al., 2023), using ChatGPT API 1pt=33.--turbo-061 We compared full model and our FREE frame- work 100 instances, randomly drawn from the CNli dataset. Figure and 8 provide the templates used for each evaluation task. full model, we observed scores of [4.73, 3,83, 3.87, 3.77], while our FREE method returned scores of [4.68, 3.84, 3.84, 3.72] across the four dimensions. Besides, the win counts for each method were 101 and 99, respectively. Given ROUGE-L scores of 41.09 (x 1.00) for the full model and 40.99 (x 1.65) for the FREE method, method is certainly capa- ble of yielding predictions of similar quality, while notably reducing computational overhead.
and (3) adaptive threshold estimation. Through extensive experiments on various generation tasks, empirically demonstrated the superior perfor- mance of FREE framework, achieving significant acceleration in latency without compromising the quality of the generated output.
Our work addressed fast and ro- bust existing framework that can be efficiently uti- lized without concerns about performance degrada- tion. However, our approach does have few lim- itations which discuss below: (1) Our method requires additional computational resources fine- tune the shallow model. However, as we have demonstrated, parameter-efficient fine-tuning meth- ods would be a promising solution to overcome this limitation. (2) While our work demonstrates ro- bustness in the depth of the shallow model, further investigation is required determine the appropri- depth for various language models. This aspect remains an area for additional research.
Limitations.
This work was supported by Institute of Information & communications Tech- nology Planning & Evaluation (IITP) grant funded by Korea government (MSIT) [No. 2021-0-00907. Development of Adaptive and Lightweight Edge- Collaborative Analysis Technology for Enabling Proactively Immediate Response and Rapid Learn- ing, 90%] and [No. 2019-0-00075, Artificial Intelli- gence Graduate School Program (KAIST), 10%].
Acknowledgement.
7 Conclusion
We proposed FREE framework to address the chal- lenges of conventional early-exiting frameworks for autoregressive language models. Our approach incorporates three key components: (1) shallow- deep module, (2) synchronized parallel decoding,

---

References
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMMN.P-JJCNIP 2019, Hong Kong, China, November 3-7, 2019, pages 6111-6120 Association for Computational Linguisties.
Sangmin Bae, Sungayun Kim, Jongwoo Ko, Gihun Lee, Seungjong Noh, and Se- YouY Yun. 2021. Self-contrastive learning: Linglevieeee supervised contrastive framework using sub-network. arXiv preprint 21..iv::0004442 15499.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. 2019. SAMSum corpus: A human- annotated dialogue dataset for abstractive summa- rization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguis- tics.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Niehues Jan, Stüker Sehastian, Sudoh Katsuitho, Yoshino Koichiro, and Federmann Christian. 2017a. Overview of the iwslt 2017 evaluation campaign. In Proceedings of the 14th International Workshop on Spoken Language Translation, pages 2-14.
Alex Graves. 2016.
Adaptive computation time for recurrent neural networks. arXiv preprint arXiv:1603.088 0603338833.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. 201 7b. Overview of the IWSLT 2017 evaluation campaign. In Proceedings of the 14th International Conference on Spoken Language Translation, pages 2-14, Tokyo, Japan. International Workshop on Spoken Language Translation.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O.K. Li, and Richard Socher. 2018. Non-autoregressive neural machine translation. In International Confer- ence Learning Representations.
Jiatao Gu and Xiang Kong. 2021. Fully non- autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Com- putational Linguisties: ACL-IJCNLP 2021. pages 120-133, Online. Association for Computational Lin- guistics.
Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Baptistte Lespiau, Laurent Sifre, and John Jumper. 2023, Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.013: 01318
Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfci Yang. 2022. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 724- 736, Seattle. United States. Association for Compu- tational Linguistics.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zeutlemoyer. 2022. Llm, int8 O): 8-bit matrix mul- tiplication for transformers at scale. arXiv preprint arXiv:2208.03339.
Maha Elbayad. Jiatao Gu, Edouard Grave, and Michael Auli. 2020. Depth-adaptive transformer. 8th Inter- national Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Jordan Hoffmann. Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan. Eric Noland, Katherine Millican, George van den Driessche, Bog- dan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. 2022. An empirical analysis of compute optimall large language model training. In Advances in Neural Information Processing Sys- tems.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019a. Multi-news: A large-scale multi-document summarization dataset and abstrac- tive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 1084, Florence, Italy, Asso- ciation for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious of neural text de- generation. In International Conference on Learning Representations.
Alexander R Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir R Radev. 2019b. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. arXiv preprint arXiv: 7906.01749,
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Ship- ing Yang, and Xiaojun Wan. 2023. Human-like sum- marization evaluation with chatgpt. arXiv preprint trXiv:2304.0255.

---

Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.
Yuandong Tian, Christopher Ré, and Beidi Chen. 2023b. Deja vu: Contextual sparsity for efficient llms at inference time. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 22137-221766 PMIR.
TinyBERT:
Distilling BERT for natural language un- derstanding. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4163- guistics.
4174, Online. Association for Computational Lin-
Schoon Kim, Karttikeya Mangalam, Jitendra Malik, Michael W Mahoney. Amir Gholami, and Kurt Keutzer. 2023. Big little transformer decoder. arXiv preprint 23xih22302008866
Zhanyu Ma and Arne Leijon. 2011. Bayesian estimation of beta mixture models with variational inference. IEEE Transactions on Puttem Analysis and Machine Intelligence. 33(11)22160-2173
Jongwoo Ko, Seungjoon Park, Minchan Jeong, Sukjin Hong. Euijai Ahn, Du-Seong Chang. and Se- YouYo Yun. 2023. Revisiting intermediate layer distillation for compressing language models: An overfitting perspective. In Findings of the Association for Com- putational Linguistics: EACL 2023, pages 158-175, Dubrovnik, Croatia. Association for Computational Linguistics.
Ramesh Nallapati. Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to- to- sequence mns and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 280-290. ACL..
Yaniv Leviathan, Matan Kalman, and Yossi Matias.
RH Norden. 1972. A survey of maximum likelihood estimation. International Statistical Review/Revue Internationale de Statistique, pages 329-354.
2023. Fast inference from transformers via spee- ulative decoding. In International Conference on Machine Leurning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 19286.19286.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation, In Proceedings of the 40th Annual Meeting of the Association for Compu- National Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
PMLR.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagu- nas, Alexander M. Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, EMNLP 2021, Online and Punta Cana, Dominican Republic, 7-11 Novem- ber, 2021, pages 175-184. Association for Computa- tional Linguisties.
Geondo Park, Gyeongman Kim, and Eunho Yang. 2021. Distilling linguistic context for language model com- pression. In Proceedings of the 2027 Conference on Empirical Methods in Natural Language Processing. pages 364-378. Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin. Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Chin-Yew Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang. Ruochen Xu, and Chenguang Zhu. 2023a. Gpte- val: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv: 16634.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research, 21 21(1):5485-5551
Yijin Liu, Fandong Meng, Jie Zhou, Yufeng Chen, and Jinan Xu. 2021. Faster aepth-adaptive transformers. In Proceedings of the AAA/ Conference on Artificial Intelligence, volume 35. pages 13424-13432.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016a. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang,

---

Pranav Rajpurkar. Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguisties.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task task-agnostic compression of pre-trained transformers. Advances in Neural In- formation Processing Systems, 33:5776-5788.
Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodolà. 2023, Accelerating transformer inference for translation via parallel de- coding. Proceedings the 61 st Anmual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACI 2023, Toronio, Canada, July 9-14, 2023, pages 13336-13355 Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: ofate-of-the-- art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online, Association for Computational Linguistics.
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. 2022. Step- unrolled denoising autoencoders for text generation. In International Conference on Learning Representa- tions.
Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. Ad- vances in Neural Information Processing Systems, 15:17456-17472
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. 2022.
Structured pruning learns compact and accurate mod- els. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1-28. Dublin, Ireland. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073- 1083, Vancouver, Canada. Association for Computa- tional Linguisties.
Guangxuan Xiao, Ji Lin, Mickaël Seznec, Hao Wu, Julien Demouth, and Song Han. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 38087-38099 PMLR.
Eva Sharma, Chen Li, and Lu Wang. 2019. BIG- PATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 2204-2213, Florence, Italy. Asso- ciation for Computational Linguistics.
Linfeng Zhang, Jicbo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. 2019a. Be your own teacher: Improve the performance convolu- tional neural networks via self distillation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages 3713-3722.
Noam Shazeer and Mitchell Stern. 2018 Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference Machine Learning. pages 4596 4604. PMLR.
Linfeng Zhang, Zhanhong Tan, Jiebo Song. Jingwei Chen, Chenglong Bao, and Kaisheng Ma. 2019b. Scan: scalable neural networks framework towards compact and efficient models. Advances in Neural Information Processing Systems, 32.
Surat Teerapitaaanonn Bradley McDanel, and Hsiang- Tsung Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recogni- fion (ICPR), pages 2464-2469. IEEE
Xinghua Zhang. Bowen Yu, Haiyang Yu, Yangyu Lv. Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. 2023. Wider and deeper Ilm networks are fairer Iim evaluators. arXiv preprint arXiv:2308.0r. 01862.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive representation distillation. arXiv preprint arXiv: 10699.
Hugo Touvron, Thibaut Lavril. Gautier Izacard, Xavier Martinet, Mane Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1322 13971.

---

A Dataset Description
Table 8: Optimized hyperparameters for training shallow-deep T5 models. The column labeled '# Batch' indicates the product of the batch size per GPU and the number of GPUs. 'In len.' and "Outlen, represent the maximum length of the input and output, respectively.
We apply FREE on various generation tasks includ- ing summarization, question answering, and ma- chine translation. We provide detailed descriptions of the datasets used.
Model
Batch
Epochs
SAMSum (Summarization): SAMSum (Gliwa et al., 2019) consists of 16K messenger- con- versations that are annotated with summary for providing concise overview of the conver- sation's content in the third person.
SAMSum
TS-large
4x2
20
512
128
CNNDD
TS large
4x4
512
128
Multi-News
LongT5 L5ngTS-hase
2x2
2048
512
BIGPATENT
LongTS-base
2x2
2048
512
SQuAD
T5 Ta-large
4x2
512
30
IWSLT
m11-laage
CNN/Daily Mail (Summarization): CNN/ Dai- lyMail (See al., 2017) consists of over 300K English news articles that were originally de- signed for machine-reading and comprehension as well as abstractive question answering, but it now also supports extractive and abstractive summarization.
the training configuration in Table 8. For all dataset. AdaFactor (Shazeer and Stern, 2018 8) opti- mizer with the learning rate of le-4. For the adap- tive threshold estimation, we set the initial thresh- old value Xo as 0.9, as 0.4, as 3% of total sample number (refer to Algorithm 1).
Performance metrics. To numerically measure the output quality of our method, we utilize the Fl score for SQuAD, BLEU score (Papineni al., 2002) for IWSLT2017, and ROUGE score (Lin, 2004) for the four summarization tasks.
Multi-News (Summarization): Multi-News (Fabbri et al., 2019a) comprises 45K news arti- cles and corresponding summaries, where each summary is professionally crafted and provides links the original articles referenced.
C Inference Latency Evaluation
BIGPATENT (Summarization): BIGPATENT
(Sharma al., 2019) contains 1.3M records of U.S. patent documents, each accompanied by abstractive summaries written by humans. In our work, we specifically focus on the Fixed Constructions category, which is one of the nine classification categories available in the dataset.
For measuring inference speed, we execute 500 inference predictions for each dataset under each examined configuration in PyTTrrch (Paszke et al., 2019) compiled function in a single server with a single NVIDIA GeForce RTX 3039 GPU and 12th Gen Intel(R) Core(TM) 17-12700K CPU. For each inference prediction, we use batch size 1, which is a common use case for online serving (Schuster et al.. 2022). Also, we use to generate output sequences through greedy sampling with beam size of 1. We measure the time including all decoding steps until completion.
SQuAD (Question Answering): The Stanford Question Answering (SQuAD, Rajpurkar et al. 201 is collection of 87.6K reading compre- hension tasks. It includes questions generated by crowd workers based on set of Wikipedia articles.
IWSLT 2017 (Machine Translation): IWSLT 2017 (Cettolo et al., 2017b) addresses text trans- lation, using a single machine translation (MT) system for multiple language directions such as English and German. Here, we specifically focus on a German-to-English translation task.
B Detailed Experimental Setup
Training hyperparameters. In this section, describe the detailed hyperparameter values for our work. We utilize the NVIDIA RTX 3090 GPUs for training the language models, and we summarize

---

Statie-cxit CALM
FREE" * FREE
Table 9: Comparison between FREE with Ta-large
92
and directly trained small-sized T5-base. We apply threshold values of FREEtas as 0.1 for SQuAD and 0.2 for CNN/DailyMail.
(
F1 (L)
91
30
SQuAD
CNNDailyMail
Method
Model
F1
Speedup
ROUGE-L
Speedup
90
20
0.2
0.4
0.6
0.2
0.6
0.8
1.0
Full Model
Ta-large
91.82
x 1.00
41.09
x 1.00
Normalized Lateney
Normalized Lateney
Full Model
T5-hase
90.50
x 1.86
40.22
2.06
(a) SQUAD
(b) IWSLT De-En
FREET
TS-large
90.95
2.76
40.17
2.07
The trade-off' between the generated output quality and normalized lateney under different exit con- ditions. The dashed line represents the Fl and BLEU scores of the full model, which is the fine-tuned shallow- deep module, respectively. Similar to Figure 5, ex- clude the inner point of the Pareto curve.
Figure 9:
effectiveness in this dataset as well. We believe that this challenge, regarding the large vocabulary size, can be mitigated by employing vocabulary size- independent confidence measure that proposed in previous work (Schuster al., 2022). Nonetheless, our proposed algorithm still outperforms the other baselines on various datasets.
WCE
KD-last
KD-mnif
KD-dyna
ROUGE-L(())
Baseline
D.2
Layerwise Knowledge Distillation
Baseline
Given the only two exit positions in our shallow- deep module, since their performance significantly impacts the overall robustness of the early-exiting approach, we carefully design the loss function for training. In Figure 10, we observed the per- formance trends of four different loss functions as varied the exit thresholds. While the differ- ences are not significant, the KD-dyna loss demon- strates better trade-off's compared weighted average or other KD-based losses. Specifically, the lower performance of KD-unif on the SAMSum dataset suggests that dynamically determining the layer mapping can facilitate more effective knowl- edge transfer between the deep and shallow models. Consequently, trained our shallow-decp module using the KD-dyna loss for all experiments, and left the exploration of additional loss functions, such as contrastive distillation losses (Tian al., 2019; Bac et al., 2021), for future work.
36
0.2
0.4
0.6
0.8
1.0
0.2
0.6
0.8
1.0
Normalized Latency
Nommalized Latency
(a) SAMSum
(b) Maill
The trade-off between performance and nor- malized latency per sentence. We varied the exit thresh- olds in the range of [0.0. 0, 0.3, 0.5, 0.7, 0.9). The latency values are normalized by the latency of a base- line, which is a simple fine-tuned full model.
Figure 10:
D Additional Experimental Results
In this section, we provide additional experimen- tal results demonstrate the effectiveness of our proposed method and its individual components.
D.1 Performance on Different Datasets
In this section, we present a comparison of the qual- ity of the generated output (F1 or BLEU) and the inference latency on the SQuAD and IWSLT 2017 datasets, similar to the experiments in Figure 5. Figure 9 illustrates that both FREE and FREE con- sistently outperform the CALM and static-exiting baselines in the SQuAD dataset, which aligns with our previous findings.
D.3 Comparison with Small-sized Models
We conducted comparison between the inference speed of FREE using T5-large model and a directly trained T5-base model. To ensure a fair compar- ison, we manually selected the appropriate confi- dence threshold for FREE (without relying on an adaptive threshold estimator) to align its perfor- mance closely with that of T5-basc. The results, presented in Table 9, demonstrate that our proposed method exhibited a competitive speedup in infer- ence performance on the CNN//aailyMai dataset. Moreover, it demonstrated a superior FI score and significantly higher speedup on the SQuAD dataset.
However, their performance advantages in the IWSLT dataset are slightly reduced compared to other datasets. This can be attributed the larger vocabulary size of mT5 compared to T5, result- ing longer processing times for the confidence measurement. The CALM approach, which also utilizes large linear classifiers, exhibits much lower

---

Table 10:
Comparison between carly-exiting frame-
works SAMSum with different decoding strategies.
top-k (k 50)
nucleus (p = 0.92)
Method
ROUGE-L
Speedup
ROUGE-L
Speedup
Full Model
44.34
1.00
X 1.00
CALM
42.35
0,78
44.48
x 0.82
FREE
43.58
1.30
45.78
x 1.31
We believe that the variance in speedup across the datasets can be attributed to the performance achievable by a directly trained smaller model, as well as shallow model within the FREE frame- work. In the case of SQuAD, the TS-base model (12 layers) achieved a ROUGE-L score of 90.50, whereas shallow model (6 layers) of our FREE framework yielded similar score of 90.24. Our method effectively leverage these inherent benefits, thereby facilitating the inference speedup through exiting at lower layers.
D.4 Various Decoding Strategies
To evaluate the applicability of FREE on various decoding methods, we conducted experiments with top-k sampling (Radford et al., 2019) and nucleus sampling (top-p sampling: Holtzman et al. 2020). top-k sampling samples the next word from the top most probable choices, instead of aiming decode text that maximizes likelihood. On the other hand, nucleus sampling chooses from the smallest possible set of words whose cumulative probability exceeds the probability p. As detailed in Table 10. FREE method exhibited consistent and robust performance while achieving larger speedup compared to CALM. These results affirm that our FREE framework can be widely applied, irrespective of the chosen decoding method.


Reviews:
[{"review_id": "QiKMeJPqNO", "reviewer": "Reviewer_e6CC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your thoughtful feedback. We are encouraged that you found our paper well-organized, and our experimental results comprehensive. The weaknesses of our paper are discussed as follows.   \n   \n\n**W1. Some missing references.**\n\nWe would appreciate it for informing relevant previous works, related to novel decoding methods with less computational overhead. We will add these references in our paper.\n\n**W2. Contributions of paper and comparisons to other decoding techniques.**\n\nThe contributions of our paper lie in analyzing the challenges of conventional early-exiting frameworks within autoregressive language models. These challenges are effectively mitigated through introducing parallel decoding or the adaptive threshold estimator. While we are motivated from existing parallel decoding techniques, our work has tailored parallel decoding to suit the early-exiting framework, switching autoregressive decoding to parallel decoding using stacked representations.\n\nWe also discussed the efficacy and superiority of our method in Table 7 compared to speculative decoding [1,2], a recent and powerful decoding technique aimed at enhancing efficiency. Moreover, we would like to emphasize that we have compared our FREE framework with existing methods [3,4].\n\n\n**References**\n\n[1] Leviathan, et al. \"Fast inference from transformers via speculative decoding.\" International Conference on Machine Learning (2023).   \n[2] Kim, Sehoon, et al. \"Big little transformer decoder.\" arXiv preprint arXiv:2302.07863 (2023).   \n[3] Elbayad, Maha, et al. \"Depth-adaptive transformer.\" Eighth International Conference on Learning Representations (2020).   \n[4] Schuster, Tal, et al. \"Confident adaptive language modeling.\" Advances in Neural Information Processing Systems 35 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well organized.\n2. Experiments are solid and comprehensive.", "reasons_to_reject": "1. Missing references. Please refer to the \"Missing References\" section.\n2. Contributions are limited since the parallel decoding problem is well-studied in the literature. There is a lack of comparisons between the proposed framework and other decoding techniques. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors proposed a Fast and Robust Early-Exiting (FREE) framework that incorporates a shallow-deep module and synchronized parallel decoding. The framework allows for faster inference by synchronizing the decoding process and introduces a novel adaptive threshold estimator. Experimental results demonstrate the effectiveness of the framework in summarization and machine translation tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Ghazvininejad, Marjan et al. “Mask-Predict: Parallel Decoding of Conditional Masked Language Models.” Conference on Empirical Methods in Natural Language Processing (2019).\n\n[2] Cho, Kyunghyun. “Noisy Parallel Approximate Decoding for Conditional Recurrent Language Model.” ArXiv abs/1605.03835 (2016): n. pag.\n\n[3] Guo, Junliang et al. “Incorporating BERT into Parallel Sequence Decoding with Adapters.” ArXiv abs/2010.06138 (2020): n. pag.\n\n[4] Gu, Jiatao et al. “Trainable Greedy Decoding for Neural Machine Translation.” Conference on Empirical Methods in Natural Language Processing (2017).\n\n[5] Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with\nstructured dropout. arXiv preprint arXiv:1909.11556, 2019."}}, {"review_id": "Ow3mqax8VJ", "reviewer": "Reviewer_Tuxi", "summary": "", "questions": "n/a", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive comments, and we are encouraged that you pointed out the effectiveness of our proposed method on accelerating inference latency. The weakness of our paper is discussed as follows. \n\n**W1. Comparison to a baseline of directly training a smaller model.**\n\nWe compared our FREE results utilizing T5-large with those of directly trained T5-base (full model). Since the T5-base model has lower performance and latency compared to T5-large, we selected the proper confidence threshold of FREE$^\\dagger$ to match the similar performance (*e.g.*, 0.1 for SQuAD and 0.2 for CNN/DM). The experimental results are as follows.\n\n|                 | SQuAD |         | CNN/DailyMail |         |\n|-----------------|:-----:|:-------:|:-------------:|:-------:|\n|                 |   F1  | Speedup |    ROUGE-L    | Speedup |\n| Full (T5-large) | 91.82 |  ⨉1.00  |     41.09     |  ⨉1.00  |\n| Full (T5-base)  | 90.50 |  ⨉1.86  |     40.22     |  ⨉2.06  |\n| FREE$^\\dagger$    | 90.95 |  ⨉2.76  |     40.17     |  ⨉2.07  |\n\nOur proposed method demonstrated competitive inference speedup on the CNN/DM summarization task, and showed superior F1 scores with much higher speedup on the SQuAD task. \n\nWe would note that FREE also can be used with smaller models, and the static-exit baselines are similar to smaller model cases because they deterministically use a small number of decoder layers of original T5-large (refer to Figure 5).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method effectively accelerates latency, particularly in QA tasks.", "reasons_to_reject": "It would be beneficial if the authors could include a baseline of directly training a smaller model (with similar average inference speed to the early-exit framework). Such a model may achieve similar performance compared to the proposed method.", "questions_to_authors": "n/a", "paper_topic_and_main_contributions": "This paper proposes an early-exiting framework for autoregressive language models, which allows for early exits based on confidence. To cache key and value states for early-exit tokens, the authors copy the lower layer hidden states into higher layers, following Schuster et al. (2022). The proposed method increases the inference speed in QA, summarization, and MT with minimal performance losses.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a", "Justification_for_Ethical_Concerns": "n/a"}, "extra_text": {}}, {"review_id": "zyxBt8EZBU", "reviewer": "Reviewer_ZsSB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful that you have acknowledged our extensive analysis on the current early-exiting framework and novelty of the proposed FREE method. The weakness of our paper is discussed as follows.\n\n**W1. Qualitative human evaluation results.**\n\nAs recent works showed the effectiveness of  LLM evaluation and high correlation to human evaluation [1-4],  we conducted two human-like summarization evaluations with ChatGPT API (gpt-3.5-turbo-0613), Likert scale scoring and pairwise comparison [2]. Here, we compared a full model and our FREE framework (with adaptive threshold estimation) on 100 randomly selected instances from CNN/DM. Note that they showed 41.09 (⨉1.00) and 40.99 (⨉1.65) ROUGE-L scores, respectively. The detailed evaluation setup and results are described as follows.\n\n(1) For measuring the ***Likert scale scoring***, we used the following template:\n\n*“Evaluate the quality of summaries written for a news article. Rate each summary on four dimensions: relevance, informativeness, fluency, and coherence. You should rate on a scale from 1 (worst) to 5 (best).”*\n\nWe obtained [4.73, 3.83, 3.87, 3.77] for a full model and [4.68, 3.84, 3.84, 3.72] for our FREE in four dimensions, respectively.\n\n(2) For measuring the ***Pairwise comparison***, we used the following template:\n\n*“Given a new article, which summary is better? Answer “Summary 0” or “Summary 1”. You do not need to explain the reason.”*\n\nWe got a win count of 101 and 99 for the full model and FREE, measured twice by changing the order of summaries for the fair comparison.\n\nAs a result, we are confident that our method is capable of yielding predictions of similar quality with those from a full model, while reducing computational overhead.\n\n**References**\n\n[1] Liu, Yang, et al. \"Gpteval: Nlg evaluation using gpt-4 with better human alignment.\" arXiv preprint arXiv:2303.16634 (2023).   \n[2] Gao, Mingqi, et al. \"Human-like summarization evaluation with chatgpt.\" arXiv preprint arXiv:2304.02554 (2023).   \n[3] Min, Sewon, et al. \"FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation.\" arXiv preprint arXiv:2305.14251 (2023).   \n[4] Zhang, Xinghua, et al. \"Wider and Deeper LLM Networks are Fairer LLM Evaluators.\" arXiv preprint arXiv:2308.01862 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Good overview of current early exiting frameworks, their strengths and weaknesses in Section 4.\n2. Novel early exiting method that tackles the biggest problems with existing frameworks, including low quality output using an adaptive threshold.", "reasons_to_reject": "1. The authors only present ROUGE-L results over time for the various models. However, I believe at least a  qualitative human evaluation is necessary when evaluating generation based tasks/models. This would ensure that good ROUGE-L scores are not a result of idiosyncrasies in the data, output or the ROUGE-L metric itself. Moreover, it would be interesting to see! Do the authors mean to say that the outputs are completely similar, or of similar quality?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a new mechanism for early-exiting during decoding at inference time in Autoregressive language models. Their framework utilizes a shallow-deep division of the LM which also enables synchronized parallel decoding, as well as on-the-fly threshold estimator that ensures that low-confidence early exits don't seep through.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mBLrhhaNTF", "reviewer": "Reviewer_Bvg2", "summary": "", "questions": "- Since you are measuring the performance on GPU, I think you should call `torch.cuda.synchronize()` before calling the model in order to measure the wall clock time properly. I don’t see this implemented in the code, can you comment a little on this aspect? You should also use `time.perf_counter()` instead of `datetime.datetime.now()`.\n- What kind of sampling procedure did you use for the different datasets? Are you using just greedy decoding or also top-p/top-k heuristics?\n- In the paper, you said that you used a single 3090 GPU. However, in the readme file of your code repository you said “We support multi-GPUs training, while we utilized 1-4 GPUs”. Which one is correct?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would appreciate your insightful comments based on a thorough inspection of our paper. We are encouraged that you found our paper and code well-organized, experiments extensive and throughout, and the contributions important and interesting. The weakness and questions of our paper are discussed as follows. \n\n**W1. Requirement of modification to the model and training.**\n\nWe would like to emphasize that there is no model modification because both shallow and deep decoders share one classifier. It only requires implementation details for exploiting synchronized parallel decoding.\n\nBesides, since shallow and deep models are jointly trained at once, it only leads to few additional training costs. Parameter-efficient fine-tuning methods could further help us to reduce the training costs. For example, we could achieve 49.99 and 39.50 ROUGE-L scores for deep and shallow models on SAMSum by utilizing the LoRA [1] (49.11 and 42.12 for full fine-tuning). Also, training a T5-xl model with LoRA showed 50.50 and 46.08, enabling FREE (with adaptive threshold estimator) to achieve 49.84 ROUGE-L score with ⨉1.50 latency speedup.\n\n**Q1. Question on implementation details when measuring latency.**\n\nThanks to the feedback, we measured the inference speed on SAMSum again with  `torch.cuda.synchronize()` for more accurate measurement. The speedup trends were almost similar to previous results as follows (parentheses denote w/o synchronize).\n\n\n|             |    Layer 8    |    Layer 12   |    Layer 16   |    Layer 20   |    Layer 24   |\n|-------------|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n| Static-exit | ⨉2.97 (⨉3.00) | ⨉1.88 (⨉1.98) | ⨉1.45 (⨉1.50) | ⨉1.12 (⨉1.19) | ⨉1.00 (⨉1.00) |\n|             |   **Thres 0.1**   |   **Thres 0.3**   |   **Thres 0.5**   |   **Thres 0.7**   |   **Thres 0.9**   |\n| CALM        | ⨉1.26 (⨉1.37) | ⨉1.10 (⨉1.14) | ⨉1.02 (⨉1.00) | ⨉0.91 (⨉0.90) | ⨉0.82 (⨉0.77) |\n| FREE$^\\dagger$       | ⨉2.35 (⨉2.38) | ⨉1.82 (⨉1.80) | ⨉1.47 (⨉1.50) | ⨉1.28 (⨉1.33) | ⨉1.17 (⨉1.19) |\n\n\n\nMoreover, we found that the difference between using `time.perf_counter()` and `datetime.datetime.now()` function is negligible.\n\n**Q2. Question on the results with top-p/top-k sampling.**\n\nWe have used greedy decoding for all datasets due to its simplicity and effectiveness. When we measure the output performance with top-k and top-p sampling, our FREE (with adaptive threshold estimator) similarly shows robust performance and larger speedup compared to the CALM method. We summarized the results on the SAMSum dataset in the following table.\n\n|          |  |    top-k (k=50)     |       |  |   top-p (p=0.92)    |       |\n|---------|:---------------------:|:-------:|:-----:|:-----------------------:|:-----:|:-----:|\n|         |       Full      |   CALM  |  FREE |        Full       |  CALM |  FREE |\n| ROUGE-L |         44.34         | 42.35 | 43.58 |          45.84          | 44.48 | 45.78 |\n| Speedup |         ⨉1.00         |  ⨉0.78  | ⨉1.30 |          ⨉1.00          | ⨉0.82 | ⨉1.31 |\n\n**Q3. Question on computational resources.**\n\nWe would note that we used a single 3090 GPU for the inference while we utilized 2 or 4 GPUs for fine-tuning the models.\n\n**Q4. Missing references and minor comments.**\n\nThanks for the constructive comments. We will cite additional related publications, and modify some citation formats and the caption of Figure 1 in the finalized version.\n\n**References**\n\n[1] Hu, Edward J., et al. \"LoRA: Low-Rank Adaptation of Large Language Models.\" International Conference on Learning Representations (2021).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is sound and well-written\n- Experiments are throughout and exhaustive\n- The method introduces contributions to improve the current limitations of early-exit approaches with several interesting observations\n- The authors provided the code in a well-organized repository. Implementation details in the appendix are also well-documented and the used resources are reasonable (single Nvidia 3090 with desktop hardware)", "reasons_to_reject": "The proposed approach FREE requires modification to the model and training", "questions_to_authors": "- Since you are measuring the performance on GPU, I think you should call `torch.cuda.synchronize()` before calling the model in order to measure the wall clock time properly. I don’t see this implemented in the code, can you comment a little on this aspect? You should also use `time.perf_counter()` instead of `datetime.datetime.now()`.\n- What kind of sampling procedure did you use for the different datasets? Are you using just greedy decoding or also top-p/top-k heuristics?\n- In the paper, you said that you used a single 3090 GPU. However, in the readme file of your code repository you said “We support multi-GPUs training, while we utilized 1-4 GPUs”. Which one is correct?", "paper_topic_and_main_contributions": "The paper addresses the problem of speeding up the generation of autoregressive models via an early exit mechanism (FREE). Specifically, their approach addresses several limitations of previous early-exit works like (i) KV cache copying in exited layers, (ii) exit positioning, and (iii) tradeoff between latency and accuracy. Results show that FREE achieves a good tradeoff between latency and accuracy without requiring threshold selection.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Caption of Figure 1 can be improved. It does not explain the setting (a) and does not refer to (a) and (b)."}, "extra_text": {"Missing_References": "- Some references that are missing:\n    - “Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time” Liu et al.\n    - “H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models” Zhang et al.\n- You are using the library Datasets from Hugging Face but you are not mentioning it in the implementation details nor citing the corresponding paper “Datasets: A Community Library for Natural Language Processing” Lhoest et al.\n- Please mind that several citations cite the ArXiv preprint and not the corresponding published article. Some examples are:\n    - Leviathan et al. 2022 → ICML 2023\n    - Elbayad et al. 2019 → ICLR 2020\n    - Santilli et al. 2023 → ACL 2023\n    - Nallapati et al. 2016 → CoNLL 2016"}}]

Author rebuttal:

Meta-review: The paper works on Early-exit based strategy to accelerate the model. Three main improvements: 1) shallow-deep module 2) synchronized parallel decoding 3) threshold estimation without valid dataset.

Pros:

1. Well organized and extensive experiments to validate the methods.

2. Provide a very good overview over Early-exit based methods. 

3. Observations on the existing early-exit methods are insightful. It could potentially benefit others to further improve this line of reserach.

4. The threshold estimation is novel.

Cons:

Section 5.2 is not well written. There is no math formula or enough English explanation on how the parallel decoding works. Figure 4 itself is not enough for readers to quickly understand.
Decision: Accept-Main