HadSkip: Homotopic and Adaptive Layer Skipping of Pre-trained Language Models for Efficient Inference
Authors: Haoyu Wang
Yaqing Wang
Tianci Liu
Tuo Zhao
Jing Gao
Venue: EMNLP 2023 Findings (2023)
Keywords: language model inference
efficiency

Abstract: Pre-trained language models~(LMs) have brought remarkable performance on numerous NLP tasks. However, they require significant resources and entail high computational costs for inference, making them challenging to deploy in real-world and real-time systems. Existing early exiting methods aim to reduce computational complexity by selecting the layer at which to exit, but suffer from the limitation that they have to sequentially traverse through all layers prior to the selected exit layer, which lacks flexibility and degrades their performance. To solve this problem, we propose a \textbf{h}omotopic and \textbf{ad}aptive layer \textbf{skip}ping fine-tuning method named HadSkip. HadSkip adaptively selects the layers to skip based on a predefined budget. Specifically, we introduce a learnable gate before each layer of the LM to determine whether the current layer should be skipped. To tackle various challenges in training such as discrete gates and the budget constraint, we propose a fine-grained initialization strategy and homotopic optimization strategy. We conduct extensive experiments on the GLUE benchmark, and experimental results demonstrate the proposed HadSkip outperforms all state-of-the-art baselines significantly.

HadSkip: Homotopic and Adaptive Layer Skipping of Pre-trained Language Models for Efficient Inference
Haoyu Wang,, Yaging Wang', Tianci Liu Tuo Zhao", and Jing Gao
Pardue University, West Lafayette, IN, USA
iGoogle Research, New York, NY, USA
* Georgia Institute of Technology, Atlanta, GA, USA
& [wang5346, liu3351, , jinggao]@purduee edu,
yaggingaangégoogle com, ttorzhao@@attchhccom edu
Abstract
neck for deploying them in latency-sensitive latency-constrained scenarios (Xin al., 2020).
Pre-trained language models (LMs) have brought remarkable performance to numerous NL.P tasks. However, they require significant resources and entail high computational costs for inference, making challenging to deploy them in real-world and real-time systems. Ex- isting early exiting methods aim to reduce com- putational complexity by selecting the layer at which to exit, but suffer from the limitation that they have to sequentially traverse through all layers prior the selected exit layer, which lacks flexibility and degrades their performance. To solve this problem, we propose a homotopic and adaptive layer skipping fine-tuning method named HadSkip. HadSkip adaptively selects the layers to skip based on predefined budget. Specifically, we introduce learnable gate be- fore each layer of the LM to determine whether the current layer should skipped. To tackle various challenges in training brought by dis- crete gates and budget constraints, we propose fine-grained initialization strategy and homo- topic optimization strategy. We conduct exten- sive experiments on the GLUE benchmark, and experimental results demonstrate the proposed HadSkip outperforms all state-of-the-an base- lines significantly.
To comply with the demand for inference- efficient models, several approaches have been proposed accelerate pre-trained LM inference. These methods include weight quantization (Zafrir et al., 2019; Kim et al., 2021), pruning (Liu et al.. 2021), knowledge distillation (Sanh et al., 2019; Jiao et al., 2019), and early exiting (Xin et al., 2020; Zhou et al., 2020; Xin et al., 2021). Among them, early exiting methods become more appealing they do not rely specific hardware support, such as custom chips (Liu al., 2023) GPUs, and they do not require training an efficient model from scratch on large pre-training corpora, thus provid- ing a more cost-effective solution. Early exiting methods attach a classifier to each transformer layer and aim to determine if the model should exit (stop inference) from the current layer based crite- rion or learned decision. Both the backbone model and the classifiers jointly trained in the fine- tuning stage.
Although early exiting methods enable the selec- tion of a layer to exit based on the input sequence, they still require the input to pass through the trans- former layers sequentially before exiting. This pre- vents the model from choosing desired uncontigu- ous layers. For example, with a 2-layer budget, for some input sequences, choosing the second and fifth layers of the model could achieve the most su- perior performance. However, early exiting meth- ods can only traverse the layers sequentially and exit most the second layer, thereby missing out on the optimal combination of the second and fifth layers and potentially leading to degraded model accuracy as a result of suboptimal layer selection.
1 Introduction
Pre-trained language models (LMs), such as BERT (Devlin et al., 2018), RoBERTa (Liu al., 2019), and DeBERTa (He al., 2020), have sig- nificantly improved performance across various natural language processing (NLP) tasks, includ- ing paraphrase identification (Wang et al., 2021), natural language inference (Jiang and de Marneffe, 2019), sentiment classification (Gao et al., 2019), and on. Pre-trained LMs typically built by stacking transformer (Vaswani et al., 2017) layers their variants, where the self-attention mecha- nism within each layer often exhibits high compu- tational complexity. Consequently, the inference complexity of pre-trained LMs has been a bottle-
To address this issue, we propose homotopic and adaptive layer skipping fine-tuning method named HadSkip. The proposed HadSkip selects which layers can be skipped based on the diffi- culty of input sequences and pre-defined budget.

---

Specifically, we introduce binary learnable gate before each transformer layer of the LM to control whether the model can skip the current layer. This allows the model to use different gate values for dif- ferent input sequences, satisfying the inference bud- get requirements. However, learning binary gates is a non-trivial task. The binary gates are discrete and it is difficult to optimize by gradient-based op- timizers directly. Besides, the presence of a budget constraint can lead early optimization collapse, particularly when the budget value small. To tackle them, we design an elaborated three-stage training strategy to tackle this challenge. The first stage finetunes the backbone model and uses it to initialize the transformer layers. Next, these trans- former layers are frozen and the gate parameters are initialized. In the third stage, the transformer layers and gates are trained jointly. To compute the gra- dient of gates, we employ a second-order gradient approximation method ReinMax (Liu et al., 2023). Further, we propose a homotopic optimization strat- egy. which converts the optimization process into a sequence of problems ranging from a large bud- get to small budget. On one hand, this ensures a smoother and more stable optimization process. On the other hand, it avoids skipping a large num- ber of layers at the beginning of training and losing valuable information from the pre-trained model.
appendix.
Early exiting methods enable the production of inference-efficient models for specific infer- ence budgets during the fine-tuning stage. These methods draw inspiration from adaptive computa- tion (Graves, 2016) techniques in recurrent neural networks and BranchyNet (Teerapittayanon et al., 2016) in computer vision. The fundamental con- cept behind early exiting methods is to identify the layer at which to exit early, rather than sequen- tially passing through all layers. Based on the cri- teria of choosing the exiting layer, they can be organized into three strategies. The first is score- based early exiting. (Liu et al., 2020; Xin et al., 2020; Kaya et al., 2019) use the entropy of the prediction probability and the maximum of the pre- dicted distribution as the score for exit determina- tion respectively. The second learning to exit. BERxiT (Xin et al., 2021) used a fully connected layer right after each layer to produce decisions if the model needs to exit from the current layer. The third is patience- based early exiting. For example, PABEE (Zhou et al., 2020), SENTEE (Li et al., 2021) and LeeBERT (Zhu, 2021) were designed to use scores of multiple layers to determine if the model could exit. Recently, PCEE-BERT (Zhang et al., 2022) also combined the score-based method with the patience- based early exiting method to im- prove accuracy. Despite these advancements, early exiting methods still require sequential traversal through multiple layers. However, for many input sequences, the optimal layers for inference could be non-contiguous. This lack of flexibility in early exiting methods can limit their performance. To address this limitation, inspired by BlockDrop (Wu et al., 2018), we propose a homotopic and adaptive layer skipping tie-ttunig method named HadSkip, which can dynamically select which layers can be skipped based on the difficulty of input sequences and the pre-defined budget. Concurrently, Smart- BERT (Hu et al., 2023) proposes to combine layer skipping and early exiting based cross-layer con- trastive learning.
The contributions of the paper are summarized as follows: 1) We propose a homotopic and adap- tive layer skipping fine-tuning method to acceler- ate the inference pre-trained LMs. It is more flexible and economical compared to existing LM inference acceleration methods. 2) We design a three-stage and homotopic optimization strategy. This strategy makes the optimization more stable and easier solve. 3) We conduct extensive exper- iments on the GLUE benchmark. Results show that the proposed HadSkip outperforms baselines sig- nificantly and achieves a good afficiency-accuracy trade-off, e.g., preserving 95.7% of BERT's perfor- mance with only half the layers on average.
2 Related Work
Preliminaries
Existing approaches for accelerating the inference of pre-trained language models can be broadly cat- cgorized into two types: 1) model compression- based methods and 2) early exiting based methods. The proposed method is more relevant to early ex- iting based methods, which are discussed in more detail as follows. The discussion about the model compression- based methods can be found in the
In this section, we first introduce the architecture of the pre-trained language model (PLM) and analyze its complexity for inference.
3.1 Pre-trained Language Model
pre-trained language model usually consists of embedding layers, multiple transformer layers, and an output layer. Specifically, for an input sequence

---

2, we obtain its corresponding embedding first:
meet the budget contraint, we propose three-stage optimization strategy in Section 4.3.
e Ctoken (x) + Epos (r) + Cseg (r))
4.2
Gated Transformer Layer
where Etokens Epos. and Escy are the token embed- ding layer, position embedding layer and segment embedding layer respectively. Then the embedding e will be fed into transformer layers, which can be formulated as ho = e. and
Based on the analysis in Section 3.2, we identify that the transformer layers are the bottleneck in terms of inference complexity. Since the difficulty of input sequences can vary due to factors such as vocabulary, sequence length, and rhetoric, it is unnecessary to utilize all I. transformer layers for (1)
h; fi; 1: 0.), i = 1.,
every input sequence. Using a smaller number of transformer layers may already yield accurate pre- diction results for certain simple sequences. There- fore, motivated by (Wu et al., 201 8), we propose gated transformer layer that adaptively selects layers enhance inference efficiency.
where h4s 8,, and ft are the output, parameters, and mapping function of the i-th transformer layer respectively. In the end, the output of the last trans- former layer passes the output layer and generates logits, which can be represented as o fout (h...
3.2 Inference Complexity
Specifically, we introduce binary gate before feeding input into each transformer layer. This gate determines whether the current transformer layer should be bypassed based on the output of the previous layer. Formally, the forward propagation can be represented as
The inference complexity consists of the complex- ity of three parts: 1) embedding layers, 2) trans- former layers, and 3) the output layer. The time complexity of both the embedding layers and the output layer is approximately (dd). The complex- ity of transformer layers is O44dd² I. + mld where m represents the sequence length, d denotes the embedding dimensionality, and L signifies the number of transformer layers. Based on the analy- sis, it is evident that the complexity of transformer layers significantly exceeds that of the embedding layers and the output layer. Hence, this paper pro- poses to skip some transformer layers to reduce inference complexity, leveraging the expected num- ber of transformer layers as a budget.
ho e, h4 =fi(h; 1;0() Di h4-1 (1 9t), where g. ([:(hh11:w:)) E {0,1} is the binary gate, B(.) : IR -> {0, 1) is a binarized function, g()) is a one hidden layer feed-forward network, and wi is the gate parameter. If the value of gate is 1, then h; fih- 0,), indicating that the cur- rent i-th transformer layer is preserved; otherwise, is skipped.
4.3 Optimization Strategy
4 Methodology
Considering a language model composed of stacked gated transformer layers, we need to im- pose a constraint on the maximum number of used layers, which should be approximately equal to specified budget. The optimization problem can be formally defined as follows:
Given an L-layer pre-trained language model (PLM) and a training set D " where z' is the input sequence, y is the corre- sponding label and n is the number of training data, the aim is to train model f(x*)= + y² which can be efficient for inference while preserving the model performance.
1 min 71
E(f(x))))
4.1 Overview
n Σ nL j=1==1
To enhance model inference efficiency, we pro- pose a homotopic and adaptive layer skipping fine- tuning method named HadSkip, which is shown in Fig. 1. The motivation behind HadSkip is to reduce the complexity of transformer layers by adaptively skipping certain layers based on the input sequence. We introduce gated transformer layers in which binary gate is incorporated into each transformer layer to determine whether the current layer should be skipped (Section 4.2). Additionally, to address the con-differentiabee nature of the binary gates and
s.t.Egg
(2)
where is the collection of model parameters con- sisting of transformer layer parameters, gate pa- rameters, embedding layer parameters and output layer parameters, of is the gate of i-th transformer layer with respect input sequence x², E(.) is the loss function, the specific form of which will be demonstrated in Section 4.3 in detail, and s is the pre-give pre-given budget. The mean square error is em- ployed as a penalty term in the problem in Eqn. 2,

---

resulting in the following formulation:
The expectation constraint is omitted and we solve the following optimization problem:
min - (E('))))) + @
mL ΣΣ 0. (3)
Σ ((f(x))))) for (o T2
min
1
where 8 is a hyper-parameterr However, because of the non-differentiabiliyy of s. the optimization pro- cess can easily converge to a suboptimal solution, making it challenging to solve directly. Addition- ally, due to the budget constraint, the model might prioritize skipping a significant number of layers first to meet the constraint. This behavior may lead to unstable optimization, and loss of knowledge learned during the pretraining phase, which is usu- ally stored in the first several layers of the model. To handle this problem, we propose a stree optimization strategy. In the initial two stages, we aim to find a good initialization for the HadSkip, while in the third stage, we propose the utilization of a homotopic optimization method, which en- ables learning of the gate values. The subsequent sections will provide an introduction to each of these three stages.
To handle the non-differentiable problem, previ- ous work usually leverages straight-through estima- tion (STE) (Bengio et al., 2013) to obtain approx- imate gradients. However, we observe that STE, although a first-order approximation (Liu et al., 2023), still presents challenges in stable backpropa- gation and may result in model collapse. To address this issue, we employ second-order gradient ap- proximation method called ReinMax (Liu et al., 2023). It integrates Heun's Method approximate the gradient of (-), which is shown in Algorithm in the Appendix.
Stage III: Train LM and Gate Jointly. Fol- lowing the completion of the first two stages, the proposed model has achieved favorable initializa- tion, and the gradient estimation is stable. However, in cases where the budget is limited and requires the LM skip multiple layers, the substantial dif- ference in model capacity leads to significant per- formance degradation. To alleviate this problem. we propose a homotopic optimization strategy that incorporates knowledge distillation. The knowl- edge distillation borrows knowledge from fine- tuned vanilla LM to guide the proposed HadSkip learning via regularizing the consistency between their hidden representation and output predictions. Additionally, the homotopic optimization strategy utilizes a greedy method to facilitate smoother op- timization.
Stage I: Initialize Transformer Layer Parame- ters. Since transformer layer parameters are real- valued, they are generally easier to optimize com- pared to gate parameters. The discrepancy in opti- mization difficulty poses a challenge for model con- vergence. Motivated by this observation, we pro- pose separate initialization design for transformer layer parameters and for gate parameters respec- tively. This section focuses the initialization of transformer layer parameters. Specifically, strict our consideration to a vanilla language model fo(-) comprising transformer layers in order to avoid non-differentiable operations. The training process for fü involves standard fine-tuning of a language model, which can be formulated as
re-
Knowledge Distillation. We incorporate the fol- lowing distillation losses into consideration: 1) the distillation loss of prediction Lp. 2) the distillation loss of hidden representations Ln. 3) the distilla- tion loss of attention Lutt, and 4) the distillation loss of embedding layer Lemb- Further elaboration on the distillation losses and the total loss can be found Section the appendix.
min 1 Σ [((fo(=';0a..))) B. n
(4)
where - denotes the task-specific loss func- tion, such as cross-entropy for classification, and O, represents the parameters of fv, comprising transformer layer parameters, embedding layer pa- rameters, and output layer parameters. The learned O, is utilized to initialize the corresponding param- eters of the model f. denoted as e) ()( V +,.
Homotopic Optimization Strategy. While lever- aging initialization and knowledge distillation can enhance the performance of HadSkip, optimizing the model remains challenging due to the signif- icant capacity and representation power disparity between f and f.. To address this issue, we pro- pose homotopic optimization strategy that trans- forms the optimization into a sequence problems, gradually progressing from casier to more challeng- ing ones. Formally, we solve a series of optimiza-
Stage I:: Initialize Gate Parameters. Following the initialization of transformer layer parameters, we freeze the parameters O) [wi]/_ and relax the original optimization problem in Eqn. to achieve a coarse-graie -aaseee initialization for gate parameters.

---

Parameters are updated
Parameters are frozen
Label
Gate i - Negaton
Stage 3: LM and gates
W a cam--ut 'I Stage
Gate Negatin
Transformer
Output Layer
Transformer i
Gated Transformer
Gated Transformer
Stage freeze LM and train
Label
Label
Transformer
DKL
Output Layer
Output Layer
Output Layer
galles
Embedding Layer
Gated
Gated
Ln + Latt
Transforer
Transformer
Transformer i
Embedding
Embedding
Lemb
Embedding Layer
Layer
Layer
Figure 1: The framework of HadSkip.
tion problems by replacing s with Eqn. 3 for k = 80, .1,..., where so > 81 >... > 8q s. Hence, the result of the last optimization becomes the initialization for the subsequent optimization problem. Here, we observe that Eqn. 3 is not highly sensitive to the choice of 3. Therefore, employ uniform | value across all opti- mization problems. Additionally, we find seeting simple yet
classification tasks from the GLUE benchmark, namely MRPC, SST-2, RTE, QQP, QNLI, MNLI, and CoLA, which use FI, Accuracy, Accuracy, FI, Accuracy, Accuracy, and Matthews correlation co- efficient as metrics respectively (the higher the bet- ter). More details about the datasets and evaluation metrics provided in Table 5 in the Appendix. The performance is reported the development sets, following the approach of (Zhang et al., 2022; Sun et al., 2019).
8o L, $1 L ,...,,,8. 8q effective choice.
5 Experiment
In this section, evaluate the proposed HadSkip and answer the following questions: RQ1) How does HadSkip perform compared to state-of-the art early exiting baselines? RQ2) What are the roles of initialization, knowledge distillation, and homotopic optimization strategy in model perfor- mance improvements respectively? RQ3) Can the proposed HadSkip generalize well with respect to different backbones? RQ4) Is it possible to combine the proposed HadSkip method with other model accelcration techniques? RQ5) How does the performance change with varying budget val- ues? RQ6) Can the proposed HadSkip capture the difficulty differences among input sequences?
Baselines We adopt two types of baselines: 1) vanilla pre-trained language model BERT (De- vlin et al., 2018), and 2) early exiting meth- ods, including budgeted exiting (Zhang et al., 2022), BranchyNet (Teerapittayanon et al., 2016), Shallow-Deey (Kaya al., 2019), BERxiT (Xin et al., 2021), PABEE (Sun et al.. 2019), and PCEE- BERT (Zhang al., 2022). Since BERT does not have an inference speedup design, it serves as the performance ceiling for inference speedup methods. Budgeted exiting uses top K layers to make predic- tions based on the budget. BranchyNet, Shallow- Deep, BERxiT, PABEF, and PCEE-BERT dynam- ically select layer to exit according to the input sequence. Following (Zhang al., 2022), make early exiting baselines and proposed HadSkip ex- pectedly use 3, 6, and 9 layers as budgets in ex- periments. Unless otherwise specified, all these methods utilize BERT as the backbone.
5.1 Datasets and Experiment Settings
Datasets Experiments are conducted on the benchmark GLUE dataset following the method- ology of (Zhang et al., 2022; Sun et al., 2019). Specifically, we evaluate our approach on seven

---

CoLA
MNLJ
MRPC:
QNLI
QOP
RTE
S8T-2
AvG
Specifically, on the CoLA dataset, the baselines
BERT"
59.4
84.3
91.3
88.5
92.5
82.2
struggle to provide accurate classification results
Bxdir
0.0
70.0
75.8
77.4
81.8
54.7
81.0
63.0
DranchyNet"
0.0
63.8
75.7
74.2
71.6
54.7
79.9
60.0
when using and 6 layers, while the proposed
Shallow-Deep"
0.0
64.1
75.6
74.3
71.4
54.7
79.5
59.9
HadSkip remains effective. This indicates that the
BERXiT"
0.0
63.5
75.6
73.3
68.2
55.3
79.5
59.3
PABIEE
75.8
73.6
55.8
79.9
59.7
proposed HadSkip exhibits greater superiority over
PCEE BERT
9.8
73.4
78.8
80.4
79.6
58.4
83.6
66.3
the baselines when operating under smaller bud-
35.6
76.1
$4.2
$4.1
58.5
54.9
86.5
73.4
Budgeted-Exit
85.3
89.5
68.1
get. Hence, in scenarios with limited resources.
BranchyNe"" Net"
D.D
78.3
K3D
87.1
89.5
67.4
88.3
70.5
selecting layers adaptively may be more effective
Shallow: Deep
0.0
78.2
$2.8
$7.2
89.6
67.2
88.4
70.5
BERXIT"
12.3
78.4
$2.9
$7.0
89.1
61.3
88.3
72.2
than choosing the exit layer.
PABEE"
0.0
78.9
83.1
67.2
89.6
67.7
88.7
70.7
PCEE-BERT
23.2
80.1
84.8
$7.1
90.8
69.4
90.4
75.1
HadSkipp BERT
50.8
$8.7
90.0
Budget
MRPC
MNLI
CoLA
Dudgeted Exit"
$1.9
83.0
$7.0
58.4
90.3
09.0
91.2
30.1
BranchyNol"
59.3
82.1
77.0
34.4
Shallow-Doy
52.3
62.9
85.7
89.3
90.1
67.8
91.2
79.9
HadSkip w/o gate initialization
BERATT"
52.2
89.6
91.4
80.1
6
86.9
81.5
$0.6
PADEE"
52.4
83.4
96.1
59.8
90.4
68.3
91.7
80.3
PCEE BERT""
52.8
83.4
86.8
90.5
69.7
91.8
80.9
88.7
83.7
56.3
HadSk IpRRIIPRRRTT
58.3
83.8
88.4
90.7
90.8
66.8
92.7
81.6
3
81.9
75.1
5.1
Hp w/o homotopic optimization
Table I:
Performance comparison on GLUE dataset. Blue, green, and orange shadow parts represent methods with 3, 6, and 9-layer budgets respectively. "AVG" notes the average performance of the tasks. Results with sggn are taken from (Zhang et al., 2022).
83.1
81.7
39.9
9
88.8
83.7
51.6
3
83.0
62.9
17.9
HadSkip w/o KD
6
87.3
80.8
47.8
9
88.7
82.7
$7.6
85.5
77.7
29.6
HadSkip only w/ prediction KD
5.2 Performance Comparison
6
86.7
81.7
48.4
87.8
83.5
58.6
This section presents the performance of the base- lines and the proposed HadSkip model, as shown in Table I, addressing RQ1.
3
84.2
78.1
35.6
HadSkip
85,9
82.4
50.8
The performance results in Table demonstrate that the proposed HadSkip surpasses all state-of- the-art baselines on the GLUE datasets. Notably, under budget configurations of 3, 6, and lay- ers, the proposed HadSkip achieves substantial im- provements in terms of average accuracy. Specifi- cally, compared to the baselines, HadSkip exhibits performance gains of at least 10.7%. 4.8%, and 0.9% when using 3, 6, and layers, respectively. vanilla BERT,
9 88.4
83.8
58.3
Table 2: Performance of HadSkip with and without the proposed gate initialization, HadSkip with and without the proposed homotopic optimization strategy, and HadSkip with and without knowledge distillation. HadSkip ip w/o KD represents HadSkip only uses task- specific loss. HadSkip w/ prediction KD represents HadSkip that uses both task-specific loss and the distil- lation loss of prediction.
Additionally, when compared the proposed HadSkip retains 89.3%, 95.7%, and 99.3% of its performance when using 3, 6, and 9 layers, respectively. This demonstrates that HadSkip effectively accelerates model inference while preserving the performance of the pre-trained language model.
5.3 Ablation Study
Effectiveness of Gate Initialization. We con- duct analysis to answer RQ2 regarding gate ini- tialization. To validate the role of the gate initial- ization, we compare HadSkip without the second stage to the complete HadSkip architecture. The comparison results are presented in Table 2. The experimental findings confirm the positive impact of gate initialization on model training. We ob- serve that gate initialization provides greater per- formance improvement when the budget is small. However, for larger budgets, the benefits of gate initialization are limited. This can be attributed the fact that with a larger budget, the model only needs to skip a few layers, which is relatively cas- ier compared to skipping multiple layers. In such
We also observe that as the budget decreases, the diversity in model performance becomes more pronounced. Specifically, when using 9 layers, the performance of the basclines, apart from the pro- posed HadSkip, exhibits minimal variation. How- ever, with and 6 layers, the performance gaps among different methods become more prominent. The proposed HadSkip demonstrates much more significant improvements under these two settings.

---

cases, a fine-gratd fine-grained warm-up strategy becomes unnecessary.
RQ2 regarding knowledge distillation. We conduct two comparison experiments: 1) HadSkip without knowledge distillation (KD) versus HadSkip, and 2) HadSkip using only prediction- based knowledge distillation versus HadSkíp. The experimental re- sults are presented in Table 2. Prediction-bee bbaedd knowledge distillation is widely employed in tradi- tional teacher-student architectures, and the knowl- edge distillation loss function used in HadSkip is popular for compressing large language mod- els. Compared to HadSkip without KD, HadSkip consistently improves model performance. For in- stance, HadSkip achieves improvements of 98.9%, 6.3%, and 1.2% when using 3, 6, and 9 layers, re- spectively. Compared to HadSkip with only predic- tion KD, HadSkip still shows improvements. This demonstrates that the losses of hidden represen- tations, attention matrices, and embeddings effec- tively guide the model in learning how to select layers to skip. However, from Table 2, we observe that the performance improvements resulting from the distillation loss of prediction are greater than those from other distillation losses. This highlights the more significant role of the distillation loss of prediction in the proposed HadSkip.
Budget
MRPC
MNLI
CoLA
RoBERTa"
91.7
87.2
64.8
81.2
49.5
0.0
DEBBERT-ROBERTA
81.2
65.0
0.0
92.4
86.2
55.7
85.1
81.0
35.3
HadSkip+RoBERTa
90.0
85.6
55.8
91.9
87.2
60.2
Table 3: Performance of using RoBERTa as the back- bone. Results with are taken from (Lee et al., 2019). Results with Ç are implemented based on its official code
Budget
MRPC
MNII
Col.A
TimBEKIT(4 layers)"
89.1
80.4
18.6
TayBERTr BERT(6 layers)*
91.6
83.5
42.8
89.8
NudSkip--TnyBERTT layers)
91.3
40.5
Table 4: Performance of using TERyBERT as the back- bone. Results with "*" are taken from (Liang et al.. 2023).
Effectiveness of Homotopic Optimization Strategy. We conduct an ablation study to answer RQ2 with respect to the homotopic optimization strategy, We compare the proposed HadSkip with HadSkip without the homotopic optimization strat- egy on the MRPC, MNLI, and CoLA datasets, as presented in Table 2. From the results in Table we observe the effectiveness of the proposed ho- motopic optimization strategy in improving model performance. The homotopic optimization strategy leads to average improvements of 201.6%, 10.5%, and 4.2% when using 3, 6, and 9 layers, respec- tively. This trend highlights the greater importance of the homotopic optimization strategy for smaller budgets. When using 9 layers, the performance of HadSkip with and without the homotopic op- timization strategy is similar for the MRPC and MNLI datasets. However, when using 3 layers, HadSkip without the homotopic optimization strat- egy fails to produce meaningful output for CoLA, whereas HadSkip still delivers competitive results. One potential reason for this difference is that the homotopic optimization strategy smooths the opti- mization problem and facilitates convergence to a better local optimum.
5.4 Extension to RoBERTa
In this section, we employ RoBERTa (Liu et al., 2019) as the backbone to investigate RQ3. The results are presented in Table 3. We also consider DeeBERT, which utilizes RoBERTa as the back- bone in their paper, as our baseline. As shown in Table 3, HadSkip+RoBERTa outperforms Dee- BERT+RoBERTa. Specifically, for CoLA and MNLI, DeeBERT+RoBERT fails to make correct predictions when using 3 and 6 layers. In contrast, the proposed HadSkip+RoBERI rable performance RoBERTa even with only 6 layers. These results demonstrate that HadSkip can effectively balance efficiency and accuracy when combined with different backbone models.
achieves compa-
5.5 Combining with TinyBERT
In this section, we investigate RQ4 using Tiny- BERT (Jiao et al., 2019) as the backbone. Apart from early exiting methods, another popular ap- proach to accelerate model inference is reducing the number of LM layers via knowledge distil- lation for LM compression. To demonstrate the complementarity of the proposed HadSkip with knowledge distillation model compression meth- ods, we utilize 6-layer TinyBERT, which com- presses BERT using knowledge distillation into a
Effectiveness of Knowledge Distillation. In this paragraph, we do the ablation study to answer

---

Laver0
Figure 2: Visualization of the probability of each layer being used on MNLI dataset.
LigarD
Figure 3: Visualization of the probability of each layer being used on QNLI dataset.
Figure 4: Visualization of the probability of each layer being used on SST2 dataset.
Lo badinng paleess
lyen wang soc
(a) MNLI
(b) QNLI
(c) SST2
Figure 5: Visualization of the probability of the number of layers being used with respect to different budgets.
6-layer small model, as the backbone and present the performance results in Table 4. Comparing HadSkip with budget of 4 layers TinyBERT layers), we observe that HadSkip can preserve over 94.6% of the model performance. This in- dicates that HadSkip can effectively complement TERyBERT and further enhance the inference speed of BERT.ER.. Moreover, when compared to the pre- trained 4-layer TinyBERT, HadSkip with 4 layers achieves higher accuracy, and even HadSkip with layers demonstrates similar performance. These re- sults suggest that combining HadSkip with com- pressed model may yield greater effectiveness than training a smaller compression model.
language model, while BERT-base and RoBERTa consist of 12 layers each. The results presented in Fig. 6. It is evident from the figure that larger budgets lead better performance. However, when the budget is less than half the total number of lay- ers, the decrease in accuracy is significantly more pronounced compared to cases where the budget ex- ceeds half the total number of layers. Specifically, as shown in Fig. 6, using half the total number of layers still preserves formance. This finding suggests that it is preferable to skip fewer than half the total number of layers. 5.7 Case Study
97% of the model's per-
In this section, we conduct case study answer RQ6. To study if the proposed HadSkip can adap- tively select layers based on input sequences, we visualize 1) the probability of each layer being used and 2) the probability of the number of layers be- ing used on MNI, QNLI and SST2 datasets. We
5.6 Sensitivity w.rt. Budgets
We perform hyperparameter experiment ad- dress RQ5, focusing on different budgets for HadSkip using BERT-Large, BERT-base, and RoBERTa as backbones. BERT-Large a 24-layer

---

88
tuned. In this paper, them fixed values (c.g., one) and achieve good results.
86
84
Ethies Statement
Aocuracy
82
This paper proposes homotopic and adaptive layer skipping fine-tuning method, called HadSkip. We show that the proposed HadSkip can be used for inference acceleration. We perform experiments on classification tasks using the GLUE benchmark. In all experiments, utilize public benchmark datasets, models, and code. No ethical concerns were identified.
+ HodSkpp8BER P+BERT Lango
80
HadSk
HadSk P-RoBERTa
performeros BERT-Lage
78
97% performaree
97% performaros on RoBEETT
76
Budget
Figure 6: Accuracy with different budgets.
A Acknowledgement
show the visualization on Fig. 2 to Fig. 5. From Fig. 2 to Fig. 4, we observe diverse probabilities of each layer being utilized across different tasks and budgets. When utilizing 3 layers, the model tends to prefer specific layers. For example, the model favors the second and fifth layers the MNLI dataset and the second, third, and seventh layers on the QNLI dataset. However, the budget in- creases, the distribution of chosen layers becomes more uniform. According to Fig. 5, the number of layers utilized concentrates around the given bud- get, while the model also employs other numbers of layers near the budget. These figures confirm that the proposed model can select different layers for various input sequences.
This work is supported in part by the US National Science Foundation under grant NSF IIS-1747614 and NSF IIS 214231. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not nec- essarily reflect the views of the National Science Foundation.
References
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. 2020. Binarybert: Pushing the limit of bert guantization. arXiv preprint arXiv:2012.15701.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 13X8.3333
6 Conclusion
In this paper, we propose a homotopic and adaptive layer skipping fine-tuning method named HadSkip for efficient inference. We introduce a learnable gate before each transformer layer of the pre- trained language model to determine whether the model can skip the respective layer. For training the binary gate, we introduce three-stage learning strategy to initialize and update model parameters. Additionally, we propose a homotopic optimization strategy stabilize the model training. We also uti- lize knowledge distillation to guide model training and achieve an improved trade-off between effi- ciency and accuracy. We conducted extensive ex- periments on the GLUE benchmark, and the results demonstrate that HadSkip outperforms state-of-the art baselines. Moreover, HadSkip complements other acceleration methods, including BiRYBERT.
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. 2020. The lottery ticket hypothesis for pre- trained bert networks. Advances in neural informa- tion procexsing systems, 13:15534-15846
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev, 2019. Low-bit quantization of neural net- works for efficient inference. In 2019 IEEE/CVF International Conference on Computer Vision Work- shop (ICCVW), pages 3009-3018. IEEE
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre Pre-training of deep bidirectional transformers language understand- ing, arXiv preprint arXiv:
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. 2020. Training with quantization noise for extreme model compression. arXiv preprint 2004v:200..07320.
Limitations
The proposed HadSkip introduces multiple hyper- parameters including 3, which might require addi- tional effort to tune. Fortunately, we observe that the prediction performance is not sensitive to these hyperparameters, and therefore they can be easily
Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1803:003333

---

Zhengjie Gao, Ao Feng, Xinyu Song, and Xi Wu, 2019, Tagge-deependet sentiment classification with bert. leee Access, 7: 7:15 154299.
Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, and Jianfeng Gao. 2023. Bridging discrete and backprop- agation: Straight-through and beyond. arXiv preprint arXiv:2304.08611
Alex Graves. 2016. for recurrent neural networks. arXiv preprint 100.93360...888.
Adaptive computation time
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Haotang Deng, and Qi Ju. 2020. Fastbert: a self- distilling bert with adaptive inference time. arXiv preprint 20X1v:200..02178.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692..
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean, 2015. Distilling the knowledge in a neural network. arXiv preprint 1 503.0003.2...
Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. 2021. Ebert: Efficient bert inference with dynamic structured pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 48 4814-
Boren Hu, Yun Zhu, Jiacheng Li, and Siliang Tang. 2023. Smartbert: promotion of dynamic early exiting mechanism for accelerating bert inference. arXiv preprint arXiv:2303.09266
Nanjiang Jiang and Marie-Catherinn
Marneffe. 2019. Evaluating bert for natural language inference: A case study the commitmentank. In Proceed- ings of the 2019 conference on empirical methods in natural language processing and the 9th interna- rional joint conference natural language process- ing SAANLP-IJCNLLE pages 6086-6091.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXiv:1910.01108.
Siqi Sun. Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model com- pression. arXiv preprint arXiv: 000i::9908003333
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language under- standing, arXiv preprint arXiv: 1909. 10351.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang. and Denny Zhou. 2020. Mobilebert: a compact task-agnostic bert for resource-limited de- vices. arXiv preprint arXiv:2004.0290 02984.
Jing Jin, Cai Liang. Tiancheng Wu, Liqin Zou, and Zhiliang Gan. 2021. Kdlsq-bert: A quantized bert combining knowledge distillation with learned step size quantization. arXiv preprint arXiv:2101.0520
Hanlin Tang, Xipeng Zhang, Kai Liu, Jianchen Zhu, and Zhanhui Kang. 2022. Mkq- bert: Quantized bert with 4-bits weights and activations. arXiv preprint 2rXiv:2203.13483
Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. 2019. Shallow- deep networks: Understanding and mitigating network overthinking. In International conference on machine learning, pages 3301-3310. PMLR.
Surat Teerapitaaynnon Bradley McDanel, and Hsiang- Tsung Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recogni- tion (ICPR), pages 2464-2469. IEEE
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021. I-bert: Integer- only bert quantization. In International conference on machine learning, pages 5506-5518. PMLR.
Jiayi Tian, Chao Fang. Haonan Wang. and Zhongfeng Wang. 2023. Bebert: Efficient and robust binary ensemble bert. In ICASSP 2023-2023 IEEE Interna- tional Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEF.
Dongjun Lee, Sohee Yang, and Minjeong Kim. 2019. Claf: Open-source clova language framework. https://gi thub. com/naver/claf.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Xiaonan Li. Yunfan Shao, Tianxiang Sun, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2021. Accelerat- ing bert inference for sequence labeling via carly-exit. arXiv preprint 105.
Chen Liang, Haoming Jiang, Zheng Li, Xianfeng Tang, Bin Yin, and Tuo Zhao. 2023. Homodistil: Homo- topic task- agnostic distillation of pre-trained trans- formers. arXiv preprint arXiv:2302.09632
Haoyu Wang, Fenglong Ma, Yaqing Wang, and Jing Gao. 2021. Knowledge-guided paraphrase identifi- cation. Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 843-853.

---

Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. 2018. Blockdrop: Dynamic infer- ence paths in residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8817-8826.
A Related Work
Model Compression-basd Methods. There are number of well-explored model compression meth- ods which can be used for accelerating model in- ference, such as weight quantization (Choukroun et al.. 2019; Fan et al.. 2020; Jin et al., 2021; Zhu et al., 2016), pruning (Zhu and Gupta, 2017). and knowledge distillation (Hinton et al., 2015).
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien De- mouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large lan- guage models. arXiv preprint (rxiv:2211.10433
Weight quantization involves mapping model weights to low-precision integers and floatingpooin numbers, making them more hardware-friendly for computation. Specifically, (Xiao et al., 2022) pro- posed 8-bit quantization for BERT; (Tang et al.. 2022) explored how to use bits to quantize BERT; (Bai et al., 2020; Tian et al., 2023; Zhang et al., 2020) studied how to quantize BERT into 1-bit or 2-bits. Pruning, on the other hand, focuses on set- ting redundant parameters to zero to create sparse network, enabling accelerated sparse matrix opera- tions on specific hardware platforms. Existing prun- ing methods such (Liu et al., 2021) and (Chen et al., 2020) proposed a dynamic structured prun- ing method and a lottery ticket hypothesis (Fran- kle and Carbin, 2018) based method respectively to learn a sparse network for BERT. Knowledge distillation. meanwhile, is to utilize a powerful large model (teacher model) to guide the learn- ing of lightweight model (student model). The lightweight student model usually has lower infer- ence complexity compared to the teacher model. DistilBERT (Sanh et al., 2019), TinyBERT (Jiao et al., 2019), MobileBERT (Sun et al., 2020) and PKD (Sun et al., 2019) used knowledge dis- tillation to learn a lightweight BERT. However, methods have drawbucks.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. Deebert: Dynamic early exit- ing for accelerating bert inference. arXiv preprint arXiv:2004. 129999.....1993.
Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. Berxit: Early exiting for bert with better fine- tuning and extension to regression. In Proceedings of the 16th conference of the European chapter of the association for computational linguistics: Main Volume, pages 91-104.
Ofr Zafrir. Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized Sbit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing -NerrlPS Edition (EMC2-NIPS), pages 36-39, IEEE.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu. 2020. Temarybert: Distillation-wware ultra-low bit bert. arXiv preprint arXiv:2009. 12812.
Zhen Zhang, Wei Zhu, Jinfan Zhang, Peng Wang, Rize Jin, and Sun Chung. 2022. Peee-bert: Acceler- ating bert inference via patient and confident early exiting, In Findings of the Association for Computa- tional Linguistics: NAACI 2022, pages 327-338.
these compression-baut Firstly, they often require training compressed mod- els from scratch, which can be computationally ex- pensive. Secondly, weight quantization pruning techniques rely on specialized hardware support, limiting their flexibility.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley, Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. Advances in Neural Information Processing Systems, 33:18330-1883 18341.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. 2016. Trained ternary quantization. arXiv preprint 16110:161...1064.
B Knowledge Distillation
Given fine-tuned LM, i.e. fo(.) learned in the first stage, knowledge distillation is to enable the proposed f(-) to mimic the prediction and hidden representation of fo. Specifically, we consider the following losses: 1) The distillation loss of predic-
Michael Zhu and Suyog Gupta. 2017. To prune, or not prune: exploring the efficacy of pruning for model compression, arXiv preprint 171.v11710.0187..
tion is defined as
Wei Zhu. 2021. Leebert: Learned early exit for bert with cross-level optimization. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the I 1tth International Joint Conference on Natural Language Processing (Vol- une 1: Long Papers), pages 2968-2980.
Lp 1
n DDKU(f(())))
(5)
where DKL(..) is the KL divergence between the probability over the two outputs. The loss Lp regu-

---

larizes the prediction of HadSkip be consistent with that of fe- 2) The distillation loss of hidden representations to enables the hidden represen- tations of f and fo to similar, which can be formulated as
Dataset
Task
Metric
MRPC
Paraphrase Identification
Fl
QQP
Paraphrase Identification
F1
MNLI
Natural Language Inference
Accuracy
QNLI
Natural Language Inference
Accuracy
RTE
Natural Language Inference
Accuracy
MSE(h. kk)) Ln nL j=
(6)
SST-2
Sentiment Classification
Accuracy
CoLA
Linguistic Acceptahility
Matthews
where hi, oh are the i-th layer hidden represen- tation of and fo with respect to input sequence respectively. 3) Similarly, the distillation loss of attention is to penalize the discrepancy between attention matrices of f and fu, which can be repre- sented as
Table 5: Dataset description.
D
Dataset
E Implementation Details
We simply set on, a2, 03,, 04, and 3 as for all experiments. We select the learning rates from {5c- 6, 1c 5, 10-26-555 --- 5, 50-55-5--- 5) and batch size from (8, 16, 32, 64). Experiments are conducted four NIVIDA RTX A6000.
Catt nL i=
MSE A .,4)),
(7)
where A and A! are the i-th layer averaged at- tention matrices of f and fo with respect to input sequence 2, respectively. 4) The distillation loss of embedding layer is
Lemb 1 ni J
Σ MSE(e',e el),
where el and et are the embedding layer outputs of f and fe with respect to input scquence zy. In summary, the total loss function is obtained by combining it with the task-specific loss, which can be formulated as
C c- - ((f(T'))) ni
=c: + aI azZh + azGatt + a4hembs where C, Σ, (x) 0), w²) is the task- specific loss function, and a1 a2 03, as are hyper- parameters.
Algorithm of Reinmax
Algorithm 1: Reinmax
Input: gate function input 1ih:
Output: [0]
To Coftmax(g(h,, 1:w)));
D One Ho)(T))
Di
3 Tı
2
T = Doftmax(stop_gradien(((oo
g( (h;-1;w;)) + ghhh
72 2m T
B π2 stop_gradient( (T2) + D;


Reviews:
[{"review_id": "NBdqDPXsxa", "reviewer": "Reviewer_a8ng", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">The novelty is limited. Although HadSkip focuses on improving early exiting, it also can be seen as a dynamic neural network an various methods have been proposed for Transformers [1]. The overall method is overly complex compared to baselines, and the introduction of tricks such as KD and Homotopic Optimization Strategy (not used in baselines, such as PABEE) does not clearly reflect the gain of the method itself.\n\nFirstly, it's important to note that the term \"dynamic neural network\" encompasses a wide range of methods. In the context of accelerating language model inference, our proposed HadSkip stands out as notably distinct from three common dynamic neural networks: early exiting, mixture of experts (MoE), and token pruning. In the case of early exiting-based methods, they necessitate the input to traverse the transformer layers sequentially before exiting. Conversely, our proposed HadSkip exhibits the capability to dynamically select layers based on the current input, thus offering greater flexibility. In contrast to MoE, the proposed HadSkip is notably more straightforward to train due to its ability to mitigate challenges related to load imbalance and under-utilization of experts[a]. Token pruning methods can be categorized as unstructured pruning methods, making it challenging to achieve substantial acceleration without specialized hardware. In contrast, the proposed HadSkip skips multiple layers in a structured manner, rendering it more feasible for implementation on general hardware. **Therefore, while both these methods and our proposed HadSkip fall under the category of dynamic neural networks, it is evident that HadSkip is distinct from existing approaches. Moreover, beyond the disparities in model architectures, we introduce a novel three-stage and homotopic optimization strategy, which represents a substantial departure from existing methodologies.**\n\nSecondly, we have conducted an ablation study to investigate the impact of KD and the homotopic optimization strategy, as detailed in Table 2 of Section 5.3. Based on the findings presented in Table 2, it is evident that our proposed HadSkip consistently outperforms the baselines even in the absence of KD or the homotopic optimization strategy. Additionally, the proposed homotopic optimization strategy aims to enhance the stability of gating, constituting a crucial component of our approach. Consequently, through a comparative analysis with the baselines in Section 5.2 and the ablation study in Section 5.3, we can establish the effectiveness of our proposed HadSkip.\n\n>The authors only report efficiency metrics by budget (3/6/9 layers), while for an efficient method of NLP, they should also report the speed-up of FLOPs or latency (which is more important for real life.). By the way, because HadSkip uses knowledge distillation when compared with some early exiting baselines, the KD contribution should also be considered.\n\nThe FLOPs of one transformer layer of BERT is about $24sH^{2}+4Hs^{2}$, the FLOPs of embedding layer is about $2sH$, and the FLOPs of output layer is about $2H^{2}k$, where $s$ is the sequence length, $H$ is the embedding dimension, and $k$ is the number of classes. For a model with $L$ layers, it requires $(24sH^{2}+4Hs^{2})L+2sH+2H^{2}k$ FLOPs for one single inference. For HadSkip, it needs $6H^{2}L$ FLOPs for gates. If HadSkip uses $l$ layers, it needs $(24sH^{2}+4Hs^{2})l+2sH+2H^{2}k+6H^{2}L$ FLOPs for a single inference. In experiments, we use $s=128$. Thus, for a binary classification task, BERT-base needs about 22.3G FLOPs. The HadSkip uses about 5.6G FLOPs, 11.2G FLOPs, 16.8G FLOPs with respect to using 3 layers, 6 layers and 9 layers respectively. They roughly correspond to 1.3x, 2x, 4x inference acceleration.\n\n>Lack of some baseline comparisons. Layerdrop [2] and dynamic token pruning should also be compared. LayerDrop share a similar idea with HadSkip. Dynamic token pruning also can be seen as a fine-grained early exiting method [3]. In addition, the authors need to clearly identify whether they will open source code or not.\n\nFirst, the setting of Layerdrop is different from ours. Layerdrop trains layer skipping during the pre-training stage and subsequently fine-tunes the model for downstream tasks. In contrast, our proposed HadSkip does not incorporate the pre-training stage. The pre-training stage is costly, and re-training each pre-trained language model upon release is impractical. Consequently, HadSkip is designed within a more practical framework. Moreover, Layerdrop cannot adaptively drop layers for individual input data, whereas HadSkip can dynamically select layers. We include the performance results of the proposed HadSkip(w/o KD) based on RoBERTa in Table A to demonstrate its superiority over Layerdrop as well. According to Table A, we observe that despite not conducting additional pre-training, our proposed HadSkip can achieve comparable performance to Layerdrop and even outperform it when using 6 layers on MNLI.\n\nAs for Transkimmer, we believe that comparing it is beyond the scope of our study. Transkimmer is a token pruning method that modifies the model input but does not alter the model architecture. In contrast, HadSkip employs distinct model architectures for different inputs while keeping the input unchanged. The two methods are orthogonal[b]. Consequently, we have selected early exiting-based methods that are more closely related as our baselines. In future work, we intend to investigate the performance of combining HadSkip with Transkimmer. We plan to release our code in the camera-ready version to further promote reproducibility. \n\nTable A: Performance comparison using RoBERTa as the backbone. To make a fair comparison, we remove KD used in HadSkip. Results of LayerDrop are taken from LayerDrop paper directly.\n| | Budget | MNLI | SST2 |\n|-----------|------|------|------|\n|LayerDrop| 3      | 78.6 | 90.5 |\n|                 | 6      |82.9 | 92.5 |\n|HadSkip w/o KD| 3      | 78.4 | 90.7 | \n|                           | 6     | 85.1 | 92.5 |\n\n[a] Zuo S, Zhang Q, Liang C, et al. MoEBERT: from BERT to Mixture-of-Experts via Importance-Guided Adaptation[C]//Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.\n\n[b] Xu C, Mcauley J. A Survey on Dynamic Neural Networks for Natural Language Processing[C]//Findings of the Association for Computational Linguistics: EACL 2023. 2023: 2325-2336.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The layer-skipping method is easy to follow and can combine with other model acceleration methods.\n2. HadSkip outperforms previous early exiting methods on various backbones and datasets.", "reasons_to_reject": "1. The novelty is limited. Although HadSkip focuses on improving early exiting, it also can be seen as a dynamic neural network an various methods have been proposed for Transformers [1]. The overall method is overly complex compared to baselines, and the introduction of tricks such as KD and Homotopic Optimization Strategy (**not used in baselines**, such as PABEE) does not clearly reflect the gain of the method itself.\n2. The authors only report efficiency metrics by budget (3/6/9 layers), while for an efficient method of NLP, they should also report the speed-up of FLOPs or latency (which is more important for real life.). By the way, because HadSkip uses knowledge distillation when compared with some early exiting baselines, the KD contribution should also be considered.\n3. Lack of some baseline comparisons. Layerdrop [2] and dynamic token pruning should also be compared. LayerDrop share a similar idea with HadSkip. Dynamic token pruning also can be seen as a fine-grained early exiting method [3]. In addition, the authors need to clearly identify whether they will **open source** code or not.\n\n[1] Dynamic Neural Networks: A Survey. Han et al. 2021.\n\n[2] Reducing Transformer Depth on Demand with Structured Dropout. Fan et al. 2019.\n\n[3] Transkimmer: Transformer Learns to Layer-wise Skim. Guan et al. 2022.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an adaptive layer-skipping method for Pre-trained Transformer models (BERT, RoBERTa, etc.). HadSkip introduces the gate network at each layer to skip the layer or not. By training with knowledge distillation and homotopic optimization, they improve previous early exiting methods on various backbones and datasets.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zzh6vS9BAI", "reviewer": "Reviewer_ah3X", "summary": "", "questions": "I am open to rebuttal/discussion related to point 2. That's my main concern. I am willing to increase my score if this is addressed satisfactorily.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Empirical time-memory savings seem missing. FLOPs could be good.\n\nThe proposed HadSkip can accelerate model inference. The FLOPs of one transformer layer of BERT is about $24sH^{2}+4Hs^{2}$, the FLOPs of embedding layer is about $2sH$, and the FLOPs of output layer is about $2H^{2}k$, where $s$ is the sequence length, $H$ is the embedding dimension, and $k$ is the number of classes. For a model with $L$ layers, it requires $(24sH^{2}+4Hs^{2})L+2sH+2H^{2}k$ FLOPs for a single inference. For HadSkip, it needs $6H^{2}L$ FLOPs for gates. If HadSkip uses $l$ layers, it needs $(24sH^{2}+4Hs^{2})l+2sH+2H^{2}k+6H^{2}L$ FLOPs for inference one time. In experiments, we use $s=128$. Thus, for a binary classification task, BERT-base needs about 22.3G FLOPs. The HadSkip uses about 5.6G FLOPs, 11.2G FLOPs, 16.8G FLOPs with respect to using 3 layers, 6 layers and 9 layers respectively. They roughly correspond to 1.3x, 2x, 4x inference acceleration.\n\n>A very relevant simple baseline seems missing - LayerDrop [1]. LayerDrop applies simple dropout at the level of layers (related to BlockDrop cited in the paper) and achieves ability to prune a high number of layers during inference. The paper shows promising performance on RoBERTa - often even beating the non-pruned vanilla baseline. In contrast, the introduced method here is much more complicated to implement and performs slightly worse than vanilla. This seems to be a critical miss.\n\nThanks for pointing out it! In fact, our proposed HadSkip differs significantly from LayerDrop. **First, their settings are different.** LayerDrop incorporates layer skipping during the pre-training stage and subsequently fine-tunes the model for downstream tasks. In contrast, our proposed HadSkip does not include the pre-training stage. The pre-training stage is resource-intensive, and it is impractical to re-train every pre-trained language model upon release. Therefore, HadSkip is designed under a more realistic setting. **Secondly, LayerDrop is not an input-adaptive method, whereas our proposed HadSkip can dynamically skip layers based on different inputs.** Additionally, with LayerDrop, there are instances where it outperforms a smaller RoBERTa model with the same number of layers as LayerDrop, rather than outperforming the vanilla RoBERTa. To facilitate a more direct comparison with LayerDrop, we have excluded the knowledge distillation module and present the performance of HadSkip using RoBERTa as the backbone, as displayed in Table A. It is evident that, even without additional pre-training, our proposed HadSkip can achieve comparable performance to LayerDrop and even outperform it when utilizing 6 layers on MNLI. \n\nTable A: Performance comparison using RoBERTa as the backbone. To make a fair comparison, we remove KD used in HadSkip. Results of LayerDrop are taken from LayerDrop paper directly.\n| | Budget | MNLI | SST2 |\n|-----------|------|------|------|\n|LayerDrop| 3      | 78.6 | 90.5 |\n|                 | 6      |82.9 | 92.5 |\n|HadSkip w/o KD| 3      | 78.4 | 90.7 | \n|                           | 6     | 85.1 | 92.5 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. To my knowledge seems like a fairly novel method and overall an interesting and reasonable combination of techniques. \n\n2. Decent performance compared to early exit methods and nice analysis.", "reasons_to_reject": "1. Empirical time-memory savings seem missing. FLOPs could be good. \n2. A very relevant simple baseline seems missing - LayerDrop [1]. LayerDrop applies simple dropout at the level of layers (related to BlockDrop cited in the paper) and achieves ability to prune a high number of layers during inference. The paper shows promising performance on RoBERTa - often even beating the non-pruned vanilla baseline. In contrast, the introduced method here is much more complicated to implement and performs slightly worse than vanilla. This seems to be a critical miss.   \n\n[1] Reducing Transformer Depth on Demand with Structured Dropout - Fan et al. ICLR 2020\n\nPost-Rebuttal:\n\nMy concerns are addressed. I am raising the soundness score to 4.", "questions_to_authors": "I am open to rebuttal/discussion related to point 2. That's my main concern. I am willing to increase my score if this is addressed satisfactorily.", "paper_topic_and_main_contributions": "The paper introduces HadSkip for learning to adaptively skipping layers for fast pre-trained language model inference. \nThe paper does this through three stages. \n\nStage 1: The authors fine-tune the model in a vanilla manner. \n\nStage 2: The authors freeze the fine-tuned parameters. They add discrete binary gates for each layer. They train the gates using ReinMax. \n\nStage 3: The author jointly further trains both the gate and non-gate parameters through successive substages with different layer budget hyperparameter (motonologically decreasing - making the optimization successively harder). Knowledge distillation is also used for smoother optimization.\n\nOutperforms many comparable methods - particularly different early exit strategies with BERT on GLUE tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Reducing Transformer Depth on Demand with Structured Dropout - Fan et al. ICLR 2020"}, "extra_text": {}}, {"review_id": "Ag6MTPh4yk", "reviewer": "Reviewer_6pFJ", "summary": "", "questions": "- Issues in the **Reasons To Reject** Part\n- Can the method be transferred to generative language models like GPT?\n\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">The new gating module introduces additional parameters and computational complexity, but the paper lacks analysis and comparison with the baseline in this regard.\n\nThe extra parameters account for only about 6\\% of the BERT-base, and the additional computational complexity (FLOPs) is quite small. The FLOPs of one transformer layer of BERT is about $24sH^{2}+4Hs^{2}$, the FLOPs of embedding layer is about $2sH$, and the FLOPs of output layer is about $2H^{2}k$, where $s$ is the sequence length, $H$ is the embedding dimension, and $k$ is the number of classes. For a model with $L$ layers, it uses $(24sH^{2}+4Hs^{2})L+2sH+2H^{2}k$ FLOPs for inference one time. For HadSkip, it needs $6H^{2}L$ FLOPs for gates. If HadSkip uses $l$ layers, it needs $(24sH^{2}+4Hs^{2})l+2sH+2H^{2}k+6H^{2}L$ FLOPs for inference one time, where $6H^{2}L$ is the additional computation for gates. In experiments, we use $s=128$. Thus, for a binary classification task, BERT-base needs about 22.3G FLOPs. The HadSkip uses about 5.6G FLOPs, 11.2G FLOPs, 16.8G FLOPs with respect to using 3 layers, 6 layers and 9 layers respectively. They roughly correspond to 1.3x, 2x, 4x inference acceleration. Therefore, compared to the transformer layers, the additional parameters and computational complexity are quite light.\n\n>The paper only reports results on three datasets beyond Table 1, and it would be beneficial to see results on a more diverse range of datasets.\n\nThanks for your valuable suggestion. We conducted experiments using the GLUE benchmark, as recommended in prior research[1]. Due to resource constraints, we selected three representative tasks, namely MRPC, MNLI, and CoLA, for conducting experiments about ablation studies, extending to RoBERTa, and exploring combinations with TinyBERT. However, they have covered three typical tasks of natural language understanding. MNLI serves as a dataset for inference tasks, MRPC for paraphrase tasks, and CoLA for acceptability assessments. Both MNLI and MRPC involve sentence pairs, while CoLA focuses on the comprehension of individual sentences. Additionally, prior research has demonstrated a strong positive correlation between performance on MNLI and that on other understanding tasks[2]. Consequently, we present the model's performance on these three datasets. In future work, we can consider incorporating additional datasets.\n\n>The paper lacks an in-depth analysis of why skip-layer inference works, which could guide us in designing better methods for accelerating inference.\n\nAs stated in [3], each layer of BERT acquires distinct information, with surface features at the lower levels, syntactic features in the middle, and semantic features at the top. Consequently, we contend that for varying input data, the layers playing significant roles exhibit diversity. This constitutes the motivation behind HadSkip's adaptive layer skipping mechanism. Furthermore, we have visualized the layer selection on various datasets in Figures 2-4 to validate our motivation. We posit that this discovery can offer valuable insights into the development of more effective approaches for accelerating inference, such as the adaptive selection of both model weights and layers.\n\n>Can the method be transferred to generative language models like GPT?\n\nThis is a very good point! We are currently researching the extension of HadSkip to generative language models. Our preliminary findings suggest that HadSkip can skip a few layers with minimal performance degradation, for example, 2 or 3 layers on GPT2. Nonetheless, the exploration of how to skip more layers with minimal performance decline remains a subject of ongoing research. We intend to incorporate this aspect into our future work.\n\n[1] Zhang Z, Zhu W, Zhang J, et al. Pcee-bert: Accelerating bert inference via patient and confident early exiting[C]//Findings of the Association for Computational Linguistics: NAACL 2022. 2022: 327-338.\n\n[2]Pruksachatkun Y, Phang J, Liu H, et al. Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work?[C]//58th Annual Meeting of the Association for Computational Linguistics, ACL 2020. Association for Computational Linguistics (ACL), 2020: 5231-5247.\n\n[3] Jawahar G, Sagot B, Seddah D. What does BERT learn about the structure of language?[C]//ACL 2019-57th Annual Meeting of the Association for Computational Linguistics. 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The skip-level inference approach overcomes the problem of the previous early exit method, which can only perform fixed-order inference on the first k layers.\n\n- A new gating network mechanism is proposed to control whether each layer needs to be inferred.\n\n- Empirical results demonstrate the advantages of the proposed method over the baseline method.\n\n- Detailed analytical experiments.", "reasons_to_reject": " - The new gating module introduces additional parameters and computational complexity, but the paper lacks analysis and comparison with the baseline in this regard.\n- The paper only reports results on three datasets beyond Table 1, and it would be beneficial to see results on a more diverse range of datasets.\n- The paper lacks an in-depth analysis of why skip-layer inference works, which could guide us in designing better methods for accelerating inference.\n\n\n\n", "questions_to_authors": "- Issues in the **Reasons To Reject** Part\n- Can the method be transferred to generative language models like GPT?\n\n", "paper_topic_and_main_contributions": "This article studies how to perform skip-layer inference during pre-trained model inference, which is a significant improvement over the previous method of early exit. This work trains a gating network to determine whether each Transformer layer needs to be inferred and sets a maximum budget for inference layers. Experimental results show that under the same budget layer, the method proposed in this paper is significantly better than the baseline method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a skip-computing model which achieved good results on basic benchmark dataset. It's a well written paper and demonstrate good empirical performance. 

Pros:

1. Review work is solid
2. An effective method.
3. Write well and easy to follow.

Cons:

1. The comparison baseline is a bit limited. Despite during discussion period authors add many results and commit they will add to the final version, there is no guarantee this will be happening and the additional information could be integrated well into present materials.

2. Authors are not proactive on adding real-time wall-clock results. Only after many rounds of discussion, they add some statistics compared to Transkimmer only. And it's only speed-up ratio but not wall-clock time which makes the results not verifiable.

3. The comparing models are relatively small.
Decision: Accept-Findings