Length-Adaptive Distillation: Customizing Small Language Model for Dynamic Token Pruning
Authors: Chang Liu
Chongyang Tao
Jianxin Liang
Jiazhan Feng
Tao Shen
Quzhe Huang
Dongyan Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: knowledge distillation
language model compression
token pruning

Abstract: Pre-trained language models greatly improve the performance of various tasks but at a cost of high computation overhead. To facilitate practical applications, there are mainly two lines of research to accelerate model inference: model compression and dynamic computation (e.g., dynamic token pruning). Existing works either adopt these methods individually or simply apply dynamic computation approaches upon a compressed small language model. We argue that they are sub-optimal since the two approaches are separately designed so the compressed model may not be tailored for dynamic computation. To tackle this problem and make compressed small language models faster, we propose Length-Adaptive Distillation, a two-stage knowledge distillation framework that aims to produce a customized small language model for dynamic token pruning. In the general distillation stage, we enforce the student to mimic and reconstruct the teacher's output based on the dynamically pruned representations. Then in the task-specific distillation stage, the student is further accustomed to token pruning while absorbing the task-specific knowledge. Experimental results on GLUE benchmark demonstrate that our method can make the small language model more customized for dynamic token pruning and achieve better speed-performance trade-off.

Length-Adaptive Distillation:
Customizing Small Language Model for Dynamic Token Pruning
Chang Liu 12, Chongyang Tao², Jianxin Liangh4 .Jiazhan Fengl Tao Shen',
Quzhe Huang Dongyan Zhao!2677
Wangxuan Institute of Computer Technology, Peking University
Center for Data Science, Peking University
Microsoft Corporation
4 School of Intelligence Science and Technology, Peking University
AAlI, University of Technology Sydney
Institute for Artificial Intelligence, Peking University
State Key Laboratory of Media Convergence Production Technology and Systems
{ {liuchang97, liangjxx,,,.. fengjiazhan {chotao (enitaa@@@iirosse com Abstract
huangquzhe, Zhaody www.luiyeeezzzzzyyyaaa jepkulyy@ppueeeee edu. tao. shen@uts edu. au
achieved, they generally suffer from high compu- tation overhead, which prevents them from being deployed into resource-scarce applications. Hence, there is an urgent need to accelerate inference speed while minimizing performance degradation.
Pre-trained language models greatly improve the performance of various tasks but at cost of high computation overhead. To facilitate prac- tical applications, there mainly two lines of research to accelerate model inference: model compression and dynamic computation (e.g.. dynamic token pruning). Existing works ei- ther adopt these methods individually or simply apply dynamic computation approaches upon a compressed small language model. We ar- gue that they are sub-optimal since two approaches separately designed so the pressed model may not be tailored for dynamic computation. tackle this problem and make compressed small language models faster, propose Length-Adaptiv Distillation, two- stage knowledge distillation framework that aims to produce customized small language model for dynamic token pruning. In the gen- eral distillation stage, we enforce the student to mimic and reconstruct the teacher's 's output based on the dynamically pruned representa- tions. Then in the task-specific distillation stage, the student is further accustomed to to- ken pruning while absorbing the task- specific knowledge. Experimental results on GLUE benchmark demonstrate that our method can make the small language model more cus- tomized for dynamic token pruning and achieve better speed pererorm-rooomcccc trade-off.
To achieve this goal, great efforts have been made, among which model compression and dy- namic computation are two major lines of research with different mechanisms. Model compression approaches typically convert the large model into smaller one with reduced layer number and hid- den dimension, while the computation graph of the compressed small model fixed during inference. One of the effective approaches to achieve this goal is knowledge distillation from large teacher model to small student model (Jiao et al., 2020a; Wang et al., 2021; Liu et al., 2022a). As for dy- namic computation methods, they don't change the architecture of the given language model and dynamically prune its computation graph during inference instead. There are two perspectives in dy- namic computation for transformer based (Vaswani et al., 2017) language models: depth-level early ex- iting (Liu et al., 2020; Xin et al., 2020; Zhou et al., 2020) and width-level token pruning (Goyal et al., 2020; Kim and Cho, 2021; Kim et al., 2021). We focus token pruning methods in this paper.
Since these two groups of methods achieve in- ference acceleration from two orthogonal perspec- tives, a natural idea is to combine them together to obtain faster inference speed. The simplest way to combine them is the pipeline approach (Kim and Cho. 2021; Guskin al., 2021, 2022): preparing a compressed small language model first and apply- ing dynamic computation algorithm upon it next. Though proved effective, we argue that the pipeline approach is sub-optimal since model compression
1 Introduction
With the rapid progress of pre-trained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Brown et al., 2020), dramatic improvement has been achieved in a wide range of natural lan- guage understanding and generation tasks. De- spite the remarkable performance those PLMs have
Corresponding author: Dongyan Zhan.

---

and dynamic computation procedures are individu- ally designed and may not be compatible with each other that much. As a result, the potential of such an approach has not been fully exploited.
vanced knowledge distillation methods on stan- dard setting (i.e., without token pruning). More- over, we prove that with the help of the customized -agtth-adpiivee distillation, our methods success- fully take advantage of both model compression and dynamic computation, and achieve dramati- cally better with the pipeline approach. We release our imple- mentation to facilitate future research'.
To address this issue and achieve better speed-performances trade-off, we propose a novel knowledge distillation framework named Length- Adaptive Distillation (abbr. LAD) that transfers the knowledge from the large language model to small language model while helping the small model to get adapted to dynamic token pruning. Drawing inspirations from Jiao et al. (2020b), we adopt two-stage distillation paradigm where a general-purpose small language model is first dis- tilled on large-scale general corpora and then task- specific knowledge is injected on task-specific datasets. In general distillation, we mainly con- sider two issues: (1) how to transfer high-quality general knowledge from the teacher and (2) how customize the student model for dynamic token pruning. For (1), we propose to transfer token- level knowledge of two types: hidden represen- tations that encode the contextual semantic infor- mation of tokens and attention dependency that is generally adopted as the indication of token im- portance (Goyal et al., 2020; Kim and Cho, 2021) for token pruning. The alignment of hidden repre- sentations between the teacher and the student is achieved by contrastive distillation and the match- ing of attention dependency fulfilled by mean square error. While for (2), we employ a teacher with an unpruned computation graph to teach a student with dynamic token pruning where the im- portance of tokens is measured by the attention scores, and the retention configuration is sampled from a uniform distribution. As only the unpruned tokens have their representations in the student's last layer, we train the student by enforeing the remaining token representations to be close to the teacher's s corresponding representations as well as to reconstruct the complete token representations of the teacher's. Then in task-specific distillation, we first employ data augmentation to enlarge the task datasets, then distill the student following the dynamic token pruning setting used in general dis- tillation but transfer more task-relevant knowledge by enforcing the sentence embedding of the student to be close to the teacher via contrastive distillation.
trade-off compared
To sum up, our contributions are three folds:
We propose a two-stage knowledge distillation framework LAD that effectively combines model compression and dynamic computation to achieve faster speedup in inference.
We customize small language model for dy- namic token pruning by cast length-adaptive set- ting the student and enforcing it to mimic and reconstruct the teacher's is representations.
We conduct comprehensive experiments on the GLUE benchmark and verify that our method can achieve a superior pered-performacce trade-off compared with other methods.
2 Related Work
2.1 Language Model Compression
There are various methods compress a large language model into small one including prun- ing (Fan et al., 2019; Gordon et al., 2020), quan- tization (Zafrir et al., 2019; Shen et al., 2020; Bai et al., 2021). weight sharing (Dehghani et al., 2018; Lan et al., 2019), knowledge distillation (Hinton et al., 2015; Sanh et al., 2019; Jiao et al., 2020b) and so on. We focus on knowledge distillation in this paper, where large model acts as the teacher and transfers its knowledge to a smaller student model. Jiao et al. (2020b) proposed ceneral-thent task specifcc distillation framework and transferred the knowledge in hidden states, attention matri- ces, and output logits in different distillation stages. Wang et al. (2020b, 2021) matched the attention dependencies derived from the query, the key and the value vectors in the self-attention module. Sun et al. (2020); Fu al. (2021) employed contrastive distillation to match the hidden states. Park et al. (2021); Liu et al. (2022a) structured the knowledge as the relations of hidden states and enforced the relations between the teacher and the student to be consistent. Compared with existing works, our method not only addresses the issue of improving the performance of small language models with
We conduct experiments on GLUE bench- mark (Wang et al., 2018). We first prove that our knowledge distillation method outperforms ad-
htttp::/ thub. Somm/EMLL-LAD/LAD

---

fixed computation graphs they did, but also en- dows the small language model with good adapta- tion ability for dynamic token pruning to achieve improved inference efficiency.
how apply token pruning upon our model.
3.1 General Distillation
General distillation aims to transfer the general knowledge held in pre-trained language model (i.e., RoBERTa to a randomly initialized small language model that has fewer layers and hidden dimensions with large-scale open-domain corpora (i.e., Wikipedia). Existing works (Sanh et al., 2019; Jiao et al., 2020b; Sun et al., 2020; Wang et al., 2021) have designed various effective algorithmss by considering different knowledge sources, knowl- edge types, distance metries, etc. However, they all focus on improving the evaluation performance of the student model in downstream tasks with fixed computation graphs. When directly adapting ex- isting models dynamic token pruning (Kim and Cho, 2021), we find that these models get poor performance when we adopt an aggressive prun- ing ratio. Therefore, to effectively combine model compression and dynamic token pruning to achieve a better speed-performance trade-off, we propose novel distillation algorithm that not only trans- fers high-quality knowledge the student but also helps the student get adapted to dynamic token pruning, especially to a high-pruning extent. We first introduce what form of knowledge we prepare to transfer and how to transfer, then we present how to effectively get the student adapted to dynamic token pruning.
2.2 Dynamic Computation
Different from language model compression meth- ods that produce a small model with fixed compu- tation graph, dynamic computation approaches fix the model architecture and achieve inference ac- celeration by dynamically pruning its computation graph (Han et al., 2021). Liu et al. (2020); Xin et al. (2020); Zhou et al. (2020) studied depth-level early exiting where samples with different diffi- culty are output from different layers (i.e., the more difficult the sample is, the more layers it would go through before getting its final output). While (Goyal et al., 2020; Kim and Cho, 2021; Kim et al.. 2021; Ye al., 2021; Guan et al., 2022; Modarressi et al., 2022: Liang et al., 2023) proposed width- level token pruning where unimportant tokens are progressively removed as the calculation goes from shallow to deep layers. Among the two lines of research, we focus on token pruning approaches. Recently, Guskin et al. (2021, 2022) explored the combination of dynamic token pruning and model compression to make more aggressive acceleration by simply applying token pruning upon a given compressed language model in a pipeline manner. Different from these works, we highlight the im- portance the adaptation from fixed computation graph to dynamic one for small language models, and propose knowledge distillation framework to fulfill this goal.
3.1.1 Knowledge Transfer
One of the vital problems in knowledge distilla- tion is knowledge transfer. In transformer distil- lation, hidden representations (Jiao et al., 2020b), attention dependencies (Wang et al., 2020b), rela- tions among representations (Park et al., 2021) all have been considered useful knowledge to be transferred. Drawing on existing works, we first consider the hidden representations as the funda- mental knowledge that mainly guides the learning of a student. Besides, considering that a majority of dynamic token pruning approaches (Goyal et al., 2020; Kim and Cho, 2021; Kim et al., 2021) have demonstrated that the attention scores derived from attention matrices of self- attentttnion mechanism can act good indicator for token importance. There- fore, to produce customized student model for token pruning, we additionally involve attention dependencies as the knowledge be transferred.
3 Methodology
Our framework adopts two-stage knowledge dis- tillation paradigm (Jiao et al., 2020b) where ran- domly initialized small language model is first dis- tilled with general knowledge, and is then taught with task-specific knowledge. Along with the dis- tillation process, the student not only absorbs the knowledge transferred from the teacher, but also gets adapted to dynamic token pruning. After the two distillation stages, we employ evolutionary scarch (Cai al., 2019; Wang et al., 2020a; Kim and Cho, 2021) obtain a set of length configu- rations for token pruning with different speedups. In the following subsections, we first introduce our knowledge distillation algorithms in general and lssk-specific distillation stages, then demonstrate
To elaborate on our design in knowledge trans- fer, we first briefly describe the calculations of the

---

LeD XlCori + Ca + Aahrect
Layer
Layer
Layer L.
Layer L.-1
Layer L,X
Layer
Layer 1
Layer
Layer
X
Student w/o Pruning
Teacher
Student w/ Pruning
Figure I: Illustration of general distillation in LAD. For an input text, the student has two branches of calculation: (1) the standard forward pass without token pruning on the left and (2) the forward pass with token pruning on the right. For the left part, we align the output of its last layer the teacher's penultimate layer using Cori- For the right part, we not only align the student's ' remaining representations after token pruning to the corresponding representations of the teacher the penultimate layer using Latigns but also encourage the remaining representations reconstruct the representations of the whole input with the help of the teacher's last layer as decoder using Lree-
transformer model together with some necessary notations. Formally, given a sequence of input text, a sub-word tokenizer first splits it into n to- kens r [Wi, W2,................... ... Wn]. Then through an em- bedding layer, each token in æ is converted as dense vector through lookup table, resulting in H [h]. ne....... ... nol where h! IRd. Then, the output of the embedding layer passed to L stacked transformer (Vaswani et al., 2017) layers which are mainly comprised of a multi-head self- attention module and a position-wise feed-forward network. The self-attention module produces the attention matrices A' E hxxxx that encodes the dependencies among the input tokens using h atten- tion heads. The output of the I-th layer is denoted H [h4 h4,,.............. .., h.)]
grained knowledge. Drawing inspirations from contrastive learning (Chen et al., 2020; He et al., 2020; Tian et al., 2019), we design contrastive objective to achieve this goal. For a hidden repre- sentation of the student h2.- we first conduct linear projection to get 2 E IRd that have the same hid- den dimensions as the teacher model. The positive representation that it is enforced match is the corresponding teacher's representation hi;- while there remain multiple choices for the negatives. Different from previous works that utilize the rep- resentations of tokens in the same input of text as negatives (Tao et al., 2022), we found that simply using randomly sampled token representations per- forms better. As demonstrated by previous works that contrastive learning requires large number of negatives, employ memory queue (He et al., 2020) M )1) with memory size avoid the unaffordable memory cost brought by in-batch negatives with enlarged batch size (Chen et al., 2020). Then the token-level contrastive dis- tillation objective for x is formulated as:
With these notations, we now go back to the knowledge distillation setting. Given teacher model with L, layers and student model with T.g layers, feed the same text into them and obtain the corresponding output hidden states be (H) and attention matrices // (1))), [L We suppose the student's 1y-th layer is aligned with the teacher's 44-th layer, then the outputs of the student (i.c., H!!- and A!) should be close the teacher's (i.e., H and A For aligning hidden states, instead of aligning sentence- level representations (Sun et al., 2020), we assume that token-level representations are more suitable for general distillation since they contain more fine-
/ix
10
log exp(s(2!,, h/.)/T) hi
Thid
EnLeeM EM exp(s(z)) h² )/T)
(I)
where is the temperature and s(-) denotes cosine similarity. Different from He et al. (2020) that using momentum encoder to make the stored rep- resentations stable and consistent during training,

---

we find that directly using the fixed teacher's repre- sentations without the help of momentum encoder yield better performance as the representations are naturally consistent. While for aligning attention dependencies, we follow Jiao et al. (2020b) to op- timize the mean square enror (MSE) between the attention matrices of the teacher and the student:
first one encourages the remaining representations of the student to recover the representations of the whole input. To achieve this goal, we first con- struct the masked imput which is the concatenation of two parts: (1) the remaining representations in the student's top layer after pruning, and (2) the query of the pruned representations constructed by adding the positional embedding of the pruned po- sitions and the embedding of the mask token [M] token. The masked input is linearly projected fit the hidden dimensions of the teacher. Then, we need to reconstruct the representations based on the masked input through a decoder. Instead initializing a new decoder to predict the pruned tokens (Liu and Shao, 2022), we propose to take advantage of the teacher model by borrowing its last transformer layer the decoder. This design not only enjoys the good ability of representation learning of the pre-trained teacher but also reduces the number of parameters that need to optimized. The borrowed decoder is frozen during the training of the student. We denote the reconstructed outputs as H and As, and form the reconstruction loss as:
Jut MSE A A
The overall objective for knowledge transfer is:
J
A H A -hha H Hi:)) +Juu A.
(3)
3.1.2 Length-Adaptive Distillation
Based on the design of knowledge transfer, we then focus the second goal: how to effectively get the student accustomed to token pruning while transferring general knowledge. We achieve this goal by employing a teacher without token pruning to teach the student with dynamic token pruning.
Token Pruning In the standard setting of dy- namic token pruning, there are two major issues: how to measure the importance of tokens in each layer and how decide the number of tokens main- tained in each layer. We follow the typical solution where the importance measurement is the sum of attention scores a token received from other tokens within a sentence (Goyal et al., 2020) and the length configuration for token pruning is obtained by sam- pling from a pre-defined range in training and using cvolutionary search (Cai et al., 2019; Wang et al., 2020a; Kim and Cho, 2021) in inference. Given an input sequence, we first sample length con- figuration N [n1, . n.,,] for the student with ratio r with the sampling strategy proposed by Kim and Cho (2021) where n, denotes the num- ber of tokens maintained in the I-th layer of the student. Hereby, the output of the student on its top layer only contains nt. token representations fi.. = [h! fe...... ...., h and the correspond- ing attention matrices A formed by them, where {21, 2,...., )nz,) is the original indices of the remaining tokens. Based on 2, we can also extract the corresponding teacher's hidden states in the teacher's 4-th layer f from H and then caleulate the attention matrices Â! with them.
JH, !!
(4)
The second one is to align the remaining repre- sentations of the student with the corresponding ones of the teacher. Notice that in Eq. 3.1.2 we feed the representations student to the last layer of the teacher, implicitly aligning the student's 's La-th layer with the 1:-1-th layer. Therefore, we form the alignment loss as:
the Ly-th layer the
Calign J (EL A
A A
(5)
In addition to the two objectives that aim get the student accustomed token pruning, we also transfer the knowledge without token pruning stabilize training:
= - A H A
(6)
The overall objective for general distillation is the weighted sum of the three objectives:
LeD Xy + Xz XrLroe-
(7)
We name the model produced by general distilla- tion as LAD
3.2 Task-Specific Distillation
Training Based on the pruned student's repre- sentations, we propose two objectives that aim to get the student accustomed to token pruning. The
Based on the model LADGD produced by general distillation, we inject task-specific knowledge with

---

the task-specific teacher, while further improving the customization for dynamic token pruning under the prediction mode of the task (e.g., classification). This distillation stage is on the training set of a downstream task. As the labeled task data is usu- ally far less than general corpora, we follow Jiao et al. (2020b); Liu et al. (2022b) to conduct data augmentation (DA) to enlarge the task data. We fo- cus on classification tasks in this paper, therefore in task-specific distillation, we focus on transferring the knowledge held in the start token (e.g., 
 in RoBERTa) instead of considering all tokens. Sim- ilar to general distillation, we sample the length configuration for token pruning with the same ra- tio as general distillation for the student while keeping the teacher unpruned. The (projected) rep- resentations of the start token of the student and the teacher are denoted as 2 and h We employ contrastive distillation to transfer the knowledge with the help of the memory queue:
ments, we manually select one length configuration that achieves good accuracy as well as considerable speed-up from S and report the results.
Experiment
4.1 Datasets and Metrics
We conduct experiments on generally adopted tasks from GLUE benchmark (Wang et al., 2018) following previous work (Jiao et al., 2020b; Wang et al., 2021), including single sentence tasks:SST- (Socher et al., 2013), CoLA (Warstadt et al., 2019), and 6 text pair tasks: MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), RTE (Ben- tivogli al., 2009), MRPC (Dolan and Brockett, 2005), STS-B (Cer al., 2017), QQP (Chen al., 2018). As for performance metries, we report Spearman's 's rank correlation coefficient (Spear) on STS-B, Matthews correlation coefficient (Mce) on CoLA, and accuracy (Acc) on the other tasks. To evaluate the inference efficiency, we use FLOPs as the metric and employ torchprofile 2 as the tool to calculate following Kim and Cho (2021).
LTD = -log exp(s(2 //T)
(8)
EnjeM exp (5 (2//, hz)/T)
4.2 Implementation Details
After distillation with Eq. 3.2, we fine-tune the model to finally adapted it to downstream tasks. We denote the model produced by task-specific dis- tillation as LADTD LADTD w/ DA depending on whether the training set of the task is augmented.
We adopt RoBERTa-base (Liu et al., 2019) the teacher model and utilize small transformer model with 6 layers, 384 hidden dimensions and 12 attention heads as the student following Wang et al. (2021). We conduct two-stage knowledge dis- tillation. In general distillation, we prepare English Wikipedia and BookCorpus (Zhu et al., 2015) as the training corpora and set the max sequence length as 128. We set the size of memory queue = 16384, the temperature
3.3 Configuration Search for Token Pruning
Given small student model produced by our two-stage knowledge distillation framework, ac- celeration through dynamic token pruning can be achieved given a length configuration A R2,,,......... nI.] where nu denotes the number of remained tokens in layer I. Naturally, there is trade-off between accuracy and efficiency with various choices of N and we need to find a ries of optimal length configurations to facilitate various application scenarios. We achieve this goal by conducting evolutionary search (Cai et al., 2019) following Kim and Cho (2021). Specifi- cally, we first initialize the candidate set of length configurations by random sampling, then itera- tively construct new configurations by mutation and crossover operations and update the candi- date set if some newly constructed configurations achieve better accuracy-ede -efuracc-ffficiency trade-off. After searching, we obtain set of optimal length config- urations S = {Wi .NG....,, ,,,r that lies in the accuracy-efficiency Pareto frontier. In our experi-
0.07, the weights for different loss terms A = 1.0, A2 X 0.5. We randomly sample the token pruning ratio r from (0.1, 0.2. 0.7) for each training instance for adapthe distillation. We use Adam (Loshchilov and Hut- ter, 2017) as the optimizer and train the student model with a batch size of 256, the learning rate as 6e-4, the maximum training steps as 400k, and the warmup ratio as 0.01. In task-specific distillation. prepare two types of datasets: the original train- ing set of each task, and their augmentation version where each sample is augmented using contextual world replacement with the same data augmenta- tion setting of Jiao et al. (2020b). The hyperparam- eters m, t and r are the same as general distillation. When using the original training sets, we train the student with a batch size of 32, the learning rate
Pht / thub. www/zhijaaaaaaeaaaaaacccccaiiaaa liu/torchprofile.

---

SST-2
MRPC
RTE
STS-B
MNLI-m
QNLI
QQP
CoLA
Model
Acc
St
Acc
St
Acc
St
Spear
St
Acc
St
Acc
St
Acc
St
Mcc
St
Pre-triined Language Models without Token Pruning
BERT
92.8
0.2x
90.3
0.2x
0.2x
0.2x
91.3
0.2x
91.2
0.2x
56.8
RoBERTaause
94.8
0.2x
0.2x
92.8
0.2x
91.9
0.2x
General Distillation Models without Token Pruning
Tiny BERT
88.7
1.0x
87.3
1.0x
67.2
1.0x
88.1
1.0x
81.0
1.0x
89.1
1.0x
90.0
1.0x
36.5
1.0x
MiniLMv2
90.9
10x
86.8
10x
66.1
1.0x
88.1
1.0%
81.9
1.0x
89.6
1.0x
90.3
1.0x
40.7
1.0x
LAD
91.6
1.0x
88.7
1.0x
67.9
1,0x
88.5
1.0%
82.3
1.0x
90.2
1.0x
90.7
1.0x
41.0
1.0x
LAD
93.7
1.0%
89.2
1.0x
68.6
1.0x
88.2
1.0%
84.8
1.0%
91.1
1.0%
91.7
1.0%
44.7
1.0x
LADrD w/ DA
93.5
1.0x
90.2
1.0x
69.7
1.0x
89.4
1.0x
86.2
1.0x
91.5
1.0%
91.7
1.0%
50.9
1.0x
Gencral Distillation Models with Token Pruning
TinyBERT
88.5
5.4x
84.7
3.3x
52.3
6.4x
87.3
3,6x
80.5
2.6x
88.0
2.8x
89.7
3,3x
35.1
8,9x
MiniLMv2
89.5
5.7x
84,5
3,4x
57.7
6.2x
87.6
3.6x
81.0
2.7%
88.3
2.8x
90.2
3.5x
40.3
8.6x
LADeD
90.4
5.8x
87,5
3.5x
63.5
7.8x
88.1
3,7x
81.5
2.8x
89,5
2.8x
90.5
3.8x
40.8
8,7x
Task specifcc Distillation / Adaptation Models with Token Pruning
TinyBETT w/ Ada
88.6
6.2x
85.1
4.0x
60.6
6.3x
87.4
3.2x
80.0
2.7x
88.1
3.2x
89.7
4.0x
36.7
8.4x
MiniLMv2 Ada
91.2
6.3x
85.3
4.1x
59.6
6.3x
87.6
3.7x
81.2
2,8x
88.1
2.9x
90.2
3.7x
39.1
8.2x
LADTD
93.0
6.1x
87.0
4.1x
5,8x
87.9
4.0%
84.2
4.1x
90.2
3,9x
91.4
5.9x
44,8
8,9
LADID DA
93.3
7.2x
89.9
4.9x
67.1
8.0x
4.9%
85.6
4.5x
90.2
4.3x
91.4
6.1x
47.6
8,9x
Table 1: Performance and speedup (S1) on GLUE benchmark. LADGD denotes the model produced by gen- eral distillation, LADrn denotes the model produced by general and task-specific distillation, DA denotes data augmentation for the task dataset, and Ada denotes the adaptive training baseline benchmarking against LADrD-
as 3c-5, the maximum training epochs as 50 for CoLA and 20 for other tasks, and the warmup ra- tio as 0.1. When on the augmented training sets, the batch size is 256, the learning rate is Ic-4, the warmup ratio is 0.06, and the maximum training epochs are the same as on the original training sets, the following fine-tuning stage, we choose the learning rate from (1e-6, 2c-6, 36-6) and the batch size from { 11, 32). For dynamic token pruning, we use the same configuration of evolutionary search Kim and Cho (2021).
proach which adapts a given language model to dynamic token pruming. Following pipeline ap- proaches (Guskin et al., 2021, 2022) that apply dynamic token pruming upon compressed small lan- guage models, we adopt the sandwich rule and inplace distillation (Yu and Huang, 2019) used in LAT (Kim and Cho, 2021) as the baseline (denoted as Ada) to be compared with our task-specific dis- tillation method.
4.4 Overall Performance
We provide the overall evaluation results of the performance and the speedup of our method and baselines in Table 1. First, it can be observed from the second block that under standard evaluation set- ting (i.e., without token pruning), our model with general distillation (i.e., LADşp) consistently out- performs advanced general distillation methods on all tasks. With task-specific distillation and data augmentation, the performance can be further im- proved. Then we compare the performance gen- erally distilled models under the same token prun- ing setting in the third block. We find that LAD@D can still outperform baselines on both performance and inference speedup under the selected pruning configurations. Finally, we compare the perfor- mance of our task-specific distillation method with existing pipeline approaches with adaptation to- ken pruning in the bottom block. It can be seen that LADTp not only achieves better performance
4.3 Baseline Methods
Baselines for General Distillation The first group of baselines is pure model compression meth- ods using advanced knowledge distillation tech- niques. We implement TinyBERT (Jiao et al., 2020b) and MiniLMv2 (Wang et al., 2021), two representative knowledge distillation methods, un- der the same distillation setting (i.e., model size, training data, optimization hyperparameters, etc.) ours. We compare two perspectives with these baselines: (1) the standard evaluation of model performance (i.e., fine-tuning on each task and testing without token pruning) and (2) the speed- performance trade-off under dynamic token prun- ing without any types of adaptation.
Baselines for Task-Specific Distillation The second type of baseline is the adaptation ap-

---

90
90
80
80
MI
ACCUrDIY
ACCurDEy
8
70
8
LAB
160
LAD LAGLID
MRRC
LAG_IB
ONLI
60
WD
LAD
(a) MRPC
RTH
QNLI
(d) MNLI
Figure 2: The speed-performance curves on four tasks.
compared with LADGD but also outperforms base- lines. With data augmentation, achieve dramati- cally better performance, where we achieve 6. speedup while improving the performance by 2.1% on average compared with static inference without token pruning using LADGD.
80
Accuracy
LAD LAD_TD, r-0.3
70
TD, r-0.5
LAD. TD DA. r-0.3
4.5 Discussions
60
MRPC
LAD, TD w/ DA, r-5.5
LAD r-0.7
There are several critical designs in our frame- work. We analyze these designs by drawing speed- performance curves under different settings four representative tasks. We choose two low-resource tasks MRPC and RTE, one moderate-resource lask QNLI, and one high-resource task MNLI.
FLOPs
Figure 3: The speed-performance curves of different pruning ratio on MRPC.
General Distillation In general distillation, we introduce three objectives Cori, Calign and Lree weighted by X1, A2, A3 to jointly transfer general knowledge and get the student accustomed to to- ken pruning. We first study the influence of differ- ent choices of these weights and find that setting A = X2 + A and X22 = > yield the best perfor- mance. Furthermore, we study the effectiveness of these objectives and plot Figure 2. Here we focus on the curves corresponding to the top 5 la- bels on the legend. Among these models, there are two models that are not obtained from specifically designed training for token pruning: MiniLMv2 and LADcD w/o align.rec. Compared with these baselines, we find the introduction of Calign and Cree both improve the performance under to- ken pruning and the combination of them performs better, verifying that our proposed length-adaptive distillation effectively help the student get accus- tomed to token pruning. It can also be observed that the improvement of our general distillation method over baselines under token pruning is more considerable on low-resource tasks (i.e., MRPC and RTE). The reason lies in that the fine-tuning steps on high-resource tasks are much more than on low-resource tasks, weakening the fitness for token
pruning learned by our general distillation. This issue can be well addressed by our task-specific distillation algorithm discussed in the following.
Task-Specific Distillation Based on the gen- erally distilled model, we further conduct task- specific distillation on the training sets down- stream tasks with or without data augmentation (DA). From Figure 2 we find that task-specific dis- tillation brings substantial improvement for all the tasks to different extent. For low-resource tasks (i.e., MRPC and RTE), although LADrn signifi- cantly outperforms the baselines trained with no specification for token pruming (i.e., MiniLMv2 and LAD w/o align,rec), makes slight improve- ment over LADGD due to limited task data. Hence with data augmentation, LADID w/ DA performs dramatically better than LADTD. While for tasks with moderate abundant amounts of instances (i.c., QNLI and MNLI), find that LADTD brings considerable improvement and data augmentation is sort of the icing on the cake. These findings verify the effectiveness of task-specific distillation and prove that data augmentation is a solution to the data scarcity issue on low-resource tasks.

---

The Choice of Pruning Ratio Recall that in both general and lask-specific distillation, the student is applied with token pruning with ratio r. To explore the effect of the ratio as well as to find the best choice, we adopt r E {0.3, 0.5, 0.7) in both gen- eral and task-specific distillation on MRPC. It can be observed from Figure 3 that the larger is, the better speed-performenn performance trade-off can be achieved whether the training set is augmented or not. More surprisingly, we also find that training with larger pruning ratio brings consistent improvement at all speedups, which indicates that token pruning can be considered a kind of regularization in distillation that helps the student learn better.
student while customizing the student for dynamic token pruning. Our method doesn't 't introduce ethi- cal issues. The datasets we used are publicly avail- able and don't 't have any privacy issues.
References
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. 2021. BinaryBERT. Pushing the limit of BERT quan- tization. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joins Conference on Natu- ral Longuage Processing (Volume I: Long Papers), pages 4334-4348, Online, Association for Computa- tional Linguistics.
5 Conclusion
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In TAC.
In this paper, we propose a two-stage knowledge distillation framework LAD that transfers general and task-specific knowledge to the student while helping the student to get adapted dynamic token pruning. We conduct comprehensive experiments on the GLUE benchmark. The evaluation results prove that our method can effectively take advan- tage of model compression and dynamic compu- tation and achieve a superior ppeed-performance trade-off for inference acceleration.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models few-shot learners. Advances in neural information processing systems, 33: 13:1817-1
Han Cai. Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. 2019. Once-for-all. Train one net- work and specialize for efficient deployment. arXiv preprint arXiv: Settv:7908.000 7908.09791.
Limitations
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop Semantic Evaluation (SemEval-2011 pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
We achieve superior speed-performance trade-off in inference acceleration by a two-stage knowledge distillation framework. In the first general distilla- tion stage, in order to jointly transfer the general knowledge and get the student accustomed to dy- namic token pruning, we introduce two calculation branches for the student. This design implies that the student needs to do two forward passes in one training iteration, increasing the computation over- head in training. In the future, we plan to explore how to unify the two computation branches to im- prove training efficiency.
Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton, 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607, PMLR.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi 7hao. 2018. Quora question pairs. URL https://www. kaggle, comlclquone-question-pe
Acknowledgements
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2018. Universal transformers. arXiv preprint arXiv: 1807.038..00
We would like to thank the anonymous re- viewers for their constructive comments. This work was supported by the National Key Re- search and Development Program of China (No. 2020AAA0106600),
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ethical Statement
In this paper, we propose a novel knowledge distil- lation framework that transfers knowledge to the

---

William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing ( (IWP200S).
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020b. TinyBERT:
Distilling BERT for natural language un- derstanding. In Findings of the Association for Com- putational Linguisties: EMNLP 2020, pages 4163- 4174, Online. Association for Computational Lin- guistics.
Angela Fan, Edouard Grave, and Armand Joulin. 2019. Reducing transformer depth on demand with struc- tured dropout. arXiv preprint arXiv: 11556.
Gyuwan Kim and Kyunghyun Cho. 2021. Length- adaptive transformer: Train once with length drop. use anytime with search. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 1 1ith International Joint Confer- ence on Natural Language Processing ( (Volume 1: Long Papers), pages 6501-6511, Online. Association for Computational Linguistics.
Hao Fu, Shaojun Zhou, Qihong Yang. Junjie Tang, Guiquan Liu, Kaikui Liu, and Xiaolong Li. 2021. Lre-bert: latent-representaaiion contrastive knowledge distillation for natural language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35. pages 12830-12838.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing BERT: Studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP. pages 143 155, Online. Association for Com- putational Linguisties.
Sehoon Kim. Sheng Shen, David Thorsley, Amir Gho- lami, Woosuk Kwon, Joseph Hassoun, and Kurt Keutzer. 2021. Learned token pruning for transform- ers. arXiv preprint arXiv:2107.00910.
Zhenzhong Lan, Mingda Chen. Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sorieut. 2019. Albert: lite bert for self-supervised learn- ing of language representations. arXiv preprint 19Xiv:990.119999
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sabharwal, and Ashish Verma. 2020. Power-bert: Accelerating bert inference via progressive word-vector elimina- tion. In International Conference on Machine Learn- ing, pages 3690-3699, PMLR.
Jianxin Liang, Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao. 2023. Attend, select and eliminate: Accelerating multi-turn response selec- tion with bual-attention-base content elimination. In Findings of the Association for Computational Linguistics: ACL 2023. pages 6758-6770, Toronto, Canada. Association for Computational Linguistics.
Yue Guan, Zhengyi Li, Jingwen Leng, Zhouhan Lin, and Minyi Guo. 2022. Transkimmer: Transformer learns to layer-wise skim. In Proceedings of the 60th Annual Meeting the Association for Computational Linguistics (Volume 1: Long Papers), pages 7275- 7286, Dublin, Ireland. Association for Computational Linguistics.
Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao. 2022a. multi-granuulrityy structural knowledge distillation for language model compression. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 1001-1011, 1, Dublin, Ireland. Associa- tion for Computational Linguistics.
Shira Guskin, Moshe Wasserblat, Ke Ding, and Gyuwan Kim. 2021. Dynamic-tinybert: Boost tinybert's infer- ence efficiency by dynamic sequence length. arXiv preprint arXiv:2111.09645.
Shira Guskin, Moshe Wasserblat, Chang Wang, and Hai- hao Shen. 2022. Quala-minilm: quantized length adaptive minilm. arXiv preprint arXiv:2210. 1..0..771 171 14.
Chang Liu, Chongyang Tao, Jianxin Liang, Tao Shen, Jiazhan Feng, Quzhe Huang, and Dongyan Zhao. 2022b. Rethinking task-specific knowledge distilla- tion: Contextualized corpus as better textbook. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10652-10658, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Yizeng Han, Gao Huang, Shiji Song, Le Yang. Honghui Wang, and Yulin Wang. 2021. Dynamic neural net- works: A survey. IEE. Transactions Pattern Analysis and Machine Intelligence.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsu- pervised visual representation learning. In Proceed- ings of the IEEF/CVF conference computer vision and pattern recognition, pages 9729-9738.
Weijie Liu, Peng Zhou, Zhiruo Wang. Zhe Zhao Haotang Deng, and Qi Ju. 2020. FastBERT: self- distilling BERT with adaptive inference time. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6035- 6044, Online. Association for Computational Lin- guistics.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint 1r.Xiv155030000
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020a. Tinybert: Distilling bert for natural language under- standing. Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 4163-4174.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.116199

---

Zheng Liu and Yingxia Shao. 2022. Retromae: Pre- training carieval-oriented transformers via masked auto-encoder. arXiv preprint 22X.v:205.12000
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive representation distillation. arXiv preprint arXiv: 1rXiv:1910.10699.
I. Loshchilov and F. Hutter. 2017. Fixing weight decay regularization in adam. ArXiv, abs/1771.055001
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998 6008.
Ali Modarressi, Hosein Mohebbi, and Moham- mad Taher Pilehvar. 2022. AdapLeR: Speeding up inference by adaptive length reduction. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics Voolume Long Pa- pers), pages 1-15, Dublin, Treland. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael. Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355, Brussels, Belgium. Association for Com- putational Linguistics.
Geondo Park, Gyeongman Kim, and Eunho Yang. 2021. Distilling linguistic context for language model com- pression. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 364-378, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hanrui Wang. Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. 2020a. HAT: Hardware-aware transformers for efficient nat- ural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 7675-7688, Online, Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Pency Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Wenhui Wang. Hangbo Bao, Shaohan Huang, Li Dong. and Furu Wei. 2021. MvniLMv2: Multi-head self- attention relation distillation for compressing pre- trained transformers. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2 2140-2151, Online, Association for Computa- tional Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.01 108.
Sheng Shen, Zhen Dong. Jiayu Ye, Linjian Ma, Zhewei Yao. Amir Gholami. Michael W Mahoney, and Kurt Keutzer. 2020. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 34, pages 8815-8821.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. 2020b. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained tramsformers. Advances in Neural In- formation Processing Systems, 33:5776-577 5788.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013, Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1644 Seattle, Washington, USA. Association for Computational Linguistics.
Alex Warstadt, Amaupreet Singh, and Samuel Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 641.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long Papers), pages 1112-1122 New Orleans, Louisiana. Association for Computational Linguis- tics.
Siqi Sun, Zhe Gan, Yuwei Fang. Yu Cheng, Shuohang Wang, and Jingjing Liu. 2020. Contrastive distil- lation on intermediate representations for language model compression. In Proceedings of the 2020 Con- ference Empiricul Methods in Natural Language Processing (EMNLP), pages 498-508, Online. Asso- ciation for Computational Linguistics.
Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. 2020. DeeBERT: Dynamic early exiting for accelerating BERT inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2246-2251, 1nniinn Association for Computational Linguistics.
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang. Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. 2022. Compression of generative pre- trained language mod- els via quantization. nual Meeting of the Association for Computational Linguistics \Volume 1: Long Papers), pages 4821- 4836, Dublin, Ireland. Association for Computational Linguistics.
Proceedings of the 60th An-
Deming Ye, Yankai Lin, Yufei Huang, and Maosong Sun. 2021. TR-BERT: Dynamic token reduction

---

for accelerating BERT inference. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5798-5809, Online. Association for Computational Linguistics.
Jiahui Yu and Thomas S Huang. 2019. Universally slimmable networks and improved training tech- niques. Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 1803- 1811.
Ofrr Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Fifficient Machine Learning and Cognitive Computing-NeurlPP Edition (EMC2-NIPS), pages 36-39, IEEE.
Wangchunshu Zhou, Canwen Xu, Tao Ge, Julian McAuley. Ke Xu, and Furu Wei. 2020. Bert loses patience: Fast and robust inference with early exit. Advances Neural Information Processing Systems, 33: 1330013--83..
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE in- ternational conference on computer vision, pages 19-27.


Reviews:
[{"review_id": "PIotVsvOvV", "reviewer": "Reviewer_42mY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments and will make every effort to address the raised concerns.\n \nRe `` The experiments’’:\n\nThe focus of our experiments is to validate the effectiveness of the proposed knowledge distillation algorithm, LAD. Therefore, in alignment with numerous prior works [1, 2, 3], we utilized a base-sized model (i.e., Roberta-base) as the teacher and conducted extensive experiments on all eight datasets in the GLUE benchmark. The experimental results substantiate the effectiveness of our proposed algorithm. We also consider compressing larger language models as a promising research direction and have plans to study knowledge distillation on larger models in the future.\n \nHope the above explanation may address your concern.\n \n**References**\n\n[1] Jiao X, Yin Y, Shang L, et al. Tinybert: Distilling bert for natural language understanding[J]. arXiv preprint arXiv:1909.10351, 2019.\n\n[2] Li L, Lin Y, Ren S, et al. Dynamic knowledge distillation for pre-trained language models[J]. arXiv preprint arXiv:2109.11295, 2021.\n\n[3] Wu S, Chen H, Quan X, et al. AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression[J]. arXiv preprint arXiv:2305.10010, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of simultaneously considering model compression and token pruning is something that hasn't done by previous methods.\n\nBetter performance is obtained according to the experiments, illustrating the effectiveness of the proposed method.\n\nAblation studies are conducted, which helps better understand the components in the proposed method.", "reasons_to_reject": "The experiments need improvement. For instance, only RoBERTa-base is used as the teacher model, and only a 6-layer transformer model with 384 hidden dimensions are used. As a distillation method, readers might be more interested in the performance of compressing larger models. Some baseline methods compared in the paper, such as MiniLM v2 (Wang et al. 2021), also did experiments on RoBERTa-large, not to mention the fact that language models have become larger and larger in recent years.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a new framework for distilling pre-trained language model, which simultaneously considers model compression and token pruning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5Hsob4irXq", "reviewer": "Reviewer_uJ9f", "summary": "", "questions": "A. How will the weighting factors $\\lambda_2$ and  $\\lambda_3$ of the introduced losses affect the performance of their approach?\nB. How is the pruning ratio decided during the inference stage of each of the tasks in GLUE?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments and will make every effort to address the raised concerns.\n \n+ Re Reason to reject 1 and Q1:\n\nHyper-parameters λ2 and λ3 control the intensity of the alignment loss and the recovery loss, respectively. In our paper, we heuristically set λ2 = λ3 = 0.5 to ensure a fair comparison between the two objectives and to equate the loss terms associated with token pruning to the magnitude of the original loss (i.e., λ1 = λ2 + λ3). In fact, our preliminary experiments indicated that the model's sensitivity to λ2 and λ3 is limited; various choices between 0.5 and 1.0 yield similar performance. We will incorporate this clarification in our final version. Your suggestion is greatly appreciated.\n \n+ Re Reason to reject 2 and Q2:\n\nThe pruning ratio is defined by a length configuration which defines how many tokens are retained in each layer. For each task in GLUE, we adopt evolutionary search following [1] to obtain a set of optimal length configurations that lies in the Pareto frontier of accuracy-efficiency trade-off. Then we manually select one length configuration that achieves good performance as well as considerable speed-up from the set. We will provide more comprehensive  details about evolutionary search in our final version. Furthermore, in response to your concern about reproducibility, we will release our implementation.\n \n+ Re reason to reject 3:\n\nAs demonstrated in Table 1 (lower block), our method yields markedly superior results in both performance and speed-up ratio. For instance, on the MNLI dataset, our method exhibits considerably higher accuracy (85.6 vs. 81.2) and is substantially faster than the leading baseline (4.5x vs. 2.8x).\n\nHope the above explanation may address your concern.\n  \n**References**\n\n[1] Kim G, Cho K. Length-adaptive transformer: Train once with length drop, use anytime with search[J]. arXiv preprint arXiv:2010.07003, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The apporach provides significant performance improvement with slightly better inference speedup on the GLUE dataset.\n2. The paper is well written and easy to follow.\n3. The paper provides comprehensive ablation study to prove the effectiveness of the proposed additional losses", "reasons_to_reject": "1. The paper lacks study of how the important hyper-parameters are chosen: the weighting factors $\\lambda_2$ and  $\\lambda_3$ of the losses. It is unclear how these hyperparameters will affect the performance of their approach.\n2. The paper lacks the clarity of the implementation details, which may harm its reproducibility. It is unclear how the pruning ratio is decided during the inference of each of task in GLUE. L499-L500 indicates that the pruning ratio is set the same as the distillation step, which means that it will be sampled from a predefined set. However, if this is the case, the inference process of the models won't be deterministic, which can be an undesired property that may introduce large variance to the performance of the model.\n3. The proposed method roughly doubles the training cost, while the improvements of inference speed up over previous work is relatively small.\n", "questions_to_authors": "A. How will the weighting factors $\\lambda_2$ and  $\\lambda_3$ of the introduced losses affect the performance of their approach?\nB. How is the pruning ratio decided during the inference stage of each of the tasks in GLUE?\n", "paper_topic_and_main_contributions": "This paper effectively combines model compression with dynamic token-pruning by proposing a reconstruction loss and an alignment loss during the knowledge distillation from a pre-trained teacher on open domain corpora. Their approach demonstrates significantly better performance-speed trade-off than strong baselines on the GLUE benchmark. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nHpg9CDoCv", "reviewer": "Reviewer_YrX6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive comments and will make every effort to address the raised concerns.\n \n+ Re Q1:\n\nAs highlighted in Lines 68-72 of the paper, prior research [1, 2, 3] treated model compression and dynamic token pruning as two separate processes (i.e., applying dynamic pruning to an already compressed smaller model). In contrast, we are the first to approach model compression and token pruning as an integrated process. We propose a customized knowledge distillation algorithm referred to as LAD, which enhances the student model's robustness to token pruning throughout the distillation process. Experimental results in Table 1 (the bottom block) demonstrate that our method can achieve a superior speed-performance trade-off compared to previous pipeline approaches.\n \n\n+ Re Q2:\n\nIndeed, we have compared our method with cascade methods (i.e., two-stage distillation -> pruning), and the results are presented in the bottom block of Table 1. It can be observed that our method achieves a superior speed-performance trade-off.\nAs for \"pruning -> two-stage distillation,\" we would like to emphasize that this pipeline requires compressing a model using a static pruning algorithm first and is beyond the scope of the dynamic token pruning that we studied in this paper.\n \n+ Re `` Reproducibility’’:\n\nWe will release our implementation to facilitate future research and provide more implementation details in appendix in our final version.\n \n\nHope the above explanation may address your concern.\n \n \n**References**\n\n[1] Kim G, Cho K. Length-adaptive transformer: Train once with length drop, use anytime with search[J]. arXiv preprint arXiv:2010.07003, 2020.\n\n[2] Guskin S, Wasserblat M, Ding K, et al. Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length[J]. arXiv preprint arXiv:2111.09645, 2021.\n\n[3] Guskin S, Wasserblat M, Wang C, et al. QuaLA-MiniLM: a Quantized Length Adaptive MiniLM[J]. arXiv preprint arXiv:2210.17114, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strong performance on GLUE benchmark.", "reasons_to_reject": "1 The proposed method is a simple combination of two-stage distillation method and length pruning, which lacks of novelty.\n2 The method should be compared with cascade methods (1 two-stage distillation -> pruning; 2 pruning -> two-stage distiilation) to showcase the effectiveness.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a two-stage knowledge distillation framework LAD that transfers general and task-specific knowledge to the student while helping the student to get adapted to dynamic token pruning. The experiments on GLUE benchmark show superior performance than MiniLMv2 and TInyBERT.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel framework for distilling pre-trained language models, introducing a simultaneous approach to model compression and token pruning. This framework efficiently combines model compression with dynamic token-pruning, employing a reconstruction loss and alignment loss during knowledge distillation from a pre-trained teacher model. Through its experimentation on the GLUE benchmark, the work exhibits superior performance over contemporary methods like MiniLMv2 and TinyBERT. Notwithstanding its achievements, reviewers have raised concerns regarding the comprehensiveness of its experiments, reproducibility due to implementation ambiguity, and the increased training costs relative to the marginal improvements in inference speed.

Pros:

Introduction of a method combining model compression and token pruning, demonstrating enhanced performance on the GLUE benchmark.

Comprehensive ablation studies providing a deeper understanding of the proposed method's components.

Well-structured paper that offers clarity and ease of reading.


Cons:

Insufficient experimentation, especially concerning the compression of larger models which are becoming more prevalent.

Absence of clarity in implementation details and potential reproducibility issues, with concerns about non-deterministic inference processes.

While the proposed method offers improvements, it comes at the cost of roughly doubling the training time with only minimal gains in inference speed.
Decision: Accept-Findings