TinyBERT: Distilling BERT for Natural Language Understanding
Authors: Xiaoqi Jiao
Yichun Yin
Lifeng Shang
Xin Jiang
Xiao Chen
Linlin Li
Fang Wang
Qun Liu
Venue:  (2020)
Keywords: BERT Compression
Transformer Distillation
TinyBERT

Abstract: Language model pre-training, such as BERT, has significantly improved the performances of many natural language processing tasks. However, the pre-trained language models are usually computationally expensive and memory intensive, so it is difficult to effectively execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we firstly propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-based models. By leveraging this new KD method, the plenty of knowledge encoded in a large “teacher” BERT can be well transferred to a small “student” TinyBERT. Moreover, we introduce a new two-stage learning framework for TinyBERT, which performs Transformer distillation at both the pre-training and task-specific learning stages. This framework ensures that TinyBERT can capture the general domain as well as the task-specific knowledge in BERT. TinyBERT is empirically effective and achieves comparable results with BERT on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT is also significantly better than state-of-the-art baselines on BERT distillation, with only ∼28% parameters and ∼31% inference time of them.


Under review as a conference paper at ICLR 2020
TINYBERT: DISTILLING BERT FOR NATURAL LAN- GUAGE UNDERSTANDING
Anonymous authors Paper under double-blind review
ABSTRACT
Language model pre-training, such as BERT, has significantly improved the per- formances of many natural language processing tasks. However, the pre-trained language models are usually computationally expensive and memory intensive, it is difficult to effectively execute them on resource-restricted devices. To accelerate inference and reduce model size while maintaining accuracy, we first- ly propose a novel Transformer distillation method that is specially designed for knowledge distillation (KD) of the Transformer-be -bssed models. By leveraging this new KD method, the plenty of knowledge encoded in a large "teacher" BERT can be well transferred to small "student" TinyBERT. Moreover, we introduce new two-stage learning framework for TinyBERT, which performs Transformer distil- lation both the pre-training and task-specific learning stages. This framework ensures that TinyBERT can capture the general-domaie as well as the task-specific knowledge in BERT.
TinyBERT is empirically effective and achieves more than 96% the performance of teacher BERT faster on inference. TinyBERT is also significantly better than state-of-the-ae baselines on BERT distillation, with only 288 parameters and ~31 % inference time of them.
GLUE benchmark, while being 7.5x smaller and 9.4x
INTRODUCTION
Pre-training language models then fine-tuning on downstream tasks has become a new paradigm for natural language processing (NLP). Pre-trained language models (PLMs), such as BERT (Devlin 2018). XLNet (Yang et al.] 2019), RoBERTa (Liu et al. 2019) and SpanBERT (Joshi et al. have achieved great success in many NLP tasks (e.g., the GLUE benchmark (Wang al. 2018 and the challenging multi-hop reasoning task (Ding et al. 2019)). However, PLMs usually have an extremely large number of parameters and need long inference time, which are difficult be deployed on edge devices such as mobile phones. Morcover, recent studies (Kovaleva et al.) (019)) also demonstrate that there redundancy in PLMs. Therefore, the computational overhead and model storage of PLMs while keeping their performances.
et al.
2019
crucial and possible reduce
There has been many model compression techniques (Han et al.] 2015a) proposed to accelerate deep model inference and reduce model size while maintaining accuracy. The most commonly used techniques include quantization (Gong al., 2014), weights pruning (Han et al. 2015b), and knowledge distillation (KD) (Romero et al. 2014). 4). In this paper we focus on knowleds distillation, an idea proposed by Hinton et al. (2015) in a reacher-studens framework. KD aims to transfer the knowledge embedded in large teacher network to small student network. The student network is trained to reproduce the behaviors of the teacher network. Based on the framework, we propose a novel distillation method specifically for Transformer-basee models /aswaani et al. 2017). and use BERT as an example to investigate the KD methods for large scale PLMs.
KD has been extensively studied in NLP (Kim & Rush 2016 Hu al. 2018), while designing KD methods for BERT has been less explored. The -he-rraiinnn--e--ffeee pre-trains BERT on a large scale unsupervised text corpus, then fine-tunes on task-specific dataset, which greatly increases the difficulty of BERT distillation. Thus we are required to design an ef-
paradigm firstly
Our code and models will be made publicly available.

---

Under review as a conference paper at ICLR 2020
fective KD strategy for both stages. To build competitive TinyBERT, firstly propose new Transformer distillation method to distill the knowledge embedded teacher BERT. Specifically, design several loss functions fit different representations from BERT layers: 1) the output of the embedding layer; 2) the hidden states and attention matrices derived from the Transformer layer; the logits output by the prediction layer. The attention based fitting is inspired by the recent findings (Clark et al.] 2019) that the attention weights learned by BERT can capture substan- tial linguistic knowledge, which encourages that the linguistic knowledge can be well transferred from teacher BERT to student TmnyBRTT, However, it is ignored in existing KD methods of BERT. such Distilled BiLSTMSOrT (Tang et al. 2019). BERT-PKD (Sun et al. 2019) and DistilBERT Then, we propose novel two-stage learning framework including the general distillation and the task-specific distillation. At the general distillation stage, the original BERT without fine-tuning acts as the teacher model. The student TinyBERT learns to mimic the teacher's behavior by ex- ccuting the proposed Transformer distillation on the large scale corpus from general domain. We obtain a general TinyBERT that can be fine-tuned for various downstream tasks. At the task task-specific distillation stage, we perform the data augmentation to provide more task-specific data for teacher- student learning, and then re-execute the Transformer distillation the augmented data. Both the two stages are essential to improve the performance and generalization capability of TinyBERT. A detailed comparison between the proposed method and other existing methods is summarized in Table The Transformer distillation and two-stage learning framework are two key ideas of the proposed method.
Table 1:
A summary of KD methods for BERT. Abbreviations: (initiiiiig student BERT with some layers of pre-trained teacher BERT), DA(conducting data augmentation for task-specific train- ing data). Embd, Attn, Hidn, and Pred represent the knowledge from embedding layers, attention matrices, hidden states, and final prediction layers, respectively.
KD at Pre-training Stage
KD at Fine-tuning Stage
KD Methods
INIT Embd
Atin
Hidn
Pred
Embd
Attn
Hidn
Pred
DA
Distilled BiLSTM
v
BERT-PKD
DistilBERT
TinyBERT (our method)
The main contributions of this work are as follows: 1) We propose a new Transformer distillation method to encourage that the linguistic knowledge encoded in teacher BERT can be well transferred to TinyBERT. 2) We propose a novel two-stage learning framework with performing the proposed Transformer distillation at both the pre-training and fine-tuning stages, which ensures that Tiny- BERT can capture both the general-domain and task-specifie knowledge of the teacher BERT. 3) We show experimentally that our TiRyBERT can achieve more than 96% the performance of teacher BERT on GLUE tasks, while having much fewer parameters (~ 13.3%) and less inference time (~...)) and significantly outperforms other state-of-the-ard baselines on BERT distillation.
PRELIMINARIES
We firstly describe the formulation of Transformer (Vaswani et al. 2017) and Knowledge Distilla- tion (Hinton et al. 2015). Our proposed Transformer distillation is specially designed KD method for Transformer- based models.
2.1 TRANSFORMER LAYER
Most of the recent pre-trained language models (c.g., BERT, XLNet and RoBERTa) are built with Transformer layers, which can capture long-term dependencies between input tokens by self- attention mechanism. Specifically, a standard Transformer layer includes two main sub-layers: multi-head attention (MHA) and fully connected feed-forwar network (FFN).
'The student learns from the [CI.S] (a special classification token of BERT) hidden states of the teacher. "The output of pre-training tasks (such dynamic masking) used the supervision signal.
2

---

Under review as a conference paper at ICLR 2020
Multi-Head Attention (MHA). The calculation of attention function depends on the three compo- nents of queries, keys and values, which are denoted as matrices Q, K and V respectively. The attention function can be formulated follows:
A = QK
(1)
vdz
Attention(Q. K, V) = (A)t,ax(AAVV,
(2)
where dg the dimension of keys and acts scaling factor, A is the attention matrix calculated from the compatibility of Q and K by dot-pree dot-product operation. The final function output is calculated as weighted sum of values V. and the weight computed by applying softmax () operation on the each column of matrix A. According to Clark et al. (2019), the attention matrices in BERT can capture substantial linguistic knowledge, and thus play an essential role in our proposed distillation method.
Multi-head attention is defined by concatenating the attention heads from different representation subspaces follows:
MultiHead(Q.. (Q, V) = Concaat (head),hhee....... heads) W,
where h is the number of attention heads, and head, denotes the i-th attention head, which is calcu- lated by the Attent innn () function with inputs from different representation subspaces, the matrix W acts as linear transformation.
Position-wise Feed-Forward Network (FFN). Transformer layer also contains fully connected feed-forward network, which is formulated as follows:
(N = max(0, x W, + b2 b2.
(4)
We can see that the FFN contains two linear transformations and one ReLU activation.
2.2 KNOWLEDGE DISTILLATION
KD aims to transfer the knowledge of a large teacher network T a small student network S. The student network is trained to mimic the behaviors of teacher networks. Let f and fs represent the behavior functions of teacher and student networks, respectively. The behavior function targets at transforming network inputs to some informative representations, and it can be defined the output of any layer in the network. In the context of Transformer distillation, the output of MHA layer FFN layer, some intermediate representations (such the attention matrix A) can be used as behavior function. Formally, KD can be modeled as minimizing the following objective function:
CKD = ((),
(5)
ZEX
where T.(.) is loss function that evaluates the difference between teacher and student networks, the text input and X' denotes the training dataset. Thus the key research problem becomes how to define effective behavior functions and loss functions. Different from previous KD methods, we also need to consider how to perform KD at the pre-training stage BERT addition to the task-specific training stage.
METHOD
In this section, we propose novel distillation method for Transformer-based models, and present two-stage learning framework for our model distilled from BERT, which is called TinyBERT.
3.1 TRANSFORMER DISTILLATION
The proposed Transformer distillation is specially designed KD method for Transformer networks. I overview of the proposed KD method. In this work, both the student and teacher networks are built with Transformer layers. For a clear illustration, we firstly formulate the problem before introducing method.
3

---

Under review as a conference paper at ICLR 2020
Transformer Layer:
Einhodding Layer:
Prediction Layer:
Layer Number: N > M
Teacher (BERT)
Hidden Size:
Attention Matrices (8) (ahhoniiit
Attention Matrices (B/-noti))
Student (TinyBERT)
Hidn
1-000
Hidden (1)11)
Haädeen
Transformer
M
Distillation
Norre
Norts
& Norm
Anno
MIA
MBA
Text Input
Teather Layer
Student Layer
(a)
Figure 1: An overview of Transfformer distillation: (a) the framework of Transformer distillation, (b) the details of Transformer-laye distillation consisting of Atth based distillation) and (hidden
(hidden states based distillation).
Problem Formulation. Assuming that the student model has M Transformer layers and teacher model has N Transformer layers, we choose M layers from the teacher model for the Transformer- layer distillation. The function n g(m) is used as mapping function from student layers to teacher layers, which means that the m-th layer student model learns the information from the n- th layer of teacher model. The embedding-laye also considered. We set 0 to be the index of embedding layer and M + +11 to be the index of prediction layer, and the corresponding layer mappings defined 0 g(0) and N + g(M + 1) respectively. The effect of the choice of different mapping functions on the performances will be studied in the experiment section. Formally, the student can acquire knowledge from the teacher by minimizing the following objective:
distillation and the prediction-lay distilaatinn are
M M+1
Lmodel
Σ ====0
Am Ctayer Tg(mm)
(6)
where Llayer refers to the loss function a given model layer (c.g., Transformer layer or embedding
layer) and Am is the hyper-parameter that represents the importance of the m-th layer's 's distillation.
Transformer-layer Distillation. The proposed Transformer-layer distillation includes the attention based distillation and hidden states based distillation, which is shown in Figure[] The attention based distillation is motivated by the recent findings that attention weights learned by BERT can capture rich linguistic knowledge (Clark et al. 2019). This kind of linguistic knowledge includes the syntax and coreference information, which is essential for natural language understanding. Thus we propose the attention based distillation to encourage that the linguistic knowledge can be transferred from teacher BERT to student TERYBERT. Specifically, the student learns to fit the matrices of multi- head attention the teacher network, and the objective defined as:
Lutn Σ
MSE MSE(A, (A, A,A))
where h is the number of attention heads, A, ! refers to the attention matrix corresponding to the i-th head teacher or student, is the input text length, and MSE () means the mean squared error loss function. In this work, the (unnormalized) attention matrix A, is used as the fitting target instead of its softmax output s (offmax(A)) since our experiments show that the former setting has a faster convergence rate and better performances.
In addition to the attention based distillation, we also distill the knowledge from the output of Trans- former layer (as shown in Figure I (b)), and the objective is as follows:
Laidn MSE S Wn, H²).
(8)
4

---

Under review as a conference paper at ICLR 2020
General Learning
Tpsk-specific Learning
Tramerormer Distillation
General ThayBERT
Transformer Distillation
Fine-tuned TinyAERT
Largo-scale Text Corpus
Data Augmentation
Augmented Task Dataset
Task Dataset
Figure 2:
The illustration of TinyBERT learning
where the matrices H Rixd' and H E RR refer to the hidden states of student and teacher networks respectively, which are calculated by Equation + The scalar values d and denote the hidden sizes of teacher and student models, and d' is often smaller than d to obtain smaller student network. The matrix W₂ E IRd xd a learnable linear transformation, which transforms the hidden states of student network into the same space the teacher network's states.
Embedding-layer Distillation. We also perform embedding-laye layer distillation, which is similar to the hidden states based distillation and formulated
Lemba MSE E
where the matrices E and H refer to the embeddings of student and teacher networks, respec- tively. In this paper, they have the same shape the hidden state matrices. The matrix W. is linear transformation playing similar role as W..
Prediction-Layer Distillation. In addition to imitating the behaviors of intermediate layers, also use the knowledge distillation to fit the predictions of teacher model (Hinton et al. (015)).
Specifically, we penalize the soft cross-entropy loss between the student network's logits against the teacher's s logits:
Cpred -sof=saaaaa (=²) wwg.ooooooooooo (oogssoftmax("") t),
(10)
where S and TT are the logits vectors predicted by the student and teacher respectively, log softmax() means the log likelihood, means the temperature value. In our experiment, we find that 1 performs well.
Using the above distillation objectives (i.e. Equations 7 and 10 we can unify the distillation loss of the corresponding layers between the teacher and the student network:
Lembal (S. To); Criau Smy Talm)) Laum(S.., Ta(m))s
m 0
Llayer (Sm Ig(m))
M 2 0
(1I)
ppeca ( M1+1 IN+1),
m M M+1
In our experiments, we firstly perform intermediate layer distillation (M 2 m 2 0), then perform the prediction-layer distillation (m M 1).
3.2 TINYBERT LEARNING
The application of BERT usually consists of two learning stages: the pre-training and fine-tuning. The plenty of knowledge learned by BERT in the pre-training stage is of great importance and should also be transferred. Therefore, we propose novel two-stage learning framework including the gen- eral distillation and the task-specific distillation, as illustrated Figure 2 General distillation helps student TinyBERT leam the rich knowledge embedded in teacher BERT, which plays important role in improving the generalization capability of TinyBERT. The task-specific distillation teaches the student the task-specific knowledge. With the two-step distillation, can further reduce the gap between teacher and student models.
General Distillation. In general distillation,
the original BERT without fine-tuning as the teacher and a large-scale text corpus as the learning data. By performing the Transformer distilla- tion on the text from general domain, obtain general TinyBERT that can be fine-tuned for
IIn the general distillation, do perform prediction-layer distillation as Equation

---

Under review as a conference paper at ICLR 2020
Table 2: Results evaluated on the test set of GLUE official benchmark, All models are learned in single-task manner. "- means the result not reported.
System
MNL.-m
MNL.-mm
QQP
SST-2
QNLI
MRPC
RTE
CoLAA
STS-B
Average
BERTDASE (Google)
84.6
83.4
71.2
93.5
90.5
88.9
66.4
52.1
85.8
79.6
BERTHASE (Teacher)
83.9
83.4
71.1
93.4
90.9
87.5
67.0
52.8
85.2
79.5
SERT
75.4
74.9
66.5
87.6
84.8
83.2
62.6
19.5
77.1
70.2
Distilled SiS.TT
73.0
72.6
68.2
90.7
BERT-PKD
79.9
79.3
70.2
89.4
85.1
82.6
62.3
24.8
79.8
72.6
DistiBBRT
78.9
78.0
68.5
91.4
85,2
82.4
54.1
32.8
76.1
71.9
TinyBRRT
82.5
81.8
71.3
92.6
87.7
86.4
62.9
43.3
79.9
76.5
Table 3: The model sizes and inference time for baselines and TinyBERT: The number of layers does not include the embedding and prediction layers.
System
Layers
Hidden
Feed-forwane
Model
Inference Time
Size
Size
Size
BASE (Teacher)
12
768
3072
109M( x 1.0)
188s 1.0)
Distilled BiLSTMSOFT
1
300
400
(0.1M(x00.8)
24.8s(x7.66 63.7s(x3.0)
BERT-PKDD/DstiBBRTT
4
768
3072
32.2M(x2.I)
TInBBERTBBRTTMAA SMALL
4 312
1200
14.5M(x7.5)
19.9s(x9.4)
downstream tasks. However, due to the significant reductions of the hidden/embeddiing size and the layer number, general TinyBERT performs relatively worse than BERT.
Task-specific Distillation. Previous studies show that the complex models, fine-tuned BERTs, suf- fer from peeemaaramttinotooonn for comain-specific tasks (Kovaleva et al. (201)). Thus, it is possible for small models to achieve comparable performances to the BERTs. To this end, we propose to de- rive competitive fine-tuned TinyBERT's through the task-specific distillation. In the task-specific dis- tillation, we re-perform the proposed Transformer distillation on an augmented task-specific dataset (as shown in Figure 2 Specifically, the fine-tuned BERT is used as the teacher and data aug- mentation method is proposed to expand the tasc-spe training set. Learning more task-related examples, the generalization capabilities of student model can be further improved. In this work, we combine a pre-trained language model BERT and GloVe (Pennington et al., (014) word embed- dings do word-level replacement for data augmentation. Specifically, to predict word replacements for single-piece words (Wu et al. 2019). and use the word embeddings retrieve the most similar words word replacements for multiple-pieces words. Some hyper- parameters are defined to control the replacement ratio of sentence and the amount of augmented dataset. More details of the data augmentation procedure are discussed Appendix
use the language model
The above two learning stages are complementary to each other: the general distillation provides a good initialization for the task-specific distillation, while the task-specific distillation further im- proves TinyBERT by focusing on learning the task-specifh knowledge. Although there is big gap between BERT and TinyBERT the TinyBERT can achieve competitive performances in various NLP tasks. The proposed Trans- former distillation and two-stage learning framework are the two most important components of the proposed distillation method.
model size, by performing the proposed two-stage distillation,
EXPERIMENTS
In this section, we evaluate the effectiveness and efficiency of TinyBERT on variety of tasks with different model settings.
4.1 MODEL SETUP
We instantiate tiny student model (the number of layers M=4, the hidden size d'=312, the feed- forward/filter size d=1200 and the head number 1=12) that has a total of 14.5M parameters. not specified, this student model is referred to as the TinyBERT. The original BERTBASE (the number of layers N=12, the hidden size d=768, the feed-forward/fflter size d,=3072 and the head number h=12) is used the teacher model that contains 109M parameters. We use 9(m) 3 as the layer mapping function, so TinyBERT learns from every layers of BERTRASE. The learning weight À of each layer is set to I, which performs well for the learning of our TinyBERT.

---

Under review as a conference paper at ICLR 2020
Table 4: Results (dev) of wider or deeper TinyBERT variants and baselines.
System
MNLI-m
MNLI-mm
MRPC
CoLA
Average
BASE (Teacher)
84.2
84.4
86.8
57.4
78.2
BERT-PKD (M=6;d' =766
80.9
80.9
83.1
43.1
72.0
DistilBRRT (N/=6;:===78:((=3777)
81.6
81.1
82.4
42.5
71.9
TinyBERT (M=4<(=312:d(=1200)
82.8
82.9
85.8
49.7
75.3
BiRyBERT (M=4:d=768:d(=337)
83.8
84.1
85.8
50.5
76.1
TinyBERT (M=6:d==312:d412200)
83.3
84.0
86.3
50.6
76.1
TinyBERT (M==x(==76:dd:=37)))
84.5
84.5
86.3
54.0
77.3
4.2 EXPERIMENTAL RESULTS ON GLUE
We evaluate TinyBERT on the General Language Understanding Evaluation (GLUE) (Wang et al.) 2018) benchmark, which a collection of diverse natural language understanding tasks, The details of experiment settings are described in Appendix B The evaluation results are presented in Table and the efficiencies model size and inference time are also evaluated in Table
The experiment results demonstrate that: 1) There is large performance gap between BERT and BERTBASE due to the big reduction model size. 2) TinyBERT is consistently better than BERT SMALL in all the GLUE tasks and achieves a large improvement of 6.3% average. This indicates that the proposed KD learning framework can effectively improve the performances of small models regardless of downstream tasks. 3) TinyBERT significantly outperforms the state-of- the-art KD baselines (i.e., BERT-PKD and DistilIBERT) by a margin of at least 3.9%, even with only 28% parameters and ~31 inference time of baselines (see Table 3). 4) Compared with the teacher BERTBASE TinyBERT is 7.5x smaller and 9.4x faster in the model efficiency, while main- taining competitive performances. 5) TinyBER! has a comparable model efficiency (slightly larger in size but faster in inference) with Distilled BiLSTMSorT and obtains substantially better perfor- mances in all tasks reported by the BiLSTM baseline. 6) For the challenging CoL.A dataset (the task of predicting linguistic acceptability judgments), all the distilled small models have a relatively bigger performance gap with teacher model. TinyBERT achieves a significant improvement over the strong baselines, and its performance can be further improved by using a deeper and wider model to capture more complex linguistic knowledge as illustrated in the next subsection.
Morcover, BERT-PKD and DistillBERT initialize their student models with some layers of well pre-trained teacher BERT (see Table D. which makes the student models have to keep the same size settings of Transformer layer (or embedding layer) their teacher BERT. In two-stage distillation framework, TinyBERT initialized by general distillation, so it has the advantage of being more flexible in model size selection.
4.3 EFFECTS OF MODEL SIZE
We evaluate how much improvement can be achieved when increasing the model size of TinyBERT on several typical GLUE tasks, where MNLI and MRPC are used in the ablation studies of Devlin et al. (2018), and CoLA is the most difficult task in GLUE. Specifically, three wider and deeper variants are proposed and their evaluation results on development set are displayed in Table 4 We can observe that: 1) All the three TinyBERT variants can consistently outperform the original s- mallest TERyBERT, which indicates that the proposed KD method works for the student models of various model sizes. 2) For the CoLA task, the improvement is slight when only increasing the number of layers (from 49.7 to 50.6) or hidden size (from 49.7 to 50.5). To achieve more dramatic improvements, the student model should become deeper and wider (from 49.7 to 54.0). 3) Anoth- er interesting observation that the smallest 4-layer TinyBERT can even outperform the 6-layers baselines, which further confirms the effectiveness of the proposed KD method.
4.4 ABLATION STUDIES
In this section, we conduct ablation studies to investigate the contributions of : 1) different pro- cedures of the proposed two-stage TinyBERT learning framework (see Figure 2}. and 2) different distillation objectives (see Equation
BERT SMERL means directly pretraining small BERT, which has the same model architecture as Tiny- BERT, through tasks of Masked Language Model (MLM) and Next Sentence Prediction (NSP).
7

---

Under review as a conference paper at ICLR 2020
Table 5: Ablation studies of different proce- dures (i.e.. TD, GD, and DA) of the two-stage learning framework. The variants are validated on the dev set.
Table 6: Ablation studies of different distilla-
tion objectives in the TinyBERT learning. The variants are validated on the dev set.
System
MNLI-m MNLI-mm
MRPC
CoLA
Average
TinyBER BERT
82.8
82.9
85.8
49.7
73.3
System TinyBERT
MNLI-m
MNLI-mm
MRPC
CoLA
Average
Embd
82.3
82.3
85.0
46.7
74.1
8288
82.9
85.8
49.7
75.3
No Pred
80.5
81.0
84.3
48.2
73.5
No GD
82.5
82.6
84.1
40.8
72.5
No Trm
71.7
72.3
70.1
11.2
56.3
No TD
80.6
81.2
83.8
28.5
68.5
No Atta
79.9
80.7
82.3
41.1
71.0
No DA
80.5
81.0
82.4
29.8
68.4
No Hidn
81.7
82.1
84.1
43.7
72.9
Table 7: Results (dev) of different mapping strategies.
System
MNLI-m
MNLI-mm
MRPC
CoLA
Average
Tiny BERy (Uniform-straiee strategy)
82.8
82,9
85,8
49.7
75.3
TinyBERT (lop-strategy)
81.7
82.3
83.6
35.9
70.9
TinyBERT (Bottom strotmyy
80.6
81.3
84.6
38.5
71.3
Effects of different learning procedures. The proposed two-stage TinyBERT learning frame- work (see Figure 2 consists of three key procedures: TD (Task-specific Distillation), GD (General Distillation) and DA (Data Augmentation). The effects of different learning procedures are analyzed and presented in Table [5] The results indicate that all the three procedures are crucial for the pro- posed KD method. The TD and DA has comparable effects in all the four tasks. We can also find the task-specific procedures (TD and DA) are more helpful than the pre-training procedure (GD) in all the four tasks. Another interesting observation is that GD has more effect on CoLA than on MNLI and MRPC. We conjecture that the ability of linguistic generalization (Warstadt et 2018) learned by GD plays a more important role in the downstream CoLA task (linguistic acceptability judgments).
Effects of different distillation objectives. We investigate the effects of distillation objectives on the TinyBERT learning. Several baselines are proposed including the TinyBERT learning without the Transformer-layer distillation (No Trm), embedding-layer distillation (No Emb) and prediction- layer distillation (No Pred respectively. The results are illustrated in Table and show that all the proposed distillation objectives are useful for the TinyBERT learning. The performance drop- s significantly from 75.3 to 56.3 under the setting (No Trm), which indicates Transformer-layer distillation is the key for TinyBERT learning. Furthermore, we study the contributions of atten- tion (No Attn) and hidden states (No Hidn) in the Transformer-layer distillation. We can find the attention based distillation has bigger effect than hidden states based distillation TinyBERT learning. Meanwhile, these two kinds of knowledge distillation are complementary to each other, which makes TERYBERT obtain the competitive results.
4.5 EFFECTS OF MAPPING FUNCTION
We investigate the effects of different mapping functions n = gmm) on the TinyBERT learning. Our original TinyBERT described in section 4.1 uses the uniform-stratege and compare with two typical baselines including top-strategy (m) = m + - M; < m S M) and bottom-strateeg (g(m) m;; < S M).
The comparison results are presented in Table We find that the top-strategy performs better than the bottom-strategy in MNLI, while being worse in MRPC and CoLA tasks, which confirms the observations that different tasks depend on the different kinds of knowledge from BERT layers. Since the uniform-strategy acquires the knowledge from bottom achieves better performances than the other two baselines in all the four tasks. Adaptively choosing layers for specific task a challenging problem and we leave it as the future work.
top layers of BERTBASE,
Other Experiments. We also evaluate TinyBERT on the question answering tasks, and study whether we can use BERTSMALL as the initialization of the general TinyBERT. The experiments are detailed in Capendix[[anndll
the augmented training set." "N Pred" means performing standard cross-entropy against the ground-truth of the original training set.
Thee prediction-layer distillation performs soft cross-entropy Equation 10
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION AND FUTURE WORK
In this paper, we firstly introduce a new KD method for Transformer-based distillation, then we further propose two-stage framework for TinyBERT learning. Extensive experiments show that the TinyBERT achieves competitive performances meanwhile significantly reducing the model size and shortening the inference time of original BART which provides an effective way deploy BERT-based NLP applications on the edge devices.
In future work, we would study how to effectively transfer the knowledge from wider and deeper teachers (e.g.. BERTLARGE and XLNNeLARGGE to student TinyBERT. The joint learning of dis- tillation and guantization/pruiing would another promising direction to further compress the pre-trained language models.
REFERENCES
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Daniel Cer, Mona Diab, Eneko Agirre Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual similarit-multilinnge and cross- cross-lingual focused evaluation. arXiv preprint arXiv: (00.::070000055 2017.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2018.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:/906.044: 04344,, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 180.0008 2018.
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension at scale. In Proceedings of the 57th Conference of the Association for Computational Linguisties, ACL pp. 2694-2703, 2019.
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional net- works using vector quantization. arXiv preprint arXiv:1412.6115. 2014.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 10.00044 2015a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems(NeurlPS), pp- 1135-1143, 2015b.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 02531, 2015.
Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention guiide answer distillation for machine reading comprehension. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing/EMNLPP pp. 2077- 2086, 2018
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. S- panbert: Improving pre-training by representing and predicting spans. arXiv preprini arX- 105.99...1529, 2019.
Yoon Kim and Alexander M Rush. Sequence-leve knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing(EMNLLP pp. 1317 1327, 2016.

---

Under review as a conference paper at ICLR 2020
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of bert. arXiv preprint arXiv:1908.085933 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint arxiv: 1692, 2019.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing( EMNIP), pp. 2383-2392, November 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you dont know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Lingristicx(ACL)) pp. 784-789, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing(EMNNP)) pp. 1631-1642, October 2013.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pracessing(EMNLP) 2019.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv: 12136, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems(NewrIPS) pp. 5998-6008, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 Conference Empirical Methods in Natural Language (MMess)ing(EMNLP), pp- 353-355, 2018
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: /000//04551111/11 /247/, 2018
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties/NAACL) pp. 1112-1122 2018.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. Conditional bert contex- tual augmentation. In International Conference on Computational Science(ICCS), pp. 84-95. Springer, 2019.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arX- 082,99 2019.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
A DATA AUGMENTATION DETAILS
In this section, we explain the proposed data augmentation method. Specifically, we firstly mask a word in a sentence, then use BERT as a language model to predict M most-likely words in the corresponding position, while keeping other words unchanged. By this way, we can easily get the candidates for each word under specific context. To induce new instances for given sentence, we also use threshold pt to determine whether we should replace the current word with randomly selected candidate. By repetitively performing this replacement operation for each word in sen- tence, we can finally get a new augmented sentence. In our preliminary experiments, we find the quality of generated candidates for the words consisting of multiple sub-word pieces, is relatively low. To alleviate this problem, we instead pick a similar word from GloVe (Pennington et al. 2014) word embeddings based on the cosine similarity. We apply this data augmentation method N times to all the sentences downstream task. In this work, we set Pt 0.4, N = 20, M = 15 for all experiments. The data augmentation procedure is illustrated as below:
Algorithm The Proposed Data Augmentation Input: x is sequence of words Pt, N, M are hyperparameters
Output: augg, augmented data
1: function AATALAAGMENNNNNINNN Pc, N)
2: 4--00 0
dola_aug +- 4: while < N do
3:
5:
Xmasked f- x
6: for i <- to 1en(x) do
7:
x[] [i]
is a single-piece word then
8:
Replace Xmaskea[i] with [MASK]
9:
condidates t M most-likely words predicted by BertMode BertModel(Xmastell
(Xmasked)
else
11:
condidates + M similar words of x|i] from GloVe
end if
13:
Sample p UNTFORM(0,1)
if p S Pt then
15:
Replace Xmasked[i] with word from candidates randomly
16:
else
17:
Keep Xmasked as x[i]
unchanged
18:
end
end for
20:
Append Xmasked to dola aug
21:
22:
end while
return data .ug
24:
end function
B GLUE DETAILS
TERyBERT setup. TinyBERT learning includes the general distillation and the task-specific distil- lation. For the general distillation, we use English Wikipedia (2,500 M words) as the text corpus and perform the intermediate layer distillation for epochs with the supervision from a pre-trained BERT teacher and keep other hyper-parameters same BERT pre-training (Devlin et al.) 2018). For the task-specific distillation, we firstly perform intermediate layer distillation on the augmented dataset for 10 epochs with batch size 32 and learning rate 5e-5 under the supervision of fine-tuned BERT teacher. and then perform prediction layer distillation for 3 epochs with batch size 32 and learning rate 3e-5. For tasks like MNLI, QQP and QNLI which have 2 100K training
11

---

Under review as a conference paper at ICLR 2020
amples, we distill intermediate layer knowledge for 5 epochs with batch size 256 on the augmented dataset. Besides, for CoLA task, we perform 50 epochs of intermediate layer distillation.
baselines. For fair comparison, firstly re-implemented the results of BERT-PKD and DistilBERT reported in their papers to ensure implementation procedure is correct. Then following the verified implementation procedure, trained a 4-layer BERT-PKD and a 4-layer DistilBERT as the baselines. The BERTSMALL, learning strictly follows the same learning strategy as described in the original BERT work (Devlin et al. 2018).
Baselines setup. We use BERT-PKD and DistilBERT
Model efficiency evaluation. To evaluate the inference speed, QNLI training set with batch size of 128 and the maximum sequence length of 128. The numbers reported in Table
inference procedure on the
3
are the average running time of 100 batches on single NVIDIA K80 GPU.
The GLUE datasets are described as follows:
MNLL Multi-Genre Natural Language Inference is large-scale, crowd-sourced entailment classi- fication task (Williams et al. 2018). Given a pair of (premise, hypothesis), the goal is to predict whether the hypothesis an entailment, contradiction, or neutral with respect to the premise.
QQP. Quora Question Pairs is a collection of question pairs from the website Quora. The task is to determine whether two questions are semantically equivalent (Chen al. 2018).
QNLI Question Natural Language Inference is version of the Stanford Question Answering Dataset which has been converted to binary sentence pair classification task by Wang et al., (2018). Given pair (question, context). The task is to determine whether the contexl contains the answer to the question.
SST-2. The Stanford Sentiment Treebank is binary single-sentence classification task, where the goal is to predict the sentiment of movie reviews (Socher et al. 2013).
CoLA. The Corpus of Linguistic Acceptability is task to predict whether an English sentence is grammatically correct one (Warstadt et al., 2018).
STS-B. The Semantic Textual Similarity Benchmark is collection of sentence pairs drawn from news headlines and many other domains (Cer et al., 2017). The task aims to evaluate how similar two pieces of texts are by score from to 5,
MRPC. Microsoft Research Paraphrase Corpus a paraphrase identification dataset where systems aim to identify if two sentences are paraphrases of each other (Dolan & Brockett 2005).
RTE. Recognizing Textual Entailment a binary entailment task with small training dataset (Ben-
tivogli et al. 2009).
C SQUAD 1.1 AND 2.0
We also demonstrate the effectiveness of TinyBERT the question answering (QA) tasks: SQuAD v1.1 (Rajpurkar et al. 2016) and v2.0 (Rajpurkar et al. 2018). Following the learning procedure in the previous work (Devlin et al. 2018).
treat these two tasks as the problem of sequence labeling which predicts the possibility of each token as the start or end of answer span. We follow the settings of task-specific distillation GLUE tasks, except with running epochs and learning rate of Se-5 for the prediction-layee distillation on the original training dataset. The results are shown in Table
The results show that TmyBBRTT consistently outperforms the baselines in both the small and medi- um size, which indicates that the proposed framework also works for the tasks of token-level la- beling. Compared with sequence-level GLUE tasks, the question answering tasks depends on more subtle knowledge to infer the correct answer, which increases the difficulty of knowledge distillation. We leave how build better QERTTTT as the future work.
D BERT AS INITIALIZATION OF GENERAL TINYBERT
Initializing general TERYBERT with SMATSMALL is a straightforward idea. However, BERTSMALL would derive mismatched distributions in intermediate representations (e.g., attention matrices
12

---

Under review as a conference paper at ICLR 2020
Table 8: Results (dev) of baselines and TinyBERT on question answering tasks.
System
SQuAD EM
1.1
SQuAD
2.0
EM
BERT BASE (Teacher)
80.7
88.4
73.1
76.4
Small Models
70.1
79.5
60.8
64.6
71.8
81.2
60.6
64.1
72.7
82.1
65.3
68.8
Medium-sized Models
BERT-PKD (M=b;d==766:dd=3372
77.1
85.3
66.3
69.8
DistiBBERT (M==6_(==76:;(;3077)
78.1
86.2
66.0
69.5
TinyBERT (M+66"'=776::443777)
79.7
87.5
69.9
73.4
Table 9: Results of different methods pre-training state. TD and GD refers to Task-specific Distillation (without data augmentation) and General Distillation. The results are evaluated on de- velopment set.
System
MNLI-m
MNLI-mm
MRPC
CoLA
Average
(392k)
(392k)
(3.5k)
(8,5k)
SERTL (MLM&NSP)
75.9
76.9
83.2
19.5
63.9
BERTsMALL (MLM&NSP+TD)
79.2
79.7
82.9
12.4
63.6
TinyBERT (GD)
76.6
77.2
82.0
61.1
TinyBERT (GD+TD)
80.5
81.0
82.4
29.8
68.4
and hidden states) with the teacher BERTBASE model, if without imitating the teacher's behav- iors at the pre-training stage. Further task-specific distillation under the supervision of fine- tuned BERT will disturb the learned distribution/noowledes of BERTsMALL. finally lead- ing to poor performances in some less-data tasks. The results in Table 9 show that the BERTSMALL
performs worse than the BERTSMALL in MRPC and CoLAA tasks, which validates our hypothesis. For the intensive-data task (e.g. MNLJ), TD has enough training data to make BERTsMALL acquire the task-specific knowledge very well, although the pre-trained distributions have already been disturbed.
To make TiRyBERT effectively work for all tasks, we propose General Distillation (GD) for ini- tialization, where the TinyBERT learns the knowledge from intermediate layers of teacher BERT the pre-training stage. From the results of Table we find that GD can effectively transfer the knowledge from the teacher BERT to the student TinyBERT and achieve comparable results with SHRTsMALL (61.1 vs. 63.9), even without performing the MI.M and NSP tasks. Furthermore, the task-specific distillation boosts the performances of TinyBERT by continuing on learning the task-specific knowledge fine-tuned teacher BARTBASE-
E MORE COMPLETE COMPARISONS WITH SAME STUDENT ARCHITECTURE
For the easy and direct comparisons with prior works, we here also present the results TinyBERT (M=6;d' ===;:d'=76;;d"" =76;;'";377" with the same architectures as used in the original BERT-PKD (Sun et al. 2019) and D Since in the original papers, the BERT-PKD is evaluated on the test set, and the DistilBERT is evaluated on the dev set. Thus, for a clear illustration, we present the results in the following Tables 10 and 11 separately.
Table 10: Comparisons between TinyBERT and BERT-PKD, and the results are evaluated on the test of official GLUE tasks.
System
SST-2
MRPC
QQP
MNLI-m
MNLI-mm
QNLI
RTE (2.5k)
(67k)
(3.7k)
(364k)
(393k)
(393k)
(105k)
Acc
F1/Ace
F1/Ace
Acc
Acc
Acc
Acc
Same Student Architecture (24 =6:d 768:d, df =3022)
BERT,
92.0
85.0/79.9
70.7/88.9
81.5
81.0
89.0
65.5
TinyBERT
93.1
87.3/82.6
71.6/89.1
84.6
83.2
90.4
66.0
13

---

Under review as a conference paper at ICLR 2020
Table 11: Comparisons between TinyBERT with DistilBERT, and the results are evaluated the de- v set of GLUE tasks. Mee refers to Matthews correlation and Pear/Spea refer to Pearson/ Searrson/Speamaan
System
CoLA
MNLI-m
MNLI-mm
MRPC
QNLI
QQP
RTE
SST-2
STS-B
(8.5k)
(393k)
(393k)
(3.7k)
(105k)
(364k)
(2.5k)
(67k)
(5.7k)
Mee
Acc
Ace
FI/Ace
Acc
FI/Ace
Acc
Acc
Pear/Spea
Same Student Architecture
=6,6
768;0(30072)
DistilBERT
42.5
81.6
81.1
38.3/824
85.5
87.7/90.6
60.0
92.7
84.5/85.0
ThayBERT
54.0
84.5
84.5
90.6/86.3
91.1
88.0/91.1
70.4
93.0
1/.96
Thus, from the direct comparisons with the reported results in the original papers, we can see the TinyBERT outperforms the baselines (DistilBERT and BERT-PKD) under the same settings of chitecture and evaluation, the effectiveness of TinyBERT is confirmed. Moreover, since BERT-PKD and DistilBERT need initialize their student models with some layers of pre-trained teacher BERT, they have the limitations that the student models have to keep the same size settings hidden size and feedforward/ilter size their teacher BERT. TinyBERT initialized by general distillation, so it has the advantage of being more flexible in model size selection.
14


Reviews:
[{"review_id": "S1xrV0aqFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new knowledge distillation method for BERT models. A number of modifications to the vanilla knowledge distillation method of Hinton et al (2015) are proposed. First, authors suggest adding L2 loss functions between alignment matrices, embedding layer values and prediction layer values. Second, authors propose run knowledge-distillation twice, once with the original pre-trained BERT model as teacher, and then again with task specific fine-tuned BERT as a new teacher. Third, authors emphasize the use of data augmentation for successful knowledge distillation. In Table 2, authors claim a significant lift across GLUE benchmarks with respect to other baseline methods with comparable model size.\n\nWhile the main contribution of this paper is the proposal of empirically useful techniques than theoretical development, the empirical results reported in this paper are somewhat puzzling. \n\nFirst of all, GLUE benchmark scores reported in Table 2 don't seem to be consistent with Table 1 of Sun et al (2019) for BERT-PKD ( https://arxiv.org/pdf/1908.09355.pdf ) or DistilBERT ( https://medium.com/huggingface/distilbert-8cf3380435b5 ). Indeed, BERT-PKD in Sun et al seems to significantly outperform TinyBERT on QNLI (89.0 vs 87.7) and RTE (65.5 vs 62.9), and the gap between BERT-PKD and TinyBERT on other tasks are much smaller if we take numbers reported in the original paper.\n\nIn Table 6, ablation studies with different distillation objectives are reported. Quite surprisingly, without Transformer-layer distillation (No Trm) the performance drops quite significantly. This is unexpected, because baselines such as Sun et al and DistilBERT do not use the Transformer-layer distillation but much more competitive to full TinyBERT than TinyBERT without Transformer-layer distillation. Would there be a reason why TinyBERT is so critically dependent on Transformer-layer distillation? Similarly, the removal of data augmentation (Table 5, No DA) is so detrimental to the performance of the model that it makes me to suspect whether the most of gain is from successful data augmentation. Indeed, 'No DA' row of Table 5 is very close to the performance BERT-PKD in Table 4, although the number of layers is different (4 vs 6). \n\nIn order for the research community to understand the contribution of proposed techniques more thoroughly, I suggest authors to conduct ablation studies with the simplest baseline. That is, rather than starting with the full TinyBERT model, start with a simple but competitive baseline like BERT-PKD, and only add one technique (DA, GD, Transformer-layer distillation) at a time so that readers shall understand what technique is the most important to be added to the baseline, and also whether some of the proposed techniques should always be used in combination.\n\n--- \nAfter Author Rebuttal: authors have addressed all of my concerns quite clearly. Additional experiments which targeted a specific design choice at a time made me much more convinced that the techniques proposed in this paper are useful not only for this particular context but also more broadly applicable.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1xOS2mL9r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the task?\nKnowledge distillation of BERT\n\nWhat has been done before?\nUnlike prior works such as Distilled BiLSTMSOFT (Tang et al., 2019), BERT-PKD (Sun et al., 2019) and DistilBERT, this work\n\ni) Do knowledge distillation at pre training stage also in addition to fine tuning stage.\nii) Student learns from all - embedding layers, attention matrices, hidden states, and final prediction layers. \n\nIn BERT-PKD, student learns from the [CLS]  hidden states of the teacher.\n\nWhat are the main contributions of the paper?\nNovel Transformer distillation method that is specially designed for knowledge distillation of the Transformer-based models.\nNovel two-stage learning framework which performs Transformer distillation at both the pre-training and task-specific learning stages\nResulting TinyBERT being 7.5x smaller and 9.4x faster on inference and significantly outperforms other state-of-the-art baselines on BERT distillation.\n\nWhat are the key techniques used to tackle this task?\nNovel Transformer distillation method that is specially designed for knowledge distillation of the Transformer-based models.\nNovel two-stage learning framework which performs Transformer distillation at both the pre-training and task-specific learning stages\n\nWhat are the main results? Are they significant?\nResulting TinyBERT being 7.5x smaller and 9.4x faster on inference and significantly outperforms other state-of-the-art baselines on BERT distillation with only ∼28% parameters and ∼31% inference time of them.\n\nResults show that three key procedures: TD (Task-specific Distillation), GD (General Distillation) and DA (Data Augmentation) are crucial for the proposed KD method.\n\nProposed distillation objectives - Transformer-layer distillation (attention matrices and hidden states), embedding-layer distillation and prediction layer distillation  are crucial for the proposed KD method.\n\nWeaknesses\nexperimental results were not easily comparable to prior work so it is hard to say if claims are well-supported experimental results\n\nQuestions\nDid authors try other values of lambda\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJeTtyRPtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose TinyBERT, a smaller version of BERT that is trained with knowledge distillation. The authors evaluate on the GLUE benchmark.\n\nOverall, I find the direction of this work exciting and making these large models smaller for practical use is an important research area. The authors provide various ablation experiments that provide insight into their method. The main contribution is experiments comparing various existing distillation methods to different parts of the model (embeddings, layers, prediction layer), so is not particularly novel in contributing new techniques for distillation. That being said, there is importance in contributing these results as they are very useful for others working in the area and on making smaller models. But I would expect the authors to be much more detailed in their experimental description and make it clear in the paper that the comparative baselines are fair and well tuned. \n\nComments:\n\n1. Can the authors please add details for how the model has been trained, such as the datasets used, the number of update steps, the batch size, etc. as well as the finetuning parameters that were cross validated for GLUE? It is difficult to tell in the current setting if the models are comparable to the baselines. The current paper doesn't seem like it could be reproduced. It is particularly important to detail how the finetuning was done, as this is very important for the smaller datasets in GLUE.\n\n2. Is the learning of the distilled model only done on the training dataset, or there is data augmentation beyond the training set? What is the effect without data augmentation?\n\n3. Unfortunately, the performance drop on the GLUE benchmark as shown in Table 2 is fairly large. The authors compare to BERT Small and DistilBERT and I like the baselines, but the claim that the model achieves comparable performance to BERT Base is not true. \n\n4. Was the BERT Small model tuned, or the same learning parameters from BERT Base were used? \n\n5. Can the authors clarify the inference time of BERT Small? The speed improvement of TinyBERT should be the same as BERT Small based on parameter size.\n\n6. The authors experiment with distilling the embedding layer to reduce the number of parameters, why not reduce the parameter size by reducing the vocabulary size? Existing approaches to BERT training use BPE with ~30k vocabulary size or RoBERTa with ~50k vocabulary size, but large gains could be applied here by reducing the size or using softmax reduction techniques that were popular on full vocabulary language modeling datasets like wikitext-103 or billion word. \n\n7. Can the authors please clarify the construction of Table 2? Are those results on the test set (e.g. evaluated on the official GLUE benchmark), or on the dev set? Where are the DistilBERT numbers on the test set coming from, as it is not reported in their paper? ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new distillation-based method for using large pretrained models like BERT to produce much *smaller* fine-tuned target-task models. 

This paper is low-borderline: It has merit and meets our basic standards, but owing to capacity limitations we had to give preference to papers we see as having a higher potential impact. Reviewers had some concerns about experimental design, but those seem to have been fully resolved after discussion. Reviewers were not convinced, even after some discussion, that the method and results were sufficiently novel and effective to have a substantial impact on the state of practice in this area.
Decision: Reject