LightPAFF: A Two-Stage Distillation Framework for Pre-training and Fine-tuning
Authors: Kaitao Song
Hao Sun
Xu Tan
Tao Qin
Jianfeng Lu
Hongzhi Liu
Tie-Yan Liu
Venue:  (2020)
Keywords: Knowledge Distillation
Pre-training
Fine-tuning
BERT
GPT-2
MASS

Abstract: While pre-training and fine-tuning, e.g., BERT~\citep{devlin2018bert}, GPT-2~\citep{radford2019language}, have achieved great success in language understanding and generation tasks, the pre-trained models are usually too big for online deployment in terms of both memory cost and inference speed, which hinders them from practical online usage. In this paper, we propose LightPAFF, a Lightweight Pre-training And Fine-tuning Framework that leverages two-stage knowledge distillation to transfer knowledge from a big teacher model to a lightweight student model in both pre-training and fine-tuning stages. In this way the lightweight model can achieve similar accuracy as the big teacher model, but with much fewer parameters and thus faster online inference speed. LightPAFF can support different pre-training methods (such as BERT, GPT-2 and MASS~\citep{song2019mass}) and be applied to many downstream tasks. Experiments on three language understanding tasks, three language modeling tasks and three sequence to sequence generation tasks demonstrate that while achieving similar accuracy with the big BERT, GPT-2 and MASS models, LightPAFF reduces the model size by nearly 5x and improves online inference speed by 5x-7x.

Under review as a conference paper at ICLR 2020
LIGHTPAFF: A TWO-STAGE DISTILLATION FRAME- WORK FOR PRE-TRAINING AND TUNING
Anonymous authors Paper under double-blind review
ABSTRACT
While pre-traie vre-trainig and fine-tuning, e.g., BERT (Devlin et al. 2018), GPT-2 Rad- ford al., 2019), have achieved great success in language understanding and gen- eration tasks, the pre-trained models are usually too big for online deployment in terms both memory cost and inference speed, which hinders them from practi- cal online usage. In this paper, propose LightPAFF, Lightweight Pre-training And Fine-tuning Framework that leverages two-stage knowledge distillation to transfer knowledge from big teacher model to lightweight student model in both pre- pre-training and fine-tuning stages. In this way the lightweight model can achieve similar accuracy as the big teacher model, but with much fewer parame- ters and thus faster online inference speed. LightPAFF can support different pre- training methods (such BERT, GPT-2 and MASS (Song al., 20T9)) and be applied to many downstream tasks. Experiments on three language understanding tasks, three language modeling tasks and three sequence to sequence generation tasks demonstrate that while achieving similar accuracy with the big BERT, GPT-2 and MASS models, LightPAFF reduces the model size by nearly 5x and improves online inference speed by 5x-7x.
INTRODUCTION
Recently, the pre-training and fine-tuning the-tuning frameworks (Devlin et al. 2018 Radford et al., 2018
2019 Yang et al., 2019b] Liu et al., 2019); Lample & Conneau 2019; Song et al.) (2199) have achieved great success in NLP by learning universal language representations on large-scale lan- guage corpus and transferring knowledge from the pre-trained model to downstream tasks. Based on the carefully designed unsupervised pre-training strategy, e.g., masked language modeling (De- vlin et al. 2018), causal language modeling (Radford et al.] 2018 2019). permutation language modeling Yang et al. 2019b) and masked sequence sequence modeling (Song et al. 2019). the pre-trained model can extract semantic information, understand meanings, generate sentences, even capture common-sense knowledge from large scale corpus. Some of the most representative pre- trained models are BERT (Devlin et al. 20188), XLNet (Yang et al. (2019)) for language understand- ing tasks, GPT/GPT--2 (Radford et al. 2018 2019) for language modeling tasks, and XLMM (Lample) & Conneau 2019), MASS (Song et al. 2019) for sequence to sequence based language generation lasks.
However, pre-trained models are usually of huge model parameter size, e.g., BERT has more than 300 million parameters while GPT-2 has more than 1.5 billion parameters, which brings challenges in memory cost and inference latency when deploying the pre-trained models online or on mo- bile devices. One straightforward solution is use smaller model for pre-training and fine-tuning, which will cause much accuracy degradation that cannot be acceptable for online usage. Previous works (Yang et al. 2019a| Tang et al., 2019) [un et al., 2019) have tried knowledge distillation to reduce the number of parameters for BERT. However, they just distill the knowledge of fine-tuned BERT to the task-specific small model, still with big accuracy drop compared with the original fine- tuned BERT model. Meanwhile, most works focus compressing BERT while few works study on reducing the size other models like GPT-2, XLNet, XLM, MASS.
Knowledge distillation (Bucilu al. 2006 Hinton et al. 2015 Yu al. 2013) is effective and practical way to transfer the knowledge from a cumbersome teacher model to a lightweight student model. However, unlike the previous works (Yang al. 2019a Tang et al. 2019 Sun et al.) 2019)

---

Under review as a conference paper at ICLR 2020
that only leverage the knowledge in the fine-tuned task-specified model, the knowledge in the pre- trained model is also helpful and even more general for language understanding and generation. In this work, we propose LightPAFF, Lightweight Pre-training And Fine-tuning Framework that incorporates knowledge distillation in both pre-training and fine-tuning stages and transfers knowl- edge from pre-trained big model into smaller model. As result, LightPAFF can greatly reduce the number of model parameters and thus memory cost and inference latency, without losing much of accuracy.
As a general framework, LightPAFF can be applied on a variety of pre-trained models (e.g.. BERT, GPT/GPT-2, XLNet, RoBERTa, XLM, MASS) and downstream tasks (e.g.. language understand- ing/moddling/geeeeratoon) In this paper, we carefully choose BERT, GPT-2 and MASS as three study cases, considering the model diversity that BERT leverages masked language modeling for pre-training and is for downstream language understanding tasks, GPT-2 leverages causal language modeling for pre-training and is for language modeling and generation tasks, while MASS leverages masked sequence to sequence modeling for pre-training and is for sequence to sequence based lan- guage generation tasks. We formulate LightPAFF on the pre-training and fine-tuning tasks of BERT, GPT-2 and MASS uniformly, and conduct experiments on three language understanding tasks (SST- 2, QQP, polyphone disambiguation) for BERT, three language modeling tasks (WikiText-2, PTB, WikiText-1033 for GPT-2, and three sequence to sequence based language generation tasks (WMT Zh-En, En-De, En-Fr translation) for MASS. Experimental results demonstrate that LightPAFF re- duces the model parameter of BERT/GPT-2/MASS by nearly 5 and improves the inference speed by 5x ~7x, while achieves similar accuracy with the original BERT/GPT-2/MASS model
TWO-STAGE DISTILLATION FRAMEWORK
Different from existing knowledge distillation methods for pre- pre-trained models (Yang et al.] 2019a Tang et al.] 2219;; (Snn et al. 2019), which only perform distillation in the fine-tuning stage, we perform distillation not only in the fine-tuning stage but also in the pre-training stage to ensure the accuracy of the final student model. The basic idea of our two-stage distillation is shown in Figure and we call this weightt Pre-training And Fine-tuning Framework as LightPAFF for short.
Fine-tune
Big Pre- teettrinnd Model
Big Fine- buned Model
Knowledge distillation
Knowledge distillatonn
Fine-tune
Small Pre-trained Model
Small Fine-tumed Model
Figure LightPAFF pipeline.
LightPAFF runs in four steps:
1. Pre-train the big teacher model using the pre-training dataset.
2. Fine-tune the big teacher model using the dataset of downstream task.
3. Distill the pre-trained big teacher model and obtain light-weight pre-trained model using the pre-training dataset.
4. Fine-tune the light- weight pre-trained model for the downstream task by using the dataset of the task and distilling the fine-tuned big teacher model.
As can be seen, the first and second steps are exactly the same as previous pre-training methods such as BERT (Devlin al. 2018) and GPT-2 (Radford et al. 2019). The third step performs distillation in the pre-training stage, and the fourth step performs distillation in the fine-tuning stage. We adopt general formulation for knowledge distillation in LightPAFF:
(())
Σ - (2.9)
MLE(MLLE (2, (0
+ X 0r),
(1)
We will release the codes and models once accepted.
2

---

Under review as a conference paper at ICLR 2020
where (LE y: 0) is the maximal likelihood loss of the student model 9 over training data pair (r, y) (=; 0r), P(z; 0)) is the KL divergence between the predicted probability distribution on the data of the teacher model 0r and the distribution of the student model 6, and À is hyper- parameter to trade off the two loss terms.
LightPAFF is general framework and can be applied many pre-training fine-tuning tasks. In the following sections, we choose three representative models considering their diversity and code availability: BERT (Devlin al., 2018), which is popular and effective for language understanding tasks such as GLUE benchmark, GPT-2 (Radford et al. 2019). which is popular and effective for language modeling tasks, and MASS (Song et al. 2019). which popular and effective for sequence to sequence generation tasks such as machine translation. Detailed descriptions of two-stage knowl- edge distillation for BERT, GPT-2 and MASS can be found Appendix (Section T). We conduct experimental study of LightPA on BERT in Section 3 GPT-2 in Section MASS in Section 5
and then conduct some general analyses of LightPAFF Section 6
LIGHTPAFF FOR BERT
We follow Equation 1 to conduct knowledge distillation for BERT in both pre-training and fine- tuning stages. The BERT teacher model provides the probability distribution of the masked tokens to the student model during pre-training distillation while provides the probability distribution of the class label to the student model during fine-tuning distillation. In the following sections, we describe the experimental setting and results of LightPAFF for BERT.
3.1 EXPERIMENTAL SETTING
Datasets For BERT pre-training, we use BooksCorpus (800M words) and Wikipedia (2500M words) for English BERT as used in Devlin et al.] (2018), and NewsCorpus (2000M words) and Wikipedia (300M words) for Chinese BERT. For downstream language understanding tasks, we choose SST-2 (Socher et al. (203)) and QQP (Iyer et al., (017)) on English and polyphone dis- ambiguation (PolyDis) (Zhang et al. 2002) on Chinese. Additionally, we add IM unlabeled data for SST-2, 2M for QQP and IM for PolyDis in the fine-tuning stage, where we simply use the teacher model to generate probability distribution to train the student model. More descriptions about datasets be found in Appendix (Section 2
Model Configuration We denote L, H, A as the number of layers, hidden size and the number of attention heads in Transformer. For BERT teacher model, choose the pre-trained BERT-base model (l==12, H=768, A=12) with 110 parameters released by [evlin et al. (008). For BERT the configuration (I==3, H=512, A=8) with 25M parameters for English and
student model, 20M parameters for Chinese, where the parameter difference is due to the different vocabulary sizes.
Training and Inference The hyperparameter À in Equation I is as 1.0 in pre-training according to the performance of pre-training on a held out validation set, and the À in fine-tuning is chosen according to the validation performance on each downstream task. We pre-train BERT on NVIDIA V100 GPUs, each with 16GB memory, and with batch size of 400 sentences and max sequence length of 128 tokens. We fine-tune on NVIDIA V100 GPUs with batch size of 32 sentences.
3.2 RESULTS
Accuracy of LightPAFF The results of LightPAFF on BERT tasks (SST-2, QQP and PolyDis) are shown in Table [ It can be seen that LightPAFF achieves similar accuracy (maintains nearly 99.5% of accuracy) with the BERT teacher model, but reduces the model size by nearly 5x (from 1 110M to 25M or 20M). Compared with the model with the same size of parameters (Transformer small and BERT small) and the previous works (Patient KD and Distilled BiLSTM) that only use
2 BERT leverages masked language modeling and next sentence prediction for pre-training, Considering some works (Yang et al. 2019b Liu al. 2019 Joshi et al. 2019) have achieved good results without next sentence prediction, we only distill BERT on the masked language modeling task in pre-training.
3

---

Under review as a conference paper at ICLR 2020
knowledge distillation in the fine-tuning stage, LightPAFF achieves much better accuracy The trade-off between accuracy and size of the student model can found in Appendix (Section Since BERT leverages additional unlabeled data for distillation, we also study the effect of unlabeled data in Appendix (Section
SST-2
QQP
PolyDis
Acc.
#Param
Fl/Acc.
#Param
Acc.
#Param
BERT (Devlin et al.
2018
93.5
110M
71.2/89.2
110M
95.9
110M
Patient KD (Sun al.
2019
87.5
35M
68.1/87.8
35M
Distilled BiLSTM Tang et al. 2019
90.7
10M
68.2/88.1
10M
Transformer small
83.4
25M
61.1/81.5
25M
89.3
20M
BERT small
89.7
25M
65.4/85.1
25M
93.6
20M
LightPAFF
92.9
25M
70.6/88.6
25M
95.4
20M
Table 1: Results of LightPAFF on BERT tasks. "BERT" represents the BERT-base model released by Devlin et al. (2018) and is taken the teacher model, "Patient KD" distills BERT-base model into Transformer model during fine-tuning, "Distilled BiLSTM" distills BERT-large model into LSTM model during fine-tuning, "Transformer small" and "BERT small" share the same model structure and number of parameters, with the only difference that "Transformer small" has no BERT pre- training. " "LightPAFF" is our method applied "BERT small".
Inference Speedup of LightPAFF We further measure the inference latency LightPAFF on both GPU and CPU. As shown in Table LightPAFF achieves about 6x speedup over the big model on GPU and about 7x speedup on CPU BERT task, demonstrating the advantages LightPAFF for inference speedup.
#Param
Latency (GPU)
Speedup
Latency (CPU)
Speedup
BERT
110M
25 ms
1.00 x
78 ms
1.00 x
LightPAFF
25M
ms
6.25 x
ms
7.09 x
Table 2: Inference speedup of LightPAFF. Evaluation is conducted on NVIDIA Tesla 16GB V100 GPU or Intel(R) Xeon(R) Platinum 8168 CPU, with batch size of 1 sentence and max sequence length of 128 tokens.
LIGHTPAFF FOR GPT-2
Since GPT-2 (Radford et al. 2019) uses language modeling for both pre-training and fine-tuning, the knowledge distillation similar in the two stages, where the GPT-2 teacher model provides the probability distribution of each token in a sentence to the student model during distillation. In the following sections, we describe the experimental setting and results of LightPAFF for GPT-2.
4.1 EXPERIMENTAL SETTING
Datasets For GPT-2 pre-training, use the same English training corpus as used in BERT de- scribed in last section For downstream language modeling tasks, we choose WikiTexxt22 (Merity et al.) 2017). PTB (Mikolov al. 2010)) and WikiText103 (Merity et al. 2017). More descriptions about datasets can be found in Appendix (Section 2).
Note that Distilled BiLSTM Tang et al. 2019) uses BERT-large teacher model while LightPAFF uses BERT-base. LightPFFF achieve better accuracy hy using BERT-large as the teacher model, which is left for future work.
'Due to the constraint of computation resource, we do not use the huge training data in GPT-2 Radford 2019), but we helieve using more data achieve better accuracy, and leave it for future work.
4

---

Under review as a conference paper at ICLR 2020
Model Configuration For GPT-2 teacher model, choose the pre-trained model (L=24, H=1024, A=16, 345M parameters) released by Radford et al. (2019). For student model (we call GPT-2 small), we use the configuration (L=4, H=768, A= 12, 67M parameters). L, H. A are same meaning as defined last Section.
Training and Inference The hyperparameter À for pre-training is set 0.4 according to the zero- shot results, and À for fine-tuning is chosen according to the validation performance on each down- 16 NVIDIA V V00 GPU, each with 16GB memory, and with the batch
stream task. We pre-train size of 512 sentences and max sequence length of 1024 tokens. We fine-tune on I NVIDIA V100 GPU with batch size of 8 sentences and max sequence length of 512 tokens. We report the word- level perplexity via de-tokenizer following Radford et al. (2019).
4.2 RESULTS
Accuracy of LightPAFF The results of LightPAFF GPT-2 tasks (WikiText-2, PTB, Wiki- Text103) are shown in Table It can be seen that lightPAFT achieves large improvements than the models under the same size of parameters without pre-training (Transformer small) and with pre-training (GPT-2 small), while approaches the perplexity of the GPT-2 teacher model in the downstream tasks but with only 1/5x number of parameters.
WikiText-2 w/o FT w/ FT
PTB
10 w/o FT
#Param
w/o FT
w/ FT
w/ FT
GPT-2
345M
22.8
15.5
47.3
17.0
26.4
13.0
Transformer small
67M
71.6
77.0
28.5
GPT-2 small
67M
58.5
25.5
129.6
31.2
61.4
19.8
LightPAFF
67M
32.6
18.8
70.0
22.8
38.7
16.4
Table 3: Results of LightPAFF on GPT-2 tasks in terms of perplexity (ppl). "w/ FT and "w/o FT" mean the pre-trained model is evaluated on downstream tasks with and without fine-tuning. "GPT-2" represents the model released by Radford et al. (2019) and is taken as the teacher model, "Transformer small" and "GPT-2 small" share the same model structure and number of parameters, with the only difference that "Transformer small" has no GPT-2 pre-training, "LightPAFF" is our method applied on "GPT-2 small".
Inference Speedup of L LightPAFF The inference latency of LightPAFF is shown in Table Light- PAFF achieves more than 5x speedup over the big model on GPU while nearly 7 x speedup in CPU, which indicates the effectiveness of LightPAFF for inference speedup on GPT-2.
#Param
Latency (GPU)
Speedup
Latency (CPU)
Speedup
GPT-2
345M
553 ms
1.00
1683 ms
1.00
LightPAFF
67M
101 ms
5.47 x
245 ms
6.87 x
Table Inference speedup of LightPAFF on GPT-2. Evaluation conducted with batch size sentence and max sequence length of 512 tokens. The GPU/CPU configurations follow that in last Section.
LIGHTPAFF FOR MASS
Similar to BERT, the MASS teacher model provides the probability distribution on each token in a masked segment to the student model during pre-training distillation, and provides the probability distribution on each token a target sequence to the student model during fine-tuning distillation. In the following sections, describe the experimental setting and results of LightPAFF for MASS.
5

---

Under review as a conference paper at ICLR 2020
5.1 EXPERIMENTAL SETTING
Datasets For MASS pre-training, we use 50M monolingual data of each language from the Newscrawl dataset, following Song al. (2019). For fine-tuning tasks, choose rich-resource task (WMT17 Chinese-English, briefly Zn-En) and two low-resource tasks (WMT14 English- French, briefly En-Fr and WMT16 English-German German, briefly En-De), where we simulate the low- resource scenario by only choosing a small part of training data. More descriptions about datasets can be found in Appendix (Section 2
Model Configuration For MASS teacher model, we choose the pre-trained model with the configu- ration (L=6, H=1024, A=16) for both encoder and decoder, which is released by Song et al. (2019). For student model, we use the configuration (L=6,, H=512, A=8) for the encoder and (L=4, H=512, A=8) for the decoder. H, A are same meaning as described in the BERT section.
Training and Inference The hyperparameter À for pre-training is set as 0.7 according to the per- formance of pre-training on a held out validation set, and for fine-tuning is adjusted according each downstream task. We pre-train on 8 NVIDIA V100 GPUs
to the validation performance with batch size of 48,000 tokens. We fine-tune on 8 NVIDIA V100 GPUs with batch size 25,600 tokens. We report the BLEU score by SacreBLEU
5.2 RESULTS
Accuracy of LightPAFF The results of LightPAFF on MASS tasks are shown in Table It can be seen that compared with the pre-trained model with the same size of parameters (MASS small), LightPAFF achieves 2.5-3.8 BLEU points improvement in low-resource tasks and 1.1 BLEU points improvement in rich-resource task. LightPAFF achieves similar accuracy with the MASS teacher model, but reduces the model size by nearly 5x (from 213M to 42M or from 307M to 67M).
WMT17 Zh-En
WMT16 En-De (100K)
WMT14 En-Fr (100K)
BLEU
#Param
BLEU
#Param
BLEU
#Param
MASS
25.2
307M
33.1
213M
26.7
213M
Transformer small
21.3
67M
23.8
42M
14.1
42M
MASS small
23.8
67M
28.4
42M
23.2
42M
LightPAFF
24.9
67M
32.2
42M
25.7
42M
Table 5: Results lightPAFF on sequence to sequence based language generation tasks in terms of BLEU score. "MASS" represents the MASS model released by (Song et al.) (2019) and taken as the teacher model, "Transformer small" and "MASS small" share the same model structure and number of parameters, with the only difference that "Transformer small" has no MASS pre- pre-training, "LightPAFF" is our method applied on "MASS small".
Inference Speedup of LightPAFF As shown in Table 6 LightPAFF achieves nearly 5x speedup over the big pre-trained model on both GPU and CPU, which demonstrates the advantages of Light- PAFF for inference speedup.
#Param
Latency (GPU)
Speedup
Latency (CPU)
Speedup
MASS LightPAFF
213M
27 ms
1.00 x
94 ms
1.00 x
42M
6 ms
4.50 x
18 ms
5.22
Table 6: Inference speedup LightPAFF on MASS. Evaluation is conducted by generating one sen- tence at a time autoregressively. The GPU/CPU configurations follow that in last Section.
6 FURTHER ANALYSIS
To better understand LightPAFF, we conduct some deep analyses on LightPAFF this section.

---

Under review as a conference paper at ICLR 2020
6.1 ABLATION STUDY
To study the impact of knowledge distillation in each stage, we conduct conduct ablation studies by removing distillation from one of the stages. The results are shown in T We have several observations:
1) The comparison between Method 1 with shows that without pre-training, using knowledge distillation fine-tuning stage can improve the accuracy of the model. 2) Comparing Method and 3 or Method 2 and 5, we see that using pre-training can always boost accuracy. 3) Comparing Method 3 and or Method 5 and 6, it can seen that using knowledge distillation in pre-training can boost the accuracy. 4) Comparing Method 3 and 5 or Method 4 and 6, it can be seen that using knowledge distillation in fine-tuning can boost the accuracy. Overall, the distillation in both stages effective and necessary in LightPAFF.
ID
PT K
FT
FT KD
BERT
GPT-2
MASS
1 (Transformer small)
83.4
71.6
23.8
2
86.9
34.3
26.2
3 (Pre-trained small)
89.7
25.5
28.4
4
90.8
21.6
29.5
5
92.0
22.1
31.0
6 (LightPAFF)
92.9
18.8
32.2
Table Ablation study on knowledge distillation in pre-training and fine-tuning stages, where "PT" means pre-training, "FT" means fine-tuning and "KD" means knowledge distillation. The results of BERT//PT-2/MASS. are the accuracy on SST-2 task, the perplexity on WikiText-2, and the BLEU score on WMT16 En-De (100K) respectively.
6.2 GENERALIZATION ANALYSIS
Previous works YYang et al. 2018 Lan et al. 2018) have shown that knowledge distillation can help improve the generalization ability, and some studies (Shirish Keskar et al. 2016 Chaudhari jtt al,, 2016) indicate the relationship between model generalization and the width local minima in the loss surface, where models with wider local minima are more robust to perturbations. Inspired by Tan et al. (2019), we perturb the parameters the models to observe the accuracy changes, in order demonstrate that the knowledge distillation in both pre-training and fine-tuning stages in LightPAFF improves the generalization ability.
We perturb model as (0) = 0f 0 * *N(0, o²), where 0f is the i-th parameter of 0', where o¹ represents the parameter named by (c.g., We. WK or Wv self-attention) in model 0, and Ø the average parameter Ø¹ We sample from the normal distribution N with standard variance where larger represents bigger perturbation shown in Figure It can be seen that the accuracy of the model without knowledge distillation drops faster matter pre-training and fine-tuning stage, which demonstrates knowledge distillation in both pre-training and fine-tuning stages make the model more robust.
the parameter. The results with varying
6.3 HYPERPARAMETER FOR DISTILLATION AND DIFFICULTY OF PRE-TRAINING TASKS
Hyperparameter LightPAFF employs a hyperparameter À (see Equation to control the trade- off between knowledge distillation from the teacher and learning from the original label the data. During pre-training, the optimal À according to our experiments is 1.0 for BERT, 0.7 for MASS and 0.4 for GPT-2. Bigger means to use more information from teacher while less information from ground-truth label in the original data. This seems suggest that the pre-trained BERT teacher model preserves the most information of the data, while the pre-trained GPT-2 teacher model pre- serves the least information of the data. To verify this, further check the training results of the three models.
Prediction Accuracy of Pre-training For this study, we pre-train BERT, GPT-2 and MASS mod- els on the same data corpus, with similar model configurations (12-layer BERT, 12-layer GPT-2
7

---

Under review as a conference paper at ICLR 2020
350
350
with kd
85
90
300
without
without
Accuracy
80
Accuracy
85-
Perplexity
250
Perplexity
2.50
200
200
75
80
150
150
70-
75
100
100
65
with kd
with kd
70
wilbout
50
50
0.0 0.2 D44 D.6 0.8 1.0 Slandard Variance
0.0 0.2 0.4 0.6 D.8 1.0
0.0 0.4 0.6 0.8 1.0
ekek 0.0 0.4 0.6 0.8 10
Slandard Variance
Standard Variance
Standard Variance
BERT pre-training
(b) BERT fine-tuning
GPT-2 pre-training
(d) GPT-2 fine-tuning
Figure 2: Generalization Analysis. The result of BERT is the accuracy of SST-2 on valid set while the result GPT-2 is the perplexity of WikiText-2 on valid set.
Model
Accuracy
Input Tokens
BERT
0.73
85%
MASS
75% 0.42
0.62
GPT-2
50%
Table 8: The comparison of accuracy, and average number of input tokens in the pre-training task between BERT, GPT-2 and MASS.
and 6/6-layer MASS, with hidden size of 768) and same vocabulary. We test the three models on held-out dataset. In particular, we compute the accuracy token predictions. The results are shown in Table g It can be seen that BERT achieves the highest accuracy while GPT-2 the lowest. Since the three models are of similar complexity (e.g., with similar architecture and similar number of parameters), our hypothesis is that the pre-training task of BERT is easiest and that of GPT-2 is the most difficult, which is verified intuitively as below.
Task Difficulty of Pre-training Many pre-training tasks in natural language processing are to predict a token conditioned on sequence of input tokens. BERT masks 15% tokens in a sequence and predicts these 15% tokens given the remaining 85% token GPT-2 praint a language model, ie.. predicting every token in a sequence conditioned all its preceding tokens. For sentence with n tokens, when predicting the i-th token, the model can see the previous i - tokens. Therefore, the average tokens that be seen is (n 1)/2, which is about 50%. For MASS, 50% tokens can be seen in the encoder, while 50% tokens are predicted in the decoder side through language modeling, where average 50% tokens be seen following the calculation in GPT-2. Therefore, predict a token, on average MASS uses 75% tokens. To summarize, as shown in Table g to predict each token, on average BERT uses 85% tokens of a sequence, MASS uses 75% while GPT-2 only 50%. Thus, the difficulty of the pre- pre-training tasks should be BERT<MMAS GPT-22, which is consistent with the observations in last two paragraphs.
7 CONCLUSION
In this work, we have proposed LightPAFF. a two-stage knowledge distillation method to reduce the size of the big pre-trained models, which transfers the knowledge from big pre-trained model into small model in both pre-training and fine-tuning stages. We formulate LightPAFF on BERT, GPT-2 and MASS models uniformly and conduct experiments on BERT, GPT-2 and MASS with downstream tasks in total. The experimental results demonstrate that LightPAFF reduces the model size by nearly 5 5x and improves the inference speed by 5x ~ 7 xx while achieves similar accuracy
We ignore the 8:1:1 strategy in BERT, which does affect the percentage much.
8

---

Under review as a conference paper at ICLR 2020
with the original BERT/GPT-2/MAS,, model. For future works, we will cover more pre-trained models and downstream tasks. On the other hand, we will also explore other methods such as quantization, pruning to compress the big pre-trained models.
REFERENCES
Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizii Model compression. In Proceedings of 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp- 535-541. ACM, 2006.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing Gradi- ent Descent Into Wide Valleys. arXiv e-prints, art. 1XXi:1611..01880 Nov 2016.
Jacob Devlin, Ming Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.04805, 2018.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0255 003.33330.0 2015.
Shankar Iyer, Nikhil Dandekar, and Kornél Csernai. First quora dataset release: Question pairs. data. quora. com, 2017.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint 190. 2019.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining, arXiv preprint WxXiv:/90.002999 2019.
Lan, Xiatian Zhu, and Shaogang Gong. Knowledge distillation by on-the-fly native ensemble. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, Curran Associates Inc., 2018.
pp. 7528- -5388.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19X1v:1907.11666 2019.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In ICLR, 2017.
Tomas Mikolov, Martin Karafit, Luks Burget, Jan Cernock, and Sanjeev Khudanpur. Recurrent neural network based language model. In INTERSPEECH, pp. (045-1048, 2010.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL https://s3-us-westt amazonaws. com/openai- understanding paper. pdf. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. arXiv e-prints, art. arXiv: 100::::600000000 160.004366. Sep 2016.
Richard Socher, Alex Perelygin, Jean Wu. Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Kaitao Song. Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mass: Masked sequence sequence pre-training for language generation. International Conference on Machine Learning, pp- 5926-5936, 2019.

---

Under review as a conference paper at ICLR 2020
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint FrXiv:190.009333 2019.
Xu Tan, Yi Ren, Di He, Tao Qin, and Tie- Yan Liu. Multilingual neural machine translation with knowledge distillation. In International Conference Learning Representations, 2019. URL
htttss://ppenreval net wwt.://wwwwweeweeettee.m.. forummid==$lgusoRR3
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv: 12166, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pupers), pp. 1112-1122, 2018.
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan Yuille. Knowledge distillation in generations: More tolerant teachers educate better students. arXiv preprint arXiv:1805.0555: 2018.
Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with multi-task knowledge distillation for web-scale question answering system. arXiv preprint arXiv:1904.09636. 2019a.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.0883 2019b.
Dong Yu, Kaisheng Yao, Hang Su, Gang Li, and Frank Seide. Kl-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7893-7897. IEFE, 2013.
Zi-Rong Zhang, Min Chu, and Eric Chang. An efficient way to learn rules for grapheme to-honemm- conversion in chinese. International Symposium on Chinese Spoken Language Processing, 2002.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX
TWO-STAGE DISTILLATION FOR BERT, GPT-2 AND MASS
We formulate the knowledge distillation for pre-training and fine-tuning tasks in BERT, GPT-2 and MASS, where BERT (Deviin al. 2018) pre-trains on masked language modeling and fine-tunes on language understanding tasks, GPT-2 (Radford et al., 2019) pre-trains and fine-tunes both on causal language modeling tasks, while MASS (Song et al., 2019) pre-trains on masked sequence to sequence modeling and fine-tunes sequence to sequence based language generation tasks.
1.1 KNOWLEDGE DISTILLATION IN PRE-TRAINING
Masked Language Modeling The knowledge distillation loss for masked language modeling (De- vlin al. 2018) LMLM is as follows:
LMLM (0) = ΣΣΣ
(1-A))16m =k)22 ; (o)))-oog
MM,
P( k/w k
:) 0), (2)
TED MEM
where D the corpus for pre-training, a sentence, M is the of indices of the masked tokens in sentence r and Zm is the corresponding prediction of the masked token, [V] is the vocabulary size. x MM is the sentence with token indices in M are masked. The masking operation in BERT is as follows: 15% tokens are randomly masked, where 80% are masked with the token "[MASK]", 10% are masked with random token, and the rest 10% are remained unchanged. is the indicator function indicating if Im equals to k, ( = \M 0r) is the probability distributions for the predicted token generated by the teacher model 0r which is fixed during training.
Causal Language Modeling The knowledge distillation loss for causal language modeling RRad- Ford al. 2019) LCLM formulated as:
M4]
CCLM -E (1-A))1fam = + k| (r))mm;0)))( xro) P(xm kl (), (3) ZED m=1k=
where |x|] is the length the sentence x, Zmm is the tokens preceding position m.
Masked Sequence to Sequence Modeling The knowledge distillation loss for masked sequence to sequence modeling (Song et al. 2019) LMSSM is formulated as:
I
LMSSM (0) = - - 1(xm = k) + Q(am = kt k))x2())))
(4)
xED
x log P(#m k| Veit ()).
where and are the start and end position of the masked segment in masked sequence sequence modeling, =t denotes the sentence segment from position to f, rhve denotes the sentence where the segment from position s and & are masked. The masking strategy in Song et al. (2019) as follows: the masked tokens in the sentence will be a [M] token 80% of the time, random token 10% the time and a unchanged token 10% of the time.
1.2 KNOWLEDGE DISTILLATION IN FINE-TUNING
Language Understanding Language understanding task usually refers to the tasks like QQP (yer et al. 2017). SST-2 (Socher et al. 2013), MNLI (Williams et al. 2018). which requires understand- ing of the sentence in order to make predictions. The loss function of the knowledge distillation for fine-tuning on the language understanding task is formulated as:
CLv(6') = Σ -(1-)) 1 = + X Q(y = k(x; x log P(y = k|2;; 8'), (5)
(Y
'BERT leverages masked language modeling and next sentence prediction for pre-training. Considering some works (Yang al. 2019b Liu al. 2019 Doshi et al. 2019) have achieved good results without next sentence prediction, we only distill BERT the masked language modeling task in pre-training.

---

Under review as a conference paper at ICLR 2020
where D' is the supervised training corpus in downstream tasks, is the input sentence, y is the label and [V] is the size of label set, e.g., y/ represents the sentiment classes for sentiment classification task, and [V] is the number of classe [ Q is output probability of the teacher model 0 that is fine- tuned from pre-trained teacher model 0r, while P is the output probability of student model that initialized from the pre-trained model and only differs from 0 in the task-specific output layer.
Language Modeling'Generation Language modeling/generrtionn refers to the tasks like WikiText-2 (Merity et al. ). PTB (Mikolov et al. (0100) and WikiText103 (Merity et al. 2017). The knowledge distillation loss is
M CLM((O) EE EE F=1
(1- k3 Q(mm k|2;; 07)) log P(am k|r<m; 0'), (6)
which is similar to that in Equation 3 except for that training corpus is from the specific downstream task, and teacher model 8/ and student model 0' are already pre-trained.
Sequence Sequence based Language Generation Sequence sequence based language gen- eration tasks include neural machine translation. text summarization, grammatical error correction, text style transfer, etc. The knowledge distillation loss is formulated as
M
Lss(O') = Σ E
(11
1ym k} + Q(ym klyeem xz 0l-))
(a.))E
k=1
log P(ym = kly/mm x; 8'),
where (x, v) the supervised sentence pair, |y| is the length of the target sentence.
2 DATASET
BERT We select different datasets to evaluate our method on language understanding tasks for BERT. SST-2 (Stanford Sentiment Treebank 2) is a movie review dataset with binary labels indi- cating positive or negative sentiment of the sentences. QQP (Quora Question Pairs) contains ques- tion pairs with binary labels indicating whether they are duplicate. More details about the above two datasets can be found in Devlin et al. (2018). PolyDis (Polyphone Disambiguation) consists of sentences with polyphonic characters whose labels are their pronunciations on Chinese Man- darin. PolyDis dataset contains 166,185 sentences for training and 43,426 sentences for testing with 79 most frequent polyphonic characters. To further reduce the gap between teacher model and lightPAFF, we add additional unlabeled source data in the tion. For PolyDis, we simply extract unlabeled data which contains polyphonic words from the pre-training corpus. For SST-2 and QQP, we get unlabeled data of movie reviews and question pairs from Kaggle, a competition website with lot of open datasets, and we remove all the data that occurs in the valid and test set of SST-2 and QQP. The additional dataset of SST-2 can be got htttp:://ww.kkgle..coommm
stage of BERT distilla-
imdb review review-dataset dataset and QQP from https:/ www. wtt.://lee..omm/eeeeeeeeeeeeeeee http:///wwee1naa/ororaaaaa....
GPT-2 Different from BERT, GPT-2 (Radford et al. 2019) proposed a web scrape filter data from web pages by emphasizing document quality. These resulting dataset, named as WebText, contains nearly 40GB text after de-duplication and basic cleaning operations. The size of WebText dataset is 2.5 times to BERT training corpus. Due to computation resource constraint, we do not use the huge training data in GPT-2, but we believe using more data can achieve better accuracy. For language model tasks (PTB, Wikitext-2, Wikitext- 103), each task has been split into train/valid/test, and the detail of each dataset shown in Table 9
*We make two clarifications here: Language understanding can also take multiple sentences input, such as QQP, where the two sentences are concatenated together and regarded as one sentence. Therefore, we just formulate the input as one sentence for simplicity. 2) Language understanding tasks can also take multiple tokens or labels as output, such as sequence labeling task like NER, polyphone disanbiguation. As there is no explicit dependency when predicting different labels in BERT model, we just formulate the task as single label for simplicity.
12

---

Under review as a conference paper at ICLR 2020
PTB
Wikitext-2
Wikitext- 103
Train
Valid
Test
Train
Valid
Test
Train
Valid
Test
Tokens
887K
70K
78K
2.08M
217K
245K
103M
217K
245K
Table 9: Statistics of the three datasets of language modeling tasks in GPT-2.
We conduct experiments datasets to evaluate the effectiveness of LightPAFF in sequence to sequence tasks. For rich-resource task, WMT17 Chinese-Fnglise (Zh-En). For low-resource scenario, We build the datasets by randomly sampling 100K bilingual data from the standard WMT14 English-French (En-Fr) and WMTI6 English-German (En-De) datasets. For similar language pairs (i.e., En-Fr and En-De), learn jointed dictionary with 32,000 tokens. For discrete language pair (i.e., Zh-En), we respec- tively learn dictionary for each language with 40,000 tokens.
MASS
rich-resource and two low-resource machine translation
TRADE-OFF BETWEEN ACCURACY AND MODEL SIZE
We compare the accuracy our LightPAFF models with different model size, and report the results of PolyDis task BERT, as shown in Table I0 It can be seen that there is trade-off between accu- racy and model size. Different models can be chosen for online deployment when taking aceuracy, memory cost and inference latency into consideration.
#Param
Acc.
BERT
110M
95.9
LightPAFF
30M
95.5
LightPAFF
20M
95.4
LightPAFF
8M
93.9
Table 10: The accuracy of LightPAFF with different number parameters on PolyDis task. We also list the results of the original BERT model with 110M parameters.
4 ANALYSIS ON THE USE OF UNLABELED DATA
During the fine-tuning in BERT, leverage unlabeled data for knowledge distillation to reduce the accuracy gap between the teacher model and student model. We study how unlabeled data can help improve the accuracy of student model, shown in Table [T] It can be seen that using unlabeled data for knowledge distillation can indeed boost the accuracy of the student model (LightPAFF))
SST-2 Acc.
QQP F1/Acc.
PolyDis Acc.
unlabeled data
without with
91.3
67.5/87.7
94.5
92.9
70.6/88.6
95.4
Table 11: The accuracy comparison of LightPAFF without and with unlabeled data on the BERT fine-tuning tasks.
To further check teacher model can benefit from unlabeled data, we also use BERT teacher model generate labels of unlabeled data and then teach itself. The result SST-2 test shows that the accuracy drops 0. 0.1% (from 93.5% to 93.4%) which means unlabeled data may not help the teacher model itself but help student models.
13


Reviews:
[{"review_id": "SylMM3oaYH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper sets to solve the problem that many language models such as GPT-2 despite having achieved great success in their language tasks are too big to deploy in applications. They propose LightPAFF, a framework that allows to transfer knowledge from a big teacher model to a lightweight student model, thus solving the deployability issue at hand. They conduct experiments showing LightPAFF achieves a 5x reduction in the number of parameters and a 5x-7x improvement on the inference speed, while roughly preserving performance. \n\nThe distillation framework the authors use in their methods is not new. It has been proposed in previous work, as the authors noted. As opposed to previous works, where distillation is performed in the fine tuning stage, the authors of LightPAFF propose a two stage distillation procedure instead that performs distillation at the pre training phase and fine tunes this distilled model for use in a downstream task using a big fine tuned teacher model and the dataset of the task. The experimental results show these results to be meaningful. They achieve better accuracy and similar compression than previous approaches. It's main weakness is that the method doesn't seem to be particularly new. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SygGucIwYH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a method for distilling language models such as BERT into much smaller versions with similar accuracy. The approach, therefore, saves both memory and computation resources. \n\nThe difference to existing methods is that the authors method also performs distillation during pretraining. The authors apply their method to BERT and run experiments, comparing the distilled BERT model with the original one as well as existing distillation methods.\n\nThe experiments are extensive with impressive results. I'm torn. On the one hand, these results show that this method (which is simple and straight-forward) reduces memory and computation time (latency) substantially while maintaining similar accuracy. It can be very useful in practice. On the other hand, the idea is so simple and straight-forward, that I'm not sure ICLR (a top machine learning conference) is the right place for it. I tend to think that the paper should get accepted though since the empirical work is really exhaustive and impressive. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SkxmRuSTKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary: This work leverages knowledge distillation both in pre-training and fine-tuning stages to learn a more compact student model that approximates the performance of a teacher model. Extensive experiments with different knowledge distillation loss functions are conducted on a number of representative language representation models including BERT, GPT-2 and MASS to show the benefit of the method. \n\nStrengths:\n\nThe numerical results show some promise of the proposed method to achieve competitive performance to the teacher network and outperform the considered baselines. The evaluated 9 downstream tasks cover a wide range of language understanding and generation problems and they do prove the effectiveness of knowledge distillation in boosting the accuracy.\n\nWeaknesses:\n\nMy main concern is about the limited novelty of approach. The knowledge distillation method developed in the paper seems not particularly novel in principle. Although extensive numerical results are provided to show the effectiveness of knowledge distillation, the value-added beyond applying different distillation loss functions in different NLP tasks is still insufficient. Particularly, the loss function that combines maximal likelihood loss with KL divergence looks similar to those used in the paper [1]. The connection and difference to that related work need to be clarified. \n\n[1] Yu, Dong, et al. \"KL-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition.\" 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2013.\n\n=== update after author response ===\n\nThank you for the response. Although interesting and showing some promise in a bunch of applications, I am still not convinced that the proposed approach is novel enough in principle.  I thus stick to my assessment of this paper as borderline leaning to rejection.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a two-stage distillation from pretrained language models, where the knowledge distillation happens in both the pre-training and the fine-tune stages.  Experiments show improvement on BERT, GPT and MASS.  All reviewers pointed that the novelty of the work is very limited.
Decision: Reject