Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
Authors: Iulia Turc
Ming-Wei Chang
Kenton Lee
Kristina Toutanova
Venue:  (2020)
Keywords: NLP
self-supervised learning
language model pre-training
knowledge distillation
BERT
compact models

Abstract: Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domain text through self-supervised pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained language representations have been proposed (Sun et al., 2019; Sanh, 2019). However, surprisingly,  the simple baseline of just pre-training and fine-tuning compact models has been overlooked. In this paper, we first show that pre-training remains important in the context of smaller architectures, and fine-tuning pre-trained compact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm, Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have a compound effect even when sequentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.

Under review as a conference paper at ICLR 2020
WELL-READ STUDENTS LEARN BETTER: ON THE IM- PORTANCE OF
COMPACT MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent developments in natural language representations have been accompanied by large and expensive models that leverage vast amounts of general-domahn text through self-super pre-training. Due to the cost of applying such models to down-stream tasks, several model compression techniques on pre-trained lan- guage representations have been proposed (Sun al. 2019a Sanh 2019). How- ever, surprisingly, the simple baseline of just pre- pre-training and fine-ture tt-ing compact models has been overlooked. In this paper, first show that pre-tra pre-training remains important the context of smaller architectures, and fine-tuning pre- pre-trained com- pact models can be competitive to more elaborate methods proposed in concurrent work. Starting with pre-trained compact models, we then explore transferring task knowledge from large fine-tuned models through standard knowledge distillation. The resulting simple, yet effective and general algorithm. Pre-trained Distillation, brings further improvements. Through extensive experiments, we more generally explore the interaction between pre-training and distillation under two variables that have been under-studied: model size and properties of unlabeled task data. One surprising observation is that they have compound effect even when se- quentially applied on the same data. To accelerate future research, we will make our 24 pre-trained miniature BERT models publicly available.
INTRODUCTION
Self-supervised learning general-domain text corpus followed by end-task learning is the two- staged training approach that enabled deep-and- Transformer-based networks (Vaswani et al,, 2017) to advance language understanding (Devlin et al.) 2018]] 1 Yang et al. 2019b] [Sun et al. 2019b Liu et al. 2019). However, state-of-the-art models have hundreds of millions of parameters, incur- ring high computational cost. Our goal is to realize their gains under a restricted memory and latency budget. We seek a training method that well rell-performing, general and simple and can leverage additional resources such unlabeled task data.
Before considering compression techniques, we start with the following research question: Could directly train small models using the same two-staged approach? In other words, explore the idea of applying language model (LMM) pre-training and task fine-tuning to compact architectures directly. This simple baseline has so far been overlooked by the NLP community, potentially based on an underlying assumption that the limited capacity of compact models is capitalized better when focusing on the end task rather than a general language model objective. Concurrent work ours proposes variations of the standard pre- tnee-raining-- feaining-fine-tunie fine-tuning procedure, but with limited generality (Sun al. 2019a Sanh 2019). We make the surprising finding that pre-training- fine-tuning in its original formulation is competitive method for building compact models.
For further gains, we additionally leverage knowledge distillation (Hinton et al., 2015). the standard technique for model compression. A compact student is trained to recover the predictions of a highly accurate teacher. In addition to the posited regularization effect of these soft labels (Hinton et al., 2015), distillation provides means of producing pscudo-labels for unlabeled data. By regarding LM pre-training of compact models as a student initialization strategy, we can take advantage of both methods. The resulting algorithm is sequence of three standard training operations: masked LM (MLM) pre-training (Devlin et al. 2018), task-specific distillation, and optional fine-tuning. From here on, we will refer to Pre-trained Distillation (PD) (Figure J. As will show in

---

Under review as a conference paper at ICLR 2020
Algorithm
Large teacher
Require: student 8, teacher Ω2 unlabeled LM data DLM, unlabeled transfer data Dr, labeled data D,
Compact Model
Unlabeled
Pre- traininnn
1: Initialize 0 by pre-training an MLM on DLM
2: for each Dr do
Unlabeled transfer data
Distilation
3: Get loss L f- Ey Pn(yl=) log Po(ylz)
4: Update student 0 <- BACKPROP(L,
Fine-tuning (Oplional)
5: end for
6: Fine-tune f on Dr
> Optional step.
Finel Compact Model
7: return
Figure I: Pre-trained Distillation
Section 6.2 PD outperforms the pre-training + fine-tuning (PF) baseline, especially in the presence of large transfer set for distillation.
In a controlled study following data and model architecture settings in concurrent work (Section show that Pre-trained Distillation outperforms is competitive with more elaborate approaches which use either more sophisticated distillation of task knowledge (Sun et al.) 2019a) or more so- phisticated pre-training from unlabeled text (Sanh 20T9). The former distill task knowledge from intermediate teacher activations, starting with heuristically initialized student. The latter fine-tune a compact model that is pre- pre-trained on unlabeled text with the help of larger LM teacher.
One of the most noteworthy contributions of our paper are the extensive experiments that exam- ine how Pre-trained Distillation and its baselines perform under various conditions. We investigate two axes that have been under-studied in previous work: model size and amount/quality of unla- beled data. While experimenting with 24 models of various sizes (4m to 110m parameters) and depth/width trade-offs, we observe that pre-trained students can leverage depth much better than width; in contrast, this property is not visible for randomly-initien initialized models. For the second axis, vary the amount of unlabeled data, as well as its similarity to the labeled set. Interestingly, Pre- trained Distillation more robust to these variations in the transfer set than standard distillation.
Finally, in order to gain insight into the interaction between LM pre-training and task-specific distil- lation, we sequentially apply these operations on the same dataset. In this experiment, chaining the two operations performs better than any one of them applied in isolation, despite the fact that single dataset was used for both steps. This compounding effect is surprising, indicating that pre-training and distillation learning complementary aspects of the data.
Given the effectiveness of LM pre-training on compact architectures, we will make our 24 pre- trained miniature BERT models publicly available in order to accelerate future research.
2 PROBLEM STATEMENT
Our high-level goal is build accurate models which fit a given memory and latency budget. There are many aspects to explore: the parametric form of the compact model (architecture, number of parameters, trade-off between number of hidden layers and embedding size), the training data (size, distribution, presence or absence of labels, training objective), etc. Since an exhaustive search over this space is impractical, we fix the model architecture to bidirectional Transformers, known to be suitable for wide range of NLP tasks (Vaswani et al. 2017 Devlin et al., 2018). The rest of this section elaborates on the training resources we assume to have at our disposal.
The teacher is highly accurate but large model for end task, that does not meet the resource constraints. Prior work on distillation often makes use of an ensemble of networks (Hinton et al.) 2015). For faster experimentation, best architectural choice. In Section 4 the teacher is tre-ted BERT BASE fine-tuned on labeled end-task data. In Section 6 use BERT instead.
single teacher, without making statement about the
Students are compact models that satisfy resource constraints. Since model size qualifiers are rela- tive (e.g., what is considered small in data center be impractically large mobile device),
2

---

Under review as a conference paper at ICLR 2020
H=128
H=256
H=512
H=768
H=128
H=256
H=512
H=768
L=2
4.4
9.7
22.8
39.2
L=2
65.24
31.25
14.44
7.46
L=4
4.8
11.3
29.1
53.4
L=44
32.37
15.96
7.27
3.75
L=6
5.2
12.8
35.4
67.5
L=6
21.87
10.67
4.85
2.50
L=8
5.6
14.4
41.7
81.7
L=8
16.42
8.01
3.64
1.88
L=10
6.0
16.0
48.0
95.9
L=10
13.05
6.37
2.90
1.50
L=12
6.4
17.6
54.3
110.1
L=12
11.02
5.35
2.43
1.25
(a) Millions of parameters
(b) Relative speedup wrt LARGE on TPU v2
Table I: Student models with various numbers transformer layers (L) and hidden embedding sizes (H). Latency is computed on Cloud TPUs with batch size and sequence length 512. For readability, we focus on five models (underlined) for most figures: Tiny (L=2, H=128), Mini (L=4, H=256), Small (L=4, H=512), Medium (L=8, H=51 and Base (1=12, H=768).
investigate an array of 24 model sizes, from TransformeryINY (4m parameters) all the way up to TransformergAss (110m parameters D The student model sizes and their relative speed-up com- pared to the LERGE teacher can be found in Table I Interested readers can situate themselves on this spectrum based on their resource constraints. For readability, most plots show a selection of models, but we verify that our conclusions hold for all 24.
Labeled data (D2) is a set of N training examples {(11.11). 31). (IN, BN)}, where x, is an input and y a label. For most NLP tasks, labeled sets are hard to produce and thus restricted in size.
Unlabeled transfer data (Dr) is set of M input examples of the form {x',..., H'M sampled from possibly not identical to the input distribution of the labeled set.
a distribution that is similar During distillation, the teacher transfers knowledge to the student by exposing its label predictions for instances s'm DDT can also include the input portion of labeled data D, instances. Due to the lack of true labels, such sets are generally easier to produce and consequently larger than labeled ones. Note, however, that task-relevant input text is not readily available for key tasks requiring paired texts such natural language inference and question answering. well domain specific dialog understanding. In addition, for deployed systems, input data distribution shifts over time and existing unlabeled data becomes stale (Kim al. 2017).
Unlabeled language model data (DLAS) is a collection of natural language texts that enable un- supervised learning of text representations. We use it for unsupervised pre-training with masked language model objective (Devlin et al. 2018). Because no labels are needed and strong domain similarity is not required, these corpora are often vast, containing thousands of millions of words.
The distinction between the three types of datasets is strictly functional. Note they are not necessarily disjunct. For instance, the same corpus that forms the labeled data can also be part of the unlabeled transfer set, after its labels are discarded. Similarly, corpora that are included in the transfer set can also be used as unlabeled LM data.
PRE-TRAINE DISTILLATION
Pre-trained Distillation (PD) (Figure models that can leverage all the resources enumerated in Section It consists of a sequence of three standard training operations that can be applied any choice architecture:
general, yet simple algorithm for building compact
1. Pre-training on DLM- A compact model is trained with a masked LM objective (Devlin) et al. 2018), capturing linguistic phenomena from a large corpus of natural language texts.
2. Distillation on Dr. This well-read student is now prepared to take full advantage of the teacher expertise, and is trained on the soft labels (predictive distribution) produced by the teacher. As will show in Section 6.2) randomly initialized distillation constrained by the size and distribution of its unlabeled transfer set. However, the previous pre-training step mitigates to extent the negative effects caused by imperfect transfer set.
3. (Optional) fine-tuning on Dr. This step makes the model robust to potential mismatches between the distribution of the transfer and labeled sets. We will refer to the two-step algorithm as PD, and to the three-step algorithm as PDF.
Note that our TransformeTBASE and BERT in Devlin et al. (018)) have the same architecture. We use the former term for clarity, since not all students in Section 6 are pre-trained.
3

---

Under review as a conference paper at ICLR 2020
Model
Step (DLM)
Step (D7 D,)
Archilecture-aanootie
PD (our work)
LM pre-training
KD
/
Sun et al. (2019a)
BERTBASE truncated
Patient-KD Fine-tuning
X
Sanh (2019
BERTBASE truncated LM-KD
X
Table 2: Training Strategies that build compact models by applying language model (LM) pre- training before knowledge distillation (KD). The first two rows apply distillation task data. The third row applies distillation with an LM objective on general- domain data.
Model
SST-2 (acc)
MRPC (fl/acc)
QQP (fl/acc)
MNLI (acc m/mm)
QNLI
RTE
Meta
(acc)
(acc)
Score
TF (baseline)
90.7
85.9/80.2
69.2/88.2
80.4/79.7
86.7
63.6
80.5
dev test
PF (baseline)
92.5
86.8/81.8
1/8./
81.8/81.1
87.9
64.2
81.6
PD (our work)
91.8
86.8/81.7
70.4/88.9
82.8/82.2
88.9
65.3
82.1
Sun al. (2019a)
92.0
85.0/79.9
70.7/88.9
81.5/81.0
89.0
65.5
81.7
PF (baseline)
91.1
87.9/82.5
86.6/90.0
81.1/81.7
87.8
63.0
82.8
PD (our work)
91.1
89.4/84.9
87.4/90.7
82.5/83.4
89.4
66.7
84.4
Sanh (2019]
92.7
88.3/82.4
87.7/90.6
81.6/81.1
85.5
60.0
82.3
Table 3: Model Quality. All students are 6/768 BERT models, trained by 12/768 BERT teachers. Concurrent results are cited reported by their authors. Our dev results are averaged over runs. Our test results are evaluated on the GLUE server, using the model that performed best on dev. For anchoring, we also provide our results for ML.M pre-training followed by fine-tuning (PF) and cite results from Sun et al. (019a) for BERT BASE truncated and fine-tuned (TF). The meta score is computed tasks only, and is therefore not directly comparable to the GLUE leaderboard.
Figure 3: Pre-trained Distillation (PD) and concurrent work on model compression.
While we are treating our large teachers as black boxes, it worth noting that they are produced by pre-training and fine-tuning. Since the teacher could potentially transfer the knowledge it has ob- tained via pre-training to the student through distillation, it is priori unclear whether pre-training the student would bring additional benefits. As Section 6.2 shows, pre-training students surpris- ingly important, even when millions of samples are available for transfer.
4 COMPARISON TO CONCURRENT WORK
There concurrent efforts aiming to leverage both pre-training and distillation in the text of building compact models. Though inspired by the two-stage pre-training- finee-uning ap- proach that enabled deep-and-wide architectures advance the state-of-the-art in language under- standing, they depart from this traditional method in several key ways.
Patient Knowledge Distillation (Sun et al. 2019a) initializes a student from the bottom layers of a deeper pre-trained model, then performs task-specific patient distillation. The training objective relies only on the teacher output, but also about the student and teacher architectures. In parallel line of work, DistilBert (Sanh) 2019) applies the same truncation-baac based initialization method for the student, then continues its LM pre-training via distillation from more expensive LM teacher, and finally fine-tunes on task data. Its downside that LM distillation is computationally expensive, as requires a softmax operation over the entire vocabulary to compute the expensive LM teacher's 's predictive distribution. A common limitation in both studies is that the initialization strategy constrains the student to the teacher embedding size. Table 2 summarizes the differences between concurrent work and Pre-trained Distillation (PD).
intermediate layers, thus making assumptions
To facilitate direct comparison, in this section we perform an experiment with the same model architecture, sizes and dataset settings used in the two studies mentioned above. We perform Pre- trained Distillation on a 6-layer BERT student with task supervision from a 12-layer BERT BASE teacher, using embedding size 768 for both models. For distillation, transfer set coincides with
4

---

Under review as a conference paper at ICLR 2020
Unlabeled
Pre-tr Pre-training
transfer data
Labeled data
Basic Training
Dietilation
Labeled data
Fine-turing
Large Teacher
Compact model
Compact model
Compact model
(a) Basic Training
(b) Distillation
(c) Pre-training + Fiinetuniig (PF)
Figure 4: Baselines for building compact models, used for analysis (Section
the labeled set (Dy=D,) Table reports reports results the 6 GLUE tasks selected Sy et al. (2019a)
and shows that, on average, PD performs best. For anchoring, we also provide quality numbers for (PF), which is surprisingly competitive to the more claborate alternatives in this setting where Dr is not larger than Dz. Remarkably, PF does not compromise generality simplicity for quality. Its downside is, however, that cannot leverage unlabeled task data and teacher model predictions.
5 ANALYSIS SETTINGS
Given these positive results, we aim to gain more insight into Pre-trained Distillation. We perform extensive analyses
orthogonal axes-model sizes and properties of unlabeled data, thus de- parting from the settings used in Section 4
All our models follow the Transformer architecture (Vaswani al., 2017) and input processing used in BERT (Devlin et al. 2018). We denote the number of hidden layers as L and the hidden embedding size as H, and refer models by their L/H dimensions. We always fix the number of self-attention heads to H/64 and the fowad/frwrr/filter size to 4H. The end-task models are obtained by stacking linear classifier on top of the Transformer architectures.
The teacher, BERTLARGE has dimensions 24L/1024H and 340M parameters. We experiment with 24 student models, with sizes and relative latencies listed in Table [ The most expensive student, TransformerBASE, is 3 times smaller and 1.25 times faster than the teacher; the cheapest student, TransformersMALL, is 77 times smaller and 65 times faster. For readability, we report results on selection of 5 students, but verify that all conclusions hold across the entire 24-model grid.
5.1 ANALYSIS BASELINES
We select three baselines for Pre-trained Distillation that can provide insights into the contributions made by each of its constituent operations.
Basic Training (Figure 4a is the standard supervised learning method: a compact model is trained directly the labeled set.
Buciiä et al. 2006; Hinton et al. 2015) (or simply "distil- lation") transfers information from a highly-parameterizedd and accurate teacher model to a more compact and thus less expressive student. For classification tasks, distillation exposes the student to soft labels, namely the class probabilities produced by the teacher pt = softmax (21/)) where pt is the output probability for class 1, 21 is the logit for class 1, and T is constant called temperature the output distribution. The softness of the labels enables better
Knowledge Distillation (Figure 4b
that controls the smoothness generalization than the gold hard labels. For each end task, we train: (i) teacher obtained by fine-tuning pre-trained BERTLARGE LARGE (24L/1024H) on the labeled dataset (note teachers do not learn from the transfer set), and (ii) 24 students of various sizes. Students are always distilled the soft labels produced by the teacher with a temperature of
Pre-training + Fine-tuning (Figure 4cb DDai & Le 2015 Devlin al. 2018). or simply PF, lever- ages large unlabeled general-domate corpora to pre-train models that can be fine-tuned for end tasks.
While Hinton et al. 2015) show that tuning the temperature could increase performance, we did not observe notable gains. They also propose using weighted sum of the soft and hard labels, but this approach cannot be applied directly in set-up, since not all instances in our unlabeled transfers set have hard labels. Our optional final fine-tuning step similarly up-weights the hard labels in the labeled set.
5

---

Under review as a conference paper at ICLR 2020
Following BERT, perform pre-training with the masked LM (MLM) and next sentence objectives (collectively referred to as MLM from here on). The resulting model is fine-tuned on end-task la- beled data. While pre-training large models has been shown to provide substantial benefits, unaware of any prior work systematically studying its effectiveness on compact architectures.
5.2 ANALYSIS TASKS AND DATASETS
The tasks and associated datasets are sum- marized in Table
Labeled Data (Dt)
Unlabeled Transfer Data (Dr)
MNLI (390k)
NLP (1.3m samples)
Sentiment classification aims classify text according to the polarities of opin- ions it contains. We perform 3-way doc- Amazon Book Re- views (He & McAuley 2016). Its consid- erable size (8m) allows us closely fol- low the standard distillation setting, where there is large number of unlabeled exam- ples for transfer. Additionally, we test our algorithm on SST-2 (Socher al., 2013).
RTB (2.5k)
NLI (1.3m samples)
SST-2 (68k)
Movie Reviews" (1.7m samples)
Book Reviews (50k)
Book Reviews* (8m samples)
ument classification
Table Datasets used for analysis. star indicates that hard labels are discarded. NLI refers to the collection of MNLI (390k). SNLI (570k) and QQP (400k). The reviews datasets part of Amazon Product Data. Unless otherwise stated, DLM consists of BookCorpus & English Wikipedia.
which is binary sentence classification task, and our results are directly comparable with prior work on the GLUE leaderboard (Wang et al. 2018). We use whole documents from Amazon Movie Reviews (1.7m) as unlabeled transfer data (note that SST-2 consists of single sentences).
Natural language inference involves classifying pairs of sentences premise and hypothesis) as entailment, contradiction, or neutral. This task is representative of the scenario in which proxy data non-trivial to gather (Gururangan al.] 2018). We chose MNLI (Williams et al.] (0188) as our target dataset. Since strictly in-domain data is difficult to obtain, we supplement Dr with two other sentence-pair datasets: SNLI (Bowman et al. 2015) and QQP (Chen et al. 2018).
Textual entailment is similar to NLI, but restricted to binary classification (entailment vs non- entailment). The most popular RTE dataset (Bentivogli et al. 2009) two orders of magnitude smaller than MNLI and offers extreme test of robustness to the amount of transfer data.
6 ANALYSIS
In this section, we conduct experiments that help us understand why Pre-trained Distillation is suc- cessful and how to attribute credit to its constituent operations.
6.1 THERE ARE No SHORTCUTS: WHY FULII PRE-TRAINING IS NECESSARY
As later elaborated Section 7 carlier efforts to leverage pre-training in the context of compact models simply feed pre-trainng tre-trained (possibly contextual) input representations into randomly-initialized students (Hu et af. 2018 Chia et al. 2018 Tang et al. 2019). Concurrent work initializes shallow- and-wide students from the bottom layers of their deeper pre-trained counterparts (Yang et al.) 2019a; Sun et al.) 2019a). The experiments below indicate these strategies are suboptimal, and that LM pre-training is necessary in order to unlock the full student potential.
Is it enough to pre-train word embeddings? No. In order to prove that pre-training Transformer layers is important, compare two flavors of Pre-trained Distillation PD with pre-trained word embeddings and PD with pre-trained word embeddings and Transformer layers. We produce word- piece embeddings by pre-training one-layer Transformers for each embedding size. We then discard the single Transformer layer and keep the embeddings to initialize our students.
For MNLI (Figure 5) less than 24% of the gains PD brings over distillation can be attributed the pre-trained word embeddings (for TransformeryNY, this drops even lower, to 5%). The rest of the benefits come from additionally pre- pre-training the Transformer layers.
Note that, for both flavors of PD, none of the student parameters are frozen; the word embeddings do get updated during distillation.

---

Under review as a conference paper at ICLR 2020
MNLI
SST-2
100
Teacher
PF
x Basic training
80
Distillation
890
70
PD (MLM pre-training)
PD (from 12-layer model)
60
PD (word embeddings)
Distillation
80
Tiny
Mini
Small
Medium
Base
2/10
40128
61128
50138
10
12/138
20200
4226
6026
$2200
192200
122240
20812
6652
17780
21/128H
4L/256H
4L/512H
8L/51HH 12L/768H
Figure Pre-training outperforms truncation. Students initialized LM pre-training (green) out- perform those initialized from the bottom layers of 12-layer pre-trained models (gray). When only word embeddings are pre-trained (red), performance de- graded even further.
Figure 6: Depth outweighs width when models are pre-trained (PD PF), as emphasized by the sharp drops in the plot. For instance, the 61/512H (35.4m parameters) model outperforms the 21/768H model (39.2 39.2m parameters). Randomly initialized models take poor advantage of extra parameters.
Is it worse to truncate deep pre-trained models? Yes, especially for shallow students. Given that pre-training is an expensive process, an exhaustive search over model sizes in the pursuit of the one that meets certain performance threshold can be impractical. Instead of pre-training all (number of layers, embedding size) combinations of students, one way short-cutting the process is to pre-train single deep (e.g. 12-layer) student for each embedding size, then truncate it at various heights. Figure [ shows that this can be detrimental especially to shallow architectures; TransformerıINY loses more than 73% of the pre-training gains distillation. As expected, losses fade away as the number layers increases.
What is the best student for fixed parameter size budget? As rule of thumb, prioritize depth over width, especially with pre-trained students. Figure Opresentss comparison between 24 student model architectures on SST-2, demonstrating how well different students utilize model capacity. They are sorted first by the hidden size, then by the number of layers. This roughly corresponds to monotonic increase in the number of parameters, with a few exceptions for the largest students. The quality of randomly initialized students (i.e. basic training and distillation) is closely correlated with the number of parameters. With pre-training (i.e. PD and PF), we observe two intuitive findings: (1) pre-trained models are much more effective at using more parameters, and (2) pre-trate pre-trained models are particularly efficient at utilizing depth, as indicated by the sharp drops performance when moving wider but shallower models.
This is yet another argument against initialization via truncation: for instance, truncating the bottom two layers of BERTBASE would lead to suboptimal distribution of parameters: the 2L/768H model (39.20) (39.2m 2m parameters) is dramatically worse than e.g. 6L/512H (35.4m parameters).
6.2 UNDER THE HOOD: DISSECTING PRE-TRAINED DISTILLATION
In the previous section, we presented empirical evidence for the importance of the initial LM pre- training step. In this section, we show that distillation brings additional value, especially in the presence of considerably-sizer transfer set, and that fine-tuning ensures robustness when the unla- beled data diverges from the labeled set.
Comparison to analysis baselines upon its constituent operations applied in isolation. We compare it against the baselines established in Section 5.1 (basic training, distillation, and pre-training- fine-tuning) on the three NL. tasks
First, quantify how much Pre- treitrained Distillation improves
7

---

Under review as a conference paper at ICLR 2020
MNLI
RTR
SST-2
Amazon Book Reviews
95
70
86
84
60
62
55
60
Tiny
Mini
Small Medium Base
Tiny
Min
Small
Medium
Base
Tiny
Mini
Small Mediam Buse
78.
Timy
Mini
Small Medium Base
2L
4L
4L
KL
12r.
2L
41
41.
8L
121.
21
41.
4L
121.
21
121.
2561l
51211
51200
12801
25601
51211
12810
12811
Tencher a. Pre-tmined Distillation + Pre treetrrainigg + Fine Fine-tuning - Distillation ... Basic Training
Figure 7: Comparison against analysis baselines. Pre-trained Distillation out-performs all baselines: pre- training+ foiniing-fieetuninng fine-tuning, distillation, and basic training over five different student sizes. Pre-training performed on a large unlabeled LM set (BookCorpus English Wikipedia), Distillation uses the task-specific unlabeled transfer sets listed in Table
Teachers are pre-trained BERT LARGE, fine-tuned on labeled data.
described in Section 5.2
We use the BookCorpus (Zhu al.) 2013) and English Wikipedia as our unlabeled LM set, following the same pre-training procedure as Devlin et al. (2018).
Results in Figure confirm 7 that PD outperforms these baselines, with particularly remarkable results on the Amazon Book Reviews corpus, where TransformerииN recovers the accuracy of the teacher at a 31x decrease in model size and 16x speed-up. Distillation achieves the same performance with TransformeTBASE. which is 10x larger than TranSfOrMeTMINI. Thus PD can compress the model more effectively than distillation. On RTE, Pre-trained Distillation improves TransformeryıNy by more than 5% absolute recover teacher accuracy with TransformerRASE:
the closest baseline ppre-training- finee-tuiingt)) and is the only method to
It is interesting to note that the performance of the baseline systems is closely related to the size of the transfer set. For the sentence-pai pair tasks such as MNLI and RTE, where the size of the transfer set is moderate (1.3m) and slightly out-of-domain (see Table 4 pre-training funnttiinn out-performs distillation across all student sizes, with an average of 12% for MNLI and 8% on RTE. Interestingly, the order is inverted on Amazon Book Reviews, where the large transfer set (8m) is strictly domain: distillation is better than pre-training trrainnng-fie-ttiiinn fine-tuning by an average of 3%. On the other hand, Pre-trained Distillation consistently best in all cases. We will examine the robustness of Pre-trained Distillation in the rest of the section.
Robustness to transfer set size It is generally accepted that distillation is reliant upon a large transfer set. For instance, distillation for speech recognition is performed on hundreds of millions of data points (Li et al. 2014 Hinton et al. 2015).
We reaffirm this statement through experiments on Amazon Book Reviews in Figure g| given that Amazon Book Reviews have the biggest transfer set. Distillation barely recovers teacher accuracy with the largest student (TransformergAsE)), using the entire 8m transfer set. When there is only transfer set, the performance is 4% behind the teacher model. In contrast, PD achieves the same performance with TransformerMINt on 5m instances. In other words, PD can match the teacher model with 10x smaller model and 1.5x less transfer data, compared distillation.
To the best of our knowledge, there is no prior work that explicitly studies how distillation impacted by the mismatch between training and transfer sets (which we will refer to as domain shifl). Many previous distillation efforts focus on tasks where the two sets come from the same distribution (Romero et al., 2014 Hinton al. 2015), while others simply acknowledge the importance of and strive for close match between them (Bucilà et al. 2006).
Robustness to domain shift
We provide empirical evidence that out-of-domain data degrades distillation and that our algorithm is more robust to mismatches between Dr, and Dr. We measure domain shift using the Spearman rank correlation coefficient (which we refer to as Spearman or simply S), introduced as a general metric in (Spearman 1904) and first used as corpus similarity metric in (Johansson et al. 1989). To compute corpus similarity, we follow the procedure described in (Kilgarriff & Rose 1998): for two datasets X and Y, compute the corresponding frequency ranks Fx and Fy of their most
8

---

Under review as a conference paper at ICLR 2020
Amazon Book Reviews
Amazon Book Reviews
88
86
84
82
PD (Mini = 11m)
82
PD-F (Mini = 11m)
Distillation (Mini = 11m)
PD (Mini 11m)
4
Distillation (Base 10m)
80
Pre pre-training 1 tine cuning
80
Diciillation (Mini
Teacher (Large 340m)
Teacher (Lagge 340m)
Basic training (Mini 11m)
78
Basic training (Mini 11m)
78
50k
1m
3m
5m
8m
0.43
0.52
0.76
1.0
Transfer Set Size (Prr)
Spearman Correlation Coefficient hetween D2 and Dr (5)
Figure Robustness transfer set size. We verify that distillation requires large transfer set: 8m instances are needed to match the performance of the teacher using TransformeRAASC. PD achieves 5m
Figure 9: Robustness to domain shift transfer set. By keeping [Dr] fixed (1.7m) and varying the correlation between Dr and DDr (denoted by S), we show that distillation requires in-domain transfer set. PD and PD-F are more robust transfer do- main.
the same performance with TransformetsNNI. transfer set (10x smaller, 13x faster, 1.5x less data).
common 100 words. For each of these words, the difference d between ranks in Fx and Fy is computed. The final statistic is given by the following formula: E dL/(n(n2 1)).
To measure the effect of domain shift, we again experiment on the Amazon Book Reviews task. Instead of varying the size of the transfer sets, this time keep size fixed (to 1.7m documents) and vary the source of the unlabeled text used for distillation. Transfer set domains vary from not task-related (paragraphs from Wikipedia with S=0.43), reviews for products of unrelated category (electronics reviews with S=0,52), followed by reviews from a related category (movie reviews with S=0.76), and finally in-domain book reviews (S=1.0). Results in Figure 9 show a direct correlation between accuracy and the Spearman coefficient for both distillation and PD. When S drops to 0.43, distillation on Dr 1.8% worse than basic training on Dz, whereas PD suffers a smaller loss over pre-training + fine-tuning, and gain of about 1.5% when final fine-tuning step is added. When reviews from an unrelated product are used as a transfer set (S=0.52), PD obtains much larger gain from learning from the teacher, compared to distillation.
6.3 BETTER TOGETHER: THE COMPOUND EFFECT OF PRE-TRAININ AND DISTILLATION
We investigate the interaction between pre- training and distillation by applying them se- quentially on the same data. We compare the following algorithms: Pre-training-t Fine-- tuning with DIM = X and Pre-trained Distil- lation with PLM = Dr X. Any additional gains that the latter brings over the former must be attributed to distillation, providing evidence that the compound effect still exists.
MNLI
80
75
70
65
For MNLI, we set DLM = Dr NLI and con- tinue the experiment above by taking the stu- dents pre-trained on D DLM- = NLI* and distill- ing them on Dr = NLI As shown in Figure 10 PD is better than PF by 2.2% on average over all student sizes. Note that even when pre- training and then distilling on the same data, PD outperforms the two training strategies ap- plied isolation. The two methods are thus learning different linguistic aspects, both useful for the end task.
PD (DEu Dy NLI') PF PF(DLM NNI))) Distillation (D, NLI))
60
Tuny
Mini
Small 4L.51H
Mediam
Base
21//28H
41 42/256H
8LS51H
12L7768
Pre-training complements distillation. PD outperforms the baselines even when we pre-train and distill on the same datasct (DLM Dr NLI*).
Figure 10:
9

---

Under review as a conference paper at ICLR 2020
7 RELATED WORK
Pre-training Decades of research have shown that unlabeled text can help learn language repre-
sentations. Word embeddings were first used (Mikolov al. 2013 Pennington et al.) 2014), while
subsequently contextual word representations were found more effective (Peters et al. 2018). Most
recently, research has shifted towards fine-tuning methods (Radford et al. 2018: Devlin al. 2018
Radford et al. 2019), where entire large pre-trained representations are fine-tuned for end tasks together with small number of task-specific parameters. While feature-based unsupervised rep- resentations have been successfully used in compact models (Johnson & Zhang 2015 Gururangan al. 2019). inter alia, the pretraining+ fine-ninnnn---tiinng approach has not been studied depth for such small models.
Learning compact models In this work we built on model compression (Bucilà al. 2006) and its variant knowledge distillation (Hinton et al. 2015). Other related efforts introduced ways to transfer more information from a teacher to a student model, by sharing intermediate layer activa- tions (Romero et al. 2014 Yim et al. 2017] Sun et al., 2019a). We experimented with related approaches, but found only slight gains which were dominated by the gains from pre-training and not complementary. Prior works have also noted the unavailability of in-domain large-scale transfer data and proposed the use of automatically generated pseudo-examples (Bucilå et al.) 2006) Kimura al., 2018). Here we showed that large-scale general domain text can be successfully used for pre-training instead. A separate line of work uses pruning or quantization to derive smaller models (Han et al., 2016. Gupta al. 2015). Gains from such techniques are expected to be complementary to PD.
Distillation with unsupervised pre-training Early efforts to leverage both unsupervised pre- training and distillation provide pre-trained (possibly contextual) word embeddings as inputs to students, rather than pre-training the student stack. For instance, [Hu et al., (2018) use ELMo em- beddings, while (Chia et al. 2018 Tang et al.) 2019) use context-independen word embeddings. Concurrent work initializes Transformer students from the bottom layers of 12-layer BERT model Yang et al. 2019a] Sun et al. 2019a] Sanh 2019). The latter continues student LM pre- pre-training via distillation from a more expensive LM teacher. For a different purpose deriving a single model for multiple tasks through distillation, Clark al. (209)) use pre-trained student model of the same size as multiple teacher models. However, none of the prior work has analyzed the impact of unsupervised learning for students in relation to the model size and domain of the transfer set.
CONCLUSION
We conducted extensive experiments to gain understanding of how knowledge distillation and the wetttrinnn-iine-uunnne
algorithm work isolation, and how they interact. We made the finding that their benefits compound, and unveiled the power of Pre-trained Distillation, simple yet effec- tive method to maximize the utilization of all available resources: a powerful teacher, and multiple sources of data (labeled sets, unlabeled transfer sets, and unlabeled LM sets).
REFERENCES
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth paseal recognizing textual entailment challenge. In Proc Text Analysis Conference (TAC09, 2009.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2015.
C Bucilă, Caruana, and Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535- 541, 2006.
Z. Chen, H. Zhang, X. Zhang, and I.. Zhao. Quora question pairs. 2018.
10

---

Under review as a conference paper at ICLR 2020
Yew Ken Chia, Sam Witteveen, and Martin Andrews. Transformer to emn: Label-scarce distillation for efficient text classification. 2018.
Kevin Clark, Minh- Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, and Quoe V. Le. BAM! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, July 2019.
Association for Computational Linguistics.
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. In Advances in neural infor- mation processing systems, pp. 3079-3087, 2015.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 2018.
Suyog Gupta, Ankur Agrawal. Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In ICML, 2015,
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In NAACI. 2018
Suchin Gururangan, Tam Dang, Dallas Card, and Noah Smith. Variational pretraining for semi- supervised text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, Florence, Italy, July 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In NIPS, 2016.
Ruining He and Julian McAuley, Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web, pp. 507-517. International World Wide Web Conferences Steering Committee, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025.: 2015.
Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention-guided answer distillation for machine reading comprehension. arXiv preprint arXiv: 07644, 2018
Johansson, Stig, and Knut Hofland. Frequency Analysis of English vocabulary and grammar, based on the LOB corpus. Claredon, Oxford, 1989.
Rie Johnson and Tong Zhang. Semi-supervised convolutional neural networks for text categorization via region embedding. Advances in neural information processing systems, pp. 919 1199-927,44 2015.
Adam Kilgarriff and Tony Rose. Measures for corpus similarity and homogeneity, In Proceedings of the Third Conference on Empirical Methods for Natural Language Processing, pp. 46 46-52, 1998.
Young-Bund Bum Kim, Karl Stratos, and Dongchan Kim. data. In ACL, 2017.
Adversarial adaptation of synthetic or stale
Akisato Kimura, Zoubin Ghahramani, Koh Takeuchi, Tomoharu Iwata, and Naonori Ueda. Few- shot learning of neural networks from scratch by pseudo example optimization. arXiv preprint arXiv:1802.088. 1802.03099 2018
Jinyu Li, Rui Zhao, Jui-Ting Huang, and Yifan Gong. Learning small-size dnn with output- distribution-ban based criteria. In Fifteenth annual conference of the international speech commu- nication association, 2014.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
11

---

Under review as a conference paper at ICLR 2020
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations words and phrases and their compositionality. In Advances neural information pro- cessing systems, pp- 3111-3119, 2013.
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv: 1.Xiv:1802... 8802.0366. 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language un- derstanding by generative pre-training. URL https://s3-us-westt amazonaws. com/openai- hose///eerrr.oorrrnggggggggrrrrr
understanding paper. pdf. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. URL https://openai. com/bblg/hetter-lannggee models, 2019.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Victor Sanh. Smaller, faster, cheaper, lighter: Introducing DistilBERT, distilled version of BERT. 2019.
hutps:/mmelimm..ooouuuiigggggi
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Spearman. The proof and measurement of association between two things. American Journal of Psychology,, pp. 201, 1904.
Siqi Sun, Yu Cheng, The Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint htt::/990.0955555 2019a.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. Ernie 2.0: A continual pre-training framework for language understanding. 2019b.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. 2018.
Adina Williams, Nikita Nangia, and Samuel Bowman. A tence understanding through inference. In Proceedings the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers). pp. 1112-1122. Association for Computational Linguisties, 2018. URL http://aclwer org. antho logyo/oooyyy/lo 1101
challenge corpus for sen-
Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with multi-task knowledge distillation for web-scale question answering system. arXiv preprint arXiv:1904.0963 2019a.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv: (tt::/906.08800 08237, 2019b.
12

---

Under review as a conference paper at ICLR 2020
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
13


Reviews:
[{"review_id": "rkgyLdyntS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission revisits the student-teacher paradigm and shows through extensive experiments that pre-training a student directly on masked language modeling is better than distillation (from scratch). It also shows that the best is to combine both and distill from that pre-trained student model.\n\nMy rating is Weak Accept. I think the submission highlights a very useful observation about knowledge distillation that I imagine is overlooked by many researchers and practitioners. The decision of Weak as opposed to a Strong accept is because the submission does not introduce anything truly novel, but simply points out observations and offers a recommended training strategy. However, I do argue for its acceptance, because it does a thorough job and presents many interesting findings that can benefit the community.\n\nComparison with prior work:\n\nThe submission focuses on comparison with Sun et al. and Sanh. These comparisons are important, but not the most compelling part of the paper. Comparison with more prior work that show large benefits would make the paper even stronger.\n\nInteresting experiments:\n\nThe paper presents many interesting experiments useful for anyone trying to develop a compressed model. First, it shows that distillation (from scratch) by itself may be overrated, since simply repeating the pre-training+fine-tuning procedure on the small model directly is effective. However, distillation remains relevant since it also shows that pre-training the student, then distilling against a teacher, is a potent combination. In the case when the transfer set is the same size as the pre-training set, it surprisingly still has some benefits. This is not experimentally explained, but I suspect there are optimization benefits that are hard to pin down exactly. The paper hypothesizes that the two methods learn different “linguistic aspects,” but I think it is a bit too speculative to put it in such terms.\n\nThe experiments are thorough, with many student sizes, transfer set sizes, transfer set/task set correlation, etc. It also compares against the truncation technique, where the student is initialized with a truncated version of the teacher. There are no error bars in the plots, but there are so many plots with clear trends, that this is not a big concern. I can’t think of any experiments that are obviously missing.\n\nMisc:\n\n- The introduction says that the pre-training+fine-tuning baseline has been overlooked. It would be great to point out papers that has actually overlooked this baseline. Including this in the results would be even better.\n- During my first read-through, I got confused because I didn’t realize “pre-training” in most of the paper refers to “student pre-training” (as opposed to simply training the teacher). Making this a bit more explicit here and there can avoid this confusion.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJe6IsB6KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to pre-train a student before training with a teacher, which is easy to understand. Although the authors provide extensive empirical studies, I do not think they can justify the claims in this paper. \n\n\n** Argument\n\nOne concern is that compared to other baselines such as \"Patient knowledge distillation\" [1], the proposed method is not consistently better. The authors argue that [1] is more sophisticated in that they distill task knowledge from intermediate teacher activations. However, the proposed method introduces other extra complexities, such as pre-training the student. I do not agree that the proposed method is less elaborate than previous methods. \n\n\nAlthough the investigation on influence of model size and the amount/quality of unlabeled data is interesting in itself, this does not help justify the usefulness of pre-training a student. I hypothesize that when considering the intermediate feature maps as additional training signals, randomly initialized students can catch up with pre-trained students. \n\nFurthermore, the mixed results shown in Table 3 do not justify the proposed method well enough. \n\n[1] Patient Knowledge Distillation for BERT Model Compression, https://arxiv.org/abs/1908.09355", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1xlKlknFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors investigate the problem of training compact pre-trained language model via distillation. Their method consists of three steps: \n1. pre-train the compact model LM\n2. distill the compact model LM with a larger model (teacher)\n3. fine-tune the compact model on target task \n\nThis idea is not significantly new since it is quite common to apply distillation to compress models, and the results are largely empirical. From Table 3 the results on test sets are better than previous works, but not by much. The authors spend quite a of space on ablation studies to investigate the contribution of different factors, and on cross-domain transfers. They do manage to show that using a teacher for distilling a compact student model does better than directly pre-training a compact model on the NLI* task in section 6.3. It would be better if they could show it for other tasks on the benchmark as well. \n\nOverall I think this work is somewhat incremental, and falls below the acceptance threshold. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review:  Though the reviewers thought the ideas in this paper were interesting, they questioned the importance and magnitude of the contribution.  Though it is important to share empirical results, the reviewers were not sure that there was enough for this paper to be accepted.
Decision: Reject