Contextual Knowledge Distillation for Transformer Compression
Authors: Geondo Park
Gyeongman Kim
Eunho Yang
Venue:  (2021)
Keywords: Knowledge Distillation
Transformer Compression
BERT

Abstract: A computationally expensive and memory intensive neural network lies behind the recent success of language representation learning. Knowledge distillation, a major technique for deploying such a vast language model in resource-scarce environments, transfers the knowledge on individual word representations learned without restrictions. In this paper, inspired by the recent observations that language representations are relatively positioned and have more semantic knowledge as a whole, we present a new knowledge distillation strategy for language representation learning that transfers the contextual knowledge via two types of relationships across representations: Word Relation and Layer Transforming Relation. We validate the effectiveness of our method on challenging benchmarks of language understanding tasks. The code will be released.

Under review as a conference paper at ICLR 2021
CONTEXTUAL KNOWLEDGE TRANSFORMER COMPRESSION
DISTILLATION
FOR
Anonymous authors Paper under double-blind review
ABSTRACT
A computationally expensive and memory intensive neural network lies behind the recent success of language representation learning. Knowledge distillation, a major technique for deploying such vast language model in resource-scarce en- vironments, transfers the knowledge on individual word representations learned without restrictions. In this paper, inspired by the recent observations that lan- guage representations are relatively positioned and have more semantic knowl- edge as a whole, we present a new knowledge distillation strategy for language representation learning that transfers the contextual knowledge via two types of relationships across representations: Word Relation and Layer Transforming Re- lation. We validate the effectiveness of our method on challenging benchmarks of language understanding tasks. The code will be released.
INTRODUCTION
Since the Transformer, simple architecture based attention mechanism, succeeded in machine translation tasks, Transformer-based models have become a new state of the arts that takes over more complex structures based on recurrent or convolution networks on various language tasks, e.g., language understanding and question answering, etc. (Devlin et al., 2018; Lan et al., 9019; Liu et al., 2019a; Raffel al., 2019; Yang et al., 2019) However, exchange for high performance, these models suffer from a major drawback: tremendous computational and memory costs. In particular, is not possible deploy such large models on platforms with limited resources such as mobile and wearable devices, and it is urgent research topic with impact to keep up with the performance of the latest models from a small-size network.
As the main method for this purpose, Knowledge Distillation (KD) transfers knowledge from the large and well-performing network (teacher) to a smaller network (student). Very recently, there have been some efforts that distill Transformer-basedd models into compact networks (Sanh et al., 2019; Ture et al., 2019; Sun et al., 2019; 2020; Jiao et al., 2019). However, they all build on the idea that each word representation is independent, ignoring relationships between words that could be more informative than individual representations.
In this paper, we pay attention to the fact that word representations from language models are very structured and capture certain types of semantic and syntactic relationships. Word2Vec (Mikolov al., 2013) and Glove (Pennington et al., 2014) demonstrate that trained embedding of words contains the linguistic patterns as linear relationships between word vectors. Recently, Reif et al. (2019) found out that the distance between words contains the information of the dependency parse tree. Many other studies also suggested the evidence that contextual word representations (Belinkov al., 2017: Tenney al.. 2019a;b) and attention matrices (Vig, 2019; Clark al., 2019) contain important relations between words. Intuitively, although each word representation has respective knowledge, the set of representations of words as a whole more semantically meaningful, since words in the embedding space are positioned relatively by learning.
Inspired by these observations, we propose a novel distillation objective, termed Contextual Knowl- edge Distillation (CKD), for language tasks that utilizes the statistics of relationships between word representations. In this paper, we define two types of contextual knowledge: Word Relation (WR) and Layer Transforming Relation (LTR). Specifically, WR is proposed capture the knowledge of relationships between word representations and LTR defines how each word representation changes as it passes through the network layers. Moreover, unlike some previous approaches with constraints

---

Under review as a conference paper at ICLR 2021
for distillation, the proposed objective is more robust to architecture changes as it does not add any structural constraints for teacher or student.
There are two distillation techniques to compress a large pre-trained language model into a compact network. Several previous works (Sanh et al., 2019; Jiao et al., 2019; Sun et al., 2020) compress large pre-trained language model into small language model the pre-training stage which requires high computation costs and times. On the other hand, some works (Ture et al., 2019; Sun al., 2019) present the task-specific distillation that transfers the knowledge to a well initialized small network to improve the performance of each task. In this paper, we focus on the task-specific distillation that has the advantage of being directly applied on top of pre-trained small BERT mod- els (Ture et al., 2019) without conducting time-consuming pre- pre-tranning process.
We validate our method on the Stanford Question Answer Dataset (SQuAD) and General Language Understanding Evaluation (GLUE) benchmark. We first demonstrate the effectiveness of our method outperforming the current state-of-the- distillation methods. We also show that our CKD performs effectively on variety of network architectures including recently proposed MobileBERT (Sun et al., 2020), a new type thin architecture of BERT, trained with task-agnostic distillation.
Our contribution is threefold:
Inspired by the recent observations that word representations from neural networks structured, we propose a novel knowledge distillation strategy. Contextual Knowledge Dis- tillation (CKD), that transfers the relationships across word representations.
We present two types of complementary contextual knowledge: horizontal Word Relation across representations in a single layer and vertical Layer Transforming Relation across representations for single word.
We validate CKD on the standard language understanding benchmark datasets and show that CKD consistently outperforms the state-of-ther -hettheart distillation methods for BERT on various model sizes.
RELATED WORK
Knowledge distillation Since recently popular deep neural networks are computation- and memory-heavy by design, there has been long line of research on transferring knowledge for the purpose of compression. Hinton et al. (2015) first proposed teacher-student framework with an objective that minimizes the KL divergence between teacher and student class probabilities. In this framework, several follow-up works proposed various objectives to distill the well-designed knowl- edge such as attention map of image (Zagoruyko & Komodakis, 2016), similarity (Tung & Mori, 2019) or the relation (Park et al., 2019; Liu et al., 2019b) between the image features.
In the field of natural language processing (NLP), knowledge distillation has been actively studied (Kim & Rush, 2016; Hu et al., 2018; Yang et al., 2020) In particular, after the emergence of large language models based on pre-training such as BERT (Devlin et al., 2018; Liu et al., 2019a; Yang al., 2019; Raffel et al., 2019), many studies have recently emerged that attempt various knowledge distillation in the pre-training process and/or fine-tuning for downstream tasks in order to reduce the burden of handling large models. Specifically, Tang et al. (2019); Chia al. (2019) proposed to distill the BERT to train the simple recurrent and convolution networks. Sanh et al. (2019); Ture et al. (2019) proposed to use the teacher's 's predictive distribution to train the smaller BERT, and Wang et al. (2020) propose the structure-level distillation that transfer the predictive distribution of sequence-level for the multi-lingual sequence labeling tasks. Sun et al. (2019) proposed a method to transfer individual representation of words in the BERT. In addition to matching the hidden state, Jiao et al. (2019) and Sun et al. (2020) also utilize the attention matrices derived from the Trans- former. Several works (Goyal et al., 2020: Liu et al., 2020; Hou et al. 2020) improve the performance of other compression methods such as sparsification and quantization by integrating the knowledge distillation objectives. Different from previous knowledge distillation methods that transfer respec- tive knowledge of word representations, we design the objective to distill the contextual knowledge of them, which can be combined with existing distillation methods.
Contextual knowledge of word representations Understanding and utilizing the relationships across words is one of the key ingredients in language modeling. Word embedding (Mikolov al.,
2

---

Under review as a conference paper at ICLR 2021
2013: Pennington et al., 2014) that captures the context of a word in document, has been tradition- ally used. Unlike the traditional methods of giving fixed embedding for each word, the contextual embedding methods (Devlin et al., 2018; Peters et al., 2018) that assign different embeddings cording to the context with surrounding words have become a new standard in recent years showing high performance. Xia & Zong (2010) improved the performance of the sentiment classification task by using word relation, and Hewitt & Manning (2019); Reif et al. (2019) found that the distance be- tween contextual representations contains syntactic information of sentences. Our research focuses on knowledge distillation using context information between words and between layers, and to our best knowledge, we are the first to apply this context information to knowledge distillation.
SETUP AND BACKGROUND
Most of the recent state-of-the-aa language models stacking Transformer layers which consist of repeated Multi-Head Attentions and Position-wise Feed-Forward Networks.
Transformer based networks Given an input sentence with n tokens, X= [21 .2,...,,n R4.x", most networks (Devlin et al., 2018; Lan et al., 2019; Liu et al., 2019a) utilize the em- bedding layer to map an input sequence of symbol representations X to a sequence of continu- ous representations E (e....., en] E 2 Then, each 1-th Transformer layer of the iden- tical structure takes the previous representations R-1 and produces the updated representations
R₁ = [rt,1 Ti......, Fi,m] E Rd-X= through two sub-layers: Multi-head Attention (MHA) and point-wise Feed Forward Network (FFN). The input at the first layer (I = 1) is simply E. In MHA operation where h separate attention heads are operating independently, each input token 71-1, for each head is projected into a query q: E R",, key k; E R and value 2i E R typically dj dg d, d,/ h.. Here, the key vectors and value vectors packed into the matrix forms K K= [k, ... kn. and V = [,1,..,, vn). respectively, and the attention value a; and output of each head Oh,; are calculated as followed:
KT- %i vdg
a Softmax
and
Oh,1
aj.
The outputs of all heads are then concatenated and fed through the FFN, producing the single word representation Ti.i. For clarity, we pack attention values of all words into a matrix form AL, (a1, 02, G.) E RRax for attention head h.
Knowledge distillation for Transformer In the general framework of knowledge distillation, teacher network (T) with large capacity is trained in advance, and then student network (S) with pre- defined architecture but relatively smaller than teacher network is trained with the help of teacher's s knowledge. Specifically, given the teacher parameterized by 9z. training the student parameterized by 8, aims minimize two objectives: i) the cross-entropy loss CCE between the output of the stu- dent network S and the true label y and ii) the difference of some statistics LD between predictions by teacher and student models. Overall, our goal is to minimize the following objective function:
C(O.) E(xx))
COB (OE((())))))) XC (CC(K'' (K"
where À controls the relative importance between two objectives. Here, K characterizes the knowl- edge being transferred and can vary depending on the distillation methods, and L2 is a matching loss function such as 41, le Huber loss. Recent studies on knowledge distillation for Transformer- based BERT can also be understood in this general framework. In particular, DistilBERT (Sanh et al., 2019) matches class predictive probabilities of S and T via Kullback-Leible divergence: C. = KL Pt T Pa). Patient KD (Sun et al., 2019) matches the individual word representations between S and in diverse intermediate layers. Recently, in addition to matching word representa- tions, TinyBERT (Jiao al., 2019) and MobileBERT (Sun et al.. 2020) utilize the attention matrix A to transfer the knowledge from T to S. More details of these methods are summarized in Ap- pendix A.
3

---

Under review as a conference paper at ICLR 2021
Twacher Word Regreserbedors
Studeet Wart Regrsseriators
KL Divergence
Pt (Logit]
P.
CLASSIFIER
sper?
CLASSIFIER
Layer
framingrming
Relation
Word Relation
Layorz
Nal-nse ro le)
Trglo rouloon
Fairw adon
wake mabten (A
Spper
Lapor2
Layer
CKD-LTR
Erota
Erola
Ertaa
Ema
Layer1
Laper
Ent
Ena
Palratic nttter 9 Trpll wise manie
Palk vis sladon at Trplo was rolitin
Apsckii
Teacher
Student
We tofally agree input X
(a)
(b)
Figure I: Overview of our contextual knowledge distillation. (a) In the teacher-student framework, we define the two contextual knowledge, word relation and layer transforming relation which are the statistics of relation across the words from the layer (orange) and across the layers for the same word (turquoise), respectively. (b) Given the pair-wise and triple-wise wise relationships WR and LTR from teacher student, we define the objective as matching loss between them.
4 CONTEXTUAL KNOWLEDGE DISTILLATION
We now present distillation method that transfers the structural contextual knowledge of word representations from teacher to student. Unlike previous methods distilling each word separately, our method transfers structural and contextual information contained in relationships between words or between layers, and provides a more flexible way of constructing embedding space than directly matching representations. The overall structure of method is illustrated in Figure 1(a). Specifi- cally, we design two key concepts of contextual knowledge from language models: Word Relation- based and Layer Transforming Relation-based contextual knowledge, as shown in Figure 1(b).
4.1 WORD RELATION (WR)-BASED CONTEXTUAL KNOWLEDGE DISTILLATION
Inspired by several previous studies suggesting that neural networks can successfully capture con- textual relationships across words (Reif al., 2019; Mikolov al., 2013; Pennington al., 2014), WR-based CKD aims to distill the contextual knowledge contained in the relationships across words certain layer. The "relationship" across set of words be defined in a variety of different ways. Considering the computation burden, our work only focuses on defining it as the sum of pair-wise and triple-wise relationships. Specifically, for each input X with n words, let RS [ri.1. .. ri. r2,al be the word representations at layer from the language model (it could be teacher or student), as described in Section 3. Then, the objective of WR-based contextual KD minimize the following loss:
LOKD-WR- =
Σ
Wiy C CD (011)
(1)))))
+AwR
Wijk C CD 20.
(3)
(1...))
where @ and are the functions that define the pair-wise and triple-wise relationships, respectively and AwR adjust the scales of two losses. Here, we suppress the layer index 1 for clarity, but the distillation loss for the entire network simply summed for all layers. Since not all terms Eq. (3) are equally important in defining contextual knowledge, we introduce the hyperparameters Wij and Wifk to control the weight of how important each pair-wise and triple-wise term is. Determining the values of these hyperparameters is open
implementation issue, but it can be determined by the locality words (i.e. Wi = 1 if i jl < and 0, otherwise), or by attention information to focus only relationship between related words.
4

---

Under review as a conference paper at ICLR 2021
While functions & and defining pair-wise wise and triple-wise relationship also have various possibil- ities, the simplest choices are to use the distance between two words for pair-wise & and the angle by three words for triple-wise v, respectively.
Pair-wise & via distance Given a pair of word representations (ra, from the same layer, Ty) is defined as follows: ((ra, Ty) ||r Ty
Triple-wise vb via angle Triple-wise relation captures higher-order structure and provides more flexibility in constructing contextual knowledge. However, expense of using higher-order in- formation, additional constraints such as computation or memory costs should be considered. One of the simplest forms for dà is the angle, which is caleulated
Ti r
Tk
(ri, Jj, rk) = cos ((r,, Ty. ra)
(4)
ri rg
P2 rs
where (5-) denotes the dot product between vectors.
Despite its simple form, efficiently computing the angles in Eq. (4) for all possible triples out of words requires storing all relative representations (r; - rry in (n,n, tensor!. This incurs an additional memory cost of (n²dd,). ). In this case, using locality for wask in Eq. (3) mentioned above can be helpful; by considering only the triples within a distance of 5 from ry. the additional memory space required for efficient computation is (88dd)), which is beneficial for 5 < In the experimental section, we show that measuring angles in local window does not hurt the performance some extent.
4.2 LAYER TRANSFORMING RELATION (LTR)-BASED CONTEXTUAL KNOWLEDGE DISTILLATION
The second structural knowledge that we propose capture is "how each word is transformed as it passes through the layers". Transformer-based language models are composed of stack of identical layers and thus generate a set of representations for each word, one for each layer, with more abstract concept in the higher hierarchy. Hence, LTR-based contextual KD aims to distill the knowledge of how each word develops into more abstract concept within the hierarchy. Toward this, given a set of representations for a single word in L layers, [rf,w: rL.w) for student and .ri.wl "*. ............ layers or student of teacher. It is the index set of layers for which we want to distill the knowledge; this time, will suppress the word index below), the objective of L.TR-based contextual KD is to minimize the following loss:
for teacher (Here abuse the notation and (1..... L} is not necessarily the entire
LOKD-ITR =
Σ
Wim L (Dr0 (m)
(I...e
+ÅLTR
Σ
Wimo CD (v(ri r2), vrt rn mmnr)))
(S)
(Lm.,)E/11.....1
where ÄLTR again adjust the scales of two losses. Here, the composition of Eq. (5) is the same as Eq. (3), but only the objects for which the relationships are captured have been changed from word representations in one layer to representations for single word in layers. That is, the relationships among representations for word in different layers can be defined from distance or angle as in Eq. (4): (rri, rm) [|r: Fmlll2 and (f11), Tm, (1p=-rml IIro-mmll?
ri-Fm
Alignment strategy When the numbers of layers of teacher and student are different, it is im- portant to determine which layer of the student learns information from which layer the teacher. Previous works (Sun et al., 2019; Jiao et al., 2019) resolved this alignment issue via the uniform (i.e. skip) strategy and demonstrated its effectiveness experimentally. For Ln-layered teacher and Lyyleeeeed student, the layer matching function is defined as
(stepp x 1) = stept x L for I. = 0,...,g where g is the greatest common divisor of L. and L., step, L,/ and step, La/g-
From the equation ||r, ryl1 ||r:11 + ||r>11 | 2(r,,rz), computing the pair-wise distance with the right hand side of equation requires additional memory cost.
5

---

Under review as a conference paper at ICLR 2021
Table Comparisons against state-of-the of-the-au distillation methods. For comparison, students are 6/768 BERT models, distilled by BERTEASS BASE (12/768) teachers. Other results are as reported by their authors. Results of development set are averaged over 5 runs and the best one of them are used for test server evaluation. "." means the result is not reported.
CoLA
MNLI-(m/-mm)
SST-2
QNLI (Ace)
MRPC
QQP
RTE
STS-B
Model
Score
(Mec)
(Acc)
(Acc)
(F1)
(Acc)
(Ace)
(Spear)
BERTBASE (Teacher)
60.4
84.8784.6
94.0
91.8
90.3
91.4
70.4
89.5
84.1
DistilBERT
42.5
81.6/8111
92.7
85.5
88.3
90.6
60.0
85.0
78.6
PD
82.5/83.4
89.4
89.4
90.7
66.7
TinyBERT (w/ DA)
54.0
84.5/84.5
93.0
91.1
90.6
91.1
73.4
89.6
83.5
CKD
52.8
83,9/84,4
93.3
90.5
89.6
90.9
67.3
89.0
82,4
CKD (w/ DA)
57,9
84.8/85.0
93.8
91.7
90.8
91.6
70.1
89.6
83,9
BERTBASE (Teacher)
53.8
84.7/83.8
93.8
90.9
88.4
89.2
67.6
85.2
81,9
PKD
81.5/8T00
92.0
89.0
85.0
88.9
65.5
PD
82.8/82.2
91.8
88.9
86.8
88.9
65.3
TinyBERT (w/ DA)
51.1
84.6/83.2
93.1
90.4
87.3
89.1
70.0
83.7
81.4
CKD
50.5
84.1782.8
93.1
90.0
87.3
89.1
65.1
82,4
80.5
CKD (w/ DA)
52.8
84.6/84.0
93.5
90.7
88.0
89.7
66.2
83.8
81.5
The distillation objective aims to supervise the student network with the teacher's knowledge. Multiple distillation loss functions can be used during training, either alone or together. The proposed CKD) can also be combined simply with the vanilla distillation loss such class prob- ability matching (Hinton et al., 2015) as an additional term. In that case, our distillation objective as follows:
Final objective
L C= - a)) LCF + Chige ACKD (LCDD-IT + LCKD-WR
(6)
where a and ACKD is tunable parameter balance the loss terms.
EXPERIMENTS
We validate CKD using the Stanford Question Answer Dataset (SQuAD) and General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018) which consists of 9 natural lan- guage understanding tasks such as Natural Language Inference, Sentiment Classification, and Para- phrase Similarity Matching, Following the Devlin et al. (2018), ,), all datasets except the WNLI dataset are used for experiments. We first compare our method with the state-of-the-art distillation objec- tives in training 6-layer BERT student network. We then report on the performance gains achieved by our method for BERT architectures of various sizes, including the recent MobileBERT (Sun al., 2020). Finally, we analyze the effect of each component of our CKD and the impact of leveraging locality for Wiyk in Eq. (3).
Setup In all experiments, to avoid time-consuming BERT pre-training process, apply CKD on top pre-trained BERT models various sizes, released by Ture et al. (2019). Also, in order to reduce the hyperparameter search cost of method, we do not allow full degrees of freedom for ÄWR and ÄLTR, and search only within the range where they have the same value. Hence, only have two hyperparameters tune (AwR and ÅCKD) in CKD. For the importance weight of each pair-wise and triple-wise terms, we leverage the locality of words, in that Wty jl < 8 and otherwise. For this, we select the 8 in (10-15). More details are provided in Appendix D
5.1 COMPARISONS AGAINST STATE-OF-THE-AN DISTILLATION METHODS
To verify the effectiveness of our CKD objective, we compare the performance with the current state-of-the-art distillation methods for BERT compression. We have four baselines objectives: Dis- tilBERT (Sanh et al., 2019), PKD (Sun et al., 2019), PD (Ture et al., 2019) and TinyBERT (Jiao et al., 2019). Following the standard setup in baselines, 6-layer BERT (6/768) as the student network. Therefore, the student models used in all baselines and ours have the same number of parameters (67.5 M and FLOPs (10878M).
the BERT BASE (12/768)² as the teacher and
In notation (a/b), means the number of layers and b denotes hidden size intermediate layers.

---

Under review as a conference paper at ICLR 2021
Table 2: Comparisons against ofte-of-the-- art distillation methods on SQuAD 1.1v dataset (EM/FL on dev set). For fair comparison. all students are 6/768 BERT models, distilled by BERT BASE (12/768) teachers. The results of PKD and TinyBERT are as reported by Jiao et al. (2019) and the result of DistilBERT is as reported by the author (Sanh et al., 2019).
SQuAD
1.1v
Model
#Params
#FLOPs (Speed up)
EM
F1
BERT BASE (Teacher)
110M
21754M (1.00x)
81.3
88.6
PKD
67.5M
10878M (2.00x)
77.1
85.3
DisitBEET
67.5M
10878M (2.00x)
79.1
86.9
TinyBERT
67.5M
10878M (2.00 2.000)
79.7
87.5
CKD
67.5M
10878M (2.00x)
81.8
88.7
Table 3: Experimental results with the recently proposed MobileBERT. Following the author, base- line is trained without distillation for downstream tasks, Results are averaged over runs on the development set.
Model
CoLA
MNLL-(m'-mm)
SST-2
QNLI
MRPC
QQP
RTE
STS-B
Score
(Mcc)
(Ace)
(Acc)
(Acc)
(FI)
(Acc)
(Acc)
(Spear)
MobileBERT (Sun et al., 2020)
$4.0
83.4/83.8
92.1
91.2
90.8
90.5
64.7
88.1
82.1
MobileBERT (w/ CKD)
54.8
84.1/84.3
92.3
91.4
90.6
90.6
67.1
88.2
82.6
Table summarizes results both for the development set and test of GLUE datasets and il- lustrates how much better our CKD performs than the recent distillation methods. We present two cases depending on whether data augmentation (following Jiao et al. (2019)) is used for training or not. Without the data augmentation, CKD surpasses the DistilBERT, PD, and PKD by a large margin for all datasets in GLUE For the case with data augmentation, CKD outdoes TinyBERT in all scores except for the RTE development and test set, and in terms of the average score of all development datasets, CKD has a 0.4 %p improvement over TinyBERT that transfers both individ- ual word representations and attention matrices. Moreover, for some datasets including MNLI and QQP, student network trained with CKD even surpass the performance of teacher with the help of data augmentation. This supports the claim that CKD is more flexible way of constructing embed- ding space than directly matching representations. In Table 2, CKD exhibits the best performance on the SQuAD 1.1v dataset with the same student architecture. Our CKD has 2.1 1%p improvement over BERY which is the current state-of- tfetherr- knowledge distillation method.
5.2 BOOSTING THE STATE-OF THE- THIN BERT ARCHITECTURE
Recently, MobileBERT (Sun et al., 2020) proposed as a new type of thin architecture BERT, trained with task-agnostic distillation. MobileBERT also employs the distillation policy and transfers individual word representations and attention matrices to further improve its performance. However, it applies the distillation loss only to the pre-training process and modifies the original architectures of teacher and student BERT models, thus making direct comparisons difficult. Instead, given the same pre-trained MoRileBERTS our CKD compared to the method originally in Sun et al. (2020). For this experiment, following Sun al. (2020), we search the additional hyperparameters in search space including different batch sizes (16/32/48). learning rates ((1 10) x e e=5) and the number of epochs (2-10). The results are summarized in Table 3. Fine-tuning with CKD distillation boosts the MobileBERT's original performance on all datasets except MRPC, which is relatively small dataset.
evaluate the performance of fine-tuning on GLUE datasets using
5.3 EFFECT OF MODEL SIZE FOR CKD
For the knowledge distillation with the purpose of network compression, it is essential to work well in more resource-scarce environments. To this end, we further evaluate our method in two settings.
First, we select small BERTs of a certain size and compare naive fine-tuning and our CKD with them on the GLUE dataset. In this experiment, we use the BERT as the teacher and consider the following student models: BERTMNN (6/256), BERTsHALLOW (12/256 ann BERT SMALL (4/512).
We use the pre-trained MohileBERT released in the official repository

---

Under review as a conference paper at ICLR 2021
Table 4: Experiment results on the effect of model sizes for CKD and TinyBERT results are re- ported in their papers. The results are evaluated on the test set of GLUE official benchmark. Detailed component of model sizes are provided in the appendix C. i marks our runs with the official code.
Model
Objective
AParams
SSpee up)
#FLOPs
CoLA
(Mec)
MNLI (-m/-mmm
(Acc)
SST-2
(Ace)
QNLI
(Asc)
MRPC
(FL)
(Aoc)
oge
(Acc)
RTE
(Spear)
STS-B
Scoce
BERT (Reported)
BERTneE (Teacher)
110.IM
21754M
(1.0x)
$4,083.88
84.683.4
93.3
90.5
88.9
66.4
67.6
85.8
85.2
BERT
Naive CKD
29.1
(6.54x)
3321M
38.0
48.3
82 48
75.1774.8
92.9
89.3
88.3
81.5
87.0
85.2
81.7
89.1
69.6
64.2
82.6
77.3
79.7
74.8
Nave
CKD
17.6M
2419
9.9%))
39.3
82.68199
76.3776.7
88.3
84.2
85.4
83.4
63.3
77.5
75.2
ThyBERT (Reportod)
14.5M
(18666)
1167M
43.3
82.5881 8
92.6
87.7
86.4
87.71
62.9
79.9
78.3
BERT
Naive
CKD
12.5M
(17.98x)
1210M
29.7
81,7880.8
74,77712
86.7
83.0
82.7
83.6
88.3
62.7
13.3
78.8
72.3
We also present the number of parameters and floating-poir operations (FLOPs) to measure the computational complexity of student models regardless of the operating environments. Speed up in Table 4 also calculated based on the FLOPs. To observe just how much distillation alone improves performance, we use data augmentation both for the baseline and our method.
The results are summarized in Table 4. The results illustrate that our CKD consistently ex- hibits significant improvements in the performance compared to naive fine-tuning on all datasets. For all downsized student models, the average score also improved by more than 4.9% point. Moreover, comparing BERT MINI and TinyBERT, we observe that the BERT trained with the CKD achieves the higher average score while it has few model parameters with comparable but slightly higher FLOPs.
Accuracy (%)
90
88
Second, we perform extensive analyses of the effect of model sizes. We experiment on the SST-2 dataset with var- ious model sizes (2~10 layers and 128~~68 intermediate dimensions) and compare the CKD and the two baselines: Naive fine-tuning and class probability matching (Logit KD). As above, we use the BERT BASE as the teacher and the BERT the student. The results are averaged over 5 runs on the development set. As illustrated Figure 2, our CKD consistently performs better than the two baselines in most model sizes. In addition, we observe that as the layer deepens or the embedding size increases, the gap in the per- formance between CKD and baselines increases.
CKD
Logit
Naive
20229
41220
6128
10/29
2/26
0025
10/366
2522
4522
2/888
Depth/Width
Figure 2: In L/H. denotes the number of hidden layers and does the embed- ding size. Listing of x-axis is width-first.
5.4 ABLATION STUDIES
We provide additional ablation studies to analyze the impact of cach component of the CKD: Word relation, Layer transforming relation, Pair-wise and Triple-wise relationship and the introduced lo- cality (wij = 5) in Eq. (3) as the weight of how important each pair-wise and triple-wise term is. For these studies, fix the student network with BERT SMALL and report the results an average over 5 runs on the development set.
Impact of each component of CKD The proposed CKD transfers the word relation based and layer transforming relation based contextual knowledge. To isolate the impact of them, we exper- iment successively removing each piece of our objective. Table summarizes the results, and we observe that WR and LTR brings considerable performance gain, verifying their individual effec- tiveness. As shown in Table 5, both pair-wise and angle-wise relationship help improve performance, respectively, and the best performance is achieved when all components are applied together.
Locality as the importance of relation terms We introduced the additional weights (Wij. Wijk) in Eq. (3) for CKD-WR (and similar ones for CKD-LTR) to control the importance of each pair- wise and triple-wise wise term and suggested to the locality for them one possible way. Here, we verify the effect of locality by increasing the local window size (6) on the SST-2 and QNLI datasets. In par- ticular, the triple-wise relation in Eq. (3) requires large amount of additional memory as discussed
8

---

Under review as a conference paper at ICLR 2021
Table Ablation study using subset of GLUE about the impact of each component CKD.
07.4
90.4
Accuracy (%)
90.2
CKD WR Pair-wise Triple-wise Pair-wise Triple-wise
CKD LTR
SST-2
MRPC QNLI
(Ace)
(F1)
(Acc)
89.3
86.2
85.9
SST-2 -
90.1
86,9
87.2
V
89.6
88.4
86.8
88.5
QNLI
90.6
88.1
87.1
20
40
90
100
Locality (26)
90.2
88.3
86.6
Figure 3: Ablation study about fect of locality.
v
90.6
89.2
87.4
in Section 4.1, using this locality for their weights is practical solution. Hence, we investigate how reducing memory usage using this locality for them affects overall performance. The result is il- lustrated in Figure 3. We observe that the local window size increases, the performance improves, but after some point, the performance almost saturated. This result exhibits that computing angles in local window does not hurt the performance some extent.
CONCLUSION
We proposed a novel distillation strategy, contextual knowledge distillation (CKD), that leverages contextual information efficiently based word relation and layer transforming relation. On the standard GLUE benchmark, it performs better than existing state-of-the-ar methods and even sur- passes the teacher's performance showed that our method boosts the performance of MobileBERT, and that performs consistently well for student models of various sizes. Through the ablation studies, it is confirmed that all com- ponents of the proposed CKD method are helpful in performance. As future work, we plan to define the weight between each relation (i.e., w) continuously using attention matrices, not identically or discretely using locality.
several datasets with the help of data augmentation. We also
REFERENCES
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad. and James Glass. What do neural machine translation models learn about morphology? arXiv preprint 174.03170, 2017.
Yew Ken Chia, Sam Witteveen, and Martin Andrews. Transformer to cnn: Label-scarce distillation for efficient text classification. arXiv preprint arXiv:/909.03500 2019.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher Manning. What does bert look at? analysis of bert's attention. arXiv preprint arXiv: 17X4441906.04341, 2019.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.0448 2018.
Saurabh Goyal, Anamitra Roy Choudhary, Venkatesan Chakaravarthy, Saurabh ManishRaje, Yogish Sabharwal, and Ashish Verma. Power-bert: Accelerating bert inference for classification tasks. arXiv preprint arXiv:2001.000 08950, 2020.
John Hewitt and Christopher Manning. structural probe for finding syntax word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pa- pers), pp. 4129-4138, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint
2015.
Lu Hou, Zhiqi Huang, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33, 2020.

---

Under review as a conference paper at ICLR 2021
Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention-guma guideed answer distillation for machine reading comprehension. arXiv preprint TrXxv:1808.07644 2018.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.1083 2019.
Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. arXiv preprini 1oX6vv6666607943 2016.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Alhert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:/909) 1942, 2019.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. Fasthert: a self-distilling bert with adaptive inference time. arXiv preprint arXiv:2004002178, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining 2019a.
approach. arXiv preprint
Yufan Liu, Jiajiong Cao, Bing Li. Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7096-7104, 2019b.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119,, 2013.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3967-3976, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, arXiv preprint arXiv: 1802:38802.05365, 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 19X.v:1900.1666 2019.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, pp. 1594,8603, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert. distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:19/0001108, 2019.
Siqi Sun, Yu Cheng. Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.0933: 2019.
Zhiqing Sun, Hongkun Yu. Xiaodan Song. Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact task aask-agnottic bert for resource-limited devices. arXiv preprint arXiv:2004.02900 2020.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint arXiv: 1903.1216,, 2019.
10

---

Under review as a conference paper at ICLR 2021
Ian Tenney, Dipanjan Das, and Ellie Pavlick. Bert rediscovers the classical nlp pipeline. arXiv preprint arXiv: 2019a.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R Bowman, Dipanjan Das, al. What do you learn from context? probing for sentence structure in contextualized word representations. arXiv preprint 1.Xv:19905.06316.. 2019b. 2019b.
Frederick Tung and Greg Mori. IEEE International Conference Computer Vision, pp. 1365-1374, 2019.
knowledge distillation. In Proceedings of the
Tulia Ture, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better:
On the importance of pre-training compact models. arXiv preprint /98hv:990..08922 2019.
Jesse Vig. Visualizing attention in transformer-basee language representation models. arXiv preprint
arXiv:1904, 02679, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 080..8044001111 2018.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Fei Huang, and Kewei Tu. Structure-level knowledge distillation for multilingual sequence labeling. arXiv preprint arXi::2004.03846, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State- of-the-art natural language processing. ArXiv, 190.071, 2019.
Rui Xia and Chengqing Zong. Exploring the use of word relation features for sentiment classifica- tion. In Coling 2010: Posters, pp. 1336-1344, 1344, 2010.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp. 5753-5763, 2019.
Ziqing Yang. Yiming Cui, Zhipeng Chen, Wanxiang Che, Ting Liu, Shijin Wang, and Guoping Hu. Textbrewer: An open-source knowledge distillation toolkit for natural language processing. arXiv preprint arXiv:2002. 12620, 2020.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 0612.0928, 2016.

---

Under review as a conference paper at ICLR 2021
Table 6: Overview of distillation objectives used for language model compression and their con- straint on architecture. Although TinyBERT acquires flexibility on the embedding size, using an additional parameter, for attention matrices matching, the number of attention heads of the teacher and student must be the same. Although Sun et al. (2020) experiment with various sizes of models, they train the modified teacher architecture to distill the word representation and attention matrix.
Knowledge Distillation Objectives
Constraint
DistilERTT (Sanh et al., 2019)
cos mi,)) Liege
Embedding size
PKD (Sun et al., 2019)
MSE(
1r2,
Tr2.al1
Embedding size
PD (Ture et al., 2019)
Clour
TinyBERT (Jiao et al., 2019)
MSE((1. W.rreed
Σ
MASE(Ai, -Ain)) Cheq
Attention head
Mobile-BERT (Sun et al., 2020)
MSEE(1). - -rt.)]
11
RCL (Ai.. ....A Clowe
Embedding Attention head
A EXPLANATION OF PREVIOUS METHOD: LEVERAGING INDIVIDUAL WORD REPRESENTATION
Table 6 summarizes the knowledge distillation objectives of previous methods and their constraints. DistilBERT (Sanh et al., 2019) uses logit distillation loss (Logit KD), masked language modeling loss, and cosine loss between the teacher and student word representations in the learning process. The cosine loss serves to align the directions of the hidden state vectors of the teacher and student. Since the cosine of the two hidden state vectors is calculated in this process, they have the constraint that the embedding size of the teacher and the student model must be the same.
PKD (Sun et al., 2019) transfers teacher knowledge to the student with Logit KD and patient loss. The patient loss the mean-square loss between the normalized hidden states of the teacher and student. To calculate the mean square error between the hidden states, they have a constraint that the dimensions of hidden states must be the same between teacher and student.
PD (Ture et al., 2019) raised the importance of the pre-training process. By pre- pre-training without distillation and employing only logit kd in the fine-tuning process, they achieved results comparable to other methods. Since only the logit KD is employed, there is no restriction on architecture.
TiRyBERT (Jiao et al., 2019) uses additional loss that matches word representations and attention matrices between the teacher and student. Although they acquire flexibility on the embedding size, using an additional parameter, since the attention matrices of the teacher and student are matched through mean square error loss, the number of attention heads of the teacher and student must be the same.
MobileBERT (Sun et al., 2020) utilizes similar objective with TinyBERT (Jiao et al., 2019) for task-agnostic distillation. However, since they match the hidden states with /2 distance, and attention matrices with K.C divergence between teacher and student, they have restrictions on the size of hidden states and the number of attention heads.
The methods introduced Table 6 have constraints by their respective knowledge distillation objec- tives. However, our CKD method which utilizes the relation statistics between the word representa- tions (hidden states) has the advantage of not having any constraints on student architecture.
12

---

Under review as a conference paper at ICLR 2021
B EXPLANATION OF DATA AUGMENTATION
Algorithm Data Augmentation
Input: sentence 2; hyperparameters N, .M. Pt
Output: augmented data Y (list of sentence)
for i = 1 to len(x:) do if 2x1|] is single-piece word then
Emasked I
Replace Tmasked[i] with '[MASKJ' token
candidates +- M most-likely words predicted for '[MASK]' by BERT BFRTT(mmased
else
candidates[]] + M similar words of z/i] from GloVe end if end for
Append & to Y +0
while n < N do
Zcondidate f-x
for i = 1 to (en(x)))) do
Sample p ~ U(0,1) 1)
if S Pt then Replace Toandidate [i] with a randomly picked word from condidatel end if end for Append Æeandidate to Y
n + n +1 end while
In this paper, for fair comparison with TinyBERT (Jiao et al., 2019), data augmentation is used. We describe the data augmentation method performed on the GLUE benchmark datasets in this section. The overall process is summarized in algorithm 1. Data augmentation consists of the processes selecting replaceable candidates for each word in a sentence and generating new sentences using the candidates.
First, replaceable candidate words (the number of candidate words is M) are selected for each word in the sentence. In this process, the method of selecting candidates is divided according to whether the word is a single-piect word or not. If the word single-piece word, M candidate words are selected in way that predicts the [[MASK] token replaced in same position through the language model. We use the BERT for the language model. If the word is not a single-piece word, the M candidate words are selected using GloVe Ve pre-trained embedding through cosine similarity with the target word. After selecting candidate words in the sentence, total of N augmented sentences are generated. For each word the sentence, it randomly selected and changed among candidate words with a probability of pt. If this process is repeated N times, total of N augmented sentences including the original sentence are created.
In the case of datasets consisting of pair of sentences such QQP MNLI in GLUE benchmark datasets, each sentence is augmented with sentences. In other words, since the paired sentence is attached to N + augmented sentences as originally, total of 2(N + 1) sentences are generated. In this study, we use 15, Pt = 0.4, and N 20 for all datasets except the QQP and MNLI which the original large datasets. For the QQP and MNLI, we use N 2.
13

---

Under review as a conference paper at ICLR 2021
C
EXPLANATION OF VARIOUS MODEL SIZE
Table 7: Details of architecture used for our experiments
Layers
Hidden Size
Feed-forward
Attention Heads
Model
Model
Size
Size
BERT BASE
12
768
3072
110. 1M
BERT SMALL
4
512
2048
8
29. 29.1M
MobileBERT (Sun al., 2020)
24
512
512
25.3M
BERT SHALLOW
12
256
1024
17.6M
TinyBERT (Jiao el al., 2019)
4
312
1200
12
14.5M
MERT
6
256
1024
12.5M
In this section, we describe the various models used in our experiments. Since pre-training each model of various sizes costs lot, pre-trained models of various sizes provided by PD (Ture al., 2019) are used for our experiments. In addition, MobileBERT introduce the Inverted- wwerted-Bottlennck BERT. Excluding BERTBASE, a total of 3 student models are used, and each model name can be different from Turc et al. (2019). Table 7 summarizes the number of layers, hidden size, feed-forward size, attention heads, and model size of each model.
D EXPERIMENT SETTING
This section introduces the experimental setting in details. Our contextual knowledge distillation proceeds in the following order. First, from pre-trained large BERT, task-specific fine-tuning is con- ducted to serve as teacher. Then, prepare the pre-trained small-size architecture which serve as stu- dent. In this case, pre-trained models of various model sizes provided by Ture et al. (2019) employed. Finally, task-specific distillation with our CKD is performed.
We implemented with PyToorch framework and huggingface's transformers package (Wolf et al., 2019). To reduce the hyperparameters search cost, ÄWR in Eq. (3) and ÄLTR in Eq. (5) are used with same value. For the important weights of pair- and triple-wise terms, the locality is applied only to the importance weight w the word relation (WR)-based CKD loss. The importance weight w of the layer transforming relation (LTRR)-base CKD loss is set to 1. In this paper, report the best. result among the following values to find the optimal hyperparameters of each dataset:
Batch size
Learning rate (3, 4, 5) 3c-5, 4c-5, 5e-5
Number of epochs: 4,
Alpha (a) 0.7, 0.9
Temperature (7) 1, 2, 3, 5,5
AWR. ALTR 0.1, 1, 10, 100, 1000
ACKD 10, 100, 1000
14


Reviews:
[{"review_id": "x5HeTbUJw6j", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two new distillation objective, word relations and layer transforming relations. Word relations constrain the pairwise/triplet relations of embeddings at each layer to be closer to the teacher network. The layer transforming relations constrain that the pairwise/triplet relations of embeddings between different layers should match the teacher network. \n\npros:\n\n1. The methods in this paper consider the pairwise or higher order (i.e. triplet)  relations to constrain  the student embeddings, while previous methods usually consider embeddings separately. The methods shall provide more constrained information from the teacher network.\n\n2. Comparison with previous methods, ablation study and other experiments like model sizes, etc demonstrate the effectiveness of the proposed method. In some cases, the student network even outperforms the teacher network (more explanation about this might be needed).\n\nCons (or questions): \n\n1. Why the angle-based method is adopted, instead of other methods (e.g. the maximum/average distance between the triplet)? Is there any experiments studying the effect of the choice of these functions?\n\n2. Previous methods sometimes use a short network (fewer layers) or thin network (smaller hidden sizes). As a result, I am not sure whether the number of parameters are comparable when comparing to the baselines. Could the authors also show the number of parameters of previous methods and their own methods?\n\nTypo:\n, etc. (Devlin et al., 2018; Lan et al., 2019; Liu et al., 2019a; Raffel et al., 2019; Yang et al., 2019): citations should be put before the punctuation. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "bBNnGeh6bz6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a contextual knowledge distillation approach by leveraging two types of contextual knowledge: word relations and layer transforming relation. Recent advancement in this area emphasizes the promising effect of this area in language modeling. \n\nThe paper is well-written and well-structured. The experiment section shows a complete set of experiments including the baselines, benchmark and ablation study. The results are relatively incremental in comparison with TinyBert. Considering that the improvement has been relatively incremental, it would be helpful to compare the models with respect to FLOPs and speedup.\nNovelty: It seems that the notion of structural knowledge distillation have been used previously by Wang et al [1]. It would be great if the authors clarify about their contribution. Also,    the related work section can be enriched by new publications such as PoWER-BERT [2], FastBert [3] and TextBrewer[4]\n1)\tStructure-Level Knowledge Distillation For Multilingual Sequence Labeling, ACL 2020 \n2)\tPoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination , ICML 2020\n3)\tFastBERT: a Self-distilling BERT with Adaptive Inference Time , acl 2020\n4)\tTextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6wQcoxMIKq", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an interesting knowledge distillation method based on a newly defined contextual knowledge of transformer-based models. The proposed contextual knowledge models the pair-wise or triple-wise relations across BERT-based contextual representations, based on which the local structures between Teacher and Student models are encouraged to be well aligned.     \n\nThe main contribution of this work lies in the newly proposed two types of contextual knowledge: Word Relation and Layer Transforming Relation. By using this new contextual knowledge, the contextual representations of Teacher model can be well transferred to Student model. Compared with existing BERT compression methods, like MobileBERT/DistilBERT/TinyBERT, this CKD has the advantage of being directly applied on top of other pre-trained small BERT models without conducting time-consuming pre-training process. \n\nThe authors evaluate their approach on GLUE datasets and compare it to other state-of-the-art models.\n\nThe paper is well-written and organized, the experiments are thorough. However, I have several concerns:\n\n* This proposed KD method is designed for the distillation on downstream tasks, so the whole distillation process should be conducted for each task, while the task-agnostic KD method, like MobileBERT, can be directly used with fine-tuning, please identify this fact in the introduction part. It would be more interesting, if experiments can be conducted during the pre-training stage and further evaluated. \n\n* More experiments on challenging tasks like QA should be added. \n\n* In the Table 1, the performance of TinyBERT on MNLI-mm is 82.6, while in an old version of tinybert paper, (https://arxiv.org/pdf/1909.10351v4.pdf), in the Table 10, the corresponding value is 83.2. And on the official GLUE benchmark the TinyBERT has comparable performance as the proposed CKD(w/DA).\n\n* In the section 5.2, the MobileBERT is further improved by the proposed CKD with self-distillation, that is the MobileBERT is used as its own teacher on the downstream tasks. This comparison is not that fair, since MobileBERT can also be improved by other self-distillation method.  \n\n* In the section 5.3, “we observe that the BERTMINI trained with the CKD shows the higher average\nscore even though BERTMINI has fewer model parameters.” this comparison is unfair since BERTMINI has 6 layers and TinyBERT is a 4-layer model, and less number of model parameters does not always mean fast inference. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5f5y3373G8q", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new knowledge distillation (KD) method for distilling BERT. This area is pretty active given that deploying BERT based models is of keen interest to many industrial applications.\n\nThis paper proposes distillation via modeling \"Word Relation and Layer Transforming Relation\" which essentially aims to \"capture the knowledge of relationships between word representations and LTR defines how each word representation changes\nas it passes through the network layers\". Not to mention *pairwise* and triple-wise relations are being modeled\n\nMy biggest question in the paper (which the doesn't paper address) is that this is bound to be expensive. Yet there is hardly any mention of training time (or time needed to cache these values from the teacher). \n\nThis seems even worse when there is not much gain over existing baselines and can be attributed to simply noise/variance. \n\nOverall, I don't think this method will be impactful at all and it is probably not worth having over existing approaches. It is far too complex. Experiments on the GLUE benchmark alone is also not convincing. \n\nThe authors can try other tasks and perhaps SuperGLUE to make their experiments more convincing. \n\nI think a runtime analysis could help to make the paper stronger to understand the differences. But I would like to make it clear that I want a honest, runtime analysis of how long this distillation would take or the practical considerations. How much more expensive is it to align and compute these values during training of the student. Please make this clear. \n\nI would also like to see a runtime analysis of the baselines as well.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method to perform knowledge distillation (KD) for transformer compression, where two types of contextual knowledge, namely, word relations and layer-transforming relations, are considered for KD. Both pair-wise and triple-wise relations are modeled. 

This paper receives two weak reject and two weak accept recommendations. On one hand, the reviewers appreciate that the authors have added more results into the paper to solve their concerns. On the other hand, several concerns still exist. (i) With regards to the compute-performance trade-off, the gains of the method does not seem too great. One reviewer feels that the authors tried to downplay the cost of their method too much. Though we care more about the inference time, the development time in practice should also not be underestimated. (ii) Compared with TinyBERT, the performance gain looks marginal on the GLUE benchmark (Table 1). (iii) It will make the paper more convincing if pre-training experiments can be performed. 

Overall, after reading the paper, the AC thinks that the novelty of the proposed method is somewhat limited. The AC is also hesitant about whether modeling word relations and layer-transforming relations simultaneously are needed. The choices for ablation study are also not totally clear. 

For example, in Figure 2, it is not clear why the authors choose SST-2 to plot the figure; in Table 5, it is unclear why SST-2, MRPC and QNLI are selected, but not others. When looking at Table 5, it is not totally convincing it is needed to model both WR and LTR, or it is needed to introduce both pair-wise and triple-wise relations. More careful ablation studies are needed. It also remains unclear what kind of word relations or layer-transforming relations are learned. 

In summary, this is a borderline paper, and the rebuttal unfortunately did not fully address the reviewers' main concerns. On balance, the AC regrets that the paper cannot be recommended for acceptance at this time. The authors are encouraged to consider the reviewers' comments when revising the paper for submission elsewhere.
Decision: Reject