SlowGuess/NAACL_2022_OCR huggingface dataset:
Md Akmal Haidar $^{1*}$  Nithin Anchuri $^{1,2*}$  Mehdi Rezagholizadeh $^{1}$  Abbas Ghaddar $^{1}$  Philippe Langlais $^{2}$  Pascal Poupart $^{3}$
<sup>1</sup> Huawei Noah's Ark Lab
$^{2}$  RALI/DIRO, Université de Montréal, Canada
<sup>3</sup> David R. Cheriton School of Computer Science, University of Waterloo {mehdi.rezagholizadeh, abbas.ghaddar}@huawei.com felipe@iro.umontreal.ca, ppoupart@uwaterloo.ca
Intermediate layer knowledge distillation (KD) can improve the standard KD technique (which only targets the output of teacher and student models) especially over large pre-trained language models. However, intermediate layer distillation suffers from excessive computational burdens and engineering efforts required for setting up a proper layer mapping. To address these problems, we propose a Random Intermediate Layer Knowledge Distillation (RAIL-KD) approach in which, intermediate layers from the teacher model are selected randomly to be distilled into the intermediate layers of the student model. This randomized selection enforces that all teacher layers are taken into account in the training process, while reducing the computational cost of intermediate layer distillation. Also, we show that it acts as a regularizer for improving the generalizability of the student model. We perform extensive experiments on GLUE tasks as well as on out-of-domain test sets. We show that our proposed RAIL-KD approach outperforms other state-of-the-art intermediate layer KD methods considerably in both performance and training-time.
Pre-trained Language Models (PLMs), such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2020) and XLNet (Yang et al., 2019) have shown remarkable abilities to match and even surpass human performances on many Natural Languages Understanding (NLU) tasks (Rajpurkar et al., 2018; Wang et al., 2018, 2019). However, the deployment of these models in real world applications (e.g. edge devices) come with challenges, mainly due to large model size and inference time.
In this regard, several model compression techniques such as quantization (Shen et al., 2019;
Zafrir et al., 2019; Kumar et al., 2022; Prato et al., 2020), pruning (Guo et al., 2019; Gordon et al., 2020; Michel et al., 2019), matrix factorization (Tahaei et al., 2021; Lioutas et al., 2020), optimizing the Transformer architecture (Fan et al., 2019; Wu et al., 2020b; Lu et al., 2020), and knowledge distillation (Sanh et al., 2019a; Jiao et al., 2019; Sun et al., 2020b; Wang et al., 2020a; Rashid et al., 2021; Passban et al., 2021; Jafari et al., 2021; Rezagholizadeh et al., 2021) have been developed to reduce the model size and latency, while maintaining comparable performance to the original model.
KD, which is the main focus of this work, is a neural model compression approach that involves training a small student model with the guidance of a large pre-trained teacher model. In the original KD technique (Bucilua et al., 2006; Hinton et al., 2014; Turc et al., 2019), the teacher output predictions are used as soft labels for supervising the training of the student. There has been several attempts in the literature to reduce the teacher-student performance gap by leveraging data augmentation (Fu et al., 2020; Li et al., 2021; Jiao et al., 2019; Kamalloo et al., 2021, 2022), adversarial training (Zaharia et al., 2021; Rashid et al., 2020, 2021), and intermediate layer distillation (ILD) (Wang et al., 2020b,a; Ji et al., 2021; Passban et al., 2021).
When it comes to BERT compression, ILD leads to clear gains in performances (Sanh et al., 2019a; Jiao et al., 2019; Wang et al., 2020a) due to its ability to enhance the knowledge transfer beyond logits matching. This is done by mapping intermediate layer representations of both models to a common space<sup>1</sup>, and then matching them via regression (Sun et al., 2019) or cosine similarity (Sanh et al., 2019a) losses. One major problem with ILD is the absence of an appropriate strategy to select layers to be matched on both sides, reacting to the skip
<sup>1</sup>In some cases, the representations are directly matched if the teacher and student have the same hidden size.
* Work done while at Huawei.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1389 - 1400 July 10-15, 2022 ©2022 Association for Computational Linguistics
Model	Layer Mapping	Complexity	Limitation
PKD (Sun et al., 2019)	Extra Hyperparameter	O(m)	Extensive Search
CKD (Wu et al., 2020a)	Extra Hyperparameter	O(m)	Extensive Search
ALP-KD (Passban et al., 2021)	Attention	O(m × n)	Slow Training time
CoDIR (Sun et al., 2020a)	Contrastive Learning	O(K × m)	Slow Training time
RAIL-KDl (our)	Random Selection	O(m)	-
RAIL-KDc (our)	Random Selection	O(m)	-
Table 1: Main characteristics and limitation of different approaches that tackle the skip and search problem. Concat indicates if the approach supports concatenated layers distillation.  $n$  and  $m$  refer to the teacher and student layer number respectively, while  $K$  refers to number of negative samples of CoDIR.
and search problem (Passban et al., 2021). Some solutions in the literature mostly rely on layer combination (Wu et al., 2020a), attention-based layer projection (Passban et al., 2021) and contrastive learning (Sun et al., 2020a). While these solutions are all effective to some extent, to the best of our knowledge, there is no work in the literature doing a comprehensive evaluation of these techniques in terms of both efficiency and performance.
A case in point is that the aforementioned solutions to the layer skip and search problem do not scale to very deep networks. We propose RAIL-KD (Random Intermediate Layer KD), a simple yet effective method for intermediate layer mapping which randomly selects  $k$  out of  $n$  intermediate layers of the teacher at each epoch to be distilled to the corresponding student layers. Since the layer selection is done randomly, all the intermediate layers of the teacher will have a chance to be selected for distillation. Our method adds no computational cost to the training, still outperforming all aforementioned methods on the GLUE benchmark (Wang et al., 2018). Moreover, we observe larger gains when distilling from large teacher models, as well as when student models are evaluated on out-of-domain datasets. Last, we report the results on 5 random seeds in order to verify the contribution of the random selection process, thus making the comparison fair with previous methods. The main contributions of our paper are as follows:
- We introduce RAIL-KD, a more efficient and scalable intermediate layer distillation approach.
- To the best of our knowledge, we are the first to perform a comprehensive study of the ILD techniques in terms of both efficiency and performance.
- We consider the distillation of models such as
BERT and RoBERTa, and compare different up-to-date distillation techniques on out-of-domain test sets.
In recent years, a wide range of methods have tried to expand knowledge transfer of transformer-based (Vaswani et al., 2017) NLU models beyond logits matching. DistillBERT (Sanh et al., 2019a) added a cosine similarity loss between teacher and student embeddings layers. TinyBERT (Jiao et al., 2019), MobileBERT (Sun et al., 2020b), and MiniLM (Wang et al., 2020b) matched the intermediate layers representations and self-attention distributions of the teacher and the student.
In PKD, Sun et al. (2019) used deterministic mapping strategies to distill a 12-layer BERT teacher to a 6-layer BERT student. PKD-LAST and PKD-SKIP refer to matching layers  $\{1 - 5\}$  of the student with layers  $\{7 - 11\}$  and  $\{2,4,6,8,10\}$  of the teacher respectively. However, these works ignored the impact of layer selection, as they used a fixed layer-wise mapping.
Researchers have found that tuning the layer mapping scheme can significantly improve the performance of ILD techniques (Sun et al., 2019). Nevertheless, finding the optimal mapping can be challenging, which is referred to as the layer skip and search problems by Passban et al. (2021). To address the layer skip problem, CKD (Wu et al., 2020a) is built on top of PKD by partitioning all the intermediate layers of the teacher to the number of student layers. Then, the combined representation of the layers of each partition is distilled into a number of subset corresponding to the number of student layers. However, finding the optimal
2e.g. matching the first (or last)  $k$  layers of the student with their corresponding teacher layers.
(a) Layer-wise Representation Distillation
(b) Concatenated Representation Distillation
Figure 1: Proposed RAIL-KD technique for efficient intermediate layer distillation. (a) This version shows a layerwise projection which is indicated as  $\mathrm{RAIL - KD}^l$  in the paper. (b) This variant named  $\mathrm{RAIL - KD}^c$ , concatenates the intermediate representations of each network before distillation.
partitioning scheme requires running exhaustive experiments. Given teacher and student BERT models with  $n$  and  $m$  layers respectively (where  $n >> m$ ), it is not trivial to choose the teacher layers that can be incorporated in the distillation process and how we should map them to the student layers (search).
ALP-KD (Passban et al., 2021) overcomes this issue by computing attention weights between each student layer and all the intermediate layers of the teacher. The learned attention weights for each student layer are used to obtain a weighted representation of all teacher layers. Although ALP-KD has shown promising results on 12-layer BERT-based compression, attending to all layers of the teacher adds considerable computational overhead to the training phase. This can become computationally prohibitive when scaling to very large models such as RoBERTa-large (Liu et al., 2020) or GPT-2 (Radford et al., 2019). Alternatively, CoDIR (Sun et al., 2020a) exploited contrastive learning (Tian et al., 2019) to perform intermediate layers matching between the teacher and the student models with no deterministic mapping. Similar to ALP-KD, this approach also requires excessive training time due to the contrastive loss calculation and the use of negative samples from a memory bank.
Table 1 summarizes the main characteristics of the existing state-of-the-art intermediate layer distillation techniques (PKD, CKD, and CoDIR) used
for pre-trained language models compared with our proposed RAIL-KD. First, PKD and CKD treat the mapping as an extra hyperparameter that requires extensive experiments in order to find the optimal mapping. Second, ALP-KD (Passban et al., 2021) and CoDIR (Sun et al., 2020a) use the attention mechanism and contrastive learning respectively to address the issue, but at the expense of extra computational cost.
Our proposed RAIL-KD method does not add any computational cost to the distillation process, while empirically outperforming previous methods. For instance, RAIL-KD is roughly two-times faster than CoDIR in a 24 to 6 layer compression. In addition, it does not require extensive experiments to find the optimal mapping scheme. In this work, we position ourselves to works that tackle the skip and search problem. In other words, we don't compare with works like TinyBERT (Jiao et al., 2019) or MiniLM (Wang et al., 2020b), which use extra losses like self-attention distribution matching. However, we expect that these methods, as well as state-of-the-art (Rashid et al., 2021; He et al., 2021) ones can take full advantage of RAIL-KD, since they use a deterministic layer mapping scheme.
3This only concerns works that perform intermediate layer distillation
3 RAIL-KD
The RAIL-KD method is sketched in Figure 1. RAIL-KD transfers intermediate knowledge of a pre-trained teacher  $T$  with  $n$  intermediate layers to a student model  $S_{\theta}$  with  $m$  intermediate layers. In contrast to traditional intermediate layer distillation techniques which keep the selected layers of the teacher for distillation fixed during training, in RAIL-KD, at each epoch, a few intermediate layers from the teacher model are selected randomly for distillation. Here for simplicity, we set the number of selected intermediate layers of the teacher model equal to that of the student model.
Let  $(X,y)$  denote a training sample  $X = (x_0,\dots ,x_{L - 1})$  which is a sequence of  $L$  tokens and  $y$  its corresponding label. In Figure 1, our Random Selection operator is applied to the intermediate layers of the teacher to randomly select  $m$  out of  $n$  layers. The intermediate layer representations of the  $m$  selected layers of the teacher and the student model corresponding to the  $X$  input can be described as  $\mathcal{H}_X^T = \{H_{1,X}^T,\dots ,H_{m,X}^T\}$  and  $\mathcal{H}_X^{S_\theta} = \{H_{1,X}^{S_\theta},\dots ,H_{m,X}^{S_\theta}\}$  respectively, where  $H_{i,X}^{T} = \cup_{k = 0}^{L - 1}\{h_{i,x_k}^T\} \in R^{L\times d_1}$  and  $H_{i,X}^{S_\theta} = \cup_{k = 0}^{L - 1}\{h_{i,x_k}^{S_\theta}\} \in R^{L\times d_2}$ . Here,  $d_{1}$  and  $d_{2}$  indicate the hidden dimension of the layers of the teacher and the student models respectively. To obtain  $H_{i,X}^{T}$  and  $H_{j,X}^{S}$ , we need to find the individual representation of each token  $x_{k}$  at each layer  $i$ , which is indicated as  $h_{i,x_k}^T$  and  $h_{i,x_k}^S$  for the teacher and student networks respectively.
At this stage, we need to obtain an aggregated representation of the sequence  $X$  at each layer. In this regard, one can either use the  $\langle \mathrm{CLS}\rangle$  token representation or use the mean-pooling of the sequence representations of the layer. Since in Sun et al. (2020a), the mean-pooling representation shows better results, we adopt it to compute the sentence representation of each layer. Mean-pooling is a row-wise average over  $H_{i,X}^{T}$ ,  $H_{i,X}^{S_{\theta}}$  to get  $\bar{h}_{i,X}^{T} \in R^{d_1}$ $\bar{h}_{i,X}^{S_{\theta}} \in R^{d_2}$  (Sun et al., 2020a):
$$
\bar {h} _ {i, X} ^ {T} = \frac {1}{L} \sum_ {k = 0} ^ {L - 1} h _ {i, x _ {k}} ^ {T}; \bar {h} _ {i, X} ^ {S _ {\theta}} = \frac {1}{L} \sum_ {k = 0} ^ {L - 1} h _ {i, x _ {k}} ^ {S _ {\theta}} (1)
$$
After obtaining aggregated layer representations for both the student and teacher networks, our RAIL-KD proposal is to randomly select  $m$  layer representations from the teacher through training to perform the intermediate layer distillation (ILD).
RAIL-KD does ILD in two different forms: using layer-wise distillation (see Fig. 1(a)) or by concatenating layer representations (see Fig. 1(b)) which are described in the following two sub-sections.
3.1 Layer-wise RAIL-KD
In this setting, the representations  $\bar{h}_{i,X}^{T}\in R^{d_{1}}$  and  $\bar{h}_{i,X}^{S_{\theta}}\in R^{d_2}$  are projected into a same-size lower-dimensional space  $\hat{h}_{i,X}^T,\hat{h}_{i,X}^{S_\theta}\in R^u$  using  $(d_{1}\times u)$  and  $(d_{2}\times u)$  linear mappings respectively. Assume that the set  $\mathcal{A} = \{a_{\kappa}|a_{\kappa}\sim \{1,2,\dots,n\},1\leq \kappa \leq m\}$  contains indices of selected  $m$  layers from the teacher, then to calculate the layer-wise loss we have:
$$
L _ {\mathrm {R A I L - K D} ^ {l}} =
$$
$$
\sum_ {X \in \mathcal {X}} \sum_ {i \in \mathcal {A}} \alpha_ {i} \left(\left| \left| \frac {\hat {h} _ {i , X} ^ {T}}{\left| \left| \hat {h} _ {i , X} ^ {T} \right| \right| _ {2}} - \frac {\hat {h} _ {i , X} ^ {S _ {\theta}}}{\left| \left| \hat {h} _ {i , X} ^ {S _ {\theta}} \right| \right| _ {2}} \right| \right| _ {2} ^ {2}\right) \tag {2}
$$
where  $\mathcal{X}$  denotes the training set, and  $\alpha_{i}$  is a hyperparameter to assign a custom weights to the layerwise distillation loss. It is worth mentioning that in our experiments we set  $\alpha_{i} = 1$ .
3.2 Concatenated RAIL-KD
In this setting, intermediate layer representations are concatenated and then distilled:  $\bar{h}_X^T = [\bar{h}_{i,X}^T]_{i\in \mathcal{A}}$ $\bar{h}_X^{S_\theta} = [\bar{h}_{j,X}^{S_\theta}]_{j = 1}^m$  which are further mapped into the same lower-dimensional space  $\hat{h}_X^T,\hat{h}_X^{S_\theta}\in R^u$  using  $(md_1\times u)$  and  $(md_2\times u)$  linear mappings to calculate the concatenated distillation loss.
$$
L _ {\text {R A I L - K D} ^ {c}} = \sum_ {X \in \mathcal {X}} | | \frac {\hat {h} _ {X} ^ {T}}{| | \hat {h} _ {X} ^ {T} | | _ {2}} - \frac {\hat {h} _ {X} ^ {S _ {\theta}}}{| | \hat {h} _ {X} ^ {S _ {\theta}} | | _ {2}} | | _ {2} ^ {2} \tag {3}
$$
Any type of loss such as contrastive (Sun et al., 2020a), or mean-square-error (MSE) (Passban et al., 2021; Sun et al., 2019) can be applied for our RAIL-KD approach.
The intermediate representation distillation loss  $L_{\mathrm{RAIL - KD}}$  is combined with the original KD loss  $L_{\mathrm{KD}}$ , which is used to distill the knowledge from the output logits of the teacher model  $T$  to the output logits of the student model  $S_{\theta}$ , and the original cross-entropy loss  $L_{\mathrm{CE}}$ . The total loss function for training the student model is:
$$
\mathcal {L} = \lambda_ {1} L _ {\mathrm {C E}} + \lambda_ {2} L _ {\mathrm {K D}} + \lambda_ {3} L _ {\mathrm {R A I L - K D} ^ {l / c}} \tag {4}
$$
where  $\lambda_{1},\lambda_{2}$  , and  $\lambda_3$  are hyper-parameters of our model to minimize the total loss, and  $\lambda_1 + \lambda_2+$ $\lambda_3 = 1$
4.1 Datasets and Evaluation
We evaluate RAIL-KD on 8 tasks from the GLUE benchmark (Wang et al., 2018): 2 single-sentence (CoLA and SST-2) and 5 sentence-pair (MRPC, RTE, QQP, QNLI, and MNLI) classification tasks, and 1 regression task (STS-B). Following prior works (Sun et al., 2019; Passban et al., 2021; Jiao et al., 2019; Sun et al., 2020a), we use the same metrics as the GLUE benchmark for evaluation. Moreover, to further show the generalization capability of our RAIL-KD method on out-of-domain (OOD) across tasks, we use Scitail (Khot et al., 2018), PAWS (Paraphrase Adversaries from Word Scrambling) (Zhang et al., 2019), and IMDb (Internet Movie Database) (Maas et al., 2011) test sets to evaluate the models fine-tuned on MNLI, QQP, and SST-2 tasks respectively.
We run extensive experiments on 3 different teachers in order to ensure a fair comparison with a wide range of prior works, and also to show the effectiveness of RAIL-KD. We experiment with the 12-layer BERT-base-uncased (Devlin et al., 2019) model as teacher  $(\mathrm{BERT}_{12})$  and the 6-layer DistilBERT (Sanh et al., 2019a) as student  $(\mathrm{DistillBERT}_6)$  to compare with PKD (Sun et al., 2019) and ALP-KD (Passan et al., 2021). Also, we consider 24-layer RoBERTa-large (Liu et al., 2020) and 6-layer DistilRoberta (Sanh et al., 2019b) as the backbone for teacher  $(\mathrm{RoBERTa}_{24})$  and student  $(\mathrm{DistilRoberta}_6)$  respectively to compare results when  $n >> m$ . Furthermore, we perform evaluation using the 12 layers RoBERTa-base  $(\mathrm{RoBERTa}_{12})$  model as a teacher to be able to directly compare our figures with the ones of CoDIR.
We re-implement PKD (Sun et al., 2019) and ALP-KD (Passban et al., 2021) approaches using the default settings proposed in the respective papers. We used early stopping based on performance on the development set, while making sure that the figures are in line with the ones reported in the papers. More precisely, the best layer setting for PKD teacher  $\mathrm{BERT}_{12}$  is  $\{2,4,6,8,10\}$  to distill into  $\mathrm{DistilBERT}_6$ . For DistilRoBERTa, we choose the intermediate layers 4, 8, 12, 16, 20 from the
teacher RoBERTa $_{24}$  model for distillation that we found to work the best on the development set.
Using ALP-KD, we compute attention weights for the intermediate layers of the teacher (i.e., 1 to 11 for  $\mathrm{BERT}_{12}$  and 1 to 23 for  $\mathrm{RoBERTa}_{24}$  models) to calculate the weighted intermediate representations of the teacher for each intermediate layer of the student model (i.e., 1 to 5 layers of the student models). Since, the hidden dimensions of the  $\mathrm{RoBERTa}_{24}$  and  $\mathrm{DistilRoBERTa}_{6}$  are different, we linearly transform them into same lower-dimensional space. We train the PKD and ALP-KD models following (Sun et al., 2019; Passban et al., 2021).
For RAIL-KD, at each epoch we randomly select 5 layers from the intermediate layers of the teacher (i.e., from layers 1 to 11 for  $\mathrm{BERT}_{12}$  model and 1 to 23 for RoBERTa24 model). Then, we sort the layer indexes and perform layer-wise distillation (Figure 1(a)) for RAIL-KD. For RAIL-KDc, we concatenated the representations of the sorted randomly selected intermediate layers and then perform concatenated representation distillation (Figure 1(b)).
We use a linear transformation to map the intermediate representations (layer-wise or concatenated representations) into 128-dimensional space  $(u = 128)$  and normalize them before computing the loss  $L_{\mathrm{RAIL - KD}^{l / c}}$  for both  $\mathrm{BERT}_{12}$  and  $\mathrm{RoBERTa}_{24}$  distillations. We fixed  $\alpha_{i} = 1$ ,  $\lambda_1, \lambda_2, \lambda_3 = 1/3$  for our proposed approaches. We search learning rate from  $\{1\mathrm{e} - 5, 2\mathrm{e} - 5, 5\mathrm{e} - 5, 4\mathrm{e} - 6\}$ , batch size from  $\{8, 16, 32\}$ , and fixed the epoch number to 40 for all the experiments. We run all experiments 5 times and report average score, in order to validate the credibility of our results. We ran all the experiments on a single NVIDIA V100 GPU using mixed-precision training (Micikevicius et al., 2018) and PyTorch (Paszke et al., 2019) framework.
Table 2 shows the performances of models trained on GLUE tasks, and evaluated on their respective DEV and TEST sets for 12-layer to 6-layer distillation. BERT $^{12}$  and DistilBERT $^{6}$  are used as backbone for the teacher and student models respectively. The baselines are fine-tuned without KD (w/o KD) and with Vanilla KD. Moreover, we di
4We didn't find a significant improvement when changing these values.
Model	CoLA	SST-2	MRPC	STS-B	QQP	MNLI	QNLI	RTE	Avg.
DEV
Teacher	61.3	93.0	90.6	88.4	91.0	84.7	91.5	68.2	83.7
w/o KD	53.3	90.1	90.0	86.5	90.4	82.3	89.1	61.7	80.4
Vanilla KD	55.8	90.3	90.3	86.6	90.5	82.7	89.6	68.5	81.9
PKD	56.1	91.3	90.7	87.4	91.2	83.3	90.2	69.3	82.5
ALP-KD	56.8	90.8	90.6	87.5	91.0	83.4	90.2	70.4	82.7
RAIL-KDl	58.8	92.8	91.0	87.8	91.2	83.5	90.3	70.4	83.2
RAIL-KDc	57.2	91.9	90.8	87.9	91.4	83.5	90.1	72.2	83.2
TEST
Teacher	52.0	92.9	87.8	82.3	88.9	84.3	90.7	66.0	81.0
w/o KD	50.7	91.7	87.2	80.4	88.3	81.4	88.4	57.6	78.6
Vanilla KD	50.9	91.0	87.7	81.0	88.5	82.2	88.7	60.6	79.2
PKD	50.6	92.0	87.2	81.7	89.1	82.7	89.0	60.6	79.5
ALP-KD	50.2	90.8	87.6	81.9	89.0	82.7	88.9	61.8	79.5
RAIL-KDl	51.3	92.3	87.9	82.1	89.2	82.6	89.0	60.8	79.7
RAIL-KDc	50.6	92.5	88.2	81.4	88.9	82.8	89.3	61.3	79.8
Table 2: DEV and TEST performances on GLUE benchmark when  $\mathrm{BERT}_{12}$  and  $\mathrm{DistillBERT}_6$  are used as backbone for the teacher and students variants respectively. Bold mark describes the best results.
Model	CoLA	SST-2	MRPC	STS-B	QQP	MNLI	QNLI	RTE	Avg.
DEV
Teacher	68.1	96.4	91.9	92.3	91.5	90.1	94.6	86.3	88.9
PKD	62.3	91.6	90.9	88.9	91.6	84.4	91.1	71.1	84.0
ALP-KD	62.7	91.7	91.1	89.1	91.4	84.3	90.8	71.1	84.0
RAIL-KDl	65.4	93.8	90.1	89.4	91.9	84.8	92.0	72.9	85.1
RAIL-KDe	65.3	93.7	91.4	89.4	92.0	84.8	92.0	72.9	85.2
TEST
Teacher	68.1	96.4	91.9	92.3	91.5	90.2	94.6	86.3	85.3
PKD	50.2	89.4	88.9	84.5	92.3	84.0	90.2	62.7	80.3
ALP-KD	53.6	89.6	89.2	84.6	92.8	83.6	90.4	64.4	81.0
RAIL-KDl	53.4	89.5	88.9	84.8	93.6	84.5	91.1	63.5	81.2
RAIL-KDe	53.6	89.6	89.6	84.8	93.4	83.9	91.6	63.8	81.3
Table 3: DEV and TEST performances on GLUE benchmark when  $\mathrm{RoBERTa}_{24}$  and DistillRoberta $_6$  are used as backbone for the teacher and student variants respectively. Bold mark describes the best results.
Model	CoLA	SST-2	MRPC	QQP	MNLI	QNLI	RTE	Avg.
RoBERTa12	62.0	95.3	90.1	89.4	87.2	93.2	72.7	84.6
CoDIR	53.6	93.6	89.4	89.1	83.2	90.4	65.6	81.0
RAIL-KDc	54.2	93.6	88.4	89.5	83.9	91.7	64.5	81.2
Table 4: GLUE test results of  $\mathrm{{RAIL}} - {\mathrm{{KD}}}^{c}$  when using  ${\text{Roberta}}_{12}$  and DistilRoBERTa  ${}_{6}$  as backbone for teacher and students. Results of CoDIR are directly copied from their paper (Sun et al., 2020a).
rectly compare RAIL-KD $^{lc}$  results with PKD and ALP-KD as more competing techniques.