SlowGuess/NAACL_2022_OCR huggingface dataset:
First, we observe that the performance gap between ILD methods and vanilla-KD is tight (0.8% and 0.3% on DEV and TEST sets respectively). Moreover, as we expect, ALP-KD performs better on DEV and similar on TEST compared to PKD with 0.2% improvement on the DEV results. Second, results show that RAIL-KD outperforms the best ILD methods by a margin of 0.5% and 0.3% on average on DEV and TEST sets respectively. We notice that, except on RTE TEST, our RAIL-KD variants obtained the highest per-task performances. Third, we observe that RAIL-KD variants perform very similarly, which indicates that our method is effective on concatenated as well as layer-wise distillation.
Similar trends are seen on the 24- to 6-layer model compression experiments, which are reported in Table 3. In this experiment, we used  $\text{Roberta}_{24}$  and  $\text{DistillRoberta}_6$  as teacher and students models respectively. Overall, RAIL-KD outperforms the best baseline by  $1.2\%$  and  $0.3\%$  on DEV and TEST sets respectively. Interestingly, the gap on DEV compared with PKD and ALP-KD is larger than the one reported on  $\text{BERT}^{12}$  experiments, and PKD TEST socres are much lower from that of ALP and RAIL-KD. This might be because PKD skips a large number of intermediate layers on  $\text{RoBERTa}_{24}$ , and the computational cost of ALP-KD attention weights over a large number of teacher layers might produce smaller weights on  $\text{Roberta}_{24}$  compared to  $\text{BERT}_{12}$ .
Furthermore, we demonstrate the effectiveness of RAIL-KD by directly comparing it with CoDIR (Sun et al., 2020a), the current state-of-the-art ILD method. It uses the contrastive objective and a memory bank to extract a large number of negative samples for contrastive loss calculations. Table 4 shows GLUE test results of both approaches when distilling  $\mathrm{RoBERTa}_{12}$  to DistillRoberta $_6$ . CoDIR results are adopted from their paper, and we followed their experimental protocol by not reporting scores on STS-B. On average, RAIL-KD performs on par with CoDIR  $(+0.2\%)$  and outperforms it on 5 out of 8 datasets, while being almost two-times faster as shown in the next section.
5.1 Training Speed-up
Table 5 shows the training time speed up against the teacher of different techniques on 8 GLUE tasks. We measured the speed up by calculating the student_train_time/teacher_train_time using RoBERTa $^{24}$  and DistilRoBERTa $^{6}$  as backbone for teacher and student respectively. We used this configuration because CoDIR pretrained student models are not available and we can only run CoDIR code out-of-the-box.
Model	Teacher	PKD	ALP	CoDIR	RAILl	RAILc
Speed-up	1.00×	1.89×	1.75×	1×	1.89×	1.96×
Table 5: Training time speedup against the teacher for different techniques using the same setting of Table 4.
Our results indicate that random layer mapping not only delivers consistently better results than the deterministic mapping technique such as PKD, but it has less computational overhead during training (two-times faster than CoDIR), while avoiding extensive search experiments to find an optimal mapping. Furthermore, using attention for layer selection (ALP-KD) or contrastive learning (CoDIR) leads to slightly worse performance result than random selection.
5.2 Impact of Random Layer Selection
To evaluate the impact of random layer selection on the performance of RAIL-KD, we report the standard deviation of the  $\mathrm{DistilBERT}_6$  student models (Table 2 models) on the 8 GLUE tasks. As Table 6 shows, the variances of RAIL-KD is in the same range for each task, for instance, RAIL-KD variance is at the same scale compared with PKD and ALP-KD on CoLA and MRPC, and even lower on RTE. This indicates that the gains of RAIL-KD are significant, and are not due to chance in our random selection of layers to distill.
5.3 Out-of-Distribution Test
We further validate the generalization ability of student models by measuring their robustness to in-domain and out-of-domain evaluation. We do so by evaluating models finetuned on MLI, QQP and SST-2 and then evaluated on SciTail, PAWS, and IMDB respectively. These datasets contain counterexamples to biases found in the training data (McCoy et al., 2019; Schuster et al., 2019; Clark et al., 2019). Performances of  $\mathrm{BERT}_{12} / \mathrm{Roberta}_{24}$  teacher and
Figure 2: Cosine similarity between the intermediate layer representations of the  $\mathrm{BERT}_{12}$  teacher and  $\mathrm{DistillBERT}_6$  student models computed on the SST-2 dataset.
Figure 3: Distribution of attention weights learned by DistilBERT $_6$  ALP-KD on CoLA (left), RTE (middle), and MRPC (right). x-axis and y-axis are the teacher and student layer index respectively.
CoLA	SST-2	MRPC	STS-B
PKD	±0.14	±0.54	±0.24	±0.64
ALP-KD	±0.95	±0.33	±0.70	±0.93
RAIL-KDl	±0.49	±0.29	±0.25	±0.50
RAIL-KDc	±0.51	±0.34	±0.40	±0.81
QQP	MNLI	QNLI	RTE
PKD	±0.53	±0.33	±0.49	±1.50
ALP-KD	±0.27	±0.41	±0.30	±1.30
RAIL-KDl	±0.64	±0.27	±0.65	±0.40
RAIL-KDc	±0.14	±0.32	±0.25	±0.23
DistilBERT $_6$ /DistilRoBERT $_6$  student variants are reported in Table 7. Also, we compute the unweighted average score of the three tasks.
First, we notice high variability in models rank and some inconsistencies in performances across tasks when compared with in-domain results. This was also reported in prior works on out-of-domain training and evaluation (Clark et al., 2019; Mahabadi et al., 2020; Utama et al., 2020; Sanh et al., 2020). Still, RAIL-KD clearly outperforms all baselines across tasks. Surprisingly, we observe that PKD and ALP-KD perform poorly (on all three tasks) compared to the Vanilla KD baseline.
Interestingly, we observe that  $\mathrm{RAIL - KD}^l$  performs consistently better (1.1% on average) than
Table 6: Standards deviation (5 runs) of DistilBERT $_6$  ILD models on 8 GLUE tasks.
Model	SciTail	PAWS	IMDB	Avg.
Teacher	70.3/82.7	43.3/43.3	84.6/88.9	66.0/71.6
w/o KD	68.7/74.9	36.5/34.7	81.3/85.8	62.2/65.1
Vanilla KD	68.6/76.1	42.2/36.6	82.0/86.1	64.3/66.3
PKD	68.0/74.8	39.9/36.5	80.9/85.4	62.9/65.6
ALP-KD	66.9/74.7	40.7/35.7	78.7/82.8	62.1/64.4
RAIL-KDl	68.6/76.6	39.0/36.9	83.2/87.3	63.6/67.0
RAIL-KDc	68.7/75.6	43.7/36.2	85.0/85.9	65.8/65.9
Table 7: Out-of-domain performances of models trained on MNLI, QQP, SST-2 and evaluated on SciTail, PAWS, and IMDB respectively.  $\mathrm{BERT}_{12} / \mathrm{Roberta}_{24}$  and  $\mathrm{DistilBERT}_6 / \mathrm{DistilRoBERTa}_6$  are used as backbone for the teacher and students respectively. For each setting, we report the unweighted average score on the 3 tasks.
RAIL-KD<sup>c</sup> on Roberta<sub>24</sub> compression, while RAIL-KD<sup>c</sup> performs better (1.1% on average) on BERT<sub>12</sub>. These results suggest that layer-wise distillation approach is more effective than concatenated distillation when we have a large capacity gap (layer number) between the teacher and the student, and vice versa.
We run extensive analysis to better understand why RAIL-KD performs better than the other baselines. We visualize the layer-wise cosine similarity between the intermediate representations of the
teacher and the student networks. Figure 2 shows the cosine similarity score between three intermediate layer representations of BERT $^{12}$  teacher (i.e. layers 2, 4 and 6) and the first three layer representations of the student for PKD, ALP-KD, RAIL-KD $^{l/c}$  students on 100 samples randomly selected from the SST-2 dataset. Due to space constraints, we only plot the scores for the first three layers of the student model. Similar trends are seen from the other layers.
We found that RAIL-KD allows the student to mimic teacher layers similar to PKD and much better than ALP-KD, despite that the mapping scheme varies at each epoch. Moreover, we observe that ALP-KD results have less similarity scores in the upper intermediate layers. PKD gives lower similarity scores in the lower layers while improving in the upper layers. In contrast, our approach gives more stable similarity scores for all layers while getting closer to the teacher representation in the upper layers.
We further investigate the attention weights learned by ALP-KD, and find out that they mostly focus on few layers (sparse attention). Figure 3 illustrates the distribution of weights, averaged on all training samples of DistilBERT $_6$  ALP-KD studnet on CoLA (left), RTE (middle), and MRPC (right) $^5$ . The figure clearly shows (light colors) that most of ALP weights are concentrated on top layers of the teacher. For instance, layers 1,2,5 of the three students mostly attend to the last layer of  $\mathrm{BERT}_{12}$ . This is an indicator that ALP-KD overfits to the information driven from last layers. In contrast, the randomness in layer selection of RAIL-KD ensures a uniform focus on teacher layers. This may explain the poor performance of ALP-KD on out-of-domain evaluation compared with RAIL-KD.
From Figure 3, we see clearly that ALP-KD mostly prefers the upper layers of the teacher. On the other hand, the deterministic nature of PDK allows it to match better particular layers of the teacher (e.g. bottom ones as shown in Figure 2), but PKD never sees the layers that are skipped by the mapping. Consequently, it is expected that even though PDK can mimic bottom layers well, it is worse overall because it completely ignores some layers of the teacher. Random layer selection allow RAIL-KD to mimic all teacher layers while delivering high performances.
We introduced a novel, simple, and efficient intermediate layer KD approach that outperforms the conventional approaches with performance improvement and efficient training time. RAIL-KD selects random intermediate layers from the teacher which equals to the number of intermediate layers of the student model. The selected intermediate layers are then sorted to distill their representations into the student model. RAIL-KD yields better regularization, which helps performance. Furthermore, our approach shows better performance for larger model distillation with faster training time, which opens up an avenue to investigate our approach for super-large models.
We thank Mindspore<sup>6</sup> for the partial support of this work, which is a new deep learning computing framework. We thank the anonymous reviewers for their insightful comments.
Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541.
Christopher Clark, Mark Yatskar, and Luke Zettlemoyer. 2019. Don't take the easy way out: Ensemble based methods for avoiding known dataset biases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4060-4073.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. https://arxiv.org/abs/1810.04805.
Angela Fan, Edouard Grave, and Armand Joulin. 2019. Reducing transformer depth on demand with structured dropout. arXiv preprint arXiv:1909.11556.
Jie Fu, Xue Geng, Zhijian Duan, Bohan Zhuang, Xingdi Yuan, Adam Trischler, Jie Lin, Chris Pal, and Hao Dong. 2020. Role-wise data augmentation for knowledge distillation. arXiv preprint arXiv:2004.08861.
Similar trends found on other datasets.
$^{6}$ https://www.mindspore.cn/
Mitchell A Gordon, Kevin Duh, , and Nicholas Andrews. 2020. Compressing bert: Studying the effects of weight pruning on transfer learning. arXiv preprint arXiv:2002.08307.
Fu-Ming Guo, Sijia Liu, Finlay S Mungall, Xue Lin, and Yanzhi Wang. 2019. Reweighted proximal pruning for large-scale language representation. arXiv preprint arXiv:1909.12486.
Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari, and Mohammad Norouzi. 2021. Generate, annotate, and learn: Generative models advance self-training and knowledge distillation. arXiv preprint arXiv:2106.06168.
Geoffrey Hinton, Oriol Vinyals, and Jeff. Dean. 2014. Distilling the knowledge in a neural network. NIPS Workshop, https://arxiv.org/abs/1503.02531.
Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, and Ali Ghodsi. 2021. Annealing knowledge distillation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2493-2504, Online. Association for Computational Linguistics.
Mingi Ji, Byeongho Heo, and Sungrae Park. 2021. Show, attend and distill: Knowledge distillation via attention-based feature matching. In Proceedings of the AAAI Conference on Artificial Intelligence.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351.
Ehsan Kamalloo, Mehdi Rezagholizadeh, and Ali Ghodsi. 2022. When chosen wisely, more data is what you need: A universal sample-efficient strategy for data augmentation. arXiv preprint arXiv:2203.09391.
Ehsan Kamalloo, Mehdi Rezagholizadeh, Peyman Passban, and Ali Ghodsi. 2021. Not far away, not so close: Sample efficient nearest neighbour data augmentation via minimax. arXiv preprint arXiv:2105.13608.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI.
Krtin Kumar, Peyman Passban, Mehdi Rezagholizadeh, Yiusing Lau, and Qun Liu. 2022. From fully trained to fully random embeddings: Improving neural machine translation with compact word embedding tables.
Tianda Li, Ahmad Rashid, Aref Jafari, Pranav Sharma, Ali Ghodsi, and Mehdi Rezagholizadeh. 2021. How to select one among all? an extensive empirical study towards the robustness of knowledge distillation in natural language understanding.
Vasileios Lioutas, Ahmad Rashid, Krtin Kumar, Md Akmal Haidar, and Mehdi Rezagholizadeh. 2020. Improving word embedding factorization for compression using distilled nonlinear neural decomposition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 2774-2784.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Roberta: A robustly optimized fbertg pretraining approach. arXiv preprint arXiv:1907.11692.
Hou Lu, Huang Zhiqi, Shang Lifeng, Jiang Xin, Chen Xiao, and Liu Qun. 2020. Dynabert: Dynamic bert with adaptive width and depth. arXiv preprint arXiv:2004.04037.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In ACL.
Rabeeh Karimi Mahabadi, Yonatan Belinkov, and James Henderson. 2020. End-to-end bias mitigation by modelling biases in corpora. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8706-8716. Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? in neurips. In NeurIPS.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In In International Conference on Learning Representations.
Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh, and Qun Liu. 2021. ALP-KD: attention-based layer projection for knowledge distillation. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13657-13665. AAAI Press.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances
in neural information processing systems, 32:8026-8037.
Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. 2020. Fully quantized transformer for machine translation. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1-14.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. https://cdn.openai.com/better-language-models/language_models—are_unsupervised multitask_ learners.pdf.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789.
Ahmad Rashid, Vasileios Lioutas, Abbas Ghaddar, and Mehdi Rezagholizadeh. 2020. Towards zero-shot knowledge distillation for natural language processing. arXiv preprint arXiv:2012.15495.
Ahmad Rashid, Vasileios Lioutas, and Mehdi Rezagholizadeh. 2021. Mate-kd: Masked adversarial text, a companion to knowledge distillation. arXiv preprint arXiv:2105.05912.
Mehdi Rezagholizadeh, Aref Jafari, Puneeth Salad, Pranav Sharma, Ali Saheb Pasand, and Ali Ghodsi. 2021. Pro-kd: Progressive distillation by following the footsteps of the teacher. arXiv preprint arXiv:2110.08532.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019a. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019b. Distilroberta, a distilled version of roberta: smaller, faster, cheaper and lighter. https://huggingface.co/distilroberta-base.
Victor Sanh, Thomas Wolf, Yonatan Belinkov, and Alexander M Rush. 2020. Learning from others' mistakes: Avoiding dataset biases without modeling them. arXiv preprint arXiv:2012.01300.
Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3410-3416.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. 2019. Q-bert: Hessian based ultra low precision quantization of bert. arXiv preprint arXiv:1909.05840.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model compression. https://arxiv.org/abs/1908.09355.
Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang, Shuohang Wang, and Jingjing Liu. 2020a. Contrastive distillation on intermediate representations for language model compression. In EMNLP.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020b. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984.
Marzieh S Tahaei, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. 2021. Kroneckerbert: Learning kronecker decomposition for pre-trained language models via knowledge distillation. arXiv preprint arXiv:2109.06243.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. 2019. Contrastive representation distillation. arXiv preprint arXiv:1910.10699.
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962.
Prasetya Ajie Utama, Nafise Sadat Moosavi, and Iryna Gurevych. 2020. Towards debiasing nlu models from unknown biases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7597-7610.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in Neural Information Processing Systems, 32.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2020a. Minilmv2: Multihead self-attention relation distillation for compressing pretrained transformers. arXiv preprint arXiv:2012.15828.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020b. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957.
Yimeng Wu, Peyman Passban, Mehdi Rezagholizadeh, and Qun Liu. 2020a. Why skip if you can combine: A simple knowledge distillation technique for intermediate layers. https://arxiv.org/abs/2010.03034.
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. 2020b. Lite transformer with long-short range attention. arXiv preprint arXiv:2004.11886.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In NeuRIPS.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. arXiv preprint arXiv:1910.06188.
George-Eduard Zaharia, Andrei-Marius Avram, Dumitru-Clementin Cercel, and Traian Rebedea. 2021. Dialect identification through adversarial learning and knowledge distillation on Romanian BERT. In Proceedings of the Eighth Workshop on NLP for Similar Languages, Varieties and Dialects, VarDial@EACL 2021, Kiyv, Ukraine, April 20, 2021, pages 113-119. Association for Computational Linguistics.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130.