SlowGuess/NAACL_2022_OCR huggingface dataset:
The first step of the proposed framework is to design the Kronecker layers by defining the shape of A and B. Once the shape of one of them is set, the shape of the other one can be obtained accordingly. Therefore we only searched among different choices for  $m_{1}$  and  $n_{1}$  which are limited to the factors of the original weight matrix ( $m$  and  $n$  respectively). We used the same configuration for all the matrices in the MHA. Also For the FFN, we chose the configuration for one layer, and for the other layer, the dimensions are swapped. For the embedding layer, since  $B^{E}$  is a row vector, we only need to choose  $n$ . The shapes of the Kronecker factors were chosen to obtain the desired compression factor and FLOPS reduction according to Eq.7. To investigate the effect of summation we also selected one configuration with summation of 4 Kronecker products. Similarly, after fixing the number of summation we chose the configuration that provided the desired compression and latency reduction. Table 1 summarises the configuration of Kronecker factorization for the three compression factors used in this work.
For KD at the pre-training stage, the Kronecker-BERT model was initialized using the teacher (pre-trained  $\mathrm{BERT}_{\mathrm{BASE}}$  model). This means that for layers that were not compressed like the last layer, the values are copied from the teacher to the student. For initialization of the compressed layers in the pre-training stage, the nearest Kronecker solution explained in section 3.2.1 is used to approximate Kronecker factors (A and B) from the pre-trained  $\mathrm{BERT}_{\mathrm{BASE}}$  model. In the pre-training stage,  $10\%$  of the English Wikipedia was used for 3 epochs.
Model	Params	MNLI-(m/mm)	SST-2	MRPC	CoLA	QQP	QNLI	RTE	STS-B	Avg
BERTBASE	109.5M	83.9/83.4	93.4	87.9	52.8	71.1	90.9	67	85.2	79.5
BERT4-PKD	7.6B	79.9/79.3	89.4	82.6	24.8	70.2	85.1	62.3	79.8	72.6
MobileBERTINY	15.1M	81.5/81.6	91.7	87.9	46.7	68.9	89.5	65.1	80.1	77.0
TinyBERT	14.5M	82.5/81.8	92.6	86.4	44.1	71.3	87.7	66.6	80.4	77.0
KroneckerBERT8	14.3M	83.0/82.7	91.9	88.5	39.8	71.5	90.2	67.2	84.5	77.7
Table 2: Results on the test set of GLUE official benchmark. The results for BERT,  $\mathrm{BERT}_4$ -PKD and TinyBERT are taken from (Jiao et al., 2019). For all other baselines, the results are taken from their associated papers. Note that our KroneckerBERT only performs pre-training KD on  $10\%$  of the Wikipedia. Also MobileBERT distils knowledge from a specially designed teacher that is trained from scratch and TinyBERT uses an extensive data augmentation in the fine-tuning stage.
Model	Params	MNLI-(m/mm)	SST-2	MRPC	CoLA	QQP	QNLI	RTE	STS-B
BERTBASE	108.5M	83.9/83.4	93.4	87.9	52.8	71.1	90.9	67	85.2
SharedProject	5.6M	76.4/75.2	84.7	84.9	-	-	-	-	-
LadaBERT4	11M	75.8/76.1	84.0	-	-	67.4	75.1	-	-
KroneckerBERT21	5.2M	81.3/80.1	88.4	87.1	28.3	70.5	86.1	64.7	81.3
Table 3: Results on the test set of the GLUE official benchmark for extreme compression factors. The results of the baselines are taken from their associated papers. LadaBERT and SharedProject refer to (Mao et al., 2020) and (Zhao et al., 2019a) respectively.
The batch size in pre-training was set to 64 and the learning rate was set to e-3. After pre-training, the obtained Kronecker model is used to initialize the Kronecker layers in the student model for task-specific fine-tuning. The Prediction layer is initialized from the fine-tuned  $\mathrm{BERT}_{\mathrm{BASE}}$  teacher. For fine-tuning on each task, we optimize the hyperparameters based on the performance of the model on the dev set. See appendix for more details on the results and the selected hyperparameters.
In this section, we compare our KroneckerBERT with the sate-of-the-art compression methods applied to BERT on GLUE and SQuAD. We also perform an ablation study to investigate the effect of pretraining and KD.
As for baselines we select two main categories of compression methods, those with compression factor  $< 10$  and those with compression factor  $>10$ . In the first category, we have  $\mathrm{BERT}_{\mathrm{PKD}}$  (Sun et al., 2019) with a low compression factor, and models with similar compression factor as our Kronecker  $\mathrm{BERT}_8$ : MobileBERT (Sun et al., 2020) and TinyBERT (Jiao et al., 2019). We also compare our results to the dynaBERT model (Hou et al., 2020). For the second category, we compare our results with SharedProject (Zhao et al., 2019a) and LadaBERT (Mao et al., 2020) with compression factors in the rage of 10-20x.
4.2 Results on the GLUE Benchmark
We evaluated the proposed framework on the General Language Understanding Evaluation (GLUE) (Wang et al., 2018) benchmark which consists of 9 natural language understanding tasks. We submitted the predictions of our proposed models on the test data sets for different tasks to the official GLUE benchmark (https://gluebenchmark.com/). Table 2 summarizes the results on GLUE test set for compression factors less than 10. We can see that KroneckerBERT $_8$  outperforms other baselines in the majority of tasks as well as on average. Moreover, the average performance of KroneckerBERT $_8$  excluding CoLA is 82.4 which is only  $0.5\%$  less than that of the teacher.
Table 3 shows the results for extreme compression on the GLUE test set. As indicated in the table, the baselines for the higher compression factors only provided results on a limited set of GLUE tasks. We can see that for higher compression factors, Kronecker  $\mathrm{BERT}_{21}$  outperforms the baselines on all available results.
In table 4 we compare the performance of KroneckerBERT with the dynaBERT model (Hou et al., 2020). We compare the results on dev set since the results on test set were not provided in their paper. We see that KroneckerBERT can outperform dynaBERT with fewer number of parameters on all GLUE tasks.
Model	Params	MNLI-(m/mm)	SST-2	MRPC	CoLA	QQP	QNLI	RTE	STS-B	Avg
KroneckerBERT5	20M	82.8/83.5	91.2	87.2	49.5	91.1	89.4	68.9	88.1	81.3
KroneckerBERT8	14.3M	82.8/83.5	91.1	87.5	43.6	90.9	90.7	69.66	88.0	80.9
DynaBERT	27M	83/83.6	91.6	83.1	48.5	91.0	90.0	67.9	88.2	80.8
Table 4: Results on the dev set of the GLUE. The results for DynaBERT are taken from (Hou et al., 2020)
Model	CMP Factor	SQuAD1.1	SQuAD2.0
EM	F1	EM	F1
BERTBASE	1×	80.5	88	74.5	77.7
BERT4-PKD	2.1×	70.1	79.5	60.8	64.6
TinyBERT	7.5×	72.7	82.1	68.2	71.8
KroneckerBERT8	7.8×	78.1	86.3	70.4	73.8
KroneckerBERT21	21×	70.7	80.5	66.9	69.3
Table 5: Results of the baselines and KroneckerBERT on question SQuAD dev dataset. The results of the baselines are taken from (Jiao et al., 2019).
Pre-training	Fine-tuning	MNLI-m (393k)	SST-2 (67k)	MRPC (3.7k)
w KD	w KD	82.8	91.0	87.5
None	w KD	80.7	86.6	70.8
w KD	w/o KD	80.0	88.8	86.5
Table 6: Ablation study of the effect pretraining and KD in the fine-tuning stage. The results show the performance of the KroneckerBERT $_8$  on GLUE dev. w and w/o denote with and without, respectively.
4.3 Results on SQuAD
In this section, we evaluate the performance of the proposed model on SQuAD datasets. SQuAD1.1 (Rajpurkar et al., 2016) is a large-scale reading comprehension which contains questions that have answers in given context. SQuAD2.0 (Kudo and Richardson, 2018) also contains unanswerable questions. Table 5 summarises the performance on dev set. For both SQuAD1.1 and SQuAD2.0, KroneckerBERT $_8$  with fewer number of parameters can significantly outperform both TinyBERT and  $\mathrm{BERT}_4$ -PKD baselines. We have also listed the performance of KroneckerBERT $_{21}$ . The results of baselines with higher compression factors on SQuAD were not available.
In this section, we investigate the effect of pretraining and KD in reducing the gap between the original  $\mathrm{BERT}_{\mathrm{BASE}}$  model and the compressed KroneckerBERT. Table 6 summarises the results for KroneckerBERT8. Our proposed method uses KD in both the pre-training and the fine-tuning stages. For this ablation study, pre-training is only performed via KD with the pre-trained  $\mathrm{BERT}_{\mathrm{BASE}}$  as
Figure 4: T-SNE visualization of the output of the middle Transformer layer of the fine-tuned models on SST-2 dev. Left: Fine-tuned  $\mathrm{BERT}_{\mathrm{BASE}}$ , middle: Kronecker  $\mathrm{BERT}_8$  fine-tuned without KD, right: Kronecker  $\mathrm{BERT}_8$  when trained using KD in two stages. The colours indicate the positive and negative classes.
the teacher. We perform experiments on 3 tasks from the GLUE benchmark with different sizes of training data, namely MNLI-m, SST-2, and MRPC. For all tasks, the highest performance is obtained when the two-stage KD is used (first row). Note that our light pretraining plays an important role in improving the performance as shown in the first and the second row (with and without pretraining respectively). As the size of the task dataset decreases the effect of pretraining becomes more significant. Also, removing KD from the fine-tuning stage (task-agnostic compression) leads to an accuracy drop on all task. However, the drop is not as pronounced as removing the pretraining stage. It seems that KD in the fine-tuning stage has a larger impact on tasks with larger datasets.
We also used t-SNE to visualize the output of the FFN of the middle layer (layer 6) of the finetuned KroneckerBERT $_8$  with and without KD in comparison with the fine-tuned teacher, on SST-2 dev. Figure 4 shows the results. See how KD helps the features of the middle layer to be more separable with respect to the task compared to the no KD case.
We introduced a novel method for compressing Transformer-based language models that uses Kronecker decomposition for the compression of the embedding layer and the linear mappings within the Transformer blocks. The proposed framework
was used to compress the  $\mathrm{BERT}_{\mathrm{BASE}}$  model. We used a very light two-stage KD method to train the compressed model. We show that the proposed framework can significantly reduce the size and the number of computations while outperforming state-of-the-art. The proposed method can be directly applied for compression of other Transformer-based language models. The combination of the proposed method with other compression techniques such layer truncation, pruning and quantization can be an interesting direction for future work.
Authors would like to thank Mahdi Zolnouri, Seyed Alireza Ghaffari and Eyyüb Sari for informative discussions throughout this project. We also would like to thank Aref Jaffari for preparing the out of domain datasets.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In TAC.
Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. 2018. Quora question pairs. University of Waterloo.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115.
Alexander Graham. 2018. Kronecker products and matrix calculus with applications. Courier Dover Publications.
Song Han, Huizi Mao, and William J Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve out-of-distribution robustness. arXiv preprint arXiv:2004.06100.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Lu Hou, Lifeng Shang, Xin Jiang, and Qun Liu. 2020. Dynabert: Dynamic bert with adaptive width and depth. arXiv preprint arXiv:2004.04037.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351.
Valentin Khrulkov, Oleksii Hrinchuk, Leyla Mirvakhabova, and Ivan Oseledets. 2019. Tensorized embedding layers for efficient model compression. arXiv preprint arXiv:1901.10787.
Sehoon Kim, Amir Gholami, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. 2021. I-bert: Integer-only bert quantization. arXiv preprint arXiv:2101.01321.
Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and tokenizer for neural text processing. arXiv preprint arXiv:1808.06226.
Zhongliang Li, Raymond Kulhanek, Shaojun Wang, Yunxin Zhao, and Shuang Wu. 2018. Slim embedding layers for recurrent neural language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Vasileios Lioutas, Ahmad Rashid, Krtin Kumar, Md Akmal Haidar, and Mehdi Rezagholizadeh. 2020. Improving word embedding factorization for compression using distilled nonlinear neural decomposition. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 2774-2784.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Yihuan Mao, Yujing Wang, Chufan Wu, Chen Zhang, Yang Wang, Yaming Yang, Quanlu Zhang, Yunhai Tong, and Jing Bai. 2020. Ladabert: Lightweight adaptation of bert through hybrid model compression. arXiv preprint arXiv:2004.04124.
Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2020. On the stability of fine-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprint arXiv:2006.04884.
Matan Ben Noach and Yoav Goldberg. 2020. Compressing pre-trained language models by matrix decomposition. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 884-889.
Gabriele Prato, Ella Charlaix, and Mehdi Rezagholizadeh. 2019. Fully quantized transformer for machine translation. arXiv preprint arXiv:1910.10485.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, page arXiv:1606.05250.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984.
Sridhar Swaminathan, Deepak Garg, Rajkumar Kannan, and Frederic Andres. 2020. Sparse low rank factorization for deep neural network compression. Neurocomputing, 398:185-196.
Urmish Thakker, Jesse Beu, Dibakar Gope, Chu Zhou, Igor Fedorov, Ganesh Dasika, and Matthew Mattina. 2019. Compressing rnns for IoT devices by 15-38x using kronecker products. arXiv preprint arXiv:1906.02876.
Urmish Thakker, Paul Whatamough, Matthew Mattina, and Jesse Beu. 2020. Compressing language models using doped kronecker products. arXiv preprint arXiv:2001.08896.
Charles F Van Loan and Nikos Pitsianis. 1993. Approximation with kronecker products. In Linear algebra for large scale and real-time applications, pages 293-314. Springer.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426.
Genta Indra Winata, Andrea Madotto, Jamin Shin, Elham J Barezi, and Pascale Fung. 2019. On the effectiveness of low-rank matrix factorization for LSTM model compression. arXiv preprint arXiv:1908.09982.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. 2020. Bert-of-theseus: Compressing bert by progressive module replacing.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. 2017. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7370-7379.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2020. Revisiting few-sample bert fine-tuning. arXiv preprint arXiv:2006.05987.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130.
Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2019a. Extreme language model compression with optimal subwords and shared projections. arXiv preprint arXiv:1909.11687.
Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2019b. Extremely small bert models from mixed-vocabulary training. arXiv preprint arXiv:1909.11687.
Sanqiang Zhao, Raghav Gupta, Yang Song, and Denny Zhou. 2021. Extremely small bert models from mixed-vocabulary training. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2753-2759.
Shuchang Zhou and Jia-Nan Wu. 2015. Compression of fully-connected layer in neural network by kronecker product. arXiv preprint arXiv:1507.05775.
In this section, we include more details of our experimental settings presented in Section 4 of the paper. For optimization, we used BERTadam and searched learning rate in range  $\{5e - 5,e - 4,5e - 4,\}$ . For pretraining the learning rate was set to 1e-13 and the number of epochs was set to 3. The batch size in all experiments were set to 32. For the GLUE benchmark, We searched epochs in range  $\{5 - 15\}$  for all tasks except CoLA. For CoLA we searched epochs in range  $\{15 - 30\}$ . This is because similar to other studies (Mosbach et al., 2020; Zhang et al., 2020) we noticed that running CoLA for more epochs is necessary to reduce its sensitivity to random seed. The sequence length at the pre-training stage is set to 512 and at the fine-tuning stage is set to 128 for GLUE benchmark. For SQuAD1.1 and SQuAD2 the sequence length is set 384 and the batch size was set to 64 and the epochs were varied in the range  $\{1 - 14\}$ . Also, the learning rate was set to e-4 and
Table 7 shows the result of the best-performing models on dev set for KroneckerBERT $_{21}$ . Tables 8 shows the learning rate for the best-performing models. The training was performed on V100 GPU and the average latency for training of KroneckerBERT $_{21}$  for a batch size of 64 was 32ms. All the values are the results of single runs.
A.2 Out of domain robustness
It is shown that pre-trained Transformer-based language models are robust to out-of-domain (OOD) samples (Hendrycks et al., 2020). In this section, we investigate how the proposed compression
Figure 5: Geometrical interpretation of projecting a matrix product onto a Kronecker product. The angle of projection is defined by the size of  $\mathbf{A}$  and  $\mathbf{B}$ .
method affects the OOD robustness of BERT by evaluating the fined-tuned models on MRPC and SST-2 on PAWS (Zhang et al., 2019) and IMDb (Maas et al., 2011) respectively. We compare OOD robustness with the teacher,  $\mathrm{BERT}_{\mathrm{BASE}}$  and TinyBERT. TinyBERT fine-tuned checkpoints are obtained from their repository. Table 9 lists the results. KroencekrBERT $_8$  outperforms TinyBERT on two of the three OOD experiments. We can see the fine-tuned KroneckerBERT $_8$  models on MRPC is robust to OOD since there is a small increase in performance compared to  $\mathrm{BERT}_{\mathrm{BASE}}$ . On IMDb our KroenckerBERT $_8$  has a small drop in accuracy (1.5% compared to 9.5% for TinyBert) after compression.
A.3 Training efficiency
Table 10 shows the training requirements of different compression methods in terms of their training data. Some models require pretraining a designed teacher from scratch before pretraining the student. KroneckerBERT however only pretrain on  $10\%$  of Wikipedia for 3 epochs. For fine-tuning in contrast to TinyBERT our KroneckerBERT model is trained for on the original data. The number of fine-tuning epochs for the majority of the GLUE tasks is less than 15.
A.4 Geometrical interpretation of Kronecker product projection
Figure 5 shows a geometrical interpretation of Kronecker product projection versus the original linear projection. It shows how Kronecker product constraints the space of possible projections. The flexibility of this space is a function of the shape of the core matrices  $\mathbf{A}$  and  $\mathbf{B}$ .
Model	MNLI-(m/mm)	SST-2	MRPC	CoLA	QQP	QNLI	RTE	STS-B	Average
KroenckerBERT21	80.6/80.6	88.9	86.2	34	90	87.2	66.4	86.4	77.8
Table 7: The results of the best-performing models on GLUE dev.
Hyperparameter	MNLI-m	MNLI-mm	SST-2	MRPC	CoLA	QQP	QNLI	RTE	STS-B
KroneckerBERT8	e-4	e-4	e-4	e-4	5e-5	e-4	e-4	1e-5	e-4
KroneckerBERT21	5e-4	5e-4	5e-4	5e-4	5e-4	e-4	5e-4	5e-5	5e-4
Table 8: The hyper-parameters for the KroneckerBERT models
Model	MRPC → PAWS	SST-2 → IMDb	RTE → HANS
BERTBASE	61.3	88.0	50.7
TinyBERT	61.3	78.5	51.2
KroneckerBERT8	61.4	86.5	50.4
Table 9: The results of out of distribution experiment. Fined-tuned models on MRPC and SST-2 are evaluated on the dev sets of PAWS and IMDb respectively.
Model	Pretraining a specific teacher	Pretraining student	Fine-tuning student
MobileBert	IB-BERTLARGE on EW+BC	EW+BC	task data
Shared project	BERTLARGE with mixed vocabulary on EW+BC	EW+BC	task data
TinyBert	None	EW	task data + Data Augmentation(20x)
KroneckerBERT	None	10% EW	task data
Table 10: Sample efficiency during training in various methods. EW and BC denotes English Wikipedia and Book corpus with 2.5B and 800M words respectively.
A.5 Datasets