SlowGuess/NAACL_2022_OCR huggingface dataset:
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. *ArXiv* preprint, abs/1910.01108.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2019. Green ai. ArXiv preprint, abs/1907.10597.
Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. 2021. Is label smoothing truly incompatible with knowledge distillation: An empirical study. arXiv preprint arXiv:2104.00676.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. ArXiv preprint, abs/1909.08053.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4323-4332, Hong Kong, China. Association for Computational Linguistics.
Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. 2020. Contrastive distillation on intermediate representations for language model compression. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 498-508, Online. Association for Computational Linguistics.
Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L Yuille. 2019. Training deep neural networks in generations: A more tolerant teacher educates better students. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5628-5635.
Yang You, Jing Li, Jonathan Hseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. 2019. Reducing bert pre-training time from 3 days to 76 minutes. ArXiv preprint, abs/1904.00962.
Yang You, Jing Li, Sashank J. Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training BERT in 76 minutes. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Ji-ashi Feng. 2020. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3903-3911.
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 19-27. IEEE Computer Society.
A.1 Effects of Model Size
We experiment on four PLMs with roughly 1.7x growth in model size:  $\mathcal{M}_1$  (RoBERTaMEDIUM, 73.5M),  $\mathcal{M}_2$  (RoBERTaBASE, 125M),  $\mathcal{M}_3$  (RoBERTaBASE_PLUS, 211M) and  $\mathcal{M}_4$  (RoBERTaLarge, 355M), whose architectures are listed in Table 6. We first pre-train a teacher PLM  $\mathcal{M}_i$  ( $\mathcal{M}_S$ ) for 125k steps with a batch size of 2,048 under the same setting then train a larger one  $\mathcal{M}_{i+1}$  ( $\mathcal{M}_L$ ) by inheriting  $\mathcal{M}_i$ 's knowledge under KI framework (denoted as  $\mathcal{M}_i \to \mathcal{M}_{i+1}, i \in \{1,2,3\}$ ). We compare  $\mathcal{M}_i \to \mathcal{M}_{i+1}$  with  $\mathcal{M}_{i+1}$  that conducts self-learning from beginning to end. As shown in Figure 4, the superiority of KI is observed across all models. In addition, with the overall model size of  $\mathcal{M}_S$  and  $\mathcal{M}_L$  gradually increasing, the benefits of KI become more evident, reflected in the broader absolute gap between the PPL curve of  $\mathcal{M}_i \to \mathcal{M}_{i+1}$  and  $\mathcal{M}_{i+1}$  when  $i$  gradually grows. This implies that with the advance of computing power in future, training larger PLMs will benefit more and more from our KI framework.
A.2 Effects of  $\mathcal{M}_S$ 's Pre-training Steps
Longer pre-training has been demonstrated as an effective way for PLMs to achieve better performance (Liu et al., 2019) and thus become more knowledgeable. To evaluate the benefits of more pre-training steps for  $\mathcal{M}_S$ , we first vary RoBERTa $_{\text{MEDIUM}}$ s pre-training steps in  $\{62.5\mathrm{k}, 125\mathrm{k}, 250\mathrm{k}, 500\mathrm{k}\}$ , and keep all other settings the same. After pre-training, these teacher models achieve the final validation PPL of  $\{5.25, 4.92, 4.72, 4.51\}$ , respectively. Then we compare the performances when RoBERTa $_{\text{BASE}}$  learn from these teacher models and visualize the results in Figure 4, from which we can conclude that, inheriting knowledge from teachers with longer pre-training time (steps) helps  $\mathcal{M}_L$  converge faster. However, such a benefit is less and less obvious as  $\mathcal{M}_S$ s pre-training steps increase, which means after enough training computations are invested, the teacher model enters a plateau of convergence in validation PPL, and digging deeper in knowledge becomes even harder. The bottleneck lies in other factors, e.g., the size and diversity of pre-training data, which hinder  $\mathcal{M}_S$  from becoming
ing more knowledgeable. We also found empirically that, after being pre-trained for  $125\mathrm{k}$  steps on the corpus with a batch size of 2,048, all the models used in this paper have well converged, and longer pre-training only results in limited performance gain in either PPL or downstream performance.
A.3 Effects of  $\mathcal{M}_L$  's Batch Size
Batch size is highly related to PLM's training efficiency, and previous work (Liu et al., 2019; Li et al., 2020b; You et al., 2019) found that slow-but-accurate large batch sizes can bring improvements to model training, although the improvements become marginal after increasing the batch size beyond a certain point (around 2, 048). BERT (Devlin et al., 2019) is pre-trained for 1,000k steps with a batch size of 256, and the computational cost is equivalent to training for 125k steps with a batch size of 2,048 (Liu et al., 2019), which is the pretraining setting chosen in our main paper. Choosing RoBERTa<sub>MEDIUM</sub> as the teacher model and RoBERTa<sub>BASE</sub> as the student model, in Figure 4 we compare the validation PPL as we vary the batch size in {256, 512, 1024, 2, 048}, controlling for the number of passes through the pre-training corpus. We also vary the peak learning rate in {1.0 × 10<sup>-4</sup>, 2.5 × 10<sup>-4</sup>, 3.8 × 10<sup>-4</sup>, 5.0 × 10<sup>-4</sup>} and pretrain for {1,000k, 500k, 250k, 125k} steps, respectively, when increasing the batch size. We observe that increasing the batch size results in improved final validation PPL, which is aligned with previous findings (Liu et al., 2019). When adjusting batch size, KI accelerates the convergence unanimously, and its benefits become more evident when training with a smaller batch size, reflected in the absolute improvement in final validation PPL. We hypothesize that this is because learning from the smoothed target probability of KI, containing rich secondary information (Yang et al., 2019) or dark knowledge (Furlanello et al., 2018), makes the pretraining process more stable. The student PLM is prevented from fitting to unnecessarily strict distributions and can thus learn faster.
A.4 Additional Experiments of the Effects of Teacher Model  $\mathcal{M}_S$ 's architecture (width)
We show in Figure 5 the validation PPL of  $\mathcal{M}_L$  when choosing the teacher PLM  $\mathcal{M}_S$  with different hidden sizes ( $\{384, 480, 576, 672\}$ ). As mentioned in our main paper, choosing a wider teacher
Figure 4: Left: effects of  $\mathcal{M}_L$ 's model size. Middle: effects of  $\mathcal{M}_S$ 's number of pre-training steps. Right: effects of  $\mathcal{M}_L$ 's batch size.
Figure 5: Left: the PPL curve when choosing the teacher PLM with different hidden sizes. Middle & Right: adapting RoBERTa<sub>BASE_WB</sub> to CS (middle) / BIO (right) domain with different number of training steps on different sizes of domain data. We compare two strategies: self-learning and KI. For example, RoBERTa<sub>CS_3400M</sub> denotes post-training RoBERTa<sub>BASE_WB</sub> with the self-learning strategy on the 3,400M token CS domain corpus. RoBERTa<sub>BASE_WB→CS_3400M</sub> denotes post-training RoBERTa<sub>BASE_WB</sub> with the KI strategy on the 3,400M token CS domain corpus.
model improves the training efficiency of the student PLM.
A.5 Additional Experiments of Knowledge Inheritance for Domain Adaptation
Domain	Strategy	3,400M	200M	100M	40M
CS	SL	6.71	7.01	7.39	8.77
KI	8.63	9.39	9.48	9.87
BIO	SL	7.29	6.61	8.16	10.34
KI	10.74	10.78	10.93	11.66
Table 4: The validation PPL on the source domain (WB) after RoBERTa<sub>BASE_WB</sub> is post-trained on the target domain (CS / BIO) with self-learning (SL) and knowledge inheritance (KI).
Different Number of Post-training Steps. In the main paper, we adapt RoBERTa<sub>BASE_WB</sub> to either CS or BIO domain by post-training it for 4k steps. We further vary the number of training steps in {1k, 2k, 3k, 4k, 5k} and visualize the validation PPL in Figure 5. We also experiment on different sizes of domain corpus, i.e., 3, 400M, 200M, 100M, 40M tokens, respectively, as done in the main pa
per. We observe that generally the validation PPL on each domain decreases with the training step growing, and the performance of KI is always better than self-learning. The improvement of KI over self-learning is further enlarged when there is less target domain data available, demonstrating that KI is more data-efficient and can work well in low-resource settings. In addition, self-learning exhibits overfitting problems when the data size of the target domain is relatively small, which is not observed under our KI framework, which means KI can mitigate overfitting under low-resource settings.
Catastrophic Forgetting on the Source Domain. Table 4 lists the validation PPL on the source domain (WB) after RoBERTa<sub>BASE_WB</sub> is post-trained on the target domain (CS / BIO) with self-learning (SL) and knowledge inheritance (KI) for 4k steps. We show the results w.r.t. different sizes of domain corpus (3, 400M, 200M, 100M and 40M tokens). We observe that after domain adaptation, the validation PPL on the source domain increases, which means PLMs may forget some key knowledge on the source domain when learning new knowledge
Figure 6: Downstream performance visualization on six GLUE tasks comparing RoBERTa  $\mathsf{LARGE}$  and RoBERTa  $\mathsf{BASE} \rightarrow$  RoBERTa  $\mathsf{LARGE}$ . For CoLA, RTE, SST-2 and STS-B, we repeat fine-tuning for 5 times; for MNLI and QNLI, we repeat fine-tuning for 3 times.
in the target domain, i.e., the catastrophic forgetting problem. In addition, we find that the problem is more evident for KI than self-learning. We expect future work to further explore how to mitigate the catastrophic forgetting.
A.6 Detailed Downstream Performances on GLUE Tasks
Figure 6 visualizes the downstream performance of RoBERTaLARGE and RoBERTaBASE  $\rightarrow$  RoBERTaLARGE on the dev sets of six GLUE tasks at different pre-training steps with an interval of 5k. It can be observed that the downstream performance of RoBERTaBASE  $\rightarrow$  RoBERTaLARGE rises faster than the baseline, which means it takes fewer pre-training steps for our KI framework to get a high score in downstream tasks. Aligned with previous findings (Li et al., 2020b), we found MNLI and SST-2 to be the most stable tasks in GLUE, whose variances are lower.
We also list the average GLUE performance for  $\mathrm{RoBERTa_{BASE}}\rightarrow \mathrm{RoBERTa}_{\mathrm{LARGE}}$  and the baseline  $\mathrm{RoBERTa}_{\mathrm{LARGE}}$  in Table 5, from which we observe that the baseline at  $70\mathrm{k}$  -th step achieves almost the same GLUE performance as our method at  $40\mathrm{k}$  -th step, which means our framework saves around  $42.9\%$  FLOPs, much higher than the reported  $27.3\%$  FLOPs saved based on the pretraining PPL metric in the main paper. In addition,
our method achieves almost the same GLUE performance as the baseline at the final step (125k) with only 70k steps, which means our framework saves  $44\%$  FLOPs in total. Both the perplexity in the pre-training stage and performance in downstream tasks can be chosen as the evaluation metric for measuring the computational cost savings. However, in this paper, we choose the former because it is more stable and accurate than the latter. We find empirically that some GLUE tasks like CoLA have higher variances than others, which might make the measurement inaccurate.
Besides, when discussing the effects of model architectures in the main paper, we only show the validation PPL of each model during pre-training, we visualize the corresponding downstream performance (MNLI) in Figure 7, from which it can be observed that learning from teacher models with more parameters helps achieve better downstream performance at the same pre-training step. In general, we observe that, under our setting, the performance gain in downstream tasks is aligned with that reflected in validation PPL during pre-training.
A.7 Teacher Models' Validation PPL Curves during Pre-training for "Effects of Model Architecture"
Figure 7 visualizes the validation PPL curves for all the teacher models used in the experiments
Figure 7: Left & Middle: downstream performances corresponding to the experiments on effects of  $\mathcal{M}_S$ 's model architecture (width (left) & depth (middle)). Right: validation PPL during pre-training for the teacher models used in experiments of effects of teacher model architecture.
Step	RoBERTaBASE	RoBERTaBASE → RoBERTaLARGE
5k	61.8	68.8
10k	75.6	78.1
15k	79.3	81.5
20k	80.4	82.8
25k	81.7	83.6
30k	82.4	83.9
35k	83.1	84.1
40k	83.6	84.5
45k	82.8	85.2
50k	83.9	84.6
55k	83.4	85.2
60k	84.0	85.7
65k	84.1	85.3
70k	84.3	85.5
75k	85.0	85.8
80k	84.7	85.8
85k	84.8	86.2
...	...	...
125k	85.5	86.1
Table 5: Average GLUE performance comparing both RoBERTa<sub>BASE</sub> and RoBERTa<sub>BASE</sub> → RoBERTa<sub>LARGE</sub> at different pre-training steps.
on the effects of model architecture. The teacher models differ from RoBERTa<sub>BASE</sub> in either the depth or width. Specifically, we vary the depth in {4,6,8,10} (denoted as {RoBERTa<sub>H_4</sub>, RoBERTa<sub>H_6</sub>, RoBERTa<sub>H_8</sub>, RoBERTa<sub>H_10</sub>}), and the width in {384,480,576,672} (denoted as {RoBERTa<sub>D_384</sub>, RoBERTa<sub>D_480</sub>, RoBERTa<sub>D_576</sub>, RoBERTa<sub>D_672</sub>}). Generally, PLMs with larger model parameters converge faster and achieve better final performance.
B Pre-training Hyper-parameters
In Table 6, we list the architectures we used for all models, covering the details for the total number of trainable parameters ( $n_{\mathrm{params}}$ ), the total number of layers ( $n_{\mathrm{layers}}$ ), the number of units in each bottleneck layer ( $d_{\mathrm{model}}$ ), the total number of attention
heads ( $n_{\mathrm{heads}}$ ), the inner hidden size of FFN layer ( $d_{\mathrm{FFN}}$ ) and the learning rate when batch size is set to 2, 048 (lr). The training-validation ratio of pretraining data is set to  $199:1$ . We set the weight decay to 0.01, dropout rate to 0.1, and use linear learning rate decay. Adam is chosen as the optimizer. The learning rate is warmed up for the first  $10\%$  steps. The hyper-parameters for Adam optimizer is set to  $1 \times 10^{-6}$ , 0.9, 0.98 for  $\epsilon, \beta_1, \beta_2$ , respectively. For a fair comparison, all experiments are done in the same computation environment with 8 NVIDIA 32GB V100 GPUs. Table 7 describes the total number of pre-training steps for each  $(\mathcal{M}_L, \mathcal{M}_S)$  pair chosen in our experiments.
C Fine-tuning Hyper-parameters
Table 8 describes the hyper-parameters for ACLARC, CHEMPROT and GLUE tasks. The selection of these hyper-parameters closely follows (Liu et al., 2019) and (Gururangan et al., 2020).
D Domain Proximity of WB, CS and BIO
Table 9 lists the domain proximity (vocabulary overlap) of WB, CS and BIO used in this paper.
E Comparison between Knowledge Inheritance and Parameter Recycling
Parameter recycling (i.e., progressive training) first trains a small PLM, and then gradually increases the depth or width of the network based on parameter initialization. It is an orthogonal research direction against our KI, and has many limitations as follows:
Architecture Mismatch. Existing parameter recycling methods (Gong et al., 2019; Gu et al., 2021) require that the architectures of both small PLMs
Model Name	nparams	nlayers	dmodel	nhedges	dFFN	lr (bs = 2,048)
RoBERTaMEDIUM	74M	9	576	12	3072	5.0 × 10-4
RoBERTaD_d	-	12	d	12	3072	5.0 × 10-4
RoBERTaH_h	-	h	768	12	3072	5.0 × 10-4
RoBERTaBASE	125M	12	768	12	3072	5.0 × 10-4
RoBERTaBASE_PLUS	211M	18	864	12	3600	3.5 × 10-4
RoBERTaLARGE	355M	24	1024	16	4096	2.5 × 10-4
GPT73M	73M	9	576	12	3072	5.0 × 10-4
GPT124M	124M	12	768	12	3072	5.0 × 10-4
GPT209M	209M	18	864	12	3600	4.0 × 10-4
GPT354M	354M	24	1024	16	4096	3.5 × 10-4
GPT773M	773M	36	1280	20	5120	3.0 × 10-4
GPT1B	1068M	40	1440	20	5760	2.5 × 10-4
Table 6: Model architectures for all the models we used in this paper.
ML	MS	Steps of teacher-guided learning
RoBERTaBASE	RoBERTaMEDIUM	35k
RoBERTaD_384	28k
RoBERTaD_480	40k
RoBERTaD_576	70k
RoBERTaD_672	85k
RoBERTaH_4	22k
RoBERTaH_6	35k
RoBERTaH_8	55k
RoBERTaH_10	65k
RoBERTaBASE_PLUS	RoBERTaBASE	55k
RoBERTaLARGE	RoBERTaBASE	40k
RoBERTaBASE_PLUS	65k
RoBERTaBASE → RoBERTaBASE_PLUS	75k
GPT124M	GPT73M	10k
GPT209M	GPT124M	15k
GPT354M	GPT209M	18k
GPT773M	GPT354M	16k
GPT1B	GPT773M	20k
Table 7: The total number of steps for teacher-guided learning for different  $(\mathcal{M}_L,\mathcal{M}_S)$  pairs.
and large PLMs are matched to some extent, however, our KI does not have such a requirement. For example, Gong et al. (2019); Gu et al. (2021) either requires the number of layers, or the hidden size/embedding size of a large PLM to be the integer multiples of that of a small PLM. Hence, it is not flexible to train larger PLMs with arbitrary architectures, making parameter recycling hard to be implemented practically. Besides, there are more and more advanced non-trivial Transformer modifications appearing (we refer to Lin et al. (2021) for details), e.g., pre-normalization, relative embedding, sparse attention, etc. It is non-trivial to directly transfer the parameters between two PLMs if they have different inner structures. Nevertheless, our KI framework will not be influenced by such
architectural mismatches.
Inability for Multi-to-one Knowledge Inheritance. It is non-trivial to support absorbing knowledge from multiple teacher models by jointly recycling their model parameters. Instead, it is easy to implement for KI. As shown in our experiments, we demonstrate that under our framework, large PLMs can simultaneously absorb knowledge from multiple teachers.
Inability of Knowledge Inheritance for Domain Adaptation. Parameter recycling is hard to support continual learning, which makes large PLMs absorb knowledge from small ones in a lifelong manner. In real-world scenarios, numerous PLMs of different architectures are trained locally with
HyperParam	ACL-ARC & CHEMPROT	GLUE
Learning Rate	2 × 10-5	{1 × 10-5, 2 × 10-5, 3 × 10-5}
Batch Size	256	{16, 32}
Weight Decay	0.1	0.1
Max Epochs	10	10
Learning Rate Decay	Linear	Linear
Warmup Ratio	0.06	0.06
Table 8: Hyper-parameters for fine-tuning RoBERTa on ACL-ARC, CHEMPROT and GLUE.
WB	CS	BIO
WB	100%	19.1%	25.6%
CS	19.1%	100%	22.5%
BIO	25.6%	22.5%	100%
different data. These small PLMs can be seen as domain experts, and it is essential that larger PLMs can continuously benefit from these existing PLMs efficiently by incorporating their knowledge so that larger PLMs can become omnipotent. As described before, it is easy to implement for our framework and we have demonstrated the effectiveness.
Model Privacy. Parameter recycling requires the availability of the parameters of an existing PLM, which may be impractical due to some privacy issues, e.g., GPT-3 only provides API access for prediction instead of the model parameters. Instead, our KI framework does not presume access to an existing model parameter since the predictions of the small model can be pre-computed and saved offline. This superiority will further make it possible for API-based online knowledge transfer.
F Comparing Label Smoothing and Knowledge Inheritance
Previous work shows the relation between label smoothing and knowledge distillation to some extent (Shen et al., 2021). To demonstrate that the success of our KI is not because of learning from a more smoothed target, we conduct experiments comparing both label smoothing and our KI in Table 10. Specifically, for label smoothing, PLMs optimize a smoothed target  $\mathbf{y}_i^S = (1 - \alpha) * \mathbf{y}_i + \alpha * \vec{1} / (K - 1)$ , where  $\alpha = 0$  denotes learning from scratch with no label smoothing, larger  $\alpha$  means a more smoothed target for PLMs to learn
Table 9: Domain proximity (vocabulary overlap) among three domains (WB, CS, BIO) discussed in this paper. Following (Gururangan et al., 2020), we create the vocabulary for each domain by considering the top 10k most frequent words (excluding stopwords).
Step	20k	40k	60k	80k	100k
α = 0.3	8.68	7.29	6.90	6.57	6.26
α = 0.2	7.27	6.47	5.95	5.68	5.46
α = 0.1	6.71	5.74	5.35	5.06	4.86
α = 0	6.13	5.21	4.83	4.57	4.36
KI	5.69	5.17	4.78	4.52	4.32
Table 10: Validation PPL for training RoBERTa<sub>BASE</sub> with different strategies. KI denotes our knowledge inheritance framework, where RoBERTa<sub>MEDIUM</sub> is chosen as the teacher.
from,  $K$  denotes the vocabulary size. Specifically, we choose  $\alpha$  from  $\{0.1, 0.2, 0.3\}$ . It can be concluded from the results in Table 10 that adding label smoothing into the pre-training objectives of PLMs leads to far worse performance than the vanilla baseline, which shows that the improvements of our knowledge inheritance framework are non-trivial: larger PLMs are indeed inheriting the "knowledge" from smaller ones, instead of benefiting from optimizing a smoothed target, which imposes regularization.