Venue: NAACL 2022

# Knowledge Inheritance for Pre-trained Language Models

 Yujia Qin\({}^{1,2,3}\), Yankai Lin\({}^{4}\), Jing Yi\({}^{1,2,3}\), Jiaje Zhang\({}^{1,2,3}\), Xu Han\({}^{1,2,3}\), Zhengyan Zhang\({}^{1,2,3}\), Yusheng Su\({}^{1,2,3}\), Zhiyuan Liu\({}^{1,2,3,5,6}\); Peng Li\({}^{7}\)+

Maosong Sun\({}^{1,2,3,5}\)*, Jie Zhou\({}^{4}\)

Corresponding author.Part of the work was done while Peng Li was working at Tencent. ++Corresponding author. ++Department of Computer Science and Technology, Tsinghua University, Beijing, China

Footnote †: Part of the work was done while Peng Li was working at Tencent.

###### Abstract

Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PLMs. However, it requires tremendous computational resources to train a large-scale PLM, which may be practically unaffordable. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring that many well-trained PLMs are available. To this end, we explore the question how could existing PLMs benefit training large-scale PLMs in future. Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs. Experimental results demonstrate the superiority of KI in training efficiency. We also conduct empirical analyses to explore the effects of teacher PLMs' pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI could be applied to domain adaptation and knowledge transfer. The implementation is publicly available at [https://github.com/thunlp/Knowledge-Inheritance](https://github.com/thunlp/Knowledge-Inheritance).

## 1 Introduction

Recently, it has become a consensus in the NLP community to use pre-trained language models (PLMs) as the backbone for various downstream tasks [9, 8]. Despite the great follow-up efforts of exploring various pre-training techniques and model architectures, researchers find that simply enlarging the model capacity, data size and training steps can further improve the performance of PLMs [10, 11]. This discovery sets off a wave of training large-scale PLMs [17, 18, 19].

Although huge PLMs have shown awesome performance [16], it requires tremendous computational resources to train large-scale PLMs [15], raising severe environmental concerns on the prohibitive computational costs. Moreover, existing PLMs are generally trained from scratch individually, ignoring that many well-trained PLMs are available. This leaves us an important question: _how could existing PLMs benefit training larger PLMs in future?_

Considering that humans can leverage the knowledge summarized by their predecessors to learn new tasks, so that the learning process could become efficient; similarly, it is worth inheriting the implicit knowledge distributed in existing PLMs. In this sense, we could distill the knowledge summarized by an existing small PLM during pre-training to efficiently learn larger PLMs. We dub the above process as _knowledge inheritance_ (KI). This intuition is similar to reversed KD [20] in the field of computer vision. They indicate that a delicate student model could still benefit from a teacher with an inferior architecture for a specific downstream task.

However, the success of reversed KD in supervised downstream tasks does not guarantee its feasibility under the scenario of large-scale self-supervised pre-training. Therefore, in this paper,

we strive to answer the following research questions: (_RQ1_) could distilling knowledge from an existing trained PLM benefit large PLMs' training from scratch? (_RQ2_) Considering human beings are able to hand down knowledge from generation to generation, could KI similarly be sequentially performed among a series of PLMs with growing sizes? (_RQ3_) As more and more PLMs with different pre-training settings (model architectures, training data, training strategies, etc) emerge, how would different settings affect the performance of KI? (_RQ4_) Besides training a large PLM from scratch, when adapting an already trained large PLM to a new domain, how could smaller domain teachers benefit such a process?

In conclusion, the contributions of this paper are summarized as follows: (1) we are the first to formulate the problem of knowledge inheritance, and demonstrate the feasibility of inheriting the knowledge from previously trained PLMs for efficiently training larger ones; (2) we show that the learned knowledge in PLMs could accumulate and further be passed down from generation to generation; (3) we systematically conduct empirical analyses to show the effects of various teacher pre-training settings, which may indicate how to select the most appropriate PLM as the teacher for KI; (4) we further show that during domain adaptation, an already trained large PLM could benefit from multiple small PLMs of different domains under the KI framework. The above empirical studies indicate that KI can well support cross-model knowledge transfer, providing a promising direction to share the knowledge learned by different PLMs and continuously promote their performance.

## 2 Related Work

Efficient Pre-training for NLP.Recently, researchers find that the performance of PLMs can be simply improved by increasing the model size, data size and training steps Liu et al. (2019); Raffel et al. (2019); Kaplan et al. (2020), sparking a wave of training ever-larger PLMs. For instance, the revolutionary GPT-3 Brown et al. (2020), which contains \(175\) billion parameters, shows strong capabilities for language understanding and generation. This means that utilizing PLMs with huge parameters for downstream tasks may greatly relieve the cost of manual labeling and model training for new tasks. However, larger models require greater computational demands Patterson et al. (2021). To this end, researchers propose to accelerate pre-training by mixed-precision training Shoeybi et al. (2019), distributed training Shoeybi et al. (2019), large batch optimization You et al. (2020), etc.

Another line of methods Gong et al. (2019); Gu et al. (2021); Chen et al. (2022); Qin et al. (2022) proposes to pre-train larger PLMs progressively. They first train a small PLM, and then gradually increase the depth or width of the network based on parameter recycling (PR). Although PR could be used for the goal of KI, these methods typically have strict requirements on the architectures of both models, which is not flexible for practical uses; instead, we resort to KD as the solution for KI without architecture constraints. In addition, different from KI, PR is not applicable for absorbing knowledge from multiple teacher models and domain adaptation. More detailed comparisons between KI and PR are discussed in appendix E.

Knowledge Distillation for PLMs.Knowledge Distillation (KD Hinton et al. (2015) aims to compress a large model into a fast-to-execute one. KD has renewed a surge of interest in PLMs recently. Some explore KD at different training phases, e.g., pre-training Sanh et al. (2019), downstream finetuning Sun et al. (2019); Krishna et al. (2020), or both of them Jiao et al. (2020); others explore distilling not only the final logits output by the large PLM, but also the intermediate hidden representations Sanh et al. (2019); Jiao et al. (2020); Sun et al. (2020). Conventional KD presumes that teacher models play pivotal roles in mastering knowledge, and student models generally cannot match their teachers in performance. When it comes to the scenario of KI, since student models have larger capacities, the performance of teacher models is no longer an "upper bound" of student models. Outside NLP, researchers recently demonstrate that a student model could also benefit from a poor teacher for a specific downstream task Yuan et al. (2020) (reversed KD). Based on the prior explorations, in this paper, we investigate the application of reversed KD in pre-training.

## 3 Knowledge Inheritance

Task Formulation.Given a textual input \({\bf x}=\{x^{1},\ldots,x^{n}\}\) and the corresponding label \({\bf y}\in\mathbb{R}^{K}\), where \(K\) is the number of classes for the specific pre-training task, e.g., the vocabulary size for masked language modeling (MLM) Devlin et al. (2019), a PLM \(\mathcal{M}\) converts each token 

\(x^{j}\in\mathbf{x}\) to task-specific logits \(\mathbf{z}^{j}=[z_{1}^{j},...,z_{K}^{j}]\). \(\mathbf{z}^{j}\) is then converted to a probability distribution \(\mathcal{P}(x^{j};\tau)=[p_{1}(x^{j};\tau),...,p_{K}(x^{j};\tau)]\) using a softmax function with temperature \(\tau\). \(\mathcal{M}\) is pre-trained with the objective \(\mathcal{L}_{\text{SELF}}(\mathbf{x},\mathbf{y})=\mathcal{H}(\mathbf{y}, \mathcal{P}(\mathbf{x};\tau))\), where \(\mathcal{H}\) is the loss function, e.g., cross-entropy for MLM. Assume that we have a well-trained small PLM \(\mathcal{M}_{S}\) optimized with the self learning objective \(\mathcal{L}_{\text{SELF}}\) (such as MLM), our goal is leveraging \(\mathcal{M}_{S}\)'s knowledge to efficiently train a larger PLM \(\mathcal{M}_{L}\) on the corpora \(\mathcal{D}_{L}=\left\{(\mathbf{x}_{i},\mathbf{y}_{i})\right\}_{i=1}^{| \mathcal{D}_{L}|}\).

**Investigated Methodology.** Specifically, imparting \(\mathcal{M}_{S}\)'s knowledge to \(\mathcal{M}_{L}\) on \(\mathcal{D}_{L}\) is implemented by minimizing the Kullback-Leibler (KL) divergence between two probability distributions output by \(\mathcal{M}_{S}\) and \(\mathcal{M}_{L}\) on the same input \(\mathbf{x}_{i}\in\mathcal{D}_{L}\), i.e., \(\mathcal{L}_{\text{KI}}(\mathbf{x}_{i};\mathcal{M}_{S})=\tau^{2}\text{KL}( \mathcal{P}_{\mathcal{M}_{S}}(\mathbf{x}_{i};\tau)||\mathcal{P}_{\mathcal{ M}_{L}}(\mathbf{x}_{i};\tau)))\). In addition, \(\mathcal{M}_{L}\) is also encouraged to conduct self-learning by optimizing \(\mathcal{L}_{\text{SELF}}(\mathbf{x}_{i},\mathbf{y}_{i})\). Both \(\mathcal{L}_{\text{SELF}}\) and \(\mathcal{L}_{\text{KI}}\) are balanced with an inheritance rate \(\alpha\):

\[\begin{split}\mathcal{L}(\mathcal{D}_{L};\mathcal{M}_{S})& =\!\sum_{(\mathbf{x}_{i},\mathbf{y}_{i})\in\mathcal{D}_{L}}\!\!\!(1- \alpha)\mathcal{L}_{\text{SELF}}(\mathbf{x}_{i},\mathbf{y}_{i})+\alpha \mathcal{L}_{\text{KI}}(\mathbf{x}_{i};\mathcal{M}_{S})\\ &=\!\!\sum_{(\mathbf{x}_{i},\mathbf{y}_{i})\in\mathcal{D}_{L}}\!\!\! \!(1-\alpha)\mathcal{H}(\mathbf{y}_{i},\mathcal{P}_{\mathcal{M}_{L}}(\mathbf{ x}_{i};1))\\ &+\alpha\tau^{2}\text{KL}(\mathcal{P}_{\mathcal{M}_{S}}(\mathbf{ x}_{i};\tau)||\mathcal{P}_{\mathcal{M}_{L}}(\mathbf{x}_{i};\tau))).\end{split} \tag{1}\]

Since larger models generally converge faster and can achieve better final performance Li et al. (2020b), \(\mathcal{M}_{L}\) becomes more and more knowledgeable during the learning process, and would surpass the teacher eventually. Thus, it is necessary to encourage \(\mathcal{M}_{L}\) increasingly learning knowledge on its own, not only following the teacher's instructions. Additionally, after \(\mathcal{M}_{L}\) has surpassed its teacher, it no longer needs the guidance from \(\mathcal{M}_{S}\) and should conduct pure self-learning from then on. Therefore, different from reversed KD, we dynamically change the inheritance rate \(\alpha\). Specifically, for a total training steps of \(T\), we linearly decay \(\alpha_{t}\) with a slope of \(\frac{\alpha_{T}}{T}\). The student only inherits knowledge from the teacher for \(\frac{T}{\alpha_{T}}\) steps, and then conducts pure self-learning, i.e., \(\alpha_{t}=\max(1-\alpha_{T}\times\frac{t}{T},0)\). Formally, at step \(t\), the loss function for inheriting knowledge of \(\mathcal{M}_{S}\) on \(\mathcal{D}_{L}\) is formulated as:

\[\small\mathcal{L}(\mathcal{D}_{L};\mathcal{M}_{S})\!\!\!=\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\!\! \!\! 

each step. Hence, we report the performance w.r.t training step Li et al. (2020), while the performance w.r.t. FLOPs Schwartz et al. (2019) and wall-clock time Li et al. (2020) can be roughly obtained by stretching the figure horizontally.

Overall Results.As shown in Figure 1 (a), we conclude that: (1) **training under KI converges faster than the self-learning baseline**, indicating that inheriting the knowledge from an existing teacher is far more efficient than solely learning such knowledge. That is, to achieve the same level of validation PPL, KI requires fewer computational costs. Specifically, under the guidance of \({\mathcal{M}_{S}}\), whose validation PPL is \(4.18\), \(\mathtt{BASE}\rightarrow\mathtt{LARGE}\) achieves a validation PPL of \(3.41\) at the end of pre-training, compared with baseline (\(\mathtt{LARGE}\)) \(3.58\). After \(\mathtt{BASE}\rightarrow\mathtt{LARGE}\) stops learning from the teacher at the \(40\)k-th step, it improves the validation PPL from \(4.60\) (\(\mathtt{LARGE}\)) to \(4.28\), which is almost the performance when the baseline LARGE conducts self-learning for \(55\)k steps, thus saving roughly \(27.3\%\) computational costs1. The results in Table 1 show that (2) \({\mathcal{M}_{L}}\)**trained under KI achieves better performance than the baseline on downstream tasks at each step**. We also found empirically that, under the same setting (e.g., data, hyper-parameters and model architectures), lower validation PPL generally indicates better downstream performance. Since the performance gain in downstream tasks is consistent with that reflected in PPL, we only show the latter for the remaining experiments. Concerning the energy cost, for the remaining experiments, unless otherwise specified, we choose MEDIUM (9 layers, \(576\) hidden size) as \({\mathcal{M}_{S}}\) and BASE as \({\mathcal{M}_{L}}\).

Footnote 1: If we load BASE and compute its logits during pre-training, \(18.7\%\) FLOPs can be saved roughly, since the forward passes of the small teacher also take up a small part.

Effects of Inheritance Rate.We set \(\alpha_{t}\) in Eq. (2) to be linearly decayed (denoted as Linear) to gradually encourage \({\mathcal{M}_{L}}\) exploring knowledge on its own. We analyze whether this design is necessary by comparing it with two other strategies: the first is to only learn from the teacher at first and change to pure self-learning (denoted as Heviside) at the \(35\)k-th step; the second is to use a constant ratio (\(1:1\)) between \({\mathcal{L}_{\text{SELF}}}\) and \({\mathcal{L}_{\text{KI}}}\) throughout the whole training process (denoted as Constant). We can conclude from Figure 1 (b) that: (1) **annealing at first is necessary**. The validation PRL curve of Linear converges the fastest, while Heviside tends to increase after \({\mathcal{M}_{L}}\) stops learning from the teacher, indicating that, due to the difference between learning from the teacher and self-learning, annealing at first is necessary so that the performance won't decay at the transition point (the \(35\)k-th step). (2) **Supervision from the teacher is redundant after \({\mathcal{M}_{L}}\) surpasses \({\mathcal{M}_{S}}\)**. Although Constant performs well in the beginning, its PPL gradually becomes even worse than the other two strategies. This indicates that after \({\mathcal{M}_{L}}\) has already surpassed \({\mathcal{M}_{S}}\), it will be encumbered by keeping following guidance from \({\mathcal{M}_{S}}\).

Saving Storage Space with Top-K Logits.Loading the teacher \({\mathcal{M}_{S}}\) repeatedly for KI is cumbersome, and an alternative way is to pre-compute and save the predictions of \({\mathcal{M}_{S}}\) offline once and for all. We show that using the information of top-\(K\) logits Tan et al. (2019) can reduce the memory footprint without much performance decrease. Specifically, we save only top-\(K\) probabilities of \({\mathcal{P}_{S}}(x^{j};\tau)\) followed by re-normalization, instead of the full distribution over all tokens. For RoBERTa,

Figure 1: (a) The validation PPL curve for pre-training \({\mathcal{M}_{L}}\) under KI framework (\(\mathtt{BASE}\rightarrow\mathtt{LARGE}\)) and the self-learning baseline (\(\mathtt{LARGE}\)). The teacher’s (\(\mathtt{BASE}\)) performance is \(4.18\). (b) Pre-training \(\mathtt{BASE}\) under KI with three strategies for the inheritance rate \(\alpha_{t}\): Linear, Heviside and Constant. The teacher’s (MEDIUM) performance is \(4.95\). (c) Pre-training \(\mathtt{BASE}\) under KI with top-\(K\) logits, we vary \(K\) in \(\left\{10,50,100,1000\right\}\), respectively.

 

the dimension of \({\cal P}_{S}(x^{j};\tau)\) is decided by its vocabulary size, which is around \(50,000\). We thus vary \(K\) in \(\{10,50,100,1000\}\) to see its effects in Figure 1 (c), from which we observe that: **top-K logits contain the vast majority of information**. Choosing a relatively small \(K\) (e.g., \(10\)) is already good enough for inheriting knowledge from the teacher without much performance decrease. Previous work also indicates the relation between KD and label smoothing Shen et al. (2021), however, we show in appendix F that the improvements of KI are not because of benefiting from optimizing smoothed targets, which impose regularization.

Experiments on GPT.To demonstrate that KI is model-agnostic, we conduct experiments on auto-regressive language modeling and choose GPT Radford et al. (2018) architecture with growing sizes of \(\{73\text{M},124\text{M},209\text{M},354\text{M},773\text{M},\\ 1\text{B}\}\) parameters in total, respectively. The detailed architectures are specified in Table 6. All the teacher models are pre-trained for \(62.5\text{k}\) steps with a batch size of \(2,048\). As reflected in Figure 2 (a), training larger GPTs under our KI framework converges faster than the self-learning baseline, which demonstrates **KI is agnostic to the specific pretraining objective and PLM architecture**.

### RQ2: Could Knowledge Inheritance be Performed over Generations?

Human beings can inherit the knowledge from their antecedents, refine it and pass it down to their offsprings, so that knowledge can gradually accumulate over generations. Inspired by this, we investigate whether PLMs also have this kind of pattern. Specifically, we experiment with the knowledge inheritance among three generations of RoBERTa with roughly \(1.7\text{x}\) growth in model size: \(G_{1}\)(BASE, \(125\text{M}\)), \(G_{2}\)(BASE_PLUS, \(211\text{M}\)) and \(G_{3}\)(LARGE, \(355\text{M}\)), whose architectures are listed in Table 6. All models are trained from scratch for \(125\text{k}\) steps with a batch size of \(2,048\) on the same corpus. We compare the differences among (1) self-learning for each generation (denoted as \(G_{1}\), \(G_{2}\) and \(G_{3}\)), (2) KI over two generations (denoted as \(G_{1}\to G_{2}\), \(G_{1}\to G_{3}\) and \(G_{2}\to G_{3}\)), and (3) KI over three generations (denoted as \(G_{1}\to G_{2}\to G_{3}\)), where \(G_{2}\) first inherits the knowledge from \(G_{1}\), refines it by additional self-exploring and passes its knowledge down to \(G_{3}\). The results are drawn in Figure 2 (b). Comparing the performance of \(G_{2}\) and \(G_{1}\to G_{2}\), \(G_{3}\) and \(G_{1}\to G_{3}\), or \(G_{3}\) and \(G_{2}\to G_{3}\), we can again demonstrate the superiority of KI over self-training as concluded before. Comparing the performance of \(G_{1}\to G_{3}\) and \(G_{1}\to G_{2}\to G_{3}\), or \(G_{2}\to G_{3}\) and \(G_{1}\to G_{2}\to G_{3}\), it is observed that the performance of \(G_{3}\) benefits from the involvements of both \(G_{1}\) and \(G_{2}\), which means knowledge could be accumulated through more generations' involvements.

### RQ3: How Could \(\mathcal{M}_{S}\)'s Pre-training Setting Affect Knowledge Inheritance?

Existing PLMs are typically trained under quite different settings, and it is unclear how these different settings will affect the performance of KI. Formally, we have a series of well-trained smaller PLMs \(\overline{\mathcal{M}_{S}}=\{\mathcal{M}_{S}^{1},...,\mathcal{M}_{S}^{N_{S}}\}\), each having been optimized on \(\overline{\mathcal{D}_{S}}=\{\mathcal{D}_{S}^{1},...,\underline{\mathcal{D}_ {S}^{N_{S}}}\}\), respectively. Considering that the PLMs in \(\overline{\mathcal{M}_{S}}\), consisting of varied model architectures, are pre-trained on different corpora of various sizes and domains with arbitrary strategies, thus the knowledge they master is also manifold. In addition, \(\mathcal{M}_{L}\)'s pre-training data \(\overline{\mathcal{D}_{L}}\) may also consist of massive, heterogeneous corpora from multiple sources, i.e., \(\overline{\mathcal{D}_{L}}=\{\mathcal{D}_{L}^{1},...,\mathcal{D}_{L}^{N_{L}}\}\). Due to the difference between \(\overline{\mathcal{D}_{L}}\) and \(\overline{\mathcal{D}_{S}}\), \(\mathcal{M}_{S}\)

\begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Step** & **Model** & **CoLA** & **MNLI** & **QNIJ** & **RTE** & **SST-2** & **STS-B** & **MRPC** & **OQP** & **Avg** \\ \hline \multirow{3}{*}{5k} & LARGE & \(0.0\) & \(73.5\) & \(81.7\) & \(53.0\) & \(81.7\) & \(45.8\) & \(71.4\) & \(87.5\) & \(61.8\) \\  & BASE \(\rightarrow\) LARGE & \(\mathbf{17.4}\) & \(\mathbf{75.8}\) & \(\mathbf{83.4}\) & \(\mathbf{54.7}\) & \(\mathbf{85.7}\) & \(\mathbf{72.0}\) & \(\mathbf{72.6}\) & \(\mathbf{88.6}\) & \(\mathbf{68.8}\) \\ \hline \multirow{3}{*}{45k} & LARGE & \(61.8\) & \(84.9\) & \(91.7\) & \(63.4\) & \(92.9\) & \(88.6\) & \(87.7\) & \(\mathbf{91.5}\) & \(82.8\) \\  & BASE \(\rightarrow\) LARGE & \(\mathbf{64.3}\) & \(\mathbf{85.9}\) & \(\mathbf{92.2}\) & \(\mathbf{75.3}\) & \(\mathbf{93.2}\) & \(\mathbf{89.3}\) & \(\mathbf{89.4}\) & \(91.5\) & \(\mathbf{85.2}\) \\ \hline \multirow{3}{*}{85k} & LARGE & \(64.5\) & \(86.8\) & \(92.7\) & \(69.7\) & \(93.5\) & \(89.9\) & \(89.7\) & \(91.7\) & \(84.8\) \\  & BASE \(\rightarrow\) LARGE & \(\mathbf{65.7}\) & \(\mathbf{87.2}\) & \(\mathbf{93.0}\) & \(\mathbf{77.0}\) & \(\mathbf{94.3}\) & \(\mathbf{90.0}\) & \(\mathbf{90.4}\) & \(\mathbf{91.8}\) & \(\mathbf{86.2}\) \\ \hline \multirow{3}{*}{\(125\text{k}\)} & LARGE & \(64.3\) & \(87.1\) & \(\mathbf{93.2}\) & \(73.4\) & \(94.1\) & \(90.3\) & \(\mathbf{90.1}\) & \(91.8\) & \(85.5\) \\  & BASE \(\rightarrow\) LARGE & \(\mathbf{67.7}\) & \(\mathbf{87.7}\) & \(93.1\) & \(\mathbf{74.9}\) & \(\mathbf{94.8}\) & \(\mathbf{90.6}\) & \(88.2\) & \(\mathbf{91.9}\) & \(\mathbf{86.1}\) \\ \hline \end{tabular}
\end{table}
Table 1: Downstream performances on GLUE tasks (dev). KI requires fewer pre-training steps to get a high score after fine-tuning. Detailed results at different training steps are illustrated in appendix A.6.

 

may be required to transfer its knowledge on instances unseen during its pre-training. Ideally, we want \({\cal M}_{S}\) to teach the courses it is skilled in. Therefore, it is essential to choose the most appropriate teacher for each composition \({\cal D}^{*}_{L}\in\overline{{\cal D}_{L}}\). To this end, we conduct thorough experiments to analyze the effects of several representative factors: model architecture, pre-training data, \({\cal M}_{S}\)'s pre-training step (appendix A.2) and batch size (appendix A.3).

Effects of Model Architecture.Large PLMs generally converge faster and achieve lower PPL, thus serving as more competent teachers. We experiment with two widely chosen architecture variations, i.e., depth (number of layers) and width (hidden size), to explore the effects of \({\cal M}_{S}\)'s model architectures. We choose BASE (\(12\) layer, \(768\) hidden size) as \({\cal M}_{L}\)'s architecture, and choose the architecture of \({\cal M}_{S}\) to differ from \({\cal M}_{L}\) in either depth or width. Specifically, for \({\cal M}_{S}\), we vary the depth in \(\{4,6,8,10\}\), and the width in \(\{384,480,576,672\}\), respectively, and pre-train \({\cal M}_{S}\) under the same setting as \({\cal M}_{L}\). The PPL curve for each teacher model is shown in appendix A.7, from which we observe that deeper / wider teachers with more parameters converge faster and achieve lower PPL. After that, we pre-train \({\cal M}_{L}\) under KI leveraging these teacher models. As shown in Figure 2 (c) and appendix A.4, **choosing a deeper / wider teacher accelerates \({\cal M}_{L}\)'s convergence**, demonstrating the benefits of learning from a more knowledgeable teacher. Since the performance of PLMs is weakly related to the model shape but highly related to the model size [19], it is always a better strategy to choose the larger teacher if other settings are kept the same. In experiments, we also find empirically that, the optimal duration of learning from the teacher is longer for larger teachers, which means it takes more time to learn from a more knowledgeable teacher.

Effects of Pre-training Data.In previous experiments, we assume \({\cal M}_{L}\) is pre-trained on the same corpus as \({\cal M}_{S}\), i.e., \({\cal D}_{L}={\cal D}_{S}\). However, in real-world scenarios, it may occur that the pre-training corpus used by both \({\cal M}_{L}\) and \({\cal M}_{S}\) is mismatched, due to three main factors: (1) **data size.** When training larger models, the pre-training corpus is often enlarged to improve downstream performance, i.e., \(|{\cal D}_{S}|\ll|{\cal D}_{L}|\); (2) **data domain.** PLMs are trained on heterogeneous corpora from various sources (e.g., news articles, literary works, etc.), i.e., \({\cal P}_{{\cal D}_{S}}\neq{\cal P}_{D_{L}}\). The different knowledge contained in each domain may affect PLMs' generalization in downstream tasks; (3) **data privacy.** Even if both size and domain of \({\cal D}_{S}\) and \({\cal D}_{L}\) are ensured to be the same, it may be hard to retrieve the pre-training corpus used by \({\cal M}_{S}\) due to privacy concerns, with an extreme case: \({\cal D}_{L}\cap{\cal D}_{S}=\emptyset\). The gap between \({\cal D}_{S}\) and \({\cal D}_{L}\) may hinder \({\cal M}_{S}\)'s successful knowledge transfer. We thus design experiments to analyze the effects of these factors, with three observations concluded:

\(\bullet\)**Obs. 1: PLMs can image the big from the small for in-domain data**. To evaluate the effects of data size, we first pre-train teacher models on different partitions of the original training corpus under the same setting by randomly sampling \(\{\frac{1}{16},\frac{1}{8},\frac{1}{4},\frac{1}{2},\frac{1}{1}\}\) of it, resulting in teacher models with final validation PPL of \(\{5.43,5.15,5.04,4.98,4.92\}\), respectively. The final validation PPL increases as we shrink the size of \({\cal M}_{S}\)'s pre-training corpus, which implies that training with less data weakens the teacher's ability. Next, we compare the differences when their knowledge is inherited by \({\cal M}_{L}\). As reflected in Figure 3 (a), however, the performance of KI is not substantially undermined until only \(\frac{1}{16}\) of the original data is leveraged by the teacher. This indicates that PLMs can well image the overall data distribution even if it only sees a small part. Hence,

Figure 2: (a) Experiments on GPT. (b) KI over generations. (c) Effects of \({\cal M}_{S}\)’s architecture (depth).

 

when training larger PLMs, unless the data size is extensively enlarged, its impact can be ignored.

\(\bullet\)**Obs. 2: Inheriting on similar domain improves performance**. To evaluate the effects of data domain, we experiment on the cases where \({\cal D}_{S}\) and \({\cal D}_{L}\) have domain mismatch. Specifically, keeping data size the same, we mix Wikipedia and BookCorpus (WB) used before with computer science (CS) papers from S2ORC Lo et al. (2020), whose domain is distinct from WB, using different proportions, i.e., \(\text{WB}:\text{CS}=\{1:2,2:1,3:1,4:1\}\), respectively. We pre-train \({\cal M}_{S}\) on the constructed corpora, then test the performance when \({\cal M}_{L}\) inherits these teachers' knowledge on the WB domain data. As shown in Figure 3 (b), with the domain of the constructed corpus \({\cal M}_{S}\) is trained on becoming gradually similar to WB, the benefits from KI become more obvious, which means it is essential that both \({\cal M}_{S}\) and \({\cal M}_{L}\) are trained on similar domain of data, so that \({\cal M}_{S}\) can successfully impart knowledge to \({\cal M}_{L}\) by teaching the "right" course.

\(\bullet\)**Obs. 3: Data privacy is not that important if the same domain is ensured**. To evaluate the effects of data privacy, we experiment in an extreme case where \({\cal D}_{S}\) and \({\cal D}_{L}\) have no overlap at all. To avoid the influences of size and domain, we randomly split the WB domain training corpus \({\cal D}\) into two halves (\({\cal D}_{A}\) and \({\cal D}_{B}\)) and pre-train two teacher models (denoted as MEDIUM\({}_{\texttt{A}}\) and MEDIUM\({}_{\texttt{B}}\)) on them. After pre-training, both of them achieve almost the same final PPL (\(4.99\)) on the same validation set. They are then inherited by the student model BASE on \({\cal D}_{B}\) (denoted as MEDIUM\({}_{\texttt{A}}\rightarrow\textsc{BASE}_{\texttt{B}}\) and MEDIUM\({}_{\texttt{B}}\to\textsc{BASE}_{\texttt{B}}\)), which is exactly the pre-training corpus of MEDIUM\({}_{\texttt{B}}\) and has no overlap with that of MEDIUM\({}_{\texttt{A}}\). We also choose \({\cal M}_{L}\) that conducts pure self-learning on \({\cal D}_{B}\) as the baseline (denoted as BASE\({}_{\texttt{B}}\)). It is observed from Figure 3 (c) that, there is little difference between the validation PPL curves of MEDIUM\({}_{\texttt{A}}\rightarrow\textsc{BASE}_{\texttt{B}}\) and MEDIUM\({}_{\texttt{B}}\to\textsc{BASE}_{\texttt{B}}\), indicating that whether the pre-training corpus of \({\cal M}_{S}\) and \({\cal M}_{L}\) has data overlap or not is not a serious issue as long as they share the same domain. This is meaningful when organizations aim to share the knowledge of their PLMs without exposing either the pre-training data or the model parameters due to privacy concerns.

### RQ4: How Could Knowledge Inheritance Benefit Domain Adaptation?

With streaming data of various domains continuously emerging, training domain-specific PLMs and storing the model parameters for each domain can be prohibitively expensive. To this end, researchers recently demonstrated the feasibility of adapting PLMs to the target domain through continual pre-training Gururangan et al. (2020). In this section, we further extend KI and demonstrate that domain adaptation for PLM can benefit from inheriting knowledge of existing domain experts.

Specifically, instead of training large PLMs from scratch, which is the setting used before, we focus on adapting BASE\({}_{\texttt{NB}}\), which has been well-trained on the WB domain for \(125\mathrm{k}\) steps, to two target domains, i.e., computer science (CS) and biomedical (BIO) papers from S2ORC Lo et al. (2020). The proximity (vocabulary overlap) of three domains is listed in appendix D. We assume there exist two domain experts, i.e., MEDIUM\({}_{\texttt{CS}}\) and MEDIUM\({}_{\texttt{BIO}}\). Each model has been trained on CS / BIO domain for \(125\mathrm{k}\) steps. Note their training computation is far less than BASE\({}_{\texttt{NB}}\) due to fewer model parameters. Hence, either MEDIUM\({}_{\texttt{CS}}\) or MEDIUM\({}_{\texttt{BIO}}\) is no match for BASE\({}_{\texttt{NB}}\) in WB domain but has richer knowledge in CS / BIO domain. For evaluation, we compare both (1) the validation PPL on the target domain and (2) the performance (test F1) on downstream tasks, i.e. ACL-ARC Jurgens et al. (2018) for CS domain and CHEMPROT Kringelum et al. (2016) for BIO domain. Before adaptation,

Figure 3: Effects of \({\cal M}_{S}\)’s pre-training (a) data size, (b) data domain and (c) data privacy for KI.

 

\({}_{\text{BASE}}\) achieves a PPL of \(5.41\) / \(4.86\) and F1 of \(68.5\) / \(81.6\) on CS / BIO domain, while MEDIUM\({}_{\texttt{CS}}\) achieves \(2.95\) (PPL) and \(69.4\) (F1) on CS domain, MEDIUM\({}_{\texttt{BIO}}\) achieves \(2.55\) (PPL) and \(83.6\) (F1) on BIO domain. This demonstrates the superiority of two teachers over the student in their own domain despite their smaller model capacity.

We compare two strategies for domain adaptation: (1) only conducting self-learning on the target domain and (2) inheriting knowledge from well-trained domain teachers. Specifically, BASE\({}_{\texttt{WB}}\) is post-trained for additional \(4\)k steps on either CS or BIO domain to learn new knowledge. In addition, considering that in real-world scenarios, it can be hard to retrieve enough pre-training data for a specific domain, due to some privacy issues. Hence, we conduct experiments with different sizes of domain corpus. In Table 2, BASE\({}_{\texttt{WB}}\) is post-trained on either CS or BIO domain while in Table 3, it is trained on synthetic domain data (BIO : CS = \(1:1\)) to absorb knowledge from two domains simultaneously (we assume \(\mathcal{M}_{L}\) is trained with the optimal teacher selection strategy, i.e., each teacher imparts the knowledge on its own domain data). It can be concluded from Table 2 and Table 3 that:

(1) **KI is more training-efficient**. Compared with self-learning, inheriting knowledge from domain teachers achieves lower final PPL and improved performance in domain-specific downstream tasks, indicating that, for domain adaptation, KI is more training-efficient so that an already trained large PLM could absorb more knowledge from new domain with the same training budget. (2) **KI is more data-efficient**. The PPL gap between KI and SL is further enlarged when there is less domain-specific data available for adaptation, which means KI is more data-efficient especially under the low-resource setting, where domain data is scarce. In other words, only providing a small portion of domain-specific data is enough for satisfactory adaptation performance under KI, while self-learning exhibits overfitting to some extent. (3) **Large PLMs can simultaneously absorb knowledge from multiple domains and thus become omnipotent**. From Table 3, we observe BASE\({}_{\texttt{WB}}\) achieves improved performance on both domains after being taught by two teachers simultaneously. KI shows superiority over self-learning. However, simultaneous learning overfits training data more easily and its performance on either domain is no match for learning only one domain at a time.

## 5 Conclusion and Future Work

In this work, we propose a general knowledge inheritance (KI) framework that leverages previously trained PLMs for training larger ones. We conduct sufficient empirical studies to demonstrate its feasibility. In addition, we show that KI could well support knowledge transfer over a series of PLMs with growing sizes. We also comprehensively analyze various pre-training settings of the teacher model that may affect KI's performance, the results shed light on how to choose the most appropriate teacher PLM for KI. Finally, we ex

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Ntokens**} & \multicolumn{3}{c}{\(3,400\)M} & \multicolumn{3}{c}{\(200\)M} & \multicolumn{3}{c}{\(100\)M} & \multicolumn{3}{c}{\(40\)M} \\ \hline
**Metrics** & F1 & PPL & F1 & PPL & F1 & PPL & F1 & PPL & F1 & PPL & F1 & PPL \\ \hline  & SL & \(69.8\) & \(3.12\) & \(71.7\) & \(3.17\) & \(71.4\) & \(3.24\) & \(68.3\) & \(3.51\) & \(67.5\) & \(4.07\) \\ CS & KI & \(\mathbf{72.9}\) & \(\mathbf{3.06}\) & \(\mathbf{72.6}\) & \(\mathbf{3.09}\) & \(\mathbf{71.9}\) & \(\mathbf{3.11}\) & \(\mathbf{71.1}\) & \(\mathbf{3.21}\) & \(\mathbf{70.8}\) & \(\mathbf{3.37}\) \\ \hline  & SL & \(84.0\) & \(2.67\) & \(82.8\) & \(2.72\) & \(83.2\) & \(2.83\) & \(83.3\) & \(3.16\) & \(82.7\) & \(3.81\) \\ BIO & KI & \(\mathbf{84.5}\) & \(\mathbf{2.65}\) & \(\mathbf{83.4}\) & \(\mathbf{2.66}\) & \(\mathbf{83.9}\) & \(\mathbf{2.69}\) & \(\mathbf{83.6}\) & \(\mathbf{2.82}\) & \(\mathbf{83.5}\) & \(\mathbf{3.01}\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: The validation PPL (PPL) and downstream performance (F1) on the target domain (CS / BIO) after BASE\({}_{\texttt{WB}}\) is post-trained for \(4\)k steps with self-learning (SL) or knowledge inheritance (KI). We experiment with different sizes of domain corpus. All downstream experiments are repeated \(10\) times with different seeds.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c c c} \hline \hline \multicolumn{1}{c}{**Ntokens**} & \multicolumn{3}{c}{\(3,400\)M} & \multicolumn{3}{c}{\(200\)M} & \multicolumn{3}{c}{\(100\)M} & \multicolumn{3}{c}{\(40\)M} \\ \hline
**Metrics** & F1C & PPL\({}_{\text{LC}}\) & F1B & PPL\({}_{\text{AB}}\) & F1C & PPL\({}_{\text{AC}}\) & F1B & PPL\({}_{\text{BC}}\) & F1C & PPL\({}_{\text{BC}}\) & F1A & PPL\({}_{\text{BC}}\) & F1B & PPL\({}_{\text{BC}}\) \\ \hline SL & \(71.7\) & \(3.15\) & \(83.7\) & \(2.71\) & \(70.5\) & \(3.97\) & \(82.7\) & \(3.36\) & \(67.7\) & \(5.95\) & \(81.7\) & \(4.84\) & \(68.3\) & \(11.7\) & \(81.1\) & \(10.5\) \\ KI & \(\mathbf{72.2}\) & \(\mathbf{3.15}\) & \(\mathbf{83.9}\) & \(\mathbf{2.70}\) & \(\mathbf{71.8}\) & \(\mathbf{3.42}\) & \(\mathbf{83.1}\) & \(\mathbf{2.92}\) & \(\mathbf{69.8}\) & \(\mathbf{3.90}\) & \(\mathbf{82.6}\) & \(\mathbf{3.32}\) & \(\mathbf{69.1}\) & \(\mathbf{5.70}\) & \(\mathbf{81.3}\) & \(\mathbf{4.64}\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: The results when BASE\({}_{\texttt{WB}}\) is post-trained on two new domains simultaneously with self-learning (SL) or knowledge inheritance (KI). We report both validation PPL (PPL\({}_{\text{B}}\) / PPL\({}_{\text{C}}\)) and downstream performance (F1B / F1C) for BIO / CS domain. We observe that SL exhibits severe overfitting when data is relatively scarce.

 

tend KI and show that, during domain adaptation, an already trained large PLM could benefit from smaller domain teachers. In general, we provide a promising direction to share and exchange the knowledge learned by different models and continuously promote their performance.

In future, we aim to explore the following directions: (1) the **efficiency** of KI, i.e., given limited computational budget and pre-training corpus, how to more efficiently absorb knowledge from teacher models. Potential solutions include denoising teacher models' predictions and utilizing more information from the teacher. How to select the most representative data points for KI is also an interesting topic; (2) the **effectiveness** of KI under different settings, i.e., how can KI be applied if the teachers and the students are pre-trained on different vocabularies, languages, pre-training objectives and modalities.

Finally, we believe it is vital to use fair benchmarking that can accurately and reliably judge each KI algorithm. Thus, we suggest future work to: (1) conduct all experiments under the same computation environment and report the pre-training hyper-parameters and hardware deployments in detail, (2) evaluate the downstream tasks with multiple different random seeds and choose tasks that give relatively stable and consistent results, which could serve as better indicators for PLMs' effectiveness. In addition, it is also essential that PLMs are tested on diverse downstream tasks which evaluate PLMs' different abilities, (3) save the checkpoint more frequently during pre-training and evaluate the downstream performance, which can better indicate the trend of PLMs' effectiveness, and (4) open-source all the codes and model parameters for future comparisons.

## Acknowledgments

This work is supported by the National Key R&D Program of China (No. 2020AAA0106502), Institute Guo Qiang at Tsinghua University, NExT++ project from the National Research Foundation, Prime Minister's Office, Singapore under its IRC@Singapore Funding Initiative, and Beijing Academy of Artificial Intelligence (BAAI). This work is also supported by the Pattern Recognition Center, WeChat AI, Tencent Inc. Yujia Qin and Yankai Lin designed the methods and the experiments. Yujia Qin, Jing Yi and Jiajie Zhang conducted the experiments. Yujia Qin, Yankai Lin and Xu Han wrote the paper. Zhiyuan Liu, Peng Li, Maosong Sun and Jie Zhou advised the project. All the authors participated in the discussion.

## References

* Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. _arXiv preprint arXiv:2108.07258_.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighian, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Chen et al. (2022) Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. 2022. bert2bert: Towards reusable pretrained language models. In _Association for Computational Linguistics: ACL 2022_, Online.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fedus et al. (2021) William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. _ArXiv preprint_, abs/2101.03961.
* Furlanello et al. (2018) Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born-again neural networks. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmassan, Stockholm, Sweden, July 10-15, 2018_, volume 80 of _Proceedings of Machine Learning Research_, pages 1602-1611. PMLR.
* Gong et al. (2019) Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Efficient training of BERT by progressively stacking. In _Proceedings 

of the 36th International Conference on Machine Learning, ICML 2019, San Jose 2019, Long Beach, California, USA_, volume 97 of _Proceedings of Machine Learning Research_, pages 2337-2346. PMLR. Cited by: SS5.
* X. Gu, L. Liu, H. Yu, J. Li, C. Chen, and J. Han (2021)On the transformer growth for progressive BERT training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 5174-5180. External Links: Link, Document Cited by: SS1, SS2.
* S. Gururangan, A. Marasovic, S. Swayamdipta, K. Lo, I. Beltayy, D. Downey, and N. A. Smith (2020)Don't stop pretraining: adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8342-8360. External Links: Link, Document Cited by: SS1, SS2.
* X. Han, Z. Zhang, N. Ding, Y. Gu, X. Liu, Y. Huo, J. Qiu, L. Zhang, W. Han, M. Huang, Q. Jin, Y. Lan, Y. Liu, Z. Liu, Z. Lu, X. Qiu, R. Song, J. Tang, J. Wen, J. Yuan, W. Xin Zhao, and J. Zhu (2021)Pre-trained models: past, present and future. ArXiv preprintabs/2106.07139. External Links: Link, 2106.07139 Cited by: SS1, SS2.
* G. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network. ArXiv preprintabs/1503.02531. External Links: Link, 1503.02531 Cited by: SS1, SS2.
* X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu (2020)TinyBERT: distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, pp. 4163-4174. External Links: Link, 2001.07865 Cited by: SS1, SS2.
* D. Jurgens, S. Kumar, R. Hoover, D. McFarland, and D. Jurafsky (2018)Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics6, pp. 391-406. External Links: Link, 1804.06620 Cited by: SS1, SS2.
* J. Kaplan, S. McCandlish, T. Henighan, T. B Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei (2020)Scaling laws for neural language models. ArXiv preprintabs/2001.08361. External Links: Link, 2001.08361 Cited by: SS1, SS2.
* J. Kringelum, S. Kim Kjaerulff, S. Brunak, O. Lund, T. I. Oprea, and O. Taboureau (2016)Chemprot-3.0: a global chemical biology diseases mapping. Database. Cited by: SS1, SS2.
* K. Krishna, G. Singh Tomar, A. P. Parikh, N. Papernot, and M. Iyyer (2020)Thieves on sesame street! model extraction of bert-based apis. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, External Links: Link, 2002.08361 Cited by: SS1, SS2.
* M. Li, E. Yumer, and D. Ramanan (2020a)Budgeted training: rethinking deep neural network training under resource constraints. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, External Links: Link, 2002.08361 Cited by: SS1, SS2.
* Z. Li, E. Wallace, S. Shen, K. Lin, K. Keutzer, D. Klein, and J. Gonzalez (2020b)Train big, then compress: rethinking model size for efficient training and inference of transformers. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Vol. 119 of _Proceedings of Machine Learning Research_, pp. 5958-5968. External Links: Link, 2002.08361 Cited by: SS1, SS2.
* T. Lin, Y. Wang, X. Liu, and X. Qiu (2021)A survey of transformers. ArXiv preprintabs/2106.04554. External Links: Link, 2106.04554 Cited by: SS1, SS2.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. ArXiv preprintabs/1907.11692. External Links: Link, 1907.11692 Cited by: SS1, SS2.
* K. Lo, L. L. Wang, M. Neumann, R. Kinney, and D. Weld (2020)S2ORC: the semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4969-4983. External Links: Link, 2001.08361 Cited by: SS1, SS2.
* B. Min, H. Ross, E. Sulem, A. Pourran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, E. Agirre, I. Heinz, and D. Roth (2021)Recent advances in natural language processing via large pre-trained language models: a survey. arXiv preprint arXiv:2111.01243. External Links: Link, 2111.01243 Cited by: SS1, SS2.
* D. Patterson, J. Gonzalez, Quoc Le, C. Liang, L. Munguia, D. Rothchild, D. So, M. Texier, and J. Dean (2021)Carbon emissions and large neural network training. ArXiv preprintabs/2104.10350. External Links: Link, 2104.10350 Cited by: SS1, SS2.
* Y. Qin, J. Zhang, Y. Lin, Z. Liu, P. Li, M. Sun, and J. Zhou (2022)ElbE: efficient lifelong pre-training for emerging data. In Findings of the Association for Computational Linguistics: ACL 2022, Online, Cited by: SS1, SS2.
* A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever (2018)Improving language understanding by generative pre-training. External Links: 1804.06617 Cited by: SS1, SS2.
* C. Raffelt, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2019)Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv preprintabs/1910.10683. External Links: Link, 1910.10683 Cited by: SS1, SS2.

 

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. _ArXiv preprint_, abs/1910.01108.
* Schwartz et al. (2019) Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. 2019. Green ai. _ArXiv preprint_, abs/1907.10597.
* Shen et al. (2021) Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. 2021. Is label smoothing truly incompatible with knowledge distillation: An empirical study. _arXiv preprint arXiv:2104.00676_.
* Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-lm: Training multi-billion parameter language models using model parallelism. _ArXiv preprint_, abs/1909.08053.
* Sun et al. (2019) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4323-4332, Hong Kong, China. Association for Computational Linguistics.
* Sun et al. (2020) Siqi Sun, Zhe Gan, Yuwei Fang, Yu Cheng, Shuohang Wang, and Jingjing Liu. 2020. Contrastive distillation on intermediate representations for language model compression. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 498-508, Online. Association for Computational Linguistics.
* Tan et al. (2019) Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Wang et al. (2019) Alex Wang, Amampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Yang et al. (2019) Chenglin Yang, Lingxi Xie, Siyuan Qiao, and Alan L Yuille. 2019. Training deep neural networks in generations: A more tolerant teacher educates better students. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 5628-5635.
* You et al. (2019) Yang You, Jing Li, Jonathan Hiseu, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. 2019. Reducing bert pre-training time from 3 days to 76 minutes. _ArXiv preprint_, abs/1904.00962.
* You et al. (2020) Yang You, Jing Li, Sashank J. Reddi, Jonathan Hkeu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020. Large batch optimization for deep learning: Training BERT in 76 minutes. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Yuan et al. (2020) Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. 2020. Revisiting knowledge distillation via label smoothing regularization. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 3903-3911.
* Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015_, pages 19-27. IEEE Computer Society.

 



## Appendix A Additional Experiments and Analysis

### Effects of Model Size

We experiment on four PLMs with roughly \(1.7\)x growth in model size: \({\cal M}_{1}\) (RoBERTaMEDIUM, \(73.5\)M), \({\cal M}_{2}\) (RoBERTaBASE, \(125\)M), \({\cal M}_{3}\) (RoBERTaBASE_PLUS, \(211\)M) and \({\cal M}_{4}\) (RoBERaLARGE, \(355\)M), whose architectures are listed in Table 6. We first pre-train a teacher PLM \({\cal M}_{i}\) (\({\cal M}_{S}\)) for \(125\)k steps with a batch size of \(2,048\) under the same setting then train a larger one \({\cal M}_{i+1}\) (\({\cal M}_{L}\)) by inheriting \({\cal M}_{i}\)'s knowledge under KI framework (denoted as \({\cal M}_{i}\rightarrow{\cal M}_{i+1},i\in\{1,2,3\}\)). We compare \({\cal M}_{i}\rightarrow{\cal M}_{i+1}\) with \({\cal M}_{i+1}\) that conducts self-learning from beginning to end. As shown in Figure 4, the superiority of KI is observed across all models. In addition, with the overall model size of \({\cal M}_{S}\) and \({\cal M}_{L}\) gradually increasing, the benefits of KI become more evident, reflected in the broader absolute gap between the PPL curve of \({\cal M}_{i}\rightarrow{\cal M}_{i+1}\) and \({\cal M}_{i+1}\) when \(i\) gradually grows. This implies that with the advance of computing power in future, training larger PLMs will benefit more and more from our KI framework.

### Effects of \({\cal M}_{S}\)'s Pre-training Steps

Longer pre-training has been demonstrated as an effective way for PLMs to achieve better performance Liu et al. (2019) and thus become more knowledgeable. To evaluate the benefits of more pre-training steps for \({\cal M}_{S}\), we first vary RoBERTaMEDIUM's pre-training steps in \(\{62.5\)k, \(125\)k, \(250\)k, \(500\)k}, and keep all other settings the same. After pre-training, these teacher models achieve the final validation PPL of \(\{5.25,4.92,4.72,4.51\}\), respectively. Then we compare the performances when RoBERTaBASE learn from these teacher models and visualize the results in Figure 4, from which we can conclude that, inheriting knowledge from teachers with longer pre-training time (steps) helps \({\cal M}_{L}\) converge faster. However, such a benefit is less and less obvious as \({\cal M}_{S}\)'s pre-training steps increase, which means after enough training computations are invested, the teacher model enters a plateau of convergence in validation PPL, and digging deeper in knowledge becomes even harder. The bottleneck lies in other factors, e.g., the size and diversity of pre-training data, which hinder \({\cal M}_{S}\) from becoming more knowledgeable. We also found empirically that, after being pre-trained for \(125\)k steps on the corpus with a batch size of \(2,048\), all the models used in this paper have well converged, and longer pre-training only results in limited performance gain in either PPL or downstream performance.

### Effects of \({\cal M}_{l}\)'s Batch Size

Batch size is highly related to PLM's training efficiency, and previous work Liu et al. (2019); Li et al. (2020b); You et al. (2019) found that slow-but-accurate large batch sizes can bring improvements to model training, although the improvements become marginal after increasing the batch size beyond a certain point (around \(2,048\)). BERT Devlin et al. (2019) is pre-trained for \(1,000\)k steps with a batch size of \(256\), and the computational cost is equivalent to training for \(125\)k steps with a batch size of \(2,048\)Liu et al. (2019), which is the pre-training setting chosen in our main paper. Choosing RoBERTaMEDIUM as the teacher model and RoBERTaBASE as the student model, in Figure 4 we compare the validation PPL as we vary the batch size in \(\{256,512,1024,2,048\}\), controlling for the number of passes through the pre-training corpus. We also vary the peak learning rate in \(\{1.0\times 10^{-4},2.5\times 10^{-4},3.8\times 10^{-4},5.0\times 10^{-4}\}\) and pre-train for \(\{1,000\)k, \(500\)k, \(250\)k, \(125\)k} steps, respectively, when increasing the batch size. We observe that increasing the batch size results in improved final validation PPL, which is aligned with previous findings Liu et al. (2019). When adjusting batch size, KI accelerates the convergence unanimously, and its benefits become more evident when training with a smaller batch size, reflected in the absolute improvement in final validation PPL. We hypothesize that this is because learning from the smoothed target probability of KI, containing rich _secondary information_Yang et al. (2019) or _dark knowledge_Furlanello et al. (2018), makes the pre-training process more stable. The student PLM is prevented from fitting to unnecessarily strict distributions and can thus learn faster.

### Additional Experiments of the Effects of Teacher Model \({\cal M}_{S}\)'s architecture (width)

We show in Figure 5 the validation PPL of \({\cal M}_{L}\) when choosing the teacher PLM \({\cal M}_{S}\) with different hidden sizes (\(\{384,480,576,672\}\)). As mentioned in our main paper, choosing a wider teacher

model improves the training efficiency of the student PLM.

### Additional Experiments of Knowledge Inheritance for Domain Adaptation

Different Number of Post-training Steps.In the main paper, we adapt RoBERTaBASE_WB to either CS or BIO domain by post-training it for \(4\)k steps. We further vary the number of training steps in \(\{1\text{k},2\text{k},3\text{k},4\text{k},5\text{k}\}\) and visualize the validation PPL in Figure 5. We also experiment on different sizes of domain corpus, i.e., \(3\), \(400\)M, \(200\)M, \(100\)M, \(40\)M tokens, respectively, as done in the main paper. We observe that generally the validation PPL on each domain decreases with the training step growing, and the performance of KI is always better than self-learning. The improvement of KI over self-learning is further enlarged when there is less target domain data available, demonstrating that KI is more data-efficient and can work well in low-resource settings. In addition, self-learning exhibits overfitting problems when the data size of the target domain is relatively small, which is not observed under our KI framework, which means KI can mitigate overfitting under low-resource settings.

Catastrophic Forgetting on the Source Domain.Table 4 lists the validation PPL on the source domain (WB) after RoBERTaBASE_WB is post-trained on the target domain (CS / BIO) with self-learning (SL) and knowledge inheritance (KI) for \(4\)k steps. We show the results w.r.t. different sizes of domain corpus (\(3,400\)M, \(200\)M, \(100\)M and \(40\)M tokens). We observe that after domain adaptation, the validation PPL on the source domain increases, which means PLMs may forget some key knowledge on the source domain when learning new knowledge

Figure 4: Left: effects of \(\mathcal{M}_{L}\)’s model size. Middle: effects of \(\mathcal{M}_{S}\)’s number of pre-training steps. Right: effects of \(\mathcal{M}_{L}\)’s batch size.

Figure 5: Left: the PPL curve when choosing the teacher PLM with different hidden sizes. Middle & Right: adapting RoBERTaBASE_WB to CS (middle) / BIO (right) domain with different number of training steps on different sizes of domain data. We compare two strategies: self-learning and KI. For example, RoBERtaCS_3400M denotes post-training RoBERTaBASE_WB with the self-learning strategy on the \(3,400\)M token CS domain corpus. RoBERTaBASE_WB=cs_3400M denotes post-training RoBERTaBASE_WB with the KI strategy on the \(3,400\)M token CS domain corpus.

 

in the target domain, i.e., the catastrophic forgetting problem. In addition, we find that the problem is more evident for KI than self-learning. We expect future work to further explore how to mitigate the catastrophic forgetting.

### Detailed Downstream Performances on GLUE Tasks

Figure 6 visualizes the downstream performance of RoBERTa\({}_{\texttt{LARGE}}\) and RoBERTa\({}_{\texttt{BASE}}\to\) RoBERTa\({}_{\texttt{LARGE}}\) on the dev sets of six GLUE tasks at different pre-training steps with an interval of \(5\text{k}\). It can be observed that the downstream performance of RoBERTa\({}_{\texttt{BASE}}\to\) RoBERTa\({}_{\texttt{LARGE}}\) rises faster than the baseline, which means it takes fewer pre-training steps for our KI framework to get a high score in downstream tasks. Aligned with previous findings Li et al. (2020b), we found MNLI and SST-2 to be the most stable tasks in GLUE, whose variances are lower.

We also list the average GLUE performance for RoBERTa\({}_{\texttt{BASE}}\to\) RoBERTa\({}_{\texttt{LARGE}}\) and the baseline RoBERTa\({}_{\texttt{LARGE}}\) in Table 5, from which we observe that the baseline at \(70\text{k}\)-th step achieves almost the same GLUE performance as our method at \(40\text{k}\)-th step, which means our framework saves around \(42.9\%\) FLOPs, much higher than the reported \(27.3\%\) FLOPs saved based on the pre-training PPL metric in the main paper. In addition, our method achieves almost the same GLUE performance as the baseline at the final step (\(125\text{k}\)) with only \(70\text{k}\) steps, which means our framework saves \(44\text{\%}\) FLOPs in total. Both the perplexity in the pre-training stage and performance in downstream tasks can be chosen as the evaluation metric for measuring the computational cost savings. However, in this paper, we choose the former because it is more stable and accurate than the latter. We find empirically that some GLUE tasks like CoLA have higher variances than others, which might make the measurement inaccurate.

Besides, when discussing the effects of model architectures in the main paper, we only show the validation PPL of each model during pre-training, we visualize the corresponding downstream performance (MNLI) in Figure 7, from which it can be observed that learning from teacher models with more parameters helps achieve better downstream performance at the same pre-training step. In general, we observe that, under our setting, the performance gain in downstream tasks is aligned with that reflected in validation PPL during pre-training.

### Teacher Models' Validation PPL Curves during Pre-training for "Effects of Model Architecture"

Figure 7 visualizes the validation PPL curves for all the teacher models used in the experiments

Figure 6: Downstream performance visualization on six GLUE tasks comparing RoBERTa\({}_{\texttt{LARGE}}\) and RoBERTa\({}_{\texttt{BASE}}\to\) RoBERTa\({}_{\texttt{LARGE}}\). For CoLA, RTE, SST-2 and STS-B, we repeat fine-tuning for \(5\) times; for MNLI and QNLI, we repeat fine-tuning for \(3\) times.

 

on the effects of model architecture. The teacher models differ from RoBERTaBASE in either the depth or width. Specifically, we vary the depth in \(\{4,6,8,10\}\) (denoted as {RoBERTaH_4, RoBERTaH_6, RoBERTaH_7, RoBERTaH_10}), and the width in \(\{384,480,576,672\}\) (denoted as {RoBERTaD_384, RoBERTaD_480, RoBERTaD_576, RoBERTaD_672}). Generally, PLMs with larger model parameters converge faster and achieve better final performance.

## Appendix B Pre-training Hyper-parameters

In Table 6, we list the architectures we used for all models, covering the details for the total number of trainable parameters (\(n_{\text{params}}\)), the total number of layers (\(n_{\text{layers}}\)), the number of units in each bottleneck layer (\(d_{\text{model}}\)), the total number of attention heads (\(n_{\text{heads}}\)), the inner hidden size of FFN layer (\(d_{\text{FFN}}\)) and the learning rate when batch size is set to \(2,048\) (lr). The training-validation ratio of pre-training data is set to \(199:1\). We set the weight decay to \(0.01\), dropout rate to \(0.1\), and use linear learning rate decay. Adam is chosen as the optimizer. The learning rate is warmed up for the first \(10\%\) steps. The hyper-parameters for Adam optimizer is set to \(1\times 10^{-6},0.9,0.98\) for \(\epsilon,\beta_{1},\beta_{2}\), respectively. For a fair comparison, all experiments are done in the same computation environment with \(8\) NVIDIA 32GB V100 GPUs. Table 7 describes the total number of pre-training steps for each (\(\mathcal{M}_{L}\), \(\mathcal{M}_{S}\)) pair chosen in our experiments.

## Appendix C Fine-tuning Hyper-parameters

Table 8 describes the hyper-parameters for ACLARC, CHEMPROT and GLUE tasks. The selection of these hyper-parameters closely follows Liu et al. (2019) and Gururangan et al. (2020).

## Appendix D Domain Proximity of WB, CS and BIO

Table 9 lists the domain proximity (vocabulary overlap) of WB, CS and BIO used in this paper.

## Appendix E Comparison between Knowledge Inheritance and Parameter Recycling

Parameter recycling (i.e., progressive training) first trains a small PLM, and then gradually increases the depth or width of the network based on parameter initialization. It is an orthogonal research direction against our KI, and has many limitations as follows:

Architecture Mismatch.Existing parameter recycling methods Gong et al. (2019); Gu et al. (2021) require that the architectures of both small PLMs

Figure 7: Left & Middle: downstream performances corresponding to the experiments on effects of \(\mathcal{M}_{S}\)’s model architecture (width (left) & depth (middle)). Right: validation PPL during pre-training for the teacher models used in experiments of effects of teacher model architecture.

 

and large PLMs are matched to some extent, however, our KI does not have such a requirement. For example, Gong et al. (2019); Gu et al. (2021) either requires the number of layers, or the hidden size/embedding size of a large PLM to be the integer multiples of that of a small PLM. Hence, it is not flexible to train larger PLMs with arbitrary architectures, making parameter recycling hard to be implemented practically. Besides, there are more and more advanced non-trivial Transformer modifications appearing (we refer to Lin et al. (2021) for details), e.g., pre-normalization, relative embedding, sparse attention, etc. It is non-trivial to directly transfer the parameters between two PLMs if they have different inner structures. Nevertheless, our KI framework will not be influenced by such architectural mismatches.

**Inability for Multi-to-one Knowledge Inheritance.** It is non-trivial to support absorbing knowledge from multiple teacher models by jointly recycling their model parameters. Instead, it is easy to implement for KI. As shown in our experiments, we demonstrate that under our framework, large PLMs can simultaneously absorb knowledge from multiple teachers.

**Inability of Knowledge Inheritance for Domain Adaptation.** Parameter recycling is hard to support continual learning, which makes large PLMs absorb knowledge from small ones in a lifelong manner. In real-world scenarios, numerous PLMs of different architectures are trained locally with

\begin{table}
\begin{tabular}{l c r r r r r} \hline \hline
* & 12 & d & 12 & 3072 & \(5.0\times 10^{-4}\) \\ RoBERTa\_old &
- & h & 768 & 12 & 3072 & \(5.0\times 10^{-4}\) \\ RoBERTa\_BASE & \(125\)M & 12 & 768 & 12 & 3072 & \(5.0\times 10^{-4}\) \\ RoBERTa\_BASE\_PLUS & \(211\)M & 18 & 864 & 12 & 3600 & \(3.5\times 10^{-4}\) \\ RoBERTa\_LANGE & \(355\)M & 24 & 1024 & 16 & 4096 & \(2.5\times 10^{-4}\) \\ \hline GPTT\_3M & 73M & 9 & 576 & 12 & 3072 & \(5.0\times 10^{-4}\) \\ GPTT\_124M & \(124\)M & 12 & 768 & 12 & 3072 & \(5.0\times 10^{-4}\) \\ GPTT\_209M & \(209\)M & 18 & 864 & 12 & 3600 & \(4.0\times 10^{-4}\) \\ GPTT\_354M & \(354\)M & 24 & 1024 & 16 & 4096 & \(3.5\times 10^{-4}\) \\ GPTT\_73M & \(773\)M & 36 & 1280 & 20 & \(5120\) & \(3.0\times 10^{-4}\) \\ GPTT\_18 & \(1068\)M & 40 & 1440 & 20 & \(5760\) & \(2.5\times 10^{-4}\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Model architectures for all the models we used in this paper.

\begin{table}
\begin{tabular}{c c c} \hline \hline \(\mathcal{M}_{L}\) & \(\mathcal{M}_{S}\) & Steps of teacher-guided learning \\ \hline  & RoBERTa\_DELIUM & \(35\)k \\  & RoBERTa\_old\_384 & \(28\)k \\  & RoBERTa\_old\_489 & \(40\)k \\  & RoBERTa\_old\_576 & \(70\)k \\ RoBERTa\_old\_572 & \(85\)k \\  & RoBERTa\_old\_4 & \(22\)k \\  & RoBERTa\_old\_56 & \(35\)k \\  & RoBERTa\_old\_55k \\ \hline RoBERTa\_old\_584E & \(55\)k \\ \hline RoBERTa\_old\_584E & \(40\)k \\ RoBERTa\_old\_584E & \(65\)k \\ RoBERTa\_old\_584E & \(65\)k \\ RoBERTa\_old\_584E & \(75\)k \\ RoBERTa\_old\_584E & \(75\)k \\ \hline GPTT\_124M & GPTT\_3M & \(10\)k \\ \hline GPTT\_25M & GPTT\_124M & \(15\)k \\ \hline GPTT\_354K & GPTT\_250M & \(18\)k \\ \hline GPTT\_733R & GPTT\_354G & \(16\)k \\ \hline GPTT\_18 & GPTT\_733R & \(20\)k \\ \hline \hline \end{tabular}
\end{table}
Table 7: The total number of steps for teacher-guided learning for different (\(\mathcal{M}_{L}\), \(\mathcal{M}_{S}\)) pairs.

 

different data. These small PLMs can be seen as domain experts, and it is essential that larger PLMs can continuously benefit from these existing PLMs efficiently by incorporating their knowledge so that larger PLMs can become omnipotent. As described before, it is easy to implement for our framework and we have demonstrated the effectiveness.

Model Privacy.Parameter recycling requires the availability of the parameters of an existing PLM, which may be impractical due to some privacy issues, e.g., GPT-3 only provides API access for prediction instead of the model parameters. Instead, our KI framework does not presume access to an existing model parameter since the predictions of the small model can be pre-computed and saved offline. This superiority will further make it possible for API-based online knowledge transfer.

## Appendix F Comparing Label Smoothing and Knowledge Inheritance

Previous work shows the relation between label smoothing and knowledge distillation to some extent Shen et al. (2021). To demonstrate that the success of our KI is not because of learning from a more smoothed target, we conduct experiments comparing both label smoothing and our KI in Table 10. Specifically, for label smoothing, PLMs optimize a smoothed target \({\bf y}_{i}^{S}=(1-\alpha)*{\bf y}_{i}+\alpha*{\bf\tilde{I}}/(K-1)\), where \(\alpha=0\) denotes learning from scratch with no label smoothing, larger \(\alpha\) means a more smoothed target for PLMs to learn from, \(K\) denotes the vocabulary size. Specifically, we choose \(\alpha\) from \(\{0.1,0.2,0.3\}\). It can be concluded from the results in Table 10 that adding label smoothing into the pre-training objectives of PLMs leads to far worse performance than the vanilla baseline, which shows that the improvements of our knowledge inheritance framework are non-trivial: larger PLMs are indeed inheriting the "knowledge" from smaller ones, instead of benefiting from optimizing a smoothed target, which imposes regularization.