SlowGuess/NAACL_2022_OCR huggingface dataset:
Yujia Qin $^{1,2,3}$ , Yankai Lin $^{4}$ , Jing Yi $^{1,2,3}$ , Jiajie Zhang $^{1,2,3}$ , Xu Han $^{1,2,3}$ , Zhengyan Zhang $^{1,2,3}$ , Yusheng Su $^{1,2,3}$ , Zhiyuan Liu $^{1,2,3,5,6*}$ , Peng Li $^{7\dagger}$ , Maosong Sun $^{1,2,3,5*}$ , Jie Zhou $^{4}$
$^{1}$ Department of Computer Science and Technology, Tsinghua University, Beijing, China
$^{2}$ Beijing National Research Center for Information Science and Technology
$^{3}$ Institute for Artificial Intelligence, Tsinghua University, Beijing, China
$^{4}$ Pattern Recognition Center, WeChat AI, Tencent Inc.
$^{5}$ International Innovation Center of Tsinghua University, Shanghai, China
$^{6}$ Quan Cheng Laboratory
$^{7}$ Institute for AI Industry Research (AIR), Tsinghua University, China.
qyj20@mails.tsinghua.edu.cn
Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever larger PLMs. However, it requires tremendous computational resources to train a large-scale PLM, which may be practically unaffordable. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring that many well-trained PLMs are available. To this end, we explore the question how could existing PLMs benefit training large-scale PLMs in future. Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs. Experimental results demonstrate the superiority of KI in training efficiency. We also conduct empirical analyses to explore the effects of teacher PLMs' pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI could be applied to domain adaptation and knowledge transfer. The implementation is publicly available at https://github.com/thunlp/Knowledge-Inheritance.
Recently, it has become a consensus in the NLP community to use pre-trained language models (PLMs) as the backbone for various downstream tasks (Han et al., 2021; Min et al., 2021). Despite
the great follow-up efforts of exploring various pre-training techniques and model architectures, researchers find that simply enlarging the model capacity, data size and training steps can further improve the performance of PLMs (Kaplan et al., 2020; Li et al., 2020b). This discovery sets off a wave of training large-scale PLMs (Raffel et al., 2019; Brown et al., 2020; Fedus et al., 2021).
Although huge PLMs have shown awesome performance (Bommasani et al., 2021), it requires tremendous computational resources to train large-scale PLMs (Schwartz et al., 2019), raising severe environmental concerns on the prohibitive computational costs. Moreover, existing PLMs are generally trained from scratch individually, ignoring that many well-trained PLMs are available. This leaves us an important question: how could existing PLMs benefit training larger PLMs in future?
Considering that humans can leverage the knowledge summarized by their predecessors to learn new tasks, so that the learning process could become efficient; similarly, it is worth inheriting the implicit knowledge distributed in existing PLMs. In this sense, we could distill the knowledge summarized by an existing small PLM during pretraining to efficiently learn larger PLMs. We dub the above process as knowledge inheritance (KI). This intuition is similar to reversed KD (Yuan et al., 2020) in the field of computer vision. They indicate that a delicate student model could still benefit from a teacher with an inferior architecture for a specific downstream task.
However, the success of reversed KD in supervised downstream tasks does not guarantee its feasibility under the scenario of large-scale self-supervised pre-training. Therefore, in this paper,
†Part of the work was done while Peng Li was working at Tencent.
Human Language Technologies, pages 3921 - 3937
we strive to answer the following research questions: (RQ1) could distilling knowledge from an existing trained PLM benefit large PLMs' training from scratch? (RQ2) Considering human beings are able to hand down knowledge from generation to generation, could KI similarly be sequentially performed among a series of PLMs with growing sizes? (RQ3) As more and more PLMs with different pre-training settings (model architectures, training data, training strategies, etc) emerge, how would different settings affect the performance of KI? (RQ4) Besides training a large PLM from scratch, when adapting an already trained large PLM to a new domain, how could smaller domain teachers benefit such a process?
In conclusion, the contributions of this paper are summarized as follows: (1) we are the first to formulate the problem of knowledge inheritance, and demonstrate the feasibility of inheriting the knowledge from previously trained PLMs for efficiently training larger ones; (2) we show that the learned knowledge in PLMs could accumulate and further be passed down from generation to generation; (3) we systematically conduct empirical analyses to show the effects of various teacher pretraining settings, which may indicate how to select the most appropriate PLM as the teacher for KI; (4) we further show that during domain adaptation, an already trained large PLM could benefit from multiple small PLMs of different domains under the KI framework. The above empirical studies indicate that KI can well support cross-model knowledge transfer, providing a promising direction to share the knowledge learned by different PLMs and continuously promote their performance.
Efficient Pre-training for NLP. Recently, researchers find that the performance of PLMs can be simply improved by increasing the model size, data size and training steps (Liu et al., 2019; Raffel et al., 2019; Kaplan et al., 2020), sparking a wave of training ever-larger PLMs. For instance, the revolutionary GPT-3 (Brown et al., 2020), which contains 175 billion parameters, shows strong capabilities for language understanding and generation. This means that utilizing PLMs with huge parameters for downstream tasks may greatly relieve the cost of manual labeling and model training for new tasks. However, larger models require greater computational demands (Patterson et al., 2021). To this
end, researchers propose to accelerate pre-training by mixed-precision training (Shoeybi et al., 2019), distributed training (Shoeybi et al., 2019), large batch optimization (You et al., 2020), etc.
Another line of methods (Gong et al., 2019; Gu et al., 2021; Chen et al., 2022; Qin et al., 2022) proposes to pre-train larger PLMs progressively. They first train a small PLM, and then gradually increase the depth or width of the network based on parameter recycling (PR). Although PR could be used for the goal of KI, these methods typically have strict requirements on the architectures of both models, which is not flexible for practical uses; instead, we resort to KD as the solution for KI without architecture constraints. In addition, different from KI, PR is not applicable for absorbing knowledge from multiple teacher models and domain adaptation. More detailed comparisons between KI and PR are discussed in appendix E.
Knowledge Distillation for PLMs. Knowledge Distillation (KD) (Hinton et al., 2015) aims to compress a large model into a fast-to-execute one. KD has renewed a surge of interest in PLMs recently. Some explore KD at different training phases, e.g., pre-training (Sanh et al., 2019), downstream finetuning (Sun et al., 2019; Krishna et al., 2020), or both of them (Jiao et al., 2020); others explore distilling not only the final logits output by the large PLM, but also the intermediate hidden representations (Sanh et al., 2019; Jiao et al., 2020; Sun et al., 2020). Conventional KD presumes that teacher models play pivotal roles in mastering knowledge, and student models generally cannot match their teachers in performance. When it comes to the scenario of KI, since student models have larger capacities, the performance of teacher models is no longer an "upper bound" of student models. Outside NLP, researchers recently demonstrate that a student model could also benefit from a poor teacher for a specific downstream task (Yuan et al., 2020) (reversed KD). Based on the prior explorations, in this paper, we investigate the application of reversed KD in pre-training.
3 Knowledge Inheritance
Task Formulation. Given a textual input  $\mathbf{x} = \{x^{1},\dots ,x^{n}\}$  and the corresponding label  $\mathbf{y}\in \mathbb{R}^{K}$ , where  $K$  is the number of classes for the specific pre-training task, e.g., the vocabulary size for masked language modeling (MLM) (Devlin et al., 2019), a PLM  $\mathcal{M}$  converts each token
$x^{j}\in \mathbf{x}$  to task-specific logits  $\mathbf{z}^j = [z_1^j,\dots,z_K^j ]$ $\mathbf{z}^j$  is then converted to a probability distribution  $\mathcal{P}(x^j;\tau) = [p_1(x^j;\tau),\dots,p_K(x^j;\tau)]$  using a softmax function with temperature  $\tau$  .  $\mathcal{M}$  is pre-trained with the objective  $\mathcal{L}_{\mathrm{SELF}}(\mathbf{x},\mathbf{y}) = \mathcal{H}(\mathbf{y},\mathcal{P}(\mathbf{x};\tau))$  where  $\mathcal{H}$  is the loss function, e.g., cross-entropy for MLM. Assume that we have a well-trained small PLM  $\mathcal{M}_S$  optimized with the self learning objective  $\mathcal{L}_{\mathrm{SELF}}$  (such as MLM), our goal is leveraging  $\mathcal{M}_S$  's knowledge to efficiently train a larger PLM  $\mathcal{M}_L$  on the corpora  $\mathcal{D}_L = \{(x_i,y_i)\}_{i = 1}^{|\mathcal{D}_L|}$
Investigated Methodology. Specifically, imparting  $\mathcal{M}_S$  's knowledge to  $\mathcal{M}_L$  on  $\mathcal{D}_L$  is implemented by minimizing the Kullback-Leibler (KL) divergence between two probability distributions output by  $\mathcal{M}_S$  and  $\mathcal{M}_L$  on the same input  $\mathbf{x}_i\in \mathcal{D}_L$  , i.e.,  $\mathcal{L}_{\mathrm{KI}}(\mathbf{x}_i;\mathcal{M}_S) =$ $\tau^2\mathrm{KL}(\mathcal{P}_{\mathcal{M}_S}(\mathbf{x}_i;\tau)||\mathcal{P}_{\mathcal{M}_L}(\mathbf{x}_i;\tau)))$  . In addition,  $\mathcal{M}_L$  is also encouraged to conduct self-learning by optimizing  $\mathcal{L}_{\mathrm{SELF}}(\mathbf{x}_i,\mathbf{y}_i)$  . Both  $\mathcal{L}_{\mathrm{SELF}}$  and  $\mathcal{L}_{\mathrm{KI}}$  are balanced with an inheritance rate  $\alpha$  ..
$$
\begin{array}{l} \mathcal {L} (\mathcal {D} _ {L}; \mathcal {M} _ {S}) = \sum_ {(\mathbf {x} _ {i}, \mathbf {y} _ {i}) \in \mathcal {D} _ {L}} (1 - \alpha) \mathcal {L} _ {\mathrm {S E L F}} (\mathbf {x} _ {i}, \mathbf {y} _ {i}) + \alpha \mathcal {L} _ {\mathrm {K I}} (\mathbf {x} _ {i}; \mathcal {M} _ {S}) \\ = \sum_ {\left(\mathbf {x} _ {i}, \mathbf {y} _ {i}\right) \in \mathcal {D} _ {L}} (1 - \alpha) \mathcal {H} \left(\mathbf {y} _ {i}, \mathcal {P} _ {\mathcal {M} _ {L}} \left(\mathbf {x} _ {i}; 1\right)\right) \tag {1} \\ + \alpha \tau^ {2} \mathrm {K L} \left(\mathcal {P} _ {\mathcal {M} _ {S}} \left(\mathbf {x} _ {i}; \tau\right) | | \mathcal {P} _ {\mathcal {M} _ {L}} \left(\mathbf {x} _ {i}; \tau\right)\right)). \\ \end{array}
$$
Since larger models generally converge faster and can achieve better final performance (Li et al., 2020b),  $\mathcal{M}_L$  becomes more and more knowledgeable during the learning process, and would surpass the teacher eventually. Thus, it is necessary to encourage  $\mathcal{M}_L$  increasingly learning knowledge on its own, not only following the teacher's instructions. Additionally, after  $\mathcal{M}_L$  has surpassed its teacher, it no longer needs the guidance from  $\mathcal{M}_S$  and should conduct pure self-learning from then on. Therefore, different from reversed KD, we dynamically change the inheritance rate  $\alpha$ . Specifically, for a total training steps of  $T$ , we linearly decay  $\alpha_{t}$  with a slope of  $\frac{\alpha_T}{T}$ . The student only inherits knowledge from the teacher for  $\frac{T}{\alpha_T}$  steps, and then conducts pure self-learning, i.e.,  $\alpha_{t} = \max(1 - \alpha_{T} \times \frac{t}{T}, 0)$ . Formally, at step  $t$ , the loss function for inheriting knowledge of  $\mathcal{M}_S$  on  $\mathcal{D}_L$  is formulated as:
$$
\mathcal {L} \left(\mathcal {D} _ {L}; \mathcal {M} _ {S}\right) = \sum_ {\left(\mathbf {x} _ {i}, \mathbf {y} _ {i}\right) \in \mathcal {D} _ {L}} \left(1 - \alpha_ {t}\right) \mathcal {L} _ {\text {S E L F}} \left(\mathbf {x} _ {i}, \mathbf {y} _ {i}\right) + \alpha_ {t} \mathcal {L} _ {\mathrm {K I}} \left(\mathbf {x} _ {i}; \mathcal {M} _ {S}\right). \tag {2}
$$
Note the logits of  $\mathcal{M}_S$  on  $\mathcal{D}_L$  can be precomputed and saved offline so that we do not need
to re-compute the logits of  $\mathcal{M}_S$  when training  $\mathcal{M}_L$ . This process is done once and for all.
4 Empirical Analysis
In this section, we answer our research questions proposed before. Specifically, (1) we first demonstrate the effectiveness of KI in § 4.1. (2) Then we show PLMs can accumulate knowledge over generations in § 4.2. (3) We also investigate the effects of different pre-training settings of the teacher models in § 4.3. (4) Finally, we show that KI could benefit domain adaptation, and a trained PLM can learn more efficiently with the help of multiple domain teachers in § 4.4. Detailed pre-training hyper-parameters are listed in appendix B.
4.1 RQ1: How Could Knowledge Inheritance Benefit Large PLMs' Training?
Setting. Our KI framework is agnostic to the specific self-supervised pre-training task and the PLM architecture. Without loss of generality, we mainly focus on the representative MLM task and use the model architecture of RoBERTa (Liu et al., 2019). Specifically, we first choose RoBERTa<sub>BASE</sub> (denoted as BASE) as the teacher (MS) and RoBERTa<sub>LARGE</sub> (denoted as LARGE) as the student (ML). We also experiment on auto-regressive language modeling using GPT (Radford et al., 2018) to show KI is model-agnostic.
For pre-training data, we use the concatenation of Wikipedia and BookCorpus (Zhu et al., 2015) same as BERT (Devlin et al., 2019), with roughly 3,400M tokens in total. All models  $(\mathcal{M}_S$  and  $\mathcal{M}_L)$  are trained for  $125\mathrm{k}$  steps, with a batch size of 2,048 and a sequence length of 512. Note the whole training computations are comparable with those of BERT. We pre-train  $\mathcal{M}_L$  by inheriting  $\mathcal{M}_S$ 's knowledge under KI (denoted as "BASE  $\rightarrow$  LARGE"). We compare it with "LARGE" that only conducts self-learning from beginning to end.
For performance evaluation, we report the validation perplexity (PPL) during pre-training and the downstream performance on development sets of eight GLUE (Wang et al., 2019) tasks. Note compared with the self-learning baseline, in KI, the logits output by  $\mathcal{M}_L$  are additionally used to calculate  $\mathcal{L}_{\mathrm{KI}}$ , we empirically find that the additional computations caused by it are almost negligible compared with the cumbersome computations in Transformer blocks. Therefore, it requires almost the same computational cost between KI and the baseline for
(b)
Figure 1: (a) The validation PPL curve for pre-training  $\mathcal{M}_L$  under KI framework (BASE  $\rightarrow$  LARGE) and the self-learning baseline (LARGE). The teacher's (BASE) performance is 4.18. (b) Pre-training BASE under KI with three strategies for the inheritance rate  $\alpha_{t}$ : Linear, Heaviside and Constant. The teacher's (MEDIUM) performance is 4.95. (c) Pre-training BASE under KI with top- $K$  logits, we vary  $K$  in  $\{10, 50, 100, 1000\}$ , respectively.
each step. Hence, we report the performance w.r.t training step (Li et al., 2020a), while the performance w.r.t. FLOPs (Schwartz et al., 2019) and wall-clock time (Li et al., 2020b) can be roughly obtained by stretching the figure horizontally.
Overall Results. As shown in Figure 1 (a), we conclude that: (1) training  $\mathcal{M}_L$  under KI converges faster than the self-learning baseline, indicating that inheriting the knowledge from an existing teacher is far more efficient than solely learning such knowledge. That is, to achieve the same level of validation PPL, KI requires fewer computational costs. Specifically, under the guidance of  $\mathcal{M}_S$ , whose validation PPL is 4.18, BASE  $\rightarrow$  LARGE achieves a validation PPL of 3.41 at the end of pre-training, compared with baseline (LARGE) 3.58. After BASE  $\rightarrow$  LARGE stops learning from the teacher at the  $40\mathrm{k}$ -th step, it improves the validation PPL from 4.60 (LARGE) to 4.28, which is almost the performance when the baseline LARGE conducts self-learning for  $55\mathrm{k}$  steps, thus saving roughly  $27.3\%$  computational costs<sup>1</sup>. The results in Table 1 show that (2)  $\mathcal{M}_L$  trained under KI achieves better performance than the baseline on downstream tasks at each step. We also found empirically that, under the same setting (e.g., data, hyper-parameters and model architectures), lower validation PPL generally indicates better downstream performance. Since the performance gain in downstream tasks is consistent with that reflected in PPL, we only show the latter for the remaining experiments. Concerning the energy cost, for the remaining experiments, unless otherwise specified, we choose MEDIUM (9 layers, 576 hidden size) as
$\mathcal{M}_S$  and BASE as  $\mathcal{M}_L$ .
Effects of Inheritance Rate. We set  $\alpha_{t}$  in Eq. (2) to be linearly decayed (denoted as Linear) to gradually encourage  $\mathcal{M}_L$  exploring knowledge on its own. We analyze whether this design is necessary by comparing it with two other strategies: the first is to only learn from the teacher at first and change to pure self-learning (denoted as Heviside) at the 35k-th step; the second is to use a constant ratio  $(1:1)$  between  $\mathcal{L}_{\mathrm{SELF}}$  and  $\mathcal{L}_{\mathrm{KI}}$  throughout the whole training process (denoted as Constant). We can conclude from Figure 1 (b) that: (1) annealing at first is necessary. The validation PPL curve of Linear converges the fastest, while Heviside tends to increase after  $\mathcal{M}_L$  stops learning from the teacher, indicating that, due to the difference between learning from the teacher and self-learning, annealing at first is necessary so that the performance won't decay at the transition point (the 35k-th step). (2) Supervision from the teacher is redundant after  $\mathcal{M}_L$  surpasses  $\mathcal{M}_S$ . Although Constant performs well in the beginning, its PPL gradually becomes even worse than the other two strategies. This indicates that after  $\mathcal{M}_L$  has already surpassed  $\mathcal{M}_S$ , it will be encumbered by keeping following guidance from  $\mathcal{M}_S$ .
Saving Storage Space with Top-K Logits. Loading the teacher  $\mathcal{M}_S$  repeatedly for KI is cumbersome, and an alternative way is to pre-compute and save the predictions of  $\mathcal{M}_S$  offline once and for all. We show that using the information of top- $K$  logits (Tan et al., 2019) can reduce the memory footprint without much performance decrease. Specifically, we save only top- $K$  probabilities of  $\mathcal{P}_S(x^j; \tau)$  followed by re-normalization, instead of the full distribution over all tokens. For RoBERTa,
<sup>1</sup>If we load BASE and compute its logits during pretraining,  $18.7\%$  FLOPs can be saved roughly, since the forward passes of the small teacher also take up a small part.
Step	Model	CoLA	MNLI	QNLI	RTE	SST-2	STS-B	MRPC	QQP	Avg
5k	LARGE	0.0	73.5	81.7	53.0	81.7	45.8	71.4	87.5	61.8
BASE → LARGE	17.4	75.8	83.4	54.7	85.7	72.0	72.6	88.6	68.8
45k	LARGE	61.8	84.9	91.7	63.4	92.9	88.6	87.7	91.5	82.8
BASE → LARGE	64.3	85.9	92.2	75.3	93.2	89.3	89.4	91.5	85.2
85k	LARGE	64.5	86.8	92.7	69.7	93.5	89.9	89.7	91.7	84.8
BASE → LARGE	65.7	87.2	93.0	77.0	94.3	90.0	90.4	91.8	86.2
125k	LARGE	64.3	87.1	93.2	73.4	94.1	90.3	90.1	91.8	85.5
BASE → LARGE	67.7	87.7	93.1	74.9	94.8	90.6	88.2	91.9	86.1
Table 1: Downstream performances on GLUE tasks (dev). KI requires fewer pre-training steps to get a high score after fine-tuning. Detailed results at different training steps are illustrated in appendix A.6.
the dimension of  $\mathcal{P}_S(x^j;\tau)$  is decided by its vocabulary size, which is around 50,000. We thus vary  $K$  in  $\{10,50,100,1000\}$  to see its effects in Figure 1 (c), from which we observe that: top-K logits contain the vast majority of information. Choosing a relatively small  $K$  (e.g., 10) is already good enough for inheriting knowledge from the teacher without much performance decrease. Previous work also indicates the relation between KD and label smoothing (Shen et al., 2021), however, we show in appendix F that the improvements of KI are not because of benefiting from optimizing smoothed targets, which impose regularization.
Experiments on GPT. To demonstrate that KI is model-agnostic, we conduct experiments on auto-regressive language modeling and choose GPT (Radford et al., 2018) architecture with growing sizes of  $\{73\mathrm{M}, 124\mathrm{M}, 209\mathrm{M}, 354\mathrm{M}, 773\mathrm{M}, 1\mathrm{B}\}$  parameters in total, respectively. The detailed architectures are specified in Table 6. All the teacher models are pre-trained for  $62.5\mathrm{k}$  steps with a batch size of 2, 048. As reflected in Figure 2 (a), training larger GPTs under our KI framework converges faster than the self-learning baseline, which demonstrates KI is agnostic to the specific pretraining objective and PLM architecture.
4.2 RQ2: Could Knowledge Inheritance be Performed over Generations?
Human beings can inherit the knowledge from their antecedents, refine it and pass it down to their offsprings, so that knowledge can gradually accumulate over generations. Inspired by this, we investigate whether PLMs also have this kind of pattern. Specifically, we experiment with the knowledge inheritance among three generations of RoBERTa with roughly 1.7x growth in model size:  $G_{1}$  (BASE, 125M),  $G_{2}$  (BASE_plus, 211M)
and  $G_{3}$  (LARGE, 355M), whose architectures are listed in Table 6. All models are trained from scratch for 125k steps with a batch size of 2,048 on the same corpus. We compare the differences among (1) self-learning for each generation (denoted as  $G_{1}$ ,  $G_{2}$  and  $G_{3}$ ), (2) KI over two generations (denoted as  $G_{1} \rightarrow G_{2}$ ,  $G_{1} \rightarrow G_{3}$  and  $G_{2} \rightarrow G_{3}$ ), and (3) KI over three generations (denoted as  $G_{1} \rightarrow G_{2} \rightarrow G_{3}$ ), where  $G_{2}$  first inherits the knowledge from  $G_{1}$ , refines it by additional self-exploring and passes its knowledge down to  $G_{3}$ . The results are drawn in Figure 2 (b). Comparing the performance of  $G_{2}$  and  $G_{1} \rightarrow G_{2}$ ,  $G_{3}$  and  $G_{1} \rightarrow G_{3}$ , or  $G_{3}$  and  $G_{2} \rightarrow G_{3}$ , we can again demonstrate the superiority of KI over self-training as concluded before. Comparing the performance of  $G_{1} \rightarrow G_{3}$  and  $G_{1} \rightarrow G_{2} \rightarrow G_{3}$ , or  $G_{2} \rightarrow G_{3}$  and  $G_{1} \rightarrow G_{2} \rightarrow G_{3}$ , it is observed that the performance of  $G_{3}$  benefits from the involvements of both  $G_{1}$  and  $G_{2}$ , which means knowledge could be accumulated through more generations' involvements.
4.3 RQ3: How Could  $\mathcal{M}_S$ 's Pre-training Setting Affect Knowledge Inheritance?