SlowGuess/NAACL_2022_OCR huggingface dataset:
Existing PLMs are typically trained under quite different settings, and it is unclear how these different settings will affect the performance of KI. Formally, we have a series of well-trained smaller PLMs  $\overline{\mathcal{M}_S} = \{\mathcal{M}_S^1,\dots,\mathcal{M}_S^{N_S}\}$ , each having been optimized on  $\overline{\mathcal{D}_S} = \{\mathcal{D}_S^1,\dots,\mathcal{D}_S^{N_S}\}$ , respectively. Considering that the PLMs in  $\overline{\mathcal{M}_S}$ , consisting of varied model architectures, are pre-trained on different corpora of various sizes and domains with arbitrary strategies, thus the knowledge they master is also manifold. In addition,  $\mathcal{M}_L$ 's pre-training data  $\overline{\mathcal{D}_L}$  may also consist of massive, heterogeneous corpora from multiple sources, i.e.,  $\overline{\mathcal{D}_L} = \{\mathcal{D}_L^1,\dots,\mathcal{D}_L^{N_L}\}$ . Due to the difference between  $\overline{\mathcal{D}_L}$  and  $\overline{\mathcal{D}_S}$ ,  $\mathcal{M}_S$
(c)
Figure 2: (a) Experiments on GPT. (b) KI over generations. (c) Effects of  $\mathcal{M}_S$ 's architecture (depth).
may be required to transfer its knowledge on instances unseen during its pre-training. Ideally, we want  $\mathcal{M}_S$  to teach the courses it is skilled in. Therefore, it is essential to choose the most appropriate teacher for each composition  $\mathcal{D}_L^* \in \overline{\mathcal{D}_L}$ . To this end, we conduct thorough experiments to analyze the effects of several representative factors: model architecture, pre-training data,  $\mathcal{M}_S$ 's pre-training step (appendix A.2) and batch size (appendix A.3).
Effects of Model Architecture. Large PLMs generally converge faster and achieve lower PPL, thus serving as more competent teachers. We experiment with two widely chosen architecture variations, i.e., depth (number of layers) and width (hidden size), to explore the effects of  $\mathcal{M}_S$ 's model architectures. We choose BASE (12 layer, 768 hidden size) as  $\mathcal{M}_L$ 's architecture, and choose the architecture of  $\mathcal{M}_S$  to differ from  $\mathcal{M}_L$  in either depth or width. Specifically, for  $\mathcal{M}_S$ , we vary the depth in  $\{4,6,8,10\}$ , and the width in  $\{384,480,576,672\}$ , respectively, and pre-train  $\mathcal{M}_S$  under the same setting as  $\mathcal{M}_L$ . The PPL curve for each teacher model is shown in appendix A.7, from which we observe that deeper / wider teachers with more parameters converge faster and achieve lower PPL. After that, we pre-train  $\mathcal{M}_L$  under KI leveraging these teacher models. As shown in Figure 2 (c) and appendix A.4, choosing a deeper / wider teacher accelerates  $\mathcal{M}_L$ 's convergence, demonstrating the benefits of learning from a more knowledgeable teacher. Since the performance of PLMs is weakly related to the model shape but highly related to the model size (Li et al., 2020b), it is always a better strategy to choose the larger teacher if other settings are kept the same. In experiments, we also find empirically that, the optimal duration of learning from the teacher is longer for larger teachers, which means it takes more time to learn from a more knowledgeable teacher.
Effects of Pre-training Data. In previous experiments, we assume  $\mathcal{M}_L$  is pre-trained on the same corpus as  $\mathcal{M}_S$ , i.e.,  $\mathcal{D}_L = \mathcal{D}_S$ . However, in real-world scenarios, it may occur that the pre-training corpus used by both  $\mathcal{M}_L$  and  $\mathcal{M}_S$  is mismatched, due to three main factors: (1) data size. When training larger models, the pre-training corpus is often enlarged to improve downstream performance, i.e.,  $|\mathcal{D}_S| \ll |\mathcal{D}_L|$ ; (2) data domain. PLMs are trained on heterogeneous corpora from various sources (e.g., news articles, literary works, etc.), i.e.,  $\mathcal{P}_{\mathcal{D}_S} \neq \mathcal{P}_{\mathcal{D}_L}$ . The different knowledge contained in each domain may affect PLMs' generalization in downstream tasks; (3) data privacy. Even if both size and domain of  $\mathcal{D}_S$  and  $\mathcal{D}_L$  are ensured to be the same, it may be hard to retrieve the pre-training corpus used by  $\mathcal{M}_S$  due to privacy concerns, with an extreme case:  $\mathcal{D}_L \cap \mathcal{D}_S = \emptyset$ . The gap between  $\mathcal{D}_S$  and  $\mathcal{D}_L$  may hinder  $\mathcal{M}_S$ 's successful knowledge transfer. We thus design experiments to analyze the effects of these factors, with three observations concluded:
- Obs. 1: PLMs can image the big from the small for in-domain data. To evaluate the effects of data size, we first pre-train teacher models on different partitions of the original training corpus under the same setting by randomly sampling  $\{\frac{1}{16}, \frac{1}{8}, \frac{1}{4}, \frac{1}{2}, \frac{1}{1}\}$  of it, resulting in teacher models with final validation PPL of  $\{5.43, 5.15, 5.04, 4.98, 4.92\}$ , respectively. The final validation PPL increases as we shrink the size of  $\mathcal{M}_S$ 's pre-training corpus, which implies that training with less data weakens the teacher's ability. Next, we compare the differences when their knowledge is inherited by  $\mathcal{M}_L$ . As reflected in Figure 3 (a), however, the performance of KI is not substantially undermined until only  $\frac{1}{16}$  of the original data is leveraged by the teacher. This indicates that PLMs can well image the overall data distribution even if it only sees a small part. Hence,
(c)
Figure 3: Effects of  $\mathcal{M}_S$  's pre-training (a) data size, (b) data domain and (c) data privacy for KI.
when training larger PLMs, unless the data size is extensively enlarged, its impact can be ignored.
- Obs. 2: Inheriting on similar domain improves performance. To evaluate the effects of data domain, we experiment on the cases where  $\mathcal{D}_S$  and  $\mathcal{D}_L$  have domain mismatch. Specifically, keeping data size the same, we mix Wikipedia and BookCorpus (WB) used before with computer science (CS) papers from S2ORC (Lo et al., 2020), whose domain is distinct from WB, using different proportions, i.e., WB : CS = {1 : 2, 2 : 1, 3 : 1, 4 : 1}, respectively. We pre-train  $\mathcal{M}_S$  on the constructed corpora, then test the performance when  $\mathcal{M}_L$  inherits these teachers' knowledge on the WB domain data. As shown in Figure 3 (b), with the domain of the constructed corpus  $\mathcal{M}_S$  is trained on becoming gradually similar to WB, the benefits from KI become more obvious, which means it is essential that both  $\mathcal{M}_S$  and  $\mathcal{M}_L$  are trained on similar domain of data, so that  $\mathcal{M}_S$  can successfully impart knowledge to  $\mathcal{M}_L$  by teaching the "right" course.
- Obs. 3: Data privacy is not that important if the same domain is ensured. To evaluate the effects of data privacy, we experiment in an extreme case where  $\mathcal{D}_S$  and  $\mathcal{D}_L$  have no overlap at all. To avoid the influences of size and domain, we randomly split the WB domain training corpus  $\mathcal{D}$  into two halves ( $\mathcal{D}_A$  and  $\mathcal{D}_B$ ) and pre-train two teacher models (denoted as MEDIUM_A and MEDIUM_B) on them. After pre-training, both of them achieve almost the same final PPL (4.99) on the same validation set. They are then inherited by the student model BASE on  $\mathcal{D}_B$  (denoted as MEDIUM_A → BASE_B and MEDIUM_B → BASE_B), which is exactly the pre-training corpus of MEDIUM_B and has no overlap with that of MEDIUM_A. We also choose  $\mathcal{M}_L$  that conducts pure self-learning on  $\mathcal{D}_B$  as the baseline (denoted as BASE_B). It is observed from Figure 3 (c) that, there is little difference between the validation PPL curves of MEDIUM_A → BASE_B
and  $\mathrm{MEDIUM}_{\mathrm{B}} \rightarrow \mathrm{BASE}_{\mathrm{B}}$ , indicating that whether the pre-training corpus of  $\mathcal{M}_S$  and  $\mathcal{M}_L$  has data overlap or not is not a serious issue as long as they share the same domain. This is meaningful when organizations aim to share the knowledge of their PLMs without exposing either the pre-training data or the model parameters due to privacy concerns.
4.4 RQ4: How Could Knowledge Inheritance Benefit Domain Adaptation?
With streaming data of various domains continuously emerging, training domain-specific PLMs and storing the model parameters for each domain can be prohibitively expensive. To this end, researchers recently demonstrated the feasibility of adapting PLMs to the target domain through continual pre-training (Gururan et al., 2020). In this section, we further extend KI and demonstrate that domain adaptation for PLM can benefit from inheriting knowledge of existing domain experts.
Specifically, instead of training large PLMs from scratch, which is the setting used before, we focus on adapting  $\mathrm{BASE}_{\mathrm{WB}}$ , which has been well-trained on the WB domain for 125k steps, to two target domains, i.e., computer science (CS) and biomedical (BIO) papers from S2ORC (Lo et al., 2020). The proximity (vocabulary overlap) of three domains is listed in appendix D. We assume there exist two domain experts, i.e.,  $\mathrm{MEDIUM}_{\mathrm{CS}}$  and  $\mathrm{MEDIUM}_{\mathrm{BIO}}$ . Each model has been trained on CS / BIO domain for 125k steps. Note their training computation is far less than  $\mathrm{BASE}_{\mathrm{WB}}$  due to fewer model parameters. Hence, either  $\mathrm{MEDIUM}_{\mathrm{CS}}$  or  $\mathrm{MEDIUM}_{\mathrm{BIO}}$  is no match for  $\mathrm{BASE}_{\mathrm{WB}}$  in WB domain but has richer knowledge in CS / BIO domain. For evaluation, we compare both (1) the validation PPL on the target domain and (2) the performance (test F1) on downstream tasks, i.e. ACL-ARC (Jurgens et al., 2018) for CS domain and CHEMPROT (Kringelum et al., 2016) for BIO domain. Before adaptation,
\(N_{tokens}\)	3,400M	200M	100M	40M	20M
Metrics	F1	PPL	F1	PPL	F1	PPL	F1	PPL	F1	PPL
CS	SL	69.8	3.12	71.7	3.17	71.4	3.24	68.3	3.51	67.5	4.07
KI	72.9	3.06	72.6	3.09	71.9	3.11	71.1	3.21	70.8	3.37
BIO	SL	84.0	2.67	82.8	2.72	83.2	2.83	83.3	3.16	82.7	3.81
KI	84.5	2.65	83.4	2.66	83.9	2.69	83.6	2.82	83.5	3.01
Table 2: The validation PPL (PPL) and downstream performance (F1) on the target domain (CS / BIO) after  $\mathrm{BASE}_{\mathrm{WB}}$  is post-trained for 4k steps with self-learning (SL) or knowledge inheritance (KI). We experiment with different sizes of domain corpus. All downstream experiments are repeated 10 times with different seeds.
\(N_{tokens}\)	3,400M	200M	100M	40M
Metrics	\(F1_C\)	\(PPL_C\)	\(F1_B\)	\(PPL_B\)	\(F1_C\)	\(PPL_C\)	\(F1_B\)	\(PPL_B\)	\(F1_C\)	\(PPL_C\)	\(F1_B\)	\(PPL_B\)	\(F1_C\)	\(PPL_C\)	\(F1_B\)	\(PPL_B\)
SL	71.7	3.15	83.7	2.71	70.5	3.97	82.7	3.36	67.7	5.95	81.7	4.84	68.3	11.7	81.1	10.5
KI	72.2	3.15	83.9	2.70	71.8	3.42	83.1	2.92	69.8	3.90	82.6	3.32	69.1	5.70	81.3	4.64
Table 3: The results when  $\mathrm{BASE}_{\mathrm{WB}}$  is post-trained on two new domains simultaneously with self-learning (SL) or knowledge inheritance (KI). We report both validation PPL  $(\mathrm{PPL}_{\mathrm{B}} / \mathrm{PPL}_{\mathrm{C}})$  and downstream performance  $(\mathrm{F1}_{\mathrm{B}} / \mathrm{F1}_{\mathrm{C}})$  for BIO / CS domain. We observe that SL exhibits severe overfitting when data is relatively scarce.
$\mathrm{BASE}_{\mathrm{WB}}$  achieves a PPL of 5.41 / 4.86 and F1 of 68.5 / 81.6 on CS / BIO domain, while  $\mathrm{MEDIUM}_{\mathrm{CS}}$  achieves 2.95 (PPL) and 69.4 (F1) on CS domain,  $\mathrm{MEDIUM}_{\mathrm{BIO}}$  achieves 2.55 (PPL) and 83.6 (F1) on BIO domain. This demonstrates the superiority of two teachers over the student in their own domain despite their smaller model capacity.
We compare two strategies for domain adaptation: (1) only conducting self-learning on the target domain and (2) inheriting knowledge from well-trained domain teachers. Specifically,  $\mathrm{BASE}_{\mathrm{WB}}$  is post-trained for additional 4k steps on either CS or BIO domain to learn new knowledge. In addition, considering that in real-world scenarios, it can be hard to retrieve enough pre-training data for a specific domain, due to some privacy issues. Hence, we conduct experiments with different sizes of domain corpus. In Table 2,  $\mathrm{BASE}_{\mathrm{WB}}$  is post-trained on either CS or BIO domain while in Table 3, it is trained on synthetic domain data (BIO : CS = 1 : 1) to absorb knowledge from two domains simultaneously (we assume  $\mathcal{M}_L$  is trained with the optimal teacher selection strategy, i.e., each teacher imparts the knowledge on its own domain data). It can be concluded from Table 2 and Table 3 that:
(1) KI is more training-efficient. Compared with self-learning, inheriting knowledge from domain teachers achieves lower final PPL and improved performance in domain-specific downstream tasks, indicating that, for domain adaptation, KI is more training-efficient so that an already trained large PLM could absorb more knowledge
from new domain with the same training budget. (2) KI is more data-efficient. The PPL gap between KI and SL is further enlarged when there is less domain-specific data available for adaptation, which means KI is more data-efficient especially under the low-resource setting, where domain data is scarce. In other words, only providing a small portion of domain-specific data is enough for satisfactory adaptation performance under KI, while self-learning exhibits overfitting to some extent. (3) Large PLMs can simultaneously absorb knowledge from multiple domains and thus become omnipotent. From Table 3, we observe  $\mathrm{BASE}_{\mathrm{WB}}$  achieves improved performance on both domains after being taught by two teachers simultaneously. KI shows superiority over self-learning. However, simultaneous learning overfits training data more easily and its performance on either domain is no match for learning only one domain at a time.
In this work, we propose a general knowledge inheritance (KI) framework that leverages previously trained PLMs for training larger ones. We conduct sufficient empirical studies to demonstrate its feasibility. In addition, we show that KI could well support knowledge transfer over a series of PLMs with growing sizes. We also comprehensively analyze various pre-training settings of the teacher model that may affect KI's performance, the results shed light on how to choose the most appropriate teacher PLM for KI. Finally, we ex
tend KI and show that, during domain adaptation, an already trained large PLM could benefit from smaller domain teachers. In general, we provide a promising direction to share and exchange the knowledge learned by different models and continuously promote their performance.
In future, we aim to explore the following directions: (1) the efficiency of KI, i.e., given limited computational budget and pre-training corpus, how to more efficiently absorb knowledge from teacher models. Potential solutions include denoising teacher models' predictions and utilizing more information from the teacher. How to select the most representative data points for KI is also an interesting topic; (2) the effectiveness of KI under different settings, i.e., how can KI be applied if the teachers and the students are pre-trained on different vocabularies, languages, pre-training objectives and modalities.
Finally, we believe it is vital to use fair benchmarking that can accurately and reliably judge each KI algorithm. Thus, we suggest future work to: (1) conduct all experiments under the same computation environment and report the pre-training hyper-parameters and hardware deployments in detail, (2) evaluate the downstream tasks with multiple different random seeds and choose tasks that give relatively stable and consistent results, which could serve as better indicators for PLMs' effectiveness. In addition, it is also essential that PLMs are tested on diverse downstream tasks which evaluate PLMs' different abilities, (3) save the checkpoint more frequently during pre-training and evaluate the downstream performance, which can better indicate the trend of PLMs' effectiveness, and (4) open-source all the codes and model parameters for future comparisons.
This work is supported by the National Key R&D Program of China (No. 2020AAA0106502), Institute Guo Qiang at Tsinghua University, NExT++ project from the National Research Foundation, Prime Minister's Office, Singapore under its IRC@Singapore Funding Initiative, and Beijing Academy of Artificial Intelligence (BAAI). This work is also supported by the Pattern Recognition Center, WeChat AI, Tencent Inc. Yujia Qin and Yankai Lin designed the methods and the experiments. Yujia Qin, Jing Yi and Jiajie Zhang conducted the experiments. Yujia Qin, Yankai Lin and
Xu Han wrote the paper. Zhiyuan Liu, Peng Li, Maosong Sun and Jie Zhou advised the project. All the authors participated in the discussion.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang, Zhi Wang, Xiao Chen, Zhiyuan Liu, and Qun Liu. 2022. bert2bert: Towards reusable pretrained language models. In Association for Computational Linguistics: ACL 2022, Online.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. *ArXiv preprint*, abs/2101.03961.
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1602-1611. PMLR.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Efficient training of BERT by progressively stacking. In Proceedings
of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2337-2346. PMLR.
Xiaotao Gu, Liyuan Liu, Hongkun Yu, Jing Li, Chen Chen, and Jiawei Han. 2021. On the transformer growth for progressive BERT training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5174-5180, Online. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. 2021. Pre-trained models: Past, present and future. ArXiv preprint, abs/2106.07139.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. *ArXiv* preprint, abs/1503.02531.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online. Association for Computational Linguistics.
David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391-406.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. ArXiv preprint, abs/2001.08361.
Jens Kringelum, Sonny Kim Kjaerulff, Soren Brunak, Ole Lund, Tudor I Oprea, and Olivier Taboureau. 2016. Chemprot-3.0: a global chemical biology diseases mapping. Database, 2016.
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit Iyyer. 2020. Thieves on sesame street! model extraction of bert-based apis. In 8th International Conference on
Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Mengtian Li, Ersin Yumer, and Deva Ramanan. 2020a. Budgeted training: Rethinking deep neural network training under resource constraints. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. 2020b. Train big, then compress: Rethinking model size for efficient training and inference of transformers. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5958-5968. PMLR.
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2021. A survey of transformers. ArXiv preprint, abs/2106.04554.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983, Online. Association for Computational Linguistics.
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heinz, and Dan Roth. 2021. Recent advances in natural language processing via large pre-trained language models: A survey. arXiv preprint arXiv:2111.01243.
David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. ArXiv preprint, abs/2104.10350.
Yujia Qin, Jiajie Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. Elle: Efficient lifelong pre-training for emerging data. In Findings of the Association for Computational Linguistics: ACL 2022, Online.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. *ArXiv preprint*, abs/1910.10683.