SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 3: Downstream task and MLM performance of  $f_{T}$  under different memory sizes  $|M|$  and the frequency of replay  $k$  (replaying every  $k$  steps of training) in ER.
tion. With these results, we address the research questions below.
Does lifelong pretraining help retain knowledge across different domain corpora? We first examine whether task-specific or lifelong pretraining improves performance over domain-specific downstream tasks. Comparing Task-Specific LMs with RoBERTa-base in Table 2, we notice consistent performance improvements, especially on Biomedical and Computer Science domains  $(D_{1}, D_{2})$ . We also see Sequential Pretraining could consistently outperform RoBERTa-base. However, the comparison between Sequential Pretraining and Task Specific LMs are mixed: on  $D_{1}, D_{2}, D_{3}$ , Sequential Pretraining could outperform Task-Specific LMs only except MNER; while on the earliest biomedical domain  $(D_{1})$ , Sequential Pretraining achieves substantially lower performance. From Figure 4, we see the performance of Sequential Pretraining on Chemprot and RCT (from  $D_{1}$ ) drops significantly from  $t = 1$  to 4. The results imply lifelong pretraining allows later domains to benefit from knowledge transfer from earlier domains, but the performance on earlier domains is limited because of forgetting.
Does continual learning algorithms help retain knowledge in sequential pretraining? Next, we compare different kinds of CL algorithms and investigate the effect of CL algorithms in alleviating forgetting and improving knowledge transfer. Ta
ble 2 shows that Online-EWC slightly improves MLM perplexity compared to Sequential PT, but brings no improvement to the fine-tuning performance. We hypothesize that regularization directly in the parameter space as in Online-EWC is not effective when the parameter space is very high dimensional. Adapter improves downstream task F1 scores on the bio-medical domain  $(D_{1})$  by  $1.2\%$  and  $0.8\%$ , but does not outperform Sequential Pretraining in other domains (similarly for Simple Layer Expansion approach), likely because a great portion of the model is kept frozen.
In contrast, the memory-replay based approach (ER) allows training the full parameters of the model and has been shown to be highly effective in continual learning of classification tasks (Wang et al., 2019; Chaudhry et al., 2019). However, we surprisingly find that ER could hardly improve over Sequential Pretraining except  $D_{1}$ . A similar pattern can be found in the MLM perplexity. We hypothesize that the positive effect of example replay has diminished because of the overfitting to the memory examples. Table 3 summarizes the effect of tuning hyperpameters in ER. When we reduce the frequency of replay (from every 10 steps to 100 steps), the MLM performance improves, which implies reduced overfitting; however, the performance of downstream task performance does not improve. When we increase the size of the memory  $|M|$  from  $100k$  to  $10M$ , the MLM perplexity also improves; still, there are still no improvements in downstream tasks. It may imply ER itself is not an effective approach for continual pretraining.
Unlike ER, distillation approaches utilize richer information such as output logits or representation similarity to preserve past knowledge. We find either Logit KD or SEED-Logit KD to be most
(a) Chemprot
(b) RCT-Sample
(c) ACL-ARC
(d) SciERC
Figure 4: Performance evolution of downstream models. Models are fine-tuned from checkpoints of lifelong pretrained LMs at different time steps  $t$ . For Chemprot and RCT-Sample from  $D_{1}$ , we use  $t \in \{1,2,3,4\}$ ; while for ACL-ARC and SciERC from  $D_{2}, t \in \{2,3,4\}$ . Methods achieving the best performance at the end of training ( $t = 4$ ) is highlighted.
Figure 5: Performance of downstream models with various number of training examples, exemplified with SciERC. The models are fine-tuned from the final pretrained model  $(f_4)$ .
effective depending on the task, while Rep-KD and Contrastive-KD are less effective. The best performing distillation approach improves F1 over Sequential Pretraining on downstream tasks from  $D_{1}$ ,  $D_{2}$  at least by  $1.0\%$ . However, performance on  $D_{3}$ ,  $D_{4}$ , which come later in the data stream, does not improve over Sequential Pretraining, possibly because the distillation loss term makes the model rigid in obtaining new knowledge.
What is the gap between lifelong pretraining and multi-task learning across all the domains?
Multi-Task Learning refers to the offline training paradigm, which retrain PTLMs over all corpora  $(D_{1:t})$  each time a new corpus  $D_{t}$  becomes available. We examine whether lifelong pretraining is comparable to multi-task pretraining in terms of performance. From Table 2 and Figure 4, we see Sequential Pretraining in general underperforms MTL except for the final domain. However, certain CL approaches, such as Logit-Distillation, could improve over MTL on all downstream tasks from the first and the second domain. We speculate the
reason is that continual learning naturally provides a curriculum (Xu et al., 2020; Shi et al., 2015) to models where each individual task is easier to learn. The results have a positive implication that lifelong pretraining is not only more computationally efficient and requires less storage of past data, but may also improve the performance of pretraining.
Does lifelong pretraining make models more data efficient? In Table 5, we further examine the performance of final pretrained models under different amounts of training examples. We include full results in Appendix B. We find in general, performance improvements are more significant in the low-resource setup.
Computational Costs. We quantify computational costs of different CL algorithms with the number of forward and backward passes in Table 4 and present additional experiments with controlled computational costs in Appendix F. We find additional computational cost is necessary for performance improvement of distillation-based CL. However, it is not possible to trade performance simply by investing more computation budget with arbitrary CL algorithms. We leave detailed discussions in Appendix F.
4.3 Temporal Data Stream
We conduct analysis on pretraining PTLM on chronologically-ordered tweet corpora, to understand whether lifelong pretraining helps adaptation to the latest data and improves temporal generalization ability. The results are summarized in Table 5.
Method	#. of Forward	#. of Backward	#. Total	#. Total (k=10)	Wall Time4k
Main results
Sequential PT	b	b	2b	2b	4.0 × 104sec.
ER	(1 + 1/k)b	(1 + 1/k)b	(2 + 2/k)b	2.2b	4.2 × 104sec.
Logit-Distill	(2 + 2/k)b	(1 + 1/k)b	(3 + 3/k)b	3.3b	6.9 × 104sec.
SEED-Logit-Distill	(3 + 3/k)b	(2 + 2/k)b	(5 + 5/k)b	5.5b	9.7 × 104sec.
Additional Controlled Experiments
Sequential PTb′=1.2b	1.2b	1.2b	2.4b	2.4b	4.4 × 104sec.
ERk=5	1.2b	1.2b	2.4b	2.4b	4.4 × 104sec.
Sparse Logit-KD	1.3b	1.1b	2.4b	2.4b	4.4 × 104sec.
Sparse SEED-Logit-KD\contrast	1.3b	1.1b	2.4b	2.4b	4.8 × 104sec.
Table 4: Number of forward and backward passes over PTLMs and wall clock time of different approaches. The number of forward and backwards passes are computed over visits of  $b$  batches from the training data stream, where  $k$  is the frequency of replay. The wall clock time is calculated over  ${4k}$  steps of training (which is the number of training steps of a single domain in the Research Paper stream) excluding the first domain, as no replay or distillation happens while learning the first domain. In the additional controlled experiments (described in Appendix. F), we control the total number of forward and backward passes of different approaches.
Years	2018 (D3)	2020 (D4)	2014 (D1) → 2020 (D4)	2016 (D2) → 2020 (D4)
Hashtag Prediction
RoBERTa-base	48.08±1.0	56.42±0.2	39.31±2.7	42.23±2.7
Sequential PT	56.79±0.5	59.85±0.4	44.00±1.1	49.87±1.8
ER	56.93±0.1	59.56±1.7	43.31±0.2	50.72±0.6
Logit-KD	58.21±0.5	60.52±0.2	44.26±0.9	50.92±0.8
Contrast-KD	57.94±0.4	59.54±0.3	45.22±0.1	52.14±1.1
SEED-KD	56.87±0.2	59.71±0.2	43.39±0.4	49.62±1.0
SEED-Logit-KD	57.75±0.4	60.74±0.6	45.35±0.6	51.56±0.7
Task-Specific (2014)	56.16±0.6	59.59±0.3	44.34±0.6	49.26±0.7
Task-Specific (Latest)	56.61±0.4	59.87±0.6	43.44±0.5	49.41±1.1
MTL	57.89±0.4	59.95±0.3	44.04±0.3	50.37±0.3
Emoji Prediction
RoBERTa-base	25.71±0.1	24.42±0.2	12.02±0.4	13.24±0.2
Sequential PT	29.30±0.1	27.69±0.1	14.20±0.2	16.08±1.4
ER	29.50±0.1	27.75±0.1	14.36±0.4	16.82±0.3
Logit-KD	29.77±0.1	27.80±0.1	14.20±0.3	16.28±1.1
Contrast-KD	29.48±0.2	27.72±0.3	14.42±0.3	17.52±0.1
SEED-KD	30.12±0.1	27.66±0.1	14.36±0.1	16.97±0.4
SEED-Logit-KD	29.98±0.1	27.84±0.2	14.36±0.1	16.97±0.3
Task-Specific (2014)	28.94±0.0	26.98±0.2	13.39±0.2	15.14±0.2
Task-Specific (Latest)	29.06±0.2	27.19±0.1	13.00±0.2	14.48±0.3
MTL	29.52±0.2	27.47±0.0	14.07±0.2	16.64±0.2
Table 5: Results on temporal data stream. We show fine-tuning performance over years 2018 and 2020  $(D_{3}, D_{4})$  and the Temporal generalization from 2014 or 2016 to 2020 data  $(D_{1} \rightarrow D_{4}, D_{2} \rightarrow D_{4})$  on Twitter Hashtag and Emoji prediction datasets. Models are fine-tuned from the final pre-trained model  $f_{T}$ . We include full results on other years  $(D_{1}, D_{2}, D_{3} \rightarrow D_{4})$  in Appendix C.
Will LMs be outdated? We compare the performance of Task-Specific (2014) to the Task-Specific models pretrained on the year of downstream datasets (noted as Task-Specific (Latest)) and notice consistent improvements in downstream tasks in 2018 and 2020 (first two columns in Table 5). Sequential Pretraining could also outperform the Task-Specific (2014) model. It verifies that language models may get outdated over time, but the issue can be addressed by task-specific or lifelong pretraining over the latest corpora.
Does lifelong pretraining help improve the downstream model's performance on latest data? We show that downstream model's performance over later data  $(D_{3}, D_{4})$  can be improved over Task-Specific models when continual learning algorithms are applied. From the first two columns of Table 5, we see Logit-KD and SEED-KD improve Hashtag prediction score over data of years 2018 and 2020. SEED-Logit KD further improves prediction F1 on Emoji prediction. Note that these findings are in contrast to the research paper stream, where CL algorithms do not improve performance in the latest domain  $D_{4}$ . The reason can be the higher similarity between domains in the tweet corpora making the knowledge transfer easier, which is further discussed in Appendix I.
Does lifelong pretraining improve temporal generalization? Temporal generalization evaluates downstream performance over latest test data when fine-tuned over outdated training data. We show lifelong pretraining brings clear improvement to temporal generalization. From Table 5, we see even Sequential Pretraining could improve over the model pretrained merely on the year 2020 data (Task-Specific (2020)) consistently. We find performance further improves with CL algorithms applied. SEED-Logit-KD performs best in general on crossyear hashtag prediction tasks. In crossyear emoji prediction, we find Contrast-KD and SEED-KD perform best. We also find that SEED-Logit-KD could slightly outperform Logit-KD.
Domain and Temporal Adaptation of Language Models. Gururangan et al. (2020) study adapta
tion of PTLMs to domain-specific corpora. Aruma et al. (2020) study algorithms to mitigate forgetting in original PTLMs, but does not investigate forgetting that happens over a sequence of domains. Maronikolakis and Schütze (2021); Röttger and Pierrehumbert (2021); Luu et al. (2021) proposes sequential pretraining over domains or emerging data, but did not investigate CL algorithms. Several recent studies have demonstrated the necessity of adapting LMs over time (Lazaridou et al., 2021) while specifically focusing on factual knowledge (Dhingra et al., 2021; Jang et al., 2021).
Continual Learning Algorithms in NLP. Continual learning in NLP has mainly been studied for classification tasks. An effective approach is to utilize a number of stored past examples (de Masson d'Autume et al., 2019; Wang et al., 2020), or pseudo examples (e.g., the ones generated with a PTLM (Sun et al., 2020; Kanwatchara et al., 2021)). Recent extensions of the algorithm (Chuang et al., 2020) perform knowledge distillation with generated pseudo examples. Other lines of works focus on regularization over the sentence representations (Wang et al., 2019; Huang et al., 2021; Liu et al., 2019a) or directly merging models in the parameter space (Matena and Raffel, 2021). Model expansion-based approaches (Liu et al., 2019a; Pfeiffer et al., 2021), including learning domain specific expert models (Gururangan et al., 2021), are also actively studied. Wu et al. (2022) present a comparative study of algorithms in the context of continual fine-tuning over NLP tasks.
In this paper, we formulated the lifelong language model pretraining problem and constructed two data streams associated with downstream datasets. We evaluated knowledge retention, adaptation to the latest data, and temporal generalization ability of continually pretrained language models. Our experiments show distillation-based approaches being most effective in these evaluation setups. A limitation of the work is that it has not been fully addressed whether there exists a variant of distillation-based CL approach that consistently outperforms Logit-KD. Based on the current observation, we conclude the performance of different KD approaches for CL is highly task-dependent. It asks for more future works into continual learning algorithms within the proposed problem setup.
Kristjan Arumae, Qing Sun, and Parminder Bhatia. 2020. An empirical investigation towards efficient multi-domain language model pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4854-4864, Online. Association for Computational Linguistics.
Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017. SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 546-555, Vancouver, Canada. Association for Computational Linguistics.
Abdul Hameed Azeemi and Adeel Waheed. 2021. Covid-19 tweets analysis through transformer language models. ArXiv, abs/2103.00199.
Francesco Barbieri, Jose Camacho-Collados, Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. 2018. SemEval 2018 task 2: Multilingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 24-33, New Orleans, Louisiana. Association for Computational Linguistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620, Hong Kong, China. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. 2021. Co2l: Contrastive continual learning. ArXiv, abs/2106.14413.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc'Aurelio Ranzato. 2019. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486.
Yung-Sung Chuang, Shang-Yu Su, and Yun-Nung Chen. 2020. Lifelong language knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2914-2924, Online. Association for Computational Linguistics.
Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13122-13131.
Franck Dernoncourt and Ji Young Lee. 2017. PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 308-313, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, D. Gillick, Jacob Eisenstein, and William W. Cohen. 2021. Time-aware language models as temporal knowledge bases. ArXiv, abs/2106.15110.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, L. Zhang, Yezhou Yang, and Zicheng Liu. 2021. Seed: Self-supervised distillation for visual representation. ArXiv, abs/2101.04731.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. ArXiv, abs/2104.08821.
Yuyun Gong and Qi Zhang. 2016. Hashtag recommendation using attention-based convolutional neural network. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pages 2782-2788. IJCAI/AAAI Press.
Suchin Gururangan, Michael Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2021. Demix layers: Disentangling domains for modular language modeling. ArXiv, abs/2108.05036.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages
8342-8360, Online. Association for Computational Linguistics.
Geoffrey E. Hinton, Oriol Vinyals, and J. Dean. 2015. Distilling the knowledge in a neural network. *ArXiv*, abs/1503.02531.
Spurthi Amba Hombaiah, Tao Chen, Mingyang Zhang, Michael Bendersky, and Marc-Alexander Najork. 2021. Dynamic language models for continuously evolving content. Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. 2018. Lifelong learning via progressive distillation and retrospection. In ECCV.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Xiaolei Huang and Michael J. Paul. 2018. Examining temporality in document classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 694-699, Melbourne, Australia. Association for Computational Linguistics.
Yufan Huang, Yanzhe Zhang, Jiao Chen, Xuezhi Wang, and Diyi Yang. 2021. Continual learning for text classification with information disentanglement based regularization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2736-2746, Online. Association for Computational Linguistics.
Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Stanley Jungkyu Choi, and Minjoon Seo. 2021. Towards continual knowledge learning of language models. arXiv preprint arXiv:2110.03215.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. TinyBERT: Distilling BERT for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4163-4174, Online. Association for Computational Linguistics.
David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391-406.
Kasidis Kanwatchara, Thanapapas Horsuwan, Piyawat Lertvittayakumjorn, Boonserm Kijsrikul, and Peerapon Vateekul. 2021. Rational LAMOL: A rationale-based lifelong learning framework. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2942-2953, Online. Association for Computational Linguistics.
Angeliki Lazaridou, A. Kuncoro, E. Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Sebastian Ruder, Dani Yogatama, Kris Cao, Tomás Kocisky, Susannah Young, and P. Blunsom. 2021. Assessing temporal generalization in neural language models. NeurIPS.
Zhizhong Li and Derek Hoiem. 2018. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40:2935-2947.
Tianlin Liu, Lyle Ungar, and João Sedoc. 2019a. Continual learning for sentence representations using conceptors. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3274-3279, Minneapolis, Minnesota. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969-4983, Online. Association for Computational Linguistics.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A. Smith. 2021. Time waits for no one! analysis and challenges of temporal misalignment.
Antonis Maronikolakis and Hinrich Schütze. 2021. Multidomain pretrained language models for green NLP. In Proceedings of the Second Workshop on Domain Adaptation for NLP, pages 1-8, Kyiv, Ukraine. Association for Computational Linguistics.