SlowGuess/NAACL_2022_OCR huggingface dataset:
Xisen Jin†1 Dejiao Zhang² Henghui Zhu² Wei Xiao² Shang-Wen Li‡² Xiaokai Wei² Andrew Arnold² Xiang Ren¹
<sup>1</sup>University of Southern California <sup>2</sup>AWS AI Labs
{xisenjin,xiangren}@usc.edu
{dejiaoz, henghui, weixiaow, shangwenl, xiaokaiw, anarnld}
@amazon.com
Pretrained language models (PTLMs) are typically learned over a large, static corpus and further fine-tuned for various downstream tasks. However, when deployed in the real world, a PTLM-based model must deal with data distributions that deviate from what the PTLM was initially trained on. In this paper, we study a lifelong language model pretraining challenge where a PTLM is continually updated so as to adapt to emerging data. Over a domain-incremental research paper stream and a chronologically-ordered tweet stream, we incrementally pretrain a PTLM with different continual learning algorithms, and keep track of the downstream task performance (after fine-tuning). We evaluate PTLM's ability to adapt to new corpora while retaining learned knowledge in earlier corpora. Our experiments show distillation-based approaches to be most effective in retaining downstream performance in earlier domains. The algorithms also improve knowledge transfer, allowing models to achieve better downstream performance over the latest data, and improve temporal generalization when distribution gaps exist between training and evaluation because of time. We believe our problem formulation, methods, and analysis will inspire future studies towards continual pretraining of language models.
Pretrained language models (PTLMs) have achieved remarkable performance on benchmark datasets for a range of NLP tasks (Liu et al., 2019b; Brown et al., 2020). However, when deployed in the wild, NLP systems must deal with emerging data that have constantly shifting data distribution, different from the text corpora they were initially pretrained on — for example, when new data domains are introduced (upper part of Fig. 1) (Gururangan et al., 2020), or when the language uses
Domain-Incremental Research Paper Stream
Chronologically-Ordered Tweet Stream
Figure 1: Two data streams created for studying life-long language model pre-training. We focus on evaluating knowledge retention on the domain-incremental research papers stream; we focus on adaptation to the latest data and temporal generalization on the chronologically ordered tweet stream.
and vocabulary change over time (lower part of Fig. 1) (Lazaridou et al., 2021). Fine-tuning from a static and possibly "outdated" PTLM may limit the model performance on downstream tasks, as the PTLM may no longer provide an effective model initialization (Beltagy et al., 2019; Müller et al., 2020). Here we look to understand whether continuously adapting a PTLM to emerging data can yield gains on various downstream tasks, and how to achieve better downstream performance for such lifelong PTLM adaptation.
A number of recent works make attempts on adapting PTLMs to a new data domain. Gururan-gan et al. (2020); Yao et al. (2021) adapt language models to corpora of different genres and topics and observe performance improvement in domain-specific downstream tasks. Arumae et al. (2020) further show that by regularizing the parameters of PTLMs, the downstream tasks performance on the general domain can be preserved. Another line of works focuses on temporal domain shift (Hombaiah et al., 2021), which analyzes the effect of pretraining over up-to-date data to the downstream
† Work done during an internship at AWS AI Labs.
$\ddagger$  Work done while at Amazon.
Human Language Technologies, pages 4764 - 4780
tasks. Röttger and Pierrehumbert (2021) further study vocabulary composition approaches for improving adaptation to up-to-date corpora. However, these work focus their study on adapting PTLM to a single new domain; while in practice, corpora from distinct domains and time stamps may emerge sequentially. Whether one can maintain a single, up-to-date PTLM remains an open problem. Related to this, Lazaridou et al. (2021) study adaptation of PTLMs over temporal data streams, but solely focus on language modeling instead of fine-tuning performance. It is also important to understand multiple aspects of the utility of lifelong PTLM pretraining, such as knowledge retention over all the seen data, and study what methods can improve the utility of PTLMs in such a continual pretraining process.
In this paper, we formulate a Lifelong Language Model Pretraining task to simulate practical scenarios of maintaining and adapting a PTLM over emerging corpora, create a testbed (along with pretraining data streams and downstream tasks) for studying continual pretraining algorithms, and present a systematic evaluation protocol for measuring the progress made on this challenging problem (see Figure 2 for an illustration). We consider two types of text corpus sequences when constructing pretraining data streams, each of which simulates a representative use case and that has slightly different focuses on the evaluation: continuously learning a single model that is applicable to both old and new domains; and improving the model's ability to handle latest data. Specifically, we construct 1) a domain-incremental text stream that consists of academic papers published in four research fields, and 2) a temporal tweet stream that consists of tweets collected from four different years. By conducting systematic experiments on these two data streams, we look to answer a series of analysis questions: 1) whether continual pretraining retains fine-tuning performance over earlier corpora compared to traditional offline pretraining, 2) whether pretraining improves downstream performance on the latest data, and 3) whether pretraining improves temporal generalization where training and evaluation have distribution gaps because of time.
To address the research questions above, we conduct a systematic evaluation of existing continual learning (CL) algorithms, spanning over model-expansion based, memory-based, and distillation-based approaches. Our results show distillation
based approaches are most effective in knowledge retention in the research paper stream, while simultaneously improve adaptation to latest data and temporal generalization in the tweet stream. We believe our problem formulation, evaluation setup, methods and analysis can inspire more future work on continual pretraining of language models.
Here we present the problem formulation for lifelong pretraining of PTLM, provide details about the data stream construction process and downstream tasks, and introduce the evaluation protocol.
2.1 Lifelong Pretraining of PTLMs
We consider the scenario where one needs to deploy and/or maintain NLP models over a sequence of  $T$  data domains. At each time step  $t$  the model visits an unlabeled text corpus  $D_{t}$  from a domain with a data distribution  $P(D_{t})$ . The data distribution  $P(D_{t})$  evolves as the time step  $t$ , forming a data stream  $D_{1..T} = \{D_{1}, D_{2}, \ldots, D_{T}\}$ . In practice, the data domain shift can refer to the topic change of the text content (from computer science research papers to biomedical papers), or temporal evolution of the text (from past to recent tweets). The task of lifelong pretraining of PTLM looks to continuously adapt a language model  $f$  as the model visits (unlabeled) text corpus  $D_{t}$  from the data stream  $D_{1..T}$ , in order to provide a good model initialization for fine-tuning on downstream tasks from the same domain. With slight abuse in notations, we also use  $D_{t}$  to directly refer to a data domain.
Here, we assume a language model  $f$  is updated sequentially over each pretraining corpora  $D_{t}$ , without accessing the full earlier corpora  $\{D_{i}\}_{i < t}$  in the data stream  $D_{1..T}$ . This aims to capture practical constraints such as privacy restriction for storing earlier data, or computation budget for training over all the text corpora in  $D_{1..T}$ . We use  $f_{t}$  to denote the language model right after updating on the domain  $D_{t}$ . In our study,  $f$  is a RoBERTa-base transformer (Liu et al., 2019b) and the model  $(f_{0})$  is initialized with pretrained RoBERTa weights.
The utility of the PTLMs  $\{f_t\}$  is evaluated based on their fine-tuned model performance on various downstream tasks. After updating on a domain  $D_{i}$ , the model  $f_{i}$  can be fine-tuned over downstream tasks from visited domains  $D_{t}$  where  $t\leq i$ . We note the set of downstream tasks related to domain  $D_{t}$  as  $S_{t} = \{S_{t}^{j}\}_{j = 1}^{N_{t}}$ , assuming the number
Figure 2: Training, evaluation setups, and metrics of life-long language model pretraining. The model sequentially visits each corpus, and is fine-tuned on downstream datasets related to the domains of pretraining. We evaluate knowledge retention and adaptation to new data with downstream fine-tuning performance on old and latest domains respectively. Besides, we evaluate temporal generalization where training/test examples are drawn from different time steps.
of downstream tasks is  $N_{t}$ . Note that in the finetuning stage, model  $f_{t}$  has no access to any of the pretraining corpus  $D_{1..T}$ .
2.2 Data Streams & Downstream Datasets
We construct data streams to simulate two representative scenarios of data domain shifts in practice (also see Fig. 1): one domain-incremental stream to simulate the sequential changes of research paper areas; and one chronologically-ordered stream to simulate tweets emerging over time.
Domain-incremental Paper Stream. This paper stream consists of the full text of research papers published in four research areas: biomedical, computer science, material science, and physics, filtered from the S2ORC dataset<sup>1</sup>, which are presented sequentially to the model. For each domain, we evaluate downstream performance over two datasets. The downstream tasks span over various tasks such as relation extraction and named entity recognition, and are summarized in Table 1. We detail these datasets in Appendix D.
Chronologically-ordered Tweet Stream. This tweet data stream consists of tweets from the year 2014, 2016, 2018 and 2020, collected by the Archive Team<sup>2</sup> and preprocessed following Nguyen et al. (2020). These four tweet corpora are presented sequentially to the language model following the chronological order of the tweet year.
Domains	Downstream Datasets	Metrics
Bio-Medicine	Chemprot (Vindahl, 2016)	Micro-F1
RCT-Sample (Dernoncourt and Lee, 2017)	Micro-F1
Comp. Science	ACL-ARC (Jurgens et al., 2018)	Macro-F1
SciERC (Luan et al., 2018)	Macro-F1
Mat. Science	Synthesis (Mysore et al., 2019)	Macro-F1
MNER (Olivetti et al., 2020)	Micro-F1
Physics	Keyphrase (Augenstein et al., 2017)	Macro-F1
Hyponym (Augenstein et al., 2017)	Macro-F1
Table 1: Summary of downstream datasets relevant to each domain in the research paper stream.
For downstream tasks, we hold out 1M tweets from each year's corpus to construct multi-label hashtag prediction datasets (Gong and Zhang, 2016) and single-label emoji prediction datasets (Barbieri et al., 2018). On two datasets, we report label ranking average precision scores (a multi-label version of MRR) of models (Azeemi and Waheed, 2021) and Macro-F1 respectively. The detailed dataset construction process is included in Appendix D.
2.3 Evaluation Protocol
We consider three key aspects for evaluating the utility of the language models  $\{f_t\}$  that are continuously updated over the data stream  $D_{1..T}$ , also illustrated in Figure 2: 1) knowledge retention and transfer over the pretraining corpora seen earlier; 2) adaptation to the latest data domain, and 3) temporal generalization when training and evaluation data are from different time steps.
Knowledge Retention. A key utility of continual language model pretraining is to obtain a single model applicable to all domains. We focus on the evaluation of the ability with the domain-incremental paper stream, because for the tweet stream, the practical need of performance over outdated data is limited. Knowledge retention is measured with the downstream task performance from earlier or the current domains that the pretrained model has visited. More formally, for each pretrained model checkpoint in  $\{f_i\}$ , we fine-tune  $f_i$  over downstream tasks  $\{S_t\}$  where  $t \leq i$  and evaluate the corresponding test set performance. It is important that the models do not suffer from catastrophic forgetting (Robins, 1995), i.e., significantly reduced helpfulness when  $f_i$  is fine-tuned for downstream tasks  $S_t$  from earlier domains with  $t < i$ .
Adaption to Latest Data Domain. In certain scenarios, performance of downstream models over the latest data domain should be emphasized. For example, classifiers in the tweet domain are usually
<sup>1</sup>We use the 20200705v1 version of the S2ORC dataset at https://github.com/allenai/s2orc
2 https://archive.org/details/twitterstream
trained and evaluated with up-to-date data for practical deployment. Formally, we focus on the downstream task performance of models fine-tuned from the final pretrained model checkpoint  $f_{T}$ , where the downstream tasks  $S_{T}$  are also from the latest domain. To succeed in these metrics, it is crucial for the model to transfer knowledge from earlier domains to the latest domain.
Temporal Generalization Ability. We consider another practical fine-tuning scenario in the tweet stream where the model is trained on outdated data and evaluated on the latest data (Rijhwani and Preotiuc-Pietro, 2020; Huang and Paul, 2018), referred to as the temporal generalization ability. Formally, we fine-tune the final pretrained model checkpoint  $f_{T}$  over the training set of downstream tasks  $S_{t}$  from an earlier time step  $(t < T)$ , and evaluate on the test set of the downstream tasks  $S_{T}$  from the latest time step  $T$ .
Lifelong language model pretraining introduces novel challenges because of the large training sets and more comprehensive evaluation protocols compared to classification tasks. We establish several strong baselines, and evaluate the performance of continual learning algorithms from different categories spanning over model-expansion, memory-based, and distillation-based approaches. We illustrate the approaches in Figure 3.
3.1 Simple Baselines
We consider several simple baselines which continual learning algorithms will be compared against. RoBERTa-base  $(f_0)$  corresponds to not pretraining on any of the domain-specific corpora. By separately pretraining  $f_0$  on each corpus  $D_{1}, D_{2}, \ldots, D_{T}$ , we obtain  $T$  Task-Specific pretrained models. We also pretrain  $f_0$  sequentially over  $D_{1..T}$ , which we refer to as sequential pretraining. While it allows knowledge transfer between domains compared to domain-specific models, without any continual learning algorithms, sequential pretraining is prone to catastrophic forgetting (Robins, 1995). Finally, we randomly shuffle corpora from all domains  $D_{1..T}$  before pretraining, noted as Multi-Task Learning (MTL). MTL corresponds to an offline training paradigm that models new corpora by re-training over all corpora seen before. The drawback is that it requires storing full data from earlier domains,
Figure 3: Comparison of adapter, memory replay, and distillation-based continual learning algorithms. Details of the methods are introduced in Sec. 3.
and that it can be extremely costly to repetitively retrain over earlier data if new data keeps emerging.
3.2 Model-expansion and Regularization-based Methods
We first introduce model-expansion based approaches, which add small trainable modules (e.g., multi-layer perceptron) to the model per new domain while keeping other parts of the model frozen. The Adapter approach is a representative approach that learns a set of "adapter" layers  $g_{t} = \{g_{t}^{k}\}_{k=1}^{K}$  for each domain  $D_{t}$  and each of the  $K$  transformer layers (Houlsby et al., 2019). We also experiment with a simple Layer Expansion approach, which learns separate top two layers of the transformer and the prediction head for each domain. We also involve a regularization-based continual learning baseline, online EWC (Schwarz et al., 2018), which directly penalize change of model parameters.
3.3 Memory Replay Methods
We also experiment with Experience Replay (ER) (Chaudhry et al., 2019), which alleviates forgetting by storing a subset of earlier examples and periodically re-training (replaying) over them. We maintain a fixed-size memory  $M$  ( $100k$  examples by default) and populate the memory  $M$  each time pretraining on a domain  $D_{t}$  finishes with examples in the current domain. We ensure  $M$  always contains a balanced sample of examples from all seen domains  $D_{1..t}$ . We replay a mini-batch of examples from the memory every 10 training steps.
3.4 Distillation-based CL Methods
While knowledge distillation (KD) (Hinton et al., 2015) techniques have been studied intensively for pretrained language models (Sun et al., 2019), applying them to continual learning has been under
explored outside image classification tasks (Li and Hoiem, 2018; Rebuffi et al., 2017; Hou et al., 2018). Distillation based CL approaches store one previous model checkpoint of the model (noted as  $f_{t-1}$ ) and regularize the differences between  $f_{t-1}$  and the current model  $f_t$ . We adapt several existing knowledge distillation techniques to PTLMs and utilize them for continual learning. We note, while individual distillation techniques are not original, their adaptation to CL algorithms can be novel.
We perform distillation with examples from the current domain  $D_{t}$  and a replay memory  $M$  (similar to ER). Despite the potential gap between  $D_{t}$  and the training data of  $f_{t - 1}$ , the approach allows utilizing more data for distillation. Formally, each time the model receives a mini-batch of stream examples  $\pmb{x}_s$  or a draws mini-batch of memory examples  $\pmb{x}_m$  from  $M$  (both noted as  $\pmb{x}$ ), we collect certain outputs of the model (e.g., output logits or intermediate representations) with  $f_{t - 1}$  and  $f_{t}$ . We compute a distillation loss  $\ell_{\mathrm{KD}}(\pmb{x}, f_{t - 1}, f_t)$  that penalizes the differences between the model outputs, and jointly optimize it with the masked language modeling loss  $\ell_{\mathrm{MLM}}$ . The final objective is written as  $\ell = \ell_{\mathrm{MLM}} + \alpha \ell_{\mathrm{KD}}$ , where  $\alpha$  is a hyperparameter to weight the distillation loss.
Logit Distillation. In logit distillation (Hinton et al., 2015), we collect the output logits of  $f_{t}$  and  $f_{t-1}$ , noted as  $y_{t}$  and  $y_{t-1}$  respectively. The distillation loss is computed as  $D_{\mathrm{KL}}(y_{t}, y_{t-1})$ , where  $D_{\mathrm{KL}}$  is the Kullback-Leibler divergence function.
Representation Distillation. We also consider minimizing the representational deviation of sentences between previous and current models (Sun et al., 2019; Jiao et al., 2020). We extract the representation of each word of two models, noted as  $h_{t-1}^{1:N}$  and  $h_t^{1:N}$ , before the masked language modeling prediction head, where  $N$  is the length of the sentence. Then, we compute MSE loss  $||h_{t-1}^{1:N} - h_t^{1:N}||_2^2$  as the distillation loss.
Contrastive Distillation. In addition to output logits and hidden representations, we further look into representational similarity within a batch of examples as additional knowledge to distill. The approach is adapted from (Cha et al., 2021), which is originally studied for supervised image classification tasks. We briefly introduce the adapted algorithm and leave the details in Appendix E. During continual pretraining, in addition to the language model pretraining objective, we add an unsuper-
vised contrastive learning objective, namely the SimCSE (Gao et al., 2021) objective to encourage sentence representations to reflect semantic similarities between sentences. Then, we compute the intra-batch representational similarity matrices of sentence representations (i.e. between each pair of examples in the mini-batch) with  $f_{t-1}$  and  $f_t$ , noted as  $\mathbf{B}^{t-1}$  and  $\mathbf{B}^t$ , and minimize the cross entropy loss  $\ell_{\mathrm{distill}} = -\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{N}\mathbf{B}_{ij}^{t-1}\log \mathbf{B}_{ij}^t$
Self-Supervised Distillation (SEED). SEED distillation proposed by (Fang et al., 2021) has a similar spirit as the contrastive distillation. The only difference is that it distills representational similarity between the batch and a large set of other examples. We leave the details of the algorithm in Appendix E. We further combine SEED Distillation with logit distillation and refer to the approach as SEED-Logit Distillation.
We summarize our findings over the created data streams. We ask whether lifelong pretraining and continual learning algorithms are effective base on our evaluation protocol proposed in Sec. 2.3.
We use the RoBERTa-base model (Liu et al., 2019b), initialized with RoBERTa-base weights throughout the experiments. We set the maximal sequence length to 128 and an effective training batch size of 2,048. On the research paper stream, models are trained for  $8k$  steps in the first domain and  $4k$  steps in the subsequent domains. On the Tweet stream, we train the models for  $4k$  steps in each domain. These correspond to less than a single pass of data in each domain. See Appendix A for detailed setups.
4.2 Domain Incremental Data Stream
As we introduced in Sec. 2.2, in the domain incremental research paper stream, we expect a model  $f_{t}$  to perform well on all downstream tasks  $S_{1..t}$  from domains  $D_{1..t}$ . In Table 2, we report the performance of models on all downstream tasks  $S_{1..T}$  fine-tuned from the final pretraining checkpoint,  $f_{T}$ . We visualize more complete change of downstream task performance over different time steps of pretraining (i.e.,  $f_{1}, f_{2}, f_{3}, f_{4}$ ) in Fig. 4. We also report the log perplexity of masked language modeling (MLM) in Table 2 as additional informa
Task	D1 - Biomedical	D2 - Computer Science	D3 - Materials Science	D4 - Physics
Dataset	Chemprot	RCT-Sample	MLM	ACL-ARC	SciERC	MLM	MNER	Synthesis	MLM	Keyphrase	Hyponym	MLM
Roberta-base	82.03±0.7	78.07±0.7	1.993	64.32±2.8	79.07±1.6	2.153	83.15±0.3	91.25±0.6	2.117	66.21±1.0	67.59±4.5	2.278
Sequential Pretraining	82.09±0.5	79.60±0.5	1.654	72.73±2.9	81.43±0.8	1.807	83.99±0.3	92.10±1.0	1.590	67.57±1.0	74.68±4.4	1.381
ER	82.73±0.3	79.98±0.3	1.737	72.50±1.0	81.64±1.1	1.857	83.99±0.4	92.65±0.4	1.621	66.11±1.1	72.82±4.3	1.391
Online EWC	81.83±0.2	78.84±0.5	1.655	71.81±2.6	80.79±0.5	1.803	83.43±0.4	91.89±0.5	1.571	66.70±0.6	72.98±6.0	1.388
Adapter	83.30±0.4	80.41±0.4	1.417	69.32±3.5	80.22±1.5	1.633	83.91±0.3	91.69±0.6	1.522	66.23±1.4	69.65±4.5	1.554
Layer Expansion	83.74±0.3	81.10±0.5	1.210	65.17±2.9	79.35±0.8	1.756	82.48±0.4	92.33±1.0	1.389	65.70±1.1	73.34±3.7	1.534
Logit-KD	83.39±0.4	81.21±0.1	1.392	73.70±3.4	81.92±0.8	1.699	83.96±0.3	92.20±1.0	1.425	64.75±1.1	71.29±3.6	1.460
Rep-KD	82.34±0.3	79.59±0.5	1.684	71.17±2.5	78.78±1.1	1.810	84.13±0.3	92.02±0.8	1.585	65.96±1.6	73.93±5.5	1.389
Contrast-KD	82.29±0.5	79.92±0.4	1.722	71.15±1.1	80.49±1.6	1.856	83.26±0.4	92.62±0.7	1.612	65.95±1.7	72.26±3.1	1.428
SEED-KD	82.78±0.3	80.38±0.4	1.720	69.98±2.4	81.61±0.7	1.829	82.99±0.4	92.35±0.7	1.609	65.35±1.0	74.79±4.1	1.401
SEED-Logit-KD	83.72±0.4	81.05±0.2	1.391	69.90±4.5	83.03±0.6	1.703	83.28±0.5	92.87±1.0	1.428	65.96±1.5	71.92±5.5	1.460
Task-Specific LM	83.74±0.3	81.10±0.5	1.210	72.20±2.6	81.24±1.7	1.629	84.02±0.2	91.56±0.4	1.418	65.95±1.1	69.43±4.5	1.426
MTL	82.91±1.6	80.67±0.4	1.289	69.46±1.8	81.12±0.8	1.616	83.92±0.3	92.66±0.6	1.355	65.37±1.6	73.31±5.2	1.418
Table 2: Results on the Research Paper stream. We report log perplexity of MLM and the performance of downstream models fine-tuned from the final checkpoint of the pretrained model ( $t = 4$ ). Performance of the best performing CL algorithm is marked bold.
|M|, k	Chemprot	RCT	ACL-ARC	SciERC	MLM-D1,2
100k, 10	82.73	79.98	72.50	81.64	1.737/1.857
100k, 100	82.06	78.64	71.97	81.62	1.599/1.789
10M, 10	82.87	79.98	71.80	81.63	1.438/1.732