SlowGuess/NAACL_2022_OCR huggingface dataset:
Kelvin Luu<sup>1</sup> Daniel Khashabi<sup>2</sup> Suchin Gururangan<sup>1</sup>  
Karishma Mandyam<sup>1</sup> Noah A. Smith<sup>1,2</sup>
<sup>1</sup>University of Washington <sup>2</sup>Allen Institute for AI
{kellu,sg01,krm28,nasmith}@cs.washington.edu,
danielk@allenai.org
When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance. In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and reviews) and periods of time (spanning five years or more) to quantify the effects of temporal misalignment. Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domain-specific pretraining, followed by task-specific finetuning. We establish a suite of tasks across multiple domains to study temporal misalignment in modern NLP systems. We find stronger effects of temporal misalignment on task performance than have been previously reported. We also find that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period. Our findings motivate continued research to improve temporal robustness of NLP models.
Changes in the ways a language is used over time are widely attested (Labov, 2011; Altmann et al., 2009; Eisenstein et al., 2014); how these changes will affect NLP systems built from text corpora, and in particular their long-term performance, is not as well understood.
This paper focuses on temporal misalignment, i.e., where training and evaluation datasets are drawn from different periods of time. In today's pretraining-finetuning paradigm, this misalignment can affect a pretrained language model—a situation that has received recent attention (Jaidka et al., 2018; Lazaridou et al., 2021; Peters et al., 2018; Raffel et al., 2020; Röttger and Pierrehumbert, 2021)—or the finetuned task model, or both. We
suspect that the effects of temporal misalignment will vary depending on the genre or domain of the task's text, the nature of that task or application, and the specific time periods.
We focus primarily on measuring the extent of temporal misalignment on task performance. We consider eight tasks, each with datasets that span at least five years (§2.4), ranging from summarization to entity typing, a subproblem of entity recognition (Borthwick, 1999). Notably, these task datasets span four different domains: social media, scientific articles, news, and reviews. We introduce an easily interpretable metric that summarizes the rate at which task performance degrades as function of time.
Our research questions are:
$(Q_{1})$  how does temporal misalignment affect downstream tasks over time?
$(Q_{2})$  how does sensitivity to temporal misalignment vary with text domain and task?
$(Q_{3})$  how does temporal misalignment affect language models across domains and spans of time?
$(Q_{4})$  how effective is temporal adaptation, or additional pretraining on a target year, in mitigating temporal misalignment?
We find that temporal misalignment affects both language model generalization and task performance. We find considerable variation in degradation across text domains (§3.2) and tasks (§3.1). Over five years, classifiers'  $F_{1}$  score can deteriorate as much as 40 points (political affiliation in Twitter) or as little as 1 point (Yelp review ratings). Two distinct tasks defined on the same domain can show different levels of degradation over time.
We explore domain adaptation of a language model, using temporally selected (unannotated) data, as a way to curtail temporal misalignment (Röttger and Pierrehumbert, 2021). We find that this does not offer much benefit, especially relative
Data and code are available here.
Human Language Technologies, pages 5944 - 5958
to performance that can be achieved by finetuning on temporally suitable data (i.e., from the same time period as the test data). We conclude that temporal adaptation should not be seen as a substitute for finding temporally aligned labeled data.
The evidence and benchmarks we offer motivate careful attention to temporal misalignment in many applications of NLP models, and further research on solutions to this problem.
Contributions. To facilitate the study of temporal misalignment phenomenon on downstream applications, we compile a suite of eight diverse tasks across four important language domains. We define an interpretable metric that summarizes temporal misalignment of a model on a task with times-tamped data. Our experiments reveal key factors in how temporal misalignment affects NLP model performance.
We begin by defining the scope of our study.
2.1 Learning Pipeline
We consider a process for building an NLP model that is in widespread use by the research community, illustrated in Fig. 1. First, a (neural network) language model (LM) is pretrained on a large text collection that is not necessarily selected for topical or temporal proximity to the text of the target application (our focus is on GPT-2; Brown et al., 2020). Second, the LM is optionally adapted by continued training on a collection strategically curated for closer proximity to the target (Beltagy et al., 2019); this stage is often referred to as domain-adaptive pretraining (DAPT; Gururangan et al., 2020). Finally, the model is finetuned to minimize a task-specific loss, using labeled data representative of what the model is expected to be exposed to in testing or deployment.
Figure 1: A typical modeling pipeline in NLP.
We study two ways in which temporal misalignment might affect the pipeline's performance as well as straightforward ways to mitigate them.
Task Shift and Temporal Finetuning The relationship between text inputs and target outputs may
change over time. To the extent that this occurs, annotated datasets used to train NLP systems in the finetuning stage will become stale over time. Due to this temporal misalignment, performance will degrade after deployment, or any in evaluations that use test data temporally distant from the training data. We seek to quantify this degradation across a range of text domains and tasks.
Language Shift and Temporal Domain Adaptation Changes in language use can cause a pretrained LM, which commonly serves as the backbone for most modern NLP models, to become stale over time (Lazaridou et al., 2021), regardless of the end task. Lazaridou et al. (2021) explored temporal adaptation, continuing LM training on new text data. This is essentially the same procedure as DAPT, where the data is selected by time period. Their work focused on the LM alone, not downstream tasks; we consider both here.
Röttger and Pierrehumbert (2021), the closest to our work, studied temporal adaptation in conjunction to finetuning for a classification task over Reddit data. They conclude that temporal adaptation does not help any more than normal DAPT. We corroborate this work and extend it by studying a wider variety of tasks over a longer span of time periods and thus are better able to draw generalizations from our results.
We believe that the two kinds of shift—task shift and language shift—are difficult to disentangle, and we do not attempt to do so in this work. Instead, we aim to quantify the effect of temporal misalignment on a range of NLP tasks, as well as the benefits of these two strategies.
2.2 Evaluation Methodology
Our experiments are designed to measure the effect of temporal misalignment on task performance. To do so, for each task, we fix a test set within a given time period,  $T_{test}$ . We vary the time period of the training data, allowing us to interpret differences in performance as a kind of "regret" relative to the performance of a model trained on data temporally aligned with  $T_{test}$ . We consider multiple different test periods for each task. We also seek to control the effect of training dataset size. We partition training data into time periods of roughly
2This setup avoids a confound of varying test set difficulty that we would encounter if we fixed the model and compared its performance across test datasets from different time periods.
the same size and always train on a single partition, keeping the training set size of each time period constant within each task. We expect that performance could be improved by accumulating training data across multiple time periods, but that would make it more difficult to achieve our research goal of quantifying the effect of temporal misalignment on performance.
2.3 Quantifying Temporal Degradation
Understanding temporal misalignment requires evaluating a model's performance across data with a range of different timestamps, which makes it difficult to compare various models in terms of their misalignment. We define a metric for temporal degradation (TD) which summarizes the expected speed of model degradation due to temporal misalignment on a task as a single value. In high-level terms, the TD score measures the average rate of performance deterioration (of perplexity,  $F_{1}$ , or Rouge-L) for each timestep of difference between that the train and evaluation sets. Higher TD scores imply greater levels of performance deterioration due to misalignment.
Let  $S_{t' \to t}$  indicate the performance a model trained on timestep  $t'$  data and evaluated on timestep  $t$ . We define  $D(t' \to t)$  as:
$$
D \left(t ^ {\prime} \rightarrow t\right) = - \left(S _ {t ^ {\prime} \rightarrow t} - S _ {t \rightarrow t}\right) \times \operatorname {s i g n} \left(t ^ {\prime} - t\right).
$$
$D(t^{\prime}\rightarrow t)$  is a modified difference in performance between two models.3 Fig. 2 illustrates  $D$  as a function of consecutive training time periods.
We find a line of best fit for  $D(t' \to t)$  for all  $t'$  using least-squares regression. The slope of this line is  $\mathrm{TD}(t)$ , the TD score for evaluation time period  $t$ . The final TD score is the average of the  $\mathrm{TD}(t)$  across all evaluation time periods  $t$ . Further details can be found in Appendix A.
2.4 Domains, Tasks, and Datasets
We describe the eight tasks and four domains used for this study. Three (out of eight) of the tasks are newly defined in this work, and all tasks required nontrivial postprocessing. We provide examples and detailed statistics in Table 1.
Figure 2: An example calculation of the TD score for a particular timestep  $t$  (discussed in Section 2.3). The plotted markers represent  $D(t' \rightarrow t)$  ( $y$ -axis) as a function of train time period  $t'$  ( $x$ -axis). The annotated numbers on each blue dot are the raw evaluation scores  $S_{t' \rightarrow t}$ , not to be confused with the  $y$  values. The red line is the line of best fit and its slope is the TD score for evaluation timestep  $t$ . In this example, we would expect to see, on average, 9.09 points of deterioration for each year of misalignment. The final TD score is averaged across all evaluation timesteps.
Domain 1: Twitter Social media platforms like Twitter have been mined to study aspects of language change over time, such as the introduction or diffusion of new words (Eisenstein et al., 2014; Tamburrini et al., 2015; Wang and Goutte, 2017). We collect unlabeled data for domain adaptation by extracting a random selection of 12M tweets, spread semi-uniformly from 2015 till 2020. We experiment with two tasks on Twitter data:
Political affiliation classification (POLIAFF) We collect English tweets dated between 2015 and 2020 from U.S. politicians with a political affiliation (Republican or Democrat). We omit any politician who changed parties over this time period or identified as independent. We consider the downstream task of detecting political affiliations, i.e., given a text of a single tweet we predict the political alignment of its author at the time of the tweet. This task can be useful for studies that involve an understanding of ideologies conveyed in text (Lin et al., 2008; Iyyer et al., 2014).
Named entity type classification (TwIERC) We use the Twitter NER dataset from Rijhwani and Preoticiuc-Pietro (2020). The dataset contains tweets dated from 2014 to 2019, each annotated with the mentions of named entities and their types (Person, Organization, or Location). We consider the task of typing a given mention, which is a subproblem of named entity recognition.
Without the modification, a task with degradation would have positive performance gaps both  $t' > t$  and  $t' > t$ ; the function would not be monotone and the rate of change would be harder to approximate. The modification yields a simpler visual understanding of the deviations over time.
4Collected via the Twitter API.
Domain	Task	Time Range	Size	Example
Twitter	political
affiliation
classification	2015-2019	120k	Input: History will note that Trump didn’t merely fiddle while the planet burned
but tried to throw the Arctic National W... Output: Democrat (vs Republican)
entity type
classification	2014-2019	8k	Input: entity: Finola, tweet: Two 64-year olds enjoying their first birthday
together in 40+ years. My twin sister, Finola, and I. Output: Person
Science	mention
type
classification	1980-2016	8k	Input: mention: deep Long Short-Term Memory (LSTM) subnetwork, abstract:
In this paper, we study the problem of online action detection from the streaming
skeleton data .... by leveraging the merits of the deep Long Short-Term Memory
(LSTM) subnetwork, the proposed model ... Output: Method
venue
classification	2009-2020	16k	Input: Rank K Binary Matrix Factorization (BMF) approximates a binary matrix
by the product of two binary matrices of lower rank, K... Output: AAAI (vs
ICML)
News	media
frame
classification	2009-2016	20k	Input: You think you have heard the worst horror a gun in the wrong hands can
do, and then this. You think there could not have been anywhere more tragic for
it to happen... Output: Gun Control (15 possible frames)
publisher
classification	2009-2016	67k	Input: A Muslim woman said Sunday that her viral article explaining why she
voted for Donald Trump has angered her liberal pals as well as other Muslims.
Output: FoxNews (vs NYTimes or WaPost)
summarization	2009-2016	330k	Input: The Consumer Financial Protection Bureau is demanding PayPal return
$15 million to consumers and pay a $10 million fine for ... Output: The CFPB
alleges many customers unwittingly signed up for PayPal Credit
Food Reviews	review rating
classification	2013-2019	126k	Input: What a beautiful store and amazing experience! Not only the atmosphere,
but the people... Output: 4 (out of 5)
Table 1: The tasks from four domains studied in this paper, with examples. See Section 2.4 for more details.
Domain 2: Scientific Articles Scientific research produces vast amounts of text with great potential for language technologies (Wadden et al., 2020; Lo et al., 2020); it is expected to show a great deal of variation over time as ideas and terminology evolve. For adaptation to this domain, we collect unlabeled data from science documents available in Semantic Scholar's corpus,[5] which yields 650k documents, spread over a 30-year period (Ammar et al., 2018). For this domain, we study two tasks:
Mention type classification (SCIERC) We use the SciERC dataset from Luan et al. (2018) which contains entity-relation annotations for computer science paper abstracts for a relatively wide range of years (1980s to 2019). We subdivide the annotated data into time periods with roughly equal-sized numbers of papers (1980-1999, 2000-2004, 2005-2009, 2010-2016). The task is to map a mention of a scientific concept to a type (Task, Method, Metric, Material, Other-Scientific-Term, or Generic).
AI venue classification (AIC) We also examine temporal misalignment on the task of identifying whether a paper was published in AAAI or ICML. We group the data into roughly equal-sized time periods (2009-2011, 2012-2014, 2015-2017, and 2018-2020). This task is, loosely, a proxy for topic classification and author disambiguation applications (Subramanian et al., 2021).
Domain 3: News Articles News articles make up a significant part of the data commonly used
to train LMs (Dodge et al., 2021). News articles convey current events, suggesting strong temporal effects on topic. For adaptation, we use 9M articles from the Newsroom dataset (Grusky et al., 2018), ranging from 2009-2016. $^{6}$  We experiment with three tasks on news articles:
Newsroom summarization (NEWSUM) The Newsroom dataset provides a large quantity of high-quality summaries of news articles (Grusky et al., 2018). We group articles by years for this task (2009-2010, 2011-2012, 2013-2014, 2015-2016). Note that this task, unlike the other tasks considered here, is not a document classification task.
Publisher classification (PUBCLS) The Newsroom dataset also provides metadata, such as publication source. We take the documents published by the 3 most prolific publishers (Fox News, New York Times, and Washington Post) and train models to classify documents among them. We bin the years (2009-2010, 2011-2012, 2013-2014, 2015-2016). This task is a proxy for applications that seek to infer fact provenance (Zhang et al., 2020). We note that, unlike in our other tasks, we downsample to ensure that the labels are equally balanced.
Media frames classification (MFC) "Framing" often refers to the emphasis or deemphasis of different social or cultural issues in the media's presentation of the news (Entman, 1983). Card et al. (2015) provide a dataset of news articles annotated
5 https://api.sementanticscholar.org/corpus/
$^{6}$ https://lil.nlp.cornell.edu/newsroom
with framing dimensions. We predict the primary frame of a document, treating the problem as a 15-way classification task. We bin by timestamp (2009-2010, 2011-2012, 2013-2014, 2015-2016).
Domain 4: Food Reviews Food and restaurant reviews have been widely studied in NLP research. We considered this domain as a possible contrast to those above, expecting less temporal change. Using data from the Yelp Open Dataset, $^{7}$  we consider one task:
Review rating classification (YELPCLS) This is a conventional sentiment analysis task, mapping the text of a review to the numerical rating given by its author (Pang et al., 2002; Dave et al., 2003). We partition the data by year (2013 to 2019) and ensure that each timestep has a roughly equal amount of reviews.
3 Empirical Results and Analysis
In this section, we summarize our experimental analysis, resulting from more than 500 experiments. In our experiments, we primarily explore the effect of temporal misalignment on GPT2 (Brown et al., 2020), a LM often used for generation. We report the macro  $F_{1}$  score for classification tasks and Rouge-L (Lin, 2004) for NewSUM.
We first focus on quantifying temporal misalignment in end tasks. As a preliminary analysis, we investigate how the marginal distribution over labels changes over time. We then study how temporal misalignment affects performance of GPT2 models in downstream tasks with temporal finetuning  $(Q_{1}, Q_{2})$ . We find that the amount of performance degradation can vary by task; in some cases the degradation can be severe.
We then study how temporal misalignment affects LMs. As a first step, we analyze how vocabularies change over time in our datasets. We then experiment with  $(Q_{3})$  how temporal misalignment affects upstream language modeling and  $(Q_{4})$  how effective temporal adaptation, or additional pretraining on a target year, is in mitigating misalignment. We find that while LMs are affected by misalignment, temporal domain adaptation is not enough to mitigate temporal misalignment.
Details on temporal domain adaptation and finetuning, and an extended version of our results, can be found in Appendices B and D, respectively.
Label Distribution Change (KL-Div)
PoliAff	0.00	0.00	0.03	0.06	0.21	0.30
15	16	17	18	19	20
SciERC	0.00	0.01	0.02	0.01
80-99	00-04	05-09	10-16
MFC	0.00	0.04	0.06	0.21
09-10	11-12	13-14	15-16
YelpCls	0.00	0.01	0.02	0.05	0.06	0.08
2013	14	15	16	17	18
Year
Figure 3: KL divergence between label distributions over time for a subset of tasks. See Appendix D for full results. For each cell, we compare the distribution of labels to that of the first time period; e.g., the 2017 POLIAFF cell contains the KL-divergence between the label distributions of POLIAFF in 2017 and 2015. While most tasks see little change over time, POLIAFF and MFC see a large shift.
3.1 Temporal Misalignment in Tasks
How much does misalignment affect task performance? We find that it depends on the task.
Label Distribution Drift We first investigate how task datasets undergo changes in the marginal distribution over labels due to time. For each task and each test period, we calculate the KL divergence between the label distributions in that period and the first test period. Full results are reported in Fig. 3. In three cases, we detected notable label distribution drift: POLIAFF, AIC, and MFC. $^{9}$  In POLIAFF, Republican tweets outnumbered Democratic ones by over a 2:1 ratio in 2015, but the reverse held by 2020. This observation shows that, regardless of the properties of NLP models, the nature of many tasks changes over time, if only because the output distribution changes.
Finetuning As described in §2.4, for each task, we create training and evaluation sets associated with different time periods. We finetune GPT2 on each of the task's training sets and evaluate each on two evaluation sets. Note that there is no domain adaptation here.
Fig. 4 shows our results on downstream tasks (with no domain adaptation). To get more reliable estimates, each number in this heatmap is an average of five independent experiments with different random seeds. A summary of the fine-tuning re
For other tasks, it is possible that the data collection/annotation procedures suppressed label distribution changes that would be visible in data "from the wild."
<sup>7</sup>https://www.yelp.com/dataset
In our preliminary results, we found that BERT, RoBERTa, and GPT2 models showed similar patterns.
Figure 4: Temporal misalignment in finetuning affects task performance (§3.1). In all cases, higher scores are better. The heatmap is shaded per column, i.e., the darkest shade of orange in a cell means the cell has the highest score in that column. Mismatch between the training and evaluation data can result in massive performance drop; the degree varies by task. For example, YELPCLS shows minimal degradation. In contrast, POLIAFF shows major deterioration over time. Additional tables for remaining tasks can be found in Appendix D.
Eval Year
Domain	Task (metric)	TD	r
Twitter	POLIAFF (F1)	7.72	0.98
TWIERC (F1)	0.96	0.74
Science	SCIENCE (F1)	0.67	0.93
AIC (F1)	1.79	0.93
News	PUBCLS (F1)	5.46	0.85
NEWSUM (Rouge-L)	1.38	0.91
MFC (F1)	0.98	0.86
Reviews	YELPCLS (F1)	0.26	0.30
Table 2: Finetuned models' temporal degradation summary scores (TD; §2.3; details in Figure 4). These scores estimate how fast a model degrades as the time period of training and evaluation data diverge (higher scores imply faster degradation). We note that since we normalize by the overall time range of a task, the temporal partitions we used do have an effect on the TD scores. For example, AIC spans ten years, even though there are only four partitions. We also show the correlation coefficient,  $r$ , that measures the strength of a linear relationship (0 meaning no correlation, 1 being perfectly correlated). In all cases but Yelp, the degree of degradation has a moderate correlation with the distance between the training and evaluation years ( $r > 0.5$ ,  $p < 0.05$ ). We use the Wald test with the null hypothesis that the slope is 0.
sults, in terms of TD scores (§2.3) is in Table 2 which indicates the speed of temporal degradation, for every year that the training and evaluation data diverges. Recall that this score (applied to task performance measures) summarizes the strength of the effect of temporal misalignment on the score, using evidence from across experiments.
$(Q_{1})$  Temporal misalignment degrades task performance substantially. Fig. 4, similar to earlier work (Röttger and Pierrehumbert, 2021), shows that models trained on data from the same time