Venue: NAACL 2022

# Time Waits for No One!

Analysis and Challenges of Temporal Misalignment

Kelvin Luu\({}^{1}\) Daniel Khashabi\({}^{2}\) Suchin Gururangan\({}^{1}\)

Karishma Mandyam\({}^{1}\)Noah A. Smith\({}^{1,2}\)

###### Abstract

When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting _temporal misalignment_ can degrade end-task performance. In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and reviews) and periods of time (spanning five years or more) to quantify the effects of temporal misalignment. Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domain-specific pretraining, followed by task-specific finetuning. We establish a suite of tasks across multiple domains to study temporal misalignment in modern NLP systems. We find stronger effects of temporal misalignment on task performance than have been previously reported. We also find that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period. Our findings motivate continued research to improve temporal robustness of NLP models.1

Footnote 1: Data and code are available here.

\begin{tabular}{c c}  & **Time Waits for No One!** \\  & **Analysis and Challenges of Temporal Misalignment** \\  & **Klein Luu\({}^{1}\)** & **Daniel Khashabi\({}^{2}\)** & **Suchin Gururangan\({}^{1}\)** \\  & **Karishma Mandyam\({}^{1}\)** & **Noah A. Smith\({}^{1,2}\)** \\   \({}^{1}\)University of Washington & \({}^{2}\)Allen Institute for AI \\  & {kellu,sg01,km28,nasmith}@cs.washington.edu, \\  & danielk@allenai.org \\ \end{tabular}

+
Footnote †: footnotemark:

## 1 Introduction

Changes in the ways a language is used over time are widely attested Labov (2011); Altmann et al. (2009); Eisenstein et al. (2014); how these changes will affect NLP systems built from text corpora, and in particular their long-term performance, is not as well understood.

This paper focuses on _temporal misalignment_, i.e., where training and evaluation datasets are drawn from different periods of time. In today's pretraining-finetuning paradigm, this misalignment can affect a pretrained language model--a situation that has received recent attention Jaidka et al. (2018); Lazaridou et al. (2021); Peters et al. (2018); Raffel et al. (2020); Rottger and Pierrehumbert (2021)--or the finetuned task model, or both. We suspect that the effects of temporal misalignment will vary depending on the genre or domain of the task's text, the nature of that task or application, and the specific time periods.

We focus primarily on measuring the extent of temporal misalignment on task performance. We consider eight tasks, each with datasets that span at least five years (SS2.4), ranging from summarization to entity typing, a subproblem of entity recognition Borthwick (1999). Notably, these task datasets span four different domains: social media, scientific articles, news, and reviews. We introduce an easily interpretable metric that summarizes the rate at which task performance degrades as function of time.

Our research questions are:

* [label=(\(Q_{1}\))]
* _how does temporal misalignment affect downstream tasks over time?_
* _how does sensitivity to temporal misalignment vary with text domain and task?_
* _how does temporal misalignment affect language models across domains and spans of time?_
* _how effective is temporal adaptation, or additional pretraining on a target year, in mitigating temporal misalignment?_

We find that temporal misalignment affects both language model generalization and task performance. We find considerable variation in degradation across text domains (SS3.2) and tasks (SS3.1). Over five years, classifiers' \(F_{1}\) score can deteriorate as much as 40 points (political affiliation in Twitter) or as little as 1 point (Yelp review ratings). Two distinct tasks defined on the same domain can show different levels of degradation over time.

We explore domain adaptation of a language model, using temporally selected (unannotated) data, as a way to curtail temporal misalignment Rottger and Pierrehumbert (2021). We find that this does not offer much benefit, especially relative

to performance that can be achieved by finetuning on temporally suitable data (i.e., from the same time period as the test data). We conclude that temporal adaptation should not be seen as a substitute for finding temporally aligned labeled data.

The evidence and benchmarks we offer motivate careful attention to temporal misalignment in many applications of NLP models, and further research on solutions to this problem.

Contributions.To facilitate the study of temporal misalignment phenomenon on downstream applications, we compile a suite of eight diverse tasks across four important language domains. We define an interpretable metric that summarizes temporal misalignment of a model on a task with timestamped data. Our experiments reveal key factors in how temporal misalignment affects NLP model performance.

## 2 Methodology Overview

We begin by defining the scope of our study.

### Learning Pipeline

We consider a process for building an NLP model that is in widespread use by the research community, illustrated in Fig. 1. First, a (neural network) language model (LM) is pretrained on a large text collection that is not necessarily selected for topical or temporal proximity to the text of the target application (our focus is on GPT-2; Brown et al., 2020). Second, the LM is optionally adapted by continued training on a collection strategically curated for closer proximity to the target Beltagy et al. (2019); this stage is often referred to as domain-adaptive pretraining DAPT Gururangan et al. (2020). Finally, the model is finetuned to minimize a task-specific loss, using labeled data representative of what the model is expected to be exposed to in testing or deployment.

We study two ways in which temporal misalignment might affect the pipeline's performance as well as straightforward ways to mitigate them.

Task Shift and Temporal FinetuningThe relationship between text inputs and target outputs may change over time. To the extent that this occurs, annotated datasets used to train NLP systems in the finetuning stage will become stale over time. Due to this temporal misalignment, performance will degrade after deployment, or any in evaluations that use test data temporally distant from the training data. We seek to quantify this degradation across a range of text domains and tasks.

Language Shift and Temporal Domain AdaptationChanges in language use can cause a pretrained LM, which commonly serves as the backbone for most modern NLP models, to become stale over time Lazaridou et al. (2021), regardless of the end task. Lazaridou et al. (2021) explored _temporal adaptation_, continuing LM training on new text data. This is essentially the same procedure as DAPT, where the data is selected by time period. Their work focused on the LM alone, not downstream tasks; we consider both here.

Rottger and Pierrehumbert (2021), the closest to our work, studied temporal adaptation in conjunction to finetuning for a classification task over Reddit data. They conclude that temporal adaptation does not help any more than normal DAPT. We corroborate this work and extend it by studying a wider variety of tasks over a longer span of time periods and thus are better able to draw generalizations from our results.

We believe that the two kinds of shift--task shift and language shift--are difficult to disentangle, and we do not attempt to do so in this work. Instead, we aim to quantify the effect of temporal misalignment on a range of NLP tasks, as well as the benefits of these two strategies.

### Evaluation Methodology

Our experiments are designed to measure the effect of temporal misalignment on task performance. To do so, for each task, we fix a test set within a given time period, \(T_{test}\). We vary the time period of the training data, allowing us to interpret differences in performance as a kind of "regret" relative to the performance of a model trained on data temporally aligned with \(T_{test}\).2 We consider multiple different test periods for each task. We also seek to control the effect of training dataset size. We partition training data into time periods of roughly

Figure 1: A typical modeling pipeline in NLP.

 

the same size and always train on a single partition, keeping the training set size of each time period constant within each task. We expect that performance could be improved by accumulating training data across multiple time periods, but that would make it more difficult to achieve our research goal of quantifying the effect of temporal misalignment on performance.

### Quantifying Temporal Degradation

Understanding temporal misalignment requires evaluating a model's performance across data with a range of different timestamps, which makes it difficult to compare various models in terms of their misalignment. We define a metric for temporal degradation (TD) which summarizes the expected speed of model degradation due to temporal misalignment on a task as a single value. In high-level terms, the TD score measures the average rate of performance deterioration (of perplexity, \(F_{1}\), or Rouge-L) for each timestep of difference between that the train and evaluation sets. Higher TD scores imply greater levels of performance deterioration due to misalignment.

Let \(S_{t^{\prime}\to t}\) indicate the performance a model trained on timestep \(t^{\prime}\) data and evaluated on timestep \(t\). We define \(D(t^{\prime}\shortrightarrow t)\) as:

\[D(t^{\prime}\shortrightarrow t)=-\left(S_{t^{\prime}\shortleftarrow t}-S_{t \shortrightarrow t}\right)\times\text{sign}(t^{\prime}-t).\]

\(D(t^{\prime}\shortrightarrow t)\) is a modified difference in performance between two models.3 Fig. 2 illustrates \(D\) as a function of consecutive training time periods.

Footnote 3: Without the modification, a task with degradation would have have positive performance gaps both \(t^{\prime}>t\) and \(t^{\prime}>t\); the function would not be monotone and the rate of change would be harder to approximate. The modification yields a simpler visual understanding of the deviations over time.

We find a line of best fit for \(D(t^{\prime}\shortrightarrow t)\) for all \(t^{\prime}\) using least-squares regression. The slope of this line is TD\((t)\), the TD score for evaluation time period \(t\). The final TD score is the average of the TD\((t)\) across all evaluation time periods \(t\). Further details can be found in Appendix A.

### Domains, Tasks, and Datasets

We describe the eight tasks and four domains used for this study. Three (out of eight) of the tasks are newly defined in this work, and all tasks required nontrivial postprocessing. We provide examples and detailed statistics in Table 1.

Domain 1: TwitterSocial media platforms like Twitter have been mined to study aspects of language change over time, such as the introduction or diffusion of new words Eisenstein et al. (2014); Tamburrini et al. (2015); Wang and Goutte (2017). We collect unlabeled data for domain adaptation by extracting a random selection of 12M tweets, spread semi-uniformly from 2015 till 2020.4 We experiment with two tasks on Twitter data:

Footnote 4: Collected via the Twitter API.

Political affiliation classification (PoliaAff)We collect English tweets dated between 2015 and 2020 from U.S. politicians with a political affiliation (_Republican_ or _Democrat_). We omit any politician who changed parties over this time period or identified as independent. We consider the downstream task of detecting political affiliations, i.e., given a text of a single tweet we predict the political alignment of its author at the time of the tweet. This task can be useful for studies that involve an understanding of ideologies conveyed in text Lin et al. (2008); Iyyer et al. (2014).

Named entity type classification (TwiERC)We use the Twitter NER dataset from Rijhwani and Preotiucieto (2020). The dataset contains tweetsdated from 2014 to 2019, each annotated with the mentions of named entities and their types (_Person_, _Organization_, or _Location_). We consider the task of typing a given mention, which is a subproblem of named entity recognition.

Figure 2: An example calculation of the TD score for a particular timestep \(t\) (discussed in Section 2.3). The plotted markers represent \(D(t^{\prime}\shortrightarrow t)\) (\(y\)-axis) as a function of train time period \(t^{\prime}\) (\(x\)-axis). The annotated numbers on each blue dot are the raw evaluation scores \(S_{t^{\prime}\shortrightarrow t}\), not to be confused with the \(y\) values. The red line is the line of best fit and its slope is the TD score for evaluation timestep \(t\). In this example, we would expect to see, on average, \(9.09\) points of deterioration for each year of misalignment. The final TD score is averaged across all evaluation timesteps.

 



**Domain 2: Scientific Articles**Scientific research produces vast amounts of text with great potential for language technologies (Wadden et al., 2020; Lo et al., 2020); it is expected to show a great deal of variation over time as ideas and terminology evolve. For adaptation to this domain, we collect unlabeled data from science documents available in Semantic Scholar's corpus,5 which yields 650k documents, spread over a 30-year period (Ammar et al., 2018). For this domain, we study two tasks:

Footnote 5: [https://api.semanticscholar.org/corpus/](https://api.semanticscholar.org/corpus/)

_Mention type classification (SciERC)_ We use the _SciERC_ dataset from Luan et al. (2018) which contains entity-relation annotations for computer science paper abstracts for a relatively wide range of years (1980s to 2019). We subdivide the annotated data into time periods with roughly equal-sized numbers of papers (1980-1999, 2000-2004, 2005-2009, 2010-2016). The task is to map a mention of a scientific concept to a type (_Task_, _Method_, _Metric_, _Material_, _Other-Scientific-Term_, or _Generic_).

_AI venue classification (AIC)__We also examine temporal misalignment on the task of identifying whether a paper was published in AAAI or ICML. We group the data into roughly equal-sized time periods (2009-2011, 2012-2014, 2015-2017, and 2018-2020). This task is, loosely, a proxy for topic classification and author disambiguation applications (Subramanian et al., 2021)._

**Domain 3: News Articles**News articles make up a significant part of the data commonly used to train LMs (Dodge et al., 2021). News articles convey current events, suggesting strong temporal effects on topic. For adaptation, we use 9M articles from the Newsroom dataset (Grusky et al., 2018), ranging from 2009-2016.6 We experiment with three tasks on news articles:

Footnote 6: [https://lil.nlp.cornell.edu/newsroom](https://lil.nlp.cornell.edu/newsroom)

\begin{table}
\begin{tabular}{c c c l} \hline \hline Domain & Task & Time Range & Size & Example \\ \hline \multirow{4}{*}{Twitter} & political affiliation classification & \multirow{2}{*}{2015-2019} & \multirow{2}{*}{\(120k\)} & **Input: History will note that Trump didn’t merely fiddle while the planet burned but tried to throw the Arctic National W.. Outputuit: Democrat (vs Republican)** \\  & entity type classification & & & **Input: entity: Finola, tweet: Two 64-year olds enjoying their first birthday together in 40+ years. My twin sister, Finola, and I. Output: Person** \\ \hline \multirow{4}{*}{Science} & mention type classification & \multirow{2}{*}{1980-2016} & \multirow{2}{*}{\(8k\)} & **Input: mention: deep Long Short-Term Memory (LSTM) subnetwork, abstract: In this paper, we study the problem of online action detection from the streaming skeleton data ... by leveraging the merits of the deep Long Short-Term Memory (LSTM) subnetwork, the proposed model .... Output: Method** \\  & venue classification & & & **Input: Knot K Binary Matrix Factorization (IMF) approximates a binary matrix by the product of two binary matrices of lower rank, Km.. Outputu:: AAAI (vs ICML)** \\ \hline \multirow{4}{*}{News} & media frame classification & \multirow{2}{*}{2009-2016} & \multirow{2}{*}{\(20k\)} & **Input: You think you have heard the worst horror a gun in the wrong hands can do, and then thisYou think there could not have been anywhere more tragic for it to happen.. Outputu: Gun Control [15 possible frames]** \\  & & & & **Input: A Mistown sonar said Sunday that her viral article explaining why she voted for Donald Trumg has angered her liberal pals as well as other Muslims. Outputu: FoxNews (vs NTTimes or WaPost)** \\ \cline{2-5}  & summarization & \multirow{2}{*}{2009-2016} & \multirow{2}{*}{\(330k\)} & **Input: The Consumer Financial Protection Bureau is demanding RayPal return \\  & & & & **JS** millietho to consumers and pay a $J million fine for ... Outputu: The CFPB alleges many customers unwittingly signed wy for PayPal Credit \\ \hline Food Reviews & review rating classification & \multirow{2}{*}{2013-2019} & \multirow{2}{*}{\(126k\)} & **Input: What beautiful store and amazing experience! Not only the atmosphere, bat the people.. Outputu: 4 (out of 5)** \\ \hline \hline \end{tabular}
\end{table}
Table 1: The tasks from four domains studied in this paper, with examples. See Section 2.4 for more details.



with framing dimensions. We predict the primary frame of a document, treating the problem as a 15-way classification task. We bin by timestamp (2009-2010, 2011-2012, 2013-2014, 2015-2016).

Domain 4: Food ReviewsFood and restaurant reviews have been widely studied in NLP research. We considered this domain as a possible contrast to those above, expecting less temporal change. Using data from the Yelp Open Dataset,7 we consider one task:

Footnote 7: [https://www.yelp.com/dataset](https://www.yelp.com/dataset)

Review rating classification**(YelpCLs)This is a conventional sentiment analysis task, mapping the text of a review to the numerical rating given by its author Pang et al. (2002); Dave et al. (2003). We partition the data by year (2013 to 2019) and ensure that each timestep has a roughly equal amount of reviews.

## 3 Empirical Results and Analysis

In this section, we summarize our experimental analysis, resulting from more than 500 experiments. In our experiments, we primarily explore the effect of temporal misalignment on GPT2 Brown et al. (2020), a LM often used for generation.8 We report the macro \(F_{1}\) score for classification tasks and _Rouge-L_Lin (2004) for NewSum.

Footnote 8: In our preliminary results, we found that BERT, RoBERTa, and GPT2 models showed similar patterns.

We first focus on quantifying temporal misalignment in end tasks. As a preliminary analysis, we investigate how the marginal distribution over labels changes over time. We then study how temporal misalignment affects performance of GPT2 models in downstream tasks with temporal finetuning (\(Q_{1}\),\(Q_{2}\)). We find that the amount of performance degradation can vary by task; in some cases the degradation can be severe.

We then study how temporal misalignment affects LMs. As a first step, we analyze how vocabularies change over time in our datasets. We then experiment with (\(Q_{3}\)) how temporal misalignment affects upstream language modeling and (\(Q_{4}\)) how effective temporal adaptation, or additional pretraining on a target year, is in mitigating misalignment. We find that while LMs are affected by misalignment, temporal domain adaptation is not enough to mitigate temporal misalignment.

Details on temporal domain adaptation and fine-tuning, and an extended version of our results, can be found in Appendices B and D, respectively.

### Temporal Misalignment in Tasks

How much does misalignment affect task performance? We find that it depends on the task.

Label Distribution DriftWe first investigate how task datasets undergo changes in the marginal distribution over labels due to time. For each task and each test period, we calculate the KL divergence between the label distributions in that period and the first test period. Full results are reported in Fig. 3. In three cases, we detected notable label distribution drift: PoliAff, AIC, and MFC.9 In Poliaff, Republican tweets outnumbered Democratic ones by over a 2:1 ratio in 2015, but the reverse held by 2020. This observation shows that, regardless of the properties of NLP models, the nature of many tasks changes over time, if only because the output distribution changes.

Footnote 9: For other tasks, it is possible that the data collection/annotation procedures suppressed label distribution changes that would be visible in data “from the wild.”

FinetuningAs described in SS2.4, for each task, we create training and evaluation sets associated with different time periods. We finetune GPT2 on each of the task's training sets and evaluate each on two evaluation sets. Note that there is no domain adaptation here.

Fig. 4 shows our results on downstream tasks (with no domain adaptation). To get more reliable estimates, each number in this heatmap is an average of five independent experiments with different random seeds. A summary of the fine-tuning re

Figure 3: KL divergence between label distributions over time for a subset of tasks. See Appendix D for full results. For each cell, we compare the distribution of labels to that of the first time period; e.g., the 2017 PoliAff cell contains the KL-divergence between the label distributions of PoliAff in 2017 and 2015. While most tasks see little change over time, PoliAff and MFC see a large shift.

 

sults, in terms of TD scores (SS2.3) is in Table 2 which indicates the speed of temporal degradation, for every year that the training and evaluation data diverges. Recall that this score (applied to task performance measures) summarizes the strength of the effect of temporal misalignment on the score, using evidence from across experiments.

\(Q_{1}\)) Temporal misalignment degrades task performance substantially.Fig. 4, similar to earlier work Rottger and Pierrehumbert (2021), shows that models trained on data from the same time period as the test data perform far better than those from the past. The performance drop is most severe for PoliAff (TD=\(7.72\)) and PubCls (TD=\(5.45\)).

\(Q_{1}\)) Temporal misalignment has a measurable effect on most tasks.Half of our tasks see an average loss of at least 1 point for each time period that the training data diverges from the test data. For datasets like SciERC that make use of data from three decades or more, this effect could add up.

Moreover, 1 point of difference can be meaningful, especially for the summarization task where we measure Rouge-L. According to the leaderboard,10 the best three performing models are within a point of each other in Rouge-L Shi et al. (2019, 2021); Mendes et al. (2019). The task has a TD score of 1.38. On average, a time period of temporal misalignment results has a larger effect on performance than changing between the three best models.

Footnote 10: [https://lil.nlp.cornell.edu/newsroom/index.html](https://lil.nlp.cornell.edu/newsroom/index.html)

\(Q_{1}\)) Performance loss from temporal misalignment occurs in both directions.Another observation in Table 4 is that degradation happens in both directions (past and future). While most of the emphasis on temporal misalignment is on how to adapt our stale models/data to the present time Dhingra et al. (2021); Lazaridou et al. (2021); Rottger and Pierrehumbert (2021), our experiments also show that models trained on newer data can be misaligned from the past, as well. Weak performance in older texts has been noted in NLP for historical documents Yang and Eisenstein (2016); Han and Eisenstein (2019). However, our findings indicate deterioration can occur sooner--just a few years rather than deca

Figure 4: Temporal misalignment in finetuning affects task performance (§3.1). In all cases, higher scores are better. The heatmap is shaded per column, i.e., the darkest shade of orange in a cell means the cell has the highest score in that column. Mismatch between the the training and evaluation data can result in massive performance drop; the degree varies by task. For example, Yelpcls shows minimal degradation. In contrast, PoliAff shows major deterioration over time. Additional tables for remaining tasks can be found in Appendix D.

 

(\(Q_{2}\)) Tasks, even in the same domain, are affected differently.Consider the two tasks of PoliAff and TwiERC (both in the Twitter domain), with TD scores of 7.72 and 0.96, respectively. Of our 8 tasks, TwiERC, MFC, and YelpCls are the most robust to temporal misalignment (TD scores of 0.96, 0.98 and 0.26, respectively). The high levels of variation show that temporal misalignment affects performance through labeled datasets, not just unlabeled pretraining data.

### Temporal Misalignment in LMs

As LMs are widely used in modern NLP systems, it is important to inspect how robust they are to temporal misalignment. We seek to understand how temporal misalignment affects the language modeling task in our four domains and if temporal domain adaptation helps in downstream tasks.

Vocabulary ShiftWe first consider an extremely simple measurement of language shift: how do vocabularies change across time periods?11 We use a similar procedure to the one Gururangan et al. (2020) used for analyzing domain similarity. Fixing a domain, we compare the (unigram) vocabularies of each pair of training sets. The vocabularies are built using the 10K most frequent terms from each time period. We note that vocabulary overlap is higher between two time periods the closer they are. Most domains see a sizeable amount of shift; however, Yelp is relatively stagnant. Fig. 5 visualizes the overlap measurement. Table 6 in Appendix D shows the correlation between model performance and the word overlap.

Footnote 11: This can be understood as a model-free way to measure covariate shift for NLP tasks that take text as input.

Temporal Domain AdaptationResearchers have studied the broader problem of distributional shift Shimodaira (2000); Zhang et al. (2013). The NLP community has historically tackled these problems via domain adaptation Jiang and Zhai (2007); Daume III (2007); Gururangan et al. (2020). Taking inspiration from these approaches, we next apply DAPT to GPT2, treating each time period as a domain: for each time period, we continue pretraining and then evaluate perplexity. We consider how the perplexity varies with the (mis)alignment between the DAPT training data and the evaluation data. We measure the TD score, which summarizes how much performance is affected by temporal misalignment (now applied to perplexity). The results of temporal domain adaptation are in Fig. 6.

(\(Q_{3}\)) Domains are a major driver of temporal misalignment in LMs.Consistent with Lazaridou et al. (2021), Fig. 6 shows degradation of LM due to temporal misalignment; it further shows considerable variation by text domain. Twitter changes most rapidly, and food reviews are much slower. This observation is consistent with past work on language change in social media Stewart and Eisenstein (2018); Eisenstein et al. (2014). To the extent that a LM's practical usefulness is associated with its fit to new data, researchers and practitioners should understand the temporal dynamics of their target text domains and plan LM updates accordingly.

Joint Effects of Temporal Adaptation and FinetuningAs discussed in SS2, continued pretraining of an LM on in-domain text has been shown to improve task performance. Our prior results show that both downstream tasks and language modeling are affected by temporal misalignment. Can temporal domain adaptation help mitigate the effects of misalignment in downstream tasks?

Here we consider how the time period of the data selected for continued pretraining affects task performance. For each task's evaluation set, we apply DAPT twice: once with the earliest available time period's unannotated data and once with the latest's. We then finetune and evaluate on data from the same time periods as in the earlier experiment.

(\(Q_{4}\)) Temporal adaptation does not overcomedegradation from temporally misaligned labeled data.In Table 3, we see small performance gains from temporal domain adaptation on LMs, and in some cases it is harmful. These observations underscore the importance of the labeled data; adjustments to the LM alone do not yet appear sufficient to mitigate the effects of temporal misalignment. In contrast to temporal domain adaptation, which does not mitigate temporal misalignment's effects, finetuning on temporally-updated labeled data is more effective.

This can be observed in each task-specific subtable of in Table 3: the top-left and bottom-right quadrants (fine-tuning on time-stamp that is used for evaluation) generally lead to higher scores.

## 4 Limitations and Future Work

We provided a well-controlled suite of experiments to study the effects of temporal misalignment on model performance. However, the setup has some 

drawbacks. For example, we expect that models trained on data accumulated across multiple time periods would perform well Lazaridou et al. (2021); Rottger and Pierrehumbert (2021); Jin et al. (2021).

We chose the time periods in our study so that they would each have sufficient and consistent training data sizes. However, amounts of data in a particular domain or task will fluctuate over time. Moreover, the rate of language use change may not be uniform. Time periods should be selected with these two considerations in mind.

Our findings indicate that temporal misalignment's effects depend heavily on the task. Though not studied here, the same issues may arise in annotation efforts; consider, for example, recent work on controversy Zhang et al. (2018) and social norms Xu et al. (2021); Zhou et al. (2021) likely hinges on constructs that may be time sensitive. Annotations that are temporally misaligned with the original data being annotated may be anachronistic.

An opportunity for future exploration is in the context of real-world events with sudden changes such as COVID-19 pandemic Cao et al. (2021) or political changes, which influence tasks such as question answering Dhingra et al. (2021); Zhang and Choi (2021).

Extensive work has been done on modeling and detecting lexical semantic change, or how words evolve in meaning Hamilton et al. (2016); Rudolph and Blei (2018); Gonen et al. (2020). Techniques

Figure 5: Vocabulary overlap between time periods, over a subset of our tasks’ datasets. Each cell shows the % overlap between the vocabularies of two time periods.

 

and intuition from this body of work may be useful in finding solutions to mitigate degradation due to misalignment. We believe that this phenomenon is an important aspect of temporal misalignment, but leave disentangling semantic shifts from other, perhaps task-related factors, for future work.

Continual learning, which allows models to learn from a continuous stream of data, could also be one way to mitigate temporal misalignment. Most prior work in this space has focused on continual learning in LMs Jin et al. (2021) or learning disparate tasks de Masson d'Autume et al. (2019); Huang et al. (2021). Future work may investigate continual learning algorithms for tasks that change over time.

Our results showed that straightforward domain adaptation was unable to mitigate the effects of temporal misalignment. Recent work in language modeling has elevated the importance of domains by using a mixture of domains Gururangan et al. (2021) or giving domains a hierarchical structure Chronopoulou et al. (2021). More sophisticated approach to domains, in line with these works, could lead to temporally robust models.

While we found that task-specific finetuning is more effective than temporal adaptation, new labeled data can be expensive. Ways to characterize or detect changes in a task could be helpful in efficiently updating datasets Lu et al. (2019); Webb et al. (2018). Future work can also treat dataset maintenance as an optimization problem between the cost and gains of annotating new data Bai et al. (2021).

## 5 Conclusion

Changes in language use over time, and how language relates to other quantities of interest in NLP applications, has clear effects on the performance of those applications. We have explored how temporal misalignment between training data--both data used to train LMs and annotated data used to finetune them--affects performance across a range of NLP tasks and domains, taking advantage of datasets where timestamps are available. We compile these datasets as a benchmark for future research as well. We also introduced a summary metric, TD score, that makes it easier to compare models in terms of their temporal misalignment.

Our experiments revealed considerable variation in temporal degradation accross tasks, more so than found in previous studies Rottger and Pierrehumbert (2021). These findings motivate continued study of temporal misalignment across applications of NLP, its consideration in benchmark evaluations,12 and vigilance on the part of practitioners able to monitor live system performance over time.

Footnote 12: Indeed, for benchmarks where training and testing data _are_ aligned, our findings suggest that measures of performance may be in some cases inflated.

Notably, we observed that continued training of LMs on temporally aligned data does not have much effect, motivating further research to find effective temporal adaptation methods that are less costly than ongoing collection of annotated/labeled datasets over time.

#### Acknowledgments

We thank Dallas Card, Sihao Chen, Alexander Fabbri, Jack Hessel, and Shruti Rijhwani for their help in curating our data. We thank Jacob Eisenstein, Rahul Nadkarni, the ARK and XLab research groups, and our anonymous reviewers for their feedback on this work. We also acknowledge the Beaker team (_[https://beaker.org_](https://beaker.org_)) for their support with experiments. This research was supported in part by the Office of Naval Research under MURI grant N00014-18-1-2670.

Figure 6: Perplexity of GPT2 after adaptive pretraining on temporally-selected data in different domains (lower is better). The TD score (in parentheses) estimates the expected perplexity rise (i.e., degradation) for every time period of misalignment between evaluation and training times. Degradation follows the expected pattern, but the magnitude varies by domain.

 



## References

* Altmann et al. (2009) Eduardo G Altmann, Janet B Pierrehumbert, and Adilson E Motter. 2009. Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words. _PLOS one_, 4(11):e7678.
* Ammar et al. (2018) Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, et al. 2018. Construction of the literature graph in semantic scholar. In _NAACL_.
* Bai et al. (2021) Fan Bai, Alan Ritter, and Wei Xu. 2021. Pre-train or annotate? domain adaptation with a constrained budget. In _EMNLP_, pages 5002-5015.
* Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In _EMNLP_.
* Bothwick (1999) Andrew Eliot Bothwick. 1999. _A maximum entropy approach to named entity recognition_. New York University.
* Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neekalantam, Prauay Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Cao et al. (2021) Ivy Cao, Zizhou Liu, Giannis Karamanolakis, Daniel Hsu, and Luis Gravano. 2021. Quantifying the effects of COVID-19 on restaurant reviews. In _Proceedings of the International Workshop on Natural Language Processing for Social Media_, Online. Association for Computational Linguistics.
* Card et al. (2015) Dallas Card, Amber E. Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The media frames corpus: Annotations of frames across issues. In _ACL_.
* Chronopoulou et al. (2021) Alexandra Chronopoulou, Matthew E. Peters, and Jesse Dodge. 2021. Efficient hierarchical domain adaptation for pretrained language models. _ArXiv_, abs/2112.08786.
* Daume III (2007) Hal Daume III. 2007. Frustratingly easy domain adaptation. In _ACL_, pages 256-263.
* Dave et al. (2003) Kwshal Dave, Steve Lawrence, and David M. Pennock. 2003. Mining the peanut gallery: opinion extraction and semantic classification of product reviews. In _WWW_.
* de Masson d'Autume et al. (2019) Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In _nips_.
* Dinggra et al. (2021) Zhuwan Dinggra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2021. Time-aware language models as temporal knowledge bases. _CoRR_, abs/2106.15110.
* Dodge et al. (2021) Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In _EMNLP_.
* Eisenstein et al. (2014) Jacob Eisenstein, Brendan O'Connor, Noah A Smith, and Eric P Xing. 2014. Diffusion of lexical change in social media. _PloS one_, 9(11).
* Entman (1983) Robert M. Entman. 1983. Framing: Toward clarification of a fractured paradigm. _Journal of Communications_.
* Gonen et al. (2020) Hila Gonen, Ganesh Jawahar, Djame Seddah, and Yoav Goldberg. 2020. Simple, interpretable and stable method for detecting words with usage change across corpora. In _ACL_.
* Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In _NAACL_.
* Gururangan et al. (2021) Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2021. Demix layers: Disentangling domains for modular language modeling. _CoRR_, abs/2108.05036.
* Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapl language models to domains and tasks. In _ACL_.
* Hamilton et al. (2016) William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In _ACL_.
* Han and Eisenstein (2019) Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In _EMNLP_.
* Huang et al. (2021) Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021. Continual learning for text classification with information disentanglement based regularization. In _ACL_.
* Iyyer et al. (2014) Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In _ACL_.
* Jaidka et al. (2018) Kokli Jaidka, Niyati Chhaya, and Lyle Ungar. 2018. Diachronic degradation of language models: Insights from social media. In _ACL_.
* Jiang and Zhai (2007) Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in nlp. In _ACL_.
* Jin et al. (2021) Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2021. Lifelong pretraining: Continually adapting language models to emerging corpora. _arXiv preprint arXiv:2110.08534_.
* Labov (2011) William Labov. 2011. _Principles of linguistic change, volume 3: Cognitive and cultural factors_, volume 36. John Wiley & Sons.



* Lazaridou et al. (2021) Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprin de Masson d'Autume, Sebastian Ruder, Dani Yogatama, et al. 2021. Pitfalls of static language modelling. _arXiv preprint arXiv:2102.01951_.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Proc. of Text Summarization Branches Out_.
* Lin et al. (2008) Wei-Hao Lin, Eric P. Xing, and Alexander Hauptmann. 2008. A joint topic and perspective model for ideological discourse. In _ECML/PKDD_.
* Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. 2020. S2orc: The semantic scholar open research corpus. In _ACL_.
* Lu et al. (2019) Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Garna, and Guangquan Zhang. 2019. Learning under concept drift: A review. _IEEE Transactions on Knowledge and Data Engineering_.
* Luan et al. (2018) Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In _EMNLP_.
* Mendes et al. (2019) Afonso Mendes, Shashi Narayan, Sebastiao Miranda, Zita Marinho, Andre F. T. Martins, and Shay B. Cohen. 2019. Jointly extracting and compressing documents with summary state representations. In _NAACL_.
* Pang et al. (2002) Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In _EMNLP_.
* Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _NAACL_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _JMLR_, 21:1-67.
* Rijhwani and Preotiuc-Pietro (2020) Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020. Temporally-informed analysis of named entity recognition. In _ACL_.
* Rottger and Pierrehumbert (2021) Paul Rottger and Janet Pierrehumbert. 2021. Temporal adaptation of bert and performance on downstream document classification: Insights from social media. In _Findings of EMNLP_, pages 2400-2412.
* Rudolph and Blei (2018) Maja R. Rudolph and David M. Blei. 2018. Dynamic embeddings for language evolution. _WWW_.
* Shi et al. (2021) Tian Shi, Yaser Keneshloo, Naren Ramakrishnan, and Chandan K. Reddy. 2021. Neural abstractive text summarization with sequence-to-sequence models. _ACM Transactions on Data Science_.
* Shi et al. (2019) Tian Shi, Ping Wang, and Chandan K. Reddy. 2019. LeafNATS: An open-source toolkit and live demo system for neural abstractive text summarization. In _NAACL_.
* Shimodaira (2000) Hidetoshi Shimodaira. 2000. Improving predictive inference under covariate shift by weighting the log-likelihood function. _Journal of Statistical Planning and Inference_.
* Stewart and Eisenstein (2018) Ian Stewart and Jacob Eisenstein. 2018. Making "fetch" happen: The influence of social and linguistic context on nonstandard word growth and decline. In _EMNLP_.
* Subramanian et al. (2021) Shivashankar Subramanian, Daniel King, Doug Downey, and Sergey Feldman. 2021. S2and: A benchmark and evaluation system for author name disambiguation. In _ACM/IEEE Joint Conference on Digital Libraries (JCDL)_, pages 170-179. IEEE.
* Tamburrini et al. (2015) Nadine Tamburrini, Marco Cinnirella, Vincent AA Jansen, and John Bryden. 2015. Twitter users change word usage according to conversation-partner social identity. _Social Networks_, 40:84-89.
* Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zwylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In _EMNLP_.
* Wang and Goutte (2017) Yunli Wang and Cyril Goutte. 2017. Detecting changes in twitter streams using temporal clusters of hashtags. In _Proceedings of the Events and Stories in the News Workshop_.
* Webb et al. (2018) Geoffrey I. Webb, Loong Kuan Lee, Bart Goethals, and Francois Petitjean. 2018. Analyzing concept drift and shift from sample data. _Data Mining and Knowledge Discovery_.
* Xu et al. (2021) Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021. Detoxifing language models risks marginalizing minority voices. In _NAACL_.
* Yang and Eisenstein (2016) Yi Yang and Jacob Eisenstein. 2016. Part-of-speech tagging for historical english. In _NAACL_.
* Zhang et al. (2018) Justine Zhang, Jonathan Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Dario Taraborelli, and Nithin Thain. 2018. Conversations gone awry: Detecting early signs of conversational failure. In _ACL_.
* Zhang et al. (2013) Kun Zhang, Bernhard Scholkopf, Krikamol Muandet, and Zhikun Wang. 2013. Domain adaptation under target and conditional shift. In _ICML_.
* Zhang and Choi (2021) Michael J.Q. Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. _EMNLP_.
* Zhang et al. (2020) Yi Zhang, Zachary Ives, and Dan Roth. 2020. "who said it, and why?" provenance for natural language claims. In _ACL_ 

Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah A. Smith. 2021. Challenges in automated debiasing for toxic language detection. In _EACL_.

 



## Supplementary Material

### A Metric for Temporal Degradation

Let \(t\) be the time period of the training data and \(t^{\prime}\) the time period of the evaluation data.13 We aim to summarize the general effect of temporal misalignment (the difference between \(t\) and \(t^{\prime}\)) on task performance, in an interpretable way that is comparable across tasks.

Footnote 13: See examples in Fig. 4.

Let \(S_{t^{\prime}\to t}\) indicate the performance a model trained on timestamp \(t^{\prime}\) data and evaluated on the timestamp \(t\). Let

\[D(t^{\prime}\to t)=-\left(S_{t^{\prime}\to t}-S_{t\sim t}\right)\times\text{ sign}(t^{\prime}-t),\]

In other words, \(D(t^{\prime}\to t)\) is a modified difference in performance between a aligned and misaligned models. The modification ensures that, as performance deteriorates, \(D\) increases, regardless of the direction of time between \(t\) and \(t^{\prime}\).

Our temporal degradation (TD) score for a fixed evaluation timestamp \(t\) for models trained on a set of timestamps \(\mathcal{T}\) is defined as:

where \(\bar{t}=\text{avg}_{t\in\mathcal{T}}t^{\prime}\) and \(\bar{D}=\text{avg}_{t\in\mathcal{T}}D(t^{\prime}\to t)\). This metric is the _slope_ of a line fitting the the performance change of models trained on a variety of timestamps, when evaluated on a fixed timestamp. It can be interpreted as the average rate of performance deterioration per time period.

Fig. 7 shows three examples of TD scores from PoliAff (the first) and YelpCls (the latter two). These illustrate cases with and without temporal sensitivity. In practice, most examples with deterioration showed a linear trend and thus the rate of degradation was suitible to be approximated by a line. The final TD score is averaged over all evaluation years \(\mathcal{T}^{\prime}\).

\[\text{TD}=\frac{\sum_{t\in\mathcal{T}^{\prime}}\text{TD}(\mathcal{T}\to t)}{| \mathcal{T}^{\prime}|}\]

AicWe randomly sample science documents in Semantic Scholar's corpus.15 Of those, we only keep documents that (1) are published in ICML or AAAI, (2) are classified as 'computer science' documents, and (3) have an abstract of at least 50 tokens.

Footnote 15: [https://api.semanticscholar.org/corpus/](https://api.semanticscholar.org/corpus/); licensed under an ODC-BY

NewsroomThe following applies to the postprocessing and data selection for both supervised temporal finetuning and unsupervised temporal adaptation of PubCls and NewSum. We use the Newsroom dataset.16. We only keep articles where (1) the year in the metadata also appears in the main text and (2) no future year is mentioned in the main text.

PubClsWe carry out additional postprocessing and ensure that each of the three labels (Fox News, New York Times, and Washington Post) have an equal distribution across years. We do so by uniform-random downsampling.

Footnote 16: [https://lil.nlp.cornell.edu/newsroom/](https://lil.nlp.cornell.edu/newsroom/)

## Appendix D Extended Results

We provide further results from our experiments described in Section 3.

Word Overlap Correlation with PerformanceIn addition to measuring vocabularies' change over time in Section 3.2, we find correlations between the word overlap and model performance of each task in Table 6.

Finetuning ResultsWe provide the full results from our fientuning experiments in Section 3.1 in Fig. 8. These results are for downstream tasks with no domain adaptation.

Finetuning with Temporal Domain AdaptationWe provide the full results from our finetuning with temporal domain adaptation in Section 3.2 in Fig. 7.

\begin{table}
\begin{tabular}{c|l|l} \hline \hline Domain & Task (metric) & Pearson’s \(r\) \\ \hline Twitter & PoliAff (\(F_{1}\)) & \(0.84\) \\  & TwiERC (\(F_{1}\)) & \(0.51\) \\ \hline Science & SciERC (\(F_{1}\)) & \(0.72\) \\  & AIC (\(F_{1}\)) & \(0.79\) \\ \hline News & PubCls (\(F_{1}\)) & \(0.65\) \\  & NewSum (Rouge-L) & \(0.72\) \\  & MFC (\(F_{1}\)) & \(0.80\) \\ \hline Reviews & YelpCls (\(F_{1}\)) & \(0.14\) \\ \hline \hline \end{tabular}
\end{table}
Table 6: Pearson \(r\) correlation coeffecients between the word overlap and performance of each task.

Figure 7: Three example calculations of the TD score (left from PoliAff and the center and right from YelpCls). The annotated numbers are the raw evaluation scores \(S_{t^{\prime}\to t}\) and the plotted markers represent the modified differences \(D(t^{\prime}\to t)\) discussed in Section 2.3. For a particular plot, the red line is the line of best fit and its slope is the TD\((t)\) score for evaluation timestep \(t\). The final TD score is averaged between all evaluation timesteps for the particular task.

 



\begin{table}
\begin{tabular}{|c|c|l||c|l