SlowGuess/NAACL_2022_OCR huggingface dataset:
period as the test data perform far better than those from the past. The performance drop is most severe for POLIAFF (TD=7.72) and PUBCLS (TD=5.45).
$(Q_{1})$  Temporal misalignment has a measurable effect on most tasks. Half of our tasks see an average loss of at least 1 point for each time period that the training data diverges from the test data. For datasets like ScIERC that make use of data from three decades or more, this effect could add up.
Moreover, 1 point of difference can be meaningful, especially for the summarization task where we measure Rouge-L. According to the leaderboard,[10] the best three performing models are within a point of each other in Rouge-L (Shi et al., 2019, 2021; Mendes et al., 2019). The task has a TD score of 1.38. On average, a time period of temporal misalignment results has a larger effect on performance than changing between the three best models.
$(Q_{1})$  Performance loss from temporal misalignment occurs in both directions. Another observation in Table 4 is that degradation happens in both directions (past and future). While most of the emphasis on temporal misalignment is on how to adapt our stale models/data to the present time (Dhingra et al., 2021; Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021), our experiments also show that models trained on newer data can be misaligned from the past, as well. Weak performance in older texts has been noted in NLP for historical documents (Yang and Eisenstein, 2016; Han and Eisenstein, 2019). However, our findings indicate deterioration can occur sooner—just a few years rather than decades or centuries.
10 https://lil.nlp.cornell.edu/newsroom/index.html
$(Q_{2})$  Tasks, even in the same domain, are affected differently. Consider the two tasks of POI LIAFF and TWIERC (both in the Twitter domain), with TD scores of 7.72 and 0.96, respectively. Of our 8 tasks, TWIERC, MFC, and YELPCLS are the most robust to temporal misalignment (TD scores of 0.96, 0.98 and 0.26, respectively). The high levels of variation show that temporal misalignment affects performance through labeled datasets, not just unlabeled pretraining data.
3.2 Temporal Misalignment in LMs
As LMs are widely used in modern NLP systems, it is important to inspect how robust they are to temporal misalignment. We seek to understand how temporal misalignment affects the language modeling task in our four domains and if temporal domain adaptation helps in downstream tasks.
Vocabulary Shift We first consider an extremely simple measurement of language shift: how do vocabularies change across time periods?11 We use a similar procedure to the one Gururangan et al. (2020) used for analyzing domain similarity. Fixing a domain, we compare the (unigram) vocabularies of each pair of training sets. The vocabularies are built using the 10K most frequent terms from each time period. We note that vocabulary overlap is higher between two time periods the closer they are. Most domains see a sizeable amount of shift; however, Yelp is relatively stagnant. Fig. 5 visualizes the overlap measurement. Table 6 in Appendix D shows the correlation between model performance and the word overlap.
Temporal Domain Adaptation Researchers have studied the broader problem of distributional shift (Shimodaira, 2000; Zhang et al., 2013). The NLP community has historically tackled these problems via domain adaptation (Jiang and Zhai, 2007; Daumé III, 2007; Gururangan et al., 2020). Taking inspiration from these approaches, we next apply DAPT to GPT2, treating each time period as a domain: for each time period, we continue pretraining and then evaluate perplexity. We consider how the perplexity varies with the (mis)alignment between the DAPT training data and the evaluation data. We measure the TD score, which summarizes how much performance is affected by temporal misalignment (now applied to perplexity). The results of temporal domain adaptation are in Fig. 6.
$(Q_{3})$  Domains are a major driver of temporal misalignment in LMs. Consistent with Lazari-dou et al. (2021), Fig. 6 shows degradation of LM due to temporal misalignment; it further shows considerable variation by text domain. Twitter changes most rapidly, and food reviews are much slower. This observation is consistent with past work on language change in social media (Stewart and Eisenstein, 2018; Eisenstein et al., 2014). To the extent that a LM's practical usefulness is associated with its fit to new data, researchers and practitioners should understand the temporal dynamics of their target text domains and plan LM updates accordingly.
Joint Effects of Temporal Adaptation and Finetuning As discussed in §2, continued pretraining of an LM on in-domain text has been shown to improve task performance. Our prior results show that both downstream tasks and language modeling are affected by temporal misalignment. Can temporal domain adaptation help mitigate the effects of misalignment in downstream tasks?
Here we consider how the time period of the data selected for continued pretraining affects task performance. For each task's evaluation set, we apply DAPT twice: once with the earliest available time period's unannotated data and once with the latest's. We then finetune and evaluate on data from the same time periods as in the earlier experiment.
$(Q_{4})$  Temporal adaptation does not overcome degradation from temporally misaligned labeled data. In Table 3, we see small performance gains from temporal domain adaptation on LMs, and in some cases it is harmful. These observations underscore the importance of the labeled data; adjustments to the LM alone do not yet appear sufficient to mitigate the effects of temporal misalignment. In contrast to temporal domain adaptation, which does not mitigate temporal misalignment's effects, finetuning on temporally-updated labeled data is more effective.
This can be observed in each task-specific subtable of in Table 3: the top-left and bottom-right quadrants (fine-tuning on time-stamp that is used for evaluation) generally lead to higher scores.
4 Limitations and Future Work
We provided a well-controlled suite of experiments to study the effects of temporal misalignment on model performance. However, the setup has some
This can be understood as a model-free way to measure covariate shift for NLP tasks that take text as input.
Figure 5: Vocabulary overlap between time periods, over a subset of our tasks' datasets. Each cell shows the  $\%$  overlap between the vocabularies of two time periods.
Domain (Task) ↓	Finetune Year ↓	Evaluation → Pretrain ↓	2015	2020	Domain (Task) ↓	Finetune Year ↓	Evaluation → Pretrain ↓	1980-1999	2010-2016
Twitter (PoliAff) F1	2015	Default	91.4	48.4	Scientific (SciERC) F1	1980-1999	Default	67.9	57.2
Default → 2015	92.2	47.5	Default → 1980-1999	73.2	66.4
Default → 2020	90.9	50.8	Default → 2010-2016	73.7	66.8
2020	Default	45.8	78.0	2010-2016	Default	60.3	72.5
Default → 2015	47.2	76.9	Default → 1980-1999	63.4	75.0
Default → 2020	44.2	78.3	Default → 2010-2016	64.8	76.0
Domain (Task) ↓	Finetune Year ↓	Evaluation → Pretrain ↓	2009-2010	2015-2016	Domain (Task) ↓	Finetune Year ↓	Evaluation → Pretrain ↓	2014	2019
News (NewsSum) Rouge-L	2009-2010	Default	36.4	29.0	Food Reviews (Yelp) F1	2009-2010	Default	58.6	58.3
Default → 2009-2010	36.4	29.1	Default → 2014	63.3	60.1
Default → 2015-2016	36.1	28.9	Default → 2019	60.2	62.3
2015-2016	Default	27.8	31.8	2015-2016	Default	58.3	58.3
Default → 2009-2010	28.2	31.8	Default → 2014	60.2	62.3
Default → 2015-2016	27.8	31.6	Default → 2019	60.8	62.3
Table 3: Combination of temporal adaptation and finetuning (§3.2) on our tasks. The row labeled "Default" corresponds to a model that has not been adapted (uses the default pretraining). The models with temporal domain adaptation are shown in rows labeled "Default  $\rightarrow$  y" and each is comparable to the "Default" row above it. The color coding is proportional to the magnitude of the performances of each task (darker shade of orange indicates higher scores). It can be observed that temporal finetuning has a greater impact than temporal pretraining. Each quadrant of 3 for each task, indicating the same finetune and evaluation years, but different pretraining conditions, are mostly uniform. In contrast, we notice a sharper difference in performance when varying the finetuning year (comparing the quadrants vertically).
drawbacks. For example, we expect that models trained on data accumulated across multiple time periods would perform well (Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021; Jin et al., 2021).
We chose the time periods in our study so that they would each have sufficient and consistent training data sizes. However, amounts of data in a particular domain or task will fluctuate over time. Moreover, the rate of language use change may not be uniform. Time periods should be selected with these two considerations in mind.
Our findings indicate that temporal misalignment's effects depend heavily on the task. Though not studied here, the same issues may arise in annotation efforts; consider, for example, recent
work on controversy (Zhang et al., 2018) and social norms (Xu et al., 2021; Zhou et al., 2021) likely hinges on constructs that may be time sensitive. Annotations that are temporally misaligned with the original data being annotated may be anachronistic.
An opportunity for future exploration is in the context of real-world events with sudden changes such as COVID-19 pandemic (Cao et al., 2021) or political changes, which influence tasks such as question answering (Dhingra et al., 2021; Zhang and Choi, 2021).
Extensive work has been done on modeling and detecting lexical semantic change, or how words evolve in meaning (Hamilton et al., 2016; Rudolph and Blei, 2018; Gonen et al., 2020). Techniques
Figure 6: Perplexity of GPT2 after adaptive pretraining on temporally-selected data in different domains (lower is better). The TD score (in parentheses) estimates the expected perplexity rise (i.e., degradation) for every time period of misalignment between evaluation and training times. Degradation follows the expected pattern, but the magnitude varies by domain.
and intuition from this body of work may be useful in finding solutions to mitigate degradation due to misalignment. We believe that this phenomenon is an important aspect of temporal misalignment, but leave disentangling semantic shifts from other, perhaps task-related factors, for future work.
Continual learning, which allows models to learn from a continuous stream of data, could also be one way to mitigate temporal misalignment. Most prior work in this space has focused on continual learning in LMs (Jin et al., 2021) or learning disparate tasks (de Masson d'Autume et al., 2019; Huang et al., 2021). Future work may investigate continual learning algorithms for tasks that change over time.
Our results showed that straightforward domain adaptation was unable to mitigate the effects of temporal misalignment. Recent work in language modeling has elevated the importance of domains by using a mixture of domains (Gururangan et al., 2021) or giving domains a hierarchical structure (Chronopoulou et al., 2021). More sophisticated approach to domains, in line with these works, could lead to temporally robust models.
While we found that task-specific finetuning is more effective than temporal adaptation, new labeled data can be expensive. Ways to characterize or detect changes in a task could be helpful in efficiently updating datasets (Lu et al., 2019; Webb
et al., 2018). Future work can also treat dataset maintenance as an optimization problem between the cost and gains of annotating new data (Bai et al., 2021).
Changes in language use over time, and how language relates to other quantities of interest in NLP applications, has clear effects on the performance of those applications. We have explored how temporal misalignment between training data—both data used to train LMs and annotated data used to finetune them— affects performance across a range of NLP tasks and domains, taking advantage of datasets where timestamps are available. We compile these datasets as a benchmark for future research as well. We also introduced a summary metric, TD score, that makes it easier to compare models in terms of their temporal misalignment.
Our experiments revealed considerable variation in temporal degradation across tasks, more so than found in previous studies (Röttger and Pierrehumbert, 2021). These findings motivate continued study of temporal misalignment across applications of NLP, its consideration in benchmark evaluations, $^{12}$  and vigilance on the part of practitioners able to monitor live system performance over time.
Notably, we observed that continued training of LMs on temporally aligned data does not have much effect, motivating further research to find effective temporal adaptation methods that are less costly than ongoing collection of annotated/labeled datasets over time.
We thank Dallas Card, Sihao Chen, Alexander Fabbri, Jack Hessel, and Shruti Rijhwani for their help in curating our data. We thank Jacob Eisenstein, Rahul Nadkarni, the ARK and XLab research groups, and our anonymous reviewers for their feedback on this work. We also acknowledge the Beaker team (https://beaker.org) for their support with experiments. This research was supported in part by the Office of Naval Research under MURI grant N00014-18-1-2670.
Indeed, for benchmarks where training and testing data are aligned, our findings suggest that measures of performance may be in some cases inflated.
Eduardo G Altmann, Janet B Pierrehumbert, and Adilson E Motter. 2009. Beyond word frequency: Bursts, lulls, and scaling in the temporal distributions of words. PLOS one, 4(11):e7678.
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, et al. 2018. Construction of the literature graph in semantic scholar. In NAACL.
Fan Bai, Alan Ritter, and Wei Xu. 2021. Pre-train or annotate? domain adaptation with a constrained budget. In EMNLP, pages 5002-5015.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In EMNLP.
Andrew Eliot Borthwick. 1999. A maximum entropy approach to named entity recognition. New York University.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Ivy Cao, Zizhou Liu, Giannis Karamanolakis, Daniel Hsu, and Luis Gravano. 2021. Quantifying the effects of COVID-19 on restaurant reviews. In Proceedings of the International Workshop on Natural Language Processing for Social Media, Online. Association for Computational Linguistics.
Dallas Card, Amber E. Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015. The media frames corpus: Annotations of frames across issues. In ACL.
Alexandra Chronopoulou, Matthew E. Peters, and Jesse Dodge. 2021. Efficient hierarchical domain adaptation for pretrained language models. *ArXiv*, abs/2112.08786.
Hal Daumé III. 2007. Frustratingly easy domain adaptation. In ACL, pages 256-263.
Kushal Dave, Steve Lawrence, and David M. Pennock. 2003. Mining the peanut gallery: opinion extraction and semantic classification of product reviews. In WWW.
Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. In nips.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2021. Time-aware language models as temporal knowledge bases. CoRR, abs/2106.15110.
Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In EMNLP.
Jacob Eisenstein, Brendan O'Connor, Noah A Smith, and Eric P Xing. 2014. Diffusion of lexical change in social media. *PloS one*, 9(11).
Robert M. Entman. 1983. Framing: Toward clarification of a fractured paradigm. Journal of Communications.
Hila Gonen, Ganesh Jawahar, Djamé Seddah, and Yoav Goldberg. 2020. Simple, interpretable and stable method for detecting words with usage change across corpora. In ACL.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In *NAACL*.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2021. Demix layers: Disentangling domains for modular language modeling. CoRR, abs/2108.05036.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In ACL.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statistical laws of semantic change. In ACL.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In EMNLP.
Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021. Continual learning for text classification with information disentanglement based regularization. In ACL.
Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In ACL.
Kokil Jaidka, Niyati Chhaya, and Lyle Ungar. 2018. Diachronic degradation of language models: Insights from social media. In ACL.
Jing Jiang and ChengXiang Zhai. 2007. Instance weighting for domain adaptation in nlp. In ACL.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2021. Lifelong pretraining: Continually adapting language models to emerging corpora. arXiv preprint arXiv:2110.08534.
William Labov. 2011. Principles of linguistic change, volume 3: Cognitive and cultural factors, volume 36. John Wiley & Sons.
Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d'Autume, Sebastian Ruder, Dani Yogatama, et al. 2021. Pitfalls of static language modelling. arXiv preprint arXiv:2102.01951.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Proc. of Text Summarization Branches Out.
Wei-Hao Lin, Eric P. Xing, and Alexander Hauptmann. 2008. A joint topic and perspective model for ideological discourse. In ECML/PKDD.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. 2020. S2orc: The semantic scholar open research corpus. In ACL.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, João Gama, and Guangquan Zhang. 2019. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In EMNLP.
Afonso Mendes, Shashi Narayan, Sebastião Miranda, Zita Marinho, Andre F. T. Martins, and Shay B. Cohen. 2019. Jointly extracting and compressing documents with summary state representations. In *NAACL*.
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In EMNLP.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21:1-67.
Shruti Rijhwani and Daniel Preotciuc-Pietro. 2020. Temporally-informed analysis of named entity recognition. In ACL.
Paul Röttger and Janet Pierrehumbert. 2021. Temporal adaptation of bert and performance on downstream document classification: Insights from social media. In Findings of EMNLP, pages 2400-2412.
Maja R. Rudolph and David M. Blei. 2018. Dynamic embeddings for language evolution. WWW.
Tian Shi, Yaser Keneshloo, Naren Ramakrishnan, and Chandan K. Reddy. 2021. Neural abstractive text summarization with sequence-to-sequence models. ACM Transactions on Data Science.
Tian Shi, Ping Wang, and Chandan K. Reddy. 2019. LeafNATS: An open-source toolkit and live demo system for neural abstractive text summarization. In NAACL.
Hidetoshi Shimodaira. 2000. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference.
Ian Stewart and Jacob Eisenstein. 2018. "fetch" happen: The influence of social and linguistic context on nonstandard word growth and decline. In EMNLP.
Shivashankar Subramanian, Daniel King, Doug Downey, and Sergey Feldman. 2021. S2and: A benchmark and evaluation system for author name disambiguation. In ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 170-179. IEEE.
Nadine Tamburrini, Marco Cinnirella, Vincent AA Jansen, and John Bryden. 2015. Twitter users change word usage according to conversation-partner social identity. Social Networks, 40:84-89.
David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In EMNLP.
Yunli Wang and Cyril Goutte. 2017. Detecting changes in twitter streams using temporal clusters of hashtags. In Proceedings of the Events and Stories in the News Workshop.
Geoffrey I. Webb, Loong Kuan Lee, Bart Goethals, and François Petitjean. 2018. Analyzing concept drift and shift from sample data. Data Mining and Knowledge Discovery.
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, and Dan Klein. 2021. Detoxifying language models risks marginalizing minority voices. In NAACL.
Yi Yang and Jacob Eisenstein. 2016. Part-of-speech tagging for historical english. In *NAACL*.
Justine Zhang, Jonathan Chang, Cristian Danescu-Niculescu-Mizil, Lucas Dixon, Yiqing Hua, Dario Taraborelli, and Nithum Thain. 2018. Conversations gone awry: Detecting early signs of conversational failure. In ACL.
Kun Zhang, Bernhard Scholkopf, Krikamol Muandet, and Zhikun Wang. 2013. Domain adaptation under target and conditional shift. In ICML.
Michael J.Q. Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. EMNLP.
Yi Zhang, Zachary Ives, and Dan Roth. 2020. "who said it, and why?" provenance for natural language claims. In ACL.
Xuhui Zhou, Maarten Sap, Swabha Swayamdipta, Yejin Choi, and Noah A. Smith. 2021. Challenges in automated debiasing for toxic language detection. In EACL.
A A Metric for Temporal Degradation
Let  $t$  be the time period of the training data and  $t'$  the time period of the evaluation data.13 We aim to summarize the general effect of temporal misalignment (the difference between  $t$  and  $t'$ ) on task performance, in an interpretable way that is comparable across tasks.
Let  $S_{t' \to t}$  indicate the performance a model trained on timestamp  $t'$  data and evaluated on the timestamp  $t$ . Let
$$
D \left(t ^ {\prime} \rightarrow t\right) = - \left(S _ {t ^ {\prime} \rightarrow t} - S _ {t \rightarrow t}\right) \times \operatorname {s i g n} \left(t ^ {\prime} - t\right),
$$
In other words,  $D(t' \to t)$  is a modified difference in performance between a aligned and misaligned models. The modification ensures that, as performance deteriorates,  $D$  increases, regardless of the direction of time between  $t$  and  $t'$ .
Our temporal degradation (TD) score for a fixed evaluation timestamp  $t$  for models trained on a set of timestamps  $\mathcal{T}$  is defined as:
$$
\mathrm {T D} (\mathcal {T} \to t ^ {\prime}) = \left| \frac {\sum_ {t \in \mathcal {T}} \left(D (t ^ {\prime} \to t) - \bar {D}\right) (t - \bar {t})}{\sum_ {t \in \mathcal {T}} (t - \bar {t}) ^ {2}} \right|,
$$
where  $\bar{t} = \mathrm{avg}_{t\in \mathcal{T}}t'$  and  $\bar{D} = \mathrm{avg}_{t\in \mathcal{T}}D(t' \to t)$ . This metric is the slope of a line fitting the performance change of models trained on a variety of timestamps, when evaluated on a fixed times-amp. It can be interpreted as the average rate of performance deterioration per time period.