Mind the Gap: Assessing Temporal Generalization in Neural Language Models

1 Introduction . In recent years , substantial efforts in neural language modelling have focused on finding better neural architectures , building increasingly larger models , and compiling ever-larger amounts of training data , which have been shown to endow language models with the ability to perform well on a wide variety of downstream tasks with minimal fine-tuning ( Vaswani et al. , 2017 ; Radford et al. , 2019 ; Brown et al. , 2020 ) . While this approach has led to impressive progress , it nevertheless relies on a static experimental paradigm . Concretely , the prevailing practice is to curate a large pretraining web crawl—randomly partitioned into a training set and a validation set in a time-agnostic fashion—and then evaluate on tasks and benchmarks that mostly overlap in time with the pretraining data.2 ∗Equal contribution . ♠ Project initiation . 4 Paper writing . ♦ Project technical infrastructure . ♥ Model design and experiments . ♣ Project support and advice . 1We release our dynamic ( streaming ) language modelling benchmark for WMT and ARXIV at https : //github.com/deepmind/deepmind-research/tree/master/pitfalls_static_language_models . 2In the case of GPT-3 ( Brown et al. , 2020 ) , such tasks include LAMBADA ( Paperno et al. , 2016 ) , TriviaQA ( Joshi et al. , 2017b ) , and WMT translation datasets , among others . These tasks were introduced between 2014 and 2017 , which overlap in time with the GPT-3 CommonCrawl dataset that covered the period of 2016-2019 . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) In this work , we argue that such practices carry two potential risks . First , they do not assess a language model ’ s ability to generalize well to future data from beyond their training period—an important ability we henceforth refer to as temporal generalization . In our dynamic and non-stationary world , temporal generalization is a key necessity : Many practical machine learning systems that use language model ( LM ) pretraining , such as machine translation and dialogue systems , are deployed on utterances that users will say in the future , whilst being trained on utterances that users have already said in the past . Furthermore , temporal generalization is also crucial to perform well on realistic use cases of language models in the real world . Examples include flagging fake news about recent events that happen outside of the training period ( Thorne and Vlachos , 2018 ; Zellers et al. , 2019 ; Augenstein et al. , 2019 ) , forecasting stock prices from the latest news articles ( Ding et al. , 2015 ) , and answering knowledge-intensive questions like “ How many people have been infected by COVID-19 ? ” and “ Has the USA ever had a female Vice President ? ” , whose answers have evolved with time . Second , the temporal overlap between the training and evaluation data increases the risk of “ test data contamination ” , where parts of the evaluation task are unwittingly included in the pretraining data . Indeed , many language modelling evaluations treat the data as independent and identically distributed ( i.i.d ) at either the sentence ( Chelba et al. , 2013 ) or document level ( Brown et al. , 2020 ; Gao et al. , 2021 ) . Nevertheless , language modelling data are not i.i.d . ( neither at the word , sentence , or document level ) ; rather it is a time series , and thus models trained on the prefix of a sample from the series should be evaluated on the continuation of that series . While previous research ( Levenberg et al. , 2010 ) has highlighted the importance of temporal splits for fairer and more realistic evaluations—and has led to research ( Osborne et al. , 2014 ; Yogatama et al. , 2014 ) that addresses language modelling from this streaming perspective ( §7 ) —using temporal splits ( or splits beyond random ones ) is still the exception rather than the rule , as evidenced by many contemporary LM ( Brown et al. , 2020 ; Gao et al. , 2021 ) and downstream tasks ( Lewis et al. , 2020a ) that are affected by test data contamination.3 Here we begin with our first question : To what extent does the current static language modelling practice overestimate performance , compared to the more realistic setup that evaluates LMs on future utterances ? To this end , we introduce our dynamic , streaming language modelling benchmarks ( §2 ) , and find that Transformer-XLs ( Dai et al. , 2019 ) perform up to 16 % worse when predicting articles that are published up to 2 years after the end of the training period . Moreover , model performance becomes increasingly worse with time ( §3 ) . Given this finding , we ask : What kinds of predictions is the model struggling with in the dynamic evaluation setup ? —-which we answer in §3.1 . Beyond LM perplexity evaluation , we further ask : How exactly does this temporal performance degradation of Transformer LMs manifest in different types of question-answering ( QA ) tasks ? We answer this through two different QA tasks , including one around recent events happening outside of the LM training period ( §5 ) . Lastly , given the challenges presented by temporal generalization for LMs : What , then , is the remedy ? This question is important because keeping LMs up-to-date by retraining with new data is expensive in compute and carbon costs ( Strubell et al. , 2019 ; Patterson et al. , 2021 ) , and risks the model getting outdated in-between long retraining cycles.4 We find that increasing model size alone—a key driver behind recent LM progress ( Kaplan et al. , 2020 ) —is not a solution for the temporal generalization problem ( §4 ) : Larger models suffer from the same performance degradation with time , and a smaller model trained on more recent data can outperform a 60 % larger model that lacks access to more recent data . We then explore a simple yet effective way of keeping our models up-to-date by continually updating the model ’ s parameters through dynamic evaluation ( Mikolov et al. , 2010 ; Krause et al. , 2019 ) , which performs a few steps of gradient descent on streams of new data ( §6 ) , and outline other promising approaches in this direction ( §7 ) . We conclude with the following recommendations for future LM research : • We should evaluate LMs on their generalization ability to future data , which circumvents test data contamination , rewards models that generalize beyond the surface patterns of their pretraining data , and better reflects how large LMs are used in practical systems . We thus argue for the broader inclusion of timestamp information in pretraining data and downstream tasks to make this possible . • Stale LMs that are deployed far outside of their training period perform substantially worse on downstream tasks that require up-to-date factual knowledge , although a broader set of experiments are needed to pinpoint what kinds of tasks are most affected . Our findings also highlight the need 3Brown et al . ( 2020 ) used n-gram filtering and deduplication to remove overlaps between the training and test sets . This can potentially induce a correlation between the training and evaluation sets that LMs can exploit . 4These risks are exacerbated by the trend of ever-larger LMs , where retraining incurs even higher costs . for more tasks , benchmarks , and metrics that evaluate how well and how rapidly LMs are able to integrate new information , which are important ingredients to encourage progress in this direction . • All in all , above and beyond impressive scaling efforts towards ever-larger models ( Brown et al. , 2020 ; Fedus et al. , 2021 ) , we argue for the development of adaptive language models that can remain up-to-date with respect to our open-ended and non-stationary world . 2 Time-stratified language modelling . We begin by introducing our time-stratification experimental setup , which examines how well Transformer LMs perform when evaluted on future utterances from beyond their training period . 2.1 Datasets . We identify news and scientific articles as two sources of dynamic streaming data with a naturally changing distribution over time—lending themselves well to evaluating how well language models generalize over time . For the scientific domain , we use the publicly available arXiv abstracts ( ARXIV ) .5 For news , we use the publicly available WMT News Crawl ( WMT ) .5 We ensure that any trends we observe also generalize well to models trained on larger datasets—which reliably improve language modelling and downstream task performance ( Liu et al. , 2019 ) —by compiling a larger news corpus that we term CUSTOMNEWS . This dataset consists of crawled English news sources from 1969-2019 , and covers various topics including politics , finance , and sport . We apply minimal preprocessing through : ( i ) Removal of non-English documents , ( ii ) deduplication using the MinHash algorithm , and ( iii ) tokenization using Moses.5 Table 1 summarizes key statistics of our datasets . 2.2 Experiment : A model up to 2 years stale . Evaluation period and test set . For each dataset , we pick the last two years ( i.e . 2018 and 2019 ) as our evaluation period , and sub-sample a test set of 24k test documents ( 1k per test month ) . TIME-STRATIFIED setup . In this setup , we evaluate LMs trained on the past based on their ability to predict future articles that are published after the time period of their training data ; this split is constructed using the time stamp of each article . Here we use all documents from the beginning of each dataset ’ s time period up until September 2017 as training data , and use the last three months of 2017 as our validation period ; we denote this as the TIME-STRATIFIED setup . We then evaluate the model on the 2018-2019 test set above , which evaluates the model ’ s ability to generalize across time by predicting articles up to two years after the end of their training period—a realistic time frame during which we expect large-scale language models to be used without retraining on recent data . CONTROL setup . We assess whether time stratification poses a challenge for current LMs by comparing it with the following CONTROL setup . In this setup , the training set includes documents that come from the same 2018-2019 period as the evaluation set ( naturally excluding the test documents themselves ) . This CONTROL setup thus resembles the prevailing ( static ) language modelling experimental practices , which train and evaluate LMs on text data from overlapping time periods . Crucially , we control such that the two training sets are of the exact same size , i.e. , they differ only in the time periods of their training data , rather than in their absolute training set sizes . Here we construct the CONTROL training data by taking the most recent documents starting from the end of the evaluation period ( excluding the test documents and including the same number of training documents per test month ) , and keep adding documents from previous time periods until we reach the same training size as the TIME-STRATIFIED setup . In Table 1 , we report the proportion of documents in the 5ArXiv : https : //arxiv.org/help/oa/index ; WMT News : http : //data.statmt.org/ news-crawl ; and SacreMoses : https : //github.com/alvations/sacremoses . CONTROL setup ’ s training data that come from the same 2018-2019 time period as the evaluation set , which is higher for ARXIV and CUSTOMNEWS due to their recent exponential growth of new documents . We sample a similarly-sized validation set as the TIME-STRATIFIED setup , which in this case comes from the 2018-2019 evaluation period ( again excluding the test documents ) . Importantly , both the TIME-STRATIFIED and CONTROL models are evaluated on the exact same test set from the 2018-2019 period , which facilitates a fair perplexity comparison between the two setups . Relative perplexity comparison . We want to measure temporal degradation , i.e . do Transformer LMs perform increasingly worse when predicting test documents further into the future ? However , any absolute perplexity degradation of the TIME-STRATIFIED model over time ( e.g. , perplexity for Jan. 2018 vs Dec. 2018 ) is an unreliable measure : Some months have longer documents , which lead to higher perplexity . We thus measure temporal degradation through relative perplexity changes between the TIME-STRATIFIED and CONTROL models for the same test month ( e.g . Dec. 2018 ) .

Summary: The paper studies the problem of temporal generalization in pre-trained language models, that is, the ability of the models to generalize well to future data beyond their training period. The paper argues that temporal generalization is necessary for pre-trained language models as they increasingly form part of dynamic and non-stationary environments that may require updated knowledge to perform better. To benchmark temporal generalization of the current models, the paper introduces three language modeling datasets - arXiv, WMT, CustomNews with timestamp information. On experimenting with the existing model, Transformer-XL, on the time-stratified setup, the paper finds that the performance deteriorates as the evaluation data moves further away from the training period. An in-depth analysis of the model predictions leads to the intuitive conclusion that the model performance suffers from the novel and emerging words/ concepts present in the future data. Next, the paper considers time-stratified closed-book QA tasks to evaluate the temporal performance of language models on downstream tasks. This paper convincingly shows that the performance on QA tasks deteriorates as the pre-trained language model becomes increasingly outdated with time, thus motivating the temporal generalization problem. Naively increasing the capacity of the model (from Transformer-XL) does not solve the problem. Therefore, the paper builds on the existing dynamic evaluation methods and proposes continuous updating of the language model with the incoming stream of the data. Such a method adapts the language model to temporal trends from incoming data, thus reducing the speed of the model from becoming outdated. Although dynamic evaluation results in improved performance, the paper shows that temporal generalization in pre-trained language models is far from solved and still poses a challenging problem for the community.