Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the problem of temporal generalization in pre-trained language models, that is, the ability of the models to generalize well to future data beyond their training period. The paper argues that temporal generalization is necessary for pre-trained language models as they increasingly form part of dynamic and non-stationary environments that may require updated knowledge to perform better. To benchmark temporal generalization of the current models, the paper introduces three language modeling datasets - arXiv, WMT, CustomNews with timestamp information. On experimenting with the existing model, Transformer-XL, on the time-stratified setup, the paper finds that the performance deteriorates as the evaluation data moves further away from the training period. An in-depth analysis of the model predictions leads to the intuitive conclusion that the model performance suffers from the novel and emerging words/ concepts present in the future data. Next, the paper considers time-stratified closed-book QA tasks to evaluate the temporal performance of language models on downstream tasks. This paper convincingly shows that the performance on QA tasks deteriorates as the pre-trained language model becomes increasingly outdated with time, thus motivating the temporal generalization problem. Naively increasing the capacity of the model (from Transformer-XL) does not solve the problem. Therefore, the paper builds on the existing dynamic evaluation methods and proposes continuous updating of the language model with the incoming stream of the data. Such a method adapts the language model to temporal trends from incoming data, thus reducing the speed of the model from becoming outdated. Although dynamic evaluation results in improved performance, the paper shows that temporal generalization in pre-trained language models is far from solved and still poses a challenging problem for the community.
main_review: The paper investigates temporal generalization in language model pretraining. Although the evaluation task and benchmarks are novel, some recent works are studying this kind of generalization on tasks like sentiment analysis, machine translation, and named entity recognition.  This paper seems an obvious extension to these works with evaluations on a generic language modeling task. Moreover, the considered online learning approach is a straightforward application of the existing dynamic evaluation method. Despite these concerns, when it comes to the novel experimental setup on language model pretraining, the paper is thorough and coherent with well-defined objectives and answers. This paper examines an important problem of temporal generalization that would be prevalent in the coming days. Apart from practical considerations of developing methods to avoid model retraining, considering temporal splits also helps prevent data contamination. The paper promises to release the introduced benchmarks which would benefit the next generation of models and learning algorithms for the temporal generalization problem.


Clarification question: For closed-book QA, in Fig 4 the performance of the model when pre-trained on a dataset up to 2019 drops when evaluated on a test dataset from 2019. The paper attributes the performance drop to confusion around the names of presidents. However, the pre-trained model is fine-tuned on the QA dataset from 2011. Does the model undergo forgetting when fine-tuned on factual knowledge from 2011?


Typos: Lines 234, 245, 361 should be Table 3 instead of Fig 3. Line 368 should be Table 4 instead of Fig 4.
limitations_and_societal_impact: Yes, the paper discusses the limitations of their work.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Motivated by the fact that in practice models are applied to text generated later than their training data, this paper shows that splitting the datasets in the common time-agnostic way results in overestimating their performance compared to using time-stratified datasets. This work also conducts in-depth analyses of the performance degradation as the time span from a training set to a (later) test set increases and draws several conclusions: 1) proper nouns, numbers, words related to politics/sports, emerging new words are affected the most; 2) increasing model size improves overall performance but does not solve this performance degradation issue; 3) this phenomenon affects some end-user applications. To mitigate this issue, this work proposes to use dynamic evaluation which updates the model in an online fashion. Based on these findings, this work recommends the wide adoption of time-stratified dataset splits to encourage more realistic evaluation and the development of more adaptive language models.
main_review: Strengths:
1. This work identifies a practical yet often overlooked issue of existing text generation evaluation protocols. The wide adoption of time-stratified data would lead to evaluations more consistent with the real-world performance of models deployed.
2. The in-depth analyses of this work find where existing models need improvements when evaluated on time-stratified datasets. This points to future research directions such as incorporating longer contexts to improve the second appearance of emerging new words, or to consider alternative continuous learning approaches rather than simply increasing model sizes to do well under the proposed evaluation protocol.
3. The release of streaming arxiv and news datasets would bolster research on the online adaptation of neural models. 

Weaknesses:
1. While it's good to quantify the performance of models on time-stratified datasets, many conclusions are obvious, such as the synthetic experiments on question answering by shifting the end year of training data, as well as the experiments on self-containing reading comprehension.

Typos:
1. L149: Fig. 2 -> you meant Table 2?
2. L234: Fig. 3 -> Table 3
3. L245: Fig -> Table

While the conclusions in this paper might appear obvious, the analyses of this paper are thorough and insightful and the released datasets appear to be useful. More importantly, the recommendation of this paper is relevant to the broad community beyond researchers in text generation. Therefore, I recommend the acceptance of this paper.

==== Post Rebuttal ====
I've read all reviews and the authors' responses. Seems that there's no significant issue that I overlooked in my initial review, so I'm keeping my score and still recommending the acceptance of this paper.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studies temporal generalization of language models -- how well they do on text data "from the future" (ie after they are trained). It performs experiments in the domains of arxiv abstracts and news domains, where the date of each document is given. 

The paper trains language models on data up to and including December 2018, and evaluates perplexity on the months afterwards -- perplexity goes up as models have to extrapolate more, much more than if documents were split iid. The paper finds similar reslults when the task is changed to QA, or when a larger model is pretrained. 

The paper introduces analysis for what new linguistic patterns cause perplexity to increase (e.g. new words or words that change in context.)  

The paper tries one simple way of addressing this -- dynamic evaluation -- but also argues that it's not enough.
main_review: To this reviewer, this paper seems strong. Generalizing to the future is a challenging task and this paper helps shed some insights on where today's models struggle. The dynamic evaluation baseline should also help ground future work in this direction.

Releasing these benchmarks should help future work in this area, particularly if the underlying metrics (like performance on "emerging new words") can be accessed by submitters.

To this reviewer, there aren't any significant issues with the paper that make me want to reject it. Though the results might not be surprising (discussed well in L439) this paper defines the task in a good way for other work to (possibly?) take a stab at it.
limitations_and_societal_impact: This paper discussed "limitations" but it did not discuss possible negative societal impact of the work -- the paper could be improved by adding such a section. Even though this benchmark is not likely to cause harm, it might be good to think about what might result from future work in this area (on both modeling and evaluation side, with respect to the field and society). You could also discuss the relevance of evaluation datasets like this one in terms of reducing harms of NLP (e.g. bias being amplified? or historical prejudices being baked in?).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper evaluates language models on time-stratified datasets, i.e., they split datasets so that the test set is newer in time than the training set. It shows LMs are worse in terms of perplexity and closed-book question answering ability on the test data, some times by surprisingly large amounts. They analyze where the largest source of errors occurs (proper nouns, named entities, etc.) and also show that online finetuning at test time can mitigate but not solve these temporal generalization issues.
main_review: Strengths:
* Well-written, good motivation, and an understudied problem in the NLP community.
* Simple and convincing experiments. While some may critique this paper as being "obvious", its a study that has seen little exploration and is definitely worth doing.
* I like that they explored ways to mitigate the problems with dynamic evaluation, even including lightweight updates, which is the most sensible baseline strategy for adapting models (although see weaknesses section)

Weaknesses:
* I'm actually surprised by how relatively ineffective dynamic evaluation is, especially for the news data. It'd be great to know why. One theory, for example, is that catastrophic forgetting is occurring and the LM is being adapted to the news domain but becoming worse in overall performance. A simple test for this would be to run the model on held-out data that is not temporally shifted and see how much perplexity changes before/after dynamic evaluation.


limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper presents a study of temporal generalization of language models to time periods beyond which they are trained on, demonstrating that current “static” LMs do not generalize well to text from the future and degrade over time. The paper argues for studying dynamic language modeling and presents new benchmarks for the same. All the reviewers and the AC found this to be novel, solid and timely work, especially given the recent proliferation of large-scale language models in a wide variety of real-world applications.
Decision: Accept (Spotlight)