Future Language Modeling from Temporal Document History
Authors: Changmao Li
Jeffrey Flanigan
Venue: ICLR 2024 poster (2024)
Keywords: Future Language Modeling
Future Language Model
Temporal Document History
Primary area: generative models
TL;DR: We propose the task of future language modeling and develop several models for this task, outperforming strong non-temporal baselines.

Abstract: Predicting the future is of great interest across many aspects of human activity.  Businesses are interested in future trends, traders are interested in future stock prices, and companies are highly interested in future technological breakthroughs.  While there are many automated systems for predicting future numerical data, such as weather, stock prices, and demand for products, there is relatively little work in automatically predicting textual data.  Humans are interested in textual data predictions because it is a natural format for our consumption, and experts routinely make predictions in a textual format (Christensen et al., 2004; Tetlock & Gardner, 2015; Frick, 2015). However, there has been relatively little formalization of this general problem in the machine learning or natural language processing communities.  To address this gap, we introduce the task of future language modeling: probabilistic modeling of texts in the future based on a temporal history of texts.  To our knowledge, our work is the first work to formalize the task of predicting the future in this way.  We show that it is indeed possible to build future language models that improve upon strong non-temporal language model baselines, opening the door to working on this important, and widely applicable problem.

Published as a conference paper at ICLR 2024
FUTURE LANGUAGE MODELING FROM TEMPORAL DOCUMENT HISTORY
Changmao Li, Jeffrey Flanigan University of California, Santa Cruz {changmao. 12, anfflanig@eussc..o edu
ABSTRACT
Predicting the future is of great interest many aspects of human activity. Businesses are interested future trends, traders are interested in future stock prices, and companies are highly interested in future technological breakthroughs. While there are many automated systems for predicting future numerical data, such weather, stock prices, and demand for products, there is relatively little work in automatically predicting textual data. Humans are interested in textual data predictions because it is natural format for our consumption, and experts routinely make predictions in a textual format (Christensen et al., 2004; Tetlock & Gardner, 2015; Frick, 2015). However, there has been relatively little formalization of this general problem in the machine learning natural language processing communities. To address this gap, we introduce the task of future language modeling: probabilistic modeling of texts in the future based on temporal history texts. To our knowledge, our work the first work to formalize the task predicting the future in this way. We show that it is indeed possible to build future language models that improve upon strong non-temporal language model baselines, opening the door to working on this important, and widely applicable problem."
INTRODUCTION
Predicting the future is a standard practice across numerous domains of human life and businesses (Christensen et al., 2004; Tetlock & Gardner, 2015; Frick, 2015). Public and private organizations constantly anticipate future trends, shifts in stock values, or forthcoming technological advancements.
The pressure predict the future has fueled developments in the automated prediction of future numeric data, encompassing areas such as weather forecasting, stock market trends, and demand for goods.
However, is striking to note the scarcity of work developed towards the automation of predicting textual data. Textual data holds unique significance, given that it is natural and rich format for human consumption. Moreover, experts frequently offer predictions in a textual format, evident in an array books, magazines, and academic publications. Despite this, predicting future data is rarely studied within the machine learning or natural language processing communities.
Our work aims to address this gap by introducing a novel task future language modeling. The future language modeling task is to construct generative language model for future text given temporal history of documents. To the best of our knowledge, this the first attempt to systematize and advance the task of predicting the future in this specific manner. Beyond formalizing this important task, we also create and develop future language models designed for this task. We evaluate these future language models against strong non-temporal baseline language models using both automatic metrics and human evaluations, and demonstrate their effectiveness at generating future textual content.
word of caution: predicting the future is bold claim. We do not wish to argue that all future text can be predicted. There are random events, named entities, serendipitous discoveries, etc, in text that cannot be predicted. But we hypothesize that there are some important aspects of the future that be predicted given enough historical text. Only by working this future language modeling task
Our code is available httptttp/////. //gi
http//gttttttttttt/// thub. com/ jjab//nabnnn nlp/ /tpuuuuue--eree Language Model inn

---

Published as a conference paper at ICLR 2024
2019
madial pratralning has led significatt performance gaina... (Liu al, 2019)
propose LinkBEAT, an LM pretraining method leverages documents... (Yasunaga at al,, 2022)
pressin Fasture
language model
2022
...we ELEcTRA-elyle language model pre training... (Chi ot al., 2022)
0.006
0.006
...we introduce BanglaBERT, a BERT-basee Natural Language Understanding (MLU) model pretrained Bangla, widely spoken yet low-resource language... (@hattacharjoe at al., 2022)
0.022
0.000
2010 2023 2012 2013 2014 2013 2015 2017 2015 2019 2020 2022
year
(b)
Figure I: (a) represents an example showing how abstracts in recent history are related to the future. In this example, the text of the abstract of the RoBERTa paper (Liu et al., 2019) anticipates the rise of papers about "language model pretraining" (Du et al., 2022; Bhattacharjee et al., 2022; Chi et al., 2022). (b) shows the word frequencies by year in NLP abstracts for some representative words, which reflects topic/approach changes over the years, i.e., "pretrain" started to dramatically go up after 2018 because of BERT, and "neural" became popular after 2013 because of deep learning.
can this hypothesis be verified. We show, by construction, that future language models can be built that perform better, across various automatic and manual evaluations, than non-temporal language models trained the most up-to-date text, thereby verifying this hypothesis. While humans sometimes predict the future, experts are often wrong (Frick, 2015), and we do not know the machine upper-bound this task. We hope to push the boundaries of predicting future trends by proposing the task and developing methods for future language modeling.
Our contributions are the following:
We introduce the future language modeling task ($2) of modeling future textual content using history of documents, as well as evaluation methods for this task ($4.4 & $4.6).
We develop series of future language models ($3) for this task that incorporate temporal information into pre-trained language models, which dynamically adjusts the generation probability generate content that follows the predicted future trend.
As a concrete example, we evaluate our model to model future abstracts for ACL conference papers, and our proposed approach outperforms the bascline model on both automatic metrics and human evaluation.
The paper is organized as follows. In $5, we present related work. In $2, we provide task overview introduce the proposed future text generation task based on previous time spans, In $3, present the details of our proposed approaches. $4 presents our experiments and results analysis.
2
TASK OVERVIEW
We begin by defining some terms. Without loss of generality, we call the times when we update our language model years, but they could be other time spans such as days hours. Each year has a collection of texts for that year. For simplicity. we call these texts documents.?
Our proposed future language modeling task is model future texts using documents from previous years. Let i denote the year index, and document dij (I.jI, be jth document from the
II experiments in $4, the ("documents") abstracts.
2

---

Published as a conference paper at ICLR 2024
ith year, where Tiyk is the kth token from the jth document in the ith year. Let D, {d.ı .... dij} represent all documents from year i. The task is to generate D, based on D1 to D;-1. which means during generation, the probability of each generated token Tijk is computed not only from standard language modeling perspective but also considering the content evolution from Dj to D,. The conditional probability for each token Vijk is conditioned not only the previously generated words in the sentence (as usual), but also on all the previous years" documents:
Plaigh
1), DT Di-1)
(I)
We call the model for the above task future language model, formally defined as statistical language model designed to assign high probability to future texts based on the temporal history of texts.
APPROACH
3.1
OVERVIEW OF MODELS
We develop three methods for future language models: word frequency model ($3.2), contextual temporal model ($3.3) and doubly contextualized temporal model ($3.4). In this section, we give some background notation common to all these models.
All our methods modify the language model probabilities to account for the temporal evolution. language model usually calculates the probabilities with softmax equation:
ET H
P( En E
In this equation, Few E TRd is the learned output embedding vector for the wth word in the vocabulary, and H2 E Rdd is the contextualized embedding at position k. We use a transformer language model, and Hu is the vector of the last layer of the transformer decoder position k. This is our baseline compare with.
Our first methods ($3.2 & $3.3) compute temporal bias Büw E R for the wth word in the ith year that is calculated from the previous years. The bias term up-weights or down-weights vocabulary items account for changes across years. The bias added into the softmax equation modify the probabilities:
ET. HK + Biss P(æg|ar.... Dj Di-1) En. (ET, Bino))
We describe how Biw is calculated in the following sections.
Our third method ($3.4) more expressive, and calculates a contextualized bias term that depends on the previous words -1- that have been generated. This allows the bias term to be contextualized in the output that is being generated. In our notation, the bias term Biew E the bias for the wth word in the kth position in the generated sentence for the ith year. The softmax probability equation becomes:
ET Hy + Bikey
P(r (I... D1 ... Di-1)= En (ET Hu + B
For training, all our future language models trained with standard cross-entropy loss:
logp(zz(an ... 11.30
(5)
where 9 represents the model parameters.
3.2 THE WORD FREQUENCY MODEL.
Our simplest method models the change over time of the frequency of the words without using any context from historical documents. It only uses the raw counts of the word over time to compute
3

---

Published as a conference paper at ICLR 2024
LSTM
LSTM
LSTM
LSTM
0900
LCTM
eigg
@000
plg?
MAA
$990
e000
Transformer
Transformer
Trenaformer
(a)
Word
Frequency
Model
(b) Temporal Contextual Model ($3.3)
(c) Doubly Contextualized Model ($3.4)
($3.2).
Figure 2: Our proposed models.
bias. This bias is added to the final softmax bias the model towards historical trends. Figure 1b shows the frequency by year for some example words, which reflects topic/approach changes over the years, i.e., the word "pretrain" started to dramatically go up after 2018 because of BERT, and the word "neural" became popular after 2013 because of the deep learning.
Our intuition is to use temporal neural network to try predict biases for words based on historical frequency data of words. This model uses an auto-regressive deep learning model to predict the change over time. We use an auto-regressive RNN-style model, specifically Transformer model for this because it is more naturally suited to our temporal task, as LSTMs do not use position embeddings. For balance of simplicity, scalability, and expressivity, we use an LSTM.
LSTM, rather than
We predict the temporal word bias for each year using an LSTM and use it as feature to bias the generation probability. Figure shows the model overview. Let five wth word for the ith year, and let m be the window size which determines how many previous years to consider to predict next year's bias. For each year, we compute a temporal bias Bi from m previous year's word embedding by using LSTM where weights are shared across word types. We use the last hidden vector of the LSTM followed by a dot product with a learned vector A E Rd to compute the bias:*
be the frequencies of the
B A L.STM(logU/mm..) lo,, log(si-1,w))
(6)
This temporal bias is added to the output of the Transformer as a bias in the softmax, as described in $3.1 Equation
3.3 THE TEMPORAL CONTEXTUAL MODEL
While the previous method models the change in the frequency of words over time, it does not have contextualized information to help it make its predictions. So while it may see words such as "pretraining" increase time. it is ignoring contextual information in prior abstracts like "pretraining has led to significant performance gains" that could help it make predictions (see Fig. la).
To account for contextualized information contained in prior abstracts, develop temporally contextualized model. For each word, we create a pooled representation for each year. We use an average of the contextualized embeddings, averaged over all instances of that word over the year. For each word, we then feed the contextualized embedding into an LSTM to predict the temporal word bias. Figure 2b shows the model overview.
In more detail, using our notation from $2, let dey (Iay1 T.g2:..., where Zijk is kith token in dij. For the token Zayks let Eija be the corresponding contextualized vectors from a pre-trained language model. Our representation for the with word in the vocabulary for the ith year the average of the contextualized embeddings, which can be expressed as:"
-... Tigk) be the jth text in ith year
Σ
Ideyl #=
Esjte I [x wi
Vin
Σ
[ = w]
'e are aware of work demonstrating autoregressive Transformers can be trained without position embed- dings, but leave this style of model for predicting biases to future work.
"To make this more efficient, we batch the L.ST across words in our implementation. 'ee use the indicator function 7[-] which is if the condition is true and if it is false.
4

---

Published as a conference paper at ICLR 2024
To use the temporal contextualized word embeddings, we use the fact that more recent years have more influence on future texts and propose a window- modeling approach. The window size determines how many previous years for word embedding consider to predict the next year's temporal bias. Let m be the window size for each year, then we compute temporal bias Bin E IRd from m previous year's s word embedding follows:
Bi LST (t-mm.)) my V(K-1,w))
(8)
where we take the last hidden vector of the LSTM and E R is learnable parameter. The temporal bias is added to the output of the Transformer bias in the softmax, as described in $3.1 Equation 3.
We also experimented with combining the word frequency model and the temporal contextual model, but we did not observe any improvement by additively combining them.
3.4 THE DOUBLY CONTEXTUALIZED MODEL.
The temporal contextual model does good job of predicting the rise and fall in the frequencies of terms. However, we observe that it does poor job of deciding when to use the terms while generating. The Contextual output in Table 2 shows an example of this. The model repeatedly introduces new terms that are fashionable, but in an incoherent manner (saying that the paper will focus on IE, but then saying that special attention will be on multi-document summarization).
We hypothesize that the contextual model can predict good terms to use, but cannot decide when to rely on the temporal contextual model versus relying on the prior state in the language model (for example, reusing a previous term in the document versus introducing new fashionable term). The model appears to need a "gating" mechanism to decide when to use the new suggested terms. To address this, introduce mechanism that contextualizes the temporal contextual model when generating a document a doubly contextualized model that contextualized both temporally and in the document generation. Figure 2c shows the model overview.
We start with matching Bin with the pre-trained model embedding space and reduce the dimension of vocabulary size. To implement this, we enable temporal bias Biw tie with the word embedding layer weights for each word Eu E I' and conduct a linear projection. We compute the tied and projected temporal bias Bin," E Id as follows:
Bin (ET Brn )
where Rd is learnable parameter.
Then we compute the sigmoid attention between transformer decoder output H2 and Brs to obtain the Bikw E IRd as follows:
Biew ao (H. CD (E:
(10)
where C, ddd are learnable parameters, and a is tuned hyperparameter.
This temporal bias is added to the output of the Transformer as a bias in the softmax, as described in $3.1 Equation Using this model, we obtain improved example output shown in Table 2.
EXPERIMENTS: FUTURE ABSTRACT PREDICTION
4.1 DATASET PROCESSING
As concrete example to be experimented with, we conduct experiments to model future abstracts for NLP papers based on previous papers' abstracts. We first collect paper abstracts for each year from ACL. anthology " and filter the noisy abstracts such as papers that are not in English. Then we use the years as the year (for other domains such as news, you can use the day or hour as the year) and split the paper abstracts by years and use abstracts from 2003-2019 as training data, the year 2020 as the development data, and the year 2021 I as the test data. Table I shows the statistics of the dataset. Figure 3 shows the number of abstracts by year for the dataset.
5

---

Published as a conference paper at ICLR 2024
Dataset Statistics
Train
Dev
Test
# of abstracts
37816
5919
5529
avg. of sentences per abstract
9.0
6.4
6,5
avg. of tokens per abstract
225.8
168.5
164.3
Table 1: Data split statistics
400
Figure 3: # of abstracts by year
4.2 MODELS
We use GPT-2 (Radford et al., 2019) as the pre-trained language model in our experiments, although our approach is not restricted any particular pre-trained language model. We train and evaluate the following models:?
Baseline baseline which fine-tunes GPT-2 on abstracts from all previous years
Baselines ine-nn A baseline which fine-tunes GPT-2 on abstracts from most recent previous years, since recent years may be more relevant for predicting future years. We evaluated from 1 to 10, and report the best 2 models (n = 2 and n = 3).
Frequency-NoLSTM A word frequency model without using an LSTM, instead directly using the previous year's frequency as bias feature in the model.
Frequency Word Frequency Model (53.2)
Context Temporal Contextual Model ($3.3)
Context? Doubly Contextual Model ($3.4)
4.3 HYPERPARAMETER SETTINGS
We use the Adam optimizer (Kingma & Ba, 2015). The batch size is set with gradient accumula- tion size 2. Between layers, we apply dropout with probability of 0.1. We fine-tune 10 epochs for each model and do early stopping. The a set to le or initialized with I when automatically learned. Bounds for all hyperparameters are the same as GPT-2. We have several hyperparameter search trials on a which are 1, le- 1,1e,,,, 1e-3, le-4, le-5. For each model, we have three training and evaluation runs. The method of choosing hyperparameters is based perplexity scores on the dev set. Fine-tuned RoBERTa Model (Liu et al., 2019) for each year is used to generate temporal word embedding representation. We use beam search decoding with top-k sampling. The beam size 5, k is 50, and is 0.92. Since topic agnostic, the start token is end of sentence token for GPT-2. All models were trained evaluated on either one A40 or A6000 GPU, Our implementation is based on Huggingface Transformers (Wolf et al., 2020).
4.4 AUTOMATIC EVALUATION METRICS
We use three automatic evaluation metries, which are perplexity (PPL), content perplexity (CPL), and content meteor (CM). Since most of the evolution of ideas in NLP papers is through changes in
Appenddi $A gives the model statistics.

---

Published as a conference paper at ICLR 2024
Approuch
Examples
paper presents machine learning approach to
disambiguzion (WSD)
based on lesical, syntactic and semantic feutures. Our approuch rolics on rule-bosed classilier trained
Baseline
manually annotated corporn. We show that our approach curperfoms state-of-fhe-ot WSD systems
English. German Spunishh.
This puper presents new approach to the automatic generation referring expressions (RES) natural langunge The approsch is hased on the use of distributional semantics, which can he used to model the meaning of utterance.
crequency
We descrihe how have constructed dara for RE generation that has been manually annotated with respect addition, show these dimensions
dimensions: referential quality, Bluncyy informisiveness correlate with human judgments about the quality of generated REs
This puper
overview the work carned out within the EUTPT European project Information Extraction (IE). involved
i... classification, naned carity recognition coreference resolution. Special attention will paid to the issues related malti- documett summarization as well cross- cross-lingual information extraction which is very important for many natural language processing tasks such as machine translation and question answering.
the main
Contextual
Finally directions future research sketched which include the development of advanced IP sysems based
deep leamning architectures the creation new evalustion corporn.
We present novel approsch to the problem of antommically detecting and clessiting semantic relations between named emities (NEs) natural language text. In particular, propose
method for identifying NEs
Contextual
that explicitly memicned the
do not appear triining The proposed method
based on an unsupevisse clustering algorihmm that uses word embeddings as features. Experimental results show that our method outertorms state of-the art methods by large marpin.
Table 2: Example output for each approach.
Dev CM1
Test
PPL4
CPL
PPL4
CM1
CPL.
Baseline-all
19.97
21.85
82.59
22.76
16.22
102.03
Baseline-2
21.66
24.41
91.36
21.53
19.69
94.37
Baseline-3
21.08
22.75
88.75
21.06
19.01
92.06
Srrpuency-NoLSTR
19.97
22.87
83.18
21.23
19.09
88.38
Frequency
19.97
23.87
82.43
20.20
19.98
87.68
Context
23.47
24.86
96.64
23.21
18.20
102.11
Context?
19.66
24.94
77.54
19.81
20.12
82.43
Table 3: Experiments results for automatic evaluation on abstracts. 1 indicates lower is better and t indicates higher is better. p-value < 0.001 for all scores over baseline based on statistical sign test (Dixon & Mood, 1946). Baseline-a means only n previous years' abstracts are used to fine-tune non-temporal LM. We evaluated from 1 to 10, and reported the best 2 (n 2 and n 3). Basel l weieineie aadeeaalll means using the whole training set to fine-tune non-temporal LM. PPL: perplexity score; CM: content meteor score; CPL: content perplexity score (See $4.4 for the detail of these metrics.)
Topic
Topic New
Problem
Problem New
Method
Method New
Avg
Baseline-all
100%
8%
50%
17%
42%
25%
46%
Baseline-2
97%
0%
60%
0%
53%
3%
36%
Bascline-3
100%
0%
95%
17%
35%
12%
44%
Frequeney
100%
17%
83%
0%
100%
25%
54%
Context?
100%
33%
100%
17%
100
50%
63 %
Table 4: Experiments results for the human evaluation. See $4.6 for details of the criteria.
content words, we manually collect the non-content words as a stopwords list. During content words based evaluation, filter out the stopwords, and the leftover tokens are naturally formulated into content words. The perplexity score evaluates fluency while the content words based metrics evaluate the adequacy of future research ideas since ideas are mainly represented by content words instead of non-content words.
Perplexity (PPL) We evaluate perplexity, which is calculated using the standard formula
PPL=2 ΣN log.(((x.))
where p(x:) is the token probability computed from the model and M Σ [as]-
Content Perplexity (CPL) Perplexity is computed over all words equally. including non-content words. To better evaluate the benefit of the improved content word selection, calculate perplexity on non-stop words. We call this content perplexity. This is computed by ignoring the stopword log probabilities, and only adding the non-stopword log probabilities together and dividing by the number
7

---

Published as a conference paper at ICLR 2024
of non-stopwords instead of the total number of words For test data D 21, NN the stopwords list is V,, then the content perplexity CPL is computed by
CPL = 2 tे Σ, logale(r
where p(z;) is the token probability computed from the model and M, -. | V&]]
Content Meteor (CM) This metric measures the match between model generated abstracts and real abstracts in the dev and test sets. We use 100 random seeds to generate N₂ 100 abstracts to compare with all abstracts in the dev or test set. After removing all the stopwords, we evaluate the Meteor score for the generated abstract only by the content words. Let G {a1, ... Gi: ... Mi:-..AN, ON₂) } be all generated abstracts with the stopwords removed, let = [d1,... ..., jj,, .... dN be all abstracts in the dev or test set (N₂, is the number of abstracts in dev test set), compute Content Meteor
No CM Σ max
No Meteor(a;,
Ng
4.5 AUTOMATIC EVALUATION RESULTS
Table shows the automatic evaluation experiment results across all experimented models. Our proposed methods perform better than the GPT-2 baselines without temporal information all automatic evaluation metrics.
The doubly contextualized model has about 3 content meteor points improvement over the year agnostic baseline-all, and 5 points content perplexity improvement, which indicates the content better matches real future abstracts. This demonstrates that the doubly contextualized enables the model to generate content words that will be used in the future. The word frequency model also shows points improvement in the content meteor and slightly better in the content perplexity. This indicates that by only adding the word frequency as bias, the model can increase the content matching slightly, We tried the accumulated baselines on the abstracts of the most recent years and the performance of only training on the most recent abstracts cannot surpass proposed model.
From a fluency perspective, the word frequency model has the same perplexity as baseline-all. In contrast, the doubly contextualized model shows a larger improvement which indicates that it can enable the model to generate more fluent abstracts than the baselines for future abstracts. Without using the LSTM to model the temporal information, the model only considers single previous year bias which hurts the performance. Without gating, although the model has a high content matching score, it has lower fluency score because the model cannot recognize which tokens should be biased. This demonstrates the importance of the gating mechanism in the doubly contextualized model.
4.6 HUMAN EVALUATION
For human evaluation, we randomly evaluate 100 generated abstracts for each approach. Since our temporal language generation task is to generate abstracts, we evaluate the abstracts with six different criteria, with criteria tailored the abstract generation task. Table shows the human evaluation results for all the experimental methods. We have six criteria for evaluation, which are divided into three abstract content types each with fluency and novelty aspects. Each criterion score binary or 1 for each abstract. We add all obtained scores together and divide them by the total gold scores to obtain the percentage of the human evaluation score. The human evaluators are NLP researchers. We conducted blind evaluation, the human evaluators did not know the approach for abstracts.
Topic: Is the topic clear and correct? We check an abstract has fluent topic or
background description without factual errors.
Topic New: Is the topic new? We check if an abstract has a topic we have never seen before matches the recent research topics.
Problem: Is the problem clear and correct? We check if an abstract has a fluent problem description without factual errors.
Problem New: Is the problem new? We check if an abstract introduces a problem that we have never seen matches the recent research problems.
Appenndi C. shows how the stopwords curated.
8

---

Published as a conference paper at ICLR 2024
Method: Is the method clear and correct? We check if the abstract has fluent method description without factual errors.
Method New: Is method new? We check if abstract proposes method we have never seen or matches the recent approaches.
Note that "new" here does not abstracts" such as abstracts in our dev or test set. Apparently, the model cannot generate completely new topics, new method, or new problem that they have never seen during the training.
completely new. Instead, it only means more related to "future
All of our proposed methods outperform the baseline when evaluated using the average score. The generated topics for all approaches are clear and correct, indicating that the GPT-2 baseline can adequately generate clear topics. However, topics are not necessarily new in the baseline approaches, whereas in our proposed approach 1/3 of the topics are new. Additionally, the problem and the method are not always clear and correct in the baseline, whereas our proposed approaches can have all generated new problems, and the methods are clear and correct. our proposed approach 1/2 of the approaches are new, which shows that our proposed approaches have the ability to predict new trends for future research.
4.7
CASE STUDY
Table 2 shows generated abstracts from all the approaches and Table 6 in Appendix shows more generated abstracts from all the approaches compared to the reference abstracts from ACL conferences. The baseline approach generates more general abstract content that does not contain many details or generate very traditional methods for NLP research. The example from Context shows that without gating, the generated output after 2-3 sentences is not related to the starting sentence because may ignore the previous context, although new content is generated, which shows that the gating mechanism can help the model determine whether the next generated token should be depended on the historical documents or the previous context. In contrast, the Context? method generates more detailed content and content that is more related to recent research, such as word embeddings or neural networks, and later generated sentences are more coherent to the previous context, which balanced between considering the historical documents to generate new content following the previous context to generate more coherent text.
5 RELATED WORK
To the best of our knowledge, there is no prior work constructing language models for future text based on temporal historical documents. However, there is much work on language models with temporal information (Röttger & Pierrehumbert, 2021; Lazaridou et al., 2021; Hofmann et al., 2021; Agarwal& & Nenkova, 2022; Loureiro et al., 2022). Huang & Paul (2019) worked on document classification using word-level temporal embeddings, and Röttger & Pierrchumbert (2021) adapts the pre-trained BERT models to domain and time. Lazaridou et al. (2021) evaluated the performance of language models on future text, in a setup similar to ours but did not construct any temporally models for future language modeling. Dhingra al. (2022) conducted experiments with temporal language models for question answering. Hofmann et al. (2021) modeled temporal and social information together by modifying BERT with latent Gaussian process. Rosin et al. (2022) concatenated time tokens to text sequences and introduced time masking using masked language modeling to make a time-aware BERT. However, none of the previous works are about building language models for future text based on temporal historical documents. In this paper, we fill this gap and propose future language models that can generate texts that are more related future content, which can be applied to many future forecasting areas.
6 CONCLUSION
In this paper, we introduce the task of future language modeling and propose a series of future language models. We evaluate our models abstracts in NLP. The proposed approaches outperform the baseline non-temporal language models across all automatic evaluation metrics and human evaluation on generating content related to the future text based temporal historical documents.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Nilay Patel, Geetanjali Rakshit, Rongwen Zhao, Brendan King, and Zekun Zhao, and the anonymous reviewers for helpful feedback on earlier drafts. This research was supported by computing resources provided by the Pacific Research Platform's Nautilus cluster, supported in part by National Science Foundation (NSF) awards
ACI-1540112, 12, ACI-1541349, OAC- 1826967, DAC-2112161 CNS-2100237, CNS-212000 the University of California Office of the President, and the University of California San Diego's California Institute for Telecommunications and Information Technology/Qualcommn Institute, and CENIC for the 100Gbps networks.
REFERENCES
Kaushik Acharya. WNUT 2020 shared task-1: Conditional random field(CRF) based named en- tity (eCog for wet lab protocols. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pp. 286-289, Online, November 2020. Association for Com- putational Linguistics. doi: 22118633/1/2222 w.3t-1 URL https: battps://aclie http:///aaaaattloooo.oo org/2020. wnut 37.
Oshin Agarwal and Ani Nenkova. Temporal Effects on Pre-trained Models for Language Processing Tasks. Transactions of the Association for Computational Linguistics, 10:904-921, 09 2022. ISSN 2307-387X. doi: 10. (62/ttclaa.0049)) URL https: /ttp::///oooooooo http://to.oo..//.////o 1162/tac2c a C0497.
Abhik Bhattacharjee, Tahmid Hasan, Wasi Ahmad, Kazi Samin Mubasshir, Md Saiful Islam, Anindya Iqbal, M. Sohel Rahman, and Rifat Shahriyar. BanglaBERT:: Language model pretraining and benchmarks for low-resource language understanding evaluation in Bangla. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1318 1327, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 122288 URL http::tt///oo wwwaaaaanlllloooooooooooooooo logy. org/ 2022. ngs- 98.
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma. Bo Zheng, Saksham Singhal, Payal Bajaj. Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. XLM-E: Cross-lingual language model pre-training via ELECTRA. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 6170-6182, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v www.o//o......oo......m. https: / //accaamm aeclannhhllgyy ogy org/ 2022 ael-2ccclllnng 427.
URL
Clayton M. Christensen, Erik A. Roth, and Scott D. Anthony. Seeing What's Next: Using Theories of Innovation to Predict Industry Change. Harvard Business Press, 2004.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. Time-Aware Language Models as Temporal Knowledge Bases. Transactions of the Association for Computational Linguistics, 10:257-273, 03 2022. ISSN 2307-387X. doi: 10.1 10..112/tacL.. _00999. URL https::tt///o /doi . org 11 2ttee///alllcom 004599
W. Dixon and A. M. Mood, The statistical sign test. Journal the American Statistical Associ- ation, (1(266):557-566 1946. ISSN 01621459. URL http:/// jstor. org/ stage//tae///////// 2280577.
the
Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 18.18/53/ v1/2022.acl-long.222 26. httpst http:///aaaaatthooooooo wow.lloyyooooooooooooooooooooeoe www.eorggr........eee a0l 1on..-ooo..25
Walter Frick.
about making accurate predictions. URL. https://hor.org/20115 /2/ anwpannggcccurttteeeeeetttie what reaatreesearch-tel seseells-s-bbboe htt.::/weeee..ooooooooooooooooan
What research tells us
Harvard Business Review, 2015.
Valentin Hofmann, Janet Pierrchumbert, and Hinrich Schütze. Dynamic contextualized word embed-
dings. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics
10

---

Published as a conference paper at ICLR 2024
and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1970-6984, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/221...m wac--ooccllllneee.. URL https: http:///aaaaaaaaaaaeeaeoo htt.tp/://allaathhooooooo/////// act-://oooo......aaaaaoaa...o 542.
Xiaolei Huang and Michael J. Paul. Neural temporality adaptation for document classification: Diachronic word embeddings and domain adaptation models. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4123, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/11/P19-1 1403. URL https: //acl antho wog..org..o..................... 90tgg//1911440 9
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: //arxiv. otg/ abs/ 412 6980.
Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson Autume, Tomas Kocisky, Sebastian Ruder, et al. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 34:29348-29366 2021.
Zhihong Lei, Weiyue Wang, Christian Dugast, and Hermann Ney. Neural language modeling for named entity recognition. In Proceedings of the 28th International Conference Com- putational Linguistics, pp. 6937-6941, Barcelona, Spain (Online), December 2020. Interna- tional Committee Computational Linguistics. doi: 1/26633///2022.co/// main.612 URL https: fttt::://acllanthelleso org/ coling-mainn 612.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907. 1692, 2019.
Daniel Lourciro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho- collados. TimeLMs: Diachronic language models from Twitter. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 251- Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 00.18633/1/2022. acl-demo.25. URL https: : /aa/aaaanttt aallanhologyy /22.2 acl acl-demo. 25.
Nicolás Marin, Gustavo divas-Gervilla, and Daniel Sánchez. Fuzzy logic for vagueness man- agement in referring expression generation. In Proceedings of the Workshop on Intelligent Information Processing and Natural Language Generation, pp. 71-76, Santiago de Com- postela, Spain, September 2020. Association for Computational Lingusties. URL https: //aclanthology o 2020 intellnnll lanttellang11...
Tawseef Ahmad Mir and Aadil Ahmad Lawaye. WORD SENSE DISAMBIUATION FOR KASH- MIRI LANGUAGE USING SUPERVISED MACHINE LEARNING. In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pp. 243-245, Indian Institute of Technology Patna, Patna, India, December 2020. NLP Association of India (NLPAI). URL https: aaaccllae aewccannhhloggyo org/ 202002220.22 1innn..c.oonn...oooo main. 32.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Guy D. Rosin, Ido Guy, and Kira Radinsky. Time masking for temporal language models. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM 22, pp. 833 841, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 0781450391320 doi: 10.1145/3488560.44988888 URL https:://// //t:::///oo// /ooi wrg/10......o.o.ooooooooo 1145/ 3488560 3498529.
Paul Röttger and Janet Pierrehumbert. Temporal adaptation of BERT and performance downstream document classification: Insights from social media. In Findings of the Association for Computa- tional Linguisties: EMNLP 2021, pp. 2400-240 2402, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/0011 www.iingsemmnll.....
URL https::///// / /////aaaaannt aacclanthlll. org/ 2021 find (anddnngs-mnnn 206..
11

---

Published as a conference paper at ICLR 2024
Philip E. Tetlock and Dan Gardner. Superforecasting: The Art and Science of Prediction. Crown Publishers, 2015.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers State-of-the-an arr natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https: / /wwww ac web wwtp:///gg.nooooogggooo.o.o...// 2020. emn 1 p- demos. 6.
12


Reviews:
[{"review_id": "0Rk7OGoR6p", "reviewer": "Reviewer_dCq4", "summary": "The authors proposed a language model that can capture the temporal pattern of text generation. They proposed 3 model variations to attack this problem and conducted experiments over a dataset of ACL paper abstracts. Their most advanced model, the doubly contextualized model, can outperform the GPT-2 based baselines and other variations in both automatic evaluations and human evaluations.", "questions": "NO", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Modeling of temporal patterns in LLM has not captured much attention from the community. Yet it is an important problem to look into.\n2. The paper is very easy to follow. The authors did a good job of describing their ideas and approaches in simple yet accurate terms and notations.\n3. The proposed models look reasonable and are proven to be effective in generating future text based on historical documents.", "weaknesses": "1. The proposed models are relatively simple and don't leverage the power of the most advanced LLM. Some problems the authors tried to solve, such as the gating problem in Sec 3.4 look like sth that would not be an issue to GPT-3 or other recently developed LLM as they are very effective in generating readable and coherent text. Finetuning a more powerful LLM with the latest text data seems to be a very effective way to model temporal patterns.\n2. Some details and questions from the experiment were not well discussed. For example, how many raters participated in the human evaluation, what are their agreements, and how subjective are their ratings? Besides, the results in Table 4. are worth more analysis and discussion. Why do the baselines not perform well in Problem and Method? Intuitively, they should be good at generating coherent and readable content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "akEfqFBZp8", "reviewer": "Reviewer_YEss", "summary": "This study pioneers the task of future language modeling, aimed at generating predictive text based on historical documents. It innovates by integrating temporal biases into a pre-trained GPT-2 model, guiding the generation of future-oriented text. The models, trained on NLP paper abstracts from the ACL Anthology spanning 2009-2021, surpass traditional non-temporal models in both automated (Perplexity, METEOR) and human evaluations.", "questions": "none", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality:\n- First to formalize future textual data prediction using temporal information.\nDevelops novel methods for measuring temporal dynamics in language modeling.\n\nQuality:\n- Presents a thorough structure, comparing three new models against multiple baselines.\n- Demonstrates model effectiveness through careful data handling, especially distinguishing between content and non-content words during evaluation.\n\nClarity:\n- Clearly articulates research motivations, background literature, methodology, and findings.\n\nSignificance:\n- Offers significant research outcomes with implications for various applications.\n- Discusses potential future applications and necessary adaptations.", "weaknesses": "To improve readability, you can align the organization of tables and figures more closely with their corresponding text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ulqHTN5pb", "reviewer": "Reviewer_QRZD", "summary": "This paper studies and introduces the task of future language modeling. This practically involves generation of text that would be written at a future time, which is not present in the pre-training of the langauge model used. This is possible through extrapolating usage from historical time intervals.\n\nThe paper introduces three methods that enhance the regular transformer-based architecture. These rely on LSTMs to model the historical frequency and predict its change with time.\n\nA new data set and evaluation setup is proposed, where the task is to predict future abstracts from the ACL conference. Results show the proposed methods improve over all baselines.", "questions": "Please expand on what stopword list was used, as that plays an important role in content perplexity and meteor scores.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "To the best of my knowledge, the application and task are novel and interesting to study.\n\nThree architectures are introduced, each building on each other. The effects of each component are studied and give us a sense of what is working and what is needed to improve performance.", "weaknesses": "More motivation and applications for this task could be provided or speculated.\n\nThe paper uses GPT-2 as the base model for all experiments. It would be better for testing robustness to potentially include a few more base models, perhaps with larger sizes or an encoder-decoder architecture for diversity.\n\nAdditional experiments and evaluation on downstream tasks can be performed using conditional generation or prompting on existing data sets with temporal dimensions e.g. the temporal NER data set (Rijhwani & Preotiuc-Pietro, ACL 2019; Luu et al NAACL 2022) or generating hashtags in future tweets using just the tweet text (similar to Preotiuc-Pietro & Cohn, EMNLP 2013). These would avoid the issues associated with evaluating generations.\n\nAnother interesting experiment to conduct would be to study prediction more into the future and quantify the expected degradation as the time window increases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)