Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Authors: Hui Dai
Ryan Teehan
Mengye Ren
Venue: ICML 2025 poster (2025)
Keywords: LLM
Forecasting
Continuous Evaluation
Temporal Generalization
Primary area: deep_learning->large_language_models
TL;DR: Daily Oracle is a continuously updated benchmark that turns daily news into forecasting questions, revealing how LLM performance degrades over time.

Abstract: Many existing evaluation benchmarks for Large Language Models (LLMs) quickly become outdated due to the emergence of new models and training data. These benchmarks also fall short in assessing how LLM performance changes over time, as they consist of a static set of questions without a temporal dimension. To address these limitations, we propose using future event prediction as a continuous evaluation method to assess LLMs' temporal generalization and forecasting abilities. Our benchmark, Daily Oracle, automatically generates question-answer (QA) pairs from daily news, challenging LLMs to predict "future" event outcomes. Our findings reveal that as pre-training data becomes outdated, LLM performance degrades over time. While Retrieval Augmented Generation (RAG) has the potential to enhance prediction accuracy, the performance degradation pattern persists, highlighting the need for continuous model updates. Code and data are available at https://agenticlearning.ai/daily-oracle.

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Hui Dai
Ryan Teehan
Mengye Ren
Abstract
formation, making it difficult to track models" performance variations over time (McIntosh et al., 2025). This creates a need for evaluation methods that always remain relevant and incorporate temporal dynamics.
Many existing evaluation benchmarks for Large Language Models (LLMs) quickly become out- dated due to the emergence of new models and training data. These benchmarks also fall short in assessing how LLM performance changes over time, as they consist of static set of questions without temporal dimension. To address these limitations, we propose using future event predic- tion as a continuous evaluation method to assess LLMs' temporal generalization and forecasting abilities. Our benchmark, Daily Oracle, auto- matically generates question-ansee (QA) pairs from daily news, challenging LLMs to predict "future" event outcomes. Our findings reveal that as pre-training data becomes outdated, L.LM per- formance degrades over time. While Retrieval Augmented Generation (RAG) has the potential to enhance prediction accuracy, the performance degradation pattern persists, highlighting the need for continuous model updates. Code and data are available https://ggenticlemmingg..g aiddaill
Daily news provides natural setting for continuous eval- uation of LLMs. Since the world constantly changing, benchmark designed around forecasting the next day's news will never be out of date by construction. In addition to enabling continuous evaluation, forecasting is itself a longstanding challenge with significant implications across various domains, including healthcare, finance, and policy- making (Tetlock & Gardner, 2016; Dempsey et al., 2017; Gillingham al., 2018; Lopez-Lira & Tang, 2023). While human experts have traditionally made such forecasts, chine learning models, particularly LLMs, have emerged as promising alternatives due to their ability to learn from vast and diverse corpora (Halawi et al., 2024; Ye et al., 2024; Yan et al., 2024). Several recent forecasting question-answer answer (QA) datasets have been developed (Jin al., 2021; Zou et al., 2022: Zhang al., 2024), however, they are limited in either size, scope, or they do not continuously keep pace with the rapidly changing world. More critically, the ex- tent to which L.L.s' predictive abilities change over time remains understudied.
oracle.
In this work, we propose Daily Oracle continuous eval- uation benchmark that uses automatically generated QA pairs from daily news to assess how the future prediction capabilities of L.MMs evolve over time. The QA pairs generated daily basis, consisting of True/False (TF) and Multiple Choice (MC) questions across various categories such as business, politics, and arts. Unlike traditional read- ing comprehension tasks, these QA pairs are designed to challenge LLMs to predict future events based on their own existing knowledge, effectively evaluating their temporal generalization and forecasting abilities.
1. Introduction
Traditional Large Language Model (LLM) benchmarks are often static, and do not reflect real-world information that evolves over time. This presents two significant challenges. First, as LLMs are updated, there is risk that static bench- marks become outdated and more vulnerable to data leak- age, where their content might end up the training data of newer models. This undermines the reliability of per- formance assessments on these benchmarks (Sainz et al., 2023; Xu et al., 2024; McIntosh et al., 2025; Li & Flanigan, 2024). Second, static benchmarks often lack temporal in-
We continuously evaluate various LLMs, both with and without access to limited archive of news articles. Our experiments reveal that I.LMS experience significant per- formance degradation between January 2020 and Decem- ber 2024, with degradation becoming more pronounced before and after the models' knowledge cutoff dates. On average, performance drops by 21.55% TF questions
'New York University. Correspondence to: Hui Dai 2hd25844nyu.eduu. Ryan Techan <rst306@nyu.eduu Mengye Ren <mengyo@nyy.edu.>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
and 11.33% on MC questions. While model performance can be improved with more recent news articles using Re- trieval Augmented Generation (RAG) (Lewis et al., 2020), the downward trend persists, suggesting the challenge in maintaining its prediction ability over time.
Dynamic QA Datasets. While static QA datasets evaluate models on fixed knowledge snapshots, dynamic QA datasets incorporate temporal dimension, allowing assessment of how models adapt to evolving information. Several dynamic QA datasets are proposed. Chen et al. (2021) construct TimeQA by using time-sensitive facts in WikiData with aligned Wikipedia passages to synthesize QA pairs. Zhang & Choi (2021) introduce SituatedQA by manually anno- tating temporally and geographically dependent questions. StreamingQA (Liska et al., 2022) and Realtime (Kasai et al., 2024) are both dynamic benchmarks with QA pairs an- swerable from news articles. StreamingQA, however, does not provide continuous evaluation with always-relevant data. RealTimeeA does not address forecasting and more like a plugin for a search engine, in the sense that tests whether model has updated its knowledge as facts change, rather than testing whether it can predict what will change given its knowledge of the past. FreshQA (Vu et al., 2024) contains a fixed set of human-written open-ended questions whose answers by nature can change based on new developments in the world, but is smaller and does not address forecasting. It is also updated weekly rather than daily. While all these datasets have some form of time-sensitivity like the Daily Oracle, they either do not provide continuous evaluation or do not evaluate forecasting capabilities, or neither.
To summarize, our key contributions are two-fold:
Continuous Forecasting Evaluation Benchmark: We present Daily Oracle, the largest and most up-to up-to-date fore- casting dataset, composed of automatically generated QA pairs. This benchmark continuously evaluates LLMs' tem- poral generalization and future prediction abilities using daily news, ensuring relevance over time and offering a challenging evaluation framework.
Empirical Findings on Performance Degradation: Since our benchmark provides new questions each day, we can study how model performance changes along the temporal axis. Our work effectively reveals clear per- formance degradation pattern in LLMs' forecasting accu- racy over time. Additionally, we study how this pattern changes as the LLMs are given access to updated knowl- edge up to different times. Surprisingly, we find that, even when the model has access recent information in an "open-book" setting, it still experiences performance degradation. Moreover, the sheer degree of decline, along with its smoothness over time, was unexpected. On the one hand, this highlights the problems with outdated LLM pre-training data and on the other hand underscores the need for continuous model updating.
Forecasting Datasets. Forecasting questions aim to assess model's s ability to predict the outcomes of future events based on its existing knowledge. Several datasets in the event forecasting field have been introduced. ForecastQA (Jin et al., 2021) used crowdworkers to collect 10,392 QA pairs from news articles. Zou al. (2022) argue that the QA pairs from ForecastQA are often nonsensical ambigu- ous since they written by humans without forecasting expertise. They further introduce AutoCast, a forecasting dataset from popular human forecasting tournaments con- taining 6,707 QA pairs. While ForecastQA and AutoCast remain static, ForecastBench (Karger et al., 2025) regularly updates a set of 1,000 forecasting questions either sourced from forecasting markets or generated via fixed templates based real-world event datasets. However, it still depends users actively submitting new forecasting questions or maintaining the underlying datasets. In contrast, our Daily Oracle dataset is generated automatically from daily news articles, which means that it is never out of date, can easily grow its size automatically without additional inputs from human forecasters. and provides more comprehensive event coverage than human forecasting tournaments.
2. Related Work
Temporal Generalization of LLMs. Lazaridou et al. (2021) define temporal generalization as the ability of Lan- guage Models to generalize well to future data from beyond their training period. They demonstrate that Transformer- XL's performance deteriorates over time, evidenced by in- creasing perplexity when evaluated on post-training data. However, perplexity-based metrics have two main limi- tations: they cannot be applied to closed-source models lacking accessible logits, and increased perplexity does not necessarily indicate degraded performance on downstream tasks (Röttger& & Pierrehumbert, 2021; Agarwal & Nenkova, 2022). Zhu et al. (2025) investigate temporal generaliza- tion using the Bits Per Character (BPC) metric. Similar to perplexity, BPC fails to capture higher-level performance on downstream tasks. In contrast, our work focuses the downstream forecasting task, evaluating how well models understand world knowledge and make predictions. This approach offers a more reliable evaluation of temporal gen- eralization with direct relevance to real-world applications and public interest.
Similar to our generation method, TI.B-forecas. (Zhang et al., 2024) has an automatic forecasting QA generation framework using news articles. However, their dataset is constrained both temporally and topically, only containing cooperation and conflict events in Middle-Faster countries
2

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Dataset
Continuous?
Interval
Forecast?
Size
Latest Update
TimeQA (Chen et al., 2021)
X
None
X
20,000
2021
SituatedQA (Zhang & Choi, 2021)
None
X
4,757
2021
StreamingQA (Liska et al., 2022)
X
None
X
36,800
2021
RealTimeQA (Kasai al., 2024)
X
None
X
1,470
2023
FreshQA (Vu et al., 2024)
Weekly
X
600
2024
QorecastQA (Tin et al., 2021)
X
None
V
10,382
2019
AutoCast (Zou et al., 2022)
X
None
6,707
2022
ForecastBench (Karger et al., 2025)
Biweekly
1.000
2024
TLB-forecast (Zhang et al., 2024)
X
None
6,604
2022
FreshBench (Zhu et al., 2025)
Unknown
2,769
2024
Daily Oracle (Ours)
Daily
31,510
2024"
Our experiments the subset generated until December 2024. Daily Oracle itself remains active, continuing to generate new questions daily from 2025 onward.
Table 1. We compare Daily Oracle with existing benchmarks the literature. For continuously updated datasets (e-g. Daily Oracle, FreshQA, FreshBench, and ForecastBench), "Interval" refers to the dataset update interval, and "Size" and "Latest Update" refer to the fixed data currently available. Our Daily Oracle benchmark is the only forecasting benehmark which is continuously updated every day using questions generated from daily news.
from 2015 2022. This restricts the dataset from evaluating more general event- event-predictie prediction abilities. Furthermore, con- sidering most of the powerful LLMs have been developed after 2020, the portions of the dataset covering earlier years may contain answers already seen during training. This prior exposure compromises the dataset's effectiveness as a forecasting benchmark. contrast, our dataset spans a broader timeframe and covers more topics, offering a more comprehensive forecasting benchmark.
forecasting ability. Additionally, at the fixed size we use for analysis we provide significantly more evaluation examples than the other automatically updated benchmarks.
3. The Daily Oracle Dataset
In this section, we present Daily Oracle, a continuously updated QA benchmark of forecasting questions that are automatically generated from daily news. For our current analysis of LLM performance, we utilize subset of the data consisting of 16,783 TF questions and 14,727 MC questions, covering a diverse range of forecasting topics, which are generated using daily news articles from January 2020 up until December 2024. However, our QA generation framework is continuous and updates daily. In Section 3.1, we describe our LLM-based dataset construction pipeline, detailing the data sources and the four-step construction pro- cess. Section 3.2 provides an analysis and general overview of the dataset. Lastly, in Section 3.3, we conduct a human evaluation, similar to our QA filtering process, to verify the quality of the generated QA pairs.
Note that none of the aforementioned datasets provide in- sights into how prediction ability changes over time. Zhu et al. (2025) introduce FreshBench, a forecasting dataset scraped from the Good Judgment Open platform, and also study temporal generalization. However, they report accu- racy in relatively short time window (from January 2023 to August 2024) with only 2,769 questions. While observe gradual performance decline in our dataset, they report significant fluctuations in model accuracy shortly after re- lease. A closer look reveals key limitations of forecasting market- based questions for studying temporal generaliza- tion: they suffer from limited early coverage, inconsistent distribution over time, and reduced dataset size after fil- tering due high proportion of low-quality questions, making it difficult to reliably analyze temporal performance trends. In contrast, our automatically generated dataset has broad event coverage, consistent growth, and more uniform question quality over time.
3.1. Dataset Construction
Data Source. Following Zou et al. (2022), we collect a large corpus of news articles from the daily-updated Common Crawl News Dataset (Nagel, 2016) with the news-ple sows-please package (Hamborg al., 2017). To fur- ther enrich our news dataset, we supplement it with daily scraped news using the Newspaper3x package. ? We filter for mainstream sources- CBS News, CNBC, CNN, Forbes, and NPR. While our data collection and evaluation are per- formed daily, for this study we utilize a static news corpus
In order clearly showcase the differences between our dataset and prior work, we highlight few key features in Table 1. The Daily Oracle is the only benchmark which is continuously updated on daily basis and evaluates
See Appendix C for details comparing I.I --geerreet and forecasting market datasets.
3

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
with 1,246,973 English articles spanning January 2019 to December 2024. This corpus is also used for the constrained open-book evaluation setting in Section 4.1.
1.2% Others flow mach will
2.0g Science 6 Tech
What
Where & Besioess
Hralthaa Biology
27.3%
10.1%
22.1%
Ensinansnnt
14.4%
How many
LLM--based Construction Process. QA pairs are gen- erated from articles published between January 2020 and December 2024.3 For each day, we select six articles for
21.2%
18.4%
will
Pollicc
16.2%
12,00
12.0%
Sports
Who will
Which will
Securits Dednzan
Arts & Becroation
(8) MC question types
(b) Category becakdown
QA generation: three are chosen randomly, and three are selected from hot topics. 4 For each selected article, we then use LLM to generate two TF QA pairs and two MC QA pairs with the few-shot prompting technique."
Figure I. Pie charts showing MC question type distribution and (b) question category distribution in Daily Oracle.
We adopt the methodology of Zhang et al. (2024), as their prompt design largely suits our setting. To further filter the QA pairs, we establish seven key criteria to ensure they qualify as valid forecasting questions, incorporating these into a QA Filtering step. The QA construction follows four steps, as illustrated in Figure 7:
Article Title
True/False Questions
Multiple Choice Questions
Probability on Day 101
1.0
1.0
1.0
0.6
9.6
0.6
0.4
0.2
0.2
0.2
0.0
0.0
0.0
40
80
80
20
60 100
(1) Article Summary, We generate a summary for each article, focusing on new events from the publishing date, instead of opinion articles discussing events from the past. This approach allows us to use the publication date as the resolution date of the generated question. Questions then be regarded as valid forecasting questions since they are prior to the resolution date.
Froquency the Post 100 Days Frequancy in the Past 100 Days Frequancy the Past 100 Days
Figure 2. Following Anderson & Schooler (1991), we plot the probability of word occurring in an (left) article title, (middle) True/False question, or (right) Multiple Choice question given how frequently it had appeared in one over the past 100 days, computed over our entire dataset. We fit a linear regression and show a linear relationship each case (I² = 0.843, 0.982, and 0.990 for left. middle, and right respectively).
(2) QA Generation. After filtering out the articles that do not introduce new events, two TF questions and two MC questions are generated together with the answers per article. To ensure balance in the TF questions, we instruct the LLM to generate the first question with a "Yes" answer and the second with a "No."
3.2. Dataset Analysis
the time of writing this paper, the subset dataset we use from Daily Oracle consists of 16,783 TF and 14,727 MC QA pairs, covering the period from January 1st, 2020, to December 31st, 2024, with average of 17.2 questions per day. Figure 1(a) shows that our dataset covers various MC question types, mainly starting with "What will" (27.3%), "Who will" (21.2%), and "Which... will" (18.4%). Figure 1(b) provides a breakdown of the categories, highlighting our dataset's broad coverage. The categorization of each question is determined using GPT--355, based on the prompt from Halawi et al. (2024). Examples of QA pairs are shown in Table 2.
Summary Statistics.
(3) Misleading Choices Generation. For MC, we provide the article, its publishing date, and the QA pair to the LLM, which then generates three misleading choices.
(4) QA Filtering, We prompt the LLM to check seven prin- ciples: correctness of answers, aon-answerability before the publication date, absence of information leakage, objectivity, inclusion of a clear temporal element, pub- lic interest, and non-obviousness of the answer. Each principle is scored with 0, I, or 2 points, and we se- lected the questions that received least 13 points in total. These principles are detailed in Appendix A.3.
Past and Future Information Usage. Each question in Daily Oracle implicitly requires the model to retrieve rel- evant knowledge. How do these requirements change day by day over the course of our benchmark? Anderson & Schooler (1991) explored similar patterns in human infor- mation environments. Inspired by their work, Figure ex- amines whether a word's frequency over the past 100 days
We use GPT-3.5 (OpenAI, 2024a) or GPT-4o-mini (OpenAI, 2024b) for steps (1) and (4), while GPT-4 (OpenAl, 2023) or GPT-4o (OpenAI, 2024b) is utilized for steps (2) and (3) to ensure high data quality.
For news articles in 2019, we use them as the corpus for the constrained open-book setting.
*See Appendix A.2 for hot-topic selection details.
"See Appendix D for all the prompts use. 'We use GPT-3.5 and GPT-4 until September 2024. After
October 2024, we switch GPT-4o-mini and GPT-4o, they are both more cost-effective and more powerful.
4

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Table 2. Daily Oracle Example Questions and Answers.
Type
Category
Question and Answer
IF
Politics & Governance
Will the presecution's 's key witness the New York hush money trial in April 2024 be someone other than
Michael Cohen? Noo
TF
Polittis & Governance
Will the House Energy and Commerce Committee vore unanimously to advance hill that could potentially
TikTok ByteDance does sell app by March 2024? -es.
MC
What
Science & Tech
What will he the starting price range for the Google Pixel 8a as of May 20247 4.$599-5644 B. B15199 $199-S249
C. 5750-$800, D. $499-$559, -D.
MC
Who
Sports
Who will go the injured list before the New York Mets' game May 29, 2024? A. Pete Alonso B. Edwin
Diaz C. Jelf McNeil D. Framcisco Lindor -B.
MC
Which
Arts & Recreation
May 2024, which streaming service will "TTh First Omen" become available for subscrihers? A.
Disncy+, B. Hulu, C, Amazon Prime Video, D. Netfix
MC
How many
Seience Toch
How many U.S. states will the path totality cross during the total solar celipse on April as reported by
Fehruary 2024? A. 15 B. 10 C. 20 D. 6-A.
MC
Where
Healtheare & Biology
Where will the second known U.S. case of bird in a human be reported by March 2024? California, B.
Texas, C. New York, D. D.Floroii Florida -B,
MC
How much
Economics & Business
How much will Apple, Inc. (AAPL) be up year to date by the end of June 2024? A. Up 44.55% B. Just over
19% C. 9.7%. D. 27%.
predicts its occurrence the next d.y-e.g.. if many questions concern the unemployment rate, will this trend continue?
4.2 presents the results, and Section 4.3 provides deeper insights into the observed degradation patterns.
We analyze this relationship for words in the titles of the articles we use to generate questions as well as in the text of the TF and MC questions themselves. Past frequency is computed by checking, for each day in the 100 day window, if a word has occurred in any article title (so, the maximum frequency is 100). We find that there is linear relationship between the frreuueccy of usage in the past 100 days and the probability of occurrence on the 10 day in all cases, replicating Anderson& & Schooler's s findings for New York Times headlines. This indicates that past information usage strongly predicts future retrieval needs, suggesting a tempo- ral structure in the knowledge demands of our benchmark.
4.1. Experimental Setup
Closed-Book Setting. We evaluate various LLMs on Daily Oracle assess their understanding of real-world events and temporal generalization abilities, i.e., how ac- curately LLMs can answer forecasting questions based on the knowledge they learned from their training data. Our evaluation differentiates between two scenarios based on the question's resolution date and model's knowledge cutoff' date: (1) Frowedwee Cutoff Questions: These questions have resolution dates before the model's knowledge cut- off, testing the model's understanding of past events. (2) Post- fast-Knoleedge Cutoff Questions: These have resolution dates after the knowledge cutoff, requiring models to pre- dict future events and test their forecasting and temporal generalization abilities.
3.3. Human Evaluation
To assess the quality of our LLM-based filtering method, we randomly sample 30 TF and 30 MC QA pairs and ask 4 hu- man annotators to evaluate them using the seven principles outlined in the QA Filtering step in Section 3. We evalu- ate the consistency among annotators using Fleiss' Kappa (Fleiss, 1971), which yields an average inter-rater agreement score of 0.26, indicating fair agreement. We then compute the human consensus score as the average of human scores and compare LLM-assigned scores, finding an average accuracy of 89.52% across the principles. For final QA pair acceptance (i.e., threshold above 13 points), the LLM and human consensus scores demonstrated an accuracy of 85.00%, further supporting the reliability of our LLM-based filtering approach. A detailed breakdown of human evalua- tion metrics is provided in Appendix A.4.
Constrained Open-Book Setting. In addition to closed- book evaluation, we explore the constrained open-book set- ting: how access to news articles up to diffcrent time cut- off's influences LLM performance using RAG (Lewis et al., 2020). We introduce the concept of the RAG cutoff (R- Cutoff), which limits the latest accessible date for retrieving articles. To prevent the models from leveraging information beyond the resolution date, for any question with resolu- tion date (dres), the accessible articles span from January 1st, 2019 9 (the start of our news corpus) up to whichever comes first between the day before the resolution date and the RAG cutoff date (dg-Cuuott). Formally, the accessible date range is [01/01/ /01/2019, min(c I, dR-Curott)) Following prior work (Jin et al., 2021; Zou et al., 2022; Zhang et al., 2024), we employ BM25 (Robertson et al., 1995) as the retriever and select the top 5 articles relevant to each question. We truncate each retrieved article to a maximum length of 512 words. These articles are then incorporated into the input prompts to serve as additional information.
4. Experiments
We first introduce three evaluation settings in Section 4.1: 1) no access to external information, 2) access to retrieved recent news articles, and 3) access to gold articles. Section
5

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
TF Questions
MC Questions
S-Month Moving Average Accuracy
M.Month Moving Average Accuracy
0.8
0.8
0.7
0.6
0.5
05
0.5
4
200041
2003
201-40
201--00
2001-00
500040
200
200-11
20044
200-0
2011-00
2011-00
20300
aed th
26300
20131
Car Month
Month
Model
Cluude 2.5 Sornet
6PT4
GPT35
Liama-3-00
Qwan3-7
Micira-ti77
Matrrsl77
Gerrma 2:28
Knowledge Cutoff
Apr
2321
2003
2024
Figure 3. Results for the closed-book setting. We plot the 5-month moving average accuracy for TF and MC questions across various models, showing LLMs' performance degradation in future event prediction.
Table For evaluating various LLMss with different knowledge cutoffs (K-Cutoffs), we show the yearly average accuracy (calculated the average across months) from 2020 2024, along with the average YoY accuracy change (%) before the knowledge cutoff date Cire-Cutoff), after the knowledge cutoff date (Post- Cost-CCtofff,, and the overall average YoY accuracy change across all months (Avg).
LLM
K-Cutoff
Average Yearly Accuracy (%)
Average YoY Accuracy Change (%)
2020
2021
2022
2023
2024
Pre-Cutoff
Post-Cutoff
Avg
Clande-3,5 3,5-Sonnet
Apr 2024
81.21
79.88
78.05
74.38
64.29
-477
-12.41
-5,58
GPT-4
Apr 2023
69.68
66.41
60.36
60.54
56.90
-5.83
-1.96
-475
GPT-3.5
Sept 2021
62.86
60.12
59.36
57.11
56.09
-4.33
-3.43
-284
Mixtral-xx7B
Unknown
57.83
52.69
43.09
39.34
36.01
-10.78
TF
Mistral-BB
Unknown
Llama 3 8B
Mar 2023
65.06
64.24
62.35
58.68
$6.99
-1.95
6.50
2.97
Qwen-2-7B
60.18
57.67
53.39
53.04
Gemma-2-28
Jul 2024
58.71
59.31
57.64
56.61
55.79
-1.41
-3,68
-1.04
Claute-3.-SSomee
Apr 2024
76.86
77.67
74.32
69.37
61.84
-6.26
-1 -1.78
-5.03
GPT-4
Apr 2023
70.60
70.62
66.76
56.36
$1.63
-4.23
-18554
-7.04
GPT-3.5
Sept
Mixtral-8x7B
Unknown
57.38
56.97
50.76
47.10
46.31
-4.68
MC
Mistral-7B
Unknown
50.07
52.36
48.06
44.40
42.99
-2822
Llama-3-4e 3-8B
Mar 2023
$2.44
54.18
50.66
47.94
46.95
-2.21
-1.25
2.30
Qwen
Unknown
$5.28
55.93
53.44
49.77
49.37
-235
Gemma-2-2B
Jul 2024
47.87
50.71
46.81
45.20
43.28
-4.66
-4.07
Gold Article Setting. We further include setting where models provided direct access the gold article, from which the question is generated." This transforms the fore- casting questions into reading comprehension ones, which can also access LLMs' general question-ans answering capabil- ities. Achieving high accuracy here ensures that the ques- tions from our Daily Oracle dataset are answerable.
instead provide definitive answers, there cases where such refusals still occur. The refusal rates provided in the Appendix B.2, and these cases are counted as incorrect to ensure comparability across model results.
4.2. Main Results
Results for the Closed-Book Setting, Figure 3 and Ta- ble 3 present our primary results for the closed-book setting, The "Avg" column in Table 3 shows the average YoY ac- curacy change of all months, revealing a clear degradation in performance over time across all models on both TF and MC questions. When comparing accuracies from the begin- ning to the end of the evaluation period, we observe that, average, the models' performance declines by 21.55% on TF questions (from 64.68% to 50.74%) and by 11.33% on MC questions (from 58.30% to 51.69%). This indicates that while LLMs demonstrate certain abilitics to understand real-world events and make predictions, they struggle to
Metrics. Accuracy score is used the evaluation metric. Though LLMs are tested daily, to show clearer trends, we plot the monthly performance in Figure 3, and apply a 5- month moving average to smooth the curve. We also report yearly averages and average year-over-year (YoY) accuracy change before and after models' knowledge cutoff dates in Table 3. Additionally, despite prompting the models to avoid responses like "I cannot predict the future" and
See Appendix B.5 for case example of evaluating L.LMs under all three different settings.
6

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
maintain these abilities.
Results for the Gold Article Setting. Figure 5 shows that when given access to the gold articles from which the questions are generated, LLM performance can approach around 90%, demonstrating the answerability of Daily Or- acle." However, most of the models still show declining trends. This is noteworthy because, ideally, LLMs are ex- pected to achieve consistent accuracy regardless of the arti- cle's publication date when answers are directly accessible. However, the outdated representations hinder their ability to consistently generate correct answers, even in reading comprehension setting.
Notably, the average YoY accuracy declines provide further insight. Before the knowledge cutoff, the average YoY decline across all models was relatively moderate. However, post-knowledge cutoff, we observe steeper declines in many models, with GPT-4 showing the most drastic drop in MC performance, declining by 18.54%, compared to just 4.23% before the cutoff. This contrast highlights that while LLMs manage to retain baseline of past knowledge with small degradation, their ability to forecast future events deteriorates much more rapidly as they move beyond their training data, struggling with temporal generalization.
4.3. Discussion
Among different models, Claude-3.5-Sonon (Anthropic, 2024) significantly outperforms all others, while GPT-4 excels in MC questions but its performance in TF is not as remarkable in MC. GPT-3.5, Qwen-2-7B (Yang et al., 2024) and Llama-3-8B (Dubey et al., 2024) show smaller temporal declines than GPT-4 in both TF and MC questions. Interestingly, Mistral-7B (Jiang et al., 2023) and Mixtral-8x77 Pattral-xxBBBo (Jiang et al., 2024) show the most pronounced drops in TF accuracy, with scores falling below the random baseline 50% due to increased answer refusals, as shown in Figure 9. Gemma-2-20 (Team et al., 2024) exhibits the most consistent performance with the smallest average YoY decline, likely due to its more recent knowledge cutoff date.
LLMs' Performance Degradation Pattern Over Time. We observe LLMs' performance evolution patterns in Fig- ure 3: (1) Gradual Decline in the Recent Past: In the months before the knowledge cutoff date, which we call the recent past, we observe a gradual decline in model performance, as seen in Llama-3-8B GPT-4, and Claude-3.5- S.5-ontett,, likely due to a lack of representation of recent news in the training data. (2) Rapid Decline in the Near Future: In the near future, which we define as the months following a model's knowledge cutoff date, sharp performance drops are ob- served in several models in MC questions. For instance, the decline in Claude-3.5-Sonnee and GPT-4 accelerates soon after their knowledge cutoffs. Most of the models, however, do not lose all the predictive power at once, as evidenced by the further decline into the farther future.
Results for the Constrained Open-Book Setting, In Fig- ure 4, we present the results of the constrained open- book setting, with Mixtral-8x7B on TF questions and 1lama-3- 8B on MC questions across different RAG cutoff' dates. x For Mixtral-8x7B, as the RAG cutoff dates extend to closer to the resolution dates, we observe a clear improvement in performance, indicating the model benefits from increas- ingly updated information retrieval. However, there are noticeable performance drops immediately after each RAG cutoff date when compared to providing information up to the day before the resolution date. This highlights the importance of keeping up-to-date information for optimal RAG performance. Interestingly, RAG does not uniformly enhance performance. Llama-3-8B may perform worse than the closed-book setting when the RAG cutoff is prior to the knowledge cutoff dates, suggesting outdated information may negatively impact performance. Conversely, for more recent RAG cutoff dates that extend beyond the knowledge cutoff, significant performance improvements are observed (as illustrated by the curves with cutoff's in September 2023 and March 2024). Notably, across all different RAG cutoff', the overall performance decline pattern persists, likely due to outdated internal representations and the model's inherent knowledge limitations.
We explore this further by analyzing the slope of accuracy as a function of time. In Figure 6, we show how the slope changes as we fit a regression to an increasingly larger win- dow of data, until we reach the full set of accuracies. Specif- ically, using the 5-month moving average of each model's accuracy on MC questions (visualized in Figure 3), start by fitting linear regression line on the first 10 months of data. We then add an additional month and compute new regression on the larger window, repeating until we reach the final month, and applying an exponential decay weighting to past data to reduce the influence of distant observations. With this, we can analyze how the slope of our regression line changes as each month is added to the data. The slope in each case is negative after the cutoff data and for Claude- 3.5-Sonnet, GPT-4, and Llama-3-8B, the slope eventually or immediately becomes more negative than it was at any point preceding the cutoff. Both Saaude-3.5-Somae and Llama- 3-8B have a crossover from positive to negative slope in late summer 2022, July and August, respectively, while GPT-4's 's seems to occur slightly earlier, in March of 2022. For GPT-3.5, GPT-4, and Llama-3-88 8B, the slope becomes
'Results for GPT-3.5 are provided and discussed in Appendix B.3, as this older model performs relatively poorly and including it on the same scale would obscure the trends of other models.
RRefer Appendix B.4 for results of other models in the constrained open-book setting.
7

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
5-Month Moving Average Accuracy
Mixtral-8x7B: TF Questions
S-Morth Moving Average Accuracy
Llama-3-88: MC Questions
0.7
0.55
0.6
0.50
0.5
0.45
9.40
2204
2205
2311-00
512--0
5011-40 40
022 60
122 60
0324 to
50M4 (
20140
2014
30340
537541
200-32
2000 28
2607
200
2601
203 23
2001-2
2000
2000
Toar-MMr Month
Month
Mixtral- 8x7B (Closed-Book):
Liama-3-88 (losed-Book)
Knowledge Cutoff: Mar 2023
RAG Cutoff:
Sep 2021
Sep 2022
Mar 2023
RAG Cutott:
Sep 2021
Sep 2022
Mar 2023
Sep
Sep
Mar 2024
Figure 4, Results for the constrained open-book setting, evaluating Mixtral-8x7B on TF questions and Llama-3-88 on MC questions with different RAG cutoff dates.
5-Month Moving Average Accuracy
TF Questions
5-Month Moving Average Accuracy
MC Questions
0.950
0.95
0.925
0.94
0.900
0.92
0.975
0.90
0.850
0.88
0.925
0.800
0.88
222- 43
1121.0
122
113-41 -1
510/11
5521
2001
223141
220-3
5529
Model
Cloude-3.--Somett
GPT4
Limma-3-00
Qwen-2-78
Mistrat-to77
Mixtral-7
Gemme-2-00
Knowledge Cutuff
Jel 2024
Figure 5. Results for the gold article setting. Most of the models struggle with temporal generalization, even when provided with gold articles containing the answers.
increasingly negative not long after the knowledge cutoff, giving evidence for rapid decline in the near future. future For example, Llama-3-8B's slope is around -0.06% per month near its cutoff in March 2023, but declines by more than 2 times to approximately -0.13% per month by the end of 2023. Likewise, the period preceding the cutoff shows milder decline, with models like GPT-3.5 and Llama-3-8B exhibiting slightly negative but consistent slopes (approx- imately -0.01 and -0.06% per month respectively, over the four months leading up to the cutoff). This suggests gradual decline in the recent past.
significantly higher compared to other models, as shown in Figure 9(b). 10 The lack of knowledge can partially recov- ered with information retrieval, as seen in the constrained open-book and gold article settings. For instance, Figures 9, 10, and 11 show that Mixtral-8x7B's refusal rate drops from 14-28% in closed-book to 3-15% with open-book retrieval, and further to 0.5-4.2% with gold articles. However, accu- racy still declines over time. Notably, the gold article setting provides an "upper bound" of open-book retrieval. The re- maining performance drop despite full access to relevant information suggests that the models' internal representa- tions are outdated. This indicates continuous pre-training of LLMs (Jang et al., 2022; Jin et al., 2022; Ke et al., 2022a;b; Yildız et al., 2024) is still needed in the context of news event forecasting.
Need for Continuous pre-training. The overall decline trend may come from two sources, the missing knowledge of future and a lack of up-to-date language representation. The absence of relevant future information can lead to two outcomes: either the model makes uninformed or incorrect predictions, or, in some cases, more likely to refuse to an- swer altogether. We observe this latter behavior notably in Mistral-7B and Mixtral-8x where refusal rates are
TF& & MC Comparison. All models except for Claude- 3.5 2.5-Sonet struggle with TF questions, where the degrada-
'OSee Appendix B.2 for more discussion of the refusal behavior.
8

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
(a) Claude-3,5-Smme
that can explain the disparity. First, TF questions can be considered more open-ended than MC because the "No" an- swer contains other possible open-ended outcomes. Second, since the distractor choices are created by an LLM, they may not be as likely to happen as the true answer.
Accuncy Movigg Aetrage
Accursty
A5 Sorey
Regresstnn
Consistent Performance Decline After September 2021. Interestingly, Figure reveals higher rate of performance decline around September 2021, which is the knowledge cutoff date of GPT-3.5, across all models, particularly for MC questions. In contrast, performance remains relatively stable prior to this date. We hypothesize that this trend arises because the period up to September 2021 may be overrepre- sented in many pre-training corpora (Raffel et al., 2020; Gao et al., 2020; Kobayashi, 2018; Gokaslan & Cohen, 2019; Zhu et al., 2015; Rae et al., 2020; Tiedemann, 201 6; Saxton et al., 2019), compared to more recent periods. Another potential cause of this imbalance is an increasing number of websites restricting access to web crawlers after the rise of ChatGPT (Longpre et al., 2024).
-3.000
300-
302
(b) GPT-4
Accuney
Accunacy
Avetage
0.9
0.5
Rogrtssionn Coufiiiien
6.02
0.000
0.004
1000
2000
(e) Llama-3-8B
Accurnay
Limitations. On the data generation side, the generated questions as well as the distractor answers could contain biases from an outdated LLM, making the benchmark less reliable in the long run unless we upgrade the models. Addi- tionally, generating questions from news articles can intro- duce bias by focusing only on events that have definitively occurred, overlooking potential events that never occur and thus never appear in the news. On the evaluation side, our paper proposes the continuous evaluation benchmark but at the time of the writing there isn't a long enough time horizon on each model, especially after the cutoff dates, for thorough analysis. Ideally, we would like analyze the relation between the effect of knowledge and RAG cutoff dates but the trend seems to be weak within the time horizon available.
Regression Contutent
0.000
a.
200
200
Start Date
(d) GPT-3.5
Rtag Tisgression Cosfitciint Accuraty
Pow Aczusery Noweg Average
0.003
Klowbigge
-6.022
10.004
XCD 41
400 4
2X0
200
Window
Dase
5. Conclusion and Future Work
We introduce Daily Oracle, a continuously updated QA benchmark leveraging daily news evaluate the temporal generalization and future prediction capabilities of LLMs. Our experiments reveal that while LLMs maintain a de- gree of predictive power over future events, their prediction accuracy exhibits a significant smooth decline over time. Although RAG mitigates the effect of outdated knowledge, strong and noticeable decline remains. Our findings the gold article setting further emphasize the importance of dis- entangling missing knowledge from the lack of up-to-date representations. In the future, alongside maintaining Daily Oracle, we plan to incorporate a broader range of models and explore how continuous pre-training and efficient adap- tation can address the performance degradation challenges presented in our work.
Figure 6. Coefficients for regressing accuracy on the MC questions against time, as the number of months grows. Using initial window of 10 months, we progressively add data for additional months to our regression and plot the cocfficient (slope) for the regression of accuracy against time. For our regression, we use the moving average of accuracy and apply exponentially decaying weights to older months (i.e., given window of l months, we weight n4 with X k=1; in this case 0.995).
tion trends towards the random baseline accuracy of 50%, indicating that predicting if future event will happen or not can be sometimes challenging for LLMs. In contrast, on MC questions, models tend to perform much better than the random baseline at 25%. There are two potential reasons

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Impact Statement
A., et al. The llama 3 herd of models. arXiv preprint arXiv:240..21788 2024.
Daily Oracle serves as an up-to-date, continuous evaluation benchmark for assessing AI Al models' forecasting accuracy and temporal generalization. These abilities have broad applications in areas such as finance, healthcare, and policy- making. The performance drop we observe highlights the risk of outdated knowledge and the importance of contin- uous model updates to keep Al systems reliable. In the future, by assessing continuous model update strategies in Daily Oracle, the broader MI. community can gain valuable insights into how to maintain AI systems that are relevant and well-informed on recent and upcoming events.
Ester, M., Kriegel, H.-P., Sander, J., Xu, X., et al. A density- based algorithm for discovering clusters large spatial databases with noise. In Proceedings of the Second Inter- national Conference on Knowledge Discovery and Data Mining, 1996.
Fleiss, L.. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378, 1971.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
Acknowledgment
This work was supported in part by the Institute of Informa- tion & Communications Technology Planning & Evaluation (ITTP) with a grant funded by the Ministry of Science and ICT (MSIT) of the Republic of Korea in connection with the Global AI Frontier Lab International Collaborative Re- search. (No. RS-2024-00469482 & RS-2024-005092279
Gillingham, K., Nordhaus, W., Anthoff, D., Blanford, G., Bosetti, V., Christensen, P., McJeon, H., and Reilly, J. Modeling uncertainty integrated assessment of climate change: A multimodel comparison. Journal of the Asso- ciation of Environmental and Resource Economists, 5(4): 791-826, 2018
We also thank the Microsoft Accelerating Foundation Mod- els Research program for providing Azure cloud compute credits for the LLM APIs. The compute was also supported by the NYU High Performance Computing resources, ser- vices, and staff expertise.
Gokaslan, A. and Cohen, V. Openwebtext cor- pus.
bitp://SkylionO0/ /ttp///y1111o00.gggtttubu.bb. github. OpenWebTextCorpus, 2019 2019.
Halawi, D., Zhang, F., Yueh-Han, C., and Steinhardr, J. Ap- proaching human-level forecasting with language models. In Advances in Neural Information Processing Systems, 2024.
References
Agarwal, O. and Nenkova, A. Temporal effects on pre- trained models for language processing tasks. Transac- tions of the Association for Computational Linguisties, 2022.
Hamborg, F., Meuschke, N., Breitinger, C., and Gipp. B. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of In- formation Science, 2017.
Anderson, J. R. and Schooler, L. J. Reflections of the en- vironment in memory. Psychological Science. 2(6):396- 408, 1991.
Jang, J., Ye, S., Yang, S., Shin, J., Han, J., Kim, G., Choi, S. J., and Sco, M. Towards continual knowledge learning of language models. In International Conference on Learning Representations, 2022.
Anthropic. The claude 3 model family: Opus, sonnet, haiku. http:////wwwwwch.atttttaao de0ba99b01c9abcccabbb5ee. htt.:aa88000bbbbb18888888 Model__arrd Caw_aardd_aaaa__a____aa_ pdf, 2024.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S.,, Casas, D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.., et al. Mistral 7b, arXiv preprint arXiv:2310.0683 2023.
Chen, W., Wang, X., Wang, W. Y., and Wang, W. Y. A dataset for answering time-sensitive questions. In Pro- ceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024.
Dempsey, W. H., Moreno, A., Scott, C. K., Dennis, M. L., Gustafson, D. H., Murphy, S. A., and Rehg. J. M. iSur- vive: An interpretable, event-time time prediction model for mHealth. In Proceedings of the 34th International Con- ference on Machine Learning, 2017.
Jin, W., Khanna, R., Kim, S., Lee, D.-H., Morstatter, F., Galstyan, A., and Ren, X. ForecastQA: A question an- swering challenge for event forecasting with temporal text data, Proceedings of the 59th Annual Meeting of
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan,
10

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
the Association for Compulational Linguistics and the 1thh International Joint Conference on Natural Language Processing (Volume I: Long Papers), 2021.
Longpre, S., Mahari, R., Lee, A., Lund, C., Oderinwale, H., Brannon, W., Saxena, N., Obeng-Marnu, N., South, T, Hunter, C., et al. Consent in crisis: The rapid decline of the ai data commons. In Advances in Neural Information Processing Systems, 2024.
Jin, X., Zhang. D., Zhu, H.,, Xiao, W., Li, S.-W.. Wei, X., Amold, A., and Ren, X. Lifelong pretraining: Continually adapting language models to emerging corpora. In Pro- ceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics:
Lopez- A. and Tang, Y. Can chatgpt forecast stock price movements? return predictability and large language models. arXiv preprini arXiv:2304.07619 2023.
Human Language Technologies, 2022.
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Xu, D., Watters, P., and Halgamuge, M. N. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. IEEE Transactions on Artificial Intelligence, 2025.
Karger, E., Bastani, H., Yueh-Han, C., Jacobs, Z., Halawi, D., Zhang, F., and Tetlock, P. E. Forecastbench: A dy- namic benchmark of ai forecasting capabilities. In Inter-
national Conference Learning Representations, 2025.
Kasai, J., Sakaguchi, K., Le Bras, R., Asai, A., Yu, X., Radev, D., Smith, N. A., Choi, Y., Inui, K., et al. Realtime qa: what's the answer right now? In Advances in Neural Information Processing Systems, 2024.
Nagel, S. Common crawl news dataset, https: / / contonncaannnc wwwp/oowggww....ooo..o.. CC N-WNNNW html, 2016.
OpenAL. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Ke. Z., Shao, Y., Lin, H., Konishi, T., Kim, G., and Liu, B. Continual pre-training of language models. In Interna- tional Conference on Learning Representations, 2022a.
OpenAL. dates.
New embedding models and api up- bttps://opent com/ index/ new-embedding andss--nd--aae apidupatess 2024a.
Ke. Z., Shao, Y., Lin, H., Xu, H., Shu, L., and Liu, B. Adapt- ing a language model while preserving its general knowl- edge. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, 2022b.
OpenAL. Gpt-4o system card. arXi:2440021276, 2024b.
arXiv preprint
Kobayashi, S. Homemade bookcorpus. https:// 2018.
Rae, J. W., Potapenko, A., Jayakumar, S. M., Hillier, C., and Lillicrap, T. Compressive transformers for long- range sequence modelling. In International Conference on Learning Representations, 2020.
github. wwm/ www.kkkkeboookoeoooooeeeeeeeoeee
Lazaridou, A., Kuncoro, A., Gribovskaya, E., Agrawal, D., Liska, A., Terzi, T., Gimenez, M., de Masson d' A'Atumee C., Kociskyy T., Ruder, S., Yogatama, D., Cao, K., Young, S., and Blunsom, P. Mind the gap: Assessing temporal generalization in neural language models. In Advances in Neural Information Processing Systems, 2021.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21: 1-67, 2020.
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-t., Rocktäschel, T., et al. Retrieval-augmented generation for knowledge- intensive nlp tasks. In Advances in Neural Information Processing Systems, 2020.
Robertson, S. E., Walker, S., Jones, S., Hancock-Beaulien M. M., Gatford, M., al. Okapi at trec-3. Nist Special Publication Sp, 109:109, 1995.
Röttger, P. and Pierrehumbert, J. Temporal adaptation of BERT and performance on downstream document clas- sification: Insights from social media. In Findings of the Association for Computational Linguistics: EMNLP 2021, 2021.
Li, C. and Flanigan, Task contamination: Language models may not be few-shot anymore. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024.
2024.
Liska, A., Kocisky, T., Gribovskaya, E., Terzi, T., Sezener, E., Agrawal, D., Cyprien De Masson, D., Scholtes, T., Zaheer, M., Young, S., et al. Streamingqa: A benchmark for adaptation to new knowledge over time in question an- swering models. In Internationaa Conference on Machine Learning, 2022.
Sainz, O., Campos, J., García-Ferree I., Etxaniz, J., de La- calle, O. L., and Agirre, E. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computa- tional Linguistics: EMNLP 2023, 2023.

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Saxton, D., Grefenstette, E., Hill, F., and Kohli, P. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations, 2019.
Zhang, Z., Cao, Y., Ye, C., Ma, Y., Liao, L.., and Chua, T.-S. Analyzing temporal complex events with large language models? benchmark towards temporal, long context understanding, In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
Team, G., Rivicre, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L.,, Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.0m 001 18, 2024.
Zhu, C.., Chen, N., Gao, Y., Zhang, Y., Tiwari, P., and Wang, B. Is your LLM outdated? deep look at temporal gener- alization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technolo- gies (Volume 1: Long Papers), 2025.
Tetlock, P. E. and Gardner, D. Superforecasting The art and science of prediction. Random House, 2016.
Tiedemann, J. Finding alternative translations in a large corpus of movie subtitle. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (6) 16), 2016.
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urta- sun, R., Torralba, A., and Fidler, S. Aligning books and movies: Towards story-like visual explanations by watch- ing movies and reading books. In Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
Vu, T., lyyer, M., Wang. X., Constant, N., Wei, J., Wei, J., Tar, C., Sung, Y.-H., Zhou, D., Le, Q., and Luong, T. FreshLLMs: Refreshing large language models with search engine augmentation. In Findings of the Associa- tion for Computational Linguistics ACL 2024, 2024.
Zou, A., Xiao, T., Jia, R., Kwon, J., Mazcika, M., Li, R., Song, D., Steinhardt, J., Evans, O., and Hendrycks, Forecasting future world events with neural networks. In Advances in Neural Information Processing Systems, 2022.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-thooglle prompting elicits reasoning in large language models. In Advances in neural information processing systems, 2022.
Xu, C., Guan, S., Greene, D., and Kechadi, M.-T. Bench- mark data contamination of large language models: A 2024.
survey, arXiv preprini arXiv:2406.04222
Yan, Q., Seraj, R., He, J., Meng, L., and Sylvain, T. Auto- cast++; Enhancing world event prediction with zero-shot ranking- based context retrieval. The Twelfth Interna- tional Conference on Learning Representations, 2024.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv:2407.106
2024.
Ye, C., Hu, 7.,, Deng, Y., Huang. 7., Ma, M. D., Zhu, Y., and Wang, W. Mirai: Evaluating I1m agents for event forecasting, arXiv preprint arXiv:2407.01231, 2024.
Yildız, Ç., Ravichandran, N. K., Punia, P., Bethge, M., and Ermis, B. Investigating continual pretraining in large lan- guage models: Insights and implications. arXiv preprint arXiv:2402. 17400, 2024.
Zhang, M. and Choi, E. SituatedQA: Incorporating extra- linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
12

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Appendix
A. Dataset Details
A.I. Illustration of the dataset construction process.
Figure shows how Daily Oracle automatically generated as discussed in Section 3.1.
Day
GA Construction
0-30 ..?
GA Fittiiggg
Midloding Cholces
GA Generation
Adii Summary
Article Selection
Data Bource
CBS NEWS
CNN
Common Grawi News
CNBC
Forbes
Figure 7. Data Construction Process of Daily Oracle.
A.2. Details for Article Selection
We select daily articles that generate the QA pairs in two ways: (1) Random Selection: We randomly sample three articles each day. (2) Hot Topic Selection: To better capture daily events and reduce noise, we select three articles from the top three hot topics of the day. We identify these hot topics by applying the density-based clustering algorithm DBSCAN (Ester Document Frequency) representations, forming
et al., 1996) to the new articles based on TF-IDF (Term Frequency-Invee clusters of news articles for each day. We filter out chaotic clusters by removing those with low average in-cluster cosine similarity scores, which typically correspond clusters containing large number of diverse articles. The top three clusters, determined by size, assumed represent the most discussed events, i.e. hot topies, since larger clusters indicate more articles covering the same event. One article is picked randomly from each of the top three clusters.
A.3. QA Filtering Principles
During the design stage of QA pair generation, manually review the questions and identify seven key criteria to ensure the QA pairs qualify valid forecasting questions. These principles guide the QA filtering step in the data construction process:
Correciness of Answers: The answer must be factually accurate and fully aligned with the information in the given article.
(2) Von-answerability Before the Publication Date: Since we treat the article's publication date the question's 's resolution date, the question should not be definitively answerable based on information available before the article's publication.
(3) Absence of Information Leakage: Questions must avoid revealing information that became known only after the article's publication, maintaining fairness for pre- publiccatiion evaluation.
(4) Objectivity: Both questions and answers must rely on objective facts, avoiding subjective ideas from the authors.
(5) Inclusion of a Clear Temporal Element: Questions must contain specific and clear reference to time, avoiding vague phrases like "in the future" or "soon."
(6) Public Interest: The questions should address topics of broad public concern.
(7) Non-obviousness of the Answer: The answer should not be immediately predictable from the question and must provide new or non-trivial insights.
13

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
A.4. Details for Human Evaluation
We assess the quality of our dataset by evaluating the effectiveness of our LLM-based evaluator in the QA Filtering step. Four human annotators independently review randomly sampled subset of Daily Oracle, consisting of 30 TF and 30 MC QA pairs. They follow the same instructions used to prompt the LLM and evaluate each QA pair based on the seven filtering principles listed in Appendix A.3.
Table 4 presents the inter-rater agreement among human annotators and the agreement between human and LLM evaluators. The average Fleiss' Kappa of 0.26 indicates fair agreement among annotators. Among the seven principles, Objectivity exhibits the highest agreement (0.66), while Non-Answerability Before the Publication Date has the lowest (0.02).
Comparing human-assigned and LLLM-ssiggee scores, the exact-match accuracy between the human consensus and L.M evaluations averages 89.52% actoss the seven principles, showing the effectiveness of our LLM-based filtering method. Non-Answerability Before the Publication Date shows the lowest agreement (83.33% accuracy), suggesting it is the most challenging principle for both humans and the LLM to evaluate consistently.
Table 4. The inter-rater agreement among four human annotators is evaluated using Fleiss' Kappa, while the agreement between human LLM evaluators is measured through accuracy scores. We report metries across seven QA Filtering principles using a sample 60 randomly selected QA pairs.
Human Agreement Fleiss' Kappa
Human vs. LLM
Metric
Accuracy (%)
Correctness of Answers
96.67
Non-answerability Before the Publication Date
0.02
83.33
Absence of Information Leakage Objectivity
0.33
86.67
0.66
98.33
Inclusion of Clear Temporal Element
0.21
90.00
Public Interest Non-obviousness of the Answer
0.18
88.33
0.29
83.33
Average
0.26
89.52
A.5. Distribution of Question Categories Over Time
In Figure 8, we provide the distributions of question categories for both TF and MC questions.
TF: Category Distribution Over Time by Month
MC: Category Distribution Over Time by Month
Gelegery
Ecatomics Budrees
Education Research
Enskorment energy
bo mgy
Folltec& devemance
Count
Scincce Tech
Securiy Dalanec
Sport
330
Figure 8. Question category distribution over time.
B. Experiment Details
B.1. Baseline Models Information
Table 5 lists the LLM model versions used in our experiments.
14

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Table 5. Baseline model versions.
Model
Model Version
Claude-3.5-Somne
claude-355sonnee22224666
GPT-4
gp--4-110-ppreview
GPT-3.5
gp--3.5-urro-0125
Mixtral-8x7B
Mixtral-kx7B--nsstuutt-dd
Mistral-7B
Mistral-7B-Instruct--kk.
Llama-3-8B
Meta-Llama-3-8B-meee
Qwen-2-78
Qwen2-7B-Instrul
Gemma-2-2B
gemma-2-2b-it
B.2. Refusal Rates
Although the models are prompted to provide definitive answers rather than responding like "I cannot predict the future," some models still occasionally refuse to do so. Figures 9(b), 10(b), 11 show the refusal rates for the closed-book, constraint open-book, and gold article settings. In closed-book evaluation (Figure 9()).. we can see that the refusal rates increase throughout the time for Mistral-7B in TF questions and Mixtral-8x7B in both TF and MC questions. Additionally, these two models exhibit notable refusal rates, with approximately 10-30% on TF questions and 1.5-8% on MC questions, resulting in their closed book performances dropping below the random baseline of 50% in certain months, as shown in Figure 3. In comparison, Qwen-2-7B and Gemma-2-2B show relatively low refusal rates=<5% for TF and <2% for MC-while all other models have near-zero refusal rates for TF and <1% for MC.
The refusal behavior likely influenced by alignment techniques, which discourage uncertain responses the post-training stage. Although refusal rates contribute to lower accuracies for certain models, our results show that performance degradation trends persist even when refusals are excluded (Figures 9(),, 10(a), (a)). We consider refusal answer an indicator of performance limitations in forecasting tasks, it reflects the model's lack of actionable knowledge. When models are supplied with more up-to-date and relevant information, their refusal rates decrease (Figure 10(b)). This suggests that refusal is one example of the broader challenge of temporal generalization and reinforces the need for continual model updates or improved external knowledge integration.
B.3. Results for GPT-3.5 in the Gold Article Setting
To more effectively illustrate the trends of other models at a suitable scale, we display GPT-3.5's performance in the gold article setting separately. As shown in Figure 12, this outdated model performs relatively poorly throughout. While its accuracy could improve with chain-of-thon chain-of-thought prompting (Wei et al., 2022), we report its performance using the same prompt format as the other models for consistency in comparison. Nevertheless, the degrading trend can still be observed.
B.4. More Results in the Constraint Open-Book Setting
Figures 13, 14, 15, 16, 17, 18, and 19 show the constrained open-book evaluation results for more models. Similar patterns are observed as discussed in Section 4.2. Specifically, for Seuude-3.5-Sonnee the constrained open-book performance lags behind its closed-book performance, likely because it already has robust representations of world events, suggesting that irrelevant or confounding retrieved information may degrade performance. This highlights the need for more careful RAG design in models that already possess robust world knowledge. GPT-3.5 is not included in the constrained open-book setting due to its unexpectedly poor performance in the gold article setting (Figure 12) and budget limitations. Additionally, due to budget constraints, open-book evaluations of proprietary LLMs (Claude-3.5-Somel GPT-3.5, GPT-4) are conducted only up to September 2024, whereas other LLMs are evaluated through December 2024.
B.5. An Example of Evaluating L.MMs Under Different Settings
Figure 20 presents a case study demonstrating how Mixtral-8x77 responds to a question under different experimental settings. The model provides an incorrect answer in the glosed-book setting. However, when supplemented with retrieved relevant articles or the gold article, it produces the correct answer.
15

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
(a) Closed-book: Accuracy excluding refusal cases
TF Questions
MC Questions
5-Month Moving Average Accuracy
5-Month Moving Average Accuracy
0.8
0.80
0.75
0.7
0.70
0.65
0.60
0.5
0.55
0.50
200000
213811
2001
2021-00
0001
2000
205000
220-99
2501
2011-00
Month
(b) Closed-book: Refusal rate
TF Questions
5-Month Moving Average Refusal Rate
MC Questions
0.30
5-Month Moving Average Refusal
0:25
0.20
0.15
0.10
0.06
0.00
2707
200
555
0
400
200
202
2003
200113
2022
2014
2034
FOR m0
$224
$20
Year-Month
Model
coude-35-ormst
GPT-
GP1-335
Liana 88
Qwem-2-7B
Mixtral Sxcr8
Mistral
Dommas2225
Knowlodge Outoff
2022
2021
Mar 2023
Figure 9. Accuracy excluding refusal rates and refusal rates under the closed- book setting. We plot the 5-month moving average refusal rates for TF and MC questions across different LLMs. We count refusal cases as incorrect both to maintain comparability across models and because failing provide
when prediction expected represents unsatisfactory outcome from the user's perspective.
C. Comparing LLM-GGnerated and Forecasting Market Datasets
Online forecasting markets such as Metaculus and Polymarket allow users to submit questions and predict the outcomes of future events. A natural question arises: why do we focus on LLM-gencrated questions rather than sourcing from these markets, as the choice in prior work (Zou et al., 2022; Halawi et al., 2024; Karger et al., 2025)?
To answer this, we analyze the dataset from Halawi et (2024), which compiled 50,343 raw questions from five forecasting platforms, of which 21,149 were resolved. Of these, 82.64% TF questions. 13.36% MC questions, and the rest are free-response or numerical. After their quality filtering, only 5,516 TF questions remained. We observe that, due high proportion of low-quality questions in the raw data, sparse coverage in earlier years, and inconsistent distribution over time, performance trends derived from this dataset are substantially more volatile and harder to interpret than those based on our LLM-generated dataset.
Lower Quality in Raw Questions. Manual inspection confirms that substantial portion of the raw dataset consists of low-quality questions, as also noted by the original authors. Examples include: "Will I have a chess.com rating of > 1300 ...?" (personal), "Will Jamaica beat Mexico?" (missing a time frame), and "Are there more disadvantages in Al than advantages?" (ill-defined). From a random sample of 50 questions, only 28% were well-defined. Specifically, 26% lacked a clear time element, 20% were overly personal, and 26% were ill-defined. Importantly, only 5,516 out of 17,477 resolved TF questions were retained after their filtering an acceptance rate of just 32%, which aligns with our own quality assessments.
Limited Early-Year Coverage. Figure 21 (left) shows that the coverage before October 2022 is sparse, averaging only 40 raw and 26 filtered questions per month. This scarcity limits the feasibility of longitudinal trend analysis, especially
16

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
(a) Constraint open-book: Accuracy excluding refusal cases
5-Month Moving Average Accuracy
Mixtral-8x7B: TF Questions
5-Month Moving Average Accuracy
Mixtral-8x7B: MC Questions
0.65
0.70
0.60
0.60
0.55
0.55
0.50
223500
2545
2524 10
502-400
5321 00
903-11
521400
50343
20/4-0
3000-0
5550
2105
5527 m
2003
205-0
2003 20
of
2022 1023-11
23
5000-07
5009
225433
2004 20
255407
201
Month
(b) Constraint open-book: Refusal rate
S-Month Moving Average Refusal Rate
5-Month Moving Average Refusal Rate
Mixtral-8x7B: TF Questions
Mixtral-8x7B: MC Questions
0:30
0.25
0.06
0.20
0.15
0.04
0.10
0.05
200
2013
5503 0R
2552
2002
2013
20E5 31
203 201400
2034-00
5004
200
2009
2000-2
AEDU CL
1000
200
1009
2009
2000-0
Year-Monte
Month
Year-M
Par--ooth
Mixtral-8x7B 8x7B (Closed-Book):
RAG Cutoff:
Sep 2021
Sep 2022
Mar 2023
Sep 2023
Mar 2024
Figure 10. Accuracy excluding refusal rates and refusal rates for Mixtral-8x77 -8 under the constrained open-book setting. (h) shows that the open-book refusal rate (blue curves) is lower than in the closed boookdbbooooo setting (cyan curve), indicating that access to more up-to-date and relevant information reduces model refusals.
across earlier model pre-training cutoffs. contrast, our method supports high scalability and retrospective generation, allowing for uniform coverage across the full time range.
closed- Harder-to-Discern Trends. To evaluate the impact of using forecasting market questions in study,
book evaluation on TF questions from both the dataset (16,089 questions) and the filtered subset (4,572 questions), starting from 2020-01 (the same start date our dataset). Notably, the original data is imbalanced, with 61.03% "No" answers in the raw set and 64.28% in the filtered set. After balancing, we retain 12,438 questions from the raw data and 3,232 from the filtered set. As shown in Figure 21 (right), neither the raw nor filtered datasets reveal a clear performance send-model accuracy fluctuates significantly over time. We believe this due to several factors:
Lower data quality: Approximately 70% of raw questions exhibit quality issues. While the overall dataset sizes are comparable (13,744 in ours 12,438 in the raw market dataset), the difference in quality introduces additional noise, making trends harder to detect.
Limited early coverage: Even within the filtered dataset, sparse early coverage and inconsistent monthly volume increase variance and reduce the reliability of time-based trends.
Confounding factors: We argue that human-submitted questions introduce more confounding factors than automat- ically generated ones. Figure 22 shows the distribution of data sources and question categories varies significantly across time (e.g. more sports-related questions in later periods). cluman-writted questions also may differ widely in style and difficulty, making them harder to control for consistency. In contrast, as shown in Figure 8, our dataset maintains relatively stable distributions over time.
17

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
(a) Gold article: Accuracy excluding refusal cases
6 6-Month Moving Average Accuracy
Questions
5-Month Moving Average Accuracy
MC Questions
0.850
0.925
0.900
0.92
0.875
0.850
0.90
0.825
800
$10000
222/00
152-0
24000
200000
205-1
22300
20000
N05-0
30540
3003
2054
Year-Month
(b) Gold article: Refusal rate
5-Month Moving Average Refusal Rate
Questions
5-Month Moving Average Refusal Rate
Questions
0.05
0.020
0.015
0.03
0.010
0.02
0.006
0.01
0.000
$30001
SCRIE
230400
110-11
207400
5ID 09
m
on
1500 11
2000
$100
203
ml
200-11
553-00
20011
200-2
200-11
1911 31
201
200-3
20000
X00-
75
4833-17
205-4
2004
Month
Month
GPT-4
8xB8
Knowledge Cutoff
2024
Apr 2023
Nar 2023
Jul 2004
Figure 11. Accuracy excluding refusal rates and refusal rates under the gold artiele setting.
TF Questions
MC Questions
5-Month Moving Average Accuracy
5-Month Moving Average Accuracy
0.63
0.62
0
0.61
0.60
0.59
78
2000-4
200--3
2000-00
2000 99
2000 11
2003 0
2003 69
540 02
2002-00 01
5000 03
09
2800 08
1990-11
(90--8
5000-00 05
2000 07
2005 99
11
2004 19)
2004-10 op
200-48
2004 11
2005-20
2000 09
3000-09
2005-00
2000-11
2022-00 01
09
2002-07
2000
2000-1
2000-000
1009 80
07
9)
11
8003 08
1005-00
2007 29
2005-11
2004-0
2004
200402
Month
Year-Monti
GPT:3.3
Knowlddge Cutoli: Sep 2023
Figure 12. Results for GPT-3.5 in the gold article setting. Compared other models achieving around 0.9 accuracy, GPT-3.5 performs
worse in both MC questions and, more notably, in TF questions.
Thus, while do not claim that LL A-M-egenrated questions of inherently higher quality, argue that dataset is better suited for analyzing performance trends over time, due to its scalability, stylistic uniformity, stable category distribution, and reduced susceptibility to human-authi authord confounders. Moreover, if one sources questions from forecasting markets, the dataset update frequency is dependent on whether people are still actively submitting high-quality forecasting questions to the platform. In contrast, our approach enables daily updates and more comprehensive event coverage, making it a valuable complement human-curated forecasting benchmarks.
18

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
5-Month Moving Average Accuracy
Claude-3.5-Sonnet: TF Questions
5-Month Moving Average Accuracy
Claude-3.5-Sonnet MC Questions
0.80
0.80
0.75
0.75
0.70
0.70
0.65
0.65
0.60
0.60
0.55
2009-43
2000-42
2008--7
2000
2008
2001 01
2003 25
2001 46
200 47
2007 49
2001 11
2009-7
2002 ce
7002
200422 G1
2000 6o
2003 c5
2009 67
2009 60
2008-11
2508
2008
2000
2008--32 12
2002-
2001 c0
2001 -5
2001 07
2001 00
200) 11
2002 43
2000-00
2002
2002
2002
2009 65
200343
2003-2
2000-33
2008 C1
2500
CO
Month
Gar-MMoth
Blaude-3.5-oomme (Closed-book):
Knowledge Cutoff: Apr 2024
Sep Sep
Mar
Sep 2023
Figure 13. Results for Claude-3.5 Manude-3.5-Somme 3-Sonnet the constrained open-book setting.
5-Month Moving Average Accuracy
GPT-4: TF Questions
5-Month Moving Average Accuracy
GPT-4: MC Questions
7.
0.75
0.70
0.65
0.65
0 860
0.55
0.50
2000
2000 08
2000
2000
2000 00
2080 21
8003 01
2003-00
2821-00
2002 06
2002
2002
2009 0L
200 50
2009-6
2000
2008 09
8008 11
200801
2609-00
45
8009-01
2009
2600
2600
2000
2502
2021
2027-006
2021-07
2002-00
2000--1
2000-0
2000-00
2012
9882 d6
5002 11
3007-00
2004
200400
2000
2006-0
ear-Month
Month
GPT-4 (Closed-book):
RAG Cutoff:
Sep
Sep
Mar
Sep
Knowledge Apr
Figure 14. Results for GPT-4 the constrained open-book setting.
5-Month Moving Average Accuracy
Mixtral-8x7BB TF Questions
5-Month Moving Average Accuracy
Mixtral-8x7B: MC Questions
0.7
0.60
0.6
0.4
0.45
2002 00
2000 09
2003 01
2001 08
2001 07
2003 11
102201
107540 a1
1022 05
117-01
102200 00
2003 ot
1003 90
503 on
101 07
79700
11
2024 68
2024 07
2214 2014-0
7P4-11
2003 m
2000
2003 00
AMP3455
502200
100-11
2021 0L
3003 40
2003 05
(0710)
202201 0l
R072 00
10z in
202200
103300
6003 85
203300
1023 11
2024
207-00 10
2024 07
201-00
201-11
2025 m
Year-Month
Year-Month
Mixtral-8x78 (Closed-Book):
RAG Cutoff:
Sep
Sep
Mar 2023
Sep
Figure 15, Results for Mixtral-8x77 8x7B in the constrained open-book book setting.
D. Prompts
All the prompts shown in this section. The QA generation prompts and evaluation prompts adapted from Zhang et al. (2024), and the prompt to categorize our generated questions is taken from Halawi et al. (2024).
19

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
5-Month Moving Average Accuracy
Mistral-7B: TF Questions
5-Month Moving Average Accuracy
Mistral-7B: MC Questions
0.65
0.55
0.60
0.50
0.55
0.45
0.50
0.45
0.40
0.40
0.35
2000 01
20000 03
2020-1
20124 01
2012 03
2077-66
07
0022 41
2012441
8972-65
205200
2000
2028 05
2033441
2019-11
2026-01
0076-03
2014 05
2036-077
09
8929
90000
2036-77
2000 05
3077-11
2022 01
8071 07
2012 09
3771-1
01
8022 07
2022-19
2722 11
2073406
2008 11
2026-41
2004
0024 09
2026-11
2026-0
Year-Me Month
Year-Mor Monthh
Mistral-7B (Closed-book):
Cutoff:
Sep
Sep
Sep 2023
2024
Figure 16. Results for Mistral-7B in the constrained open-book setting.
5-Month Moving Average Accuracy
Llama-3-88: TF Questions
5-Month Moving Average Accuracy
Llama-3-38 MC Questions
0.68
0.66
0.55
0.64
0.50
0.62
0.45
0.56
0.40
2000
2005 68
2200-00
2000-33
2001
2002 08
150
7022
2002 04
200 68
2002 n02-06
2002
1009-1
2003 c1
200143
700366
61
2005 11
2004-33
2004 60
7704-00
2004
2005-11
2000
25000
2000- 05
2000-000
2000
2001
2002 00
2007 -0
2002 07
250
2002
2002
1007 26
2002 07
1107
100
2003 20
2003
7103-00
c4
310 1D04-00
2004 06
2004-7
00
21
Year-Month
(ear-Month
Llama-3-88 (ookk:
RAG Cutoff:
Sep 2021
Sep 2022
Mar 2023
Sep 2023
Mar 2024
Knowledge
Figure 17. Results for Llama-3-88 in the constrained open-book setting.
5-Month Moving Average Accuracy
Qwen-2-7B: TF Questions
5-Month Moving Average Accuracy
Qwen-2-7B: MC Questions
0.650
0.60
0.625
0.600
0.55
0.575
0.550
0.50
0.525
2000 C1
2000 23
2000-66
2009
2022
2000 25
c0
57
2001-20 21
(000-0)
2002
E0000 21
2012 21
ca
07
2000-c co
1000-11
2002 ci
2004-C3
2004
200/-99
2004-33 21
2085--2 01
2000 01
2001
2000-29
2000-33
01
2007-0
7001-00 de
2001
1800-33
2002
2001
11
2000--3
2003
2009 07
2022
2003-33
5004-55
2001-7
2004 09
2004-11
Year-Month
Year-Me
Parr-Motth
Qwen-2-7B (Closed-book):
RAG Cutoff:
Sep 2021
Sep 2022
Mar 2023
Sep 2023
Mar 2024
Figure 18. Results for Qwen-2-7B in the constrained open-book setting.
20

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
S-Month Moving Average Accuracy
Gemma-2-2B: TF Questions
5-Month Moving Average Accuracy
Gemma-2-26: MC Questions
0.650
0.625
0.50
0.600
0.575
0.45
0.550
0.40
0.525
0.500
2000
200000
2000
2009
2022
2010 00
202 06
-00
200
3022
2022 00
2002
2012-
2023 00
2009-00 -0
2003-44 re
2002
2004-00
2004-00
2004 co
2004 11
2020-00
2000-00
700--00
2000
2000
2012-00
2013-00 -0-0000
-0
2007
207
2022 04
3103-00
2002
5000000 0000000
202-11
1039 00
7002
2003 (0
2023 11
70D4-00
2004
201400
2004 00
2004-1
Year-Month
Year-Mont
Gemma-2-22 (Closed-book):
RAG Cutoff:
Sep 2021
Sep 2022
Mar 2023
Sep
Mar
Knowledge Cutoff: Jul 2024
Figure 19. Results for Gemma-2-2D in the constrained open-book setting.
21

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
[News Article]
Title: DFL leaders, Minneapolis announce deal on rideshare pay, but Lyft and Uber say they will leave Publishing Date: 2024-05-06
Text: ST. PAUT., Minn. Key lawmakers and DFI. legislative leaders on Monday announced that they have compromise statewide rideshare regulations in Minnesota, but Uber and Lyft quickly rejected the plan and said that they will still exit the market the proposal becomes law.
The amendment, which will be heard in a House committee Tuesday, includes minimum wage pay rate of S1.27 per mile and 49 cents per minute. Leaders said they worked with the Minncapolis City Council on finding compromise, but the companies were not included the negotiations.
The proposal is lower than the Minneapolis rate of $1.40 per mile and S0.49 per minute, but higher than study done by the Minnesota Department of Labor and Industry, which determined $0.89 mile and $0.49 for the Twin Cities metro would bring driver pay up the citys minimum wage of $15.57 hour.
"This was a really big move for the Minneapolis City Council. This is coming down 13 cents from the rate that they passed. And this is consistent with the rates that Uber and Lyft are operating with and other states and other jurisdictions." House Majority Leader Jamie Long, DFI- -Minnealssss who was key negotiator, told reporters. "We know that the companies have said the same thing other states. They's ve said that they were going to leave at certain rates, and they' 've stayed and been able to operate profitably."
Uber and Lyft quickly responded that they will still plan to leave Minneapolis - as they said would happen on July 1, when the cityš ordinance is effective and the state if this becomes law.
[Gemerated QA Pairs]
Question: What will be the response from Uber and Lyft if the proposed rideshare regulations become law in Minnesota by May 2024?
Answer: (a) They will plan to leave the market.
Misleading Choices:
(b) They will stay and adjust their pricing to meet regulations.
(c) They will collaborate to develop an alternative service model
(d) They will reduce their services rather than exit.
[Closed-Book Settingl
The question is asked directly to Mixtral-8x7B.
Model response: (wrong) "(d) They will reduce their services rather than exit. This is because leaving the market entirely can be a drastic move and may not be the first response to new regulations. Reducing services is more likely response, as it allows rideshare companies comply with the new laws while still maintaining presence in the market. Additionally, they may also try negotiate or influence the regulations before they become law."
[Constraint-Open Book Setting]
The RAG-cutoff refers to the latest accessible date for retrieving articles. With RAG-cutoff = 2024-03-01, extracted the top relevant articles (here, only the titles are displayed, but the articles included in the evaluation prompt):
2222-00--25 Uber Threatens To Most Service Minnesota Minimum Pay Law Takes Effect
(2023-05-25) Minnesota Governor Vetoes Rideshare Minimum Pay Bill After Uber Threatens To Leave State
(2023-08-09) Minncapolis City Council moving towards minimum wage for rideshare drivers
(2024-02-27) Minneapolis City Council to consider rideshare minimum wage ordinance Tuesday
(2024-01-11) Some Lyft, Uber drivers to strike Thursday at MSP Airport
Model response: (correct) "(a) They will plan to leave the market."
[Gold Article Setting]
In the gold article setting, models provided direct access to the gold article, from which the question is generated. The question then is treated as a reading comprehensive question rather than forecasting question.
Model response: (correct) "(a) They will plan to leave the market."
Figure 20. example of evaluating Mixtral-8x7B across different settings.
22

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Forecasting Market- Row. Monthly Resolved Questions
Forecasting Market Raw: TF Questions
S-Merth Moving Average Accurayy
1000
0.5
202
Monch
Forecasting Market Fltered Month Questions
Forecasting Market Filtered: Questions
Market-Filtered:
Month Moving Average Accuraty
400-
0.7
320
36
250
0.
120
200
122
230-5
Daily Gracle: Monthly Resolved TF Questions
S-Monith Meving Ancuracy
Daily
Questions
&
X040
Claude--3 Coaai-33--aaeee
Mara (M
Commer2.00
Krewslngg Caleri Aar
Knowindge Outal: Var 202)
Ower 2/1
Keovelite Curahh 3d 2014
Figure 21. Dataset size and closed-book evaluation accuracy over on the forecasting market dataset.
23

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Farecasting Market Filtered: Manthly Resoived Questions by Data Saurce
Data Source
gjopen
manifald
metaculus
polymarket
100
10004
310000
1000
2000 0s
y00 keour
do
2000 yoa.m
12
2012 oo
201 os
2011
2120111
501-12
20040
2012 0s
2100000
200007
2002 e
202-00
2022-12
2004
202400
2000 es
2103
3103477
200400
500 ok
203300 10
2053 32
105441
Month
Forecasting Market Flitered: Monthly Resolved Questions
Question Category
Question Category
Arts Recreation
Education Research
420
Environment Energy
agy
Pulkits & Governence Other
Security & Defense
220
Sciences
Sports
10000
100
16:000
10:000
10-000
10:000
41.00
1e:000
91000
26 ztte
11 0000
11000
20000
Monith
Figure 22, Distribution shift of the filtered forecasting market dataset.
24

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
You are an expert in extracting summary and keypoint from articles.
Rules
Provide a comprehensive summary the entire article in one paragraph, ensuring that all essential aspects are addressed. Your summary should include key statistics, notable dates, and any significant statements to fully convey the context and content of the news story.
2. Please provide keypoint that summarizes the event from the article with the following rules:
Focus specifically on events are newly occurring on publication date the article. the article does not introduce a new event but instead discusses ongoing topics or about non-news content like advertisements, state 'No new event reported.'
The point should be concise, accurate and complete, especially for numbers, names and dates.
Basically NO "he, she, they, them, etc" are allowed. Please clearly write out the entity you referencing in the point.
You not allowed to start with any of the phrases: the article discusses, the article shows, the article emphasizes, the article discusses, the speaker says, the speaker discusses, the author mentions, etc.
Examples
Here are several examples of extracting keypoints from articles. Note that the articles in different examples are irrelevant.
Example
Article: Professional golfer Lexi Thompson has announced her retirement from professional golf at the end of the 2024 season at the age of 29. Thompson, an 11 11-tmme LPGA Tour champion, made the announcement ahead of her 18th consecutive US Women's Open appearance. She turned professional in 2010 and won her first major the 2014 Kraft Nabisco Championship. Despite enduring injuries that led to drop in her world ranking, Thompson continued compete high level. her retirement announcement, Thompson expressed gratitude for the opportunities golf provided her highlighted her excitement for the next chapter her life. Publishing Date: 2024-05-29, Wednesday
Keypoint: Lexi Thompson announced her retirement from professional golf at the end of the 2024 season at the age of 29.
## Example
Article: India's capital territory of Delhi sweltered to its highest-ever temperature of 49.9 degrees Celsius (121.8 degrees Fahrenheit) on Tuesday, an oppressive heat wave forced authoritics to impose water rationing.
The Indian Metcorological Department (IMD) said the record measured in the suburb of Mungeshpur, surpass- ing Delhi's 's previous high of 49.2 degrees Celsius (120.5 degrees Fahrenheit), observed May 2022.
Publishing Date: 2024-05-29, Wednesday
Keypoint: Delhi experienced its highest-ever temperature of 49,9 degrees Celsius on 2024-05-28, Tuesday.
New Article
Based on the provided rules and examples, please summarize article and identify one key point that concludes new event on the publishing date.
Article: O
Publishing Date:
Output: Output should follow the format of
Summary:
Keypoint:
Figure 23, Prompt in the Article Summary step, adapted from Zhang et al. (2024).
25

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
[Setup:] Today {}. You will be testing people who are from the past, i... week or month ago, recently published article will serve as the basis for your questioning. Your objective is to ask them questions that assess the accuracy and plausibility of their predictions about events.
You will write question-answer pairs:
1. The question should challenge the person to predict an outcome or development related to the article's 's content as if they were being asked one week or one month ago. Please provide the question that can be answered on { but only guessable not answerable before ()-
2. The answer MUST be based on factual information from the article. Ensure that the answers do predict outcomes that have not been explicitly stated the article.
[Rules:] Article: 0-
Publishing date: 0
Please generate four questions about the above article, along with answers. You should follow the instructions below:
1. Please turn the key point "{) "{}" into the question, with focusing whether the event will happen.
2. The question should NOT be designed for reading comprehension. Please focus what happened rather than the implications after event.
3. The question MUST be in future tense.
Start the first question with "Will", with the answer "Yes".
Start the second question with "Will", with the answer as "No".
Start the third and fourth questions with phrase like "What will". "Who will", "Where will", "Which xxx will", "How much
will", or "How many will".
4. There must be time element in the question. It can be phrases like "In O ...", "By { ...," in f
5. You MUST NOT unclear implicit time element phrases like "in the future" or "in the upcoming weeks".
You should avoid: questions that require numerical reasoning; questions that require substantial world knowledge.
7. The answer MUST be short and concise, avoiding using redundant words or repeating the information in the question.
8. The question must be grammatically correct and contain the information required to answer. NO "he, she, they, it, them, etc"
allowed. Please clearly write out the entity you referencing in the question.
9. The question MUST be able be answered by the article.
The question MUST NOT include the information that came out just now. should be understandable people from the past. Avoid using "How will" or "Why will" questions, as they imply that the event has already occurred.
[Suggested questions and questions to avoid are detailed below:]
Keypoint: Delhi experienced highest-ever temperature of 49.9 degrees Celsius on Tuesday, leading water ra-
tioning due to the oppressive heat wave.
Suggested Question: Will Delhi break the highest temperature record again by May 2024?
Avoid This Question: Will extreme heat events continue to pose threat to India's 's development in the upcoming years?
Reason Avoid: The time constraint "in the upcoming years" vague and question can answered based today's knowledge.
Figure 24, Prompt in the QA Generation step (part 1), adapted from Zhang al. (2024).
26

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Keypoint: Owners of nearly 84,000 older Nissan vehicles the United States equipped with recalled, unrepaired Takata air bags, including models such as the 2002-2006 Nissan Sentra, are advised by NHTSA to immediately stop driving them due to safety concerns.
Suggested Question: Will older Nissan vehicles such the 2002-2006 2006 Nissan Sentra exhibit quality issues by May 2024? Avoid This Question: Will owners of 2002-2006 Nissan Sentra, 2002-2004 Nissan Pathfinder, and 2002-2003 Infiniti QX4 heed the NHTSA's advice to immediately stop driving their vehicles in late May 2024?
Reason to Avoid: This question is overly specific. People from the past would not have known the "NHTSA's advice".
Keypoint: Children's sketches of violent scenes, likely made by children aged before eruption of Mt. in 79 AD, have been uncovered at the archaeological park of Pompeii.
Vesuvius
Suggested Question: Will children's 's sketches of violent scenes be discovered at the archacological park of Pompeii by May 2024? Avoid This Question: Will the newly discovered children's ' sketches the archacological park of Pompeii be available for public
viewing by May 2024?
Reason Avoid: This question includes future events about newly discovered children's sketches Pompeii, which wouldn't it
known to a past audience.
Keypoint: North Korea has been sending "filth and garbage" across the border to South Korea using giant balloons strategy, prompting South Korean authorities to warn of the objects landing residential areas. The move, according to North
Korean state media KCNA, was to retaliate against South Korean activists who often send materials to the North. Suggested Question: What will North Korea do to retaliate against South Korean activists who often send materials to the North by May 2024?
Avoid This Question: Will North Korea continue using balloons send items the border South Korea by May 2024?
Reason Avoid: The word "continue" should not be used here. The question MUST NOT include the information that came out just now.
[Output:] Now please write four clear and concise question-answer pairs following the instructions and examples above. Once again the question should NOT designed for reading comprehension but of forecasting interests. Also, vague and implicit time elements like "in the future", "in the upcoming weeks" or "in the coming years" should NOT used. The question should be able to answer on (h. but only guessable not answerable before {}. You should output the question along with its answer, in the format of
Question I: "Will xxx?"
Answer 1: Yes.
Question 2: "Will xxx?"
Answer No.
Question 3: Either "What will xxx?", "Who will xxx?", "'Where will xxx?", "Which xxx will", "How much will XXX?", "How many will xxx?" Answer xxx.
Question 4: Either "What will xxx?", "Who will xxx?", "Where will xxx?", "Which xxx will", "How much will xxx?", or "How many will xxx?" Answer 4: xxx.
Figure 25. Prompt in the QA Generation step (part 2), adapted from Zhang al. (2024).
27

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Rules
Article: 0
Given the article, please generate three noising answers to the given questions, whose correct answers can be obtained from the article. Name the three noising answers as (b), (c) and (d) respectively. While (b), (c) and (d) should all be unambiguously incorrect, they should also make sense and be plausible.
Examples
Here are examples showing the output format. This example is NOT related to the noising answers you will generate.
Question: What will be the annual change in the UK's Consumer Prices Index (CPI) for November 2021?
Correct Answer: 'Less than 1.7%
Noising Answers:
(b) Between 1.7% and 2.2%, inclusive'
(c) 'More than 2.2% but less than 2.9%
2.9% more'
Question: Who will win the 2020 Georgia Democratic primary? Correct Answer: 'Joe Biden'
Noising Answers:
(b) 'Michael Bloomberg"
(c) Pete Buttigieg"
(d) "Someone else'
Question: Before July 2020, will it be officially announced that the Tokyo 2020 Summer Olympics and/or Paralympics will be postponed, canceled, and/or relocated?
Correct Answer: Yes, the Olympic Games only
Noising Answers:
(b) "Yss, the Paralympic Games only'
(c) Yes, both'
(d) 'No'
Input: Question 1: 0
Correct Answer 1: 0
Question 2: 0
Correct Answer 2:
Output: Now please generate three noising answers to the question, given the above article, instructions examples. DO NOT output the backgrounds, the question or any other explanations. Noising Answers 1:
(b)
(c) XXx.
(d) XXX.
Noising Answers 2:
(b)
(c) xxx.
(d) xXx.
Figure 26. Prompt in the Misleading Choices Generation step, adapted from Zhang et (2024).
28

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Task
Please help evaluate the quality of question-answer pairs derived from the given article. The questions will be presented to someone who not seen the corresponding news article, in order to evaluate accuracy and plausibility the event prediction ability.
Inputs
Article: 0 Publishing Date: 0
Question 1: 0
Answer 1: {}
Question 2: (}
Answer (
Question 3: (}
Answer 3: {}
Question 4:
Answer {
Scoring Categories
## Correctness: Given the above article, please check if the answer is correct to the question with 100% certainty. There is evidence in the article that the answer correct with 100% certainty.
-2 points:
point: The answer generally aligns with the news facts but minor inaccuracies or missing details.
0 point: Significantly misaligned with the news facts.
Imagine traveling back in time date (()). At that time, you asked the question without having seen this specific article, but you do have news articles. The question should ideally only guessable- not definitively answerable- based on the information available at that time. That is, the answer should be able to be found in the given article, but should not be obtainable from earlier articles.
## Only Answerable Publishing Date:
week before the article's publishing
all carlier
Note that past tense descriptions in the article DO NOT INFL INFUUENCE this assessment.
-2 points: The question is answerable
but only guessable not answerable before ().
point: Could be somewhat predicted before O.. but not with complete certainty.
0 point: A person (could be anyone, even an expert in the field) would be able to find an article (or many) published before { that answers the question with 100% certainty.
### point examples
Example I:
Question: What will be one of Lexi Thompson's career highlights in professional golf? Answer: Winning 11 LPGA Tour titles.
Reasoning: This question is answerable with prior knowledge and does not test predictive ability related the publishing date.
## No New Information:
Ensure question does not include new information that only became known on the publish- ing date, making it understandable for past audience.
2 points: No new information from the publishing date are included.
Minor information from the publishing date might be inferred but not explicitly stated. point: Includes clear new information from the publishing date, unsuitable for past understanding.
point:
### 0 point examples
Example 1:
Question: Will owners of the 2002-2006 Nissan Sentra, 2002-2004 Nissan Pathfinder, and 2002-2003 Infiniti QX4 heed the NHTSA's advice to immediately stop driving their vehicles in late May 2024? Reasoning: This question contains new information on the publishing date. People from the past would not have known the "NHTSA's advice".
Figure 27. Prompt in the QA Filtering step (part 1).
29

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Example 2:
Question: "What will Lexi Thompson's 's ranking be at the time of her retirement announcement in May 2024?"
Reasoning: This question contains the information that Lexi will annouce her retirement, which is not known to the people from the past.
Example 3:
Question: "Will the newly discovered children's 's sketches at the archaeological park of Pompeii be available for public viewing by May 20247"
Reasoning: This question includes future events about newly discovered children's sketches Pompeii, which wouldn't be known to past audience
## Objectiveness:
The answer should not rely more the author's 's personal views than objective facts. Completely objective, based strictly reported facts.
-2 points:
point:
Primarily objective. with minor subjective interpretations. Largely subjective or opinion-based lacking a factual basis.
0 point:
## Clear Time Element: This category checks if the question has clear element in it, without having vague phrases
like "in the future" "in upcoming weeks".
points: The question has clear time elements, like "by May 2024" or "in July 2023".
point: The question includes a general timeframe, like "next month" or "this winter," which allows for some estimation but lacks precise dates.
0 point: The question includes vague time phrases like "in the future" "in the upcoming weeks," which do not specify clear or precise timeframe.
### 0 point examples
Example 1:
Question: Will extreme heat events continue to pose threat India's development in the upcoming years? Reasoning: The time constraint "in the upcoming years" is vague.
Example 2:
Question: "What will Illinois require from parents who monetize their children's online activities starting in July?"
Reasoning: The mention "July" specifies only the month lacks the necessary detail of the year.
## Public Interest: Determine if the question addresses topic of public concem.
22 points: The question covers topic that widely affects or interests the public.
point:
The question of moderate interest, relevant to specific groups
point: The topic overly personal or localized, lacking relevance to broader public.
### 0 point examples
Example 1:
Question: Will the exhibition 'Fragile Beauty' at London's Victoria Albert Museum include both midcentury and contemporary works May 2024?
Reasoning: The specific details of an personal art exhibition's 's contents are generally of limited public interest.
## Answer Not Too Obvious: This category evaluates whether the to question is too predictable or straightfor-
ward based on the question itself.
2 points: The answer provides new or non-obvious insights, requiring additional context or understanding not explicit in the question.
1 point: The answer is somewhat predictable but includes minor additional information slight twist.
0 point: The answer directly restates closely mirrors the question, offering details insights.
Figure 28. Prompt in the QA Filtering step (part 2).
30

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
###l point examples
Example 1:
Question: What will New York officials do ensure safety ISIS-K?
Men's Cricket World Cup following global threats from
Answer: New York officials will implement increased safety precautions for the event.
The answer is straightforward and expected, directly restates the premise of the question without providing any new or specific details on how the safety precautions will be implemented or what they might entail.
Reasoning:
Instructions
Evaluate each question-anse answer pair by assigning points in each of the categories based on the criteria provided. Please be strict giving points. the requirements category are not fulfilled, assign a point
Please strictly follow this output template:
"Question |*
## Correctness
Reasoning:
Point:
## Only Answerable on Publishing Date
Reasoning:
Point:
## No New Information
Reasoning:
Point:
## Objectiveness
Reasoning:
Point:
## Clear Time Element
Reasoning:
Point:
## Public Interest
Reasoning:
Point:
## Answer Not Too Obvious
Reasoning:
Point:
"Question 2*
## Correctness
Reasoning:
Point:
## Only Answerable on Publishing Date
Reasoning:
Point:
## No New Information
Reasoning: Point:
## Objectiveness
Reasoning:
Point:
## Clear Time Element
RReasoning:
Point:
Figure 29, Prompt in the QA Filtering step (part 3).
31

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
## Public Interest
Reasoning:
Point:
## Answer Not Too Obvious
Reasoning:
Point:
*Question
## Correctness
Reasoning:
Point:
#il Only Answerable on Publishing Date
Reasoning:
Point:
## No New Information
Reasoning:
Point:
## Objectiveness
Reasoning:
Point:
## Clear Time Element
Reasoning:
Point:
## Public Interest
Reasoning:
Point:
## Answer Not Too Obvious
Reasoning:
Point:
"Question
## Correctness
Reasoning:
Point:
Only Answerable on Publishing Date
Reasoning:
Point:
## No New Information
Reasoning:
Point:
## Objectiveness
Reasoning:
Point:
## Clear Time Element
Reasoning:
Point:
## Public Interest
Reasoning:
Point:
##l Answer Not Too Obvious
Reasoning:
Point:
Figure 30. Prompt in the QA Filtering step (part 4).
32

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
System Prompt: You're an expert forecasting events, You will NEVER refuse to answer forecasting question by saying "I cannot predict the future", even if without 100% certainty.
User Prompt: You should output your answer either "Yes' or 'No' WITHOUT anything clse.
Question: {
Choices: "Yes' or 'No'
[Output:] Your answer:
Figure 31. Closed book evaluation prompt for TF questions, adapted from Zhang al. (2024).
System Prompt:
You're an expert forecasting events. You will NEVER refuse to answer forecasting question by saying "I cannot predict the future"", even if without 100% certainty.
User Prompt: You should output your answer either '(a)', 'b)', '(e) '(d) WITHOUT anything else.
Question:
O
Choices:
(a) [}
(c)
(d) +
[Output:] Your answer:
Figure 32, Closed-book evaluation prompt for MC questions, adapted from Zhang et al. (2024).
System Prompt:
You're expert forecasting events. You will NEVER refuse to forecasting question by saying "I cannot predict the future", even if without 100% certainty.
User Prompt: You should output your answer as either "Yes" or 'No' WITHOUT anything else. Below are the top 5 relevant news article fragments retrieved for the question, which may may not assist you in making forecast.
Article I: {
Article ()
Article 3: [}
Article 4: []
Article 5: {
Question: {}
Choices: "Yes' or 'No'
[Output:] Your answer:
Figure 33, Constrained open-book evaluation prompt for TF questions, adapted from Zhang al. (2024).
33

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
System Prompt:
You're expert forecasting events. You will NEVER refuse answer forecasting question by saying "I cannot predict the future"" even if without 100% certainty.
You should output your answer as either "(a),, '(b),, "(c) or '(d) WITHOUT anything clsc. Below are the top 5 relevant news article fragments retrieved for the question, which may may not assist you in making forecast.
User Prompt:
Article {
Article
Article 3: [
Article 4: (
Article 5: 0
Question: {
Choices:
(a) 0
(c)
(d) ()
[Output:] Your answer:
Figure 34. Constrained open-book evaluation prompt for MC questions, adapted from Zhang et al. (2024).
System Prompt:
You're an expert in forecasting events. You will NEVER refuse to answer forecasting question by saying "I cannol predict the future", even if without 100% certainty.
User Prompt: You should output your answer as either or WITHOUT anything else. Below the updated news article relevant to the question, which may help you in providing an answer.
Article: {
Question:
Choices: 'Yes' or
[Output:] Your answer:
Figure 35, Gold article evaluation prompt for TF questions, adapted from Zhang et al. (2024).
System Prompt:
You're an expert forecasting events. You will NEVER refuse to answer forecasting question by saying "I cannot predict the future", even if without 100% certainty.
User Prompt:
You should output your answer either '(a),, '(b)', '(c)" '(d) WITHOUT anything else. Below is updated news article relevant the question, which may help you providing answer.
Article:
0
Question: 0
Choices:
(a) [}
(b)
(c)
(d)
[Output:] Your answer:
Figure 36. Gold article evaluation prompt for questions, adapted Zhang et al. (2024).
34

---

Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
Question: ()
Options:
Science & Tech
Hcalthcare & Biology
Economics & Business
Environment & Energy
Polities Governance
Education Research
Arts & Recreation
Security Defense
Soccil Sciences
Sports
Other
Instruction: Assign category for the given question.
Rules:
Make sure you only return one the options the option list.
2. Only output the category, and do not output any other words in your response.
3. You have to pick string from the above categories.
Answer:
Figure 37, Prompt to categorize the generated questions, taken from Halawi et al. (2024).
35


Reviews:
[{"review_id": "czDz8F2Piu", "reviewer": "Reviewer_2v6q", "summary": "The authors propose a method of constructing a continuous temporal knowledge & temporal prediction efficacy benchmark for LLMs. They show results of an implementation of the benchmark, and they describe the release of the benchmark for public use.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our work!", "claims_and_evidence": "Yes, to the best of my knowledge.", "methods_and_evaluation_criteria": "The proposed methods are quite suitable for the problem at hand. The authors describe a convincingly-comprehensive approach to the problem of constructing an automatically-updating continuous benchmark, including:\n\n1. Sourcing from a reputable and established corpus of news on the web (common crawl)\n2. Robust filtering procedures\n3. Methods to identify truly \"current\" news rather than opinion pieces discussing past news\n4. Methods to extract both multiple-choice and true/false QA questions from current news documents", "theoretical_claims": "No theoretical claims were made", "experimental_designs_or_analyses": "Yes, the authors' submission depends critically on experimental design, specifically in their experiment setup for generating insights from their benchmark. The authors expose interesting aspects of their benchmark by breaking down the task scenario into three settings:\n\n1. Closed-book QA: the LLMs are challenged to answer the temporal QA questions without retrieved documents, i.e. predict the future form their own internal knowledge\n2. Constrained open-book QA: the LLMs get access to RAG subject to a retrieval cutoff date\n3. Gold open-book QA: the LLMs get access to the exact documents that contain the answer to the question\n\nThe authors also carefully track moving averages across the analyzed timespan subject to the various knowledge cutoffs. This allows the authors' study to expose sources of the models' (in)accuracies, e.g. degradation of entity representations over time (parametric knowledge) vs the ability to reason about current information in the context (RAG documents and cutoff).", "relation_to_broader_scientific_literature": "The authors' contribution is related to the temporal QA and forecasting areas. Largely, the paper introduces an impactful benchmark to these spaces. It is the first continuously-updating, daily LLM forecasting benchmark, as shown in Table 1.", "essential_references_not_discussed": "I did not think of any essential references that were missing.", "other_strengths_and_weaknesses": "I think it's a great benchmark and experimental paper", "other_comments_or_suggestions": "No other comments/suggestions", "questions_for_authors": "No other questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gBpx73CHlW", "reviewer": "Reviewer_xifq", "summary": "This paper proposes a benchmark dataset for assessing a model’s generalization ability in predicting future events and analyzes how model performance evolves over time. Specifically, it compares model performance under three conditions: no access to external information, access to retrieved recent news articles, and access to gold articles. The experimental results indicate that LLMs' prediction accuracy exhibits a significant, gradual decline over time.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback, and hope to address the concerns below:\n***\n### Sample size of human evaluation\nWhile the evaluation involved 60 questions, we respectfully note that **this sample size aligns with standard practices in similar dataset validation studies**. For example, *TLB-forecast* conducted human evaluations using 24 samples, *SituatedQA* utilized 100 samples for assessing human performance, and *FreshQA* similarly performed human evaluations on 100 samples. To further mitigate bias, we had 4 annotators per question and conducted inter-annotator agreement analyses, as shown in Appendix A.4.\n\n***\n### Distribution of question categories and question types\nWe provide the distributions of question categories and question types over time in [Fig.S1](https://imgur.com/a/GdqLtRq). Additionally, we conducted further analyses by balancing the categories monthly (selecting TF 5,520 out of 16,783; MC 4,680 out of 14,727) and further **balancing both categories and question types** within MC questions (2,400 out of 5,520). The performance **degradation pattern consistently persists**, showing that degradation primarily arises from increasing temporal distance rather than shifts in category or question distributions. For difficulty-level analysis, while it would be possible to assign difficulty using language models, we believe such automated measures would introduce unnecessary noise. We would rather see the observed model performance itself effectively reflects the inherent difficulty of the questions.\n***\n### Evaluation of mid-size LLMs\nWe selected models based on recent popular choices, covering both open-source and closed-source options. We appreciate the reviewer’s suggestion regarding mid-sized LLMs. Accordingly, we've included **Llama-2-13B and Qwen-2.5-14B**, and provided an updated plot ([Fig.S2](https://imgur.com/a/BnKiI6t)) and table. As an earlier-generation model, Llama-2-13B underperforms compared to Llama-3-8B, though it still demonstrates higher performance before the knowledge cutoff than after. For Qwen-2.5-14B, we observe relatively strong performance on MC questions but near-random accuracy (~50%) on TF questions. Interestingly, we note that Qwen-2.5-14B exhibits a strong bias towards responding \"No,\" selecting this answer 91.66% of the time on TF questions.\n\n*Table: Yearly Accuracy and YoY Accuracy Change for Llama-2-13B and Qwen-2.5-14B*\n|||K-Cutoff|2020|2021|2022|2023|2024|Pre-Cutoff YoY Change|Post-Cutoff YoY Change |Avg YoY Change|\n|--|-|-|-|-|--|---|--|--|--|--|\n|TF|Llama-2-13B|Sept 2022|56.80|58.59|54.29|51.95|52.65|-0.79%|-8.52%|-1.75%|\n|TF|Qwen-2.5-14B|Unknown|54.02|52.48|52.11|51.74|51.36|-0.99%|-|-0.99%|\n|MC|Llama-2-13B|Sept 2022|42.24|42.31|39.35|37.53|38.74|-1.37%|-12.22%|-1.53%|\n|MC|Qwen-2.5-14B|Unknown|56.54|59.13|56.59|54.60|52.85|-1.38%|-|-1.38%|\n\n***\n### Contribution to ML research\nWe believe much progress in ML research benefits from open benchmarks. To name a few, ImageNet, GLUE, SQuAD, …. Specifically, our benchmark provides contributions in two important research directions:\\\n**(1) LLM forecasting:** Our large-scale, daily-updated dataset reflecting real-world events enables the training and evaluation of models to better support human decision-making.\\\n**(2) Continual learning:** Our dataset highlights the challenge of maintaining up-to-date knowledge in LLMs. Our analysis demonstrates that even when provided with gold articles, performance degradation persists, emphasizing the necessity of continuous pre-training to mitigate outdated representations. With Daily Oracle, one could explore how continuous pre-training and efficient adaptation can address the performance degradation challenges presented in our work.\n***\n### Would CoT prompting help?\nWe randomly sample 25 MC questions per month (1,500 in total) to study how CoT impacts the performance (See [Fig.S3](https://imgur.com/a/0Z9goAA)). For LLaMA-3-70B, we prompt the model to explicitly generate a rationale before providing the final answer. Compared to directly answering, we observe a slight performance improvement initially, though this advantage diminishes after 2023. Additionally, for DeepSeek-R1-Distill-Llama-8B, we utilize the original prompt without modification; however, since this model is fine-tuned to naturally generate reasoning prior to the final answer, we treat its outputs as CoT results. **Its performance similarly matches that of the non-CoT approach.**\\\n\\\nThis observation aligns with recent findings [4], indicating that CoT reasoning primarily benefits tasks involving mathematical, logical, or algorithmic reasoning, with limited gains on other task types. While improved CoT prompt engineering might yield better performance, the results presented here provide baseline insights, leaving further optimization for future research.\\\n\\\n[4] Sprague, Zayne, et al. \"To cot or not to cot? chain-of-thought helps mainly on math and symbolic reasoning.\" arXiv preprint arXiv:2409.12183 (2024).", "claims_and_evidence": "1. The paper claims that they conducted a human evaluation to assess the quality of the constructed dataset. However, the evaluation consists of only 60 questions, which may introduce bias into the assessment.\n2. In Table 3, the conclusion that model performance degrades over time due to increasing temporal distance from pretraining is not strongly supported. The study does not address whether the distribution of article types, question types, and difficulty levels remains consistent across different years.", "methods_and_evaluation_criteria": "1. Human evaluation in section 3.3 is insufficient, making it difficult to ensure dataset quality.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. How should LLMs of different sizes be selected for the experiments in Table 3? The study experiments with models ranging from 2.7B to 7.8B and 56B but seems to lack mid-sized LLMs, such as LLaMA 2-13B or Falcon-40B.", "relation_to_broader_scientific_literature": "The paper provides a benchmark that allows continuous updates for evaluating models'\ngeneralization in future event prediction.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "### Strengths\n1. The proposed benchmark can be continuously updated, ensuring long-term usability.\n\n### Weaknesses\n1. Constructing a benchmark dataset could be interesting and valuable to the ML community, but it alone is not a sufficient major contribution to ML research, especially given that the quality of the benchmark data is verified with limited human annotation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Would incorporating Chain-of-Thought reasoning improve model performance in future event prediction?\n2. How does the distribution of question types and difficulty levels vary across different years? Could this affect the observed performance trends?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CVg1WjwSip", "reviewer": "Reviewer_s4SM", "summary": "The paper uses the task of forecasting real-world events to demonstrate that LLM knowledge deteriorates on more recent questions, and this trend also holds for retrieval. It generates these forecasting questions between January 2020 and December 2024 using LLMs, sourcing information from news articles.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed feedback.\n***\n### Brier score\nWhile we agree that Brier score is valuable to account for uncertainty in binary predictions, we clarify that accuracy remains a valid metric in this setting, revealing a clear performance degradation trend in our experiments. We provide Brier score result in [Fig.S4](https://imgur.com/a/fjtEONv). The increasing trend further confirms the previously identified trend of performance degradation. We also note that Brier score can be sensitive to calibration issues of LLMs, which is an important but orthogonal direction that we leave for future work.\n***\n### Quality of dataset\nWe refer the reviewer to Section 3.1 & Appendix A.3 for our quality control process. During the QA filtering step, 7 principles are identified based on common mistakes observed during manual reviews while testing various QA generation prompts, and overall 18.11% TF questions and 24.20% MC questions were filtered out. Further validation, presented in Section 3.3 and Appendix A.4, includes a human evaluation of the filtering process. We find strong agreement between human reviewers and LLM-assigned scores, with an average accuracy of 89.52% across the 7 principles.\n\nWe are aware of examples like those the reviewer mentioned and designed our filtering criteria such as “Answerability” to address such issues. While some imperfections are inevitable in any LLM-generated dataset, we note that for final QA pair acceptance, the agreement between LLM and human evaluations achieved 85% accuracy, indicating that the majority of the retained questions are valid and of acceptable quality. Moreover, if the questions were broadly unanswerable, we would not observe clear differences in LLM accuracy across models or the consistent degradation trends over time.\n***\n### Comparison with forecasting markets\nWe would like to explain why we chose to focus on LLM-generated questions in this work:\n1. [Prior work](https://arxiv.org/abs/2402.18563) collected data from 5 existing forecasting platforms, sourcing 48,754 raw questions (2015-2024). They note that many questions in the raw dataset were unsuitable, resulting in a much smaller filtered dataset of 5,516 questions compared to our 31,510. Also, in Figure 11, existing platforms offer limited coverage in earlier years (<300 questions per quarter before Q4 2021), making longitudinal analysis difficult. In contrast, our method supports high scalability and retrospective generation, allowing for uniform coverage across the full time range.\n2. [Concurrent work](https://arxiv.org/abs/2405.08460) collected 2,532 questions from GoodJudgmentOpen to study temporal generalization. However, the trend is difficult to discern due to the limited number of data points (ranging from 2 to 8) for each model (Table 3). In contrast, while their bi-monthly accuracy results exhibit significant fluctuations, our dataset presents a clearer trend of monthly accuracy degradation, providing deeper insights into how LLM performance evolves over time.\n\nFinally, while we do not claim that LLM-generated questions are of inherently higher quality, we believe our automatic QA generation approach offers several key advantages.  If one sources questions from forecasting markets, the dataset update frequency is dependent on whether there are active users. In contrast, our approach enables daily updates, scalability, and more comprehensive event coverage, making it a valuable complement to human-curated forecasting benchmarks.\n***\n### RAG results with one-year retrieval window\nWe conduct the suggested experiment with a fixed one-year retrieval window on a subset of 1,500 TF questions (randomly selected 25 questions for each month). As shown in [Fig.S5](https://imgur.com/a/sB7jb1D), our key findings remain consistent: while models can benefit from retrieving more updated articles (RAG cutoff 2024-03), a degradation trend still persists over time. This reinforces the broader conclusion that temporal distance from pretraining continues to impact performance, even with external knowledge augmentation.\n***\n### Choice of models in QA generation\nOur initial version of the dataset was generated before GPT-4o became available. We compared question generation across GPT-3.5 & GPT-4 and the newer GPT-4o & GPT-4o-mini models using the same set of articles. Each model produced 48 TF and 48 MC questions, manually evaluated using the same seven QA filtering criteria. Newer models outperformed the older ones with a 54.55% win rate, highlighting their potential to improve question quality in future dataset iterations.\n***\n### Others\n1. Complete RAG results can be found in Appendix B.5.\n2. Number of articles available for retrieval - [Fig.S6](https://imgur.com/a/qsYFhLT)\n3. Paleka (2025) introduces a forecasting dataset to test LLMs’ logical coherence. While “generate-then-verify” via LLMs as a common approach, our main contribution lies in capturing and quantifying degradation patterns.", "claims_and_evidence": "Yes, the claims made in the paper are focused, and supported with evidence. However, there are concerns with the experiment design, stated below.", "methods_and_evaluation_criteria": "Please report brier score / log-odds for all plots. Accuracy is a bad metric for forecasting as one can realistically never be sure whether the event will occur or not (inherent uncertainty).\n\nThe key uncertainty about this paper lies in the quality of the LLM generated forecasting questions dataset. The paper does not discuss how it evaluates this quality. Further, GPT-3.5 is used to create some questions, and it is unclear why, when GPT4o/4omini are both cheaper and much more reliable models. I encourage the authors to think carefully about how the quality of questions can be measured, and also re-generate questions before September 2024 with GPT4o if this improves quality compared to GPT-3.5. This would make the data/benchmark much more usable for future work. For example, a quick look at the data provided in the supplementary led me to find obviously faulty questions. Eg:\n\na) \"What career strategy will be recommended in January 2020, traditional ladder climbing or pursuing innovative and entrepreneurial approaches?\" -- recommended by whom? I don't think this question has any single answer.\n\nb) Which aspect of social media strategy will most brands and individual thought leaders overlook in January 2020? - Same issue as a). \n\nOne way of measuring question quality could be checking the brier score of the same models on questions beyond the cutoff-date, comparing different sources like Metaculus, Manifold, Polymarket as baselines. In fact, I am unsure why these other data sources are not already included in the analysis of this paper, as they do provide many thousands of questions and its unclear whether LLM generated questions provide any better quality.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The broad experimental design of varying model performance across time (including RAG cutoffs) makes sense and is interesting. However there are a few major issues in the details which could be confounders for the results:\n\n1) The performance with RAG greatly depends on the number of recent articles in the index, which is being varied across time. Could you please present temporal plots with the number of articles published in a fixed time window, say the last year from each date, that are retrievable from the index?\n\n2) Back-generating questions using news might lead to the creation of a biased question set based on events that are reported in the news. Why not use questions on platforms like Polymarket for this? Can you provide an empirical comparison to what happens if the same analysis was done using questions obtained from Polymarket's API?", "relation_to_broader_scientific_literature": "There have been previous papers on the potential of forecasting as a language model benchmark. Backtesting and lookahead bias (a form of contamination) are extremely important for forecasting in other domains (such as stock market). This paper contributes towards better backtesting for language model forecasting.", "essential_references_not_discussed": "Consistency Checks for Language Model Forecasters, Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, Florian Tramèr, ICLR 2025 -- Also uses a sophisticated LLM generated forecasting questions pipeline. Please compare with this.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The use of forecasting as a task to test model performance across time is interesting.\n\n2. Generating forecasting questions using News and LLMs is a clever insight. \n\n**Weaknesses**: \n1. Question quality is not evaluated, and if the data is bad all the results could be unreliable.\n2. The results/trends are confounded by the number of fresh articles in the retrieval index across time, which is not reported.", "other_comments_or_suggestions": "Please add methodology to measure question quality to validate design choices, and address questions below.", "questions_for_authors": "1. Why is model performance increasing after July 2024 in the gold article setting?\n\n2. Why use Mixtral 7b for Figure 4 (left) when its performance is almost worse than random after a certain point. Why not report the same Llama 3 8b model.\n\n3. Could you add more details about the retrieval, such as the number of articles available at each month in the relevant period using your scraping pipeline? Would more sophisticated retrieval than BM25 lead to improved results for models?\n\n4. Does question quality improve by using a better model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "whRUrqtJTe", "reviewer": "Reviewer_dDkL", "summary": "The manuscript introduces Daily Oracle, a benchmark dataset composed of automatically-generated question-answer pairs concerning daily news over a 4 year period. The questions are all phrased in a \"forecast\" manner (e.g., \"Will X happen?\", \"What will Y be on DD-MM-YY?\") and are either yes/no or multiple (4) choices questions.\n\nDifferent language models are evaluated on this benchmark in one of three regimes: closed-book (no context), constrained open-book (RAG on news articles up to a certain cutoff date), and gold article (the article used to generate the question-answer pair is explicitly provided to the model). The models are good at predicting the answers in the gold article setting. In the two other settings, the models are better at predicting the sought answer at early time than at latter ones, with more pronounced decrease around the data cutoff point. Part of this decrease has been identified as the model refusing to predict the future.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions, which we will incorporate into the future version of our manuscript.\n***\n### Concerns about refusal rate\nWe appreciate the reviewer’s thoughtful comments regarding the refusal cases of Mistral and Mixtral models. We would like to clarify the following points: as seen in Figure 8, **only the Mistral and Mixtral models exhibit notable refusal rates**, with approximately 10–30% on TF questions and 1.5–8% on MC questions. In comparison, Qwen-2-7B and Gemma-2-2B show relatively low refusal rates—<5% for TF and <2% for MC—while **all other models have near-zero refusal rates for TF and <1% for MC**. We count refusal cases as incorrect both to maintain comparability across models and because failing to provide an answer—when a prediction is expected—represents an unsatisfactory outcome from the user's perspective.\n\nFurther,\n1. We provide an additional plot [Fig.S7](https://imgur.com/a/ZTpSKyj) that excludes questions the models refused to answer, focusing only on cases where the model provided a definite answer. While this means the results among models are no longer directly comparable, the performance degradation trend remains evident, particularly for TF questions in Mixtral-8x7B. We hope this analysis helps address the reviewer’s concern regarding the impact of refusal behavior on the observed trends. We would be happy to include this plot in a future version of the paper to help clarify concerns related to refusal cases.\n\n2. Our claim of `\"their ability to forecast future events deteriorates much more rapidly as they move beyond their training data, struggling with temporal generalization\"` specifically contrasts pre- and post-knowledge-cutoff performance for models with known cutoff dates. The Mistral and Mixtral models mentioned by the reviewer do not disclose such information, and therefore are not used to support the claim here. For the models with known knowledge cutoff (Claude, GPT, LLaMA, and Gemma), refusal rates are minimal—most of them having nearly 0% for TF questions and less than 1% for MC questions—and thus have negligible impact on the observed degradation patterns.\n\n3. **Effect of rewording questions:** We thank the reviewer for suggesting the rewording experiment, and tested the impact of softer phrasing (e.g., rephrasing “Will...” to “Would it be likely...”) using Qwen-2-7B and Mistral-7B on TF questions. As shown in [Fig.S8](https://imgur.com/a/NMy925N), this change reduced refusal rates by approximately 10%. However, the overall performance degradation trend still persists, suggesting that refusal alone does not account for the observed decline.\n\n4. **On models `“not being trained to be oracles”`:** We believe that model refusals alone do not explain the observed performance decline. As shown in Fig.S7, even when refusal cases are excluded and only answered questions are considered, the downward trend in performance persists.\n\n5. **On reluctance vs. performance:** While Claude demonstrates the lowest refusal rate on TF questions (Figure 8), and high performance (Figure 3), this pattern does not generalize across models. GPT-3.5, GPT-4, and LLaMA-3-8B also have near-zero refusal rates, yet show varying levels of forecasting accuracy. Thus, we do not find strong evidence that lower reluctance to predict directly correlates with better performance.\n\nOverall, our findings indicate that performance decline is a general phenomenon that persists even when refusals are removed from the analysis.\n***\n### Others\n1. For concerns about the number of human evaluation samples, please refer to our response to Reviewer xifq (\"Sample size of human evaluation\"). Regarding the shift of newer 4o models since September 2024, please refer to our response to Reviewer xifq (\"Distribution of question categories and question types\") and Reviewer s4SM (\"Choice of models in QA generation\").\n2. Background of human annotators: The 4 annotators are graduate students from the authors’ institution, majoring in finance, accounting, statistics, and data science, respectively. They were not involved in the research beyond their role in conducting the evaluations.", "claims_and_evidence": "### Claim 1: Introducing a continuous forecasting evaluation benchmark\n\n> We present Daily Oracle, the largest and most up-to-date forecasting dataset\n\nFrom Table 1, this dataset is the only daily one, it is one of 6 forecasting datasets (which, as I understand it, means that the questions are worded as a future-prediction task), it is the second largest (and the largest dates back to 2021). I deem this claim supported.\n\n\n### Claim 2: Empirical Findings on Performance Degradation\n\n> Our work effectively reveals a clear performance degradation pattern in LLMs’ forecasting accuracy over time.\n\nThe statement as worded appears factual. However, I see issues with the general narrative, as well as with more granular claims made thorough the manuscript. This is my main issue with the manuscript in its current state, which I try to convey through comments on some excerpts (emphasis mine).\n\n> Additionally, despite prompting the models to avoid **responses like “I cannot predict the future”** and instead provide definitive answers, there are cases where such refusals still occur. The rejection rates are provided in the Appendix B.3, and **these cases are counted as incorrect** to ensure comparability across model results.\n\nThese models were explicitly trained to say things like “I cannot predict the future” when asked to predict the future. The refusal rates shown in Figure 8 are quite substantial, and they behave like one would expect them to for models that were trained to refuse to predict the future.\n\n> However, post-knowledge cutoff, we observe steeper declines in many models, with GPT-4 showing the most drastic drop in MC performance, declining by 18.54%, compared to just 4.23% before the cutoff. This contrast highlights that while LLMs manage to retain a baseline of past knowledge with small degradation, **their ability to forecast future events deteriorates much more rapidly as they move beyond their training data, struggling with temporal generalization.**\n\n... or they just do as they were trained to do? We're talking about more than 50% variation for Mixtral-8x7B on MC questions, and about 15% variation for Mistral/Mixtral on TF. If you add these values to those in Figure 3 (i.e., if you reinterpret refusals as correct answers), the corresponding TF curves would be flat, and Mixtral's MC curve would go up. And Figure 8 shows \"lower bound\" effect, only reporting the refusals that were caught, and not accounting for the cognitive dissonance induced by a prompt saying the opposite of what the model was trained to do.\n\n> For Mixtral-8x7B, as the RAG cutoff dates extend to closer to the resolution dates, we observe a clear improvement in performance, indicating the model benefits from increasingly updated information retrieval. However, there are noticeable performance drops immediately after each RAG cutoff date when compared to providing information up to the day before the resolution date. **This highlights the importance of keeping up-to-date information for optimal RAG performance.**\n\nNo! This highlights that Mixtral refuses to answer when it doesn't have the information required to answer!\n\n> The overall decline trend may come from two sources, the missing knowledge of future and a lack of up-to-date language representation.\n\nOr these models were not trained to be oracles? I mean, even rewording the questions to become \"which is the most likely...\" assessments instead of \"will ...\" could potentially help. A more passive approach to study this phenomenon could be to assess if there is a difference in the wording of the questions that Mixtral *did* answer past the cutoff. Another data point: Claude is the least reluctant to predict the future (Figure 8) and the \"best\" at doing so (Figure 3).\n\nTo be clear, I do believe that there is something worth saying here, it is just that it may not be what is currently being said. This is the topic of my Question 1 below.", "methods_and_evaluation_criteria": "Excluding the subject of my Question 1 below, the way the language models are evaluated makes sense to me. The reported metrics in the main paper are all averaged over 5 months, but the Appendix provides some raw data. This was important for me to make my mind about potential issues with the September 2024 transition to GPT-4o for generating the dataset.\n\nThe manuscript's main point is to introduce a benchmark, so the real evaluation concerns the benchmark itself. Summary statistics are provided in Figure 1, and distribution stability is assessed through Figure 2. Human evaluation according to different criteria is provided in A.4. Only 60 questions where human-annotated, there were only 4 human annotators, and there is no mention of their sociocultural background nor of their potential affiliation/overlap with the authors.\n\nThe generation method of the benchmark makes sense to me overall, except that there is no explicit check to assess whether the switch to GPT-4o in September 2024 altered the distribution of questions and answers.", "theoretical_claims": "I didn't notice any particular theoretical claims.", "experimental_designs_or_analyses": "See my answer to Methods and Evaluation Criteria above.", "relation_to_broader_scientific_literature": "What is presented in the Related Work section makes sense to me, though I may be unaware of some missing related work.", "essential_references_not_discussed": ".", "other_strengths_and_weaknesses": "The benchmarking dataset itself is likely to be useful to the community.", "other_comments_or_suggestions": "Table 1: consider adding `\\citep{...}` after the dataset names. As a reader, I often use those as lookup tables.\n\nThe ordinate axes in Figure 8 appear mislabeled.\n\nMany news articles may be generated by LLMs toward the end of the period, whereas this likely wasn't as much the case back in 2020.\n\nWhen the model data cutoff date is unknown, consider showing the model's release date: the cutoff must precede it (with caveats for API-based models).\n\nConsider giving high-level description of the annotators. Things like their level of education, age range, etc., as well as their relations to the authors, if any, and if they were they paid.", "questions_for_authors": "### Question 1: What is your proposed solution to my issues mentioned in Claim 2 above?\n\nDo you agree with my observations and assessments? Are you open to contextualize, reword, and/or tone down these kind of claims? If yes, to what exactly?\n\nOr if you disagree, please let me know of your arguments and/or additional evidence.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Daily Oracle, a large-scale, daily updating benchmark sourced from real-world news for evaluating LLM temporal knowledge and forecasting capabilities over a long period (2020-2024). The work uses closed-book, RAG, and gold-article settings to dissect LLM performance, revealing clear degradation patterns over time, particularly concerning knowledge cutoffs.

The reviewers overall found the benchmark to be useful, distinguished by its daily generation of forecasting questions directly from news articles over a multi-year period, making it the first continuously updating daily resource of its kind for this task. They also appreciated the insights and analysis of LLM weaknesses. For instance, the benchmark helps differentiate between performance loss due to outdated internal parametric knowledge versus limitations in reasoning ability even when provided with current context (as seen in RAG settings). The persistence of degradation with RAG suggests issues beyond just missing facts, potentially involving outdated representations affecting reasoning.

The most prominent concern was about the quality and reliability of the LLM-generated forecasting questions, supported by reviewer examples of noise and ambiguity. The paper however does validate the approach on a small human evaluation sample (60 questions), which I feel is enough to get a feel for the data. Also, the interpretation of performance degradation, i.e. sharp drops post-knowledge cutoff was contested. The concern was if this degradation reflects a true failure of forecasting/temporal generalization or intended model behavior due to alignment training against speculation. Initial analyses also lacked controls for potential confounders (e.g., impact of varying retrieval index size in RAG, question characteristic shifts over time) and comparison to alternative data sources like forecasting markets. However much of these points were addressed during rebuttal and I hope the authors include some aspects of this discussions to improve the thoroughness of the paper for the next version.
Decision: Accept (poster)