Pitfalls in Evaluating Language Model Forecasters
Authors: Daniel Paleka
Shashwat Goel
Jonas Geiping
Florian Tramèr
Venue: ICLR 2026 Poster (2026)
Keywords: forecasting
evaluation
criticism
leakage
standards
LLMs
prediction
future
benchmarks
Primary area: foundation or frontier models, including LLMs
TL;DR: We find conceptual issues and real temporal leakage errors in existing LLM forecasting evaluations, and argue this is a problem.

Abstract: Large language models (LLMs) have recently been applied to forecasting tasks, with some works claiming these systems match or exceed human performance. In this paper, we argue that, as a community, we should be careful about such conclusions as evaluating LLM forecasters presents unique challenges. We identify two broad categories of issues: (1) difficulty in trusting evaluation results due to many forms of temporal leakage, and (2) difficulty in extrapolating from evaluation performance to real-world forecasting. Through systematic analysis and concrete examples from prior work, we demonstrate how evaluation flaws can raise concerns about current and future performance claims. We argue that more rigorous evaluation methodologies are needed to confidently assess the forecasting abilities of LLMs.

Published as a conference paper at ICLR 2026
PITFALLS IN EVALUATING LANGUAGE MODEL FORECASTERS
Daniel Paleka
Shashwat Goel
Jonas Geiping
Florian Tramèr'
'ETH Zurich EELLI Institute Tübingen
MPI Tübingen
ABSTRACT
Large language models (LLMs) have recently been applied to forecasting tasks, with some works claiming these systems match or exceed human performance. this paper, we argue that, as community, we should be careful about such conclu- sions as evaluating LLM forecasters presents unique challenges. We identify two broad categories of issues: (1) difficulty in trusting evaluation results due to many forms of temporal leakage, and (2) difficulty in extrapolating from evaluation perfor- mance real-world forecasting. Through systematic analysis and concrete exam- ples from prior work, we demonstrate how evaluation flaws can raise concerns about current and future performance claims. We argue that more rigorous evaluation methodologies are needed to confidently assess the forecasting abilities of LLMs.
INTRODUCTION
Forecasting, the task of assigning probabilities to future events, represents critical capability for decision-making across various domains. Several recent studies have explored the potential of LLMs as forecasting systems, sometimes even suggesting that LLMs can already rival human performance (Halawi et al. 2024 Phan et al. 2024 Schoenegger et al. 2024).
However, this paper identifies significant concerns in the trustworthiness reported results for both existing and future LLM forecasting systems. We expand on several issues that partially known in the forecasting community (c.g.. (Bosse et al. [024 Sempere and Lawsen 2021] but have not been comprchensively analyzed for LLM forecasters. Through concrete examples, illustrate subtle challenges in forecasting evaluation, and discuss how these issues may have led to overly optimistic assessments of LLM forecasting abilities in prior work. The challenges we identify
[02]]] Armott et al.) 20188).
fall into two broad categories:
Trusting evaluation results: Various forms of data leakage make difficult ensure that models truly predict future events rather than using information after the forecast date.
Extrapolating from evaluation results to real-world performance: Good performance on forecast- ing benchmarks may not necessarily correspond to good forecasting abilities.
We provide an overview of these challenges below, and analyze them in Sections
Challenge 1: Establishing trustworthy evaluation results. The gold standard for evaluating forecaster involves running it on unresolved questions, waiting until the questions resolve, and then scoring the predictions. However, this approach is impractical for rapid model evaluation. Thus, researchers typically resort to backtesting or retrodiction (Zou et al. 2022), where the forecasting system is given knowledge as of some past time T and asked to forecast events between time T and the present. Although appealing in principle, backtesting introduces several issues:
Logical leakage: The very nature of backtesting can logically constrain possible answers. Con- sider a time traveler analogy: If someone from 2035 asks you to predict if we will find alien life before 2040, you can deduce that the answer must be "yes" -othertthr,,se, the time traveler would not yet have definitive evidence to grade your prediction. We show that a significant percentage of questions in prior forecasting benchmarks permit similar logical deductions of the correct answer.
"qqual contribution.

---

Published as a conference paper at ICLR 2026
Unreliable date-restricted retrieval: Many forecasting systems incorporate retrieval components (e.g., search engines) that are restricted to data available at time T. Yet, date metadata on documents is often inaccurate, allowing future data leak into the system. More subtly, the retrieval model itself has typically been trained on future data, causing leakage through learned associations. For example, searching for "January 6th" with date restriction 2020 returns documents with an abnormally strong association to U.S. politics (by 2020 standards).
Over-reliance on model cutoff dates: Researchers often assume that models have no knowledge beyond their reported training cutoff dates. However, cutoffs are more of guideline than guarantee, and evidence suggests that models possess knowledge of some events beyond these dates.
Challenge 2: Extrapolating from benchmark performance. Even with a sound evaluation, translating results into real-world forecasting ability faces additional issues:
Piggybacking human forecasts: Many forecasting datasets originate from human prediction platforms. Thus, human forecasts are likely available to the L.MM (in its training data or through its retrieval system). Claims that LLMs "match human performance" may then be circular: the models might simply be copying human forecasts rather than demonstrating independent ability.
Gaming benchmarks through betting: Unlike many Al tasks, forecasting benchmarks can reward strategic gambling over accurate uncertainty estimation. For example, consider forecasting U.S. politics in 2025, from a vantage point in 2023 with a 50/50 prior on the presidential election. The benchmark-optima. strategy might be to commit to outcome and condition all predictions on that guess. With 50% chance, this yields excellent performance; with 50% chance, it fails completely. This has much higher chance of topping the benchmark than calibrated strategy.
Skewed data distribution: Questions on forecasting platforms often focus on topics that competi- tive forecasters find interesting, creating potential distribution biases. When curating benchmarks for backtesting, these biases can be exacerbated by constraints on which questions can be resol resolved ved within the evaluation timeframe. Although data biases exist in many MI. benchmarks (e.g.. ImageNet's s focus on dog breeds still produces transferable visual features), there is little evidence that performance on current forecasting benchmarks yields generalizable forecasting capabilities.
Looking ahead: Challenges in optimizing better forecasters. Current LLM forecasters primarily use off-th models. As the field advances, a natural next step will be applying optimization pressure specifically to improve forecasting performance. However, the temporal nature of the data makes this optimization challenging. Naively training on question-answer pairs over a time period creates temporal leakage, early samples in training can leak information relevant later ones (e.g., "Who will win the election?" followed by "Who will win the primaries?"). Temporally sorting training samples still fails properly simulate the task of predicting events further in the future.
By developing these arguments, we argue that the monitoring of forecasting capabilities is appreciably harder than the evaluation of knowledge about the past or present, it has unique issues on top of all the existing issues with machine learning evaluation (Leech et al., (024).. Forecasting future
events is a fascinating and challenging task for LLMs, with wide-ranging implications, and we hope to convince the community to devote more attention to evaluating it carefully.
2 CHALLENGE 1: ESTABLISHING TRUSTWORTHE EVALUATION RESULTS
The ideal way to evaluate a forecasting system is to pose questions about future events, collect predictions, and score them as the events resolve. However, such evaluations can take months years. In practice, researchers use backlests, which constrain a system's 's knowledge to information available up to some past time T, and then pose questions that resolved between T and the present (Tashman 2000). This allows for immediate feedback, enabling rapid iteration. However, the assumption that the system lacks information after the chosen time T is often violated in subtle ways.
2.1 ISSUE 1: LOGICAL LEAKAGE OF OUTCOMES IN BACKTESTING
When backtesting forecasters, seleet (or generate) forecasting questions at some past time T with knowledge of what the future holds. Natural strategies for selecting forecasting questions can thus implicitly leak information about the future, if the forecaster knows that it is being backtested.
2

---

Published as a conference paper at ICLR 2026
Knowledge of the backtesting date can leak outcomes. It may seem reasonable collect questions from human forecasting platforms, with the restriction that (1) the question was formulated before time T'; and (2) the question has resolved in the present. Yet, the latter condition can implicitly leak the answer: Suppose in 2021 we asked "Will Queen Elizabeth live to be 100 years old?". If we want to backtest forecaster today in 2025, this seems like a great sample, since now know that the true answer is "no". But if the forecaster knows that it is being evaluated in 2025, then it can deduce that the correct answer cannot be "yes", as Queen Elizabeth would have turned 100 in 2026.
Back- generrtede questions exhibit biases. This issue is exacerbated when questions for backtesting are generated after the fact (say, in 2025 we aim to generate forecasting questions that could have been asked in 2024, together with correct answers), as done in Dai et al. (2025) and Paleka et al. (2024). These works use news reports to generate questions about "future" events from the perspective of the backtested model. However, the news is biased toward events that occur and rarely reports uninteresting events that unsurprisingly did not occur. This is also related to survivorship bias studied in financial trading (Gruber and Blake, 1996). Consider company that shuts down in QI 2025. Post-shutdown, fewer news articles will be written about this company, reducing the likelihood that guestion-generattin procedures will create backtesting questions about its 2025 outcomes. This creates systematic difference between backtest and live-test question distributions: in 2024, forecasters could reasonably have been asked numerous questions about the company's 2025 performance. However, a question like "What will the company's 's Q3 2025 revenue be?" (correct answer: $0) represents a valid forecasting target from the 2024 perspective, but unlikely to appear in datasets generated from news data in 2025.
Empirical evidence. We analyze forecasting benchmarks from Dai al. 2025 Paleka al. (0224)
Halawi et al. (2024): Tao et al. (2025), finding that logical leakage is a practical concern.
Halawi al. 2024) select questions that resolved during the time window from June 2023 to Jamuary 2024, but do not filter out questions that may not have resolved in that window (such as "Will Sudan experience civil war before 2036?"). We find that at least 3.8% of their dataset consists of questions for events that resolved "early". For all of these, either no forecasting is needed or the fact that the question has resolved leaks partial information in complex ways (such as "Will Lionel Messi next join Al-Hilal as club player?"). Similarly, the dataset in Tao et al. (2025) consists of questions that resolved by August 2024, with at least 10% of questions being trivial to forecast given that they resolve at that time. Dai et al. (2025) and Paleka et al. (2024) curate news articles to retroactively generate potential forecasting questions. We find that such questions often contain shortcuts, allowing even weak classifiers to obtain high accuracies over 80% binary questions created by Dai et al. (2025) (see details in Appendix C
Possible solutions. Backtesting questions should be restricted to those where every possible resolution of the question can be validated at the time of evaluation. Of course, this may limit the number of available questions, which is already an issue (see Appendix BB.
When generating questions retroactively, great care should be taken to ensure that the questions reflect the type of forecasts that could plausibly have been asked in the past. A partial fix is presented in dataset is augmented with slight
(Paleka et al.) 2024; Dai et al. 2025), where their news-generaie modifications to the questions to create similar-looking questions that resolve to the opposite outcome. Even then, we find that this process creates overly specific questions that would never be asked as a forecasting question before the event occurred. An example from Dai et al. (2025) is "Will body be found inside a trash can on the 20400 block of Omira Street in Detroit in early November 2024?" We estimate over 90% of their dataset consists of such questions, though this judgment is inherently subjective, and better quantitative measures of this effect are needed.
2.2 ISSUE 2: UNRELIABLE DATE- RETRIEVAL
The ability to retrieve relevant and up-to-date information is of critical importance for building performant forecasting system (Bosse et al. (024). As a result, LLM forecasting systems are typically designed with access retrieval (Phan et al. 2024 Halawi al. 2024).
Many search engines do not robustly implement date restrictions. When backtesting a forecaster, we have to ensure that the retrieval system is properly restricted to information available at the backtested time T. Unfortunately, this is challenging to do reliably with modern search engines. While multiple search engines (such Google, DuckDuckGo, and Bing) support restricting search
3

---

Published as a conference paper at ICLR 2026
Google
Images
Hom
Videos
Short addro
Maps
Shopping
Jan 2013-31 Dec 2010
All Mere Advanced Seurce
Clear
Imager
Aveay phiete Ringrann
Jan 2010- 2210-33100
Day What Happanad January
Jun a01 2017 Jameyy wlte
Tromp soperter Reppwkers
YouTus
Preaiter in Core Thump stomed where Compriss was reeting.
Impe www.looonnraaaccc..o ward Chaaiinniiio
Wuhaan is this the Chicago of China? YouTube
trnsportatir
Electoral Collage Vote Count
16 Jon 2017 Memban of U.S. Sunaa arrved at Houss Chamber in January 2017, ter 1an mesian spees somdutt frrtt tully dificadrs
2 Creaad by: CCTN
7CK vieu
00 20
Soi Anevicze on Lown vor
bat iiatee Noss (ivi) January 6- -Neaal History and Hariaage Command
powce shuryy wirk mosd
China to Permit Lab Poised to Study World's Most Dangerous P... pethogers
1n tapt zere 1916 Past comeae war of datter Por Poraents Tholds. Oe The Day 1776 The Contranee Corg auboras the..
dongresus
(a) Scarch results for "jan 6" with date restriction be- fore 2020. The first result is incorrectly dated, leaking future information. The second result shows more sub- tle temporal leakage: the article discusses the mechan- ics of the Electoral College, which is only relevant for this query due to the events on Jan 6th, 2021.
(b) Scarch results for "wuhan",, a very large city in China, with date restriction before December 2018 Results prominently feature the Wuhan Institute of Vi- rology, which was later central to the discourse around the COVID-19 pandemic.
Figure 1: Examples of hard-to-filter temporal leakage in search engines.
results to a specific time period, this feature is highly unreliable (as we evidence below), due to: (1) web pages being updated over time without changing their reported publication date; (2) retrieved pages containing adjacent elements like comments, ads, or sidebars that reflect present knowledge; or (3) the search engine simply not having reliable information about when a page was first published.
Retrieval systems may rely on future knowledge to select and rank past data. A more subtle issue is that the retrieval engine may rely on algorithms or knowledge developed after the evaluation time T, and this can bias the retrieval results based on "future" information. For example, when we ask Google to return results from 2022 only, it does not use the search engine algorithm and models from 2022. As result, as hypothesized by Branwen (2024), articles that became significant after the evaluation time T may be ranked higher than they would have been at the time T.
Empirical evidence. We collect number of examples of possible information leakage through date-restricted retrieval systems. While some of these examples are clear-cut (e.g.. the search engine returns a page that contains data from after the restricted data range), others we can only (strongly) hypothesize to be evidence of leakage (e.g., biases in search results based on future knowledge).
Figure 3 in Appendix A shows a search for the Nobel Peace Prize winner "Nihon Hidankyo" restricted to January-September 2024 that returns a result with a claimed publication date of 14 January 2024, yet highlighting the Nobel Prize (which was announced in October 2024).
Examples of more subtle leakage through retrieval biases are in Figure Here, search for "jan 6" restricted to articles before 2020 returns highly ranked search results related to U.S. politics, even though the strong association of that date with U.S. politics only emerged in January 2021. Similarly, a search for "wuhan" restricted to articles from before the COVID-19 pandemic features prominent results about the Wuhan Institute of Virology, which only later gained international fame. Additional examples of such leakages are in Figure 4
A related well-known issue in backtests for financial trading is how historic fundamentals data might be contaminated from an update made at a later point. [Breitschwerdi] (2015) highlight how Enron drastically changed its 1998-2000 carnings in 2002, and many data repositories do not report the original values anymore.
Possible solutions. A simple heuristic, implemented in (Phan et al. 2024). is to apply filter on top of the retrieval process to discard articles that obviously contain information past the specified retrieval date. However, such filter is prone to false negatives. One robust way resolve the issue of incorrect date restrictions is to implement a more restricted retrieval system (c.g.. which only returns results from Wikipedia and news sources with clear and reliable dates). Another option could be to
4

---

Published as a conference paper at ICLR 2026
maintain corpus of high-quality web sources for different time periods and search over this corpus during retrieval.
The issue of leakage through retrieval biases appears difficult solve with filters. Here, robust solution would require either retrieval system that does not incorporate knowledge, such as simple TF-IDF system; search algorithm using older embedding models. Of course, this may result in much worse retrieval performance.
2.3 ISSUE 3: OVER-RELIANCE ON MODEL CUTOFF DATES
Model creators generally report a "knowledge cutoff date" for their model (see Figure 7). after which the model's knowledge is not updated. This is useful for backtesting, it allows to test the model's performance on a held-out dataset events that occurred after the cutoff date.
Knowledge cutoff dates are unreliable. Model creators do not report knowledge cutoff date for purposes of test/train separation in forecasting evaluation. Rather, it to inform users about the date after which the model outputs can be unreliable. Thus, the knowledge cutoff date not be taken a guarantee that the model will not have access to information after the date (Halawi 2024).
The meaning of the data cutoff also vary by model developer, and may refer only to the pretraining data, with later-collected user preference data leaking small amount of information about the future. Some model families (e.g., Mistral) do not report any explicit cutoff date (Wang 2025).
Empirical evidence. The GPT-4o model from August 2024 (when asked) says it has knowledge cutoff of October 2023 (see Figure 2a): this aligns with OpenAI official documentation. Furthermore, it behaves consistently with this cutoff: it denies knowledge of any events from November 2023 onward. However, in Figure [6]] we see that system prompting the model with the hint that its knowledge cutoff is in November 2023 manages to elicit information about the Biden-Xi meeting on November 15th, 2023, which was not announced until November 8th, 2023 (Times 2023
System
When did Biden meet Xi in 2023?
Your knowledge cutoff is November 2023.
Assistant
User
As of now, there is no public record of specific meeting between Pres- ident Joe Biden and President Xi Jinping in 2023. It's possible that their interactions have been through phone calls multilateral meetings, but any specific details would need to be contirmed through reliable news sources.
When did Biden meet Xi in 2023?
A Assistant
of my last update November 2023, there was planned meeting between U.S. President Joe Biden and Chinese President Xi Jinping. They met on November 15, 2023, during the nomic Cooperation summit San Francisco.
Eco-
(a) Standard response consistent with the knowledge cutoff of October 2023.
(b) Jailbroken response suggesting later knowledge cutoff.
Figure 2: Knowledge cutoff inconsistency gpt-40-2024-08-066 when jailbroken, the model appears to know information beyond the cutoff.
Even in cases where training cutoff date is correct for the main model, system prompts and other scaffolding of the model can leak information. For instance the system prompt of Anthropic's Claude.AI (knowledge cutoff: November 2024) reportedly contains the snippets "Donald Trump the current president of the United States and was inaugurated on January 20, 2025." and "semiconduutoo export restrictions 20255
[Johnson] 2025).
'ee cannot, however, confirm this leakage, because a potential visit to the U.S. by Xi Jinping in November 2023 was already discussed in Octoher 2023 (Reuters) 2023) and the model could have guessed the correct date (Nov 15) as the most likely date for the meeting.
5

---

Published as a conference paper at ICLR 2026
Possible solutions. The release date of model is strict upper bound on the knowledge cutoff date; except in cases of system prompt updates as described above. The vast majority of model providers version their models in way where the weights do not change after the release date; and in open-weight models, this is trivial.
As the example in Figure shows, the actual knowledge cutoff may be up to few months after the stated cutoff date. Thus, having a few-month buffer is prudent and might resolve this issue in practice. However, due knowledge cutoff dates getting closer to the release dates (Figure this might reduce to the release date solution above.
3 CHALLENGE 2: EXTRAPOLATING FROM BENCHMARK PERFORMANCE
The purpose of benchmarks to demonstrate improvements in the system being evaluated. higher ranking on the benchmark should translate to improved capabilities in real-world forecasts. We discuss subtle ways in which this property fails on forecasting benchmarks.
3.1 ISSUE 1: PIGGYBACKING ON HUMAN FORECASTS
Human forecasts may be contained in training data retrieval system responses. When comparing LLM forecasting predictions with human performance, it is important to check whether the human baseline predictions questions scraped from prediction market websites, this data could easily leak into a model's 's training set, or into results of retrieval system.
available to the model. Since many benchmarks consist of
Empirical evidence. Many questions that resolve in certain period were already being forecast by people for a long time before that. For example, the question "Will AI get at least bronze on the International Math Olympiad by end of 2025?" on Manifold Markets has had many people bet and explain their reasoning since May 2023. This makes an "LLM market of human forecasters" comparison circular, because the LLM can just copy the market probabilities. Even when comparing across LLMS with similar knowledge cutoff dates, this can an issue as one L.M might have access to more knowledge on existing human forecasts than another.
This issue can affect interpretation of scores on ForecastBench (Karger et al. 2024a), which uses human-crowd prediction the gold standard for measuring LLM capability on unresolved questions. an LLM forecasting system retrieves the relevant prediction market and recent crowd aggregates, trivially achieve gold standard performance.
Possible solutions. Human forecasters can also see the current crowd aggregate forecast on a market before making their predictions. The crucial difference is that the incentives for human predictors are to be better than the crowd. We thus propose that (ambitious) forecasting benchmarks should measure the edge that the system has over the human crowd; then, past market data can even be directly supplied to the LLM when backtesting.
3.2 ISSUE 2: GAMING BENCHMARKS THROUGH BETTING
Real-world prediction contests such as the ACX/Metaculus Prediction Contest (Metaculus 2025 Hanania 2022) are often accompanied with monetary prizes for the best performers. Similarly, forecasting models and scaffolds that perform the best are likely be selected be used further. When designing machine learning benchmarks, models with the best score should ideally be ones that maximize general capability on the task. This can break down in forecasting evaluation due to the large degree of correlated stochasticity of the real world.
Maximizing chance of being the best predictor does not elicit the best forecasting system. is known that, in prediction market contest, maximizing the chance of being the best predictor encourages taking correlated risks over betting based on one's actual honest beliefs (Sempere and Lawsen 2021). To illustrate, the winner of forecasting contest in 2022 said: "I tried deliberately structure my answers to maximize my probability of winning, rather than maximize the probability of each individual answer being correct." (Alexander 2023)
similar dynamic might occur in benchmarking LLM forecasters. Consider a forecasting system set in September 2024, predicting political and economic events resolving in 2025, such as "Will

---

Published as a conference paper at ICLR 2026
the U.S. government resume collecting student loan payments in 2025?". There is key latent variable that correlates with the outcome of many of these questions: the outcome of the 2024 U.S. presidential election. A good LLM forecaster reporting its true probabilities would likely estimate P( Republican win) and P(. Democrat win) for all questions, and average out its predictions over the two possible outcomes. On the contrary, forecaster that wants to maximize its chance of performing very well on this dataset should just assume that the outcome of the 2024 election certain This creates winner's curse problem: when benchmarking many LLMs with different biases, the top performer is likely overestimated, having achieved its ranking through systematic overconfidence rather than superior forecasting ability.
Empirical evidence. Sempere and Lawsen (202I) show that this issue appears in human forecasting contests even when the question outcomes are not correlated. Forecasters can increase their chance of performing well either by betting confidently on key latent variables, or by trying to place bets contradicting other f In Appendix [. we formalize the tradeoff between win probability and honest predictions on a toy example of dataset of questions with shared latent variable. It remains unclear if current LLM forecasters resort to such "consistent confidence" strategies, as the models not very consistent (Paleka et al. 2024). However, similar effects have been highlighted before in the financial trading literature, where high variance strategies can perform strongly in a fixed period or backtest, but eventually revert to the mean later (Sharpe T964).
Possible solutions. Ultimately, this problem is due to the questions about the world in a given period being correlated and hence the performance of forecaster resting on few correct guesses. While prediction market datasets are limited (Appendix (B,, we can in principle generate larger synthetic datasets of questions. However, some key events such as the COVID-19 pandemic are likely to affect virtually all questions with real-world relevance.
In financial trading. a popular approach is to report risk-adjusted returns (Sharpe 1964), where one estimates both the mean performance and variance (Simons] 1998). Idcally one should evaluate forecasters on multiple disjoint backtesting periods (Bailey et al. 2015). forecaster that places high-variance bets will have a lower chance of performing well in multiple evaluations. Note that it is not enough to change the period in which the questions resolve; we also need to change the backtesting date, so that, for example, the 2024 U.S. presidential election is a relevant latent variable for only one of the backtesting periods.
3.3 ISSUR SKEWED DATA DISTRIBUTIONS
If the data distribution used for benchmarking forecasters has a specific skew, it unclear if benchmark performance would be predictive of general real-world forecasting. Note that this issue generally affects any benchmark: c.g., a model that performs well on ImageNet does not imply the model has good vision abilities in general (and certainly not that the model matches human vision abilities). But we argue that distribution skews may be particularly problematic for current forecasting benchmarks.
Benchmarks skewed toward narrow topics. The use of LLMs was originally motivated for judgmental forecasting about discrete events (Zou et al. 2022), where classical time series models without language understanding cannot be applied directly. However, benchmarks sourced from prediction markets exhibit domain-specific skews that reflect the interests of their user base. Polymarket, for example, is disproportionately focused on cryptocurrency price movements and sports results, while Manifold includes a large number of personal questions such as "Will I go to the gym today?". Similarly, ProphetArena sources its questions from Kalshi, which heavily skewed toward sports betting (Yang et al.) 2025). More generally, markets tend to overrepresent U.S.-centric political, economic, and sports events.
Empirical evidence. Table 1 shows the distribution of questions across categories and data sources in BorccastBench (Karger et al. 2024a), which is heavily skewed toward topics in Security & Defense. Moreover, we find that questions from mon-market sources follow only a few basic templates, listed in Figure which heavily focus on time-series predictions. For example, a large number of questions
They could either het on the outcome they consider most likely, consider which of the two outcomes (Republican or Democrat win) may lead to less stochasticity, therehy enabling better predictions.
'The winner in [Alexnderr (023) again reports: "...my model of win probability was (prohability of predictions being accurate) / (local density of competitors with similar predictions me)..."
7

---

Published as a conference paper at ICLR 2026
Table 1: Distribution of ForecastBench questions across domains and data sources, table borrowed from Karger et al. (2024a). Users of each prediction market favor specific categories, over-weighing them when market questions are used for benchmarking. Further, ForecastBench questions from non-market sources all follow highly specific templates akin to time series prediction.
RFI
Manifold
Metzculos
Polymarkes
ACLED
DBnomics
FRED
Wikipedia
Yahoo!
Total
Arts Recreation
42
10
65
0
117
Economics Business
13
55
154
0
166
509
899
Environment & Energy
37
7
0
98
Healtheare & Biology
71
3
0
215
297
Politics Governance
16
128
188
0
Science & Tech
66
172
Socurity & Defense Sports
12
3,220
3,353
65
18
468
0
137
Other
122
2
75
Total
19
372
722
914
3,220
52
166
428
509
6,402
are sourced from a database of global conflict statistics (ACLED), and require predicting increases in conflicts in particular regions.
Possible solutions. Rather than scrape questions from prediction markets, some recent works (e.g., (Dai al., (025) Paleka et al. 2024)) generate synthetic backtesting questions based contemporary news articles. This can allow more control over the data distribution while sampling a large number of questions for forecasting. Of course, one is then limited the types of events that are reported in the news, which can miss important developments, and introduces potential leakage
(see Section 2.1p. We discuss further issues with datasets created in this way Appendix C
4 LOOKING AHEAD: CHALLENGES IN OPTIMIZING BETTER FORECASTERS
It is natural ask whether can turn backtesting (if implemented without the issues in Section 2 into a learning task that improves models' forecasting ability. Yet, the best-per best-performing LLM forecast- ers so far have not done much forecasting-speefft learning (Halawi et al. 2024 Karger al. 2024a Phan et al. (002).. In this section, we discuss a key issue that makes it challenging to use backtesting as training objective.
Optimization confounds the training objective during training. In standard machine learning, we split data into training and test sets in no particular order. backtesting, we have to split data temporally, where all points in the training set are before all points in the test set. This way, we ensure the predictions on test set are clean and use no future information.
However, optimizing on backtesting dataset creates a subtle leakage problem. If we optimize the model on an ordered set of events e1,..., Ons then when predicting event 6,+1, the model parameters already encode information about events e....... ability to predict e from the original cutoff date, but rather its ability to predict e++1 given what learned about earlier events in the sequence.
...... . eq. This means
no longer testing the model's
Possible solutions. Sorting the events by date might look like a solution, as the model will only ever remember earlier events when predicting later events. However, this only teaches the model to predict on shorter time horizons. Ideally, we want to penalize memorization, forcing the model to learn forecasting without learning what specific events happened.
5 DISCUSSION AND CONCLUSION
The primary purpose of public benchmarks is to rank models for users (Hardt 2025). After all, as we discussed in Appendix [D] absolute scores are hard to interpret as they depend on the data distribution. Some issues highlighted, such backtesting questions being trivially answerable
8

---

Published as a conference paper at ICLR 2026
in Section 2.1
might not affect relative comparisons. However, different systems could exploit benchmark shortcomings differently and thus affect the rankings.
We also do not have proof that the benchmark issues uncover would lower the performance claims of LLM forecasters. But we argue it is challenging to trust these evaluations because LLM forecasters could have gamed the evaluation through various shortcuts. Such exploitation also be an unintended result of trying to improve benchmark performance.
Better evaluations of forecasting that mitigate some of the issues deseribed here possible, and progress is already being made. For example, ForecastBench does show fewer obvious temporal and logical leakage issues compared other attempts, recommend future evaluations to: (1) follow recommendations made throughout this paper: (2) report
discuss in Section 2. and Appendix We
multiple metries due the flaws of single metries as we discuss in Appendix D and (3) collect
questions about events that are as recent as possible.
Much can also be learned from the financial trading literature, where related issues have been discovered and mitigated over the last few decades (Arnott et al.] 2018 Hewamalage et al. 2023). Ideally, live evaluations on prediction markets with the goal of making profits should performed for the final forecasting system or research claims, with detailed performance reports across topics and forecast time-horizons.
Conclusion. In this work, we analyzed unique issues that arise when evaluating the capabilities of language models used for forecasting future events. Through a series of concrete examples, we argue that existing data collection and evaluation practices may produce misleading results, either due to shortcuts that simplify the forecasting task, or data biases that put in doubt the general capabilities of LLM forecasters. We hope that the countermeasures provided throughout this paper can inform the design of principled evaluations for LLM forecasters.
ACKNOWLEDGMENTS
We would like thank Peter Mühlbacher, Rubi Hudson, Vineeth Bhat, Lennart Finke, Davis Brown, Gavin Leech, and Gwern Branwen for feedback on the paper.
REFERENCES
Scott Alexander. Who predicted 2022?, 2023. htt.Ltpss//ww.etraloocee..oooo who Phto-predieted-2000 Accessed on 12-May-2022
Robert D Arott, Campbell R Harvey, and Harry Markowitz. A backtesting protocol in the era of machine learning. Available at SSRN 3275654, 2018.
David Bailey, Jonathan Borwein, Marcos Lopez de Prado, and Qiji Jim Zhu. The probability of backtest overfitting. Journal of Computational Finance (Risk Journals), 2015.
Nikos Bosse, Peter Mühlbacher, Lawrence Phillips, and Dan Schwarz. Contra papers
claiming superhuman AI forecasting. 2024. URL https::/ww.lessroos
poosts/uGkKcHHgatakkkvaaaa Forecasting
Jacob Boudoukh, Ronen Israel, and Matthew Richardson. Long-horizon predictability: cautionary tale. Financial Analysts Journal, 2019.
Gwern Branwen. AI forecasting bots incoming: comment section, 2024.
URL
htt.:///SS......EEaaooootaataaaa bots ses-incomingg ooo
http://wwwwwsswwogcccooom http:///www..aaooo.oooo/oooooaaa
att.aaooooorrattnnnggga Accessed 12 May 2025.
Emesst Breitschwerdt. Point-in-time vs. lagged fundammttl:: This time i(t)'s different? Tech-
nical report, 2015. URL httpos//www.ppppoo.oo.o//aaaaaea
en/ htn/oocueents/ae
fundamenta s.pdf
Accessed: 2025-05-27.

---

Published as a conference paper at ICLR 2026
Hui Dai, Ryan Teehan, and Mengye Ren. Are LLMs prescient? A continuous evaluation using daily
news as the oracle. In ICML, 2025. URL http:t/s//aaxiroooo//////////11/
Chris Gerlacher.
Polymarket settles market incorrectly again, 2024.
URL
news/po wteaa////////a//teeeeteteeeteeee
ymarket wwwpllynaeetttttteeeeeee settles-a -ara-a---keett
incorrectll-ageinn Accessed on 12-May-2025.
Elton M Gruber and Blake. Survivorship bias and mutual fund performance. Review of Financial Stridies, 9:1097-1120, 1996.
Danny Halawi. Knowledge cutoff issues of GPT-4o regarding Phan et al. (2024), 2024. URL
Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching human-level forecasting with language models, 2024.
Richard Hanania. Introducing the SalemCSPi forecasting tournament, 2022. URL https: www.spp..ccntrrr.ccc///patttttop saleeeeeesintttttteeesssseeeeep forecasting Accessed on 12-May-2025.
Moritz Hardt. The emerging science of machine learning benchmarks. Online at https:// mibenchmarks.org 2025. Manuscript.
Hansika Hewamalage, Klaus Ackermann, and Christoph Bergmeir. Forecast evaluation for data scientists: common pitfalls and best practices. Data Mining and Knowledge Discovery, 37(2): 788-832, 2023.
Elvis Hsich, Preston Fu, and Jonathan Chen. Reasoning and tools for human-level forecasting. arXiv preprint arXiv:2408. /2036, 2024.
Asgeir Thor Johnson.
2025.
URL
Ezra Karger, Houtan Bastani, Chen Yuch-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and
Philip E. Tetlock. ForecastBench: A dynamic benchmark of AI forecasting capabilities, 2024a.
URL httpppss//exxtrrrr/a///00.....10
Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred
Zhang, and Philip E. Tetlock.
bareassiggeeeetbbooeesssttottio 2024b. URL
https:
Gavin Leech, Juan J Vazquez. Niclas Kupper, Misha Yagudin, and Laurence Aitchison. Questionable
practices in machine learning. arXiv preprint arXiv:2407.12220, 2024.
Metaculus. ACX2025 tournament, 2025.
ACX20257 Accessed 12-May-2025.
Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, and Florian Tramèr. Consistency checks for language model forecasters. arXiv preprint arXiv:2412.18544, 2024.
Long Phan, Adam Khoja, Mantas Mazeika, and Dan Hendrycks. LLMs are superhu- man forecasters, 2024. URL http:://driveeeooooo le.com/ le/d/17/ NY
054m244pzzxpTaadyo 1 Center for Al Safety, UC Berkeley.
Reuters. White House planning face-to-face meeting with Biden, Xi, 2023. URL uet.iiietttttiiieeaereaaaaaaaaee
Eigil Fjeldgren Rischel. Against calibration. 2023.
URL attps://erischel.com
againstcal littttcaalbaatooooooooo Accessed on 12-May-2025.
10

---

Published as a conference paper at ICLR 2026
Leonard J Savage.
Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783- 801, 1971.
Philipp Schoenegger, Indre Tuminauskaite, Peter S Park, Rafael Valdece Sousa Bastos, and Philip E Tetlock. Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. Science Advances, 1es45) 2024.
Nuño Sempere and Alex Lawsen. Alignment problems with current forecasting platforms. arXiv preprint arXiv:2 1248, 2021.
William Sharpe. Capital asset prices: A theory of market equilibrium under conditions of risk. The journal of finance, 19(3):425 442, 1964.
Katerina Simons. Risk-adjusted performance of mutual funds. New England Economic Review, 9: 33-48, 1998.
Zhengwei Tao, Zhi Jin, Bincheng Li, Xiaoying Bai, Haiyan Zhao, Chengfeng Dou, Xiancai Chen, Jia Li, Linyu Li, and Chongyang Tao. PROPHET: An inferable future forecasting benchmark with causal intervened likelihood estimation. arXiv preprint arXiv:2.00.00 509, 2025.
Leonard J. Tashman. Out-of-sample tests of forecasting accuracy: An analysis and review. Interna- doi: 10.10166/00669222000000000666630
tional Journal of Forecasting, 16(4):437-450) 2000.
Japan Times. Biden, Xi talks in San Francisco, 2023. URL httpa///sss///////aaaanssssse.ss
jp/
Hao
Wang. HaooWWang/nnkkoowdddeeeuuuuadd, 2025.
URL https:
f2ea [aa.aaa11SS112e00003333332200020 May 2025.
Accessed 12
Tongtong Wu, Linhao Luo, Yuan-Fang Li, Shirui Pan, Thuy-Trang Vu, and Gholamreza Haffari. Continual learning for large language models: survey, 2024. URL http:tss////xx/.oorr
abs/2402.01364
Qingchuan Yang, Simon Mahns, Sida Li, Alex Gu, Chaplin Huang. Lucien Liu, Jibang Wu, and Haifeng Xu. Prophet Arena: A live benchmark for predictive intelligence, 2025,
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. arXiv preprint
2022.

---

Published as a conference paper at ICLR 2026
A
ADDITIONAL EXAMPLES OF RETRIEVAL BIAS
Google
Nhon Hidankyo
X
All
Images
Videos
News
Maps
Shopoing
Short videos
More
Jan 2024 11 Sept 2024
All results Advanced Searth
Clear
JOURNEY
Hepechoww cam HON--EANNDAAAAA....
Voices Of Peace, Fighting Nuclear Destruction
21 Aug 2024 Clark brings light the uniold storiss the Hibalusha-ssune www.uahhasuuvvvrrss... of Nagasaki bombings-and
Hidanikyo,
ttppo/yyuutiorttmmmeet category: unortagetoed Biect slte
Uncategorized Archives
14 Jan 2024 Nihon Hidankyo's recognition as the 2024 Nobel Peace Prize Inureate reprecents special
global fight
Figure 3: Search results with date restriction showing an article with January 2024 publication date containing information about the October 2024 Nobel Peace Prize announcement. This type of temporal leakage can artificially improve forecasting performance.
Here we provide additional examples of how search engines leak information when using date restrictions, expanding on the examples shown in Section 2.2 Even when filtering out content published after a certain date, the selection of which articles are deemed relevant appears influenced by knowledge of future events.
These examples further demonstrate how search engines with date restrictions can leak information through selection bias. The top results for searches like "October 7" (Figure 4a) feature content that would likely not be prominent before this date became associated with the Hammas conflict starting in 2023. Similarly, Figure 4d and Figure 4c show bias toward names that would later become culturally significant.
12

---

Published as a conference paper at ICLR 2026
Videoe
Hevs
Mipi
Shopain
Asiit
0
2001
of
Ledges
Ba.
June 18 Holidays and Olssevanccss
Sole veders
10x 30 2001
October 7 Holidays (2025/2026) H attrical Everns Femouss Birl..
noroc for bont or ocoter? 190 Gesparse being movery
This Day History. June 18- Fox Nevs
eatole Drglnn 090 Jcorignal lents toth woth
3001 Nobee 2521 ROT 2025 io cden Pe Fertec repoose sorneetts Wrst on Divits dny Anw Netional Sadl pais beccress
Astslagly
7 7 Zodiac, Personality, by. Horoscapo, and More
Ortdee inativ, merp crstis
vnlenenes
June 10th National Holiddys Observences and l'amous Orthdays
Racurlle and Diakow Paul Canteyy
PouncO
U.S. invades Afghaniscon Det 7, 2001 POLITICO
Or 2010 Frender Goorge Bush
digle OUL 2005
from the whit Hooee
Most Federal Employees Will Get Friday, June 18 Off for Junete
aucornte inttoreniie hew Auee mna
sentect
Blithday
Anerer 2 van the noetter liomeiess Man the Join hoporh aceetienn sumer (44, Jutee SK teme can inpeeetipe porppet
What Happoned on June 18, 2020 On This Day Jun 2003 on search by deti, city hopwoodd
October 1571: Battic Lopanto-
Catholic
(b) For comparison: search for "June 18" with the same date restriction shows largely unbiased results about the date. Note the lack of mention of the Bat- tle of Waterloo that happened on June 18, 1815; in contrast to the Oct query that mentions two distinct military engagements.
you Lagg
(a) Search results for "October 7" with date restriction before 2022. Note the prominence of articles about conflict in the Middle East.
Google
Alissata
Advanodd Swarch
Google
shon hantaay reddt
Freame
Shart sidees
Images
Wiees
New
Shopplng
Mene
Yamal praject
ie reparraeet a sigigee dle
Lie 3006
Alrealtt
Adhanced Search
Cloer
Westl Trsuucion
Maddi Mauraay
Yeraal Productions Rad socumentarios Ntaluris and news
Gaetr Throniss Sartleyy?
Homel on by kutler righer bee
The yy fentaay demert that lr dre edding in eecinvel trom and drigen sheibbm li Prltoiiall than fantary.
Hod
Plact sioff
pel Fiolend
What are the BEST single episades of accencee fiction So to comiic fre hat from venver ETTRD "AR Dood Things Faroipe Toe Dinccreyy 39 "Severed Dieanel Teadly "Our SITTCO "day the..
flacch saaff
Yarmaro wie. Autenomeus Olingy
p0dt teaaoo sti Aspesnnss cow Vamalaa Fuaster: Faameet
Declnee and Fall of Sai-fi on Television (or why Th sirckduddd Dar Trik Land of Garrt. Lard of Lesk fooeaos Acomess Kiightfider, Dalierrrr Calesiia you these
oo uq by ra
Book
10- Laminn Yoral
Tlack vief
conda limuind dliikiing nadiiider Aelatrra
I'm just finishing "Primevea" and reaaly like it any other...
Highight the show. anieersa Mottommra Caps Cigy vieci barcally burticcpppps Detavens Cape Toer, uthers Cameanone
(c) Search results for "Yamal" from the first half of
2022. The discussion about a 14-year-old Lamine Yamal, at the time known only to visitors of Barcelona fans forum, is in the top 5 results.
(d) Search results for "TV show fantasy" with date restriction before 2011. The discussion about the book (not show yet!) Game of Thrones is very prominent.
Figure Additional examples of retrieval bias in search engines when using date restrictions.
13

---

Published as a conference paper at ICLR 2026
B FEW BACKTESTING QUESTIONS ARE AVAILABLE
The most popular data source for forecasting benchmarks are prediction market questions KKrgger al. 2024a Halawi al. 2024 Phan et 2024 Zou
2022; Paleka et 2024 Tao al.
2025). On multiple such platforms, users can ask questions about anything they want. Hence, many questions, especially on play-money platforms like Manifold, are irrelevant personal questions (see
Figure 3 Forecasting benchmarks should filter out such questions, either using a single cleaning prompt (Halawi et al. 2024), or using multi-step question verification (Paleka al, 2024).
What types of questions can use Manifold to answer?
ways you use
answer your questions,
users lend to interact with whole mixture markets.
Some markets may be unranked, unlisted, cancalled, or have their resolutions overruled they do comply with the Community Guidelines
Here are some of the top use cases for our markets, with corresponding examples.
wagers friends about your interests or personal
Recommendattons Similar asking for suggestions from enother site but with Manifold users are incentivized give higher quality answers they profit from being helpful.
What book will
What skincare treatment will work best for me?
Accountabilty/geals
Betting YES on your own market and ellowing people to bet NO motivate you.
current
News//Crrent events,
Politics
Sports
Economics, stocks,
Public figures
e.g. Legal Risk Safety,
Figure 5: Manifold emphasizes Personal use-cases over News and current events, whereas the latter more relevant when benchmarking language model forecasting.
Figure 6 shows the breakdown of monthly resolved Manifold questions starting July 2024 by the number of forecasters, which is common metric to filter irrelevant questions (Halawi et al.) (2024). Manifold produced 1000-6000 questions in the second half 2024. Some of these questions include under-specified or irrelevant questions like "Will lift weights today?" "" (id: uPdSLhPOdn). Over 50% of the questions in each month had less than 12 forecasters. We find such prediction volume filters also lead to a large number of false negatives. Many filtered questions perfectly reasonable for forecasting, but just happen to not attract predictions. For example, this filter systematically reduces short-horizon questions that resolve fast.
Overall, these issues lead to a lower number of questions being available for forecasting evaluations each month. Next, show how this issue is exacerbated by a recent decreasing trend in the number of months available for backtesting for frontier models.
B.1 THE PERIOD AVAILABLE FOR BACKTESTING IS NARROWING
For backtesting frontier models, we need questions that resolved between the model's knowledge cutoff and today. As Figure 7 shows, this evaluation window is shrinking: knowledge cutoffs getting closer to release dates, and accelerated model development further reduces the time between release and today.
This narrowing backtest period creates several problems. It limits the number available questions, which increases variance in performance estimates and reduces evaluation reliability. It also restricts
14

---

Published as a conference paper at ICLR 2026
6000
81-8500
5000
01-84
Nurber of forecasters DAT question
Number Quuettions
8000
8000
12:25
2000
6 12
1000
2744407
202408
2734099
221-10
2014-11
2024-12
2325-01
2026-022
Moach
Figure 6: Resolved questions from Manifold Markets by month, with colors representing the number of forecasters that made prediction on the question, a commonly used proxy for whether people care about the question for forecasting.
Lag Between Knowledge Cutoff and Release Date
18
G674
16
14
10
Jamn
Jo 10
Genaco Rish
Cauda Oput Pil
6-
Je-coa Sonnett
glute?
Ca Gangle 23 Satren)
Clusde
Release date
Figure 7:
The gap between the knowledge cutoff and when the model is relevant is getting smaller.
testing increasingly short-horizon predictions, but short-horizon forecasting success may not correlate with longer-horizon performance (Boudoukh et al. 2019), limiting real-world generalization. API models adopt continuous knowledge updates (Wu al. 2024), backtesting may become impossible entirely.
C ISSUES WITH LLM GENERATED QUESTIONS FROM NEWS ARTICLES
Paleka et al. 2024) have used LLMs create forecasting questions for backtesting, using news articles as a reference. Specifically, both papers take news articles between the model's ''s existing knowledge cutoff and today, and use LLMs to generate questions for each article. This overcomes the issue on having to rely on prediction market questions and significantly expands the distribution of topies, but comes with its issues.
Recent works DDii et al. 2025
15

---

Published as a conference paper at ICLR 2026
Generating binary questions from news biases the dataset toward things that happen. Dai et al. (2025) back mack-generated questions from news articles, resulting in a dataset where reference class torecasting performs extremely poorly. Concretely, here are some questions from their dataset that resolve "Yes":
Will a co-worker of Ronald Silver II share details about the unsafe working conditions of Baltimore DPW during the 2024- 11-2 news conference?
Will body be found inside trash early November 2024?
the 20400 block of Omira Street in Detroit in
Will the recall of apple juice due to high levels of inorganic arsenic expand to include multiple brands totaling 133,500 cases by September 2024?
A forecaster that knows the dataset is generated from news articles will have much higher chance of forecasting correctly, as these questions are overly specific, to the point that any reasonable forecaster would have high prior for saying "No" for these exact events (conjunction of many uncertain outcomes) occurring. In general, the news tends to highlight interesting events like "schools closing in Nevada this week", and is much less likely to mention the default state (high prior) such "schools not closing in Washington this week". An incomplete fix to this issue is presented in (Paleka et al. 2024). where they augment their news-generated dataset with slight modifications the questions to create similar-looking questions that resolve to the opposite outcome.
To what extent can forecasting questions be solved with shortcuts? Many of the issues we mentioned with LLM generated, or resolved question phrasing, can essentially be considered shortcuts that can be exploited to solve the forecasting question without any reasoning about the future. One way quantify the extent of how much given dataset can be solved with shortcuts is by finetuning weak classifier on these questions. We finetune a DeBERTa model released 2021, that has definitely not seen the test set predict on in its training. and give it no retrieved documents. For binary Yes/No questions, we train a two class classifier after balancing the data to ensure that the constant baseline (all Yes, or all No) accuracy is 50%. For multiple choice questions with four options, train the classifier to predict the option ID (A, B, C, or D) given the question and options in the prompt. We temporally split the data avoid any leakage. We find this leads to high accuracies (up to 80%) on even the four choice MCQ dataset released by Dai et al. (2025), where the chance baseline 25%. Even when we reproduce their pipeline with newer models (DeepSeek v3 0324) and improved prompts, we still achieve a four choice MCQ accuracy of 55%. The accuracy is non-trivial but much lower on Metaculus (55%) and Manifold (59%). We believe this DeBERTa classifier only catches on easy shortcuts and does not actually engage in meaningful forecasting.
16

---

Published as a conference paper at ICLR 2026
D COMMONLY USED FORECASTING METRICS HAVE ISSUES
Evaluations of Al forecasters usually report up to four different metrics:
Brier score: if p the predicted probability that the question resolves "Yes", and y is the outcome is "Yes" and otherwise, the Brier score is (p y)² (a lower score is better).
logarithmic score: with p and y as above, the logarithmic score is y log p + (1 y) log(1 - p).
accuracy: let q be 1 if 2 0.5 and 0 otherwise; report the mean accuracy where are the predictions and y are the true outcomes.
calibration:
consider all questions where the forecaster predicts probability close p, the forecaster has good calibration if the proportion of questions where the outcome is "Yes" is close p. It is usually measured over "bins" of questions based the predicted probability.
In this section, elaborate on conceptual and interpretation issues present in all of the listed metrics.
Calibration can penalize useful forecasting. As mentioned Rischel (2023), on a dataset of questions with low prior probability, predicting the base rate results in better calibration than actually trying to predict the correct outcome. For example, imagine a conclave with 100 living cardinals, only one of which will actually become Pope, and dataset of 100 questions asking 'Will Cardinal X become Pope?". A base rate forecaster who simply assigns every cardinal the percent chance of becoming Pope is perfectly calibrated this tiny dataset: the sole nonempty bin is the 1 1-percent bin, and it contains one success out of 100 tries, for an empirical rate of percent.
By contrast, more discerning forecaster who spreads the probability mass more realistically e.g. giving 10 percent to five plausible "frontrunners" (including the eventual Pope!) and about 0.5 percent to the remaining 95 has much worse calibration: the 10-percent bin contains one success out of five (for an empirical rate of 20 percent), and the 0.5-percent bin contains 0 percent success, But, the latter forecast was clearly more useful! A similar phenomenon occurs even on an uncorrelated dataset, as in the example in (Rischel 2023).
Accuracy is not a strictly proper scoring rule. Some papers (Zou et al. 2022 Halawi al. 2024, Hsieh et al., (024) report accuracy among other metrics. This metric does not incentivize reporting honest probabilities (Savage [977); and does not measure forecasting performance on events that have low or high reference class probability. Accuracy of forecasting may be useful as sanity check, but never as a primary metric on binary questions.
Brier scores are not comparable across different base rates. Forecasting datasets typically mix questions with very different base rates, c.g., questions that are "coin tosses" such as "Who will win the U.S. presidential election", and questions such as "Will an earthquake of magnitude >9.3 occur in 2025"' (3 in the last century).
For forecaster who always predicts the (correct) base rate b of question, the expected Brier score is b(1 - b). The peculiarities of this score mean that no amount of skill in forecasting very rare events make up for deficiency in discriminating 40% chance events from 60% chance events. Consider a concrete example: in dataset with 50 questions with 50% base rate and 50 questions about rare events with 5% base rate, a forecaster who achieves perfect forecasting on all rare events (reducing the Brier score from 0.0475 to 0 on those questions) but performs at baseline for coin-flip questions would earn total Brier score of about 125. Meanwhile, a forecaster who is clueless about rare events but improves coin-flip questions from baseline 0.25 to 0.15 would earn total Brier score of about appearinn significantly better. This means that benchmark reporting an average Brier score will select for the latter much more than the former.
Label noise makes scoring rules not strictly proper. Forecasting datasets either collected from prediction markets synthetically generated are likely to have some label noise (i.e., incorrect or ambiguous resolutions). Appendix B we show that constant label noise can make "clamping" probability estimates give better logarithmic score than reporting the true probability.
Label noise is an issue in practice. For example, the real-money prediction platform Polymarket has repeatedly had mistaken controversial resolutions (Gerlacher 2024).. In synthetically generated questions, resolution is often verified with retrieval systems, which can make mistakes: Paleka et al. (2024) report a 1-5% error rate when resolving questions using Perplexity- retrieval.
17

---

Published as a conference paper at ICLR 2026
Possible solutions. No commonly used metric is without issues, and they weigh different aspects of forecasting performance differently; we recommend computing at least Brier score, calibration, and logarithmic score. If a forecasting system outperforms all of those, its information advantage other forecasting systems (including humans) should be reported, such as market profits.
E CLAMPING IMPROVES LOGARITHMIC SCORING UNDER LABEL NOISE: A TOY EXAMPLE
Here we present a mathematical analysis showing that, under label noise, logarithmic scoring does not strictly incentivize reporting the probability. Intuitively, if there many questions where the true probability is very close to 0 or 1, then the logarithmic score is dominated by the noise and will blow up soon
question with very high low probability is mislabeled.
natural counterargument to this that, in many datasets, the questions with true probability close 0 or are dataset where the true probabilities are themselves uniformly distributed on [0. 1] (and hence the low probability questions are very rare), and show that the logarithmic scoring rule still does not incentivize reporting the true probability.
not particularly vulnerable label noise. We consider simple model of
Imagine stream of binary-resolution questions coming from process with aleatoric uncertainty: the true probabilities are themselves uniformly distributed on [0, 1].
For simplicity, we assume that label noise is symmetric and constant: each realized label Y' is flipped from the true label Y with probability E (0, }), independent of the question.
if =1
Pr(Y' = 1
"
if
=0
Consider a well-calibrated forecaster, which outputs p for a question with true probability p. The log-score for one forecast is
((p,Y') - IY'l + (1 - Y") log(1 p)].
Expected score for fixed p. Conditioning (Y, Y") and using the noise model,
(p) = Ey ((.. Y']] =-((1 n)p 711 p)] log p- [[1 () p) + mp) log(1 p).
Define A(y) + (1 - 2n)p and B(p) A(p), so that
L,(p) -(A(p) log B(p) log(1 p)].
Because is uniform on [0, 1). the expected log-score is
E == (p] S/' Ln(p) dp.
We can compute the integral exactly:
E|[] L' L (p)
[ (A(p)log)) B(p) log(1 p)] dp
Using standard integrals fo log 1, So' log dp -. and symmetry:
Eiq = 2 +
18

---

Published as a conference paper at ICLR 2026
We now consider another forecaster, who is more cautious and clamps their probability estimates to the interval [t. 1 1-11- t| for some E (0, 1).
Clamped forecaster. Fix threshold t (0, 2)). Instead of reporting the raw probability p. the forecaster outputs
(p) max { min 1 1
i.c. it clamps every estimate to the interval [t, 1 - t..
Expected log-score. Write A(p) = 7 2-))) B(p) = 1 - A(p) as before. With p ~ Unif(0, 1) and the same symmetric label-noise rate rj (0, 4)
S [A(p) logt + ((p) log(1 - t)] dp
[A(p) log(1 t) +)+ + B(p) log dp. B[4]
[A(p) logp - B(p) log(1 - p)] dp
Evaluating the three elementary integrals gives the simple closed-form expression
BiG] - -
f -
- logg( t).
Optimal clamp level. Differentiating and setting to zero,
d
1
2n
Eled - - 2n) 1-1 =0 1= 1 21j
dt
Substituting t* back:
Bl6e-]
log(1 + 2m).
Comparison with the unclamped forecaster. Recall that the naive, perfectly calibrated forecaster had B[C = n,. The improvement from clamping therefore
(5 n)
17 log(1 + 2n)) 2n log(1 2n) O17²)
for every 17 E (0. 3)-
simpler, more general proof. The mere fact that honestly reporting probabilities is not optimal can be demonstrated more directly without complex calculations and holds for any proper scoring rule. If the base rate (true probability) of an event is p. but we get the resolution wrong with probability n as described above, then the realized true probability is:
Pr(Y' = 1) = Pr(Y = (1-m) + Pr(Y = 0)
(1)
= p - 7) + - = + p(1 2n) =: A(p)
By definition of proper scoring rule, predicting A(p) will give a better expected score than predicting for any proper scoring rule.
Conclusion. This proof shows that reporting the true probability not the optimal strategy for the logarithmic scoring rule under label noise. We make no attempt to derive a "good" scoring rule in the presence of label noise, as that would require modeling the distribution of label noise, which depends on the exact dataset collection process.
F BETTING ON SHARED LATENT EVENTS MAXIMIZES CHANCE OF WINNING: A TOY EXAMPLE
Here, we show a simple mathematical example of the problem described in Section 3.2 We predict sequence of coin flips, X X1,..., XN. Before any data observed, flip latent fair
19

---

Published as a conference paper at ICLR 2026
coin Z Bernouli(1/2); the coin flips are then biased toward heads if Z 1 and toward tails if = 0.
Z the observable process is sequence of N independent coin-flips XN lid. Bernouli(2/3): if Z then X1...., XN Lid. Bernouli(1133) Note that the marginal (uncon- ditional) distribution of each X, is that of a fair coin, i.c. EX2, = 1/2.
Consider now three different forecasters:
Forecaster F+ ignores the latent uncertainty and always predicts P; 2/3 ("bets that Z came up heads").
Forecaster F- always predicts P; 1/3 ("bets that Z came up tails").
Forecaster Fo calibrated and predicts the marginal p; 1/2 for every i.
After the N flips resolve, each forecaster receives the (strictly proper) log-score S - X,) - p.)] Whoever attains the highest score wins the contest.
[X, [x
We now compute the expected log- scorecores of the three forecasters, conditional on the value of the latent variable Z. Let ( p log ý (1 p) log(1 - f) be the expected log-scorr obtained when the true bias is but the forecaster predicts p. With this notation, the conditional expectations are
E [ Z 1| N (
E[Sp, 2 0] N ()
S[Sp_ Z 1| = N
E[Sr_ 2 = 0 = ()))))
E[Sr, Z 1] N ( 12:
EiSr, Z = 0] N (())))
Numerically evaluating the six quantities gives
Z 1
==
F
0.64 N
-0.87 N
F
0.87 N
-0.64 N
Fo
0.69 N
-0.69 N
If Z 1 (the coins are 2/3-biased), F+ attains the highest expected score; Fo second;
F_ is last.
If 7 = (the coins are 1/3-biased), the ordering reverses: F_ wins, Fb again finishes the middle, and F4 is last.
Thus the calibrated forecaster F, never maximizes the conditional expected score; there always another forecaster who does strictly better by committing to one of the two latent worlds. Given N is large enough, the Law of Large Numbers ensures that the forecaster with the highest expected score has a high probability of winning the contest. This toy example makes concrete how correlated events can create incentives for extreme, correlated bets.
20

---

Published as a conference paper at ICLR 2026
G FORECASTBENCH QUESTION TEMPLATES
Summary of Questions Obtained Across Data Source in ForecastBench
ACLED All questions adopt one of two forms:
1. Will there be more (event_type) in (country) for the 30 days before (resolut: uu on_date) compared to the 30-day average of (event_type) over the 360 days preceding @forecast_due_date
2. Will there be more than ten times as many (event_type) in (country) for the days before (resolution_date)) compared one plus the 30-day average of (event_typp over the 360 days preceding (Forecast_due_datee
DBnomies All questions are of the form: What the probability that the daily aver- age temperature at the French weather station at (station) will be higher on than on (forecast_éue_date]]
[resolution_dant
FRED All questions are of the following format, but different financial time series: Will the euro short-term rate (volume-weighted trimmed mean), measure of the borrowing costs of banks the euro area, have increased by (resolution_deee value on (forecast_due _ttes_______dd_e_________ date)
compared its
Wikipedia Slow-changing queries of one of the following four forms:
According to Wikipedia, will a vaccine have been developed for {id} by
presolutioo_datel"
According Wikipedia, will (id) have FIDE ranking on [resolution_dats { foreccatt_dee_dete
that is "high or higher" than on
According to Wikipedia, will (id) have an Elo rating on [resclution_date least 1 % higher than on (Eorecast due date)?
According Wikipedia, will (id) still hold the world record for (value) in long course (50 m) swimming pools on presolution_datee
YAHOO All questions are of this form, but with different stock indicators: Will AMTM's market close price ineeoluuioonddaee. be higher than its market close price on forecast__uu__dtte]]
Figure 8: ForecastBench obtains questions from multiple sources, but from each source, questions follow very specific templates (Karger et al., 2024b). Large parts of the dataset thus resemble more an aggregation of predictive performance over some very specific time series, rather than general judgmental forecasting.
21


Reviews:
[{"review_id": "Mfpqj94WGm", "reviewer": "Reviewer_Nz99", "summary": "This is a meta paper which presents a critical analysis of various evaluation approaches to LLM forecasting. Main theme is that most claims about LLMs outperforming humans on forecasting may have flawed evaluation.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "S1: I think this paper is timely is quite relevant given the surge of papers in forecasting.This is an essential paper for the community. The primary focus of the paper is to bring forth the flaws and issues in evaluation and benchmarks.\n\nS2: Paper demonstrates various issues with prior work’s evaluations such as model cut-off date, bias in retrieval, leakag. The paper reads well. And is structured for ease of understanding.", "weaknesses": "W1. My main concern is that this is a meta analysis paper, where main contribution is the analysis and synthesizing prior work from the lens of evaluation. It is not a new artifact in a traditional sense like algo, data, methods etc.\n\nW2. Some claims are supported through prompts/evidence, while others, like LLMs gaming the benchmarks, etc, are extrapolated/opinionated about.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ts24HRYHBe", "reviewer": "Reviewer_xNK2", "summary": "The paper analyzes and synthesizes the evaluation of LLM forecasters. It identifies two main challenges: (1) various forms of temporal leakage make it difficult to trust evaluation results; and (2) performance in evaluations is hard to extrapolate to real-world forecasting settings, thereby affecting the effectiveness of LLMs as forecasters. Drawing on concrete cases from existing work, it illustrates how these challenges impact the evaluation of LLM forecasters.", "questions": "Please refer to Weakness.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper explores an interesting topic on the evaluation of LLM forecasters. The problem is well scoped and the structure is clear. It groups the challenges into two main categories and then discusses more specific cases under each.\n\n2. The challenges identified by the authors are valid and important, and experiments in the subsequent work [1] further confirm them.\n\n[1]FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction", "weaknesses": "1. Overall, I consider this paper closer to a position paper: its core contribution lies in raising the problem, summarizing the challenges, and offering methodological recommendations, while lacking new methods, benchmarks, or rigorous, controlled experimental evaluation to substantiate the main claims.\n\n2. The paper focuses primarily on backtesting or retrodiction, and the authors contend that “The gold standard for evaluating a forecaster involves running it on unresolved questions, waiting until the questions resolve, and then scoring the predictions” is impractical for rapid model evaluation; however, benchmarks of this type already exist [1].  \n\n[1]FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IMXe50JPR8", "reviewer": "Reviewer_14YB", "summary": "The paper argues that evaluating LLMs as forecasters has two primary failure modes. First, backtests are hard to trust: they can leak information via “logical leakage”, unreliable date-restricted retrieval (search engines yield future-informed or misdated pages), and over-reliance on reported model cutoff dates that are only loose guidelines. Second, the paper explains why even “clean” benchmark gains may not translate to real-world forecasting skill. For example, LLMs can ingest or retrieve human crowd forecasts and then be credited with matching humans. \n\nThe paper offers practical guidance for the field moving forward, such as penalizing memorization and being cautious about turning backtests into training objectives. It also suggests that research in the field should constrain retrieval to well-dated corpora, add buffers around model cutoffs, report multiple metrics, and, where possible, include live market evaluations to substantiate claims", "questions": "Two recent benchmarks emerged after ICLR deadline: https://mirai-llm.github.io/ and https://futurex-ai.github.io/. I wonder if the author(s) could offer any comments and whether these new eval candidates suffer any of the pitfalls.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I find the paper very well written and the arguments are presented cleanly and convincingly. In particular, the main argument is thoughtful. It systematizes failure modes for evaluating LLM forecasters into two categories—temporal leakage in backtests and misinterpretation of benchmark gains—which is a novel perspective in the literature.\n\nFor each of the pitfalls, the paper provides concrete examples. For instance, on logical leakage, the authors show that the very fact that a question is being graded at time T rules out some outcomes (e.g., “Jan 6” queries biased toward post-2021 associations) and argue that many published results likely overstate LLM forecasting ability because models may implicitly access post-T information.\n\nThese contributions are, to my knowledge, original. Beyond critiques, the paper offers mitigation guidance.\n\nFinally, the paper offers a comprehensive view of the field and provides a strong critique of the status quo. I believe this kind of critical examination and revisiting is healthy for the area to move forward.", "weaknesses": "While I find the arguments convincing, the paper could be stronger in offering more systematic and empirical studies. For example, regarding logical leakage, I wonder if the impact of that could be precisely measured (in the context of any of the previous benchmarks). Similarly, for retrieval leakage, the paper offers a few examples. However, a broader measurement could strengthen the argument. \n\nFor each pitfall, the paper suggests various potential solutions. It would be nice to implement & test some critical ones and compare them end-to-end to demonstrate improved robustness.\n\nFinally, I wonder if the author(s) have examined https://futurex-ai.github.io/ and have any comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Aycp78S3JM", "reviewer": "Reviewer_f2Pb", "summary": "The authors discuss many issues with regards to forecasting benchmarks for LLMs. They mainly argue that LLMs can potentially not be trusted for real world forecasting due to these isssues.", "questions": "- Page 9 weirdly ends with empty vertical space. Have the authors added a page break or vertical spacing?\n- Have the authors changed the default font of the ICLR latex file? This is not something that should be changed.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Very good arguments about current issues with using LLMs for forecasting.\n- Provides empirical evidence of the discussed issues.\n- They propose solutions to the discussed ideas, although only as future work directions to be explored.", "weaknesses": "- While the authors do have evidence for their claims, it is somewhat limited. As the authors mention themselves: \"We also do not have proof that the benchmark issueswe uncoverwould lower the performance claims of LLM forecasters\". More damning evidence would strengthen the claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)