ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities
Authors: Ezra Karger
Houtan Bastani
Chen Yueh-Han
Zachary Jacobs
Danny Halawi
Fred Zhang
Philip Tetlock
Venue: ICLR 2025 Poster (2025)
Keywords: language models
evaluation
benchmark
forecasting
LLM
decision-making
datasets
Primary area: datasets and benchmarks
TL;DR: We build a dynamic benchmark to evaluate LLMs' ability to forecast future events

Abstract: Forecasts of future events are essential inputs into informed decision-making. Machine learning (ML) systems have the potential to deliver forecasts at scale, but there is no framework for evaluating the accuracy of ML systems on a standardized set of forecasting questions. To address this gap, we introduce ForecastBench: a dynamic benchmark that evaluates the accuracy of ML systems on an automatically generated and regularly updated set of 1,000 forecasting questions. To avoid any possibility of data leakage, ForecastBench is comprised solely of questions about future events that have no known answer at the time of submission. We quantify the capabilities of current ML systems by collecting forecasts from expert (human) forecasters, the general public, and LLMs on a random subset of questions from the benchmark ($N=200$). While LLMs have achieved super-human performance on many benchmarks, they perform less well here: expert forecasters outperform the top-performing LLM ($p$-value $<0.001$). We display system and human scores in a public leaderboard at www.forecastbench.org.

Published as a conference paper at ICLR 2025
FORECASTBENCH: A DYNAMIC BENCHMARK OF AI FORECASTING CAPABILITIES
Ezra Karger
Houtan Bastani
Forecasting Research Institute
Forecasting Research Institute
Federal Reserve Bank of Chicago ezraã fo forecas st igg.seercc..... org
houtan& forecast www.aassssnnneeerrc...... org
Chen Yuch-Han x New York University yc7592enyu.edi
Zachary Jacobs
Forecasting Research Institute
Danny Halawi
Fred Zhang
University of California, Berkeley dhalawi @wwrkeeey....... edu
University California, Berkeley z0 z0@berkeley.co edu
Philip E. Tetlock
Forecasting Research Institute University of Pennsylvania
getlock@wharte upenn.
ABSTRACT
Forecasts of future events essential inputs into informed decision-makee makinng. Machine learning (ML) systems have the potential to deliver forecasts at scale, but there is framework for evaluating the accuracy of ML systems standardized set of forecasting questions. To address this gap, we introduce ForecastBench: a dynamic benchmark that evaluates the accuracy of ML systems generated and regularly updated set of 1,000 forecasting questions. To avoid any possibility of data leakage, ForecastBeench comprised solely of questions about future events that have no known answer at the time of submission. We quantify the capabilities of current ML systems by collecting forecasts from expert (human) forecasters, the general public, and LLMSs on random subset of questions from the benehmark (N 200). While LLMS have achieved super-huma performance many benchmarks, they perform less well here: expert forecasters outperform the top-performing LLM (p-value < 0.001). We display system and human scores in a public leaderboard at www.forecastbench.org
automatically
INTRODUCTION
Forecasting the future is challenging but important task (Armstrong, 2001; Tetlock and Gardner, 2015). Economic forecasts influence investment and hiring decisions (Christensen et al., 2018). And forecasts of the Covid-19 pandemic in early 2020 prompted local lockdowns slow the spread of the virus (Adam, 2020). However, human forecasting is often expensive, time-consuming, applicable only in specific domains, and subject to concerns about human biases. Motivated by these limitations, recent work explores the use of machine learning (ML) models, especially large language models (LLMs), in automated forecasting (Fluri al., 2024; Halawi al., 2024; Hendrycks et al., 2021; Phan et al., 2024; Pratt et al., 2024; Yan et al., 2024; Zou et al., 2022).
'qqual contribution. Corresoondence to
org.
The views expressed in this paper do not necessarily reflect the views of the Federal Reserve Bank of Chicago the Federal Reserve system.

---

Published as a conference paper at ICLR 2025
To assess LLMs' forecasting capabilities, prior work built static benchmarks of questions where the answer was known (resolved) after the knowledge cut-offs of the LLMs they test (Halawi et al., 2024; Yan et al., 2024; Zou et al., 2022). For example, "Will nuclear weapon be detonated in 2023," though resolved on Jan 1, 2024, is a valid out-of-sample question for testing a model with a known knowledge cut-off before the end of 2023.
This static evaluation methodology comes with three key drawbacks. First, as the knowledge cut-off's of frontier models are updated, static benchmarks become obsolete, necessitating further data-sourcing. This makes it difficult to continuously track and compare the top models in the field. Second, knowledge cut-offss are usually estimated using the time range of pre-training data. Such estimates may not be accurate, and post-training may inject further post-cutoff knowledge into the model, contaminating the test sets. Lastly, model developers face financial incentives to exaggerate their accuracy on benchmarks and claim that their models are ttate-of-the-art performers. While some fudging easily identified, other subtle benchmark manipulation or overfitting harder to catch, and some studies show significant evidence of benchmark contamination and/or memorization by popular LLM models (Elazar et al., 2024; Li et al., 2023; Roberts et al., 2023).
To address these drawbacks, we introduce ForecastBench, a dynamic benchmark that is continuously updated with new questions about future events. Our automated system gathers new questions from nine sources on a daily basis. These sources include prediction markets, forecasting platforms, and real-world time series. We regularly elicit predictions on these questions from both LLMss and human forecasters. As they resolve, we update public leaderboard to show participant performance. This process makes ForecastBench an accurate real-time benchmark of forecasting ability.
Our initial benchmark consists of 1,000 standardized forecasting questions randomly sampled from a much larger real-time question bank. To establish baseline levels of performance, we record predictions from dozens of LLMs on the initial set, using methods like boost performance (Halawi et al., 2024; Lewis et al., 2020). We independently elicit predictions from two groups of human forecasters: the general public and seasoned forecasters (superforecasters) (Tetlock and Gardner, 2015) who have consistently performed well in competitive forecasting tournaments. As questions resolve, we rate the submissions of registered models and the human comparison groups, updating our public leaderboard.
to
Our initial results indicate that state-of-the-arr models, such as Claude-3.5 Sonnet and GPT-4 Turbo, perform only roughly as well as simple median of forecasts from survey of humans with no (or minimal) forecasting experience, even when the LLMs are augmented with news retrieval and prompt engineering and when they have access to forecasts from human crowd (on market-based questions). The models also perform significantly worse than the median forecast of superforecasters.
These findings leave significant room for researchers to improve Al- forecasting systems using innovative approaches, such as developing methods for continuously updating models with current events and enhancing LLMs to reason over extended time frames. To support progress in this area, we publish an auxiliary dataset of LLM and human forecasts, rationales, and accuracy for use in future LLM fine-tuning and testing.
1.1 RELATED WORK
Automated forecasting systems that make accurate forecasts can help inform human decision- making (Hendrycks et al., 2021; Schoenegger et al., 2024a). Recent work studies the use of LLMs for automated forecasting (Halawi et al., 2024: Jin et al., 2021; Pratt et al., 2024; Yan et al., 2024; Zou al., 2022). These papers all build static benchmarks of resolved questions. A recent paper from Halawi et al. (2024) uses questions resolved between June 2023 and January 2024. Unfortunately, the latest LLMS have knowledge cut-offs past this point. This fact motivates our work to build a dynamically updating benchmark that can accurately evaluate frontier model performance.
In addition, Schoenegger and Park (2023) and Abolghasemi et al. (2023) compare the accuracy of GPT-4 and other LLMs to human forecasters. Schoenegger et al. (2024b) found that ensemble of 12 LLMS rivaled human forecasts in a forecasting tournament with a small number of questions, limiting the study's statistical power. Unlike work, that tournament was only once and is no longer updated. Also, our much larger question set allows us to make precise statistical claims about the performance of L.LMs relative to each other and to humans.
2

---

Published as a conference paper at ICLR 2025
Finally, recent work focuses the use of LLMs and transformer- based systems in statistical time- series forecasting (Das et al., 2024; Dooley et al., 2023; Goswami et al., 2024; Gruver et al., 2023; Jin et al., 2024; Nie et al., 2023; Rasul et al., 2023; Woo et al., 2024), but many of the most important forecasting questions do not have well-defined time series that can be used in standard statistical forecasting models (e.g., what is the probability that nuclear weapon will be used offensively in the next ten years?). Our benchmark is more general, and evaluates forecasting performance across questions with and without underlying time series and historical baseline data.
Language model evaluation Evaluating highly capable LLMs is challenging task-with many models saturating key benchmarks soon after their release (Maslej et al., 2023; Owen, 2024) and with benchmarks potentially leaked to models' training data (Balloccu et al., 2024; Jacovi et al., 2023; Jiang et al., 2024b; Magar and Schwartz, 2022; Sainz et al., 2023; Xu et al., 2024a;b; Zhang et al., 2024). Our dynamic forecasting benchmark avoids both of these issues. First, automated forecasting highly unsaturated: Halawi et al. (2024) showed that under simple zero-shot evaluation, frontier models such as GPT-4 are much less accurate than aggregates of human predictions. Second, our benchmark is dynamic. The final resolution of each question is only determined in the future and cannot be leaked in any training data (by construction). This eliminates concerns of contamination.
PRELIMINARIES
Forecasting A forecasting question asks for probabilistic prediction of a future event. A forecaster may assign probabilities to potential outcomes of the event. Forecasting platforms, including prediction markets, host wide range of questions. Many questions are of public interest, such as "Will Democrat win the 2028 US presidential election?"
Metries For binary questions, we use the Brier score as the performance metric, defined as o) where f E [0, 1] is the probabilistic forecast and E {0. 1) is the outcome. Lower Brier scores are better, and score of 0.25 associated with the uninformed forecast of 0.5. Brier scores are strictly proper, incentivizing truthful reporting from participants.
Models We evaluate 17 LLMs on our initial benchmark: GPT-3.5- TeTT335-Tuurb-Bee Turbo-Instruct (Brown et al., 2020), GPT-4 (OpenAI, 2023), GPT-4o, Llama-2-70B (Touvron et al., 2023), Llama-3-7B, Llama-3- 70B, Mistral-7B (Jiang et al., 2023), Mixtral-8x7B (Jiang et al., 2024a), Mixtral-8x22B, Mistral-Large, Brenl.5-110B-CCS (Bai et al., 2023), Claude-2.1 (Anthropic, 2023), Claude-3- Claude-3.5- Sonnet, Claude-3-Opus (Anthropic, 2024), Gemini 1.5 Flash, and Gemini 1.5 Pro (Gemini Team, 2023). We outline the various baselines we run with these models in Section 5.
BENCHMARK, LEADERBOARD, AND DATASETS
We maintain a question bank with growing of forecasting questions. Continuously adding questions to the question bank allows it to stay relevant and ensures that we have a large selection of unresolved questions sample from.
Every night, our automated system updates the question bank with new questions and resolution values. We drop invalid, low-quality, and resolved questions, categorizing the remaining questions by topic. The process is fully automated, as detailed in Section 3.1.
Every two weeks, we sample from the question bank and release question for those interested in submitting their forecasts to the benchmark. We also survey a standard set of LLM-based models to allow for comparisons of performance over time. Submitted forecasts are resolved continuously with daily updates to our leaderboard. We provide the resulting data on forecast questions and submissions to researchers. See Section 3.2 for details. Finally, we discuss the question resolution procedure in Section 3.3 and our leaderboard design in Section 3.4.
3.1 QUESTION BANK
In an automated process that runs nightly at 0:00 UTC, questions are added to the question bank, resolution values are updated, and metadata generated. Details follow.
3

---

Published as a conference paper at ICLR 2025
3.1.1 QUESTIONS AND RESOLUTION VALUES
We bring in questions from two broad types of sources: markets and datasets.' ! We select multiple, reliable sources from each type, ensuring the diversity of our benchmark. See Table for details,
Markets We compile market questions from handful of prediction markets and forecast aggrega- tion sites that elicit human predictions on questions across wide range of topics. When selecting questions from market sources to add the question bank, we choose those with high levels of active human trading on these platforms (liquidity) as these questions tend to be of higher quality than those with low levels of human trading. We store the latest market and resolution values for each question.
Datasets We create dataset questions from well-established and well-maintained datasets that track real-world events (e.g., ACLED (Raleigh et al., 2023), geopolitical database that tracks worldwide conflict, including facts like the number of protests in Niger each month). With these sources, we can generate questions using fixed question template (c.g.. "Will the number of protests in Niger increase by at least 10% over this month's 's value by the resolution date?").
Question Bank Table lists the sources of market and dataset questions, along with the number of questions available in our question bank, whence we sample questions for our benchmark runs. In addition to the main questions (with sample size N), we also construct additional combination questions by choosing pairs of questions within each source. We describe combination questions in more detail in Section 3.2.
Table 1: Question bank composition, grouped by source type (market dataset).
Source
URL
N
RFI
aandforecastingntiaatveeorr
18
153
Manifold Markets
manifold.marken
405
81,810
Metaculus
metaculus.com
722
260,281
Polymarket
polymarket.com
915
418,155
Market Total
2,060
760,399
ACLED
acleddata.com
3,220
5,182,590
DBnomics
db.nomics world
52
1,326
FRED
fted.stlouisfed.org
166
13,695
Wikipedia
wikipedia.oo
428
91,378
Yahoo! Finance
finance. yahoo.com
509
129,286
Dataset Total
4,375
5,418,275
Question Bank Total
6,435
6,178.674
3.1.2 QUESTION METADATA
After we automatically update the question bank with new forecasting questions and resolution values, the data are processed in several ways. This generates more information about the questions and creates another sampling option when creating the question set. Following Halawi al. (2024), use gpt-4o-mine i. to categorize questions by subject and to filter out low-quality questions. We report the number of questions by category and source in Table 15, where we display the breakdown of the standard questions from our question bank (N from Table 1) across question categories by source, after removing low-quality questions.
Licensing information can be found in Section C.I.
"Hereafter, for brevity, refer to questions that come from both prediction markets and forecast aggregation sites "market" questions. We also refer theses sources collectively as "market" sources. 'ee Table 8 for an example question pulled from market source.
*See Table for example question pulled from dataset source.
4

---

Published as a conference paper at ICLR 2025
3.2 QUESTION SETS
LLM question set We release of 1,000 forecast questions for L.LMs every other Sunday midnight UTC. We sample an equal number of questions from each source to ensure representative- ness. Within each source, we then uniformly sample questions across all question categories, aiming for an equal distribution from each category within each source. This ensures that models cannot be overfit to a specific type of question or topic. Limiting the number of questions generated to 1,000 also ensures that costs for testing LLMs on the benchmark are capped for development teams with fewer resources.
Human question set The human question set is comprised of 200 forecast questions sampled directly from the LLM question set. When sampling, we do our best to maintain proportionality across question sources and across categories within each question source; this ensures the question set addresses as many domains as possible.
Forecast horizons For questions derived from dataset sources, the distribution of resolution dates the forecast duc_ due_date date + n days, where n E {7. 30, 90, 365, 1095, 1825, 3650). In other words, for each dataset question we ask for 8 forecasts that differ only in their resolution date. For questions derived from market sources, we ask for only forecast: the probability that each question will resolve positively (will the event underlying the question occur, or not). This setup will allow us to evaluate both human and LLM forecasting performance over the short, medium, and long term.
Combination questions There are two types of questions, each comprising half of the question set. The first type is a standard forecasting question with a binary outcome, e.g., "Will inflation (core CPI) be above 3% next month?" We construct the second type, combination questions, by pairing two standard questions. For combination questions, we ask for forecasts on all Boolean combinations of the two questions (i.c., P(Q1 r Q2). P(Q1 n -Q2), ...). Considering the extensive number of standard questions and potential combinations in question bank (as we could potentially combine 3, 4, or more standard questions together), we effectively have access to millions of possible forecasting questions from which
sample. We show the number of two-question combination questions in the question bank as it stands at time of writing in the right-hand column of Table 1. This setup implies that for market combination questions, each forecaster will provide 4 forecasts, whereas for dataset combination questions, each forecaster will provide 32 forecasts (4 for each Boolean combination of Q1 and Q2 at each of the forecast horizons).
Combination questions require forecasters to consider the covariance structure of different events, some of which are more independent than others. For instance, the best forecasts of whether the S&P500 (a key U.S. stock market index) will reach an all-time high and whether Spain will win the next Men's 's World Cup are likely independent. However, the best forecast of whether the S&P500 will reach all-time high and whether the U.S. will enter an economic recession must account for the likely strong correlation between these events.
Of the 1,000 questions in the LLM question set, 500 are combination questions. Each combination question is composed of two standard questions from the same question set. This means that LLMs will also provide forecasts for the individual components of these combination questions, since they're 're drawn from the existing standard questions. Importantly, none of the 200 questions in the human question set are combination questions.
Timeline of forecasting round To compare human performance LLMs, periodically run surveys asking the general public and superforecasters to forecast on our question sets (see Section 4). We produce the question sets 10 days before the forecast due date to allow for time to create and run a human survey. LLM teams receive their question set 24 hours before the due date, even though was generated at the same time the human question set. This constrained time frame gives teams less time to be able to game the system. We thus elicit forecasts, obtaining comparable forecast sets on the due date from both LLMs and humans who faced the same information environment.
'We exclude combination questions hecause expert human forecasters expensive; we maximize their relevance by having them focus on standard questions.
5

---

Published as a conference paper at ICLR 2025
3.3 RESOLUTION
We resolve forecast nightly by gathering the latest information about which events have have not occurred. All questions are ultimately resolved to ground truth.
Evaluating performance on market questions For market questions, ground truth not available until the question has been resolved on the platform. Until then, evaluate performance by calculating the squared distance between the forecasted value and the platform's 's crowd forecast (an aggregate of human forecasts reported the platform) from the preceding day. This provides good estimate of forecast performance on unresolved questions since crowd forecasts tend toward ground truth the resolution date approaches.
Once market question has officially been resolved, we score the forecast against the resolution value, creating definitive score for the question. We are thus able to estimate forecast performance on the entire set of market questions (resolved and unresolved), incorporating all information available to markets on a nightly basis.
Evaluating performance dataset questions Datasets can be updated new information becomes available. Hence, questions derived from datasets are continuously resolved to the value from the latest available data. As the resolution dates (ranging from a 7-day a 10-year horizon) for dataset questions come due, a new round of forecasts is evaluated. We thus are able to evaluate forecasting performance over different time horizons.
Missing forecasts We select 1,000 questions for the LLM question set make the forecasting task impractical to complete manually within the 24-hour window after the question set is released. And we obligate all LLMs forecast on all questions to ensure comparability of scores across models and human- based aggregates. When model does not submit forecasts on certain questions or time horizons, we consider that model error and impute naive forecast for the model to ensure comparability across models over time.
For market questions, since we only ask for forecasts on the outcome of the question, missing forecasts are assigned the value of the crowd forecast on the forecast due date. Some may argue this is overly-generous, but we did not want forecasters to have a competitive advantage by simply scraping the market websites themselves.
For dataset questions, we impute the value 0.5 (which represents an uninformed 50% forecast) to forecasts across all time horizons. Empirically, top models report valid forecasts all questions and are not affected by this imputation procedure.
3.4 LEADERBOARDS
We generate leaderboards that rank models and humans by average overall score. The main leader- board highlights the top forecasting submission across all questions and can by sorted by performance on the question type (market dataset) and by resolution status (resolved or unresolved). The leaderboard is updated nightly after scoring forecasts against the latest data, market resolution values, and crowd forecasts.
3.5 DATASETS
As key output of ForecastBench, we generate four datasets that grow over time.?
General public forecast dataset Every time we run the public survey described in Section we provide multiple independent forecasts and rationales for every one of the 200 forecast questions and report the accuracy of the median public forecast. See Section B.2.1 for details.
TThe crowd forecasts are updated nightly in the Question Bank as described in Section 3.1.1. "ee Appendix A for licensing details and Appendix B for data dictionaries.

---

Published as a conference paper at ICLR 2025
Superforecaster forecast dataset In addition to forecasts and rationales, the superforecasters provide pertinent information about their forecasting process, like search terms used and useful URLs consulted. See Section B.2.2.
LLM forecast dataset Similar to the general public dataset, we ask LLMs to produce forecasts on each of 1,000 forecast questions in the LLM question set. Their rationales are also included in the dataset whenever provided. See Section B.2.3 for details.
Question & resolutions dataset In creating the benchmark, we have automated question creation and resolution from all of the sources outlined in Table We provide these as a dataset that can be combined with the forecast datasets mentioned above. See Section B.1 for details.
HUMAN FORECASTER BASELINE
To compare LLM forecasting performance to human performance, we ran surveys of two different groups: the general public and superforecasters. We scored each group's median forecast, treating as representative of its overall performance.
4.1 GENERAL PUBLIC
We recruited 500 human forecasters via Prolific and advertisements on Facebook to participate as representatives of the general public. These human subjects completed brief introductory survey to gather demographic information* and evaluate performance on few forecasting and comprehension tasks. They then completed one-hour survey containing 20 random questions from the 200-item human question set descrihed Section 3.2, providing their forecasts and rationales for each question.
The number of responses per question varied to ensure representativeness across categories and sources; at least 40 responses were gathered per question, averaging 49 responses per question.
4.2 SUPERFORECASTERS
We recruited 39 superforecasters, who have strong track record of accurate performance on a diverse set of geopolitical questions, to participate in 9-day forecasting experiment in which participants were prompted to give their individual forecasts for random questions from the same 200-item human question set described above. Roughly halfway through the 9-day experiment, participants were moved into group forecasting stage in which we allowed them to see one another's 's forecasts and rationales and to communicate about each question. They were also given the opportunity to forecast on questions beyond the 20 questions assigned to them individually.
Because of the lower number of superforecasters, questions generally had fewer responses than in the public survey; a minimum of forecasts were recorded for each question, with an average responses per question."
5 LLM BASELINE
In this section, evaluate the forecasting capabilities of LLMS and report the methodology and results.
5.1 METHODOLOGY
We evaluate suite of mnstruction-folloiing chat models without any additional fine-tuning (see Section 2 for details on the models). For each baseline outlined below, we prompt the model to generate a probabilistic forecast that the question will resolve "Yes."
*See Appendix I. for an overview of public participant demographics.
"See Figure for example question from the human surveys.

---

Published as a conference paper at ICLR 2025
Baselines We implement seven baselines: (1) zero-shot prompting; (2) prompting with scratchpad instructions; (3) prompting with scratchpad instructions and retrieved news articles; (4) zero-shot prompting with crowd forecasts; (5) scratchpad prompting with crowd forecasts; (6) scratchpad prompting with retrieved news articles and crowd forecasts; and (7) aggregating predictions from multiple LLMs. Each baseline is described in more detail below.
1 Our first baseline prompts the model zero-shot to generate a forecast directly without generating other content, such as intermediate thinking (Figure 4). By prompting the model to output its forecast directly, we assess raw forecasting capability without sensitivity to prompting strategies.
2 Our second baseline, prompts the model with scratchpad instructions (Nye et al., 2021) that outline a procedure the model should use to reason about the question (Figure 5). Our scratchpad prompt comes from (Halawi et al., 2024), which formed its prompts through a combination of analyzing the Brier score as prompt changes were made, and by adding language to fix common errors the LLMs would make, e.g., asking them to rephrase the question for understanding.
3 Since LLMs' knowledge is not continuously updated, it important to provide them with up- to-date information relevant to the question (Zou et al., 2022). Our fourth baseline, scratchpad with news, uses the same scratchpad prompt as above, supplemented with retrieved news articles. The retrieval system is the same as described in Halawi et al. (2024): an LLM generates search queries for a news API, filters articles for relevancy, and summarizes the articles.
4
The question sets we provide to LLMs contain what we term freeze values. For market questions these are just the crowd forecast on the market the day the question set was created, as described in Section 3.2. For dataset questions, these are baseline values relevant to the forecasting task. 10 Our third baseline is the zero-shot with freeze values. This simply the zero-shot prompt from Baseline 1 supplemented with the freeze value and an explanation of the freeze value. For examples of the freeze value and its explanation, see Table and Table 9.
5 Our fifth baseline is the seratchpad with freeze values (the scratchpad prompt from Baseline 2 supplemented with freeze values as explained in Baseline 4).
6 Our sixth baseline is the scratchpad with news with freeze values.
7 In our final baseline, we aggregate the predictions generated by LLMs into an LLM "ensemble" forecast. We do this as Metaculus (2023) shows that an ensemble of all forecasters consistently outperforms using just the top 5, 10, .... 30 best forecasters (based on past scores).
To produce the LLM ensemble forecast, we use models (GPT-4o, Claude-3.5-Sonnet and Gemini- 1.5ini-1.5-Proo ..5) and 3 prompts crafted by superforecasters (Figure 6, Figure 7, and Figure 8). This results in forecasts per question. We generate 3 LLM ensemble baselines using the median, geometric mean, and geometric mean of log odds (Satopää et al., 2014). For details, see Appendix E.
5.2 RESULTS
Comparing humans and LLMs In Table 2, we show that superforecasters achieve an overall mean Brier score of 0.096, significantly outperforming both the general public (Brier = 0. 1,12,, p < 0.001) and the top LLM performer the 200-item subset (Claude 3.5 Sonnet: Brier 0.122, p < 0.001)." The top-performing LLMs all had access to the crowd forecast on market questions (the "freeze values" from Baselines 4, 5, and 6 above). The pop-performing model without access to the crowd forecast on market questions was less accurate than models with access to the human forecast with Brier score of 0.136. The comparison between humans and LLMs relies on the 200 questions forecasted by humans, which is random sub-sample of the 1,000 questions in the question set provided to LLMs (excluding combination questions)."
For example, a question generated from a Wikipedia page about whether a chess player's Elo rating will increase by a given date, the freeze value is the chess player's Elo rating on the question set generation date. An explanation of what the freaze value represents is also provided.
See statistical note in Appendix G.
"ccurrayy measures are based on more than 200 forecasts because human and LLM forecasters submitted multiple forecasts on each dataset question, one for each time horizon. The results presented here include forecasts over the 7-, 30-, 90-, and 180-day time horizons.
8

---

Published as a conference paper at ICLR 2025
Table 2: LLM/Human Leaderboard (top 10)
Biner Score 1
Model
Organization
provided
Information
Prompt
Dataset (N
Market (Nv76)
Overall (Nm495)
Confidence Interval
Prarwise povauue comparing than No.
1
Superforeeesttrr median feeoesst
ForecnutRench
0.118
0.074
0.096
[0,076, 0.116]
0%
Public mediam furecad
FunecasiBench
0.153
0.089
0.121
[0.101, 0,141]
<0 00.00
22%
2land-3-5--Saa 20240630
Anthropic
Scratchoul
0.138
[0,099. 0.145)
<0.001
31%
Clande Socnet 20240620
Authropic
News freeze values
Scratchpad
0.142
(0.104.0.150)
<0.001
29%
GPT-4 Turbo 2024 Or-
OpenAl
Freeze values
Zero shot.
0.162
0.085
0.128
[0.105, 0.151]
<0.001
32%
Clande-3-3 103--Soae--200-44000
Authuopic
Freeze valses
Zero shot
0.145
0.117
0.131
[0.105. 0.159]
<0.001
51%
GPT-4
OpenAf
Freeze values
Zero shot
0.167
0.096
0.132
[0.109. 0.155
<0.001
51%
GPT-44
OpenAl
News with froeze values
Scratehgud
0.162
0.105
0.133
[0.113, 0.154
<0.001
25%
Claude 3-5 3-5-ol-et----tte------- 202-40000
Anthropic
Scratchpul
0.138
0.133
0.136
[0,113, 0.158)
<0.00 00.001
28%
GPT-40
OpenAl
Freeze values
Scratchpad
0.161
10,115,
<0.001
27%
Nanw:
on standard questions powided
quesioo
7- 30- and 150-dx d0- forecast hurizons, See Tuble tup 50.
2 The full laadrbuard avaiable www.oomeassee woforresbeenchh.. hench.nng Online reslls updated nighily.
slighily dillerent than vensinn prevensed here. For resolved market questions, forocasts are campared agminet grmind tuth while for unicsolied mario ket quesnioss, they are compared community aggregutes,
ovengee fomeaaterr under the null hypulhesis that there's no ditlerenes.
overall score calealated
dataset Brier seare by hootstrapping
mean market diffesenees ovenall between model best
loore.
Paawise
The
6. Pot. more accurate than 1: The percens of questions where this forecaster had a better overall score than the bes, forecaster.
As particular failure mode, we find LLMs are significantly worse at combination questions. Al- though our human surveys did not explicitly ask for forecasts combination questions, we bound human performance by assuming independence of the component of each combination question. This underestinates human accuracy because a human forecaster predicting the outcome of a combination question could account for dependence between the permuted events. In Table 20, we present this comparison of human and LLM forecasts. We see that LLMs perform poorly these combination questions, and including them in the benchmark widens the gap between human and L.MM perfor- mance: superforecasters (Brier 0.076) outperform the general public (Brier 0.096) and the top LLM (GPT-4o, Brier = 0.130) significantly. To benchmark the size of this gap in performance, the 0.054 Brier score gap in performance between superforecasters and GPT-4o is significantly larger than the 0.026 gap in performance between GPT-4o and GPT-4.
Table 3: LLM Leaderboard (top 10)
Brier Soore J
Model
Organization
Information
provided
Pronge
Danaset
(2V=5,192)
Market (N44977
(N-6,389)
Overall
Coufidence Inserval
Pairwise p-value comparing than
Pot. more accurate
No.
Claude-1-5- Saalll-----Smmt---00 2020000222220000
Anhropic
Freeze values
Scratchpud
0.169
0.078
0.123
[0.117, 0.129]
[/%
GGPT-4-Trho-2124..00
OpenAl
Freeze values
Scratchpud
0.172
0.080
0.126
[0.120], 0.132]
0.096
43%
GPT-40
OpenAl
Preeze values
Scranchpod
0.186
0.009
0.128
0.133)
43%
Gemini-1.5-Pr
Google
Freeze values
Scraschpad
0.162
0.134
0.139)
35%
GPT-4o
OpenAl
News with freeze values
Scratchpud
0.190
0.084
0,137
[0.131,
<0.001
39%
Gemini-1 1.5-
Gougle
News with freeze values
Scratchpud
0.166
0.111
0.139
10.133, 0.144]
<0.001
315
Claude 3- 3-Dpps-200000 202440299
Anthropic
Freeze values
Zero shoe
0.185
0.093
0.139
[0.133, 0.146)
<00.001
41%
Owent. JOB Chat
Owen
Freeze values
Scratchpad
0.176
0.108
0.142
10.136, 0.148)
<0.001
30%
Claude 3-5 Sonnet 40240620
Amhropic
News freeze
Scratchpod
0.184
0.101
32%
Sanule----Sommet 200244000000
Anthropic
Freeze values
Zero shot
0.192
0.094
0.143
[0.136, 0.150]
<0.001
42%
Notes:
1. Shows perfommuce on the 1,000 (500 standard, 500 combitation) questions in the L.IM question set at the 7-, 30. 90-, and 1801-day forcesst horizons. See Table 19 for
slightly dificrent than the vension presensed here. 3 For resolved market question, lorecusts are compured aganns gruund suih while for unresolved market questions, they are compared community aggregates. Pairwise
full leaderboard is avaiaall www. footootbeencc..n.. Online results updated nighlly,
ovenal
calealated as ovange comparing
menh dotaso Brier scare and mean mar marker ccoee diffesenees ovenll between model best
by hootstraping
forocoster uider the mull lypaihesis that there's no difference. 6 Pet. mare securate thnn I: The percest of questions where this forceaster had a better overall seare than the best forecaster.
Comparing LLMS Table 3 excludes humans and evaluates LLMs on the entire question set
(N=1,000 questions). Here we see similar ranking of models, with Claude 3.5 Sonnet slightly outperforming GPT-4 Turbo. As in Table 2, most of the rop-performing models use the scratchpad prompt (Figure 5) and use as inputs the human crowd forecasts for market questions. Access to recent topical news related to the questions did not improve performance.
LLM performance and forecasting accuracy Figure la demonstrates the seemingly linear rela- tionship between Chatbot Arena scores (Chiang et al., 2024) and the overall Brier score from Table 2. We observe a significant correlation (r = -0.68, p = 0.003), indicating that models with higher Arena scores tend to produce more accurate forecasts. The linear relationship implies that LLMS

---

Published as a conference paper at ICLR 2025
Brier score (lower is better)
Brier score (lower is better)
uhe txl
SPy
010
UGA Losc Laze
210
Sambroauee
000
130
1100
-200
1300
1400
15:00
1900
1840
1ex25
1ex00
Texd7
texds
Chatbot Arena score (higher is better)
Log training compute (FLOP)
(a) Brier score vs. Chatbot Arena
(h) Brier score vs. Log training compute
The graphs show the linear relationship between the Brier scores from Table 2 and (a) Chatbot Arena scores and (b) estimates of training compute. The dotted blue line represents the Superforecasters' overall Brier score. A red dot with bootstrapped 95% confidence interval is placed at the intersection of this dotted blue line with the dashed linear fit line to demonstrate the potential intersection of LLM Arena score/training compute and Superforecaster-leed forecasting performance. For (b), if estimates from Epoch AI (2024) were not available, we produced estimates following attps: epoch. a ai/blog/ herploo//ee eestimmttttsti weteiiinntttnnnnnnn
Figure 1:
compute. The trend trend-line in (a) is y 0.506 0.000298x (R² 0.47) and in (b) it is y = 0.844 0.01213z (R² = 0.41).
could match superforccaster performance when the Arena score approaches 1406 (bootstrapped 95% Cl: 1346-1633).
Figure lb shows the log-linear relationship between estimated training compute and the overall Brier score from Table 2. Projecting out the log-lineat linear relationship, we find that LLMs could match superforecaster performance when training compute approaches 6.49 1026 though there is large confidence interval (bootstrapped 95% Cl: 9.69 x 1025 x 102*) given the marginally significant relationship (r 00.7,, p 0.046).
DISCUSSION
We introduced ForecastBench, a dynamic and continuously updated benchmark for evaluating LLM forecasting capabilities. By focusing exclusively on questions that are unresolved the time of submission, we eliminate the risks of data leakage and ensure a robust evaluation environment. Our initial results demonstrate that while state-of-the-arr LLMs exhibit promising potential, they underperform superforecasters. This performance gap highlights the challenges in leveraging current LLMMs for accurate, real-time forecasting.
We produce public leaderboard listing the real-time accuracy of top LLMs and humans well as standardized dataset of forecasting questions and rationales. Future work should leverage this auxiliary dataset of predictions and rationales to fine-tune models, explore new architectures, and develop adaptive systems better suited for general reasoning in dynamic, real-world environments. Ultimately. ForecastBench serves step toward harnessing the full potential of Al-based systems for forecasting and decision decision-making making.
REPRODUCIBILITY STATEMENT
One reason we' ve open-sourced code (link in Appendix A) is to allow for independent verification of our results. See Appendix for reproducing the human forecast sets, Appendix for reproducing L.M forecast sets, and Appendix K for resolving the forecasts and creating the leaderboard.
10

---

Published as a conference paper at ICLR 2025
8 ETHICS STATEMENT
Human survey subjects in both the public and superforecaster surveys are made aware prior to their participation the study via an informed consent form (approved by our IRB, number 855431) that their forecast/rationale data may be publicly released and used to train large language models or other Al systems, with said data carefully reviewed and anonymized.
We have manually reviewed text provided by human participants ensure that personally identifiable information released as part our human forecast datasets, per IRB requirements. Similar manual reviews of text data will take place part of every future human forecasting round.
ACKNOWLEDGMENTS
We thank Open Philanthropy for providing a grant to fund this work, allowing us to maintain the benchmark least until mid-2027.
We thank Otto Kuusela for helpful discussions, Sam Glover and Molly Hickman for writing super- forecaster prompts, and Harrison Durland and Victoria Schmidt for their research assistance.
REFERENCES
Mahdi Abolghasemi, Odkhishig Ganbold, and Kristian Rotaru. Humans large language models: Judgmental forecasting an era of advanced AI. arXiv preprint arXiv:2312.06941, 2023.
David Adam. Special report: The simulations driving the world's response to COVID-19, Nature, 580(7802):316-319 2020.
Anthropic.
Model card and evaluations for Claude models,
2023.
https: //t::///wwwww www cán anthropic http://wwww.caaaaaooo.oo.....om http://////eeeeeeerrrroooooo// wttbziitebbbwbbsie//// bccfccee7988ceef0077796222222662
pdf.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. //ww://d//wwwcn....
anthropic com/ de @baabblbbel 9ab.00cccaaooocooo htt.ce/ccccccabacc000000001d11dd 10121886666727//211111 /Model Card_Claude_3.pp
Jon Scott Armstrong. Principles of Forecasting: a Handbook for Researchers and Practitioners. Springer, 2001.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi
Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang. Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023. URL https:/ //txxvxxvvv/./////v///e////aaavl wwwrxivv.oog/eee org/abs 2309. 6609.
Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek. Leak, cheat, repeat: Data contamination and evaluation malpractices in closed-soure LLMs. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL), 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
11

---

Published as a conference paper at ICLR 2025
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng. Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang. Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arema: An open platform for evaluating Ilms by human preference, 2024. hwtptttttttaii..../////a//// abs/2403 04 132.
Peter Christensen, Kenneth Gillingham, and William Nordhaus. Uncertainty in forecasts of long-run economic growth. Proceedings of the National Academy of Sciences, 115(21):5409-5411 2018.
Abhimanyu Das, Weihao Kong, Rajat Sen, and Yichen Zhou. A decoder-only foundation model for time-series forecasting, In International Conference on Machine Learning (ICML), 2024.
Samuel Dooley, Gurnoor Singh Khurana, Chirag Mohapatra, Siddartha Venkat Naidu, and Colin White. ForecastPFN: munthetically-trainee zero-shot forecasting. In Advanced in Neural Informa- tion Processing Systems (NeurIPS), 2023.
Yanai Elazar, Akshita Bhagia, Ian Helgi Magmusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, al. What's in my big data? In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Epoch AL Data on notable ai models. 2024. sataableaaimmdder
URL https:// a /
Accessed: 2024-11-22
Lukas Fluri, Daniel Paleka, and Florian Tramer. Evaluating superhuman models with consistency checks. IEEE Conference Secure and Trustworthy Machine Learning (SaTMMI)) 2024.
Gemini Team. Gemini: family of highly capable multimodal models. arXiv:2312.11805. 2023.
arXiv preprint
Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Shuo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In International Conference on Machine Learning (ICML), 2024.
Nate Gruver, Marc Anton Finzi, Shikai Qiu, and Andrew Gordon Wilson. Large language models are zero-shot time series forecasters. In Advanced in Neural Information Processing Systems (NeurIPS), 2023.
Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching human-level forecasting with language models. arXiv preprint arXiv:2402.18552 2024.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML safety. arXiv preprint arXiv:2109.13916 2021.
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5075-5084, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL pestts::/arxxie org/abs/ 2210. 06825.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux. Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril. Thomas Wang. Timothée Lacroix, and William El Sayed. Mixtral of experts. arXiv preprint arXiv:2401 20000080 2024a.
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Investigating data contamination for pre-training language models. arXiv preprint arXiv:2401. 000099, 2024b.
12

---

Published as a conference paper at ICLR 2025
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y. Zhang, Xiaoming Shi, Pin- Yu Chen, Yux- uan Liang, Yuan-Fang Li, Shirui Pan, and Qingsong Wen. Time-LLM: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations (ICLRR, 2024.
Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. ForecastQA: question answering challenge for event forecasting with temporal text data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL), 2021.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
Yucheng Li, Frank Guerin, and Chenghua Lin. An open source data contamination report for large language models. arXiv preprint arXiv:2310.17588 2023.
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2022.
Nestor Maslej. Loredana Fattorini, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons. James Manyika, Helen Ngo. Juan Carlos Niebles, Vanessa Parli, Yoav Shoham, Russell Wald, Jack Clark, and Raymond Perrault. Artificial intelligence index report 2023. arXiv preprint
arXiv:2310.037155 2023.
Barbara Mellers, Eric Stone, Terry Murray, Angela Minster, Nick Rohrbaugh, Michael Bishop, Eva Chen, Joshua Baker, Yuan Hou, Michael Horowitz, Lyle Ungar, and Philip Tetlock. Identifying and cultivating superforecasters on Psychological Science, 2015.
method of improving probabilistic predictions. Perspectives
Metaculus.
Wisdom of the crowd vs. the best the best of the best, URL https:/ https::/www.meteett www.://taculss... wttp://waaw.....o/ooottooooooooo
2023. wisdom-on
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference Learning Representations (ICLR), 2023.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00111 2021.
OpenAl. GPT-4 technical report. arXiv preprint 0.XN::2303.08774,20000
David Owen. How predictable is language model benchmark performance? arXiv preprint arXiv:2401.04757, 2024.
Long Phan, Andrew Zeng, Mantas Mazeika, Adam Khoja, and Dan Hendrycks, Superhuman automated forecasting. https: : /www. safe ai wwtblag/bblaaaaaoaaea forecasting, aooeeaaa///boo///eceaaiiii 2024.
Sarah Pratt, Seth Blumberg, Pietro Kreitlon Carolino, and Meredith Ringel Morris. Can language models use forecasting strategies? arXiv preprint arXiv:2406.0444 2024.
Clionadh Raleigh, Roudabeh Kishi, and Andrew Linke. Political instability patterns are obscured by conflict dataset scope conditions, sources, and coding choices. Humanities and Social Sciences Communications, 10:74, 2023.
Kashif Rasul, Arjun Ashok, Andrew Robert Williams, Arian Khorasani, George Adamopoulos, Rishika Bhagwatkar, Marin Biloš, Hena Ghonia, Nadhir Vincent Hassen, Anderson Schneider, Sahil Garg, Alexandre Drouin. Nicolas Chapados, Yuriy Nevmyvaka, and Irina Rish. Lag- Llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.0822 2023.
13

---

Published as a conference paper at ICLR 2025
Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. Data contamination through the lens of time. arXiv preprint arXiv:2310.10628 2023.
Oscar Sainz, Jon Campos, Iker García-Ferrene Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation in trouble: On the need to measure Ilm data contamination for each benchmark. In Findings of the Association for Computational Linguistics (Findings of EMNLP), 2023.
Ville A. Satopää, Jonathan Baron, Dean P. Foster, Barbara A. Mellers, Philip E. Tetlock, and Lyle H. Ungar. Combining multiple probability predictions using a simple logit model. International Journal of Forecasting, 30(2):344 334-356, 2014.
Philipp Schoenegger and Peter S Park. Large language model prediction capabilities: Evidence from real-world forecasting tournament. arXiv preprint 1rXiv:2310. 2023.
Philipp Schoenegger, Peter S Park, Ezra Karger, and Philip E Tetlock. Al-augmented predictions: LLM assistants improve human forecasting accuracy. arXiv preprint arXiv:2402.07862. 2024a.
Philipp Schoenegger, Indre Tuminauskaite, Peter S Park, and Philip E Tetlock. Wisdom of the silicon crowd: L.LM ensemble prediction capabilities match human crowd accuracy. arXiv preprint arXiv:2402.19379, 2024b. 2024b..
Philip E. Tetlock and Dan Gardner. Superforecasting: The Art and Science of Prediction. Crown, 2015.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian. Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
Gerald Woo, Chenghao Liu, Akshat Kumar, Caiming Xiong, Silvio Savarese, and Doyen Sahoo. Unified training of universal time series forecasting transformers. In International Conference on Machine Learning (ICML), 2024.
Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. Benchmark data contamination of large language models: A survey, arXiv preprini arXiv:2406.0422 2024a.
Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. arXiv preprint arXiv:2404. 0000044.8822 18824, 2024b.
Qi Yan, Raihan Seraj. Jiawei He, Lili Meng, and Tristan Sylvain. Autocast++: Enhancing world event prediction with zero-shot ranking-based context retrieval. In International Conference on Learning Representations (ICLR), 2024.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXiv:2405.00332 2024.
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks. In Advances Neural Information Processing Systems (NeurIPS), 2022.
14

---

Published as a conference paper at ICLR 2025
A LICENSING AND MAINTENANCE
Hosting The latest Icaderboards, updated nightly, are available on www.forccastbench.org mentation is available on the repository wiki:
Docu- Datasets Our datasets, distributed under the CC BY-SA license, are available on hrrpccassbbencc.oo/ddtttteeee html. Historical updates the resolution datasets and leaderboards -attssesc-ddtsstss... Bi-weekly question sets also released via this repository. The repository is mirrored to Hugging Face and available at huggingfice.co/daatees///foo
are available on github.com/forecasttio
Codebase The code underlying automated system runs Google Cloud Platform and is available at
under the MIT license.
Participating Bi-weekly forecasting rounds are open to LLM teams. Instructions for participating are can be found on the wiki
Maintenance & long-term preservation We ensure the long-term availability and maintenance of the benchmark as it is funded by Open Philanthropy until mid-2027. no further funding is provided beyond that point, datasets will continue be made available on GitHub and Hugging Face.
B DATASETS
We intend our datasets to be used for training general LLMs, fine-tuning forecasting LLMs, and for any applicable research purposes. No restrictions are placed on who may use our datasets, nor to what end.
Availability
www.forcaattercc.og//aasssssss
B.1 QUESTION AND RESOLUTION SETS
Every question set will be published. Their resolutions will also be published such that there's a complete training set when combined with the forecast sets outlined Section B.2. The data dictionary for the question set is outlined in Table 4 and Table 5. The data dictionary for the resolution set is outlined in Table 6 and Table 7.
Data format The question and resolution datasets are released as JSON (. jsoo) files.
Ethical and responsible use
There are no restrictions on use of the question and resolution datasets.
Data collection Our question and resolution datasets have been pulled, and are updated, from various, public-facing sources. From those sources where the terms of use/service prohibit the redistribution of their information (currently, Manifold Markets and Metaculus), we have obtained explicit permission to do so. Before add new sources our growing dataset, we will ensure the ability to distribute questions and resolutions publicly. Data sources in our question bank can be found in Table 14.
B.1.1 EXAMPLES
Table 8, Table and Table 10 show concrete examples of the data dictionary detailed in Table 4.
B.2 FORECAST SETS
Every set of forecasts provided to ForecastBench is made public and all forecasts coming from the general public and superforecasters anonymized before release.
Each forecast set contains the header information outlined in Table 11, with all forecasts in an array called forecast.s. The forecast sets described in the following subsections.
15

---

Published as a conference paper at ICLR 2025
Table Question set data dictionary.
Field forecast_die due date
Description
Required
Data Type string
Date in ISO format. "2024-07-21"
e.g.
question_set
The name of the file that con- tains the question set. e.g. "2024-07-21- 11mmmm1.mmmm.mmmmmm json" A list of questions to forecast on, as defined in Table 5.
string
questions
array<object>
Data format
The question and resolution datasets are released JSON (. json) files.
Ethical and responsible use
There are no restrictions on use of the forecast sets.
B.2.1 GENERAL PUBLIC FORECAST SET
This forecast set consists of both forecasts made by individuals on the given question set and of aggregation of those forecasts, as described in Appendix D. The dataset will be updated every time survey run with every forecast containing the information outlined in Table and Table 12. Note that the aggregated forecast sets do not contain the user id or reasoning fields.
Data collection Data is collected from human forecasters via Qualtrics. We collect the rationale behind forecasts and manually anonymize the data ensure there is no personally identifiable information in the dataset before posting it online.
B.2.2 SUPERFORECASTER FORECAST SET
In addition to the fields outlined in Table 11 and Table 12, the superforecaster dataset contains forecasts with the fields shown in Table 13 (minus the suspected LLM field). Likewise, this dataset will be updated every time a group of superforecasters is surveyed.
Data collection superforecasters provided forecasts, rationales, and additional information about they way they forecast in our latest survey round. We will further manually check all of their responses to ensure the anonymity of the dataset.
B.2.3 LLM FORECAST SET
This dataset provides the same data (aside from user_id as outlined in Table 11 and Table 12, only provided by language models. Each individual json file was created by a model for the given question set.
Data collection Beyond informing teams their forecasts will be made public, we will not check the rationales.
16

---

Published as a conference paper at ICLR 2025
Table 5: Data dictionary for question entries in questions array from Table
Field
Description
Required
Data Type
id
A unique identifier string given source. If instead of string it's an array of strings, then this is a combination question and combinate maaiion_oat of will contain one question per id in the array of strings.
string arraycstringg
source question
Where the data comes from. The question to forecast, presented as an f-string with placeholders [oorecast_ cestt_____ddttee date) and (resolution_date for dataset questions.
string string
resolution_
ForecastBench resolution criteria. Specifies how forecasts will eval- uated for each question type.
string
criteria
background
Background information about the forecast question provided by the source, if available. Default: 'N/A' The datetime when the forecast question went on the market spec- ified by source. Default: 'N/A' The datetime when the forecast question closes on the market speci- fied by source. Default: 'N/A' The resolution criteria provided by the source, if available. Default: 'N/A'
string
market_info_
string
open_
datetime
market_info
string
_close_
datetime
market_info
string
resolution
criteria
url
The URL where the resolution value is found.
string
freeze_datetime
The datetime UTC when this ques- tion set was generated. The latest value of the market or comparison value the day the ques- tion set was generated. Explanation
string
Creeze_dateting
string
value
Creeze_datetime. value
of what the value specified in freeze_datet inetvluee_aalu represents.
string
_explanation
cource_intry
A prompt that presents the source of this question.
string
combination_of
An array of question objects, de- fined by this data dictionary. De- fault: 'N/A"
string arraycobject>>
disolution_aates The resolution dates for which fore- casts should be provided for this forecast question.
arraycstring>
17

---

Published as a conference paper at ICLR 2025
Table 6: Resolution set data dictionary.
Field forecast_due
Description
Required
Data Type string
e.g. The name of the file that con- tains the question set. e.g. 22024-07-21-11m, json" A list of resolutions to the forecast questions, defined in Table
Date in ISO format. "2024-07-21"
_date question_set set
string
resolutions
arraycobject>
Table 7: Data dictionary for resolution entries in questions array from Table
Field
Description
Required
Data Type string array
A unique identifier string given source. If instead of string it's an array of strings, then this is com- bination question.
source direction
Where the data comes from. If id has array value, this is an array of the same length. Each en- try 1-1 If the value is 1, the question was asked the normal direction. If the value is -1, the question was negated in the com- bination question e.g., for a ques- tion asking for P(-Q1 n Q2), the value would be [-1, 1]: all possible directions for this question would
string string
be: [1, 1). [-1, [1, 11. [-1,-1111111 The value is null when id is string.
forecast_due
The date the forecast is due in ISO 8601 format YYYY-MM-DD. The date the value is associated with in ISO 8601 format YYYY-MM-DE DD. The resolution value for the given date.
string
_att
desolution_date
string
resolved_to
number
resolved
If true the question has been re- solved. False otherwise.
boolean
18

---

Published as a conference paper at ICLR 2025
Table 8: Prediction market example.
Field id
Entry
1558
source
metaculus
combination_of
N/A
question
Will the Cavendish account for less than 50% of banana exports world- wide before 2035?
background
Bananas are well-liked import fruit all over the world, and the Cavendish cultivar has been crushing that market for sixty years. But its rise is literally founded upon the compost heap of the Gros Michel, another cultivar. The so-called 'Big Mike' variety had been the leading export towards Europe and North America, but the Panama disease, fungus belonging to the *Fusarium clade, killed that. Luckily the Cavendish, grown the same soil as the wilting Gros Michel, replaced it *the banana most of the western world connected with bananas. However, it appears another *Fusarium* rears its spores. Cavendish, with their genetic homogenity (they're all clones) and sterile nature, aren't resistant to it, and the fungus is ravaging more and more planta- tions. There efforts under way to deal with *Fusarium*, but with various societies' doubts and misgivings about GMOs, the cure may be viewed curse instead.
parke__info
This question will resolve as **s if the Cavendish banana accounts for less than 50% of worldwide annual banana exports in any year from 2018 to 2034 (inclusive). 2018-112 0018-11-13T08:000000+0000
resolution_
criteria
market_infi
open_datetime
market_info_
2034112-31123:::000000000
dime resolution_
Resolves to the outcome the question found at
criteria
collapse-by-20355.
url
http:///ww..atcciiii...oooooo//s bananas-- collapse-by-2035/
freeze_datetiime_ 0.48
value
Lerezee_attttim____
The community prediction.
value_explanation
freeze_datetime 2024-07-12T 011-0077-127000:00:0000000000000
cource_intre
We would like you predict the outcome of prediction market. A pre- diction market, in this context, is the aggregate of predictions submitted by the website Manifold. You're going predict the probability that the market will resolve as 'Yes'.
resolution_dates
19

---

Published as a conference paper at ICLR 2025
Table 9: Data source (DBnomics) example.
Field id
Entry
meteofrance TEMPERATURE CAA.AAAUURRECccssss007000000 oclsius.07005.D
source
dbnomics
combination_en
N/A
question
What is the probability that the daily average temperature the French weather station at Abbeville will be higher on resolution, ion__aate than on forecast_ arrecast_due_ddtteeee
background
The history of Average temperature by day and by station for France Degree Celsius ABBEVILLE Daily from Météo-France is available wwwp:///etefrreeeeeTEEAAAAAAATT cecrce/eeeessusSeeee 07005..0000
market_info
resolution_
criteria
N/A
market_info_
open_datetime
N/A
parket_info_
http://taaammiss.wwtd//eeeerrrn http:///teeottttceeeeMEEEEEPEEET TWW close_datetine url
N/A
resolution
the http://d.oommss..vev.o.. world/meteofrrance TEMPERATURE TEMMERRATURE.cClisss.TTTTT wewsius.07000.... once the data is published.
Resolves
to
value
found
at
criteria
https://db.nomn
freeze_datet
value freeze_datetimme
17.95
The daily average temperature at the French weather station at Abbeville.
value_explanation
freeze_datet ime
2024007-2200:0000000000
notro
DBnomics collects data topics such population and living ditions, environment and energy, agriculture, finance, trade and others from publicly available resources, for example national and international statistical institutions, researchers and private companies. You're 're going predict how questions based this data will resolve.
resolution_dates 1"2024-07-28", "2024-08-20". "2024-10-19", "2025-01-17", "2025-07- 21". "2027-07-21". "2029-07-20", "2034-07-19"]
20

---

Published as a conference paper at ICLR 2025
Table 10: Combination (DBnomics) example.
Field
Entry
id
['meteofrance TEMPERATURE wetpeeeellsssIIeeee .7..ii........D" ofrance_TEMPERR TEMEERATURE "er.eeeesuu..00740.."""""" dbnomics
"mete-
source
combination_of
[An array containing dictionary entries of both questions.] We are presenting you with two probability questions. Please predict the probability that both will happen, that one will happen but not the other, and that neither will happen. In other words, for each resolution date please provide predictions.
question
background
N/A
market_info
resolution
criteria
N/A
market_info_
open_datetime
N/A
market_info
close_datetime url
N/A
N/A
resolution criteria
N/A
freeze_datetime
value
N/A
@geezeddaettme
value_
explanation
N/A
freeze lirezzedaaetime
2022007-12000:000000000:00
human_prome
We are presenting you with two probability questions. Please predict the probability that both will happen, that one will happen but the other, and that neither will happen. In other words, for each resolution date please provide 4 predictions.
resolution_datee ['2024-07-28", "2024-08-20", "2024-10-19", "2025-01-17", "2025-07-21", "2027-07-21", "2029-07-20", "2034-07-19"]
Table 11: Data dictionary of headers for forecast set.
Field
Description
Required
Data Type
organization
The organization name as it should be dis- played the leaderboard.
string
model
The model name as it should be displayed on the leader board. The name of the question set file these fore- casts are associated with.
string
question_set
string
forecast_due
The date the forecasts were due in ISO 8601 format Y MM-DD.
string
_date
forecasts
All forecasts for this question set.
arraycobject>
21

---

Published as a conference paper at ICLR 2025
Table 12: Public forecast set data dictionary of entries in forecasts array from Table 11.
Field
Description
Required
Data Type
1d
A unique identifier string given source. If instead of string it's an array of strings, then this is a com- bination question.
string arraycstringg
source
Where the data comes from.
string
forecast
The forecast [0, 1].
number
Resolution_date
The resolution date this forecast cor-
string Inull
responds to. null for market ques- tions.
reasoning
The rationale underlying the fore- cast. During data anonymiza- tion, we insert [redacted to maintain anonymi ty] wher- ever text has been redacted.
string
direction
If id has array value, this is an array of the same length. Each entry an integer E f-1, 1}. If the value is 1, the question was asked in the normal direction. If the value is -1, the question was negated in the com- bination question c.g., for a ques- tion asking for P(-Q1 n Q2), the value would be [-1, 1]. All possible values are: [1, 1]. 1, [1, -1,.
array<number> Inull
[1], and null.
user_id
A randomly generated string associ- ated with the human respondent who submitted the forecast. This value contains no information which could identify said participant and was as- signed to the dataset after personal identifiers had been removed. Only required for participants from the general public and superforecaster surveys.
string
Table 13: Superforecaster forecast set data dictionary of entries in forecasts array from Table 11 (additional fields to those in Table 12).
Field
Description
Required Data Type
searches
An array of search terms used in re- searching the topic.
array<string> Inull
consulted urls
A list of useful URLs
arraycstring> Inull
22

---

Published as a conference paper at ICLR 2025
C QUESTION BANK
C.I LICENSES
The licenses outlined in Table 14 apply the datasets we have sourced for questions. We were granted express permission to use questions from Manifold Markets and Metaculus. Though not required by their license, also met with a representative from ACLED who approved our use of their dataset for the benchmark and dataset distribution.
Table 14: Question sources and permissions
License
License grants permission to use
Express permission granted
RAND Forecasting Initiative
Public Domain
Manifold
Terms of Service
X
Metaculus
Terms of Use
X
Polymarket
Terms of Service
ACLED
Terms of Use
DBnomics
Open License
FRED
Terms of Use
Wikipedia
Terms of Use
Yahoo!
Terms of Use
C.2 CATEGORIES
We generate metadata on all of the questions in our question bank, categorizing questions, as described in Section 3.1.2.
It is important to have forecasting questions across a broad array of categories to test L.M capabilities. We are currently adding more questions from datasets with the goal of equilibrating these categories.
Table 15: Categories and question counts by source, dropping invalid questions.
Manifold
Metaculus
Polymarket
DBnomics
Wikipedia
ACLED
FRED
Yahoo!
Total
RFI
Arts & Recreation
42
10
65
0
117
Economics & Business
13
55
154
166
0
509
899
Environment & Energy
2
37
52
0
0
98
Healtheare & Biology
71
215
297
Politics & Governance
16
128
188
0
0
0
0
335
Science & Tech
66
172
15
0
1
0
259
Security & Defense Sports
109
12
3,220
0
0
3,353
468
0
137
688
Other
151
122
2
0
75
0
356
Total
19
372
722
914
3,220
52
166
428
509
6,402
23

---

Published as a conference paper at ICLR 2025
D HUMAN SURVEY INSTRUCTIONS AND SCREENSHOTS
D.1 INSTRUCTIONS
Participants in the public survey were first prompted read consent form detailing the tasks involved in the study and potential risks to participants, alongside estimates of payment for completing each study ($5 bonus payments for forecasting performance in the introductory survey; $10 for completing the primary survey). In said consent form, the overall task was described as follows: "You will be asked to read some material, to follow some instructions, and to provide forecasts for future events." Then, participants were prompted give their responses to 20 forecasting questions randomly selected from the 200-question subset questions provided to the LLMs.
Participants were given a brief description of the task before each question:
You are going to be predicting the probability of the answer to the question below being "Yes" (or "resolving positively").
For tasks with questions generated from data providers, forecasters prompted provide probability forecast at multiple resolution dates.
The public survey was hosted Qualtrics's ' (https: htttps::/www.uuatrrioss com/trrrssiccccccccooomoo survey software platform.
Participants in the superforecaster survey went through similar initial experience before being moved into the group forecasting stage described in Section 4. Because of the expertise of this set of forecasters, superforecasters were also guaranteed base payment of $1,000 for participating in the individual and group stages of the experiment as well as bonus payments for individual-level accuracy.
The superforecaster survey was hosted on Quorum http:s::///quuuuppp.oooo platform
designed to host multi-stage forecasting tournaments.
D.2 SCREENSHOTS
Figure 2 and Figure show few of the questions presented to participants in the public study.
24

---

Published as a conference paper at ICLR 2025
You are going to be predicting the probability of the to the question below being "Yes" (or "resolving positively").
Will politician claim they lost major election due to "deepfake" image, video, or audio recording in G20 country before 2025?
"Deepfakes" or "Deep fakes" are synthetic media created with generative Al that imitate real media, but have people saying or doing things they did not actually do. This can be in the form of static image, a short video, or an audio recording with no video.
In May 2022, deepfake video of Elon Musk promoting scam cryptocurrency went viral social media, which Elon replied on Twitter "Yikes. Def not me.". In June 2022, Google banned deepfake-generate Al from its Colab tool. In May 2023, deepfake image showed an explosion at the US Pentagon, which was picked by media and caused temporary huge drop in stock markets.
May 2023, no major political scandal has erupted due a particular deepfake successfully convincing large block voters that politician or said something they didn't actually do.
URL: httpg://www.mmtaccluus.c////////
Resolution Criteria: Resolves the outcome of the question found at
https:/wwww.metaculs. http://wwwweeellssoo/oo///steee/
This question
resolves as *"YES* if, by Dec 31, 2024, politician an election with >3M votes
country, claims that they lost election due deepfake video, image, or audio recording them.
Last recorded value: 29.0%
The community prediction.
Freeze date: 2024-07-12 00:00:00
Market open date: 2023-05-23 18:00:00
Market close date: 2024-12-31 20:00:00
Prediction
Probabilly (0-100%)
Please write your rationale here. We encourage you to include the step-by-step process you used to come up with your forecast.
Figure 2: An example market-based question from the human survey.
25

---

Published as a conference paper at ICLR 2025
You are going to be predicting the probability of the answer to the question below being "Yes" (or "resolving positively).
According to Wikipedia, will Sarasadat Khademalsharieh have rating the dates listed below that's at least 1% higher than 2024-07-217
The International Chess Federation (FIDE) governs international chess competition. Each month, FIDE publishes the lists 'Top 100 Players', 'Top 100 Women', 'Top 100 Juniors' and 'Top 100 Girls' and rankings of countries according to average rating of their top 10 players and top 10 female players.
create the rankings, FIDE uses the Elo rating system, which method for calculating the relative skill levels of players in zero-sum games such as chess. The difference in the ratings between two players serves as predictor of the outcome of match. Two players with equal ratings who play against each other are expected to score an equal number of wins. A player whose rating is 100 points greater than their opponent's expected to score 64%;; the difference is 200 points, then the expected score for the stronger player is 76%.
A player's Elo rating number which may change depending the outcome of rated games played. After every game, the winning player takes points from the losing one. The difference between the ratings of the winner and loser determines the total number of points gained or lost after game. the higher-rated player wins, then only few rating points will be taken from the lower-rated player. However, the lower-rated player scores upset win, many rating points will transferred. The lower-rated player also gain few points from the higher rated player the event of draw. This means that this rating system is self- correcting. Players whose ratings are too low or too high should, in the long run, do better worse correspondingly than the rating system predicts and thus gain or lose rating points until the ratings reflect their true playing strength.
Elo ratings are comparative only, and valid only within the rating pool which they were calculated, rather than being absolute measure of player's strength.
URL: https://en http:://enwwkkipddiaa http://wwwwwkkedddaaoog///////// rankings
Resolution Criteria: Resolves to the value calculated from https///annwikkkpedaac.cckkiiird on the resolution date.
Last recorded value: 2489.0
Sarasadat Khademalsharieh's ELO rating.
Freeze date: 2024-07-12 00:00:00
Please only enter numbers between and 100 below, where represents 1%, represents 10%, etc...
Prediction
Probabiiity (0-100%) 2024-07-
28
Probability (0-100%) 2024-08-
20
Figure 3: An example question generated from data provider, in this case DBnomics, from the public survey. Two of eight forecast horizons for which we elicited forecasts are included above. The rationale text boxes (one for each forecast horizon) have also been excluded from the screenshot for brevity.
26

---

Published as a conference paper at ICLR 2025
E LLM "ENSEMBLE" BASELINE
E. MODELS
To construct an ensemble baseline that includes diverse candidates, we evaluate models using the most recent forecasting dataset containing cross-domale questions with true resolutions from Halawi et al. (2024). We assess models from the following organizations: OpenAI, Mistral AI, Qwen, Google, Anthropic, and Meta. Using the same scratchpad prompting method from Halawi et al. (2024), then select the top three models: GPT-4o, Gemini-1.5.Pre Claude-3.5-Sonnet, See Table 16 for the results.
Table 16: Brier Scores from each LLM "crowd" candidate.
Model
Scratchpad
GPTT40
0.207 (0.026)
Llama-3-70b
0.232 (0.020)
Mistral-Large
0.233 (0.026)
Qwen- 1.5-110b
0,222 (0.025)
Gemini-1.5-Pro
0.214 (0.025)
Claude-3.5-Sonnet
0.178 (0.025)
GPT-4-0613
0.222 (0.009)
GPT-4-1106-Previeer
0.209 (0.012)
GPT-3.5-Turbo-110
0,261 (0.010)
GPT-s.s-Turoo-Inttruect
0.257 (0.009)
Claude-2
0.219 (0.014)
Claude-2.1
0,215 (0.014)
Gemini-Pro
0.230 (0.007)
Mistral-7B-Instruue
0,243 (0.008)
Mistral-8x7B-Iie
0.238 (0.010)
Mixtral-8x7B-DD
0,248 (0.010)
Yi-34B-Chal
0.241 (0.009)
Llama-2-7B
0.264 (0.011)
Llama-2-138
0.268 (0.008)
Llama-2-70B
0.282 (0.011)
The highlights indicate the models we decide to use. Subscript numbers denote 2 standard errors. Above the divider line are the new models evaluate, while below the divider line are the models evaluated by Halawi et al. (2024) on the same dataset using the same prompt.
E.2 AGGREGATION METHODS
With the forecasts generated by the top 3 models selected in Section 1. then compare perfor- mance of 5 aggregation methods: Median, Trimmed Mean, Geometric Mean, and Geometric Mean of Log Odds (Satopää et al., 2014). See Table 17 for the results.
E.3 OTHER DETAILS
LLM Parameters. We set the temperature and the max output token length to 2000.
F PROMPTS
In this section, we present the following prompts: zero-shot (Figure 4), scratchpad (Figure 5), and three prompts (Figure 6, Figure 7, Figure 8) written by superforecasters from the Forecasting Research Institute used to construct the LLM "ensemble" baseline. Note that the scratchpad with information retrieval prompt is simply the scratchpad with additional line "We have retrieved
27

---

Published as a conference paper at ICLR 2025
Table 17: Brier Scores from each Aggregation Method.
Aggregation Method
Brier Score
Median
0.197 (0.024)
Trimmed Mean
0.197 (0.024)
Geometric Mean
0.194 (0.023)
Geometric Mean of Log Odds
0.194 (0.023)
The geometric mean and geometric mean of log odds are highlighted as they achieve the best performance. Subscript numbers denote standard errors.
expert superforecusier, familiar will he reselved as true. You MUST give prohahility estimate hetween 0 and UNDER ALL. CIRCUMSTANCES. for some reason you can't answer, pick the base rate, return number between
Tetlock
a prediction probubility question
Question: [question]
Question Buckground; [backgroun]]
Resolution Criteria: |resolution, criteria)
Question close date: (close_
Oulput your answer number between
asterisk beginning decimall. output anything
Answer:
[[ Insert answer here }]
Figure 4: Zero-shot Prompt from Halawi al. (2024)
the following information for this question: (retrieved j before the instructions begin. For combination questions, we slightly modify each of the above prompts by including the second question's information (Figure 9). Finally, present the prompt used to determine whether question is marked as valid or invalid Figure 10.
G STATISTICAL DETAILS
While our pairwise bootstrapped p-values are precise, our statistical tests ignore one potentially important source of imprecision. We assume that each question is independent of the other questions. In world where forecasting accuracy quite correlated within a topic, or where most events are correlated, this may result in overstating how confidently we can reject the equivalence of different models. We hope to explore this question future work, but we are somewhat reassured by the fact that our gathering of forecasting questions from diverse domains and sources makes it unlikely that correlated questions would change our interpretation these results in any meaningful way.
Our statistical tests are significantly more precise than the 95% confidence intervals for each model would imply because accuracy on each question is quite correlated across models. To understand this phenomenon, consider hypothetical world where Model A outperforms Model by constant (c) on each question. No matter how close the performance no matter how much variance there is in the accuracy of Model A across questions (which drives the 95% confidence interval surrounding Model A's accuracy), a pairwise bootstrap would show that Model A is more accurate than Model B. This is because forecasts of Model A and Model B are perfectly correlated.
Models A and are (how small is), and
H LEADERBOARDS: TOP 50
We show the best 50 performers on several leaderboards. Table 18we show the leaderboard for the human question set of 200 standard (non-combination) questions. Table 19 shows the leaderboard for the full LLM question set of 1,000 questions. Finally, in Table 20, we present the human leaderboard with combination questions included where humans provided forecasts both components of the combination question. We derive human forecasts for these combination questions by treating each component of the combination question independent.
28

---

Published as a conference paper at ICLR 2025
Question:
Iquestion]
Question Background:
[background)
Resolution Criteria:
Iresolution, criteria]
Question close date: [close_date
Instruction:
1. Given the above question, rephrase and expand to help you do better answering. Maintain all information in the original question.
[[ Insert rephrased and expanded question. I!
Provide fow reasons why the answer might be no. Rate the streagth of each reason.
[[ Insert your thoughts }}
Provide few reasons why the
might he yes. Rate the strength of each
[[ Insert your thoughts |]
Aggrogate your considerations. Think superforeeaster (e.g. Nate Silver).
[[ Insert your aggregated considerations ]1
Output an initial probability (prediction) given stops 1-4.
Insert initial probubility }}
Evaluate whether your calculated probahility is excessively confident or not confident enough. Also, consider anything else that might affect the forecast that you did not hefore consider.
{( Insert your thoughts I}
decimal. example, there resolution dates, you would outpui different *p* for each resolution date) Do not output anything els.
Output your
number
and
the beginning
[[ Insert your answer )
Figure 5: Scratchpad Prompt modified from Halawi al. (2024)
T AGGREGATING HUMAN FORECASTS
We provide aggregated forecasts on benchmark questions from 500 members of the general public
and superforecasters, as described Section 4:
Public:
www.bheceeecaennaaaaaeeeeeeeeeen public.json
21 ..oreeastBencc...hee human hhluaa_ppbliccjsieeeeeaaeaa
www.evasweattaaaaaaeeeerareeeeee 21. ..ForecastBencc.humaaa suunan_ssppreoooon
Superforecasters:
As described in Section 4, members of the general public were recruited via Prolific and Facebook. First, participants completed an introductory survey designed to gather demographic information and evaluate performance to take part in the main survey, featuring 20 random benchmark questions. Some participants were disqualified from participating in the main survey based on suspicious elements of their presurvey responses (e.g., answering questions rapidly, large numbers of submissions from the same IP address, etc.).
few forecasting and comprchension tasks. Then, they were invited
Superforecasters were invited to participate directly and were prompted to give forecasts for at least 20 questions from the benchmark (though, many chose participate on additional questions). Superforecasters were also given access to other Superforecasters' forecasts and rationales and were allowed to comment on and update based on others' forecasts.
For each group, the median forecast for each forecasting question was taken to create the aggregated forecast sets. Code and individual forecasts are forthcoming once we can ensure that the text responses have been fully anonymized.
J REPRODUCE LLM FORECASTS
We evaluate 17 LLMs initial benchmark: GPT-3.5-Turbo-Instuee (Brown al., 2020), GPT-4 (OpenAI, 2023), GPT-4o, Llama-2-70B (Touvron et al., 2023), Llama-3-7B, Llama-3-70B, Mistral-7B. Mistral-8x77 (Jiang et al., 2024a), Mistral-8x22B, Mistral-Large, Siven1.5-1100 1hBt- Claude-2.1 (Anthropic, 2023), Claude-3-Haiku. Claude-3.5-Sommen Claude-3-Opus (Anthropic, 2024), Gemini 1.5 Flash, and Gemini 1.5 Pro (Gemini Team, 2023).
To make inferences, we use APIs. For the GPT-suite, we use OpenAl's API; for Gemini-suite, we use Google's API; for Llama-suite, Mistral-7B, Mixtral-8x7B, Mixtral-8x22B and Qwen 1.5-11..11111 5-110B-Chat
29

---

Published as a conference paper at ICLR 2025
Question: (question)
Question Background: [hackground]
Resolution Criteria: [resolution criteria]
Question close date: {closc_date
We have retrieved the following information for this question: [retrieved_jnie iinf
Instractions:
Given the above question, rephrase and expand it to help you do better answering. Maintain all information in the original question. [[
Insert rephrased and expanded question.]]
2. Let's 's start hy coming with a hase- that could he helpful for foreensting this questionn. Come with the best reference-che -laannee--llssse you can for this sort event, and give general base-rate that docso' take into account factors unique to this question.
For instance, if the question were about the probability of new technology being widely adopted within five yeurs, you might look historical data on the adoption rates of similar technologies as a reference class. Come up with a base rate that could be relevant for this question.
The base-rate must be formatted as a clear probability (or number, in cases where you believe that to be more usefal than a probability), instance, imagine you forccasting the probabiity that incumbent president will re-elected in an upcoming election hypothetical country. The past data shows that the incumbent has been elected 60% of the time.
Hore, you would write 'The reference class have chosen is the incumbent being elected. My base-rare is that the probability of the incumbent being re-elected 0.6." Give a justification for the base-rate, as well as clear number. Imgurtantly, the base rate should he as specific as it's possible to be without losing confidence that the number is correct. For instance, you were forecasting on the probability of hypothetical democratic country going base-rate for a democratic country going to war in a given year, rather than simply thinking about a given country going to war. Insert your
the next year, you should iddeally produce
rate }}
3. Now, let's think about factors specific to this question that may give good reason to deviate from the base rate. Please give some ressons that the probability of this question resolving positively may be higher than the base rate. Please note spocifically how they affect your forecast in terms of percentage point change. il Insert your thoughts }}
Now, let's think about reasons that the probahility of this question resolvinng positively may he lower than the hase rate. Please note specifically how they affect your forecast terms of percentage point change. [I Insert your thoughts
5. Consider any other factors that may affect the probability of this question resolving positively or negatively, that you have not already discussed in the previous two steps. |l Insert your thoughts })
6. Aggregate your considerations. Think like a superforecaster (e.g. Nate Silver). Give a ranking to each consideration based on how
much you believe it ought to affect your forecast. II Insert your aggregated considerations }]
Are there any ways in which the question could resolve positively negalively that you haven't 't considered yet, or that require
outside-the-hox thinking? For example, if the question was 'Will Microsoft have a market capitalization of over 55tn by 2030', you might consider questions like:
How likely is it that Microsoft no longer exists in 2030! How likely is it that inflation crodes that value of the dollar as such that $5n is there a merger between Microsoft and another large company?? How likely is that Microsoft is broken as is perceived to have monopoly power?
worth significantly
than today?
likely
Here, we're 'r thinking about things that are produbly quite unlikely to huppen, but should still be integrated into your forrcast. Write up some possihilities and consider how they should he integrated into your final forecast. ( (Inssrrt your thoughts and considerations about
how this should affect your forecast] |
Output an initial probability (prediction) given steps 1-7. [1 Insert initial probability. }}
9. Okay, now let's think about some other ways to comider how to forecast on this question. What would you say are the odds that you statement true or false, you would find it's true? You must give an odds ratio. This odds ratio probahly shouldn't i be purely on the basis of the considerations in the previous steps, but you should think again about what you would expect to see if you could fast forward into the future. it helps, imugine that you's 'r laking a bet. il Insert your odds ratio. }} 10. Given your rephrased statement from step think of 2.3 statements that if you conditiones on their heing TRUE, you would think it more or less likely that your statement would be TRUE as well. These statements must not DETERMINE OR BE LOGICALLY EQUIVALENT
could fast- forward and find whether
the original statement. Be creative! (I Insert 2 to 3 related statements. }}
11. For each of your related statemenis, give new odds of the original statement conditional on the related statement being TRUE. (I For
each related statement, insert new adds for the original statement. 1)
12. Now consider each of your odds from the previous stepststeps 9- (1), and come up with your all-things-oomiddree odds ratio for the original statement. (1 Insert final odds for the original statement. }}
13. Now, convert that odds ratio to a probability between 0 and I. I [[nnsert probability)) I
14, Now, consider the probability that you came with in step 8, as well as the probability that you came up with in stop 13. Which of these probubilaties do you towards? How you weigh them against one another? Write up your thoughts on which probubility more likely to be "coorec"" and then decide on FINAL probabiliy that will be used as your forecast. ( ([Insert your thoughts AND final probahility] I|
15. Output your answer number between 0 and 1) with an asterisk at the beginning and cad of the decimal. [( Insert your answer )}
Figure 6: Superforecaster prompt
we use Together Al's API; for Mistral-Large, we use Mistral Al's API; and for the Claude-suite, we use Anthropic's 's API. To reproduce our results, people will need to gather these API keys.
We then record model predictions using several different methods: zero-shot prompting, scratchpad prompting, scratchpad prompting with retrieval augmentation, and scratchpad prompting with retrieval augmentation and aggregate human forecasts. For the scratchpad and information retrieval setting, we use the retrieval infrastructure from Halawi et al. (2024) and provide relevant news articles to the models in-context to reason about. Additionally, only models with context window larger than 8,000 tokens were evaluated under the retrieval setting due to the inclusion of news articles in the prompt.
30

---

Published as a conference paper at ICLR 2025
Question: [question]
Question Background: [hackground]
Resolution Criteria: |resolution criteria]
Hore's 's some related information from the news that I've collected for you: [setricved_iafe
Question close date: (close_ ]
Insiructions:
Rephrase the question as a statement about the future, egg. you would rephrase "Will Riden be the U.S. president on January 20257"
"Biden is the U.S. president January 2025." Insert question rephrased statement.
2. What would you say are the odds that if you could fast-forwan forward and find out whether that statement is trse or false, you would find out it's true? You must give an ratio. helps, imagine that you' re taking bet. [[ Insert your odds ratio. }]
Given your rephrased statement, think of 2-3 statements that if you conditioned their being TRUE, you would think it more less likely that your statement would be TRUE as well. These statements must not DETERMINE OR BE LOGICALLY EQUIVALENT to the original statement. Be creative! Insert to related statements.
4, For each of your related statements, give new odds of the original statement conditional on the related statement being TRUE insert odds for original statement. H)
5. Now consider each of your odds from the previous steps and come up with your all oll-hiing--onsideree odds ratio for the original
statement. Outpat your answer (a number between 0 and 1) with an asterisk the beginning and end of the docimal. {I Insert final odds for the original statement. }]
Figure 7: Superforecaster prompt 2
Question: [question)
Question Background: [background)
Resolution Criteria: (resolution _crttercciaa
Relevant information we retrieved from news articles: (retrieved_infos
Question close date: {close_ ]
Insiructions:
Given the above question, rephrase and expand it to help you da hetter answering. Maintain all information in the original question. [[ Insert rephrased and expanded question.).
2. Provide a few reasons why the answer might be no. Rate the strength of each reason. For now, ignore the evidence, ideas, and perspectives vontained
allached news articles. Insert your thoughts I
3. Provide few reasons why the answer might he yes. Rate the strength of each reason. For now, ignore the evidence, ideas, and perspectives contained in the atlached news articles. II Insert your thoughts ]]
Aggregate the considerations you developed in the previous steps. Think like superforecaster (e.g. Nate Silver). Insert your aggregated considerations }
Output an initial probability (prediction) given steps (1 Insert initial probability.
6. Now. consider the perspectives, ideas, and evidence that was provided in the retrieved news articles. How should these affect your judgment
probability of the question resolving positively? List reasons why these news articles might increase the probability of the question resolving positively. [[nsertt your thoughts)
7. Now, let's locus on how the ideas, perspectives, and evidence provided in the news articles might decrease the probubility of the question resolving positively. [[nsert your thoughts]]
8. Given what you'te ve thought about in the provious two steps, update your probability from the initial probability you gave in stop 5, { [Insert updated probabiity }
9. Evaluate w whether your calculated probability is excessively confident or not confident enough. Also, consider anything else that might
affect the forecast that you
hefore consider. [[ Insert your thoughts ]I
10. Ourput your answer (a number between and with an asterisk the boginning and end of the decimal. Do not ourpot anything
else. [[ Insert your answer }]
Figure 8: Superforecaster prompt 3
Additionally, to speed up the inference, rate limit and requests for better subscription plans from each source. However, one can run inference sequentially by setting it as worker, but this requires longer time generate all the baselines.
multithreading with 50 workers, which requires high
J.1 ZERO-SHOT AND SCRATCHPAD BASELINES
Prompts We use the shro-shot and scratchpad prompts shown in Appendix F.
Hyperparameters For the zero-shot setting, we set the maximum output token length to 50 since we only request probabilistic forecasts. For the scratchpad prompt, we increase the maximum output token length to 1300 as it requires reasoning and probabilistic forecasts. We initially considered high token length of 3000, but after observing that the maximum response length was around 1250, we settled on 1300 as the optimal maximum token length. In both cases, the model temperature is set to to ensure stable outputs.
How to Reproduce To run zero-shot and scratchpad baselines, follow the steps below:
1. Insert all the necessary API keys in prc/hellerra perss/eys.pp.
31

---

Published as a conference paper at ICLR 2025
Question
[question_]]
Question 2:
[question 2)
Question Background: (background
Question 2 Background:
(background_2)
Question Resolution Criteria: ieoolution_ertteriia__.
Question 2
Resolution Criteria: [resolution criteria_2
Question Current value on ffreeze datetime__ [valae_nt at freeze datetime_
Question Value Explanation: |valuc_ at freeze, dwt.ddttttmmmmmc explamation_
ftt_aatffeeeeeeeeeeeeeee Question 2 Current value on ffreeze datetime__ I]: [value_nt at_ freeze, datetime_]]
Question Value Explanation: vvluue at freeze datetime, explanation 2|
Here's related information from the news that I've collected for Question [retrieved_iife into_]|
Here's a some related information from the news that I've collected for Question 2: [reurieved_info__ info__
Question resolution date: [list of resolution _
Figure 9: Combination prompt that includes information about both non-market questions. The instructions are truncated and can be supplemented with any of the prompts shown above.
quality forocast question.
Baza Corecast questions (queation).
Please flag quastions that donft seem appropriate by octputring "Flas". Otherwise, 12 it seems like reasonable question or you ze ungure, cutput *sx." general, poerly-datiog questions, questions personal,
sexual in queations
questians about the dea.h/Life wxpectancy of an Individual should be Clayged er, nore generally,
questiona
in the incerest should flarged. Gropelitical quastions, quastions about swaes,
that
the untertaineet Indestzy, waze, public Zigures, and, more genarally, questions in the public Incerest should
"ak."
Examples questiona that should flanqed:
*g:11 Finish sh ny horwwork tonight?"
*delaculus party 2023*
*Mi1l Hell frenze ever?"
or teila?"
*1:1 get lake MID!*
*Mi1l this video zeach 10ck views by the BCDS"
2Aella goes
podoest,
segret 1t.7
*Saily cuinflip"
*boak va Backerborq: Will either of then sbit their sants on the matt"
Exampeex of Maskle
lons that should NOT be flaggeds
*Mil
Prince
have baby
year?"
*g:11 the Brain Preserves lon ion's targe Narmal preservat fon prize won by Peb 9th, 20137*
*1l1 Uneew be more novel new drugs agproved by the PIX in 2016 than in 20152*
M1l1 Tsrael invade Bafah in Nay 20247*
Trag
month?"
Tigerr
Moods Mill Another
Tourramen:
M1l1
Drayne Johnson win the 2024 US Prealdertial Election?"
681 11 Oppenbeiner win picture ANT Bitcoin reach $70K AND Ni nieedo announce console by POY
20242a
*05 anybody born betore 2000 Live to be 1507*
*8:
Taylor Swift get married before Risnoir reaches 5100K rigape
*1l1 Rassiafx Lotal lecritury decrmase by least 203 Swforw 2028"*
Trump jatled incercerated
Chine invades Taivan before 2035, will the reagond with military force?"
*1l1 Lheew be a Launani that killa aa al leass 50, 000 peuple before 20503*
11il there be military contlict resulting in least 50 deaths between United States and Chine
20243"
*Mill an systen be reported to have exccesstully blacknailed sonsone for 591060 by BOY 20287"
11 Vladinir Putin declare Nartial Law in at least 3/4 of Russia before 20257*
Agair, when in doubt, do NoT flag the questiion mark it "ok".
Your response
rollowing structure:
Tneert thinking:
II insert your concise thoughts here })
Classttication:
*fleg"
Figure 10: Question validation prompt
2. Run the zero-shot and the scratchpad baselines in wwtpbbseeeeal///e/eeeeese /tce://aeeeeel//lmmaaaeeeeeee/// manager/maln mmw.n.manpp......m.mmm..........
baselines:
J.2 SCRATCHPAD WITH INFORMATION RETRIEVAL BASELINE
Prompts We use the same scratchpad prompt as scratchpad baseline with an additional line "We have retrieved the following information for this question: (retrieved_ing "irrivveddiffo""" before the instructions begin.
32

---

Published as a conference paper at ICLR 2025
Table 18: Leaderboard: Human question set (top 50)
Brier Seore
Model
Organization
Information
provided
Prompt
Dutaset (N-422)
Market (N-76)
Overall (N-1983
Confidence Interval
Parrwtse perzlue compariin to No.
Pet more
than
Superforecaster median Eocccast
FoneasslBench
0.118
0.U74
0.096
[0,070, 0.316
0%
Public modian forecsst
FamezsiRench
0.153
0.089
0.121
[0.101. 0.141]
<0.001
225g
Clands 3-5 Somne: 20240620
Anthropic
Freeze values
Scratchpad
0.138
0.107
0.122
[0.09%) 0.146
<0.00)
31%
Claude-3-S-Sonee S00a0-22228800
Autloppic
News with freeze values
Serutchpadd
0.342
0.112
0.127
[0.104, 0.150)
-00000
20%
GPP-4-Tubb-21133 Turbo-2112440000000000000
OpenAl
Fieeze valles
Zeso shot
0.362
QLIRS
0.128
10.105 0.151
-00000
32%
Anthoppi
Frozs valuure
Zzro shot
0. 145
0.117
0.131
[0.10% 0.159
<0.001
31%
GPT-4
OpenAl
Theeze values
Zero shot
0.167
0.095
0.132
[0.109. 0.155)
<0000]
31%
GPT-4o
OpenAl
News with freeze values
Scratchpud
0.162
0.105
0.135
[0.113, 0.154
<0.00]
25%
Claude-3-5 S.----Snnett 2000022 20240221
Anthoopic
Scracchpadd
0.138
0.153
0.336
10.113, 0.158
<00/201
28%
GPT:4
OpenAl
Frezz values
Seraichju hyadl
0.
0.113
0.137
10.15 0 8.85
<0.001
27%
Anthopie
Nows
Scratchyaa hypd
0.142
0.157
0.139
10.117, 0.161
<0.001
369
Clande 3-Opus 202-02299
Antrropi
Theese values
Zero shot
0.165
0.115
0.139
[0.114, 0
<0.00]
23%
Claude-3-5- Sesuull----SSnes-2110 201440211
Amthopa
Superforecaste?
0.123
0.340
10.120 0.361
25%
GPT-4a
OpenA
Seratchuud
0.125
0.145
125
<0100)
Amthopic
Superforeeasae
0.135
0.145
<00001
25%
4PT-4
OpenAl
Scratchpad
0.126
0.146
10.122 0,165
-0000
22%
Cluud=3-00uu-200a0
Authooiic
Fiezze vallaes
Scratchyadd
0.129
0.144
10.125 1,165
23%
Mistral Langtal-Lang-LLaeee
Masral
Fiezze values
Zeso shit
0.173
0.117
0.345
10121,
-0000
23%
Gemini- 1.5mP
Google
Nows with frocze valluee
Senatchuui byudl
0.130
0.146
10 16
-000
1
339
Gemini 1.5 Pro
Google
Scratchpad
0.162
0.131
0.147
10.12%
0,160
<0000
1
23%
Lastral-Larg-LLeeet
Matral Al
Fieese vallaes
Seratchpod
0.361
0.13
0.147
1012%
111651
101000
2258
GPT-4-Tu Tut4--Tubo22212-00400
OpenAl
Zeso shot
0.162
0.157
U.149
1012%
(6120)
(0000
23%
OpenAl
Frozze valure
Senachpud
0.176
0.123
0.150
[0
01173
-0000
25%
GPT
OpenAl
Treese values
Scratchpad
0.175
0.126
0.150
10.1 0.170
<0000
21%
Gemin: 1.5 Pro
Google
Theese values
Scratchpud
0.162
0.138
0.150
10.122 0.169)
<0000
22%
Claude-3-S-B Stuude--S-Sommee--000 202
Anthopi
Zaso shot
0.345
0.155
0.350
10.123 0.177
-00001
24%
Gemini 5-Pro
Google
Nows
Seraichpud
0.163
0.141
0.152
10.33 1.13
40000
2254
Chanlo--Qqqa- 200-1122
Anthopic
Nows
Superfonccster
0.157
0.147
0.152
10.151 0.0
20000
224
GPT-4 4 Turbo 2024 04 09
OpenAl
News with freeze values
Scraichpad
0.176
0,128
0.155
10.130) 0.0
<0.001
25%
GPT-4o
OpenAl
Fiezse
0.397
0.109
0.155
200000
Seratchuud
0.116
0.153
10
1000
25%
Misiral-Fx 22B..aarttt
Seratchyud
0.127
0.154
1000
2024
OpenAl
Scratchpad
0.176
0.133
0.154
10112, 7.
10000 1000
22%
Gemini- 1.5
Google
Fiezze vallass
Zaso
0.124
0.355
10112, (1182
100000
255
Cluude-3-Opus- 20002920
Anthoopic
Scrutchpad
0.360
0.149
0.355
1011%
100000
22%
GPT:4
Openi
byud!
0.174
0.137
0.155
[0 1- 11
2000
189
Qwen1.5 1100 Chat
Qwen
Freeze values
Zero shot
0.197
0.115
0.156
10.124 9.129
-0000
20/4
Chade-2.1 1
Authopic
Frecse values
Seratchpu byac
0.217
0.156
10115% 111251
100000
24%
Gemini- 1 1.5Hashh
Google
Fiezse values
Zeso shot
0.191
0.125
0.15%
10120 11186
-00000
23%
Misial Al
Froese valure
Zaro shot
0.
0.131
0.15%
[0.151 0.185
<0000
24%
Liama 3 706 Chat II
Meta
Freeze values
Zero shot
0.19:
0.124
0.159
[0.133, 0.185
<0000]
25%
GPT 4 Turbo 2024-00 04 09
OpenAl
News
Scratchpad
0.178
0,144
0.161
[0.140, 0.182
<0.00]
24%
Inguted Rocceaster
ForceasBench
0.250
0.U73
0.362
[0.142 0.181
<00,00
26%
Owen 5-1100B-hhu
Qwen
Frezze valuee
Scratchpud
0.183
0.141
0.162
(O) 144, 0. 180]
<0.001
18%
Clunde 2.1
Amthopic
Scraichpad
0.217
0,109
0.165
[0.143, 0.185
<00001
22%
Qwenl 5-100 Chat
Qwen
News with freeze values
Scraichpad
0.179
0,148
0.165
[0.144, 0.185
S0,00]
22%
Clande-211 1
Authopic
Fieese vallass
Zao
0.221
0.108
0.164
[0.143. 0.1881
27%
Mistral-Larg-- Latest
Misrra
0.161
0.168
0.165
Chanls3-S- Seanl-----SSmme--20e 2050m0---2400 2014600
Amthopic
Nows
Seratchyud
0.133
0.165
144
<0.001
Clande Opus 20240229
Amtropic
Zero
0.163
0.167
0.165
[0.140, 0.191]
<0000]
18%
Notes:
Sharss
on standard questions provided
question
7-, 30-, and forecast horizons.
2: full leaderboard avaiahle www.foree dorre Online results updated nightly.
slightly different than vensoon presensed here.
3. For resolved market questions, forecasts are compared sgains, ground tuth while for unresolied market quesiooss they are compared community aggregates.
mash dotason Rrier scame and mean market covee (bontstrappll. foroenster under the mull hypaihesis that there's difference
ovemil sevee calealated
ovange
Pairwise
comparing
p-vsiue calicilated by hoatstapping
differennss overal between model
6. Pet. more accurate than I: The percent of questions where this forecaster had a better overall score than the best forecaster.
Information Retrieval We use the same information retrieval system from Halawi et al. (2024). The pipeline consists of four steps: search query generation, news retrieval, relevance filtering and re-ranking, and text summarization. One must acquire a Newscatcher API key to implement the same retrieval method.
Information Retrieval Hyperparameters The hyperparameters were selected following the results in Section E. of Halawi et al. (2024), in which they used a greedy search approach to identify the optimal hyperparameters. We display the hyperparameters below:
NUM SEARCH QUERY KEYWORDS: The number of keywords used in the search query. For our
system, this is set to 6.
MAX WORDS NEWSCATCHER:
The maximum number of words allowed in search queries for the
NewsCatcher API. This is set to 5.
MAX WORDS SNEWS:: The maximum number of words allowed in search queries for the Google News API. This is set 8.
SEARCH SEARCH_QQERY L MODEL NAME: The name of the model used to generate search queries. We use
9pt-4-11066previe
33

---

Published as a conference paper at ICLR 2025
Table 19: Leaderboard: LLM question set (top 50)
Beter Score
Model
Organization
provided
Incormation
Promutt
Dataset (A-5,492)
Market (X-697)
Overall
(8866,389)
Confidence Interval
Pairwise p-valse comparing to No.
Pet.
tham
Claude 3-5 Soones- 202400220
Anthroppi
Freeze values
Scratchpud
0.109
0,078
0.123
[0.117, 0.129)
0%
GPT-4-TThb--22444444
OpenAl
Fresse values
Scratchpad
0.172
0.080
0.126
[0.120. 0.132)
0.09%
43%
GPT-4a
OpenAl
Freeze values
Scratchpad
0.186
0.069
0.128
[0.122. (33)
-0.01
43%
Gemini-1.55.
Google
Froeze values
Scratchpad
0.362
0.306
0.334
[0.128, 0.139)
<01.001
35%
GPT-4o
OpenAl
News with fronze values
Scratchpad
0.190
0.084
0.137
[0.131. 0 0.145
<0.001
39%
Gemini 1.5 Pro
Google
News with freeze values
Scraichpad
0.166
0.129
[0.133. 0.144
-0.001
345
Clande-5-Opus-20 -pps-22244222
Authropie
values
10.133,
<0.001
Qwent.
values
Scratchpad
10.136.
Clunde Sonnet 20240620
Authropic
News freeze values
Scratchpudd
0..12,00.405
32%
Authropic
Freeze values
Zero sbot
0.192
0.143
[0.136. 0.100)
40.001
42%
GPT-4-Turbb-2M0 2024-04-09
OpenAl
Scratchpud
0.172
0.115
0.143
10.12. 0.109)
<0.001
31%
GPT-4-Tirbo-B 202002024-04400
OpenAl
Freeze values
Zero sbot
0.304
0.084
0.144
10.137, 0.1000100 150)
<0.001
42%
Clande 3-5 Sonnet 20240620
Authrogic
Scratchpud
0.308
0.120
0.144
00.129 0.150)
<0.001
10%
Gemini-1,55 ..5Pro
Google
Scratchpad
0.162
0.128
0.145
0..199 0. 1511
<0.001
325
GPT-4
OpenAl
Freene values
Scratchpad
0.194
0.100
0.147
10.141 0. 0.154
<00001
36%
Gemini-1.5 1.5-PPo
Google
News
Scratchpid
0
0.124
0.347
U..44, 0.153
<01001
32%
Imputed Foreenster
FonessRench
0.350
0.048
0.149
0.115, 0 1553
20/001
46%
GPT-4o
OpenAl
Scratchpud
0.186
0.150
0.144, 0.156
-0000
31%
Gemini-1 1-PPip
Fresse values
0.237
10.14
20000
30%
GPT-4Turbb--000 20--000000-00000000 2024-04-09
OpenAT
News freeze
Scratchpad
-0/00
1000
GPT-4o
OpenAl
News
Scratchpud
0.190
0.114
0.522
10--4-0 1-1580
-0/00
31%
Authropic
News
Scratchped
0.184
0.134
0.154
(0
(00)
20000 1
305
GPT-4
OpenAl
Freeze values
Zero sbot
0.222
0.087
0.154
10.14 (.110
-0.00
1
38%
Qweal.5-1100 Sopell,--118-Chha
Quea
Scratehpud
0.176
0.134
0.155
00150, 0 1660
<0000
100001
285
L.M Crowd
FenczaeRench
News
0.242
0.068
0.155
00.51, 0 159)
-0000
18%
Clande 3 Opus 20240229
Anthropic
Freeze values
Scratchpad
0.201
0.112
0.156
0.150, 0. 0.162
-00000
27%
Mistral-Lare LageLatttt
Mistral Al
Freeze values
Scratchpid
0.359
0.115
0.157
0.151, O 1621
401001
264
Gemini-1 1, -.SPro
Google
Zero shot
0.217
0.007
0.157
0.1151 0 1633
20/002
37%
Crowd
Poretusteenc
News
0.243
00153,0.160
37%
Crowd
ForecodRench
News
0.344
20000
10000
GPT-4
OpenAl
Scratchpad
10.15.0 (201
28%
LIama-3- 2-1-0---------0 Chat-HH
Meta
Freeze values
Zero
10-21-0 199
23%
Gemini 1,5
Google
Superforccateer
10-12-0 1000 1021
21%
Authropic
News
Siperforcestir
0.190
0.59
00-53
20000
20%
GPT-4-Tie N9r-4-Turbo-2225 2024-04-09 04-09
OpenAi
Zero sbot
0.204
0.117
0.160
-0001
32%
Gemini-1 1,5
Google
Preeze values
Scratehpud
0.194
0.12H
1.161
10.54, O 1661
200001
32%
Chade---- 0-00----000000
Anthropic
Zern shot
0.186
0.136
0.161
10. 15.1 O 168)
20.00
35%
GPT-4 Oorbo-Tubb- 2024 04 09
OpenAl
News
Superforscaster 2
0.208
0.116
0.162
10.156, 0.1671
<00001
28%
GPT-4-Tahh-22244 202400000
OpenAl
News
Scratchpid
0.211
0.114
0.363
10.15, 0 0.166)
<01001
275
Clande335 3-5---nnnt--0- 202-0620
Anthropic
Zero shot
0.192
0.134
0.163
10.156 0. 77)
<00001
34%
Queal.5-1108 Chat
Owea
News with trezee values
Scratchpad
10.15%
<01001
values
Scratchpad
25R
Mistral Lates:
Mistral
Freeze values
[0.157. 0.12
31%
Gemini-1.5 Fl5h
Google
Freeze values
Zero shot
10.15%, 0.1731
<00001
40%
Mistal
Freese values
Scratchpad
0.210
0.165
10.159, 172)
<0.001
30%
4o
OpenAl
News
Superforzcaster
0.211
0.124
0.168
10.162, 0.173)
<0.001
28%
Cladde2211
Antrropi
Scratehpad
0.237
0.100
0.168
(0.163, 0.174]
<0.001
38%
Gemini-1. 1.5-Flahh
Google
Scratchpad
0.194
0. 0.146
0.170
10. 10.164, 176)
<0.001
28%
4P
OpenAl
Freeze values
Zeco sbot
0.225
0.116
0.371
10.163, 0.178)
<00001
37%
Chaddo--- -upubbb3-Q0
Antisopic
Scratchpad
0.301
0.141
0.171
[0.165, 0.177)
<0.001
26%
Notex:
Shows performanc on the 1,00 (500 standard, 500 combinntioon) quedicess in the 1. question set at the 30. 90. and 180-day funecoal berizons
2. The full lewkeboord is available al www. fornensheennhh ong Online reullts are upditted nightly. so may he slighily diffcrent tham the vension presented here. unresolfed 5. Parwise p- value compourang to No. I (booistraped): The p value calculated by boosstrapping the ditferences in overall score between each model and the best Gomennter under the mill
market questions, forccasts are compared agains ground outh Brie
qusituds, they are compared community aggregates.
overall
there's diffrrmee tran No. The percent questions where forocaster had better
than forecastor:
SEARCH QUERY TEMPERATURE:
The temperature setting for the search query model, which controls the randomness of the output. We set this to 0.0 for deterministic outputs.
SEARCH_ BUEECE__UERRY__EE PROMPT TEMPLATES: The templates used to generate search queries. QUERY_PPRMPPT
In our configuration, we use PROMPT PROMPT_DICTII ("t__IICC"""'e""""" ""searchhquurr""" ["O"] and PROMPT_DICE "ssearch_qurrr" ("1"). The exact search query can be found in search_query.py.
NUM ARTICLES PER___EERRR__ QUERY: The number of articles retrieved per search query. This is set to 10.
SUMMARIZATION MODELL NAME: The name of the model used for summarizing articles. We use
SUMMARIZATION CIOIRIIIATIOONTEEEERRAURRSSS The temperature setting for the summarization model, which controls the randomness of the output. We set this to 0.2.
SUMMARIZATION PROMPT TEMPLATE: The template used for summarizing articles. In our configuration, we use PROMPT_DIP DICC [" "summari commarizatiinn!. zation"] ["9"]. The exact search query can be found in summarization.py.
34

---

Published as a conference paper at ICLR 2025
Table 20: Leaderboard: Human question set with LLM question set combination questions (top 50)
Brier Seore
Model
Organization
Information
provided
Prompt
Dutasct (N-1,754)
(N-2966)
Market
Overall (N-2.050)
Confidence Inserval
Pairwtss perzlue compariin to No.
Pet. more than
Superforecaster median Ebeccast
FoneasslBench
0.091
0,062
0.076
[0.067. (1086]
0%
Publicc modian foreesst
FenessiRench
0.119
0.072
0.096
[0.08K 0.105)
<0.001
2398
GPT
OpenAl
Preeze values
Scratchpad
0.175
0.085
0.130
[0.119,00 0.141]
<0.00]
24%
Claude-3-S 3.50:-----Sone--000 2000022200
Auiluppi
Fieese values
Scrnchppdd
0.354
0.107
0.131
[0.115. (1.445
<00000
24%
GPP--4TTubo-2100 1ube--1124-04-09
OpenAl
Freese values
Scranclpad
0.164
0.101
0.135
[0.121. 0.1451
-00000
23%
GPT-44
OpenAl
Nows with frocze valurs
Senachpud
0.171
0.104
0.137
[0.125. 0.144)
<0.001
2096
Gemin: 1.5 Pro
Google
Theaze values
Scraschpad
0.152
0.130
0.141
[0.130, 0.152
<0000]
21%
Gemin: 1.5 Pro
Google
News with freeze values
Scrachppd
0.154
0,133
0.145
[0.133, 0.154
<0.00]
21%
Claude-3-S 3-5-Sonnet- 202440211
Anthoopi
News with freeze values
Scrnchipad
0.260
0.130
0.345
10.152, 0.155)
+000001
209
Chaule11-Sa 1-mule-1---Somm---e --0000---20000
Anthopi
Frezze vallue
7ar shit
0.174
0.119
0.146
10 0 (60)
-00001
2256
Gemini1 5.Prro
Google
Seratchyual
0.152
0.143
0.14%
10.157 0. 858)
-0.001
309
GPT 4 Turbo 2024-0002000 04-0000-9
OpenAl
Scraichpod
0.164
0.152
0.148
[0.138, 0.
+0.001
17%
Gemini- 1.5.Pro
Google
Scraccyadd
0.154
0.143
0.14%
10.157
Chanl-3-SS-B 2---Sommn---000000
Amthopic
Seratchuud
0.149
0.149
GPT-4)
OpenA
Seratchyud
0.122
0.149
1000
Clande Opus 20240229
Antromic
Pheese values
Zero
0.173
0.124
0.149
10125
10000
21%
OpenAi
News
Scratchpad
0:371
0.127
0.349
10115%
1000
195
GPT T0r0-22244000000
OpenAl
Frezze values
Zero shit
0.200
0.100
0.350
10115%
00000
24%
Qwen1.5: 1-108--hu
Qwen
Frezze valluce
Seratc bqud
0.171
0.131
0150
20000
16%
Clande 3.5 Sonnet 20240620
Anthropic
News
Scratchpad
0.160
0.169
0.154
10110 9100
-0000
19%
Ingutted Koceaster
FareaslBench
0.250
0.059
0.155
101+7,
1869
-0000
2258
GPI-4
OpenAl
Freeze values
Zero shot
0.213
0.999
0.156
-0000
1000
215
Gemini-155 5-Pmo
Google
Frozse valure
Zzro shat
0.305
0.110
0.157
[0 11 I1
-000
10000
2096
Clande 5.5 Sonner 20240620
Anthopic
News
Superforecaste:
0.167
0,149
0.158
(.169)
2000
10001
17%
GPT-4
OpenAl
Theeze values
Scrotchpud
0.190
0.125
0.158
1O1RS 11871
19%
Clande-3-ppu- 20240229
Anthropis
Freeze vallees
Scratchpadd
0.385
0.154
0.159
11311
200000
10000
1898
LLM Crowd
Crod
OpenAl
FomeasiBeenh
FomeantBench
News
Nows
Nows with frocze values
Scratchyud
0.309
0.242
0.241
0,083
0.114
0.090
0.162
0.161
0.161
10155
1.1 11
01 0166
0.0 1.0
2000
2000
500000
20000
10000
100
304
18%
18%
Croed
FancaasBench
0.245
0,082
0.162
200000
10000
Gemini- 5-Pro
Google
Superfoneenster
0.151
0.164
5000
2000
Mistral-Lang--1
Freeze
Serstchued
0,143
0.164
10000
4PT-4
OpenAl
Scratchpad
0.140
0.165
50000
10000
15%
Qwen1 5-110B-Chuu
Owen
Scratchyadd
0:371
0.161
0.366
10 15% 112
100000
15%
Gemini-
Google
Zeso shit
0.205
0.128
0.367
10 1122
10000
19%
Gemini- 5.F.lash
Google
Freaze vallues
Senatc bpud
0.179
0.154
0.167
15% 10 1800
-0000
2000
1896
Clande 2.1
Anthropic
Scratchpad
0.228
0.105
0.167
10.157, 1.17
10000
2014
Llaon-3- -0---Cha--H
Meta
Freese valless
Zeio shot
013055
0.122
0.168
10.155 111921
1
19%
Liama- Cob-Char-Ht Hr
Mena
Fiease values
Scratchpad
0.208
0.129
U.169
10.15% 11379
100000
17%
Chanlo-3-Qpus- 20140029
Anthmpic
Zaro shot
0.173
0.165
0.169
[0. 0.18
<0000
18%
Gemini 1.5 Flash
Google
Freeze values
Zero shot
0.217
0,122
0.169
10.155 0.185
<0000]
23
GPT-4 Turbo 2024 01 09
OpenAl
Zero shot
0.200
0.139
0.169
[0.157, 0.182
<0.00]
19%
OpenAl
News
Scrucchpadd
0.209
0.151
0.370
10.159, 11
<000001
175
Gemini- 155-lash
Google
Scraichpod
0.179
0.161
0.170
10.159 0.16
<0.001
16%
GPT 4 Turbo 2024 04
OpenAl
News
Superforecaster
0.139
0.370
10.150
<00001
Qwen1 1100 Chat
Owen
News with freese values
Scratchpad
0.196
0,166
0.172
[0.161. 0.185
<0.00]
16%
Cladde33S- Set.:3.--SS--2222444005
Anthopic
Zao
0.174
0.171
0.372
10.155. 0.387
Mistral-Lange- Latest
Misral
Freeze values
0.145
0.174
180
Chadls2.
Amhoopic
Seratchuud
0.120
GPT 4o
OpenAl
Superforecaster
0.206
0.145
0.175
0.186]
<0.001
16%
Noros:
This perfomaance all stundard questions the human question set those combination questions from LLM question set where humans this combined question Human forecusts combunation
forecasts both from their torecasts on the componess questions by mouming independence (whuch not always the case. putting humans at disadvantage) Evallutted at the 7,, 30. 00- and 100-day fonccasi hontonn. full lestirhhard For reselved market questions, foeecasts are compared agains ground muhh while umbsaliedd market quesinces, they are compared community aggregates,
LLM
fornesstennhh.ngg Online results updated
differentt than version
here.
4. The overall score calculated as the overage of the menn datase: Brit score and the mean market Brigr score. y cocmpurang
p valule calculated by buststrappigg the differences in overall score between model best forgeistrr the null hypothesis (boolstraped): difference.
6 Pat. more accuratc than 1: The percent of questions where this forocaster hnd a better overall seare than the hest forecuster.
NUM SUMMARIES THRESHOLD: The threshold number of summaries to generate. This is set to 10.
PRE FILTER WITH EMBEDDING: A boolean flag indicating whether to pre-filter articles using embeddings. This is set True
WIITER_WITH___ EMBEDDING THRESHOLD: THEEDDING_TRRESHOOLD:: The threshold for pre-filtering articles using embeddings. This is set to 0.32.
RANKING MONKIN NAME: The name of the model used for ranking articles. We use
spt-3.5-turbo-111
TEMPERATURE: RANKING_TEMPERAATE
The temperature setting for the ranking model, which controls the randomness of the output. We set this to 0.0 for deterministic outputs.
RANKING PROMPT TEMPLATE: The template used for ranking articles. In our configuration, use PROMPT DICT [ "rankingg) ["]"].
RANKING RELEVANCE THRESHOLD: The relevance threshold for ranking articles. This is set to 4.
35

---

Published as a conference paper at ICLR 2025
RANKING COSINE SIMILARITY THRESHOLD: The cosine similarity threshold used in rank- ing. This is set to 0.5.
SORT BY: The criterion used to articles. We sort by date.
BANKING_METHOLE The method used for ranking articles. We use llm-rating.
RANKING MONTON LLM: The specific method for ranking articles using the LLM. We use [icle_250_tokent meaning ranking articles based on their titles and the first 250 tokens.
NUM_ SUMMARIES T The threshold number of summaries to generate for final output.
This is set 20.
EXTRACT BACKGROUND URLS: boolean flag indicating whether to extract background URLs from the articles. This is set True.
Inference Hyperparameters: We the maximum output token length 2000 to accommodate reasoning and probabilistic forecasts. We set the model temperature to 0 ensure stable outputs.
How to reproduce run the Scratchpad with Information Retrieval baseline, follow these steps:
1. To run the information retrieval part:
(a) Insert all the necessary API keys in 1 1im_retrieval/
Specifically, add the News-
catcher and OPt keys.
(b) Run 1im_retrievall http///ttooooo/erreeaaa//// iip.nb
(c) Save all the retrieved news under folder called news.
2. To run the scratchpad with the information retrieval baseline:
(a) Insert all the necessary API keys src/he pers
http///oosttttsoooo/ooooocs prc//heepeer//cossttatsss py.
(b) Place the "news" folder in the same directory htt.i/aaaseeeesmmmaii... (c) Run src/base pic/base_eval/alll pwwurrent_taaee baselines /main.
as
src/base_evee
J.3 LLM "ENSEMBLE" BASELINE
To produce the LLM ensemble forecast, we query three models: GPT-4o, Hlaude3355-oomi and Gemini-1.5-Pro. We use three prompts crafted by superforecasters who were given explicit instructions to write prompts that would help an LLM produce accurate forecasts This results in 3 x 3 forecasts per question. We then show 3 LLM crowd baselines using the median, geometric mean, and geometric mean of log odds.
Prompts We use the superforecaster-writte prompts shown in the appendix of our paper as Superforecaster Prompt 1-3.
Inference hyperparameters We set the maximum output token length to 2000 to accommodate reasoning and probabilistic forecasts. We initially considered a high token length of 3000, but after observing that the maximum response length around 1950, finalized 2000 as the optimal maximum token length. We set the model temperature to to ensure stable outputs.
How to reproduce To run LLM "Ensemble" bascline, follow the below steps:
Insert all the necessary API keys s:c/hhelpers/constatts
2. Place the "news" folder in the same directory strc/basee_vall//mmccrooo notebook. ipynb.
3. Run
K REPRODUCE RESOLUTION AND LEADERBOARD
Given the forecast files output from Appendix and Appendix J. the forecasts can resolved and the leaderboard created outlined below, after first having downloaded the benchmark codebase.
The Google Cloud Run Job in htc/resolve_forecasse forecast //w/min.... py resolves all forecasts on the questions from the question in Section B. To do this, depends
36

---

Published as a conference paper at ICLR 2025
the forecast files provided in Section B.2.1 and Section B.2.3;
the complete resolution files from our Question Bank on GCP Cloud Storage, which we cannot distribute freely because some providers not allow us to distribute their data directly, rather only modifications of their data. However, the code to create these resolution files is provided under stc/queste ons and can be created given the API keys to the data sources.
Having resolved the forecasts for the day, either ground truth if was forecast on a dataset ques- tion, or the resolution value or market value for market questions, we can now create the leaderboard. To do this, we use the Google Cloud Run Job defined in src/ leaderbba rd /ma .n.
L GENERAL PUBLIC SURVEY DEMOGRAPHICS
We collected demographic information from the 500 human forecasters in the general public survey. Summaries of participants' age, gender, ethnicity, and country of residence are shown in the tables below.
Table 21: Age Distribution
Age
Percentage
18-24 years old
32.0%
25-34 years old
43.4%
35 44 years old
14.4%
45-54 years old
5.4%
Over 55
4.8%
Table 22: Gender Distribution
Gender
Percentage
Male
53.4%
Female
46.2%
Prefer not to say
0.4%
Table 23: Ethnicity Distribution
Ethnicity
Percentage
White
48.6%
Black
33,4%
Mixed
8.6%
Asian
4.4%
Other
3.4%
Prefer not to say
1.6%
We did not collect similar demographic information from the superforecasters participating in the study, but are reasonably certain that the superforecasters in this study are roughly representative of superforecasters
whole. Describing forecasters previously recruited by Good Judgment Project, Mellers et al. (2015) noted that they "tended to be men (83%) and U.S, citizens (74%), with an average age of 40 years."
M PERFORMANCE BREAKDOWN
Tables Table 25 and Table 26 show the performance of the top LLM Cllade 3.5 Sonnet (using the Scratchpad prompt with freeze values) compared with Superforecasters, evaluated by forecast category and horizon.
37

---

Published as a conference paper at ICLR 2025
Table 24: Country of Residence Distribution
Country
Percentage
South Africa
31.2%
United States
15,4%
Poland
9.0%
Portugal
7.8%
United Kingdom
4.6%
Mexico
4.2%
Chile
3.8%
Italy
3.4%
Greece
3,0%
Hungary
3.0%
(25 others)
14.6%
In Table 25, we see that Claude 3.5 Sonnet slightly outperformed superforecasters on Environment Energy questions.
As reminder, we ask for forecasts on dataset questions at 8 diffreent forecast horizons, as explained in Section 3.2. Here, we breakdown performance at the forecast horizons that have resolved by the publication date: days, 30 days. 90 days, and 180 days. We see that superforecasters outperformed Claude 3.5 Sonnet at all forecast horizons, except for the 90-day horizon where Claude 3.5 Sonnet performed better.
Table 25: Brier score by category
Category
N
Claude 3.5 Sonnet
Superforecaster
Difference
Arts & Recreation
8
0.102
0.066
0.036
Economics & Business
181
0.224
0.134
0.090
Environment & Energy
81
0.126
0.141
-0.015
Healthcare Biology
0.003
0.001
0.002
Politics & Governance
17
0.066
0.022
0.044
Science Tech
14
0.207
0.175
0.032
Security & Defense
95
0.056
0.030
0,026
Sports
0.058
0.034
0.024
Table 26:
Brier score by forecast horizon
Forecast Horizon
N
Claude 3.5 Sonnet
Superforecaster
Difference
day
106
0.093
0.076
0.017
30-day
112
0.163
0.121
0.042
90-day
110
0.151
0.172
-0,021
180-day
169
0.110
0.082
0.028
38


Reviews:
[{"review_id": "ejXjEHgPm2", "reviewer": "Reviewer_tDYK", "summary": "A dynamic dataset of forecasting qs about the future is aggregated and proposed as a benchmark for LLMs. Performance of a variety of LLMs is compared with human.", "questions": "The level of technical details and insight are the biggest things that would make me change my score.\n\nDetailed comments by section:\n\nabs/intro/relwork\n - \"necessary task\" seems a bit overstated; if you want to make this claim it would be worth stating how exactly the referenced papers support that claim\n- reasonably clear intro/positioning and motivation overall; a bit repetitive but lacking technical details (e.g. how are predictions elicited from humans? what kinds of mistakes do LLMs make?); it would be nice to mention that briefly \n- relwork: what is \"statistically imprecise\"?  - good lmeval section\npreliminaries\n - unclear details about the nature of forecasts in the dataset; are they open-ended questions / natural language? are the possible outcomes enumerated in advance? is there a char limit? etc.  - unclear explanation of the Brier score; the Forecasting paragraph says forecasters \"may\" assign probabilities to possible outcomes; if they don't how do you assign f? How do you assign a 0/1 outcome to questions like \"who will be the next US president?\" - using the character f is confusing notation (f is almost universally a function) but not a huge deal - \"strictly proper\" and why we would care about that / why that incentivises truthfulness is not explained at all.  - 3 different methods of prompting/rag are not explained at all - \"prompting\" for the humans that are compared to the LLMS is not explained at all\n\nsec 3 \n - does not seem necessary to have a flowchart here to say that everything goes to the question bank, and it takes a lot of space. It might be helpful to have one if it showed what are the different platforms and datasets, any pre-preprocessing that needs to be done, etc., but this is covered fine in Table 1. \n - Similarly for Fig 3; neither of these is contributing substantively to the paper; they could be moved to appendix to make space for more technical details.\n - it seems like the details in 3.2 contradict the \"preliminaries\" section\n \nsec 5\n- given the small dataset sizes here it would be interesting to explore exactly what LLMs do and don't get right", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Interesting and useful benchmark, whose position in the landscape is clearly described. \nLLM evals are extensive.", "weaknesses": "This is an interesting and useful dataset, but reads like a tech report or class assignment; the level of technical detail is superficial, and there is little to no exploration of what we learn about LLMs from this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EXARqz0FzY", "reviewer": "Reviewer_8v9S", "summary": "The paper presents a sophisticated and automated benchmark for evaluating forecasts that improves over prior benchmarks in that it will not become stale and does not rely on assumptions about models (e.g., knowledge cutoffs). Using the benchmark, they find that interestingly, SOTA models are still short of expert forecasters.", "questions": "- Is there anything easy (cheap) we can try to improve the capabilities of these models to forecasting beyond RAG-like things? I guess finetuning open-source models would be ideal but that is probably infeasible.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- very well written--well motivated, structured, and polished overall\n- addresses a key problem in prior benchmarks: benchmark staleness, which is obviously very important for the problem of forecasting, since the goal is to build/evaluate systems that can predict new events, not post-hoc. their approach is conceptually cleaner (imo) than relying on LLMs' knowledge-cutoffs, etc.\n- overall interesting finding that at least in this setting, LLMs still quite lag behind experts, only matching the median of average people \n- the approach to building/generating the question set and the entire pipeline is very thoroughly thought out.", "weaknesses": "- Table 2 is a bit big / dense and hard to parse. The paper could benefit from using more visual plots to convey the results", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aj301ELUKg", "reviewer": "Reviewer_UK3t", "summary": "The paper introduces ForecastBench, a novel dynamic benchmark designed to evaluate the forecasting capabilities of AI systems, particularly large language models. In contrast to prior static benchmarks, ForecastBench continuously updates with new forecasting questions about future events that have no known answers at the time of submission, thereby eliminating concerns of data leakage or contamination. The study compiles a standardized set of 1000 forecasting questions sourced daily from nine data providers, including prediction markets, forecasting platforms, and real-world time series. To evaluate the accuracy, the authors assess the performance of 17 state-of-the-art LLMs on a subset of these questions, comparing their forecasts to those of expert human forecasters (superforecasters) and the general public, using the Brier score as the primary metric.", "questions": "- Have you investigated the impact of the prompt date on model predictions? Specifically, could questions asked further in advance of the resolution date introduce greater uncertainty in LLM predictions, potentially affecting prediction accuracy?\n- How is the quality of questions released for evaluation determined?\n- Have you considered evaluating model predictions by examining the logits of the answer tokens? Although this may be more feasible with open-source models, some providers do offer token-level logit outputs. Incorporating logits could provide insights into model calibration and further enhance prediction reliability.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- A sophisticated evaluation framework that incorporates multiple forecasting dimensions, including:\n    - Selection of questions from high-liquidity market sources,\n    - Temporal sampling methodology for question release,\n    - Granular analysis of model and human performance across varying time horizons.\n    - Comprehensive assessment of instruction-following chat models through six distinct evaluation methodologies (Section 5.1), including zero-shot prompting, scratchpad reasoning prompting, and news-augmented generation.\n- Implementation of diverse temporal resolutions for forecasting questions, spanning from week-long to decade-scale predictions, enabling robust evaluation across different time scales.\n- Novel incorporation of combination questions for assessing model performance on higher-complexity tasks, revealing significant human performance advantages and highlighting current limitations in model capabilities.\n- The development of a daily question-release mechanism featuring 1000 forecasting queries, which:\n    - Mitigates potential gaming of the leaderboard system,\n    - Generates an expanding dataset of resolved predictions enabling model training and calibration of future forecasting systems.", "weaknesses": "- The evaluation of unresolved questions relies on comparisons with crowd predictions, potentially complicating future assessments of whether models outperform the crowd baseline.\n- The number of expert predictions was limited, with only 39 experts contributing, leading to an average of 8 expert predictions per question. This relatively small sample size may affect the robustness and generalizability of the evaluation results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Yt4x7XHHS", "reviewer": "Reviewer_4xz8", "summary": "This paper proposes a forecasting benchmark for evaluating LLM's forecasting capability. The benchmark is carefully constructed to avoid potential data leakage. The benchmark results indicates with statistical significance that even the SOTA LLMs nowadays cannot beat human experts in this task.", "questions": "1. What if we allow LLMs to access different tools to actively acquire relevant information in a feedback loop? In this way the LLM may construct more context for more accurate answer.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The forecasting problem studied in this paper is both very interesting and practical. Forecasts on macroeconomic problems such as presidential election is both complicated and impactful to real-world scenarios. Hence it can serve as a good benchmark for evaluating LLM's capability.\n \n2. The benchmark design considers data leakage, which improves the reliability of the benchmark.\n\n3. The benchmark results indicates some areas where LLMs still have inferior performance compared to top human, revealing potential future research directions.\n\n4. Code is provided for better reproducibility.", "weaknesses": "1. It may be worth justifying the adopted method in this benchmark for LLM to make forecasts, since LLM reasoning methods (e.g. RAG, etc.) can have great influence on the final performance", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GI5wwGBaOd", "reviewer": "Reviewer_zsfr", "summary": "This work presents a benchmark for evaluating how well LLMs predict future events, curating questions about current events and corresponding answers. A set of state-of-the-art LLMs are then prompted to predict the probability of future events. The prediction accuracy is then compared against expert and non-expert human forecasters, ultimately finding that the LLMs generally perform similarly to average forecasters but worse than expert forecasters.", "questions": "Answers to the following questions would help me better calibrate my review:\n1. Is there intuition for why the LLMs would perform better on average for the questions sampled for humans? It seems like there's a consistent drop in the Brier Score.\n2. How do LLM and human questions differ? Could these differences explain any differences in accuracy? Maybe this relates to L265---what's different about their information-availability?\n3. What is \"News with freeze values\"?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "This paper has many strengths:\n* Dynamic benchmarks are needed and event forecasting seems like a sensible LLM use to evaluate, likely interesting members of the research community\n* This paper seems to describe a mature and already-used system that could elicit some interesting use\n* The finding that LLMs can forecast events similarly to average forecasters is interesting, and the fact that experts are still better is a promising sign that this area deserves further attention\n* The use of real human experiments is a big strength", "weaknesses": "This paper could be strengthened by addressing the following weaknesses:\n1. The main weakness of this work is the limited experiment section. Given the main contribution is the curation of a new dataset and resultant evaluation, it is unsatisfying that the result is only one page of experiments with minimal interpretation. The result is it's unclear if this benchmark is useful beyond this experiment. Relatedly, it's never really shown why it's important for this to be a \"dynamic\" benchmark, despite this being a key selling point. Avoiding data leakage is indeed important, but it's surprising that this doesn't allow for any dedicated experiment or discussion around how often questions are resolved during the benchmarking.\n2. The method development is described without many details, especially explanations of design decisions and their impact. Instead, most of Section 3 simply states what was done. It's not clear that future researchers could improve on this work if there's no explanation of what alternatives exist and why they are insufficient. This isn't inherently a major concern, but the main contribution of the work appears to be the dataset development, so defending these decisions would help the reader gain confidence in the resulting data and findings.\n3. The paper is hard to follow, with many details sprinkled throughout various sections and the reader is left to hunt them down. It could help to include more details on the forecasting problem and summaries of exactly what's provided in Section 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AzeZWYrYkC", "reviewer": "Reviewer_1r9B", "summary": "The paper introduces ForecastBench, a dynamic benchmark for evaluating the forecasting capabilities of both human and machine learning systems. This benchmark continuously updates with new forecasting questions, addressing the limitations of static benchmarks. The paper includes comprehensive experiments comparing the performance of various LLMs and human forecasters.", "questions": "Questions in Weakness Section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Interesting paper and relevant problem statement. Human annotated data generation in a streamlined manner is a challenge and the methodology proposed in the work can be used for updating the evaluation set for LLMs with time.\nThe work also discusses the evaluation on complex questions comprising of simple questions.\n\nPaper is well-written.", "weaknesses": "Overall concerns-\nForecasting based on questions is a highly subjective problem. The questions and their respective responses could vary depending on the demographics, education background, understanding of the domain, etc of the forecasters. Does the authors have any openly accessible user-study report highlighting the biases and how they are being addressed. This follows for both - human forecasters as well as \"superforecasters\"\n\nI'm wondering how does the 1000 questions designed for evaluation covers forecasting of an open world set? Or is it only targeted to a few domains?\n\nThe question examples shown in Figure 4&5 seem at a very high-resolution. Although, it is appreciated that the source of the questions is provided, but \"is 2024 going to be hotter than 2023\" - What parameter is being considered here for comparison? is it mean temperature? Can LLMs infer these for a new undefined concept?\n\nMethodology relies on online links and APIs, thus limited in its coverage of questions.\n\nReproducibility is missing; validation is difficult.\n\nUser -study report should be provided publicly. What's the demographics of the human forecasters?\n\nReliability and risks associated with the methodology - Is stress test performed on the methodology for use in real-world? What are the metrics?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)