ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities

## Abstract

Abstract Forecasts of future events are essential inputs into informed decision-making. Machine learning (ML) systems have the potential to deliver forecasts at scale, but there is no framework for evaluating the accuracy of ML systems on a standardized set of forecasting questions. To address this gap, we introduce \BenchmarkName : a dynamic benchmark that evaluates the accuracy of ML systems on an automatically generated and regularly updated set of 1,000 forecasting questions. To avoid any possibility of data leakage, \BenchmarkName is comprised solely of questions about future events that have no known answer at the time of submission. We quantify the capabilities of current ML systems by collecting forecasts from expert (human) forecasters, the general public, and LLMs on a random subset of questions from the benchmark ( N = 200 𝑁 200 N=200 italic_N = 200 ). While LLMs have achieved super-human performance on many benchmarks, they perform less well here: expert forecasters outperform the top-performing LLM (p-value = 0.01 absent 0.01 =0.01 = 0.01 ).
We display system and human scores in a public leaderboard at www.forecastbench.org .

## 1 Introduction

Forecasting the future is a challenging but necessary task (Armstrong, , Tetlock and Gardner, ). Economic forecasts influence investment and hiring decisions (Christensen et al., ). And forecasts of the Covid-19 pandemic in early 2020 prompted local lockdowns to slow the spread of the virus Adam ().
However, human forecasting is often expensive, time-consuming, applicable only in specific domains, and subject to concerns about human biases. Motivated by these limitations, recent work explores the use of machine learning (ML) models, especially large language models (LLMs), in automated forecasting (Hendrycks et al., , Zou et al., , Yan et al., , Pratt et al., , Phan et al., , Fluri et al., , Halawi et al., ).

To assess LLMs’ forecasting capabilities, prior work built static benchmarks of questions where the answer was known (resolved) after the knowledge cut-offs of the LLMs they test (Zou et al., , Yan et al., , Halawi et al., ). For example, “Will a nuclear weapon be detonated in 2023,” though resolved on Jan 1, 2024, is a valid out-of-sample question for testing a model with a known knowledge cut-off before the end of 2023.

This static evaluation methodology comes with three key drawbacks. First, as the knowledge cut-offs of frontier models are updated, static benchmarks become obsolete, necessitating further data-sourcing. This makes it difficult to continuously track and compare the top models in the field. Second, knowledge cut-offs are usually estimated using the time range of pre-training data. Such estimates may not be accurate, and post-training may inject further post-cutoff knowledge into the model, contaminating the test sets.
Lastly, model developers face financial incentives to exaggerate their accuracy on benchmarks and claim that their models are state-of-the-art performers. While some fudging is easily identified, other subtle benchmark manipulation or overfitting is harder to catch, and some studies have shown that there is significant evidence for benchmark contamination and/or memorization in popular LLM models (Roberts et al., , Elazar et al., , Li et al., ).

To address these drawbacks, we introduce \BenchmarkName, a dynamic benchmark that is continuously updated with new questions about future events. Our automated system gathers new questions from nine data sources on a daily basis, including prediction markets, forecasting platforms, and real-world time series. We regularly elicit predictions on these questions from both automated systems and human forecasters. As they resolve, we update a public leaderboard to show participant performance. This process makes \BenchmarkName an accurate real-time benchmark of forecasting abilities.

Our initial benchmark consists of 1,000 standardized forecasting questions randomly sampled from a much larger real-time question bank. To establish baseline levels of performance, we record predictions from dozens of LLMs on the initial set, using methods like retrieval-augmentation to boost model performance (Lewis et al., , Halawi et al., ). We independently elicit predictions from human forecasters of varying expertise, including a group of seasoned forecasters (superforecasters) (Tetlock and Gardner, ) who have consistently performed well in competitive forecasting tournaments. As questions resolve, we rate the submissions of registered models and the human comparison groups, updating our public leaderboard.

Our initial results indicate that state-of-the-art models, such as GPT-4o and Claude-3.5, perform only roughly as well as a simple median of forecasts from a survey of humans with no (or minimal) forecasting experience, even when the LLMs are augmented with news retrieval and prompt engineering, and when the models have access to forecasts from a human crowd (on market-based questions). The models also perform significantly worse than the median of expert forecasters (superforecasters) who answer the same questions.

These findings leave significant room for researchers to attempt to improve AI-based forecasting systems using innovative approaches, such as developing methods for continuously updating models with current events and enhancing LLMs to reason over extended time frames. To support progress in this area, we publish an auxiliary dataset of LLM and human forecasts, rationales, and accuracy for use in future LLM fine-tuning and testing.

### 1.1 Related work

##### Automated forecasting

ML systems that make accurate forecasts can help inform human decision-making
(Hendrycks et al., , Schoenegger et al., ).
Recent work studies the use of LLMs for automated forecasting Jin et al. (), Zou et al. (), Yan et al. (), Halawi et al. (), Pratt et al. (). These papers all build static benchmarks of resolved questions. A recent paper from Halawi et al. () uses questions resolved between June 2023 and January 2024. Unfortunately, the latest LLMs, such as GPT-4o, have knowledge cut-offs past this point. This fact motivates our work to build a dynamically updating benchmark that can accurately evaluate frontier model performance.

In addition, Schoenegger and Park () and Abolghasemi et al. () compare the accuracy of GPT-4 and other LLMs to human forecasters. Schoenegger et al. () found that an ensemble of $12$ LLMs rivaled human forecasts in a forecasting tournament in 2023 with a small number of questions, but the results are statistically imprecise. Unlike our work, that tournament was run only once and is no longer updated. Also, our much larger question set allows us to make precise statistical claims about the performance of LLMs relative to each other and relative to human performers.

Finally, recent work focuses on the use of LLMs and transformer-based systems in statistical time-series forecasting (Nie et al., , Gruver et al., , Dooley et al., , Rasul et al., , Jin et al., , Das et al., , Goswami et al., , Woo et al., ), but many of the most important forecasting questions do not have well-defined time series that can be used in standard statistical forecasting models (e.g., what is the probability that a nuclear weapon will be used offensively in the next ten years?). Our benchmark is more general, and evaluates automated forecasting performance across questions with and without underlying time series and historical baseline data.

##### Language model evaluation

Evaluating highly capable LLMs is a challenging task—with many models saturating key benchmarks soon after their release (Maslej et al., , Owen, ) and with benchmarks potentially leaked to models’ training data (Magar and Schwartz, , Sainz et al., , Jacovi et al., , Xu et al., , Balloccu et al., , Jiang et al., , Zhang et al., , Xu et al., ).
Our dynamic forecasting benchmark avoids both of these issues. First, automated forecasting is highly unsaturated: Halawi et al. () showed that under simple zero-shot evaluation, frontier models such as GPT-4 are much less accurate than aggregates of human predictions. Second, our benchmark is dynamic. The final resolution of each question is only determined in the future and cannot be ‘leaked’ in any training data (by construction). This eliminates concerns of contamination.

## 2 Preliminaries

##### Forecasting

A forecasting question asks for a probabilistic prediction of a future event. A forecaster may assign probabilities to potential outcomes of the event. Forecasting platforms, including prediction markets, host a wide range of questions. Many questions are of public interest, such as “Who will be the next US president in the 2024 election?”

##### Metrics

For binary questions, we use the Brier score as the performance metric, defined as $(f-o)^{2}$, where $f\in[0,1]$ is the probabilistic forecast and $o\in\{0,1\}$ is the outcome. Lower Brier scores are better, and a score of $0.25$ is associated with the uninformed forecast of $0.5$. Brier scores are strictly proper, incentivizing truthful reporting from participants.

##### Models

We evaluate 17 LLMs on our initial benchmark: GPT-3.5-Turbo-Instruct (Brown et al., ), GPT-4 (OpenAI, ), GPT-4o, Llama-2-70B (Touvron et al., ), Llama-3-7B, Llama-3-70B, Mistral-7B, Mistral-8x7B (Jiang et al., ), Mistral-8x22B, Mistral-Large, Qwen1.5-110B-Chat, Claude-2.1 (Anthropic, ), Claude-3-Haiku, Claude-3.5-Sonnet, Claude-3-Opus (Anthropic, ), Gemini 1.5 Flash and Gemini 1.5 Pro (Gemini Team, ).

We record model predictions under three different methods: zero-shot prompting, scratchpad prompting, and scratchpad prompting with retrieval augmentation. For the last setting, we use the retrieval infrastructure from Halawi et al. () and provide relevant news articles to the models in-context to reason about. Additionally, only models with a context window larger than 8,000 tokens were evaluated under the retrieval setting due to the inclusion of news articles in some prompts.

## 3 Benchmark, leaderboard, and datasets

We maintain a question bank with a growing set of forecasting questions. Continuously adding questions to the question bank allows it to stay relevant and ensures that we have a large selection of unresolved questions to sample from.

Every night, our automated system updates the question bank with new questions and resolution values. We drop invalid, low-quality, and resolved questions, categorizing the remaining questions by topic. The process is fully automated as detailed in Section 3.1.

Every two weeks, we sample from the question bank and release question sets for teams interested in submitting their forecasts to the benchmark. We also survey a standard set of LLM-based models to allow for comparisons of performance over time. The forecasts that these groups submit are resolved continuously with daily updates to our leaderboard. We provide the resulting data on forecast questions and submissions to researchers. See Section 3.2 for details. Finally, we discuss the question resolution procedure in Section 3.3 and our leaderboard design in Section 3.4.

### 3.1 Updating the question bank

In an automated process that runs nightly at 0:00 UTC, questions are added to the question bank, resolution values are updated, and metadata generated, as shown in Figure 1. Details follow.

Figure: Figure 1: The automated system that pulls questions from forecasting platforms and datasets for question generation from data providers.

#### 3.1.1 Updating questions and resolution values

We bring in questions from two broad types of sources: markets and datasets.(^1^11Licensing information can be found in Section C.1.) We select multiple, reliable sources from each type, ensuring the diversity of our benchmark. See Table 1 for details.

##### Markets

As one of our two main question sources, we rely on a handful of prediction markets and forecast aggregation sites that elicit human predictions on questions across a wide range of topics.(^2^22Hereafter, for brevity, we refer to questions that come from both prediction markets and forecast aggregation sites as ”market” questions. We also refer to theses sources collectively as ”market” sources.) When selecting questions from market sources to add to the question bank, we choose those with high levels of active human trading on these platforms (liquidity) as these questions tend to be of higher quality than those with low levels of human trading.(^3^33See Table 7 for an example question pulled from a market source.) We also store the latest market and resolution values for each question.

##### Datasets

As a second question source, we rely on 5 well-established and well-maintained datasets that track real-world events (e.g. ACLED (Raleigh et al., ), a geopolitical database that tracks worldwide conflict, including facts like the number of protests in Niger each month). With these dataset sources, we can generate questions using a fixed question template (e.g., “Will the number of protests in Niger increase by at least 10% over this month’s value by the resolution date?").(^4^44See Table 8 for an example question pulled from a dataset source.)

##### Question Bank

Table 1 lists the sources of market and dataset questions, along with the number of questions available in our question bank, whence we sample questions for our benchmark runs. In addition to the main questions (with sample size $N$), we also construct additional combination questions by choosing pairs of questions within each source. We describe combination questions in more detail in Section 3.2.

**Table 1: Question bank composition, grouped by source type (market or dataset).**
| Source | URL | N | $\binom{N}{2}$ |
| --- | --- | --- | --- |
| RFI | [randforecastinginitiative.org](https://www.randforecastinginitiative.org/) | 15 | 105 |
| Manifold Markets | [manifold.markets](https://manifold.markets) | 392 | 76,636 |
| Metaculus | [metaculus.com](https://www.metaculus.com) | 800 | 319,600 |
| Polymarket | [polymarket.com](https://polymarket.com) | 534 | 142,311 |
| Market Total | 1,741 | 538,652 |  |
| ACLED | [acleddata.com](https://acleddata.com) | 3,150 | 4,959,675 |
| DBnomics | [db.nomics.world](https://db.nomics.world/) | 52 | 1,326 |
| FRED | [fred.stlouisfed.org](https://fred.stlouisfed.org) | 166 | 13,695 |
| Wikipedia | [wikipedia.org](https://www.wikipedia.org) | 335 | 55,945 |
| Yahoo! Finance | [finance.yahoo.com](https://finance.yahoo.com) | 504 | 126,756 |
| Dataset Total | 4,207 | 5,157,397 |  |
| Question Bank Total | 5,948 | 5,696,049 |  |

#### 3.1.2 Updating forecast question metadata

After we automatically collect forecasting questions and resolution values in our standardized question bank, the data get processed in several ways. This generates more information about the questions and creates another sampling option for the creation of the final question set. Following Halawi et al. (), we use gpt-3.5-turbo-0125 to categorize questions by subject and to filter out low-quality questions. We report the number of questions by category and source in Table 14, where we display the breakdown of the standard questions from our question bank ($N$ from Table 1) across question categories by source, after removing low-quality questions.

### 3.2 Generating the question set

Every other Sunday at midnight UTC, we release a set of 1,000 forecast questions for LLMs through the process outlined in Figure 2. We sample an equal number of questions from each data source, to ensure representativeness.
Within each source, we then uniformly sample questions across all question categories, aiming for an equal distribution from each category within each source. This ensures that models cannot be overfit to a specific type of question or topic. Limiting the number of questions generated to 1,000 also ensures that costs for testing LLM systems on the benchmark are capped for development teams with fewer resources.

For questions derived from dataset sources, the distribution of resolution dates is as follows, as represented by days past the forecast due date: [7, 30, 90, 180, 365, 1,095, 1,825, 3,650]. In other words, the distribution of resolution dates spans from one week into the future to ten years into the future. For questions derived from market sources, we ask only for the probability that each question will resolve positively (will the event underlying the question occur, or not occur). Prior to a given market question’s resolution, performance is evaluated by calculating the squared distance between the forecasted value and the crowd forecast on the platform from the previous day. In the main public leaderboard we report performance using all questions—including unresolved questions—but we also separately report performance on resolved questions for interested readers.

Figure: Figure 2: We sample from the question bank every two weeks to produce question sets.

For every data source, there are two types of questions, each comprising half of the question set. The first type is a standard forecasting question with a binary outcome, e.g. “Will inflation (core CPI) be above 3% next month?” We construct the second type, combination questions, by pairing two standard questions. For combination questions, we ask for forecasts on all Boolean combinations of the two questions (i.e. $P(Q1\cap Q2)$, $P(Q1\cap\neg Q2)$, …). Considering the extensive number of existing standard questions and potential combinations in our question bank (as we could potentially combine 3, 4, or more standard questions together), we effectively have access to millions of possible forecasting questions to sample from. We show the number of two-question combination questions in the question bank as it stands at time of writing in the right-hand column of Table 1.

Combination questions require forecasters to consider the covariance structure of different events, some of which are more independent than others. For instance, the best forecasts of whether the S&P500 (a key U.S. stock market index) will reach an all-time high and whether Spain will win the next Men’s World Cup are likely independent. However, the best forecast of whether the S&P500 will reach an all-time high and whether the U.S. will enter an economic recession must account for the likely strong correlation between these events.

To estimate baseline human and expert performance on these questions, we run surveys asking the general public and expert forecasters to predict on our question sets (see Section 4). The human question is comprised of 200 forecast questions sampled directly from the LLM set of 1,000 questions.(^5^55We can evaluate humans and models on more than 200 questions because each data question asks about the value of a datapoint at several different time horizons; and we consider accuracy of the model or human forecaster at each time horizon to be a different forecast.) We produce the question sets 10 days before the forecast due date to allow for time to create and run a human survey. LLMs receive their question set 24 hours before the due date, even though it was generated at the same time as the human question set. This gives us the ability to elicit forecasts, obtaining comparable forecast sets on the due date from both LLMs and humans who faced the same information environment. A detailed timeline is visible in Figure 3.

Figure: Figure 3: Timeline from question set generation to forecast due date, $D$.

### 3.3 Resolution

We resolve forecast sets nightly by gathering the latest information about which events have or have not occurred. All questions are ultimately resolved to the final true value.

##### Markets

Ground truth is not available for market questions before the events are known to have occurred or not occurred. Before events resolve, we use the platform prediction (an aggregate of human forecasts reported on the platform) from the preceding day as a proxy resolution value for unresolved questions. These values are updated nightly as described in Section 3.1.1. This allows us to score the complete set of forecast questions, incorporating all information available to people on a nightly basis. Once a market question is officially resolved, we score the forecast against the resolution value, creating a definitive score for the question across all forecast horizons.

##### Datasets

Datasets can be updated as new information becomes available. Thus, questions derived from datasets are continuously resolved to the value from the the latest available data.

##### Missing forecasts

We select 1,000 questions for the LLM question set to make the forecasting task impractical to complete manually within the 24 hour window after the question set is released. And we obligate all LLMs to forecast on all questions to ensure comparability of scores across models and human-based aggregates. So, when a model does not submit predictions on certain questions or time horizons, we consider that a model error and impute a naive value for the model to ensure comparability across models over time.

For market questions, as we only ask users to forecast the outcome of the question, missing forecasts on question outcomes are assigned the value of the crowd forecast on the forecast due date, day $D$ in Figure 3. Some may argue this is overly-generous, but we did not want teams to have a competitive advantage by simply scraping the market websites themselves.

For dataset questions, we impute the value $0.5$ (which represents a lack of information) to forecasts across all time horizons. Empirically, top models report valid forecasts on all questions and are not affected by this imputation procedure.

### 3.4 Leaderboards

We generate leaderboards, ranking models and humans by average overall score. The main leaderboard highlights the top forecasting submission across all questions and can by sorted by performance on the source type (market or dataset) and by score on resolved questions.

### 3.5 Datasets

As a key output of \BenchmarkName, we generate four living datasets that grow over time. See Appendix A for licensing details and Appendix B for data dictionaries.

##### General public forecast dataset

Every time we run the public survey outlined in Section 4, we provide multiple independent forecasts and rationales on every one of the 200 forecast questions and report the accuracy of the median public forecast. See Section B.2.1 for details.

##### Superforecaster forecast dataset

We provided a random subset of the benchmark questions to 39 expert forecasters with a strong track record of accurate performance on a diverse set of geopolitical questions. In addition to forecasts and rationale, the superforecasters provide pertinent information about their forecasting process, like search terms used and useful URLs consulted. See Section B.2.2.

##### LLM forecast dataset

Similar to the general public dataset, we prompt LLMs to produce forecasts on each of 1,000 forecast questions in the LLM question set. Their rationales are also included in the dataset whenever provided. See Section B.2.3 for details.

##### Question & resolutions dataset

In creating the benchmark, we have automated question creation and resolution from all of the sources outlined in Table 1. We provide these as a dataset that can be combined with the forecast datasets mentioned above. See Section B.1 for details on these datasets.

## 4 Human forecaster baseline

### 4.1 Public

We recruited 500 human forecasters via Prolific and Facebook advertising to participate as representatives of the general public. These human subjects completed a brief introductory survey to gather demographic information and evaluate performance on a few forecasting and comprehension tasks. We then drew 200 questions from the 1,000 question set provided to language models; human subjects completed a one-hour survey containing 20 random questions from this 200-item subset where they provided their forecasts and rationales for each question.

The number of responses per question in the 200-item subset varied to ensure representativeness across subject matters and sources; at least 40 responses were gathered per question, averaging 49 responses per question.

### 4.2 Superforecasters

We additionally recruited 39 expert forecasters with a strong track record of accurate performance on a diverse set of geopolitical questions (“superforecasters”) to participate in a 9-day forecasting experiment in which participants were prompted to give their individual forecasts for 20 random questions from the same 200-item subset of forecasting questions we provided to the general public. Roughly halfway through the 9-day experiment, participants were moved into a group forecasting stage in which we allowed these expert forecaster participants to see one another’s forecasts and rationales and to communicate with one another about each question. They were also given the opportunity to forecast on questions beyond the 20 questions assigned to them individually.

Because of the more limited quantity of the superforecasters, questions generally had fewer responses than in the public survey; a minimum of 3 forecasts were recorded for each question, with an average of 8 responses per question.

See Figure 4 for an example question from the human surveys. All data from human surveys are anonymized per IRB requirements.

##### Results

We evaluate human performance on the forecasting tasks by comparing predictions to the ground truth on resolved questions and comparing to community aggregates on unresolved questions. We find that the median public survey participant had an overall Brier score of 0.107 on the 200-item subset we drew from the LLM Question Set with 1,000 questions. The median superforecaster participant had an overall Brier score of 0.093 on this same subset of forecasting questions. See Section 5 and Table 2 for a comparison between human performance and LLM performance, as well as Appendix H for a top 50 leaderboard of human and LLM performance.

## 5 LLM baseline

In this section, we evaluate the forecasting capabilities of language models and report on the methodology and results.

### 5.1 Methodology

We evaluate a suite of instruction-following chat models without any additional fine-tuning (see Section 2 for details on the models). In each baseline, we prompt the model to generate a probabilistic forecast that the question will resolve to “Yes."

##### Baselines

We implement six baselines: (1) zero-shot prompting, (2) prompting with scratchpad instructions, (3) zero-shot prompting with crowd forecasts, (4) scratchpad prompting with crowd forecasts, (5) prompting with scratchpad instructions and retrieved news articles, and (6) aggregating predictions from multiple LLMs. Each baseline is described in more detail below.

- 1
Our first baseline, prompts the model zero-shot to generate a forecast directly without generating other content, such as intermediate thinking (Figure 6). By prompting the model to output its forecast directly, we assess raw forecasting capability without sensitivity to prompting strategies.
- 2
Our second baseline, prompts the model with scratchpad instructions (Nye et al., ) that outline a procedure the model should use to reason about the question (Figure 7). Our scratchpad prompt comes from (Halawi et al., ), which formed its prompts through a combination of analyzing the Brier score as prompt changes were made, and by adding language to fix common errors the LLMs would make, e.g., asking them to rephrase the question for understanding.
- 3
The question sets we provide to LLMs contain what we term freeze values for market questions, which are just the crowd forecast on the market the day the question set was created, as decsribed in Section 3.2. Our third baseline is the zero-shot prompt with freeze values.
- 4
Our fourth baseline is the scratchpad prompt with freeze values.
- 5
Since LLMs’ knowledge are not continuously updated, it is important to provide LLMs with up-to-date information relevant to the question (Zou et al., ). Our fifth baseline, scratchpad with information retrieval, uses the same scratchpad prompts that also contain retrieved news articles.
The retrieval system is the same as described in Halawi et al. (): an LLM generates search queries for a news API, filters articles for relevancy, and summarizes the articles.
- 6
The aggregated prediction of forecasters (“crowd" prediction) is a strong benchmark. For instance, Metaculus () shows that an ensemble of all forecasters consistently outperforms using just the top 5, 10, …, 30 best forecasters (based on past scores). In our final baseline, we aggregate the predictions generated by LLMs into an LLM “crowd" prediction.
To produce the LLM crowd forecast, we query GPT-4o, Claude-3.5-Sonnet, and Gemini-1.5-Pro. We use three prompts crafted by superforecasters who were asked to write prompts that would help an LLM produce accurate forecasts (Figure 8, Figure 9, Figure 10). This results in $3\times 3=9$ forecasts per question. We then show 3 LLM crowd baselines using the median, geometric mean, and geometric mean of log odds (Satopää et al., ). See details at Appendix E.

### 5.2 Results

##### Comparing Humans and LLMs

In Panel A of Table 2, we show that superforecasters achieve an overall mean Brier score of $0.093$, significantly outperforming both the general public (Brier $=0.107$, $p<0.01$) and the top LLM performer on the 200-item subset (Claude 3.5 Sonnet: Brier $=0.111$, $p\leq 0.01$).(^6^66See statistical note in Appendix G.) The top-performing language models all had access to the (human) crowd forecast on market questions. The top-performing model without access to the crowd forecast on market questions was significantly less accurate than models with access to the human forecast. The comparison between humans and LLMs relies on the 200 questions forecasted by humans, which is a random sub-sample of the 1,000 questions in the question set provided to LLMs.(^7^77Accuracy measures are based on more than 200 forecasts because human and LLM forecasters submitted multiple forecasts on each dataset question, one for each time horizon. The results presented here include forecasts over the 7-day and 30-day time horizons.)

##### Comparing LLMs

Panel B excludes humans and evaluates LLMs on the entire question set ($N=1,000$ questions). Here we see a similar ranking of models, with GPT-4-Turbo now slightly outperforming Claude-3.5 Sonnet. Both of those models outperform GPT-4o and Gemini-1.5 Pro. As in Panel A, most of the top-performing models use the scratchpad prompt (Figure 7) and use as inputs the human crowd forecasts for market questions. Access to recent topical news related to the questions did not improve performance.

**Table 2: Leaderboard with Top 10 Models**
| Panel A: Includes human comparison group and evaluates the set of 200 questions received by humans |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  |  |  | Brier Score $\downarrow$ |  |  |  |  |  |
| Model | Organization | Information<br>provided | Prompt | Dataset<br>(N=210) | Market<br>(N=77) | Overall<br>(N=287) | Confidence<br>Interval | Pairwise<br>p-value<br>comparing<br>to No. 1 | Pct. more<br>accurate<br>than No. 1 |
| Superforecaster median forecast | ForecastBench | – | – | 0.135 | 0.051 | 0.093 | [0.073, 0.112] | – | 0% |
| Public median forecast | ForecastBench | – | – | 0.167 | 0.048 | 0.107 | [0.09, 0.125] | 0.009 | 30% |
| Claude-3-5-Sonnet-20240620 | Anthropic | Freeze values | Scratchpad | 0.155 | 0.067 | 0.111 | [0.09, 0.132] | 0.01 | 33% |
| GPT-4 | OpenAI | Freeze values | Zero shot | 0.169 | 0.056 | 0.113 | [0.093, 0.132] | 0.003 | 37% |
| Claude-3-5-Sonnet-20240620 | Anthropic | Freeze values | Zero shot | 0.155 | 0.074 | 0.114 | [0.09, 0.139] | 0.008 | 36% |
| Claude-3-5-Sonnet-20240620 | Anthropic | News with freeze values | Scratchpad | 0.16 | 0.07 | 0.115 | [0.095, 0.135] | 0.004 | 31% |
| GPT-4o | OpenAI | Freeze values | Scratchpad | 0.173 | 0.059 | 0.116 | [0.098, 0.133] | 0.002 | 30% |
| GPT-4-Turbo-2024-04-09 | OpenAI | Freeze values | Zero shot | 0.177 | 0.055 | 0.116 | [0.097, 0.135] | <0.001 | 37% |
| GPT-4o | OpenAI | News with freeze values | Scratchpad | 0.179 | 0.059 | 0.119 | [0.101, 0.137] | <0.001 | 42% |
| Mistral-Large-Latest | Mistral AI | Freeze values | Scratchpad | 0.163 | 0.081 | 0.122 | [0.107, 0.137] | <0.001 | 29% |
| Always 0.5 | ForecastBench | – | – | 0.25 | 0.165 | 0.208 | [0.198, 0.218] | <0.001 | 17% |
| Panel B: Excludes human comparison group and evaluates the entire set of 1,000 questions |  |  |  |  |  |  |  |  |  |
|  |  |  |  | Brier Score $\downarrow$ |  |  |  |  |  |
| Model | Organization | Information<br>provided | Prompt | Dataset<br>(N=2,737) | Market<br>(N=912) | Overall<br>(N=3,649) | Confidence<br>Interval | Pairwise<br>p-value<br>comparing<br>to No. 1 | Pct. more<br>accurate<br>than No. 1 |
| GPT-4-Turbo-2024-04-09 | OpenAI | Freeze values | Scratchpad | 0.175 | 0.051 | 0.113 | [0.107, 0.118] | – | 0% |
| Claude-3-5-Sonnet-20240620 | Anthropic | Freeze values | Scratchpad | 0.179 | 0.047 | 0.113 | [0.108, 0.119] | 0.442 | 53% |
| GPT-4o | OpenAI | Freeze values | Scratchpad | 0.201 | 0.039 | 0.12 | [0.115, 0.125] | <0.001 | 45% |
| Gemini-1.5-Pro | Google | Freeze values | Scratchpad | 0.168 | 0.075 | 0.121 | [0.116, 0.127] | <0.001 | 42% |
| GPT-4o | OpenAI | News with freeze values | Scratchpad | 0.196 | 0.05 | 0.123 | [0.117, 0.128] | <0.001 | 41% |
| Claude-3-Opus-20240229 | Anthropic | Freeze values | Zero shot | 0.192 | 0.061 | 0.127 | [0.12, 0.133] | <0.001 | 44% |
| Gemini-1.5-Pro | Google | News with freeze values | Scratchpad | 0.172 | 0.082 | 0.127 | [0.121, 0.132] | <0.001 | 41% |
| GPT-4-Turbo-2024-04-09 | OpenAI | Freeze values | Zero shot | 0.199 | 0.055 | 0.127 | [0.12, 0.133] | <0.001 | 47% |
| Qwen1.5-110B-Chat | Qwen | Freeze values | Scratchpad | 0.176 | 0.078 | 0.127 | [0.122, 0.133] | <0.001 | 32% |
| GPT-4-Turbo-2024-04-09 | OpenAI | No | Scratchpad | 0.175 | 0.083 | 0.129 | [0.124, 0.134] | <0.001 | 7% |
| Always 0.5 | ForecastBench | – | – | 0.25 | 0.16 | 0.205 | [0.202, 0.208] | <0.001 | 22% |

As a particular failure mode, we find LLMs are specifically worse at combination questions. Although our human surveys did not explicitly ask for forecasts on combination questions, we bound human performance by assuming independence of the component of each combination question. This underestimates human accuracy because a human forecaster predicting the outcome of a combination question could account for independence between the permuted events. In Table 19, we present this comparison of human and LLM forecasts. We see that LLMs perform poorly on these combination questions, and including them in the benchmark widens the gap between human and LLM performance: superforecasters (Brier score of $0.068$) outperform the general public (Brier score of $0.083$) and the top LLM (Claude 3.5, Brier score of $0.119$) significantly. To benchmark the size of this gap in performance, the $0.051$ Brier score gap in performance between superforecasters and Claude 3.5 (the top-performing LLM) is significantly larger than the $0.031$ gap in performance between Claude 3.5 and Claude 2.1.

## 6 Discussion

We present \BenchmarkName, the first dynamic benchmark for evaluating automated forecasting systems and establish a series of human and LLM baselines. As the main result, we find that frontier LLMs fail to outcompete human forecasters. We believe that fine-tuning, better retrieval-augmentation and improved reasoning skills can be venues to improve LLMs’ forecasting abilities in the future.

From a practical point of view, we hope that our benchmark can shed light on how automated systems compare in accuracy to humans and when they might become the preferred source of predictions. Finally, from the perspective of LLM evaluation, forecasting requires models to integrate pre-training knowledge, engage in effective reasoning, and maintain calibration to avoid overconfidence. All of these are essential capabilities that are desired across various domains. We believe that performance on our benchmark can shed light on other capabilities of LLMs.

## Appendix A Licensing and maintenance

##### Hosting

The \BenchmarkName human and LLM-only overall leaderboards are available on [www.forecastbench.org](www.forecastbench.org). Combination leaderboards will be added soon. Documentation and Tutorials are forthcoming. We plan to open our bi-weekly forecasting rounds to LLM teams in November 2024.

##### Datasets

Our datasets, distributed under the [CC BY-SA 4.0 license](https://creativecommons.org/licenses/by-sa/4.0/deed.en), will be made available and updated regularly on [www.forecastbench.org](www.forecastbench.org).

##### Codebase

The code underlying our automated system runs on Google Cloud Platform and is available at [https://github.com/forecastingresearch/forecastbench](https://github.com/forecastingresearch/forecastbench) under the [MIT license](https://github.com/forecastingresearch/forecastbench/blob/main/LICENSE).

##### Maintenance & long-term preservation

We ensure the long-term availability and maintenance of the benchmark as it is funded by [Open Philanthropy](https://www.openphilanthropy.org/) until mid-2027. In case no further funding is provided beyond that point, we will upload our final datasets to GitHub and Hugging Face.

## Appendix B Datasets

We intend our datasets to be used for training general LLMs, fine-tuning forecasting LLMs, and for any applicable research purposes. No restrictions are placed on who may use our datasets, nor to what end.

### B.1 Question and resolution sets

Every question set will be published. Their resolutions will also be published such that there’s a complete training set when combined with the forecast sets outlined in Section B.2. The data dictionary for the question set is outlined in Table 3 and Table 4. The data dictionary for the resolution set is outlined in Table 5 and Table 6.

##### Data format

The question and resolution datasets are released as JSON (.json) files.

##### Ethical and responsible use

There are no restrictions on use of the question and resolution datasets.

##### Availability

[www.forecastbench.org/datasets/question_sets.html](www.forecastbench.org/datasets/question_sets.html)

##### Data collection

Our question and resolution datasets have been pulled, and are updated, from various, public-facing sources. From those sources where the terms of use/service prohibit the redistribution of their information (currently, Manifold Markets and Metaculus), we have obtained explicit permission to do so. Before we add new sources to our growing dataset, we will ensure the ability to distribute questions and resolutions publicly. Data sources in our question bank can be found in Table 13.

**Table 3: Question set data dictionary.**
| Field | Description | Required | Data Type |
| --- | --- | --- | --- |
| forecast_due<br>_date | Date in ISO format. e.g. "2024-07-21" | ✓ | string |
| question_set | The name of the file that contains the question set. e.g. "2024-07-21-llm.json" | ✓ | string |
| questions | A list of questions to forecast on, as defined in Table 4. | ✓ | array<object> |

**Table 4: Data dictionary for question entries in questions array from Table 3.**
| Field | Description | Required | Data Type |
| --- | --- | --- | --- |
| id | A unique identifier string given source. If instead of a string it’s an array of strings, then this is a combination question and combination_of will contain one question per id in the array of strings. | ✓ | string | array<string> |
| source | Where the data comes from. | ✓ | string |
| question | The question to forecast, presented as an f-string with placeholders {forecast_due_date} and {resolution_date} for dataset questions. | ✓ | string |
| resolution_<br>criteria | \BenchmarkName resolution criteria. Specifies how forecasts will be evaluated for each question type. | ✓ | string |
| background | Background information about the forecast question provided by the source, if available. Default: ‘N/A’ |  | string |
| market_info_<br>open_<br>datetime | The datetime when the forecast question went on the market specified by source. Default: ‘N/A’ |  | string |
| market_info<br>_close_<br>datetime | The datetime when the forecast question closes on the market specified by source. Default: ‘N/A’ |  | string |
| market_info_<br>resolution_<br>criteria | The resolution criteria provided by the source, if available. Default: ‘N/A’ |  | string |
| url | The URL where the resolution value is found. | ✓ | string |
| freeze_datetime | The datetime UTC when this question set was generated. | ✓ | string |
| freeze_datetime_<br>value | The latest value of the market or comparison value the day the question was frozen. | ✓ | string |
| freeze_datetime_<br>value<br>_explanation | Explanation of what the value specified in freeze_datetime_value represents. | ✓ | string |
| source_intro | A prompt that presents the source of this question. | ✓ | string |
| combination_of | An array of question objects, as defined by this data dictionary. Default: ‘N/A’ |  | string | array<object> |
| resolution_dates | The resolution dates for which forecasts should be provided for this forecast question. | ✓ | array<string> |

**Table 5: Resolution set data dictionary.**
| Field | Description | Required | Data Type |
| --- | --- | --- | --- |
| forecast_due<br>_date | Date in ISO format. e.g. "2024-07-21" | ✓ | string |
| question_set | The name of the file that contains the question set. e.g. "2024-07-21-llm.json" | ✓ | string |
| resolutions | A list of resolutions to the forecast questions, as defined in Table 6. | ✓ | array<object> |

**Table 6: Data dictionary for resolution entries in questions array from Table 5.**
| Field | Description | Required | Data Type |
| --- | --- | --- | --- |
| id | A unique identifier string given source. A list if this was a combination question. | ✓ | string | array<string> |
| source | Where the data comes from. | ✓ | string |
| direction | If id has an array value, this is an array of the same length. Each entry $\in\{-1,1\}$. If the value is $1$, it means the question was asked in the normal direction. If the value is $-1$, it means the question was negated in the question. e.g. for a question asking for $P(\neg Q1\cap Q2)$, the value would be $[-1,1]$; all possible directions for this question would be: $[1,1]$, $[-1,1]$, $[1,-1]$, $[-1,-1]$. The value is null when id is a string. | ✓ | string |
| forecast_due<br>_date | The date the forecast was due in ISO 8601 format YYYY-MM-DD; day $D$ from Figure 3. | ✓ | string |
| resolution_date | The date the value is associated with in ISO 8601 format YYYY-MM-DD. | ✓ | string |
| resolved_to | The resolution value for the given date. | ✓ | number |
| resolved | If true the question has been resolved. False otherwise. | ✓ | boolean |

#### B.1.1 Examples

Table 7, Table 8, and Table 9 show concrete examples of the data dictionary detailed in Table 3.

**Table 7: Prediction market example.**
| Field | Entry |
| --- | --- |
| id | 1558 |
| source | metaculus |
| combination_of | N/A |
| question | Will the Cavendish account for less than 50% of banana exports worldwide before 2035? |
| background | Bananas are a well-liked import fruit all over the world, and the Cavendish cultivar has been crushing that market for sixty years. But its rise is literally founded upon the compost heap of the Gros Michel, another cultivar. The so-called ’Big Mike’ variety had been the leading export towards Europe and North America, but the Panama disease, a fungus belonging to the *Fusarium* clade, killed that. [Luckily the Cavendish, grown in the same soil as the wilting Gros Michel, replaced it as *the* banana most of the western world connected with bananas](https://en.wikipedia.org/wiki/Gros_Michel_banana). However, it appears [another *Fusarium* rears its spores](https://www.wired.co.uk/article/cavendish-banana-extinction-gene-editing). Cavendish, with their genetic homogenity (they’re all clones) and sterile nature, aren’t resistant to it, and the fungus is ravaging more and more plantations. There are efforts under way to deal with *Fusarium*, but with various societies’ doubts and misgivings about GMOs, the cure may be viewed as a curse instead. |
| market_info_<br>resolution_<br>criteria | This question will resolve as **Yes** if the Cavendish banana accounts for less than 50% of worldwide annual banana exports in any year from 2018 to 2034 (inclusive). |
| market_info_<br>open_datetime | 2018-11-13T08:00:00+00:00 |
| market_info_<br>close_datetime | 2034-12-31T23:00:00+00:00 |
| resolution_<br>criteria | Resolves to the outcome of the question found at [https://www.metaculus.com/questions/1558/cavendish-bananas-collapse-by-2035/](https://www.metaculus.com/questions/1558/cavendish-bananas-collapse-by-2035/). |
| url | [https://www.metaculus.com/questions/1558/cavendish-bananas-collapse-by-2035/](https://www.metaculus.com/questions/1558/cavendish-bananas-collapse-by-2035/) |
| freeze_datetime_<br>value | 0.48 |
| freeze_datetime_<br>value_explanation | The community prediction. |
| freeze_datetime | 2024-07-12T00:00:00+00:00 |
| source_intro | We would like you to predict the outcome of a prediction market. A prediction market, in this context, is the aggregate of predictions submitted by users on the website Manifold. You’re going to predict the probability that the market will resolve as ‘Yes’. |
| resolution_dates | N/A |

**Table 8: Data source (DBnomics) example.**
| Field | Entry |
| --- | --- |
| id | meteofrance_TEMPERATURE_celsius.07005.D |
| source | dbnomics |
| combination_of | N/A |
| question | What is the probability that the daily average temperature at the French weather station at Abbeville will be higher on resolution_date than on forecast_due_date? |
| background | The history of Average temperature by day and by station for France - Degree Celsius - ABBEVILLE - Daily from Météo-France is available at [https://db.nomics.world/meteofrance_TEMPERATURE_celsius.07005.D](https://db.nomics.world/meteofrance_TEMPERATURE_celsius.07005.D). |
| market_info_ |  |
| resolution_ |  |
| criteria | N/A |
| market_info_ |  |
| open_datetime | N/A |
| market_info_ |  |
| close_datetime | N/A |
| url | [https://db.nomics.world/meteofrance_TEMPERATURE_celsius.07005.D](https://db.nomics.world/meteofrance_TEMPERATURE_celsius.07005.D) |
| resolution_<br>criteria | Resolves to the value found at [https://db.nomics.world/meteofrance_TEMPERATURE_celsius.07005.D](https://db.nomics.world/meteofrance_TEMPERATURE_celsius.07005.D) once the data is published. |
| freeze_datetime_ |  |
| value | 17.95 |
| freeze_datetime_<br>value_explanation | The daily average temperature at the French weather station at Abbeville. |
| freeze_datetime | 2024-07-12T00:00:00+00:00 |
| source_intro | DBnomics collects data on topics such as population and living conditions, environment and energy, agriculture, finance, trade and others from publicly available resources, for example national and international statistical institutions, researchers and private companies. You’re going to predict how questions based on this data will resolve. |
| resolution_dates | ["2024-07-28",<br>"2024-08-20",<br>"2024-10-19",<br>"2025-01-17",<br>"2025-07-21",<br>"2027-07-21",<br>"2029-07-20",<br>"2034-07-19"] |

**Table 9: Combination (DBnomics) example.**
| Field | Entry |
| --- | --- |
| id | ["meteofrance_TEMPERATURE_celsius.07117.D", "meteofrance_TEMPERATURE_celsius.07240.D"] |
| source | dbnomics |
| combination_of | [An array containing dictionary entries of both questions.] |
| question | We are presenting you with two probability questions. Please predict the probability that both will happen, that one will happen but not the other, and that neither will happen. In other words, for each resolution date please provide 4 predictions. |
| background | N/A |
| market_info_ |  |
| resolution_ |  |
| criteria | N/A |
| market_info_ |  |
| open_datetime | N/A |
| market_info_ |  |
| close_datetime | N/A |
| url | N/A |
| resolution_ |  |
| criteria | N/A |
| freeze_datetime |  |
| value | N/A |
| freeze_datetime |  |
| value_ |  |
| explanation | N/A |
| freeze_datetime | 2024-07-12T00:00:00+00:00 |
| human_prompt | We are presenting you with two probability questions. Please predict the probability that both will happen, that one will happen but not the other, and that neither will happen. In other words, for each resolution date please provide 4 predictions. |
| resolution_dates | ["2024-07-28",<br>"2024-08-20",<br>"2024-10-19",<br>"2025-01-17",<br>"2025-07-21",<br>"2027-07-21",<br>"2029-07-20",<br>"2034-07-19"] |

### B.2 Forecast sets

Every set of forecasts provided to \BenchmarkName is made public and all forecasts coming from the general public are anonymized before release.

Each forecast set contains the header information outlined in Table 10, with all forecasts in an array called forecasts. The forecast sets are described in the following subsections.

##### Data format

The question and resolution datasets are released as JSON (.json) files.

##### Ethical and responsible use

There are no restrictions on use of the forecast sets.

##### Availability

[www.forecastbench.org/datasets/forecast_sets.html](www.forecastbench.org/datasets/forecast_sets.html)

**Table 10: Data dictionary of headers for forecast set.**
| Field | Description | Required | Data Type |
| --- | --- | --- | --- |
| organization | The organization name as it should be displayed on the leaderboard. | ✓ | string |
| model | The model name as it should be displayed on the leader board. | ✓ | string |
| question_set | The name of the question set file these forecasts are associated with. | ✓ | string |
| forecast_due<br>_date | The date the forecasts were due in ISO 8601 format YYYY-MM-DD. | ✓ | string |
| forecasts | All forecasts for this question set. | ✓ | array<object> |

#### B.2.1 General public forecast set

This forecast set consists of both forecasts made by individuals on the given question set and of the aggregation of those forecasts, as described in Appendix D. The dataset will be updated every time a survey is run with every forecast containing the information outlined in Table 10 and Table 11. Note that the aggregated forecast sets do not contain the user_id or reasoning fields.

##### Data collection

Data is collected from human forecasters via Qualtrics. We collect the reasoning behind forecasts and manually anonymize the data to ensure there is no personally identifiable information in the dataset before posting it online.

**Table 11: Public forecast set data dictionary of entries in forecasts array from Table 10.**
| Field | Description | Required | Data Type |
| --- | --- | --- | --- |
| id | A unique identifier string given source. If instead of a string it’s a list of strings, then this is a combination question. | ✓ | string | array<string> |
| source | Where the data comes from. | ✓ | string |
| forecast | The forecast $\in[0,1]$. | ✓ | number |
| resolution_date | The resolution date this forecast corresponds to. null for market questions. | ✓ | string |null |
| reasoning | The rationale underlying the forecast. |  | string |
| direction | If id has an array value, this is an array of the same length. Each entry is an integer $\in\{-1,1\}$. If the value is $1$, it means the question was asked in the normal direction. If the value is $-1$, it means the question was negated in the question. e.g. for a question asking for $P(\neg Q1\cap Q2)$, the value would be $[-1,1]$. All possible values are: $[1,1]$, $[-1,1]$, $[1,-1]$, $[-1,-1]$, and null. |  | array<number> |null |
| user_id | A randomly generated string associated with the human respondent who submitted the forecast. This value contains no information which could identify said participant and was assigned to the dataset after personal identifiers had been removed. | ✓ | string |

#### B.2.2 Superforecaster forecast set

In addition to the fields outlined in Table 10 and Table 11, the superforecaster dataset contains forecasts with the fields shown in Table 12. Likewise, this dataset will be updated every time a group of superforecasters is surveyed.

##### Data collection

39 superforecasters provided forecasts, rationales, and additional information about they way they forecast in our latest survey round. We will further manually check all of their responses to ensure the anonymity of the dataset.

**Table 12: Superforecaster forecast set data dictionary of entries in forecasts array from Table 10 (additional fields to those in Table 11).**
| Field | Description | Required | Data Type |
| --- | --- | --- | --- |
| searches | An array of search terms used in researching the topic. |  | array<string> |
| consulted_urls | A list of useful urls |  | array<string> |

#### B.2.3 LLM forecast set

This dataset provides the same data (aside from user_id) as outlined in Table 10 and Table 11, only provided by language models. Each individual .json file was created by a model for the given question set.

##### Data collection

Beyond informing teams their forecasts will be made public, we will not check the rationales.

## Appendix C Question Bank

### C.1 Licenses

The licenses outlined in Table 13 apply to the datasets we have sourced for questions. We were granted express permission to use questions from Manifold Markets and Metaculus. Though not required by their license, we also met with a representative from ACLED who approved our use of their dataset for the benchmark and dataset distribution.

**Table 13: Question sources and permissions**
|  | License | License grants permission to use | Express permission granted |
| --- | --- | --- | --- |
| RAND Forecasting Initiative | Public Domain | ✓ |  |
| Manifold | [Terms of Service](https://manifold.markets/terms.html) | ✗ | ✓ |
| Metaculus | [Terms of Use](https://www.metaculus.com/terms-of-use/) | ✗ | ✓ |
| Polymarket | [Terms of Service](https://polymarket.com/tos) | ✓ |  |
| ACLED | [Terms of Use](https://acleddata.com/terms-of-use/) | ✓ | ✓ |
| DBnomics | [Open License](https://www.etalab.gouv.fr/wp-content/uploads/2014/05/Licence_Ouverte.pdf) | ✓ |  |
| FRED | [Terms of Use](https://fred.stlouisfed.org/docs/api/terms_of_use.html) | ✓ |  |
| Wikipedia | [Terms of Use](https://foundation.wikimedia.org/wiki/Policy:Terms_of_Use) | ✓ |  |
| Yahoo! | [Terms of Use](https://legal.yahoo.com/us/en/yahoo/terms/product-atos/apiforydn/index.html) | ✓ |  |

### C.2 Categories

We generate metadata on all of the questions in our question bank, categorizing our questions, as described in Section 3.1.2.

It is important to have forecasting questions across a broad array of categories to test LLM capabilities. We are currently adding more questions from datasets with the goal of equilibrating these categories.

**Table 14: Categories and question counts by source, dropping invalid questions.**
|  | RFI | Manifold | Metaculus | Polymarket | ACLED | DBnomics | FRED | Wikipedia | Yahoo! | Total |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Arts & Recreation | 0 | 57 | 10 | 16 | 0 | 0 | 0 | 0 | 0 | 83 |
| Economics & Business | 1 | 22 | 76 | 35 | 0 | 0 | 166 | 0 | 504 | 804 |
| Environment & Energy | 0 | 3 | 44 | 5 | 0 | 52 | 0 | 0 | 0 | 104 |
| Healthcare & Biology | 0 | 8 | 83 | 1 | 0 | 0 | 0 | 214 | 0 | 306 |
| Politics & Governance | 6 | 51 | 208 | 428 | 0 | 0 | 0 | 0 | 0 | 693 |
| Science & Tech | 5 | 101 | 200 | 21 | 0 | 0 | 0 | 1 | 0 | 328 |
| Security & Defense | 4 | 15 | 157 | 9 | 3,150 | 0 | 0 | 0 | 0 | 3,335 |
| Sports | 0 | 77 | 21 | 4 | 0 | 0 | 0 | 120 | 0 | 222 |
| Other | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| Total | 16 | 334 | 800 | 519 | 3150 | 52 | 166 | 335 | 504 | 5,876 |

## Appendix D Human Survey Instructions and Screenshots

### D.1 Instructions

Participants in the public survey were first prompted to read a consent form detailing the tasks involved in the study and potential risks to participants, alongside estimates of payment for completing each study ($5 + bonus payments for forecasting performance in the introductory survey; $10 for completing the primary survey). Then, participants were prompted to give their responses to 20 forecasting questions randomly selected from the 200-question subset of questions provided to the LLMs.

Participants were given a brief description of the task ahead before each question as follows:

> You are going to be predicting the probability of the answer to the question below being "Yes" (or "resolving positively").

For tasks with questions generated from data providers, forecasters were prompted to provide a probability forecast at multiple resolution dates.

Participants in the superforecaster survey went through a similar initial experience before being moved into the group forecasting stage described in Section 4. Because of the expertise of this set of forecasters, superforecaster participants were also guaranteed a base payment of $1,000 for participating in the individual and group stages of the experiment as well as bonus payments for individual-level accuracy.

### D.2 Screenshots

The following screenshots (Figure 4, Figure 5 show a few of the questions presented to participants in the public study.

Figure: Figure 4: An example market-based question from the human survey.
Refer to caption: extracted/5979248/sec/appendix/example_survey.png

Figure: Figure 5: An example question generated from a data provider, in this case DBnomics, from the public survey. Two of eight time horizons for which we elicited forecasts are included above.
Refer to caption: extracted/5979248/sec/appendix/example_acled.png

## Appendix E Details about LLM "Crowd" Baseline

### E.1 Models

To construct a crowd baseline that includes diverse candidates, we evaluate models using the most recent forecasting dataset containing cross-domain questions with true resolutions from Halawi et al. (). We assess models from the following organizations: OpenAI, Mistral AI, Qwen, Google, Anthropic, and Meta. Using the same scratchpad prompting method from Halawi et al. (), we then select the top three models: GPT-4o, Gemini-1.5.Pro, Claude-3.5-Sonnet. See Table 15 for the results.

**Table 15: Brier Scores from each LLM "crowd" candidate.**
| Model | Scratchpad |
| --- | --- |
| GPT-4o | 0.207 (0.026) |
| Llama-3-70b | 0.232 (0.020) |
| Mistral-Large | 0.233 (0.026) |
| Qwen-1.5-110b | 0.222 (0.025) |
| Gemini-1.5-Pro | 0.214 (0.025) |
| Claude-3.5-Sonnet | 0.178 (0.025) |
| GPT-4-0613 | 0.222 (0.009) |
| GPT-4-1106-Preview | 0.209 (0.012) |
| GPT-3.5-Turbo-1106 | 0.261 (0.010) |
| GPT-3.5-Turbo-Instruct | 0.257 (0.009) |
| Claude-2 | 0.219 (0.014) |
| Claude-2.1 | 0.215 (0.014) |
| Gemini-Pro | 0.230 (0.007) |
| Mistral-7B-Instruct | 0.243 (0.008) |
| Mistral-8x7B-Instruct | 0.238 (0.010) |
| Mixtral-8x7B-DPO | 0.248 (0.010) |
| Yi-34B-Chat | 0.241 (0.009) |
| Llama-2-7B | 0.264 (0.011) |
| Llama-2-13B | 0.268 (0.008) |
| Llama-2-70B | 0.282 (0.011) |

### E.2 Aggregation Methods

With the forecasts generated by the top 3 models selected in Section E.1, we then compare performance of 5 aggregation methods: Median, Trimmed Mean, Geometric Mean, and Geometric Mean of Log Odds (Satopää et al., ). See Table 16 for the results.

**Table 16: Brier Scores from each Aggregation Method.**
| Aggregation Method | Brier Score |
| --- | --- |
| Median | 0.197 (0.024) |
| Trimmed Mean | 0.197 (0.024) |
| Geometric Mean | 0.194 (0.023) |
| Geometric Mean of Log Odds | 0.194 (0.023) |

### E.3 Other Details

LLM Parameters. We set the temperature as 0 and the max output token length as 2000.

## Appendix F Prompts

In this section, we present the following prompts: zero-shot (Figure 6), scratchpad (Figure 7), and three prompts (Figure 8, Figure 9, Figure 10) written by superforecasters from the Forecasting Research Institute used to construct the LLM "crowd" baseline. Note that the scratchpad with information retrieval prompt is simply the scratchpad with an additional line "We have retrieved the following information for this question: {retrieved_info}" before the instructions begin. For combination questions, we slightly modify each of the above prompts by including the second question’s information (Figure 11).

Figure: Figure 6: Zero-shot Prompt from Halawi et al. ()

Figure: Figure 7: Scratchpad Prompt modified from Halawi et al. ()

Figure: Figure 8: Superforecaster Prompt 1

Figure: Figure 9: Superforecaster Prompt 2

Figure: Figure 10: Superforecaster Prompt 3

Figure: Figure 11: Combination Prompt that includes information about both non-market questions. The instructions are truncated and can be supplemented with any of the prompts shown above.

## Appendix G Statistical Details

While our pairwise bootstrapped p-values are precise, our statistical tests do ignore one potentially important source of imprecision. We assume that each question is independent of the other questions. In a world where forecasting accuracy on questions is quite correlated within a topic, or where most events are correlated, this may result in us overstating how confidently we can reject the equivalence of different models. We hope to explore this question in future work, but we are somewhat reassured by the fact that our gathering of forecasting questions from diverse domains and sources makes it unlikely that correlated questions would change our interpretation of these results in any meaningful way.

Our statistical tests are significantly more precise than the 95% confidence intervals for each model would imply because accuracy on each question is quite correlated across models. To understand this phenomenon, consider a hypothetical world where Model A outperforms Model B by a constant ($\epsilon$) on each question. No matter how close the performance of Models A and B are (how small $\epsilon$ is), and no matter how much variance there is in the accuracy of Model A across questions (which drives the 95% confidence interval surrounding Model A’s accuracy), a pairwise bootstrap would show that Model A is more accurate than Model B. This is because forecasts of Model A and Model B are perfectly correlated.

## Appendix H Top 50 ForecastBench Leaderboards

We show the Top 50 leaderboard with human performance evaluated on 200 questions received by humans in Table 17 and the one excluding humans evaluated on the full 1,000 questions in Table 18. Additionally, we show the leaderboard with the inclusion of combination questions in Table 19 , where we generate human forecasts on the combination questions by treating the human forecasts on each component of a combination question as independent (multipling them together to produce the forecast on the relevant combination question). Lastly, we show the leaderboard with combination questions Table 20, where we used the same human-generated forecasts as above but also generated combination forecasts for LLMs from their forecasts on the individual questions, assuming independence of their forecasts on each component, and ignoring the forecast they actually gave on the combination question.

Table: Table 17: Leaderboard with Human Performance: Top 50

Table: Table 18: Leaderboard excluding Human Performance: Top 50

Table: Table 19: Leaderboard with Performance om Combination Questions: Top 50

**Table 20: Leaderboard with Performance on Combination Questions (Independence): Top 50**
|  |  |  |  | Brier Score $\downarrow$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Organization | Information<br>provided | Prompt | Dataset<br>(N=870) | Market<br>(N=301) | Overall<br>(N=1,171) | Confidence<br>Interval | Pairwise<br>p-value<br>comparing<br>to No. 1 | Pct. more<br>accurate<br>than No. 1 |
| Superforecaster median forecast | ForecastBench | – | – | 0.097 | 0.04 | 0.068 | [0.06, 0.077] | – | 0% |
| Public median forecast | ForecastBench | – | – | 0.127 | 0.039 | 0.083 | [0.075, 0.091] | $<$0.001 | 28% |
| GPT-4 | OpenAI | Freeze values | Zero shot | 0.131 | 0.041 | 0.086 | [0.077, 0.095] | $<$0.001 | 34% |
| GPT-4-Turbo-2024-04-09 | OpenAI | Freeze values | Zero shot | 0.135 | 0.04 | 0.088 | [0.079, 0.097] | $<$0.001 | 33% |
| Claude-3-5-Sonnet-20240620 | Anthropic | Freeze values | Zero shot | 0.114 | 0.062 | 0.088 | [0.076, 0.1] | $<$0.001 | 34% |
| Claude-3-5-Sonnet-20240620 | Anthropic | Freeze values | Scratchpad | 0.118 | 0.064 | 0.091 | [0.08, 0.102] | $<$0.001 | 28% |
| GPT-4o | OpenAI | Freeze values | Scratchpad | 0.134 | 0.052 | 0.093 | [0.084, 0.102] | $<$0.001 | 28% |
| Claude-3-5-Sonnet-20240620 | Anthropic | News with freeze values | Scratchpad | 0.13 | 0.06 | 0.095 | [0.085, 0.105] | $<$0.001 | 28% |
| GPT-4o | OpenAI | News with freeze values | Scratchpad | 0.142 | 0.049 | 0.096 | [0.087, 0.104] | $<$0.001 | 27% |
| Mistral-Large-Latest | Mistral AI | Freeze values | Zero shot | 0.134 | 0.062 | 0.098 | [0.088, 0.108] | $<$0.001 | 25% |
| Mistral-Large-Latest | Mistral AI | Freeze values | Scratchpad | 0.13 | 0.068 | 0.099 | [0.091, 0.107] | $<$0.001 | 24% |
| Claude-3-Opus-20240229 | Anthropic | Freeze values | Zero shot | 0.146 | 0.057 | 0.102 | [0.091, 0.112] | $<$0.001 | 30% |
| Claude-3-5-Sonnet-20240620 | Anthropic | No | Scratchpad | 0.118 | 0.086 | 0.102 | [0.09, 0.113] | $<$0.001 | 26% |
| Claude-2.1 | Anthropic | Freeze values | Scratchpad | 0.163 | 0.042 | 0.102 | [0.095, 0.11] | $<$0.001 | 29% |
| Llama-3-70b-Chat-Hf | Meta | Freeze values | Zero shot | 0.139 | 0.067 | 0.103 | [0.093, 0.114] | $<$0.001 | 26% |
| GPT-4o | OpenAI | No | Scratchpad | 0.134 | 0.073 | 0.104 | [0.094, 0.113] | $<$0.001 | 24% |
| Claude-3-5-Sonnet-20240620 | Anthropic | News | No | 0.135 | 0.074 | 0.104 | [0.094, 0.114] | $<$0.001 | 25% |
| Qwen1.5-110B-Chat | Qwen | Freeze values | Zero shot | 0.153 | 0.056 | 0.105 | [0.096, 0.114] | $<$0.001 | 26% |
| Claude-3-Opus-20240229 | Anthropic | Freeze values | Scratchpad | 0.135 | 0.076 | 0.105 | [0.096, 0.115] | $<$0.001 | 24% |
| GPT-4 | OpenAI | Freeze values | Scratchpad | 0.151 | 0.061 | 0.106 | [0.097, 0.114] | $<$0.001 | 24% |
| Claude-3-5-Sonnet-20240620 | Anthropic | News | Scratchpad | 0.13 | 0.083 | 0.107 | [0.096, 0.118] | $<$0.001 | 24% |
| Claude-2.1 | Anthropic | No | Scratchpad | 0.163 | 0.052 | 0.107 | [0.099, 0.116] | $<$0.001 | 27% |
| Claude-3-5-Sonnet-20240620 | Anthropic | No | Zero shot | 0.114 | 0.101 | 0.107 | [0.094, 0.12] | $<$0.001 | 26% |
| Llama-3-70b-Chat-Hf | Meta | Freeze values | Scratchpad | 0.151 | 0.064 | 0.108 | [0.099, 0.116] | $<$0.001 | 23% |
| Mixtral-8x22B-Instruct-V0.1 | Mistral AI | Freeze values | Zero shot | 0.154 | 0.066 | 0.11 | [0.099, 0.121] | $<$0.001 | 29% |
| Gemini-1.5-Flash | Google | Freeze values | Scratchpad | 0.151 | 0.069 | 0.11 | [0.1, 0.12] | $<$0.001 | 25% |
| Claude-3-5-Sonnet-20240620 | Anthropic | News | No | 0.136 | 0.085 | 0.11 | [0.099, 0.121] | $<$0.001 | 24% |
| GPT-4o | OpenAI | Freeze values | Zero shot | 0.166 | 0.055 | 0.11 | [0.1, 0.121] | $<$0.001 | 30% |
| GPT-4o | OpenAI | News | Scratchpad | 0.142 | 0.08 | 0.111 | [0.1, 0.122] | $<$0.001 | 23% |
| Gemini-1.5-Pro | Google | Freeze values | Zero shot | 0.154 | 0.068 | 0.111 | [0.099, 0.123] | $<$0.001 | 28% |
| Gemini-1.5-Pro | Google | Freeze values | Scratchpad | 0.139 | 0.084 | 0.111 | [0.102, 0.121] | $<$0.001 | 23% |
| Claude-3-Opus-20240229 | Anthropic | No | Scratchpad | 0.135 | 0.088 | 0.111 | [0.102, 0.121] | $<$0.001 | 23% |
| GPT-4-Turbo-2024-04-09 | OpenAI | Freeze values | Scratchpad | 0.152 | 0.072 | 0.112 | [0.101, 0.122] | $<$0.001 | 26% |
| GPT-4-Turbo-2024-04-09 | OpenAI | No | Zero shot | 0.135 | 0.089 | 0.112 | [0.102, 0.123] | $<$0.001 | 22% |
| GPT-4 | OpenAI | No | Zero shot | 0.131 | 0.095 | 0.113 | [0.103, 0.122] | $<$0.001 | 24% |
| Gemini-1.5-Pro | Google | News with freeze values | Scratchpad | 0.143 | 0.084 | 0.113 | [0.103, 0.123] | $<$0.001 | 22% |
| Mistral-Large-Latest | Mistral AI | No | Scratchpad | 0.13 | 0.097 | 0.113 | [0.104, 0.123] | $<$0.001 | 23% |
| Mixtral-8x22B-Instruct-V0.1 | Mistral AI | Freeze values | Scratchpad | 0.15 | 0.077 | 0.114 | [0.105, 0.123] | $<$0.001 | 23% |
| Gemini-1.5-Flash | Google | Freeze values | Zero shot | 0.15 | 0.078 | 0.114 | [0.101, 0.127] | $<$0.001 | 27% |
| Gemini-1.5-Pro | Google | No | Scratchpad | 0.139 | 0.09 | 0.115 | [0.105, 0.124] | $<$0.001 | 22% |
| GPT-4 | OpenAI | No | Scratchpad | 0.151 | 0.079 | 0.115 | [0.106, 0.123] | $<$0.001 | 21% |
| Imputed Forecaster | ForecastBench | – | – | 0.203 | 0.029 | 0.116 | [0.107, 0.124] | $<$0.001 | 32% |
| Gemini-1.5-Pro | Google | News | Scratchpad | 0.143 | 0.091 | 0.117 | [0.107, 0.127] | $<$0.001 | 23% |
| GPT-4-Turbo-2024-04-09 | OpenAI | No | Scratchpad | 0.152 | 0.082 | 0.117 | [0.108, 0.127] | $<$0.001 | 21% |
| Claude-2.1 | Anthropic | Freeze values | Zero shot | 0.179 | 0.055 | 0.117 | [0.107, 0.127] | $<$0.001 | 29% |
| GPT-4-Turbo-2024-04-09 | OpenAI | News with freeze values | Scratchpad | 0.161 | 0.075 | 0.118 | [0.108, 0.129] | $<$0.001 | 26% |
| Claude-3-Opus-20240229 | Anthropic | No | Zero shot | 0.146 | 0.093 | 0.119 | [0.108, 0.131] | $<$0.001 | 24% |
| Claude-3-Opus-20240229 | Anthropic | News | No | 0.134 | 0.106 | 0.12 | [0.11, 0.131] | $<$0.001 | 23% |
| Llama-3-70b-Chat-Hf | Meta | No | Zero shot | 0.139 | 0.102 | 0.12 | [0.11, 0.131] | $<$0.001 | 23% |
| Gemini-1.5-Flash | Google | No | Scratchpad | 0.151 | 0.09 | 0.12 | [0.11, 0.13] | $<$0.001 | 22% |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The level of technical details and insight are the biggest things that would make me change my score.

Detailed comments by section:

abs/intro/relwork
 - "necessary task" seems a bit overstated; if you want to make this claim it would be worth stating how exactly the referenced papers support that claim
- reasonably clear intro/positioning and motivation overall; a bit repetitive but lacking technical details (e.g. how are predictions elicited from humans? what kinds of mistakes do LLMs make?); it would be nice to mention that briefly 
- relwork: what is "statistically imprecise"?  - good lmeval section
preliminaries
 - unclear details about the nature of forecasts in the dataset; are they open-ended questions / natural language? are the possible outcomes enumerated in advance? is there a char limit? etc.  - unclear explanation of the Brier score; the Forecasting paragraph says forecasters "may" assign probabilities to possible outcomes; if they don't how do you assign f? How do you assign a 0/1 outcome to questions like "who will be the next US president?" - using the character f is confusing notation (f is almost universally a function) but not a huge deal - "strictly proper" and why we would care about that / why that incentivises truthfulness is not explained at all.  - 3 different methods of prompting/rag are not explained at all - "prompting" for the humans that are compared to the LLMS is not explained at all

sec 3 
 - does not seem necessary to have a flowchart here to say that everything goes to the question bank, and it takes a lot of space. It might be helpful to have one if it showed what are the different platforms and datasets, any pre-preprocessing that needs to be done, etc., but this is covered fine in Table 1. 
 - Similarly for Fig 3; neither of these is contributing substantively to the paper; they could be moved to appendix to make space for more technical details.
 - it seems like the details in 3.2 contradict the "preliminaries" section
 
sec 5
- given the small dataset sizes here it would be interesting to explore exactly what LLMs do and don't get right
soundness: 2
strengths: Interesting and useful benchmark, whose position in the landscape is clearly described. 
LLM evals are extensive.
summary: A dynamic dataset of forecasting qs about the future is aggregated and proposed as a benchmark for LLMs. Performance of a variety of LLMs is compared with human.
weaknesses: This is an interesting and useful dataset, but reads like a tech report or class assignment; the level of technical detail is superficial, and there is little to no exploration of what we learn about LLMs from this.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - Is there anything easy (cheap) we can try to improve the capabilities of these models to forecasting beyond RAG-like things? I guess finetuning open-source models would be ideal but that is probably infeasible.
soundness: 3
strengths: - very well written--well motivated, structured, and polished overall
- addresses a key problem in prior benchmarks: benchmark staleness, which is obviously very important for the problem of forecasting, since the goal is to build/evaluate systems that can predict new events, not post-hoc. their approach is conceptually cleaner (imo) than relying on LLMs' knowledge-cutoffs, etc.
- overall interesting finding that at least in this setting, LLMs still quite lag behind experts, only matching the median of average people 
- the approach to building/generating the question set and the entire pipeline is very thoroughly thought out.
summary: The paper presents a sophisticated and automated benchmark for evaluating forecasts that improves over prior benchmarks in that it will not become stale and does not rely on assumptions about models (e.g., knowledge cutoffs). Using the benchmark, they find that interestingly, SOTA models are still short of expert forecasters.
weaknesses: - Table 2 is a bit big / dense and hard to parse. The paper could benefit from using more visual plots to convey the results

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: - Have you investigated the impact of the prompt date on model predictions? Specifically, could questions asked further in advance of the resolution date introduce greater uncertainty in LLM predictions, potentially affecting prediction accuracy?
- How is the quality of questions released for evaluation determined?
- Have you considered evaluating model predictions by examining the logits of the answer tokens? Although this may be more feasible with open-source models, some providers do offer token-level logit outputs. Incorporating logits could provide insights into model calibration and further enhance prediction reliability.
soundness: 3
strengths: - A sophisticated evaluation framework that incorporates multiple forecasting dimensions, including:
    - Selection of questions from high-liquidity market sources,
    - Temporal sampling methodology for question release,
    - Granular analysis of model and human performance across varying time horizons.
    - Comprehensive assessment of instruction-following chat models through six distinct evaluation methodologies (Section 5.1), including zero-shot prompting, scratchpad reasoning prompting, and news-augmented generation.
- Implementation of diverse temporal resolutions for forecasting questions, spanning from week-long to decade-scale predictions, enabling robust evaluation across different time scales.
- Novel incorporation of combination questions for assessing model performance on higher-complexity tasks, revealing significant human performance advantages and highlighting current limitations in model capabilities.
- The development of a daily question-release mechanism featuring 1000 forecasting queries, which:
    - Mitigates potential gaming of the leaderboard system,
    - Generates an expanding dataset of resolved predictions enabling model training and calibration of future forecasting systems.
summary: The paper introduces ForecastBench, a novel dynamic benchmark designed to evaluate the forecasting capabilities of AI systems, particularly large language models. In contrast to prior static benchmarks, ForecastBench continuously updates with new forecasting questions about future events that have no known answers at the time of submission, thereby eliminating concerns of data leakage or contamination. The study compiles a standardized set of 1000 forecasting questions sourced daily from nine data providers, including prediction markets, forecasting platforms, and real-world time series. To evaluate the accuracy, the authors assess the performance of 17 state-of-the-art LLMs on a subset of these questions, comparing their forecasts to those of expert human forecasters (superforecasters) and the general public, using the Brier score as the primary metric.
weaknesses: - The evaluation of unresolved questions relies on comparisons with crowd predictions, potentially complicating future assessments of whether models outperform the crowd baseline.
- The number of expert predictions was limited, with only 39 experts contributing, leading to an average of 8 expert predictions per question. This relatively small sample size may affect the robustness and generalizability of the evaluation results.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. What if we allow LLMs to access different tools to actively acquire relevant information in a feedback loop? In this way the LLM may construct more context for more accurate answer.
soundness: 3
strengths: 1. The forecasting problem studied in this paper is both very interesting and practical. Forecasts on macroeconomic problems such as presidential election is both complicated and impactful to real-world scenarios. Hence it can serve as a good benchmark for evaluating LLM's capability.
 
2. The benchmark design considers data leakage, which improves the reliability of the benchmark.

3. The benchmark results indicates some areas where LLMs still have inferior performance compared to top human, revealing potential future research directions.

4. Code is provided for better reproducibility.
summary: This paper proposes a forecasting benchmark for evaluating LLM's forecasting capability. The benchmark is carefully constructed to avoid potential data leakage. The benchmark results indicates with statistical significance that even the SOTA LLMs nowadays cannot beat human experts in this task.
weaknesses: 1. It may be worth justifying the adopted method in this benchmark for LLM to make forecasts, since LLM reasoning methods (e.g. RAG, etc.) can have great influence on the final performance

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 1
questions: Answers to the following questions would help me better calibrate my review:
1. Is there intuition for why the LLMs would perform better on average for the questions sampled for humans? It seems like there's a consistent drop in the Brier Score.
2. How do LLM and human questions differ? Could these differences explain any differences in accuracy? Maybe this relates to L265---what's different about their information-availability?
3. What is "News with freeze values"?
soundness: 3
strengths: This paper has many strengths:
* Dynamic benchmarks are needed and event forecasting seems like a sensible LLM use to evaluate, likely interesting members of the research community
* This paper seems to describe a mature and already-used system that could elicit some interesting use
* The finding that LLMs can forecast events similarly to average forecasters is interesting, and the fact that experts are still better is a promising sign that this area deserves further attention
* The use of real human experiments is a big strength
summary: This work presents a benchmark for evaluating how well LLMs predict future events, curating questions about current events and corresponding answers. A set of state-of-the-art LLMs are then prompted to predict the probability of future events. The prediction accuracy is then compared against expert and non-expert human forecasters, ultimately finding that the LLMs generally perform similarly to average forecasters but worse than expert forecasters.
weaknesses: This paper could be strengthened by addressing the following weaknesses:
1. The main weakness of this work is the limited experiment section. Given the main contribution is the curation of a new dataset and resultant evaluation, it is unsatisfying that the result is only one page of experiments with minimal interpretation. The result is it's unclear if this benchmark is useful beyond this experiment. Relatedly, it's never really shown why it's important for this to be a "dynamic" benchmark, despite this being a key selling point. Avoiding data leakage is indeed important, but it's surprising that this doesn't allow for any dedicated experiment or discussion around how often questions are resolved during the benchmarking.
2. The method development is described without many details, especially explanations of design decisions and their impact. Instead, most of Section 3 simply states what was done. It's not clear that future researchers could improve on this work if there's no explanation of what alternatives exist and why they are insufficient. This isn't inherently a major concern, but the main contribution of the work appears to be the dataset development, so defending these decisions would help the reader gain confidence in the resulting data and findings.
3. The paper is hard to follow, with many details sprinkled throughout various sections and the reader is left to hunt them down. It could help to include more details on the forecasting problem and summaries of exactly what's provided in Section 3.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: Questions in Weakness Section.
soundness: 2
strengths: Interesting paper and relevant problem statement. Human annotated data generation in a streamlined manner is a challenge and the methodology proposed in the work can be used for updating the evaluation set for LLMs with time.
The work also discusses the evaluation on complex questions comprising of simple questions.

Paper is well-written.
summary: The paper introduces ForecastBench, a dynamic benchmark for evaluating the forecasting capabilities of both human and machine learning systems. This benchmark continuously updates with new forecasting questions, addressing the limitations of static benchmarks. The paper includes comprehensive experiments comparing the performance of various LLMs and human forecasters.
weaknesses: Overall concerns-
Forecasting based on questions is a highly subjective problem. The questions and their respective responses could vary depending on the demographics, education background, understanding of the domain, etc of the forecasters. Does the authors have any openly accessible user-study report highlighting the biases and how they are being addressed. This follows for both - human forecasters as well as "superforecasters"

I'm wondering how does the 1000 questions designed for evaluation covers forecasting of an open world set? Or is it only targeted to a few domains?

The question examples shown in Figure 4&5 seem at a very high-resolution. Although, it is appreciated that the source of the questions is provided, but "is 2024 going to be hotter than 2023" - What parameter is being considered here for comparison? is it mean temperature? Can LLMs infer these for a new undefined concept?

Methodology relies on online links and APIs, thus limited in its coverage of questions.

Reproducibility is missing; validation is difficult.

User -study report should be provided publicly. What's the demographics of the human forecasters?

Reliability and risks associated with the methodology - Is stress test performed on the methodology for use in real-world? What are the metrics?

Meta-review: This well-written and well-motivated paper proposes a new benchmark for LLM-based forecasting that aims to mitigate the staleness of existing benchmarks. It has been evaluated by 6 knowledgeable reviewers. Most of them recommended its acceptance (including 2 straight acceptances and 1 marginal acceptance), while other 2 voted for marginal rejection. The authors have provided an extensive rebuttal and substantially improved the paper in response to the reviews. Based on the maturity of the presented work and its scope, I recommend accepting this work for publication at ICLR.

Decision: Accept (Poster)