Consistency Checks for Language Model Forecasters
Authors: Daniel Paleka
Abhimanyu Pallavi Sudhir
Alejandro Alvarez
Vineeth Bhat
Adam Shen
Evan Wang
Florian Tramèr
Venue: ICLR 2025 Oral (2025)
Keywords: forecasting
markets
trading
LLM
evaluation
eval
consistency
robustness
Primary area: datasets and benchmarks
TL;DR: It is difficult to evaluate AI forecasters instantaneously; we propose market-based consistency evals on LLM forecasters and show plenty of inconsistency.

Abstract: Forecasting is a task that is difficult to evaluate: the ground truth can only be known in the future. Recent work showing LLM forecasters rapidly approaching human-level performance begs the question: how can we benchmark and evaluate these forecasters *instantaneously*? Following the consistency check framework, we measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions. We propose a new, general consistency metric based on *arbitrage*: for example, if a forecasting AI illogically predicts that both the Democratic and Republican parties have 60\% probability of winning the 2024 US presidential election, an arbitrageur could trade against the forecaster's predictions and make a profit. We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits the predictions of the forecaster, and measures the consistency of the predictions. We then build a standard, proper-scoring-rule forecasting benchmark, and show that our (instantaneous) consistency metrics correlate strongly with LLM forecasters' ground truth Brier scores (which are only known in the future). We also release a consistency benchmark that resolves in 2028, providing a long-term evaluation tool for forecasting.

Published as a conference paper at ICLR 2025
CONSISTENCY CHECKS FOR LANGUAGE MODEL FORECASTERS
Daniel Paleka ETH Zurich
Abhimanyu Pallavi Sudhir' University of Warwick
Alejandro Alvarez Independent
Vineeth Bhat HITT Hyderabad
Adam Shen Columbia University
Evan Wang Cornell University
Florian Tramèr ETH Zurich
ABSTRACT
Forecasting is task that is difficult to evaluate: the ground truth can only be known in the future. Recent work showing LLM forecasters rapidly approaching human-level performance begs the question: how can we benchmark and evaluate these forecasters instantaneously? Following the consistency check framework, we measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions. We propose a new, general consistency metric based arbitrage: for example, if a forecasting AI illogically predicts that both the Democratic and Republican parties have 60% probability of winning the 2024 US presidential election, an arbitrageur forecaster's predictions and make a profit. We build an automated evaluation sys- tem that generates a of base questions, instantiates consistency checks from these questions, clicits the predictions of the forecaster, and measures the consis- tency of the predictions. We then build a standard, proper-ccoring-ruk forecasting benchmark, and show that our (instantaneous) consistency metrics correlate with LLM forecasters' ground truth Brier scores (which are only known in the future). We also release consistency benchmark that resolves in 2028, providing long- term evaluation tool for forecasting.
trade against the
INTRODUCTION
Prediction markets markets that pay contingent event. For market such "$1 if Jeb Bush is elected President 2028", the price reflects the "market estimate" for the probability that event. Prediction markets are promising tool for aggregating information from disparate sources to arrive at the most correct possible belief after taking into account all relevant information (Arrow al., 2008; Hanson, 2002).
Until 2024, LLM forecasters generally performed poorly relative to human forecasters (Zou et al., 2022b; Schoenegger and Park, 2023). However, recent works (Halawi et al., 2024; Schoenegger et al., 2024; Phan et al., 2024) suggest that LLM--bseed forecasters can rival human forecasts on forecasting websites such as Metaculus, PredictIt, and Manifold Markets.
A key question emerges: once LLM forecasters are better than human ones, how can we efficiently evaluate their predictions? In particular, long-term forecasting questions are very important for decision-making (Tetlock et al., 2024; Muehlhauser, Luke, 2019). and finding ground truth for eval- uation such contexts is infeasible by virtue of the questions resolving far in the future.
One approach, proposed by Fluri et al. (2023), is that even when we cannot evaluate the correct- ness LLM decisions, we can evaluate their logical consistency. For example, if an LLM fore- caster gives probabilities 0.5 and 0.7 to "Will Trump be elected US president?" and "Will someone other than Trump be elected US president?", this is necessarily inconsistent. Fluri et al. (2023) demonstrated that GPT-4 and OPT-3.5-turbo. when asked one-sentence forecasting questions, were inconsistent on simple logical checks such as negation.
"quul contribution. Corresponding: daniel. paleka@ind aethae..... sh. Author contributions in Con- tributions.

---

Published as a conference paper at ICLR 2025
Our contributions in this work are as follows:
1) Principled metries for consistency. In Section 2, we introduce a theoretical framework for mea- suring consistency violations of binary forecasts, based on market arbitrage, and a frequentist metric, based on hypothesis testing. We apply these metrics to 10 different logical consistency rules (see Table 3): NEGATION, PARAPHRASE, CONSEQUENCE, ANDOR, AND, OR, BUT, COND, CONDCOND and EXPEVIDENCE.
metrics: arbitrage metric, based
2) A consistency evaluation pipeline for binary forecasters. In Section 3, we introduce a sistency evaluation pipeline for LLM forecasters. We create two forecasting datasets with known ground truth resolutions: one scraped from prediction markets, and one synthetically generated from news articles. Both datasets include only events that happen past the training data cutoff of all fore- casters we test, and resolve before September 2024, We then generate tuples of forecasting questions satisfying logical consistency rules with associated consistency metrics.
3) Consistency correlates with ground truth forecasting performance. Our consistency metrics are novel performance metries for forecasters that can be computed right away, no matter the time horizon. Of course, forecasters could also be evaluated using bucktesting, asking past questions with known ground truth resolutions. Yet, backtesting L.M forecasters can be challenging if we do not have clear information about the models' training data contents. Moreover, there may be new types of questions that want to evaluate forecasters for which do not have appropriate past results (e.g.. questions related to pandemics before 2020). is thus natural to ask: can consistency metrics tell us anything about future forecasting performance?
In Section 4, we show that for all forecasters we test, our consistency metrics correlate positively with forecasting performance (as measured by the Brier score) on both our benchmark datasets. The correlation varies across consistency checks, with some logical checks (e.g., consistency of conditional probabilities) having over R 0.9 correlation with forecasting performance, while other logical tests provide little signal. We hypothesise that this analysis can extend to smarter forecasters and longer time horizons, to provide instantancous feedback on forecaster performance.
4) Scaling inference-time compute can improve consistency for some logical checks, but fails to generalize. Since we find that consistency correlates with forecasting performance, natural to ask whether we can improve forecasters by making them more consistent. Unfortunately, we find that natural ways of improving consistency tend overfit to specific consistency checks and do not generalize.
Specifically, we design Arbi arbittran orecasser: a forecaster that "patches" some base fore- caster's output by generating logically related questions and "arbitraging" the base forecaster's fore- casts for these related questions against each other. In Section 5 and Appendix G, we show that ArbitrageF oorrcaaateet improves consistency on checks that we optimize against, but this im- provement does not generalize to other held-out consistency checks, nor does it improve the actual forecasting performance.
5) A long-horizon forecasting consistency benchmark. We create a long-horizon benchmark of 3,000 consistency checks for forecasts resolving in 2028. Our benchmark spans questions various topics for which we will have no ground truth for more than three years, and thus serves as a nice testing ground for advanced LLM forecasters.
We release the full code and the datasets 2 used in the paper.
2 A THEORETICAL FRAMEWORK FOR FORECASTING CONSISTENCY
Notation. Let Prop denote the set of forecasting questions we are interested in, θ denote the set of possible outcooessresolutionss for an individual questions. this paper, we focus on Prop as a set of binary forecasting questions, so θ (T,1). A Forecaster is then map F : Prop + [0. 1). One special forecaster is the ground truth resolutions 0 Prop + O, returning and probability for {T, L}. respectively.
htttte///tttttto..mmm//////////s fore--oorrassse foreaacn httpss//oogginfiss.oo/ateeeeeet leka leka/ /ccf//
2

---

Published as a conference paper at ICLR 2025
For conditional questions that resolve None, also have optional resolutions @ := e U [None} = (T, None). We focus on binary questions following Halawi et al. (2024). Our methods in principle be extended to study consistency between general probability distributions in forecasting, such as the ones discussed in Gooen (2024).
2.1 CONSISTENCY CHECKS AND INCONSISTENCY METRICS
In line with Fluri et al. (2023), a consistency check is conceptualized as a pair of n-ary relations: R Prop" {T, } in question space, S [0, 1 {T,1) in forecast space, and a predicate for F' such that (( satisfied by all feasible 0, and also any "correct" forecasts generated by world model that accurately accounts for aleatoric uncertainty. Violation of consistency is measured by some violation metric S(F(z1).... F(a,)) For example, intuitively. the "negation" check NEGATION is given by the relation R2) == 21 = -22 on questions, and the relation ((F(x)) F(x2)) == (F())) ) + F(2r2) on forecasts. The full table of the consistency checks we use is given in Appendix B.
En) In)) = ((I(()))..... ((((1))...FF.)))) F(r,)). In particular, this assertion must be
V [0, 1]" + I which must satisfy (((1))....F((..)) = <
Improving upon Fluri et al. (2023), we derive V from in a principled way, handling all types of logical consistency checks simultaneously. We introduce two new inconsistency metrics: the arbi- trage metric and the frequentist metric for measuring logical inconsistency probabilistic forecasts.
2.1.1 ARBITRAGE METRIC
The arbitrage metric is conceptualized as the minimum profit that an arbitrageur can be guaran- teed making bets against the forecaster's predictions. More precisely: suppose that the forecaster's probabilities (..),, ((x,)) were prices offered by logarithmic market maker with market sub- sidy parameter $1. If these probabilities are inconsistent, then there are prices Pi....P. Pr. that an arbitrageur could bring to the market such that is guaranteed to make a profit against the market- maker, no matter the outcome of each question. We define (IF(x),,.. (F(nn) as the maximum achievable "minimum profit" that the arbitrageur can guarantee by choosing appropriate Pı... Pn- We further denote by A(F(z)).... Fx.)) the of prices PI.... Pr. that maximize the minimum profit:
(arg max, max) min (logp; lo;)( (0())T + (log (1 - p.)) log (1 F(xx))) 600=1
De 10
WEB
(1)
R(w) is the set of all possible consistent resolutions this tuple. A more general version of is given in Appendix D, along with specific worked-out examples of the arbitrage metric for each consistency check, and details on how we compute it; as an example, the arbitrage metric for the Negation Check can be derived exactly (Appendix D.2):
Here Q = (w E 0m
((F()), 2og (((=)(1-((<x)) + V(1 - ((2)(()))F(-))
To illustrate: V(0.5, 0.6) R 0.01, V(0.5, 0.51) x 10-4 The metric more sensitive to violations for probabilities very close or due to the logarithmic market maker. In our evals, for all types of checks, we say that sampled check does not pass if V > 0.01. We have to pick some hyperparameter as an inconsistency threshold; we set correspond to giving 110% probability in total to the events of Republican and Democratic parties winning the US presidential election.
2.1.2 FREQUENTIST METRIC
We also compute a different, frequentis consistency metric. Consider Monte Carlo forecaster that samples a world model n times, and for any event, returns the fraction of samples in which the event occurs. The frequentist metric is the number of standard deviations given tuple forecast is off from
A logarithmic market maker with subsidy w is a market maker who adjusts prices in response to trades such that the trader's reward for moving the probability of true- resolvinn sentence from to p' is w log p' log po. For further hackground on scoring rules and the associated market makers, see Appendix D, Berg and Proebsting (2009), Hanson (2002).
3

---

Published as a conference paper at ICLR 2025
the mean Monte Carlo forecast, scaled be independent of n. We say that consistency violation happened if the number of standard deviations away from the mean of the null is at least as in the (0.5,0.) 0.6) case described in Section 2.1.1. The full description is given in Appendix E.
2.1.3 INTUITION ON CONSISTENCY METRICS
Our metrics address two major obstacles with measuring inconsistency: tolerance to noise and principled aggregation of inconsistency scores.
Tolerance to noise. In the standard Bayesian setting. beliefs are either consistent or not: there either is a Dutch book (a way to bet against the forecaster's beliefs to get infinite profit) or the probabilities are perfectly consistent. In practice, forecasters' beliefs (even on the same question) are never perfectly consistent across runs. If an election model has presidential candidate at 48% with one random seed and 50% on the other, this is not a reason to discard it as completely flawed. Hence, metrics increase smoothly with inconsistency.
instead of being binary measure of consistency,
Principled aggregation and comparison of inconsistency scores Fluri et (2023) developed set of inconsistency checks, used an ad hoc metric for each check they used, and normalized the scores to [0, 1]. There are two important issues with their approach:
The metrics in their work are mostly linear and would treat the inconsistencies of (0.5, 0.6) and (0.89, 0.01) ) on the NEGATION check as equally bad, which is counterintuitive in many applications.
2. It is unclear how to compare and aggregate scores from different consistency checks.
Our approach ensures that all consistency scores share a common "unit". For example, in the ar- bitrage metric, to aggregate inconsistencies, sum up the profit made by arbitrageur across questions.
3 PIPELINE OVERVIEW
We illustrate the steps in our data collection pipeline below, and provide more details on each indi- vidual steps:
syothetic
tuple
(P.Q) E. (p.g) V v(p.g)
Online platforms, news, topics
+scraping
instantiation
(... - P) We first prepare datasets of base questions in multiple ways:
(a) Scraping questions from online platforms such as Manifold and Metaculus;
(b) A ground-truth resolved dataset synthetically generated from news articles;
(c) Synthetic generation on questions on list of topics such as Politics, Science, Eco- nomics, etc.
For the first two of the above, we also include the ground truth resolution for each question. We discuss all these in more detail in Section 3.1.
(P - ()) The base questions are synthetically instantiated into tuples that must satisfy certain consistency checks. For example, every single base question P is instantiated into tuple (P-PP): and pairs of mutually relevant base questions P, Q are instantiated into tuples like PQ A,AQ, Q).
((,Q) E (p) The forecaster is separately queried to elicit forecasts on each question, resulting in forecast tuples that should, if the forecaster is consistent, satisfy consistency properties. For example, for a size-two tuple where Q = -P, it should satisfy p + =
* ((p,)) q) 24 V(p.9)) We score each tuple of forecasts for consistency with both of our viola- tion metrics.
Examples of data at each step of the pipeline are given in Appendix C. The prompts and LLM calls used in each step before forecasting given in Appendix H.
4

---

Published as a conference paper at ICLR 2025
3.1 GENERATING AND SCRAPING FORECASTING QUESTIONS
Forecasting question format. Each forecasting question includes a title that states the main ques- tion, a body that provides detailed resolution criteria, and a resolution date, along with optional fields such metadata creation date.
Real prediction market questions. We scrape questions from two forecasting platforms, Metacu- lus and Manifold Markets, and only use questions that both resolved and were imitially to resolve between May 1, 2024, and August 15, 2024. This leaves us with over 500 questions, of which 242 pass our verification step (see end of this subsection). An example a processed question, including its relevant details, is provided in Appendix C.I.
Generating forecasting questions from NewsAPI articles. To generate forecasting questions with known resolutions, concrete events rather than opinion pieces. To mitigate biases towards positive resolutions (as most questions derived from article would typically resolve to True), employ reference class span- ning using an LLM to modify key entities in the questions while keeping the overall thematic structure intact. Each question's 's ground-truth resolution is verified using the Perplexity API with internet access, yielding ground truth resolution labels with less than a 5% error rate in our testing. We compile total of 2,621 ground-truthh resolved forecasting questions resolving between July 1, 2024, and August 21, 2024. Of these, we use a subset of 1,000 to test the relationship between consistency violation and accuracy. Further details regarding the pipeline can be found in Appendix K.
articles sourced from NewsAPI. We focus articles describing
Synthetic question generation. We generate questions by few-shot prompting, we sample six examples of forecasting questions, style examples, as well NBA...) to diversify the generated questions. We generate question titles, deduplicate them using text text-embeddingg -d--eed--nggg----- 11 embeddings from OpenAI, and then for each title we use gpt-4o to create the question body and resolution date. With this method we create 1,000 forecasting ques- tions that resolve either by in 2028. More details are in Appendix H.
set of tags (Brazil,
Verification and improvement from human feedback. In all of the above steps, we filter gen- erated questions using 9o to check for properties such as the coherence between the body and title, the clarity and precision of the resolution criteria, and whether the question is about actual world events. Questions failing this step are discarded. To develop this step, we used a feedback form for human reviewers (authors of this paper) to suggest modifications to generated questions. These suggestions inform refinements to prompts and few-shot examples in our pipeline. An exam- ple of the feedback form is provided in Appendix I.
3.2 INSTANTIATING TUPLES OF QUESTIONS FOR CONSISTENCY CHECKS
The base forecasting questions are subsequently used to synthetically generate tuples of logically related questions. For example, a pair of base questions (P, Q) can be used to generate a 4-tuple (P, Q, PVQ) for ANDOR, tails). The main question content (titles and bodies) were generated synthetically (using gpt- 4o), while the resolution dates and other properties were calculated systematically (e.g. the max of the resolution dates of the base questions).
3-tuple (P. AP, PVQQ Q) for BUT (see Appendix B for de-
We then conduct two measures to ensure the instantiated tuples are correct and sensible: relevance scoring, and verification that the tuples of questions indeed describe logically related events.
Relevance scoring. When combining base questions into tuples, we have to take care to avoid off-distribution questions like "Is SpaceX going to be worth $200B by 2030, given that Sri Lanka's rice production grows 40% by 2040?". For tuples instantiated from more than one base question, sort 2000 potential base question combinations by their "relevance score", obtained by querying an LLM and asking it to score how relevant the questions are to one another, and choose the top 200 for each consistency check. See Figure 15 for details.
5

---

Published as a conference paper at ICLR 2025
Verification. The instantiated tuples of questions then passed to another LLM call to reject they do not fit their intended structure; for example, we detect if the resolution criteria of the second question are not truly negation of the resolution criteria of the first question. Examples of verification prompts are given in Appendix H.
3.3 ELICITING FORECASTS
We test range of forecasters based on various LLM models (gpt-4o, gpt-4o-mint. claude-3. ..---3.55500 5-sonnet, 11 allama-3.. 1-8 1lama-3. 1--33.1-773 70B, 11ama-3 1.taa...--45500 1-40BB, ol-mini and ol-previ ew) with and without chain-of-thought prompting: see Appendix F for details. We run each of these forecasters on 5000 tuples in total (for each of the 10 checks, we use 200 tuples from scraped questions and 300 from NewsAPI questions), except for ol ol-preview, which we test on 50 tuples per check only due to cost constraints. We could not test forecasters from Halawi et al. (2024) due API deprecations; Section 6.
RESULTS
We evaluate range of forecasters on the datasets described above. for both consistency and ground truth Brier score. We note that the Brier score as the standard metric of forecasting accuracy depends both on model capabilities and the training data cutoff: it should not be surprising for a stronger model to have worse Brier score its training data cutoff is earlier than for weaker model. The full list of forecasters is in Appendix F.
For all data analysis in this section, we exclude forecasters that have Brier score worse than random guessing (0.25), such as the basic setup with llama 3. 1-8., it would unfairly advantage our case of "correlating consistency with accuracy"..
Average consistency scores correlate strongly with forecasting performance. We can aggregate the consistency scores across all checks for each forecaster by aggregating either the arbitrage or the frequentist violations.
We plot the average Brier score against the three aggregate consistency scores in Figure 1.
aggregated vieation betor (scrped)
aoyoopte 200l
Cardiliie 2
314
3x45433-058
Colol CoTol grev
348
C3T4:
CoSol (oc0loooge.oo AN
Zomai
(a) Aggregate frequentist metric the scraped fore- casting question dataset.
(b) Aggregate arbitrage metric on the NewsAPI ques- tions dataset.
Figure I: Scatter plots showing the relationship between consistency metrics and average Brier scores for different forecasters. Each point represents a forccaster, with the x-axis showing the average Brier score and the y-axis showing the consistency metric.. The y-axis values aggregated across all checks for each forecaster averaged over the instantiated consistency check tuples. Lower scores are better for both axes.
Bayesian consistency checks are the best proxies for forecasting performance. Figure 2a shows the strong correlation between certain consistency checks from Table 3 and average Brier scores across different forecasters. This relationship suggests that COND, which tracks logical consis- tency in conditional probability estimates, serves as proxy for overall forecasting accuracy, without knowing how the questions resolved.

---

Published as a conference paper at ICLR 2025
Cantthedecatttnge ordeed vitaton son cope Hemoodd)
CasoCareChelle Phccurttttte veritiin ve sug orer scom Ineasadd
CDI Cororiiagen
Condutione 9MM
2128
UP1-49-06
C37457702 COF1L3-9
C-Cal-OPi-4p 40
CDT-33-1550
3004
Corseot SUTiLS//2
Cafoo--ne miii
Con-orrnt
(AT precies
NA Wilr
Dog Miler pcome
(a) COND arbitrage metric on the scraped forecasting question dataset.
(b) CONNCOND frequentist metric the NewsAPI questions dataset.
Figure 2: Both COND and CONDCOND consistency metrics see Table show strong correlation with fore- casting accuracy measured by the Brier
Certain consistency metries are not well correlated with forecasting performance. The mea- sured correlations between the consistency checks and Brier scores are given in Table 1. We see that some checks yield higher signal on the ground truth performance than others. Aggregating different consistency metrics seems to improve the correlation.
We note that the selection of forecasters we test is quite limited, so we cannot guarantee the trends here representative of future LLM forecasters. Part of the correlation be attributed to better models being both more consistent and better forecasters. For comparison, the correlations of the Brier score of our forecasters and the MMLU Hendrycks et al. (2020) (college split) error rate the best approximation of our forecasters in Appendix F are 0.38 and 0.55 on the NewsAPI and scraped datasets, respectively.
We include all data (questions, tuples, forecasts, and scores) in the supplementary material.
Table 1: Correlation of consistency metries with Brier score, both of base question datasets and the derived consistency check tuples.
Scraped
NewsAPI
Arbitrage
Frequentist
Arbitrage
Frequentist
NEGATION
0.60
0.67
-0.36
-0.13
PARAPHRASE
0.57
0.61
0.13
0.24
CONSEQUENCE
0.51
0.52
0.21
0.30
ANDOR
0.20
0.25
0,02
0.06
AND
0.68
0.72
0.54
0.71
OR
0.14
0.24
-0.24
-0.31
BUT
0.20
0.67
0.63
0.77
COND
0.92
0.87
0.71
0.69
CONDCOND
0.78
0.71
0.75
0.69
EXPEVIDENCE
0.20
0.77
-0.11
0.06
Aggregated
0.62
0.85
0.49
0.66
Even good reasoning models are inconsistent. We give the full set of consistency metrics for Ope- nAl's ol ol-mini in Table 2. The Frac column counts the fraction of tuples for which the violation exceeded a certain threshold; see the full exposition of what the thresholds mean in Appendices D and The frequentist metric is not directly comparable to the arbitrage metric, but the respective violation counts F"Frac" in the table) are.
OpenAl's ol-mini forecaster, despite being one of the best reasoning models so far, violates consistency checks more than the (0.5, 0.6) threshold from Section 2 very often.

---

Published as a conference paper at ICLR 2025
Table Consistency metrics for ol-mini.
Scraped
NewsAPI
Arbitrage
Frequentist
Arbitrage
Frequentist
Check
Avg
Frac
Avg
Frac
Avg
Frac
Avg
Frac
NEGATION
0.07
58%
0.26
61%
0.08
52%
0.27
56%
PARAPHRASE
0.07
56%
0.26
61%
0.06
53%
0.24
56%
CONSEQUENCE
0.03
27%
0.13
29%
0.03
18%
0.10
19%
ANDOR
0.09
65%
0.34
71
0.07
57%
0.29
67%
AND
0.02
24%
0.11
27%
0.03
23%
0.11
24%
OR
0.11
48%
0.30
50%
0.05
48%
0.21
50%
BUT
0.11
60%
0.40
79%
0.11
63%
0.38
80%
COND
0.04
41%
0.22
52%
0.07
66%
0.29
70%
CONDCOND
0.03
30%
0.19
45%
0.04
54%
0.23
71%
EXPEVIDENCE
0.04
47%
0.27
69%
0.05
45%
0.28
63%
Aggregated
0.06
0.25
0.06
0.24
Long-horizon consistency benchmark. The results of the previous section indicate that, even on longer time horizons where it's not possible to have ground truth resolutions, we can still evaluate and compare different forecasters via consistency metrics.
We create dataset of 900 synthetic questions resolving in 2028 and create 3000 tuples in total from this dataset using the method described in Section 3.2, to evaluate the consistency the forecasters in questions with longer horizon, where it's not possible to have the ground truth resolutions. Examples of questions and the results for spt-4o are in Appendix
We intend this dataset working prototype for continual long-term forecasting benchmark.
ARBITRAGEFORECASTER CAN WE DESIGN A MORE CONSISTENT FORECASTER?
Let (I1,... be question tuple for some consistency check R, e.g. (P, -P). Given forecasts ((.1)....F(a the arbitrage metric maximization problem Equation computes the following (as the argmax and max of the arbitrage respectively):
1. Improved forecasts F' (z1), ...))...F'(( (Z,,) which consistent, i.c. satisfy S; and
2. The profit carned by arbitrageur who bets these improved forecasts against the original ones this the actual metric.
This leads us to wonder: can we use these "improved consistent forecasts" to build a new forecaster which builds on the base forecaster E, but is more consistent on R?
We introduce: the Arbitrace denoted by RR, which computes its forecast on question follows:
with base F arbitraged on consistency check R,
1. Instantiates tuple (Ii,-..Ir) satisfying R:
2. Queries FF to obtain (( (F:,);
3. Arbitrages these base forecasts per Eq and returns the arbitraged forecast for ø1-
Despite what one might assume, however, an AazbitzagePorecaste is not "definitionally" consistent on the check it is arbitraged on, but rather consistency must be investigated empiri- cally. Suppose, for example, that forecaster produces forecasts F(P) 0.5, F(para( P)) = 0.6,
F(para(para(P)) (P)))))ra(P))))) = 0.7. Then == (F)aarrA would produce forecasts (P) = 0.55, F'(para(P)) z 0.65, which are not consistent.
Appendix G contains a precise definition of ArbitragePorecastt including the case se- quentially arbitraging multiple checks (F)) R.), and theoretical discussion of its consistency
8

---

Published as a conference paper at ICLR 2025
properties. In particular, we list strong theoretical reasons to expect consistency gains from recursive ArbitrageForecaste setups, i.. ( == ((F) 1 RR in particular with NEGATION, as well in a non-recursive ArbitragePooeccsteer with EXPEVIDENCE.
Due to these priorities and the high costs of running recursive Arbit raritttareeooeeesttees (see Ap- pendix G.1), limited ourselves studying only a small number of Arbi ArbittrageForecatter setups, with a limited number of checks rather than the whole list; specifically: (g) (g)p- PN)
(9)[E)= where ==gpt 4o-mini, N, E NEGATION, PARAPHRASE, EXPEVIDENCE spectively, and r and s vary from 0 4.
The full results of our experiments with these forecasters are reported in Appendix G.2; our key takeaways from these preliminary runs look hopeful:
In the case of the checks we tested, arbitraging on a check indeed makes a forecaster consistent on that check, with increasing consistency gains with recursive depth, as shown in Fig 3. Crucially, this also applied when the arbitraging was on more than a single check: [N)(N did well on both NEGATION and PARAPHRASE; arbitraging on the next check did not increase inconsistency on the first. We are cautiously optimistic that this may extend to the full list of checks in Table 3.
This consistency gain was greatest with NEGATION, followed by PARAPHRASE, and lowest with EXPEVIDENCE. This finding is in line with our hypothesis in Appendix G that ArbitraaeForecastee would be particularly effective on consistency checks which are symmetric. and instantiate deterministically.
We do not observe reliable improvements ground truth forecasting performance, or on consistency checks other than the ones we arbitrage on. I.e. (F)R, does not reliably do better R2.
6
FUTURE WORK
We have developed a comprehensive benchmark of static consistency checks for L.LM forecasters, and demonstrated its correlation with ground truth accuracy, suggesting that our consistency metrics could serve as a proxy for accuracy when we do not have access to ground truth. We envision several directions in which our framework could be extended:
Consistency in making. AI systems may be used not only to make forecasts that inform decisions, but also take decisions directly. Here too, we can have notion of inconsistency: for example, intransitive preferences and analogously, an inconsistent decision-make may be exploited by an arbitrageur.
Training for consistency. Modulo consideration of the cost-benefit to safety, our methods could be used train LLMs for consistency, minimizing our violation metrics. This may or may not impact overall forecasting performance and other Al capabilities. One may also imagine an AlphaZero-style set-up, where an LLM is trained on the outputs of (F)", i.e. a recursive ArbitrageForecastee. wrapped around it.
Most our experiments with involved arbitraging only single check (apart from one experi- ment with both NEGATION and PARAPHRASE). due to the cost limitations described in G. It is easy to imagine how a bad forecaster could still overfit single check: simply forecasting 50% prob- ability for all questions will pass PARAPURA: ASE, EXPEVIDENCE and NEGATION but we expect that being consistent under variety of checks is difficult without a consistent world model. One ap- proach to using more checks cheaply, particularly in training, may be to randomly sample a number of consistency checks for each question.
Further experiments with ArbitrageForecatter: ArbitrageForecaster
Dynamic generation of consistency checks. Although we found strong correlations be- tween ground truth accuracy and consistency among existing LLM forecasters, our results with rebitrages orecaster demonstrate that this isn necessarily the case: it is possible to do well
'See e.g. Fishburn (1970) and the Von Meumann-Morgenteem utility theorem for introduction to deci- sion rationality.

---

Published as a conference paper at ICLR 2025
NegCheckee aug vislatoon (seneed)
Megghheccendeeeeatiiargg viuus (scruped)
0.40
0.25
0.010
0.330
2.025
0.25
0.020
0.070
2.01%
0.015
8.010
8.005
Sast-Gri-4pmmmn CP-N
CP-M2
Forecasters
(a) Average violation of T (denoted CF-Nr) on NEGATION for from 0 to 4.
(b) Average violation of (g) (denoted CF-NPr) on NEGATION for from 0 to 4.
Paraphaseehhcderrceeuu violation
0,016
0.016
0.014
0.014
8rg Veatbe
0.007
0.000
BISk GPT40-mii CP.L
P2
CF-P3
CF-P4
0.000
Basic GPI mPi-4pimirs CF KP1
CF hP2
CF-MP3
CF WP
Forocasters
Forocosters
(c) Average violation of (g)p (denoted CF-Pr) PARAPHRASE from
(d) Average violation of (g) (denoted CF-NPr) PARAPHRASE for r from 4.
Figure 3: NEGATION and PARAPHRASE violations for various Arbitrage tions, denotes gpt-4o-ming mi, N, P denote NEGATION and PARAPHRASE respectively, and the definition of the Arwitrageeorreeeecccc. setup in Def G.2.
setups. In all cap-
on consistency without improving ground truth. In particular, this means that consistency as train- ing metric could "Goodharted" by a learning Al model (Karwowski et al., 2023). One way to prevent this may be via adversarial training: i.c. have adversarial agent instantiate consistency checks that believes the agent will perform poorly on.
Evaluating RAG-augmented forecasters. We have conducted some preliminary experiments evaluating state-of of-te-of-the-ar forecasters such as Halawi et al. (2024). Unfortumately, we could not reproduce the system from Halawi et al. (2024) the time of writing, due to deprecations in the Google News API (we could not obtain access to the alternative Newscatcher API). At the time of writing, are not with the results of Halawi et al. (2024) (there exist proprictary systems that may be competitive, such FutureSearch (2024)). We thus leave the evaluation of better forecasters like Halawi et al. (2024) and Phan et al. (2024) to future work, once such forecasters are more widely available.
of other publicly-vvailable LLM forecasting systems that competitive
AUTHOR CONTRIBUTIONS
DP and APS developed consistency checks and the arbitrage and frequentist metrics. DP, AA, APS, and EW worked on the LLM question to evaluation pipeline. APS thought of and implemented ArbitrageForecaat
VB created the news-derived question dataset. AS and DP created the scraped question dataset. AA and DP created the 2028 synthetic question dataset. DP started and led the project. FT proposed correlating consistency with forecasting accuracy and advised the project. All authors helped with the writing. DP and APS wrote the first draft of the paper.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Danny Halawi for extensive discussions and help with our setup. We thank Brendan Murphy, Ezra Karger, Fred Zhang, and Tatsunori Hashimoto for helpful discussions and feedback on the paper and forecasting in general. We thank Berkeley SPAR for connecting collaborators, and BERI for partially funding the project.

---

Published as a conference paper at ICLR 2025
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in Al safety, arXiv preprint arXiv: 06505, 2016.
Kenneth J. Arrow, Robert Forsythe, Michael Gorham, Robert Hahn, Robin Hanson, John O. Led- yard, Saul Levmore, Robert Litan, Paul Milgrom, Forrest D. Nelson, George R. Neumann, Marco Ottaviani, Thomas C. Schelling, Robert Shiller, Vernon L. Smith, Erik Snowberg, Cass R. Sunstein, Paul C. Tetlock, Philip E. Tetlock, Hal R. Varian, Justin Wolfers, and Eric Zitzewitz. The Promise of Prediction Markets. Science, 820(5878) May 2008. doi: 10. 20.1116/science.u 157679.
Henry Berg and Todd A Proebsting. Hanson's automated market maker. The Journal of Prediction Markets, 3(1):45-59, 2009.
Collin Burns, Haotian Ye, Dan Klein, and Jacob Steinhardt. Discovering Latent Knowledge in Language Models Without Supervision. In The Eleventh International Conference on Learning Representations, September 2022.
Tsong Chen, Shing C Cheung, and Shiu Ming Yiu. Metamorphic testing: new approach for gen- erating next test cases. Technical report, The Hong Kong University of Science and Technology, 1998.
Maria Christakis, Hasan Ferit Eniser, Jörg Hoffmann, Adish Singla, and Valentin Wüstholz. Specifying and testing k-safety properties for machine-learning models. arXiv preprint arXiv:2206.06054. 2022.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod- cls. Transactions of the Association for Computational Linguistics, 9:1012-1031, 2021.
Peter C. Fishburn. Utility Theory for Decision Making. Wiley, January 1970. ISBN 978-0-471- 26060-8.
Lukas Fluri, Daniel Paleka, and Florian Tramèr. Evaluating superhuman models with consistency checks. 2024 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML),
volume 31, page 194-232. IEEE, April 2023. doi: 10.
URL
http:/ //tp://xxxoo doi.oi..ooo www.doi....o...ooo 1 0 11109//aMMS55 TML55330. 2024. 00017.
FutureSearch.
FUTURESEARCH: Manifold markets trading bot, 2024.
URL https://
Accessed on 26-Sept-2024.
Ozzie Gooen. Scorable Functions: A Format for Algorithmic Forecasting, May 2024.
Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching Human-Level Forecasting with Language Models, February 2024.
Robin Hanson. Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggre- gation. The Journal of Prediction Markets, 1(1):3-15, January 2002. doi: 10.5750/jpm.v 0.57700/pm.vlil.4477 111..117.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common ruptions and perturbations. arXiv preprint arXiv: /903. ,
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
Elvis Hsieh, Preston Fu, and Jonathan Chen. Reasoning and tools for human-level forecasting. arXiv preprint arXiv:2408.1200 2024.
Geoffrey Irving, Paul Christiano, and Dario Amodei. Al safety via debate. arXiv preprint 1 2018.
Mycongjun Jang and Thomas Lukasiewicz. Consistency analysis of ChatGPT. arXiv preprint 2.Xiv:2303.06273. 2023.
12

---

Published as a conference paper at ICLR 2025
Kaarel, gekaklam, Walter Laurito, Kay Kozaronek, Alex.Mnneen and June Ku. Searching for model's concepts by their shape a theoretical framework, February 2023.
Daniel Kahneman, Ilana Ritov, David Schkade, Steven Sherman, and Hal R Varian. Economic preferences or attitude expressions?: an analysis of dollar responses to public issues. Elicitation of preferences, pages 203-242, 2000.
Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tetlock. Forecastbench: dynamic benchmark of ai forecasting capabilities. arXiv preprint arXivv22009 19839, 2024.
Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Max Viktor Skalse. Goodhart's Law in Reinforcement Learning. In The Twelfth International Conference on Learning Representations, October 2023.
Li-Cheng Lan, Huan Zhang, Ti-Rong Wu, Meng- Yu Tsai, Wu, Cho-Jui Hsieh, et al. Are AlphaZero-like agents robust to adversarial perturbations? arXiv preprint arXiv:2211.03669, 2022.
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. A logic-driven framework for consistency of neural models. arXiv preprint 19Xiv: 2019.
Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, and Percy Liang. Bench- marking and improving generator-validator consistency of language models. arXiv preprint arXiv:2310.01846, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021.
Jason Liu. Instructor: Structured LLM Outputs, May 2024. URL https://github.. com/ jxr1/instructor, Version 1.4.1.
Muehlhauser, Luke. How Feasible Is Long-range Forecasting?, October 2019.
Long Phan, Adam Khoja, Mantas Mazeika, and Dan Hendrycks. LLMs superhuman forecasters,
2024. URL https: htwp::://wwwwww// wttp://wwwww........o//////teeat wttp::/wwww....aaaaoap/apa wwtbl////aaaaa/aaaeeeeeaaaoo http://oorrratinnnoee Accessed on 26-Sept-2024.
Harsh Raj. Vipul Gupta, Domenic Rosati, and Subhabrata Majumdar. Semantic consistency for assuring reliability of large language models. arXiv preprint arXiv:2308.09118, 2023.
Leonard J. Savage. Elicitation of Personal Probabilities and Expectations. Journal of the American Statistical Association, 66(336):783 801, 1971. ISSN 0162-1459. doi: 10.2307/2284229.
Philipp Schoenegger and Peter S. Park. Large Language Model Prediction Capabilities: Evidence from Real-World Forecasting Tournament, October 2023.
Philipp Schoenegger, Indre Tuminauskaite, Peter S. Park, and Philip E. Tetloc.. Wisdom of the Sil- icon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracy, May 2024.
Arnab Sharma and Heike Wehrheim. Testing monotonicity of machine learning models, 2020.
Philip E Tetlock, Christopher Karvetski, Ville A Satopää, and Kevin Chen. Long-range subjective- probability forecasts of slow-motion variables in world polities: Exploring limits on expert judg- ment. Futures & Foresight Science, 6(1):e157, 2024.
Susan Vineberg. Dutch Book Arguments. In Edward N. Zalta and Uri Nodelman, editors, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, fall 2022 edition, 2022.
Qi Yan, Raihan Seraj. Jiawei He, Lili Meng. and Tristan Sylvain. Autocast++: Enhancing world event prediction with zero-shot ranking- based context retrieval. arXiv preprint arXiv:2310.01880, 2023.
13

---

Published as a conference paper at ICLR 2025
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural net- works. arXiv preprint 2204114400, 2022a.
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting Future World Events With Neural Networks. Advances in Neural Information Processing Systems, 01272933-73066 December 2022b.
14

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Metamorphic and consistency checks. Checking logical properties of outputs of programs under cemantic-preserving transforms has long history (Chen et al., 1998). Before Fluri al. (2023), variants of the consistency check framework were used for simple ML models (Christakis et al., 2022: Sharma and Wehrheim, 2020), vision (Hendrycks and Dietterich, 2019), and chat LLMs (Jang and Lukasiewicz, 2023), among other areas. Li et al. (2019) consider logical consistency checks beyond paraphrasing and negation for simple ML models.
Forecasting and large language models. LLMs and forecasting date back to Zou et al. (2022a) and Yan al. (2023). Recently, strong performance of LLM forecasters on prediction market datasets has been claimed in (Halawi et al., 2024; Tetlock et al., 2024; Hsieh et al., 2024; Phan et al., 2024). Concurrent with our work, Karger et al. (2024) have introduced an automatically updating bench- mark for forecasting.
Scalable oversight and failures of superhuman AI. The difficulty of evaluating models with su- perhuman performance in domains without source of ground truth has long been acknowledged, and falls under the umbrella scalable oversight (Amodei et al., 2016). Forecasting using Al or- acles is one such domain. The use of consistency checks for scalable oversight has been studied in the simpler context of superhuman game Als (Lan et al., 2022; Fluri et al., 2023), and in general question-answeriigg tasks via debate (Irving al., 2018).
Consistency evaluations for LLMs. Even on tasks where the ground truth is in principle knowable, consistency evaluations have long helped in cases where checking consistency is easier than getting the ground truth labels (Elazar et al., 2021; Li et al., 2023). Raj et al. (2023) measure paraphrasing consistency and ground truth accuracy on TruthfulQA (Lin et al., 2021) and find little no correla- tion. Some forms of consistency checks have been applied on model internals to discover features related LLM truthfulness and reliability (Burns al., 2022; Kaarel et al., 2023).
15

---

Published as a conference paper at ICLR 2025
B TABLE OF CONSISTENCY CHECKS
Table 3 includes all the consistency checks tested for in our benchmark. In most of them, we leave the logical relations between forecasting questions R implicit by constructing the sentences directly. For instance, R(11, := I1 - is implied by simply writing # 22 as P, -P. In the rest of the appendix, we use the sentence- (P, Q instead of æ1, x2) notation.
Table Consistency checks and the logical consistency conditions.
Name
Tuple
Condition (S)
NEGATION
(P,-P)
F(P)+ =
PARAPHRASE R(P.Q) =P + Q (P,Q)
R(P) (F())
CONSEQUENCE
(P,Q)
R(P) ((Q))
R(P,Q) : =
ANDOR
(P.0,PAO,PQ)
AND
(A,Q,PAQ)
max(F(P) + (()))) S R(PA
Q) min((())).(Q))
OR
(P.Q,PVQ)
Fax I F(PVQ) S
min (1,((P)++((Q))
BUT
(
COND
(P.Q)P,PAQ)
F(P))()))))((PQ))
CONDCOND
(P.Q),)R(PPQ)) E(P)F(Q)P)((RPP))) B(PAQA
PAQAR)
R)
EXPEVIDENCE
P(-.
= -EP=== F(Q))
The consistency checks in Table 3 represent core logical relationships between probabilities, but many other forms of consistency checks are possible. Here are two examples that could extend our framework:
Comparative checks: Building generator-validator checks from Li al. (2023),
could ask a forecaster to predict both F(P), (QQ), and separately whether P or Q is more likely. The forecaster's probability estimates should match their comparative judgment.
Monotonicity checks: Fluri et al. (2023) propose a variant of CONSEQUENCE for real- valued quantities, where predictions must respect the monotonic ordering a sequence of future values. This connects scope insensitivity (Kahneman et al., 2000), cognitive bias where humans fail to scale probability estimates appropriately with the magnitude of outcomes.
We do not include specific consistency check for Bayesian updates, conditional probabilities are already covered by COND, CONDCOND, and EXPEVIDENCE.
16

---

Published as a conference paper at ICLR 2025
C DATA TYPES USED IN OUR PIPELINE
C.I FORECASTING QUESTIONS
Figure shows the data stored on forecasting questions. Of these, only title and body are shown to the forecaster.
Forecasting question Data Type
id: Universally Unique Question Identifier (UUID), auto-generated using a default factory.
title: Title of the forecasting question.
body: Detailed resolution criteria, background information, etc.
resolution date: The date when the question expected to be resolved. We only consider questions that have a clear date when the resolution should be decided.
the forecasting question: in this paper, only binary and conditional--inaat Options not used in this paper include multiple-choice, interval, continuous-valle or opinion.
question_type type: Type
data source: Source of the question, either the website from which was scraped or synthetic.
created_date The date when the question was created, or null if not important for the meaning the question.
url: URI. of the source if the question scraped, else null.
metadata: Any additional information, e.g., topics, tags, category; but also data fields specific to Metaculus, Manifold, etc; the source articles for NewsAPl-geneerted questions; instantiation metadata for questions in consistency tuples.
resolution: A boolean indicating whether the question resolves to YES or NO, or null if unresolved.
Figure 4: Description of the forecasting question data type.
For instance, forecasting question from Metaculus, such the one shown in Figure 5, will be stored in the form depicted in Figure 6 using our method. The original question, which asks whether SpaceX will land people on Mars before 2030, is presented with detailed conditions for resolution, including specific criteria such as the confirmation of the landing by SpaceX and the completion of an extravehicular activity (EVA) on the Martian surface.
The data type in Figure is compatible (after appropriate processing) with scraped questions from Metaculus and Manifold, and standardization helps with synthetic question generation and tuple instantiation. We do not include information about human forecasts because we explicitly focus on evaluation without relying on any human-generated probabilities.
17

---

Published as a conference paper at ICLR 2025
Will SpaceX land people on Mars before 2030?
1,22 predictions
900
Closee Jen 2030
unreed
130%
830
2017
-211
2010
2.1
2010
J.41
$200
Ad1
2001
2023
2053
Forecesters
Predkttin
Make Prediction
community:
Predict
This question will resoive Yes if SpeceX-branded mission successfully lands human the surface Mars before 2030. The landing Itself of must before January 1, 2030, UTC.
Mars
landing, however recessary person suruive long term make return trip Earth, necessary the return or long-term survival.
person the
SSeeeeX branded mission is defined to mean that the Spacex-sesociete associated logos on the spacecraft
invalved (both the boosters and the Mars-bound craft) have larger surface area than the logos of any other entity
Figure S: Example of question the Metaculus platform.
18

---

Published as a conference paper at ICLR 2025
Example forecasting question (scraped)
id: 1781
title: Will SpaceX land people on Mars before 2030?
body: This question will resolve as Yes if SpaceX successfully lands at least human on the surface of Mars on or before December 31, 2030. The landing must be confirmed by SpaceX through official announcement live broadcast. The human(s) must be alive upon landing and must perform at least one extravehicular activity (EVA) on the Martian surface, which must be documented and released to the public. the event of dispute regarding the success of the mission, the resolution will defer to the judgment of an international space agency such NASA ESA. landing attempt is made by the specified date, or if all attempts fail to meet the above criteria, the question will resolve as No.
resolution date: 2030-12-31 23:39:59400:00
question type: binary
data source: metaculus
url: attp://www.nttacuuus..coo http://cwc...as.coooooessiiiso//
metadata:
topics:
* id: 184, slug: elon-musk, name: Elon Musk, link 27681, num qum.quesions:
159
id: 485, slug:
proggram, name:
SpaceX reusable launch system, link id: 27682, num qum-question: 130
id: 1365, slug: spacex, name: SpaceX, link id: 75197, num questions: 112
* id: 564, slug: colonization-of-mar, name: Colonization of Mars, link id:
27683, num-questions: 70
id: 1768, slug: sanees--mantttasspottttiooooooo
name: SpaceX Mars
transportation infrastructure, link id: 40982, num fomm-qetsion::5 5
resolution: null
Figure Example of forecasting question scraped from Metaculus.
By processing this question through our pipeline, we retain all relevant details, such as the resolution date and specific criteria for binary outcome, while structuring the data more standardized format to facilitate further analysis. Additionally, associated metadata, including related topics and links other questions, is also preserved.
19

---

Published as a conference paper at ICLR 2025
Example forecasting question (synthetie)
id:
title: Will Russia launch manned mission to the Moon before 2030?
body: This question will resolve as Yes if, before January 1, 2030, the Russian Federation
successfully launches and completes a manned mission to the Moon, where 'successful' is defined mission where astronauts land the lunar surface and return safely to Earth. The mission must be officially recognized by Roscosmos or another authoritative space agency. In the event of joint mission involving Russia and other countries, the mission will still resolve as Yes if Russian astronauts are part of the crew that lands on the Moon. If no such mission is launched, or a mission launched but does not meet the above criteria, the question will resolve as No. In the case of ambiguity or lack of clear public information by the resolution date, the question will resolve as No unless official equivalent authoritative body
statements or evidence are provided by Roscosmos that confirm the mission's 's success as per the defined criteria.
resolution date: 2030-12-31 23:59:59400:00
question type: binary
data_source: synthetic
url: null
metadata:
-tags:
* Russia
categories:
Space
resolution: null
Figure 7: Example a synthetic forecasting question. All question generations are seeded with the metadata field.
As an example, we also show a forecasting question generated synthetically using the source tags "Russia" and "Moon" could ask whether Russia will launch a manned mission to the Moon by 2030. The structure and format this synthetic question, as illustrated Figure mirror those of real forecasting questions while maintaining the essential metadata for context.
C.2 EXAMPLES OF INSTANTIATED TUPLES
In the following examples, we focus on the question title for clarity. Figure 8 illustrates an instanti- ated AND tuple, starting from forecasting questions (P and Q) that address distinct events regarding artificial intelligence policy in the U.S. and Canada, together with a conjunction question (P (Pand__Q Q) about their joint occurrence by specified date. Figure 9 presents instantiated EXPEVIDENCE tuple, examining the global space industry's 's revenue potential alongside the political dynamics in the U.S. House of Representatives, including conditional questions that evaluate the influence of one event on another.
We note that making the detailed resolution criteria ("body" field) actually correspond to the com- posite event not straighforward, and is only in reach of the newest generations of LLMs.. A different design option would be to just list the original questions and resolution criteria separately in the "body" field, and then say what the logical operation is. We opt against it for two reasons:
A separate, unnatural format for composite questions might induce qualitatively different behaviors LLM forecasters.
Future works in this framework might not rely simple logical operations, but rather an advanced LLM grader that computes "do these forecasts make sense taken together". Our current design allows for easier extension to this direction.
20

---

Published as a conference paper at ICLR 2025
Example tuple (AND)
title: Will the United States pass federal law regulating the ethical use of artificial intelligence in energy management before January 1, 2028?
Q:
title: Will Canada implement a nationwide artificial intelligence policy before Jan- uary 1, 2028?
P_and_Q _:
Will both the following occur before January 1, 2028: (a) the United States passes federal law regulating the ethical use of artificial intelligence in energy man- agement and (b) Canada implements a nationwide artificial intelligence policy?
title:
Figure 8: Example of an instantiated AND forecasting question tuple. We omit the rest of the fields for brevity.
Example tuple (EXPEVIDENCE)
title: Will the global space industry generate annual revenues exceeding $1 trillion by the end of 2027?
Q:
title: Will the Democratic Party gain a majority the US House of Representatives after the 2026 midterm elections?
P --given_Q:
title: Given the Democratic Party gains a majority in the US House of Representa- tives after the 2026 midterm elections, will the global space industry generate annual revenues exceeding $1 trillion by the end of 2027?
given_not_
title: Conditional the Democratic Party failing to gain a majority the US House of Representatives after the 2026 midterm elections, will the global space industry generate annual revenues exceeding $1 trillion by the end of 2027?
Figure 9: Example of an instantiated EXPEVIDENCE forecasting question tuple. We omit the rest of the fields for brevity.
21

---

Published as a conference paper at ICLR 2025
D ARBITRAGE AS A VIOLATION METRIC
For the following definition we use a slightly more general notation than in the main body, to convey that methods could be generalized beyond binary forecasting questions.
Notation. Let Prop denote the set of forecasting questions are interested in, θ denote the set of possible outcomes/resolutions for an individual question, and AO denote the set of probability distributions on O. A Forecaster is map Prop - Δ0. For conditional questions that can resolve to None, we also have optional resolutions O' == θ U (None} = None)
The arbitrage metric may be seen as being motivated by Dutch Book Arguments for probabilistic consistency rules (see e.g. Vineberg (2022)). Imagine the forecaster's predictions F(z1).... Fxx,) were prices offered by bookie on prediction markets for sentences 21.... Zn. If these probabilities are inconsistent, then there are bets that an arbitrageur can make that guarantee profit in all possible (consistent) worlds regardless of the individual outcomes. For example, if Iı, I2 are two sentences such that 21 > Z2, but the bookie prices F(x1) < F(22), then an arbitrageur can simply buy æ1 and sell 22 make a risk-frre profit.
However, the bookie never changes their prices in response to trades, the arbitrageur can make an infinite amount of profit with its strategy. This is neither realistic nor useful for creating metric to measure inconsistency. Instead, we turn to market scoring rules, introduced in Hanson (2002)), where the bookie is market-maker who updates market prices in way that ensures that the reward for moving the market price of a sentence that resolves True from Po to p' is given by a proper scoring rule sp') - spo)). We then define inconsistency metric to be the minimum profit an arbitrageur can guarantee against such a market-maker, if the latter offers inconsistent probabilities ((I1)....E F(x.).
Definition D.1 (Arbitrage-based Violation Metric). Let R Prop" + [T, be an n-ary rela- tion such that R(0(z1), ... ((x,)) satisfied by the ground-truth resolutions 0 Prop + θ for all tuples (.1,... Fnn) 6 Let s Prop x@ x [0, 1] + IR be proper scoring rule that gives the score earned based on the probability assigned to the true resolution, e.g. s(x, ((r.8.pi p(0)) = log) Let (xI,....,, xnn E Prop" be a question tuple, and denote Ω := {w E 0 R(w)} the set of possible consistent resolutions (including None resolutions) of this tuple. Then for forecasts (F(#1).... F(z,)) the arbitraged forecasts A(F(21)....FfG F(z,)) (pI... Pn) and the minimum guaranteed profit of the arbitrageur ((I(())....E((( are given by:
(arg max, max) min PEΛg"
Σ
(ri, Wi, wi, pr(w:)) 8 (xi, wi, (wi))
WEn
Where by convention, any score on resolution w; None is taken to be 0.
Definition D.1 is presented in full generality: and (xx;) here are probability distributions on e. Breaking it down: each s (Iz, 45; (i())) (ra, wy; F(z,) (wx)) gives the arbitrageur's profit on the market for question 2;, given that it resolves wi. The profit is summed across all markets in the tuple, and then minimized over all consistent worlds; this minimum is maximized across all possible arbitrageur bets.
It is helpful to explicitly state Eq in the case of binary forecasting questions, as follows.
(arg max, max) min pE[0.1]= WE
(p.) - (F(x:)) ( (s p:) s(1 F(x.))) 0w()=1 (3)
We will illustrate our violation metric with three specific examples, for PARAPHRASE, NEGATION and COND. For other consistency checks, the math becomes too convoluted and we use a numerical method in project code.
AA proper scoring rule (Savage. 1971). is one that incentivizes honest reporting of probabilities: widely used proper scoring nules include the Brier score (1 - p)² and the logarithmic scoring rule log
'This is well-defined because resolutions can be taken as subset O C Prop, by treating them as forecasting questions that always resolve to themselves by definition. For example, the forecasting question T is always worth $1 and the forecasting question is always worth $0.
22

---

Published as a conference paper at ICLR 2025
profit if
(0.535.0.095)
p
F(Q)
(P(P)
profi if -P
Figure 10: Profit earned by the arbitrageur in case of inconsistency over ParaphraseChecker, taking s(p)
(pg(p) and ((P),F(Q) = 0.7, in (4).
D. PARAPHRASECHECKER
Let and Q be equivalent sentences, and suppose that the forecaster produces forecasts F(P)
and F(Q). A trader who instead brings prices to F((P) = =(()) = for both questions carns combined profit on both questions:
s x + ( - (F(()))
if
& (1 p) +1 - ((Q)) if-P
For this first example, we graph this profit function of p for illustration, shown in Fig. 10 demonstrating that any p E (0.529, 0.576) profitable for the arbitrageur, and further that the arbitrageur can guarantee minimum profit of 0.095 regardless of the outcome of P by choosing the consistent probability = 0.555.
We may compute this intersection analytically:
s (p) - (F(P)) + s (F(Q)) s -) -(L + p) (1-F(Q))
p_ loog
F(P)F(Q)
2 log
(1 F(P)) F(Q))
(((P)((Q)
p VF(P)( + (1- F(Q)
Substituting this back into either expression in (4) get the expression for the arbitrage:
V(F(P),F(Q) = 2 2og (VI(P))(Q) + (1-B(P)(1-F(Q))
(5)
As bonus, this can straightforwardly be extended the multi-question paraphrasing check:
(Pi + Pa) (i((P) ... F(P.)). Here the corresponding possible
profits are:
i ns (1 - - (1-P(()))) -P
ns (F
Equating them and solving for p, get:
23

---

Published as a conference paper at ICLR 2025
p
F(P:)
log
log
1-p
1 R(P.)
1/m
A A +1
F(P.) Il
p =
where A
F(P:)
(8)
Observe that the arbitraged probability is simply the arithmetic mean log-odds space! One may wonder if the violaton is some kind of variance measure in log-odds space, but this does not seem be the case:
SE(PP)...PFPP.. = n log (IIF()))'* 1/m = (I[(-F(P)))
1/2)
(9)
D.2 NEGCHECKER
Suppose the forecaster produces forecasts (PP) and F(-P). trader who instead brings prices to F'(P) p. (-P) =
combined profit on both questions:
s (p) - (F()) (p) - (1-F(((P)) ((- +s (F(-P))
if P
(10)
if -P
Equating them and solving before,
p
(PP)(1 - -F(-P)
2 log
log
(1-F D-F(P))F((P)
v /E(P)(1 F(-P))
p
+ -
Substituting into (10), we get:
((P()) 2 log (R(()( - R(-P)) (1 - ((P)(PP)((()))
(11)
The similarity of these results to PARAPIIRASE is suggestive: both the arbitraged probability and the violation for NEGATION can be derived from PARAPHRASE simply replacing F(Q) with 1-F( seeing the latter as the "probability implied for P by -P". This raises the natural question: Can all consistency checks be reduced to the case of PARAPHRASE arbitraging F(P) against an "implied probability" for P?
However, as we will see, COND shows that this approach does not always hold. Its violation ex- pression depends more than just F(P) and F(P A Q)/F(Q P). so there is single, neat interpretation akin to "arithmetic mean the log-odds space."
D.3 CONDCHECKER
Suppose the forecaster produces forecasts F(P), I[Q | P), F(P are (P. Q P, P A Q +
Q). The possible outcomes Ω (L, 1). Consider an arbitrageur who makes bets F'PP) p. (Q P) [) F(PAQ) pq.
In each outcome:
24

---

Published as a conference paper at ICLR 2025
8 (p) 8 (F(P)) + - P)) (pq) (F(AA))
iP ifP, Q
8 (p) 8 (P(P)) (11-))))1--------) - R P)) -1 ((--(((()
s(1 - p) - s1 - (1-F(P)) +1 pg) - s F(PA))
if-P
(12)
Equating these and rearranging:
L- 1-3(P) A
p(1-9 P[P
1-q pm (1- -IP-IIPPP---PPPC =: B
P9
F(QIP)R(PQ))
Solving, where we indicate the right-hand-sides of each equation above by A and B respectively:
1 B[
VB B. (A+1)
1
9 1+ V (B/(A+1)
1
pq
1+ BB. (A+1)
Substituting back into 12 and simplifying:
V(F(P), P),E(P Q))
-21og (VE(P)E(Q PJE(PAQ) + - [(P)(( P))(1 - B(PA
D.4 NUMERICAL ESTIMATION
Explicitly deriving the violation metries for other checkers from Equation (2) is infcasible by hand, and the expressions yielded by SymPy are very convoluted. For these checks, we use numerical algorithm based on solving a differential equation for pæ (t), as detailed below.
The arbitraging process may be understood adjusting market prices in such way that the scores in each possible w E Ω remain equal throughout the process - i.e. such that their derivatives remain equal. For derivatives (1) of the prices, the derivatives of each score (1) are:
TAO11)
(t) = [awi awn(pa)]
(1)
where
8 (pr) if wi = T, (wr(p:) = 111111101 (p)-p) p.)
if 1, if wi N/A
Then, where A(p) (a (p:) (with Ω rows and n columns), we have s'(1) A(p)p'(t)- We want (t) to be a multiple of [1 1] to ensure is the same in all outcomes w. Since the coefficient
of proportionality only controls how quickly the process converges, we can solve p((t) = A
The dynamics are then:
(0( F(x:)) (initial conditions)
25

---

Published as a conference paper at ICLR 2025
A(P) [1:1]
Consistency is reached when det A reaches 0.
26

---

Published as a conference paper at ICLR 2025
E FREQUENTIST CONSISTENCY METRIC
In a deterministic world, we cannot let any inconsistency pass; every time we prove any rule of probability does not hold exactly, must discard the forecaster flawed. This is too strict for the consistency check framework to be useful. Instead, we propose a violation metric and the corre- sponding inconsistency threshold based on statistical hypothesis testing.
Assume that each event P has a true probability value T(P), say under some world model that accounts for aleatoric uncertainty.
Definition E.1 (Frequentist consistency). A frequentistt-cossistett forecaster IF samples a Gaussian estimate T(P) + E of each event P. with variance d2 T(P)) for hyperparameter o²:
(PP) T(P) ~ N (0, - T(P))) independently for all events P.
(13)
This is principled from the frequentist perspective. Consider forecaster that just samples the (rel- evant subset of) the world n times using the best available world simulator, and estimates the prob- ability of each event P as the proportion of times that P occurs in the n samples. If we estimate the probability as the average chance of an event P with true probability p occurring out of n times, then this estimate has scaled binomial distribution with mean and variance p(1 p)/n. To reach Equation (13), replace the averaged binomial with the Gaussian of the same variance, and denote o² 1/n.
This simple model enables us to derive hypothesis tests for each of the consistency checks described in Table 3. The null hypothesis is always that the forecaster is frequentist-consiseen. Note that o² is not our estimate of the variance of any forecaster, it is just a hyperparameter that controls how strict our null hypothesis is. We leave estimating the variance of particular forecaster and testing frequentist consistency based on that alone to future work.
The expression aN c²) denotes a Gaussian random variable with mean and variance a²c². The expression aN(0, e²) + bN(0, c²) denotes a Gaussian random variable with mean 0 and variance a² +b2 All sums range over the cyclic permutations of the variables under the sum. All N(O, e²) terms appearing with the same power of are independent. Two N(O, e²) terms appearing with a different power of a may be correlated; this not important for our purposes, since we discard high-order powers of a.
Notation
Bootstrapping the true probability The final expressions for hypothesis test statistics might in- volve the true probability T(P). It is not available, we just plug in F(P) for T(P) in the end. If we had a prior on T T(P), we could combine it with ((P) to get a more robust estimate.
NEGATION We take the violation metric and the corresponding threshold to produce hypoth- esis test against this:
1= -
+
N
We estimate the unknown T values with the corresponding FF estimates. Note that, although T(P) = 1 T(-P), it is of course not necessarily the case that F(P) = 1- R--P).
The error distribution is oN
F (1-P))()()(()))) (-P))) and the two-sided test is [F(P) + F(-P) < 70 / ((P)(((P))(()) + (1 F(-P))E(PP)
for some scale factor 7 (number of standard deviations) that scales the power of the test. For exam- ple, 7 2.58 gives 99%%-conidenn interval.
We now want to compute some consistency violation metric that makes inconsistency comparable across different checks. The natural idea is to aggregate all terms dependent on FF to side; and make the hypothesis test be just some threshold on the computed violation metric.
It is possible that the denominator of the resulting expression is 0 when the forecaster is certain and is 0 or 1; to avoid division with zero, we add a small regularization term BAN = 10-3 See the last paragraph of this section for discussion of hyperparameters.
27

---

Published as a conference paper at ICLR 2025
Our consistency violation metric is then:
F(P)+ -
UNEGATION
-
BMIN
The hyperparameter o² determines how strict with rejecting inconsistencies which could be attributed to "noisy" predictions. Note that the violation metric itself does not depend on o².
A violation (inconsistency), therefore, occurs when:
UNEgatION
yo.
CONDCOND This is more complex consistency check; we derive the hypothesis test and viola- tion metric detail below. For the other checks, we just report the short derivation.
(a,b, (T(P), P) PA R)) (a' Id','',''' (I( P), PAQ).
We can write:
((P) N (0. 2a a)) +G,
F(Q P) = N (0. (( b)) +
F(R PA Q) = N (0. o c1 (()))) +
F (PP Q R) = N (0, o0 -d) +
We now compute the difference of the two expressions that should be equal. All sums and products cyclic over a, b, ((P)F(Q P)F(R) FP AQAR) abc d
Σ beN ((l-() a)) -(0,()()) -1N00-((--))-N000(-0111
+ o²
N(O. b(1 ())10-))((.c(11-111 -c1
8 II N(O, a(1 - a)).
In the above, Gaussians with the same variance are identical, and all other combinations are independent. As abc d = 0 by the law of total probability, the leading error term is next to This a Gaussian with mean and standard deviation:
V +d -v jabe Σ be(1 a) + d(1 -d)
We now discard the terms of a², 3 and in general any higher order power of a. This is principled because the coefficients can always be (in some confidence interval) upper bounded by a constant independent of a. Hence, if is small enough, the resulting test will be very close to the true hypothesis test.
We do not have the true probabilities a, b, d, we just plug in (a', U, c', d''d') = (F(P),((C F(R P F(P Q R)). Thus the hypothesis test is (where the sum is cyclic over b):c)::
'Depending on how we use the relation abc d. we can end up with different expressions in the end, We choose the one that, after plugging in, (i) yields expression for variance that is always nonnegative, and (ii) not polynomial multiple of any single value of F.
28

---

Published as a conference paper at ICLR 2025
ld'l'' - d'| na a'b' E vc(1-aa) d' d')
Our violation metric is then:
l'b/'' - d''
UCONNCCNND
Valle En - dd) + - d!) + BAIN
c' where again (a',b','', (F(P), P) PAQ),
A R)) are the forecasts.
COND Similarly as for CONDCOND: we denote (a, 6, c) = (T(P), T(P Q), T(P A Q)) and the associated (a', B, c') for the forecasts. Then compute
F F(P)F(Q) P) - F(P ((PAQ) QQ)
ab c+ (b, a)) + aN b(1 b)) N(0,c(1-0)) c)))
+ N a(1 - ((O, 6(1- -(1-0) b)).
The term next is a Gaussian with mean 0 and standard deviation:
o - b) + ba - c(1 - = -)
(db 6(1 a)) c(1 - c).
Again, we have plug in (a',b, (F(P), P(P A Q)) instead of (a, b, c). Our violation metric is then:
l'' - d'
"COND Va'V (d'I-(((1))
b)) 61 -1)) + c'(1 c²) + BuIN
And the test is again, for suitable 7 corresponding to the desired power of the test:
UCOND 7a.
PARAPHRASE Here we can simply check whether P and Q are the same.
F(P) - (FQ) = T(P) + - T(Q) -
= - ( T(P)) + (T(Q)L1T(Q1I)
This yields the following violation metric:
F(P) - F(Q))
UPARAPHRASE
(C(P)[-F(P))--- F(Q)) + BaIN
ANDOR
- - F(PAQ) =T(P)+T(Q) -(P
Es E4
= - 54
~ (T())) (1 T(P))+
+T(P' Q Q)) + T(P Q)( 10011((P1)))) Q)))).
We again plug in IF instead of T compute the error term allowed: yo / M where
29

---

Published as a conference paper at ICLR 2025
= F(P AQQAQ 1OIL--UUE F(PAQ))
and violation metric:
-
VANDOR
-PP) - ((PPA)) + 8MIN
BUT
T(P) T(-PAQ) + 63
- -2 N - QU
with error term:
yo 1PT
and violation metric:
-(PVO))K())-K-PAAG
UBET - - + BAIN
CONSEQUENCE In the case of inequalities involving S. there are two ways in which the con- sistency check can be passed. If (PP) < F(Q), the consistency check is automatically passed. Otherwise, we check for pseudo-equality using the same violation metric in PARAPHRASE.
F(P) ((Q)
[FP)) [(Q)
FCONSEQUENCE
/F(P)(1 F(P) F(Q) ((Q)) BMIN
where [F(P)= > [[Q)) is the Iverson Bracket (1 if true, otherwise).
AND Similarly to CONSEQUENCE, if the chain of strict inequalities
max(F(P) + F(Q) - 1,0) < F(P A Q) < min(F(P).
holds, then the check automatically passes. We set Vandlhh O and FAND_RHS 0 if passes the first and second strict inequality respectively.
not, then we for pseudo-equality for the violating pair:
LHS max(F(P)
1,0) = (((AQ)
RHS F(P Q) min(F()).
Equality check it fails the first inequality:
70 -T
if (P(P)+F(Q)) +
- - 1>00, 0,
ELUS
N/A
otherwise pass as F(P Q) 0.
30

---

Published as a conference paper at ICLR 2025
ANDDLHS =
-LP)=E(Q)---(PAAQ) - (((AA)))) BMIN
Equality check if it fails the second inequality:
Define F(R) = min(F(P).F(()).
ERHS = 70
F(PAQ) - F(R)
UAND.RHS = F(R) < F(P Q)1
+ MMIN
Consistency is violated if either incquality is violated, and the respective hypothesis test for pseudo- equality fails. We use VANDJIS for the first and VAND_RHS for the second inequality. We define VAND max (UADDLL:. Rus
OR
We proceed similarly as for AND.
If the strict inequality (ax(F(())F((Q) F(PVQ) min(1,F(P) + F(Q)) holds, then automatically passes. We set UOR.LHS 0 and fOR rhs = 0 if it passes the first and second strict inequality respectively.
If not, we test for pseudo-equality:
LHS (()) F(VVQ)
RHS: F(PV = min(1, F(Q))
Equality check LHS: Define F(S) max(
L =
F(S) F(PVQ)
TOR_LIS [F(S) ((PV Q)]
( BMIN
Equality check RHS:
70 (1) ifF(P) + F(Q) 1
ERRS
N/A
otherwise pass as (PP V Q) < 1.
UOR = [F(P) < F(PV Q)}- F(P Q) -(P)-F(())
(((0)(1F(Q)))) BMIN
Consistency is violated if either inequality is violated, and the subsequent hypothesis test for pseudo- equality fails. We use tOL.Is for the first and VOR_RS for the second inequality. Analogously to AND, define Vor max JaIs)
31

---

Published as a conference paper at ICLR 2025
EXPEVIDENCE Write
Q). -Q), T)) T(Q)): then
6'd (
= -
d)))
-(+toN(((1-)))
(bd+
+ +o]dN(b(1-b)) b))
+ (b--c)N(d(1-0) d))
+ (1-d))((c11
-N(a(1-a))
+ O
gives us a normal distribution with standard deviation
a va a) + ( b) + (1-
The violation metric then:
bd + c(1 - d) -a
Va(1 - + d b) + (1 d) c) (b c)2d(1 d) + BaaN
Hyperparameters for hypothesis testing Our goal is for the rejection criteria be similar to the arbitrage violation metric in Appendix D on simple examples. We choose = 2.58 for all checks, ensure 99%-confidence intervals for two-sided tests; future work may consider using different y for checks that require one-sided tests. We pick a = 0.05 (corresponding to n = 400 in Definition E.1). The allowed violation threshold for all checks is then ya 0.129. For reference, a NEGATION pair (P(P). ((-P)
(0.5, 0.59) has violation metric of 0.128, and would thus not be rejected as inconsistent. This closely corresponds to the tolerance threshold of 10-2 of profit for the arbitrage metric, described in Section 2.1.
We pick AAIN 10-3 because LLM forecasters from Halawi al. (2024) answer with at most digits of precision for events close to 0 and in probability.
32

---

Published as a conference paper at ICLR 2025
F FORECASTERS
We describe the forecaster architectures evaluated in the paper below. All of these forecasters accept model parameter working with most popular LLMs, such as gpt 4o, claude 3. sonnet and 1lama 1atmaaa3.140000 3.1-405B
In plots, the following names refer to these forecasters:
GPT-4o-05: Basic Forecaster with 2pt-40-2000 2024--00--------11
GPT-4o-08: Basic Forecaster with -pt--o-2222--
GPT-4o-mini: Basic Forecaster with op1-40-minn-2022-----2
Sonnet: Basic Forecaster with - sonnet
L3-8B: Basic Forecaster with 11ama 8-
L3-70B: Basic Forecaster with 11ama-33 1-70B
L3-405B: Basic Forecaster with 1 11laamm33 3 -455
CoI-ol-preview: CoT Forecaster with o1-preview
CoT-ol-mini: CoT Forecaster with ol-mini
CoT-GPT-4o-08: CoT Forecaster with 4p0--4-2002--0---00
CoT-GPT-4O-mini: CoT Forecaster with gpt-4o-mm
CoT-Sonnet: CoT Forecaster with claudee3..--oonee
CoT-L3-8B: CoT Forecaster with 1lama-3. 1-8B
CoT-L3-70B: CoT Forecaster with alama 1...aaaa31.... 3.1-70B
CoT-L3-40522 CoT Forecaster with 1lama-3 3, 1-405B
All forecasters receive the question (see Appendix C.1) as string render of the JSON object in Figure 11.
"title:: "Question title".
"body": "Question body and resolution criteria"" dastt "YYYY-MM-DD".
"reated_date":: "YYYY-MM-DD"
Figure 11: The format in which questions are presented forecasters. If created eed_tte available, it omitted.
F.1 BASIC FORECASTER
The Basic Forecaster is simple forecasting model that uses language model to generate prob- ability estimates for given questions. We use the Instructor library Liu (2024) to make the output conform specific Pydantic model that has a prob field forced be float between and 1.
You are an informed and well-calibrated forecaster. I need you to give me your best probability estimate for the following sentence question resolving YES. Your answer should be float between 0 and 1, with nothing else in your response. Question: {question)
Figure 12: The prompt used for Basic Forecaster.
33

---

Published as a conference paper at ICLR 2025
F.2 CoT FORECASTER
The CoTForecaster is composed of two steps:
The first model call a native chat message with a chain-of-thought reasoning prompt in Figure 13.
2. Then, opt-4o-mini is used in an Instructor Liu (2024) call to parse the output into single probability estimate similarly as in the Basic Forecaster, plus the reasoning summary.
We this two-step process because of concerns with structured outputs degrading reasoning ability in language models.
You are an informed and well-calibrated forecaster. I need to give me your best probability estimate for the following question resolving YES. you think is likely the question resolves YES, the probability should large; you think unlikely the question resolves NO, the probability should small. want you to first provide detailed reasoning for your answer, and then give me the probability. Your answer should be in the format: 'Reasoning; [your reasoning here] Probability: [float between 0 and LJ'
Note: unless explicitly stated in the prompt, do worry about the exact formatting of the output. There be an extra step that will summarize your output into the final answer format. For context, the final answer format described by the following Pydantic model: (response model 1 model fie fields-} Again, just try to answer the question best as you with all the necessary information; the output will be cleaned up in the final step. Question: {question]
Figure 13: The prompt used for CoT Forecaster.
34

---

Published as a conference paper at ICLR 2025
Algorithm Atbbtrrageooooreccststee algooio (F)
input a (x)
Query base forecaster
w f 1 for
(Ri, Ti) in C do
(r, 22 .n) t [z(x)
Instantiate tuple of size n = ReR Query base forecaster tuple > arbitrage the forecasts as per Def p now carries information from other markets
(2,,... pn)) (F(22).... F(
(p.p2,.... Pn)) t A (..
(p. 02.p...... Pn.)
+ n
end for return
G ARBITRGEFORECASTER
To formally define Arbit ArbitraageFreccaterr we need to first formalize our "instantiation" pro- cess mathematically:
Definition G.1 (Tuple sampler). Let R Prop" + (T, A + [T. be consistency check. Then we call J Prop ~4 Prop" a bogll-base-question tuple sampler" for R if for all x, [(x) = x and ((](z)) holds surely.
A multiple-base caltiple-base-qusstee described 3.2, can simply be composed with question sampler g Prop Prop (e.g synthetic generator or sampler from our dataset) to produce single-base-questre sampler [(x) == I(x,
tuple sampler I Prop"" + Prop". like the instantiation process
Next, in order to correctly handle sequentially arbitraging checks and prevent bias towards later applied checks, we need introduce "weighted" arbitraging. This follows easily from Eq D. by simply having the scoring rule for each question be wy log(p). We denote the calculation of arbitraged probabilities under these weighted scoring rules by A (("I.....)
Let Prop + AO be the "Base Forecaster", and let C == [ J). SR, J&)] be a list of consistency checks along with respective Lingle-baee-uuestinn tuple samplers. Then we construct new forecaster (IF) Prop + AO that produces its forecast for a given question x as given in Algorithm I; we call this the Arbi wageearecasssee. with base FF and check list C.
Definition G.2 (Arbitrageeooecatttttt
The first thing we observe that this isn't necessarily robust to different instantiations. For this reason, we a priori expect that ArbitrageForecaae question checks like NEGATION and PARAPHRASE.
will be more effective on single-base-
We might hope that the ArbitrageForecasstttc tionally consistent on the checks is arbitraged on. However, this not the case even for ArbitrageForecaase
introduced in Def G.2 would be defini-
applied to single check R(x1,.... E.). because the tuple of forecasts that is arbitraged to compute (F) (F).),))) (21), the tuple arbitraged to compute (F),,.)))) (I2). the tpple arbitraged to compute (R.S.. (x,)) are different. While the tuple instantiated to (RRS..T)
(#i,... xn) (at least if the tuple sampler J is deterministic and happens to be the same as the one used in the instantiation of the check), the tuples
compute (F) VR.SS. (x1) could indeed be J(x1)
instantiated to compute (z)) for i + will be J (xx), all of which are different from one another.
To make this concrete, consider the simplest case (F) (where P is short for PARAPHRASE). let para be a deterministic tuple-sampler for PARAPHRASE. (IF) p(x) is calculated by arbitraging F(x) and F (z)ara(z)). But (para(x)) is calculated by arbitraging (para(x)) and F(para(para(z)))
A priori, this gives the following hypothesis: ArbitragePorecaster will be especially effective for fundamentally "symmetric" checks like NEGATION where nog(neg(P)) is likely to be a very similar sentence to P. Although have not conducted a full scale experiment of Arbit rebitrageFF ragee ooreeeorreserr with each checker, our preliminary results in Table 4 suggest very good performance of t weroitrageoorccate.
NEGATION.
35

---

Published as a conference paper at ICLR 2025
Suppose, however, that we had an "extended" ArbitrageForecaster that made its forecast for z based on the tuple (z, para(z), para? para" (z)) then its forecast for para(z) would be based on (para(x), para (x),... para" para"+(()) (x) these tuples would be "almost" the same, except with para"*1 (x) instead of x, and this extended A Arbi #rbitraggeorrccstee would be "almost" consistent PARAPHRASE.
This precisely the idea behind recursively applying ArbitrageForeeteeec
to itself: we recursively define (x) == (I(P)))((())) for I,...n) then if this iteration approaches a fixed point, this fixed point (F) is consistent. More precisely:
Let (R,S,J) be an n-ary consistency check and corresponding deterministic tuple sampler satisfying Def G.I, and have A(pr....PA and V(pi,...Pu denote the arbitraging function and arbitrage metric corresponding to R as per Def D.I under logarithmic scoring rule. Then, for some "base forecuster (IF) =F, recursively define
Theorem G.3 (Consistency of recursive ArbitzageForecastern
(E) :=
for II....))
If this iteration converges pointwise in log-odds space i.e. if for all x E Prop, the sequence (By' (1) has limit strictly between 0 and 1, then ((F)(((()) for I....n) 0.
Proof. Recall per Def D.1 that, where Ω is the set of possible outcomes allowed by R:
((F)) (I(z)) for i I....n)
-mnn
(og(A((())(]))))))))) (I)) for == I,...n). .):) log log (w()=T
weRl
+ (log(1 A((F)) (I(()) for = =1; (I....n) - log(1 - (T)(()))) ( (1)=1
min
(F) (J(x))) - IFg (z)))) bw(t)=T
wER
+ (log(1
(F) (J)))) - log(1 (F) 6w))==1
Since (IF (x)) converges to something that neither 0 nor do log log( and 1og(1-((4) (x)). And as this is true for all x, so in particular it is true for J(r). Thus the expression above is a finite sum of terms that each approach O.
This is somewhat weak result: other than for NEGATION and PARAPHRASE, none of our static consistency checks involved a deterministic instantiation process they all require sam- pling other related base questions, and having the checks use the same instantiation process as the ArbitrageFore www.aeeooreasteerrrr would be cheating.
Furthermore, this gives PARAPHRASE, we have the following where log odds p denotes log =
actual conditions for the convergence of the iteration. At least for
Theorem G.4 (Convergence of recursive ArbitrageF wwwrraagFFrecasteer.... for PARAPHRASE), If the sequence a; log odds (paraa (x)) convergent, then the condition of Theorem G.3 holds for the recursive Arbi t wwwtttrreaaorrrrreeeeee defined arbitraged on PARAPHRASE with tuple sampler para.
Proof. Recall from Sec D.1 that the arbitraged probability for PARAPHRASE is simply the erage the original probabilities in log-odds space, i.c. log odds ((para(z)) = log odds Fards((r) udds odds (para(a)) We can apply this recursively get:
log odds F(para"(x:) (7)
(x)
Σ
2"
Which is simply binomial moving average of log odds F ((para (x)) a, and converges iff a, does. Convergence in log-odds space is equivalent to convergence of probability to something other than 0 or 1, so the result follows.
36

---

Published as a conference paper at ICLR 2025
G.1 CHOICES OF EXPERIMENTS
:- - (1,,, Sk,Jk) involves 1 + (n. 1) calls to F. plus at least 2 + nR: 2) (where mi, is the number of separate base questions that must be generated synthetically in each tuple) LLM calls for the J.s.
A single call to (F) where Č=
For all the checks listed Table 3, this amounts to a total of 49 LLM calls per question. For recursive Arbit sebittae ragee orecaster set-up of depth r, this amounts to 49" LLM calls per question, which can get prohibitively expensive. Even on gpt spt-4o- mommmin and assuming 600 input tokens and 600 output tokens on average, this amounts to = $0.02 per question at depth 1, and = $2500 per question at depth 4.
Furthermore, it was not clear that experimenting on all checks made logical sense: recursive Arbit t rageForecaster set-ups with COND, CONDCOND and EXPEVIDENCE would involve forms like P | (Q R), which do not have a basis in probability theory. We decided to prioritize studying the following hypotheses and research questions, motivated by the theoretical discussion above:
1. We hypothesised above that Arbitrage ArbitrageForecaster will be particularly effective on checks that are symmetric and have deterministic instantiations thus we studied (gpt-4o-mi ni NEGATION-
2. We hypothesized that there would be consistency gains from increasing depth thus we studied recursive Arbit reeeiirrgeoooccccoeeee where it was most practical to.
setups on NEGATION an PARAPHRASE,
3. We were interested to know if the consistency gains observed when arbitraging on one check alone would persist after arbitraging on a sequence checks - to predict if this would hold when arbitraging the full sequence of checks, we did preliminary of 1opt-4o-mine NECATIONN N PARApHRaSE and tested if maintains consistency on NEGATION and PARAPHRASE.
4. We expected (I)) EXPEEVINEE to improve ground truth and consistency scores across the board. This is based on our intuition that arbitraging on EXPEVIDENCE essentially "informs" the forecast on a question x with consideration information y except instead of subjectively feeding this information (e.g. chain-of-thought), it adjusts for it via a strict probabilistic rule. Although recursive setup would not make sense for EXPEVIDENCE, (F)) [E simply sequentially arbitrages on EXPEVIDENCE repeatedly (breaking the seed each time ensure unique new questions y),, which amounts to informing the forecast for x with information !/, y/2 etc.
The results reported in Sec 5 of the main body and G.2 of the Appendix provide evidence in favour of hypotheses 1 and 2, answer 3 in the affirmative, and do not provide clear evidence on 4.
Future work should compare (F)) [E)/EEEEVEEE))] against a comparable chain-of-thought model in which the forecaster is asked to consider these related questions before makes its forecast.
G.2 RESULTS TABLES FOR BRDITRAGEFORECASTER
config- urations we experimented with are reported in Tables 4, 5, 6 and 7. The results included are for the NewsAPI dataset and the arbitrage metric. Results for the scraped and 2028 synthetic datasets (Appendix L)), as well as for the frequentist metric, look very similar; they are available in the sup- plementary data of this paper.
Consistency violation and ground truth results for each of the ArbitrageForecaatee
37

---

Published as a conference paper at ICLR 2025
Table 4: Consistency results (arbitrage metric) for (gppt--oomnnn (denoted CF-Nr) forecasters on
NewsAPI questions.
Check
gpt -it--o
CF-NI
CF-N2
CF-N3
CF-N4
Avg
Frac
Avg
Frac
Avg
Frac
Avg
Frac
Avg
Frac
NEGATION
0.036
43%
0.012
33%
0.007
22%
0.004
11%
0.004
9%
PARAPHRASE
0.013
27%
0.012
36%
0.008
23%
0.006
16%
0.005
17%
CONDCOND
0.084
85%
0.111
88%
0.121
91%
0.129
94%
0.136
93%
EXPEVIDENCE
0.015
27%
0.009
35%
0.008
25%
0.007
26%
0.007
25%
CONSEQUENCE
0.005
10%
0.003
9%
0.003
7%
0.002
4%
0.001
3%
AND
0.006
20%
0.019
45%
0.027
53%
0.031
59%
0.035
65%
OR
0.007
13%
0.004
10%
0.002
6%
0.002
6%
0.001
4%
ANDOR
0.017
38%
0.024
58%
0.031
61%
0.033
67%
0.035
66%
BUT
0.053
75%
0.081
84%
0.091
89%
0.100
88%
0.107
91%
COND
0.062
88%
0.085
92%
0.107
91%
0.119
94%
0.131
96%
aggregated
0.030
0.036
0.041
0.043
0.046
Brier score
0.185
0.204
0.202
0.201
0.201
Table Consistency results (arbitrage metric) -4--mmi-mmmmm-----m PARAPIRASI (denoted CF-Pr) forecasters on NewsAPI questions.
Check
gpt-4o-ma min
CF-PI
CF-P2
CF-P3
CF-P4
Avg
Frac
Avg
Frac
Avg
Frac
Avg
Frac
Avg
Frac
NEGATION
0.036
43%
0.028
49%
0.026
50%
0.023
46%
0.024
44%
PARAPHRASE
0.013
27%
0.006
22%
0.004
11%
0.002
6%
0.002
3%
CONDCOND
0.084
85%
0.083
83%
0.079
85%
0.080
83%
0.079
84%
EXPEVIDENCE
0.015
27%
0.014
28%
0.012
24%
0.011
28%
0.012
28%
CONSEQUENCE
0.005
10%
0.002
4%
0.001
3%
0.001
2%
0.001
2%
AND
0.006
20%
0.004
12%
0.005
13%
0.004
12%
0.004
12%
OR
0.007
13%
0.005
10%
0.004
9%
0.003
10%
0.003
9%
ANDOR
0.017
38%
0.015
41%
0.014
42%
0.013
39%
0.013
39%
BUT
0.053
75%
0.053
76%
0.049
77%
0.051
79%
0.048
79%
COND
0.062
88%
0.066
93%
0.071
95%
0.069
95%
0.071
95%
aggregated
0.030
0.028
0.026
0.026
0.026
Brier score
0.185
0.176
0.175
0.174
0.175
Table 6: Consistency results (arbitrage metric) for (gpt-40-minl) NEGATION, (ARAPPRA (denoted CF-NPr) forecasters NewsAPT questions.
Check gpt 40 mini
CF-NP1
CF-NP2
CF-NP3
CF-NP4
Avg
Frac
Avg
Frac
Avg
Frac
Avg
Frac
Avg
Frac
NEGATION
0.036
43%
0.014
30%
0.007
18%
0.004
9%
0.003
6%
PARAPHRASE
0.013
27%
0.006
17%
0.003
7%
0.002
2%
0.001
2%
CONDCOND
0.084
85%
0.095
90%
0.096
86%
0.108
94%
0.115
94%
EXPEVIDENCE
0.015
27%
0.010
27%
0.007
27%
0.006
22%
0.005
21%
CONSEQUENCE
0.005
10%
0.003
7%
0.001
3%
0.001
2%
0.001
0%
AND
0.006
20%
0.011
30%
0.010
28%
0.011
34%
0.012
39%
OR
0.007
13%
0.004
11%
0.002
5%
0.001
4%
0.001
2%
ANDOR
0.017
38%
0.017
43%
0.016
46%
0.016
46%
0.016
47%
BUT
0.053
75%
0,070
85%
0.072
91%
0.077
91%
0.083
97%
COND
0.062
88%
0.082
96%
0.076
97%
0.076
97%
0.077
98%
aggregated
0.030
0.031
0.029
0.030
0.031
Brier score
0.185
0.188
0.195
0.200
0.202
38

---

Published as a conference paper at ICLR 2025
Table Consistency results (arbitrage metric) (cot-4o-mini) EXPEVIDENCEJAA (denoted CF-rxEEI) forecasters on NewsAPI questions.
Check
opt-4o-man
CF-1xEE1
CF-2xEE1
CF-3xEE1
CF-4xEE1
Avg
Frac
Avg
Frac
Avg
Frac
Avg
Frac
Avg
Frac
NEGATION
0.036
43%
0.030
51%
0.026
49%
0.024
50%
0.025
53%
PARAPHRASE
0.013
27%
0.008
22%
0.006
22%
0.005
19%
0.005
18%
CONDCOND
0.084
85%
0.057
82%
0.053
79%
0.050
76%
0.044
74%
EXPÉVIDENCE
0.015
27%
0.008
22%
0.007
19%
0.007
16%
0.007
20%
CONSEQUENCE
0.005
10%
0.003
8%
0.002
7%
0.002
5%
0.002
6%
AND
0.006
20%
0.002
6%
0.002
6%
0.002
4%
0.001
5%
OR
0.007
13%
0.004
9%
0.003
8%
0.002
8%
0.003
9%
ANDOR
0.017
38%
0.014
42%
0.011
39%
0.010
34%
0.011
35%
BUT
0.053
75%
0.040
71%
0.039
74%
0.040
77%
0.035
68%
COND
0.062
88%
0.049
88%
0.046
89%
0.044
88%
0.040
87%
aggregated
0.030
0.021
0.020
0.019
0.017
Brier score
0.185
0.172
0.171
0.171
0.173
39

---

Published as a conference paper at ICLR 2025
H PROMPTS FOR THE EVALUATION PIPELINE
In this section, we present the prompts used for the different parts of our pipeline. For each LLM call, use gpt-4o with structured output Pydantic format enforced by the Instructor library Liu (2024) and JSON API calls. The whitespace in the figures is not representative of the whitespace in actual queries.
Synthetic question generation prompt
want help generate some forecasting questions for forecasting market site like Metaculus or PredictIt. will provide you with category and some tags. Your task generate questions that be answered with probability between and 1. For each tag, generate relevant question the tag pertinent to the category. If the tag is not relevant, generate a general question about the category. Examples:
{example_1)
[cxample_2
{cxample_3)
{cxample. 4
(example S
[example 6)
Category: [category) Tags: [tags}
Figure 14: The prompt used for generating the title field of forecasting questions, given the category and tags metadata.
list of initial quality-filtee filtered questions is supplied to seed the list of examples.
Relevance scoring prompt
I'm doing a project that involve eliciting probabilities from LLMs measure the calibration, consistency and such properties LLM forecasters. As part of this project we will be taking logical combinations forecasting questions and cliciting probabilitics on them. need your help in deciding, for two given forecasting questions, whether it makes sense to think about their logical combinations/whetlae it's worth doing
For example, we might want to elicit probability 'Will Donald Trump win the 2024 US presidential election? AND Will US economic growth exceed 3.5% in 2025?'
because Trump winning the election might potentially (positively negatively) affect economic growth in the following year.
But we probably wouldn't 't care about the probability of 'Will Donald Trump win the 2024 US presidential election? AND Will the men's 's deadlift record be broken in 2025?'
because those Can you help me with this? will just give you two forecasting questions, you must give me examples of reasons someone might be interested in the logical combination of those questions; based on realistic these reason(s) are, provide-
wholly unrelated.
1. One
2. score between 0 and 10 to advise combination (with being 'the logical combination is nonsensical, nobody would ever ask something like that', 10 being 'yeah that's a perfectly legitimate question could imagine seeing that on Manifold or Metaculus')
whether it makes sense to consider their logical
Figure 15: The prompt used to decide whether two questions related enough to be combined in instanti- ated tuple.
40

---

Published as a conference paper at ICLR 2025
Tuple instantiation prompt OR
You are helpful assistant. I will give you two forecasting questions with Yes/No answers. You should then give me the logical OR of these two questions, i.. the question that would be answered YES if EITHER question is answered YES, and NO otherwise. Notes:
Your response should be as clear possible, since the words 'and" and "or' are used ambiguously in natural language. For example. 'Will P happen will Q happen? is usually confusing, as sounds like you are asking which the two will happen (whereas you'l re actually seeking a YES/NO answer on whether either of the two will happen). Instead, if there is any chance of confusion, you should give me something like: Will either of the following occur: (a) P (b) Q?
When the questions allow for a simple rephrasing or factorization (c.g. using words like respectively', 'both' or "either'), go for it.
If both of the given questions already logical combination of questions, join them in most natural way possible. E.g.
combine ((P1 OR P2) OR Q) how you would combine (P1 OR P2 OR Q)
((P1 AND P2) OR Q) might have to be combined something like: Will EITHER of the following occur: (1) BOTH of the following occur: (a) P1 AND (b) P2 (2) Q. Unless more natural formulation exists.
careful when combining conditional expressions (which often have words like 'given' and 'if"). '(Given A then P) OR (Given B then Q) should be combined as is, rather than messing up the conditions. E.g. a phrasing like "Will either of the following occur given their respective conditions: (a) Given A then P? (b) Given B then Q?' good.
This also applies when only one of the questions is conditional. Like 'P OR (Given A then Q) s'should be phrased something like: "Will either of the following occur given their respective conditions met? (a) (b) Given A, then Q?".
Most importantly: make sure you retain ALL the information in the question bodies from BOTH base questions! You cannot discard single relevant detail. All this is for an experiment to test the logical consistency of forecasters: The combined question you give will he handed the forecasters without having seen the base questions, so it critical that all the information in questions be included in your logical combination; the resolution criterion for each component should be neatly and clearly provided.
Also, make sure that the title self-sufficient independent of the body, i.e. question that be meaningfully answered without looking at the body. So you CANNOT give me question title like 'Is the following true?" or 'What will happen if the following happens?"
One type of question you may be given is single choice from multiple choice question. For example, you may he given 'Which of these countries will legalize human cloning by 2030? (Japan)'. This is asking Japan will recognize and legalize humaan cloning by 2030. Such question may also itself be a logical combination - c.g. 'Which of these countries will legalize human cloning by 2030? (UK. France, Germany) is asking if any either of the UK, France, or Germany will legalize human cloning by 2030. Make sure correctly combine such combinations as previously described.
Figure 16: The prompt used for instantiating OR tuples. We use similar prompts for other checks.
41

---

Published as a conference paper at ICLR 2025
Verification prompt- CONSEQUENCE
will provide you with two propositions, P and Q. Your task is to assess whether Q is proposition that will always be true if P is true. In other words, validate whether Q is a logical implication of P. ensuring that Q will always occur if is true. Reject if and Q are completely equivalent. Q should be logical consequence of P, but not necessarily the other way around. Reject if you need any additional assumptions to derive Q from P. Reject if Q is just formed by making some resolution criteria more vague not operationalizing them (but accept if it made by actually loosening some resolution criteria while still precisely defining everything). Reject if Q is 'ERROR: NO CONSEQUENCE FOUND' something like that. Example
P: computer can receive emails.
Q: A computer is connected to the internet. reasoning: If computer receive emails (P), then it must be connected to the internet (Q), internet connection is necessary for receiving emails. Therefore, is logical consequence of P. valid: True
Example 2:
P: The ground is wet.
Q: It is raining.
reasoning: can easily imagine the ground being wet (P true) without raining (Q false). So does not imply Q.
valid: False
Example 3:
P: is daytime.
Q: The sun has risen and set yel.
reasoning: The two statements are logically equivalent, as daytime (P) is defined by the sun being above the horizon and not having set yet (Q). So Q is logical consequence of P, but also completely equivalent it, therefore useful
valid: False
Example 4:
P: Will at least 50 percent of the world's population live in Asia by 2050?
Q: Will Asia have at least 3 billion residents by 2050?
reasoning: They probably thought
logical consequence of because the world population is billion, half of that 4 billion, so if Asia more than 4 billion people it must more than 3 billion people. However, this assumes that the world population in 2050 is billion, which we do not know for certain. Without knowing the world population in 2050, we cannot judge if 50 percent of that is more or less than 3 billion.
valid: False
Example
P: Will ANY of the following happen 2025? (a) A manned mission to Mars (b) A new Starship launch by SpaceX?
Will manned mission Mars happen in 2025?
reasoning: Suppose only new starship launch happens, but no manned mission to Mars. Then true, but Q is false. So Q is not logical consequence of
valid: False Example 6:
P: Will there be epidemic of meningococcal discases in 2025? (An epidemic is defined as discase with an infection rate of over 15 per 100,000 people over span of two consecutive weeks.) Q: Will there be an epidemic of meningococcal diseases in 2025? (An epidemic is defined as a disease that infects large number of people over short span of time.) vague,
reasoning: Q just makes the definition of epidemic
do not count it.
valid: False
Example 7:
P: Will there be an epidemic of meningococcal diseases in 2025? (An epidemic is defined as discase
with infection rate of over 15 per 100,000 people over span of two consecutive weeks.)
Will there an epidemic of meningococcal diseases in 2025? (An epidemic defined disease with an infection rate of over 10 per 100,000 people over span two consecutive weeks.)
reasoning: Q is a logical consequence of P, as the definition of an epidemic in Q is a subset of the definition of epidemic in P. If an epidemic is defined as infecting more than 15 per 100,000 people, it must also be true that infects than 10 per 100,000 people, 15 is more than 10. valid: True
Figure prompt used to verify the generated CONSEQUENCE tuple logically valid.
42

---

Published as a conference paper at ICLR 2025
I
FEEDBACK FORM EXAMPLES
Example Entry
title: Will the United Kingdom deploy an Al-powered public surveillance system with real-time facial recognition capabilities 2025?
body: Resolution criteria: This question will resolve as Yes if, by December 31, 2025, the United Kingdom has officially deployed public surveillance system that utilizes artificial intelligence for real-time facial recognition across any part of its public spaces. The system must be capable of identifying individuals deployment of such a system must be confirmed by an official government announcement or doc- umentation. In the absence of public announcement, credible reports from at least three major news organizations (e.g., BBC, The Guardian, Reuters) will be considered sufficient evidence. The system must be operational and not in trial phase. If multiple systems deployed, the resolution will consider the first system limited to specific cities or areas), the question will resolve as Yes the system intended to be expanded nationwide. Edge cases, such as temporary deployments for specific events or the use of similar technology in private spaces, will not count towards this question resolution.
matching them a database with least an 85% accuracy rate. The
meets these criteria. In the event of a partial deployment (e.g.-
resolution.date: 2025-12-31 00:00:00+00:00
metadata:
tags: [United Kingdom]
category: (Artificial Intelligence)
Example Feedback
bad .or wwr.reeeaantt.nom inchuude information:
unintuitive or wrong resolution criteria:
too specifie_criteria
criteria or edge cases:
ambiguities: Should specify which public news agencies would count resolution.
edge cases not covered:
general feedback:
formatting issues:
rewritten_tive _itte:::
Resolution criteria: This question will resolve as Yes if, by December 31, 2025, the United Kingdom has officially deployed public surveillance system that utilizes artificial in- telligence for real-time facial recognition across any part of its public spaces. The system must be capable of identifying individuals and matching them to database with at least 85% accuracy rate. The deployment of such system must confirmed by an official government announcement or documentation. major news organizations (BBC, The Guardian, Reuters, Bloomberg, New York Times, Washington Post) will be considered sufficient evidence. The system must be operational and not in trial phase. If multiple systems deployed, the resolution will consider the first system that meets these cri- teria. In the event partial deployment (e.g., limited to specific cities areas), the question will resolve as Yes the system is intended to be expanded nationwide. Edge cases, such as temporary deployments for specific events the use of similar technology in private spaces, will not count towards this question's 's resolution.
rewritten body:
the absence
public announcement, credible reports from at least three
rewritten_resollee www.:rrrroooooooom date:
discard reason:
43

---

Published as a conference paper at ICLR 2025
J CONSISTENCY AROUND A QUESTION
There is no particular reason why we need a starting dataset to measure consistency over questions and the corresponding instantiated tuples; single starting question suffices. We give a prelimi- nary exploration of a pipeline for measuring consistency around given question. This pipeline is especially useful when we have a dataset of questions and want consistency metric for each of these questions. For example, to understand how much consistency helps with understanding the correctness of forecast, we want a per-question consistency metric to compare with a dataset of Brier scores.
We follow a similar process as in Section 3.1 and Section 3.2. We start with dataset of ques- tions we want consistency metrics around, and then few-shot prompt gpt-4o (see Figure 18) to generate related questions for each source question. We follow the deduplication process based on text eexeeenbbd 3-small 11 embeddings from OpenAl to ensure diverse questions.
As in Section 3.1, after title creation, we generate question bodies and resolution dates using a few- Next, this dataset of each source question followed by generated related questions are used to create logical tuples in the same form as in Section 3.1. We ensure that each source question is included in the tuple, along with the necessary number of related questions for the specific check: 1 for NEGATION, for COND, and so on.
shot prompt
For tuples where the order of the questions matter, such COND(P, PIP, Q), allow the source question to take the position of P or Q. Overall, we get dataset of tuples for each source question, such that the source question is included in the tuples. We follow the same steps for verification and evaluation. For evaluation around a source question, we aggregate the consistency metries by source question.
Synthetic question generation prompt for source question
Objective: Generate set of forccasting questions for forecasting market site like Metaculus or PredictIt. will provide source question. Your task generate (num questions) new related questions that are logically related to the provided source question. Each new question should be suitable for probabilistic evaluation and should logically combine with the source question in a meaningful way. Guidelines:
The new questions should explore related scenarios, alternate outcomes, consequences and preroquisites of the source question.
Consider alternate outcomes, timelines, or deeper implications that are connected to the theme of the source question.
Each question should be binary and can be answered with probability between 0 and 1.
The source question will optionally include hody (detailed resolution criteria). If the source question has body, use to inform the generation of related questions. You need to generate only single sentences, not detailed resolution criteria. Examples:
[example_1]
(cxample_22 2
{cxample_3)
Source question: (source.question) question
Related questions:
Figure 18: The prompt used for generating the title field of related questions, given a source question.
44

---

Published as a conference paper at ICLR 2025
K CREATING FQS WITH KNOWN RESOLUTION FROM NEWS ARTICLES
This section describes a pipeline for creating forecasting questions with known ground-truth resolu- tions using news articles retrieved from NewsAPI. We derive an initial set of forecasting questions directly from the news articles. Then, to ensure broader coverage and mitigate dataset biases in- herent this approach of generating questions, we generate additional questions by spanning their reference classes, modifying key components like location or entity while preserving thematic and temporal consistency.
Finally, we verify and, where necessary, assign ground-truth resolu- tions all generated forecasting questions via the Perplexity API (perplexity/11 aallaama-.... 1ew.--s-----g---e- (atphugeee12---ooneeeeee see Appendix K.3 The ground truth resolutions given by perplexity/Ilama -.1---naar---a--eee -age-l2kkk-n- always correct, but have an error rate of less than 5% when applied to the scraped question dataset.
1128k-online are not
K.1 NEWSAPI-BASER FORECASTING QUESTION GENERATION
We use NewsAPI due to its diverse set of sources and free availability, making it suitable for our application. Additionally, we curate list of reliable news sources, such as Associated Press, which tend to provide more informative and factual content rather than opinion- based articles. These sources yield higher volume of articles grounded in real-world events that can effectively trans- formed into forecasting questions.
We gather daily news articles from July 2024 to 31 August 2024 through NewsAPI. These articles include fields such as the title, content, description, and publication date, and are consolidated into a single file for further processing.
At this stage, we encounter an issue: conflicting news articles from different dates report opposing information. For instance, one article states that President Joe Biden confirms his candidacy for the 2025 U.S. Elections, while later article claims he withdraws. These discrepancies lead to the generation of forecasting questions with contradictory resolutions.
To address this, remove older articles that are highly similar to more recent ones by calculating a Named Entity Recognition (NER) similarity score", based on the ratio of shared entities to unique ones. Articles surpassing certain similarity threshold are treated duplicates, allowing to discard outdated and repetitive information and resolve the issue as in the Biden problem above.
We feed processed articles to got-4o determine their suitability for creating forecasting ques- tions with binary resolutions, judging them based on parameters such as clarity of content, contextual relevance, binary resolution potential, and specificity. The prompt for this is in 19.
"haps///ppacyioomoooleiiee
45

---

Published as a conference paper at ICLR 2025
Example Validated News Article with Reasoning
Article
Source
ID: bloomberg
Name: Bloomberg
Author: Bloomberg
Title: HDFC plans sell $1.2 billion of loans to plug funds gap
Description:
The bank is in talks with local asset managers including ICICI Prudential AMC,
Nippon Life India Asset Management and SBI Funds Management issue so-called pass through certificates
URL:
Image URL: http://ttttpp..............oaoor octe:///tttttt/////cco/ccaaaaaaa
sector-G22D92000) 1pt../......................
Published 2024-08-31T13:27:5666
Content: HDFC Bank plans to sell much 100 billion ($1.2 billion) of loan portfolios
using a dcht instrument rarely used, as nation's banks devise ways to overcome challenges in raising deposits...
Validation Result:
Validation Reasoning:
The article provides clear information about HDFC Bank's plan sell $1.2 billion of loans, which concrete and definitive event.
mentions the involvement of local asset managers, giving context the transaction.
The specificity of the information is sufficient generate forecasting questions, such whether the sale will be completed by a certain date if specific asset managers will par- ticipate.
The event has a binary resolution potential, as the sale will either occur or not.
However, the article's content is truncated, but still contains enough information to support actionable predictions.
Therefore, the article meets most criteria for generating forecasting questions.
Articles identified as suitable for forecasting questions are then processed by our Rough Forecasting Question Generator module using gpt-4o. This generator follows structured guidelines (described in 20) to extract clear and unambiguous YES/No questions based solely on the article's s information. Each question consists of clear and precise title that adheres temporal guidelines, ensuring the resolution date aligns with the article's month. The body provides essential context without superfluous details, and the ground- -oound-truth resolution is directly derived from the source article.
Further, we include a pose date (set to October 1st, 2023) in the prompt to ensure temporal clarity. This is only relevant for NewsAPl-based FQs and should not be confused with the created date in Appendix C.I. For example, when an event is referenced as happening in 2024, the pose date prompts the LLM add relevant context, preventing disambiguation issues for forecasters unfamil- iar with the event. The resulting intermediate data structure, containing the question's 's title, body, and resolution, then passed the Final Forecasting Question Generator for further refinement.
46

---

Published as a conference paper at ICLR 2025
Example Rough FQ Data
Article Title: Death toll is in listeria outhreak tied Boar's Head deli meat, CDC says
Article Description: It's the largest listeria outbreak since 2011. I. On July 29, the recall was ex- panded to include all foods produced at the firm's 's plant in Jarratt, Virginia.
Article Content: least eight people have died after being infected with listeria Boar's Head deli meats tied to a massive recall last month, federal health officials said Wednesday. The new food poisoning to... [+7300 chars]
Article URL: attps: /twpww/pww///s//eeeee/ cwn
recall-
Article Published At: 2024-08-87211115000
Forecasting Question Title: Will the listeria outhreak tied to Boar's Head deli meat result more than confirmed deaths by August 2024?
Forecasting Question Body:
This question resolves as YES if, by August 31, 2024, there official reports confirming than deaths attributed to the listeria outbreak linked to Boar's Head deli meats.
Official confirmation must come from eredible sourees such as the CDC equivalent health authorities, and reported by at least reputable outlets.
the death toll remains 5 fewer, the question resolves NO.
Forecasting Question Resolution: true
Our experiments indicate that claude 3.5 5- sonnet produces better-phrased questions than gpt-4o; however, it occasionally generates hallucinated content and introduces fabricated details not found in the original article. To leverage Claude's strengths in phrasing while addressing this concern, incorporate validation prompt into the Final Forecasting Question Generalor process. This prompt (21) assesses the intermediate (rough) forecasting questions on multiple criteria, en- suring clarity and removing elements that suggest direct derivation from news article, including the article's 's publication date. After validating these questions, we rephrase them to minimize overly specific details, thereby enhancing their generality and facilitating their predictability.
The Final Forecasting Question Generator subsequently validates the resolutions of the rephrased forecasting questions (using 22). This process involves prompting -pt-4o evaluate the gener- ated questions against their respective source news articles. The LLM determines whether binary resolution applicable if the question cannot be answered based on the information provided in the article. This approach effectively filters out questions that do not derive directly from the news articles and imposes the necessary constraints clarity and specificity. By focusing solely on the factual content available at the time of publication, the generator ensures that the resolutions are both definitive and accurate. We then verify the NewsAPP-genneated FQs with common FQ verification step ensure correct structure and format.
We generate a dataset forecasting questions using NewsAPI articles published between July 1, 2024, and August 31, 2024, inclusive, described in the above pipeline.
47

---

Published as a conference paper at ICLR 2025
Example Final FQ
ID: 4337/007---22--422--91----- |aaff-a
Title: Will Hawaii enact legislation restricting the public carrying of non-freee non-firearm weapons by Au- gust 2024?
This question will resolve as YES if, by August 31, 2024, Hawaii officially passes enacts legislation that imposes new restrictions on the public carrying of non-firearm weapons, such as bladed weapons or other non non-firean firearm implements previously affected by the recent legal change. The legislation must specifically address the carrying of these weapons in public spaces. For YES resolution, the new must officially enacted and reported by least two reputable news sources (e.g.. Associated Press, Reuters, local Hawaiian news outlets). If no such legislation is passed and enacted by the specified date, or if any enacted legislation does not specifically restrict the public carrying of non-firearm weapons, the question will resolve as NO.
Body:
Resolution Date: 2024:09-31
Question Type: binary
Data Source: synthetic
Created Date: 2024-06-30123:59:5
URL: None
Metadata:
Article Information:
Article URL: httpp://aaaeee.cc////rttttaaaaaa
weapons- senons--eccod- -tt.oo/ceeeetteeeeeeeeeeeaaaaaaa
Article Date: 2024-08-28 10:46:38
Article Description: Second Amendment activists in Hawaii are celebrating a recent legal change that allows them to carry not just guns but other weapons from battle-axes to butterfly knives openly in public. Hawaii has long had strict weapons laws and some of the lowest rate...
Article Title: Bikinis, surfboards and battle-axes? Hawaii loosens long-strict weapons
laws after court ruling...
Article Content: HONOLULU (AP) Hawaii's tourist hotspot of Waikiki is known for bikinis. shopping and surfboards. But resident Andrew Roberts has recently introduced different item evening walks through his neighborhood... [+5086 chars]
Pose Date: 2023- 2023-10-01 00:00:00
Resolution: false
K.2 GENERATING DIVERSE FQS THROUGH REFERENCE CLASS SPANNING
critical issue in forecasting inquiries is the inherent bias towards current phenomena, which results in an overrepresentation of outcomes associated with actively reported events. For instance, if forecasting question posits whether Colorado will conduct referendum on abortion rights by July 2024 and the answer resolves as Yes due to media coverage, this introduces distortion within the dataset. Similar inquiries-sun as whether Nevada will pursue a comparable referendum or whether Colorado will address unrelated topics like gaming regulation-man be inadequately represented or entirely omitted, thus perpetuating a bias towards current phenomena. This imbalance prevents us from effectively testing forecasters' ability to predict a wider array of potential scenarios, limiting the evaluation to outcomes associated with current events and reported phenomena.
To mitigate this bias, advocate for the implementation of the Reference Class Spanner methodol- ogy, which utilizes g to systematically create a set additional forecasting questions within the reference class by modifying essential entities components (prompted with 23). This approach ensures that the dataset reflects a more extensive spectrum of outcomes rather than being disproportionately skewed towards events reported as occurring.
The Reference Class Spanner method generates new forecasting questions by varying one to two core components of the original question while preserving its resolution date and thematic struc- ture, thereby facilitating broader scenario exploration. For example, it transforms the question "Will
48

---

Published as a conference paper at ICLR 2025
Tesla complete major software upgrade for over 1.5 million vehicles in China by August 2024?" into "Will Ford complete a major software upgrade for over 1.5 million vehicles in the states by Au- gust 2024?" This approach promotes diversity in potential outcomes and significantly mitigates bias toward positive outcomes by producing a set of high-quality forecasting questions within the same reference class. By prompting the LLM to change multiple key components simultaneously- as the company name or location-we ensure that the questions generated remain plausible and relevant. We verify the structure of the generated questions and subsequently input them into our Perplexity Verification Module to attach ground truth resolutions.
Table 8: NewsAPI Generated FQs. Represents the number of data points generated until crcation of reference spanned FQs using K.2.
Data
July 2024
August 2024
Total
Initial News Articles
533
486
1019
Validated News Articles
381
363
744
Rough FQ Data
457
375
832
Final Validated FQs
117
104
221
Reference Spanned FQs
2517
2246
4763
49

---

Published as a conference paper at ICLR 2025
Examples of reference spanned FQs
Original Question
ID:
Title:
Will India report a successful winter crop season for wheat and rapeseed by August
2024?
Body: This question will resolve YES if, by August 31, 2024, India reports successful winter crop season for wheat and rapesced, characterized by yields meeting exceeding the average the past five years. The success must be confirmed by official agricultural statistics from the Indian Ministry of Agriculture and Farmers' Welfare or at least three reputable news sources (such as Reuters, Bloomberg, or The Economic Times). For this question, 'successful" defined the combined production of wheat and rapeseed being at least 5% above the five- year average. the winter crop season does not meet these criteria, or if insufficient data available to make a determination, the question resolves as NO.
Spanned Questions
Spanned Question
ID:
Title: Will Pakistan report successful winter crop season for wheat and sugarcane by
August 2024?
Body: This question resolve as YES if. by August 31, 2024, Pakistan reports a suc- cessful winter crop season for wheat and sugarcane, characterized by yields meeting or exceeding the average of the past five years. The success must be confirmed by official agricultural statisties from the Pakistan Ministry of National Food Security & Research least three reputable news sources (such Reuters, Bloomberg, or The Economic Times). For this question, 'successful" is defined as the combined production wheat and sugarcane being at least 5% above the five-year average. If the winter crop season does not meet these criteria, or if insufficient data is available to make determination, the question resolves NO.
Spanned Question 2
ID: 426713c2-ecca-4208 1810888766-aaa 776d-af0b388dbb66
Title: Will Turkey report a successful winter crop season for wheat hazelnuts by Au-
gust 2024?
Body: This question will resolve YES if, by August 31, 2024....
Spanned Question 3
ID: 1:014403--062-44--aaaaa--------1
Title: Will Iran report a successful winter crop season for wheat and pistachios by August 2024?
Body: This question will resolve YES if, by August 31, 2024....
K.3 VERIFYING THE FQ RESOLUTIONS USING PERPLEXITY-BASED QUESTION RESOLVER
To ensure high-quality benchmark, we verify or attach resolutions to every forecasting ques- tion generated in the previous stages. This verification process uses the Perplexity API querying models with internet access to deter-
(llama-3. 1atraa-..-soar---aa -sonar-huge--a be resolved with current information. If the question is resolvable,
mine if the question obtain and attach the resolution. In cases where Perplexity cannot resolve the question, or if the resolution differs from the one originally derived from the source article, we discard that question.
For questions formed through reference class spanning, we directly attach the resolution obtained from Perplexity. For those generated from news articles, we focus on verifying the accuracy of the initial resolutions to ensure consistency and reliability in our dataset. As of the creation of the NewsAPI FQ dataset up until K.2, Perplexity maintains an accuracy of over 95%, with half of the discrepancies arising due to contradictory internet data (which makes the resolution unclear even to the authors). Due to the potential of such label noise, we adopt the Brier score instead of the log scoring rule for all ground truth metrics.
We create a ground-truth resolved dataset 00w.:000..2200011--------- www.oonnndrrsooeedd.. jsonl) comprising of 2621 forecasting questions which is used for tuple instantiation. Further, we filter out 1000 questions (20240701.2024083) json1) from this set, consisting of all of the NewsAPI
50

---

Published as a conference paper at ICLR 2025
Table 9: Question Verification and Resolution Data for July and August 2024. resolved questions is lower than the combined totals for both months, questions with existing resolutions that differ from those suggested by Perplexity are discarded.
Notably, the final count of
Data
July 2024
August 2024
Total
Total Questions Generated
2517
2246
4763
Filtered for Verification
2516
2246
4762
Questions Discarded After Perplexity
1005
1090
2095
Resolved with Final Resolution Attached
1511
11566
2667
Final Total Questions Resolved
2621
generated FQs and subset of the reference-spannee questions, to use as a ground-truth dataset in our experiments.
51

---

Published as a conference paper at ICLR 2025
News Article Validation Prompt
System Prompt:
You Al agent responsible for evaluating news articles determine their suitability for generating forecasting (prediction) questions can answered with definitive YES or NO. Assess each article against the following criteria to ensure clarity, relevance, and factual accuracy:
Clarity of Content: the information presented clearly and straightforwarddyy? Reject articles that are overly convoluted or difficult to understand.
Focus on Definitive Events: Does the article discuss concrete events that have occurred or are planned? Evaluate articles referencing past events based on their clarity and context.
Contextual Relevance: Does the article provide adequate context for events discussed? While some background gaps are acceptable, the article should allow for a reasonable understanding of the events.
Specificity of Information: Is the information detailed enough to formulate precise forccasting questions? Reject articles that are vague support clear predictions. Binary Resolution Potential: Does the article imply resolution that can be confirmed TRUE (YES) or FALSE (NO)? Articles may contain subjective elements but should lead to binary outcome.
Completeness of Information: Does the article provide sufficient detail to create multiple high-quality forecasting questions? Brief articles are acceptable as long as they contain enough information.
Numerical Clarity: applicable, does article present clear thresholds or metrics for numerical data? Some ambiguity is acceptable, but numerical references should be understandable.
Sufficiency for Definitive Resolution:
Does the article provide enough information to formulate forecasting questions that yield definitive resolutions from the current date until the specified resolution date {month name), {year]? ? Ensure the content supports actionable predictions based on concrete events, assuming the current date is {pose date }-
Truncated Information: Truncated information NOT cause rejection. Accept articles can form prediction questions, even they reference past events not covered by the I.L''s knowledge.
An article that meets most of these criteria considered "complete" and suitable for generating forecasting questions, even it contains minor ambiguities or references past events that may not be fully known.
User Prompt:
Please evaluate the following news article based on the established criteria for completeness: (source news article)
Based on your assessment, determine the article "complete" and suitable generating forecasting questions. Provide a brief justification for your decision.
Validation prompt used to judge whether processed article be used forecasting question with binary resolution.
Figure 19:
52

---

Published as a conference paper at ICLR 2025
Rough FQ Generation Prompt
System Prompt:
Objective: Generate forecasting questions that be definitively answered with YES NO, based provided news articles, while testing forecaster set the past.
Forecaster's Context: The forecaster's present date is set to [pose date), so all questions must be framed if this is the current date. Although the articles may reference future events, your questions must be phrased in way that forecaster cannot detect the actual date question creation.
Clarity & Precision:
Each question must be clear, specific, and unambiguous. Avoid subjective terms "significant" or any similar ambiguity.
Do reference sensitive topics such as religion, politics, gender.
No Temporal Hints:
Do include any information or context that implies the question was created after {pose_date)
Ensure no indication that the article used to inform the question, keeping creation date fully hidden.
Resolution Period:
If you phrase the resolution date as "by question must remain definitive and applicable from the current date until [month_name {year).
[year}" then resolution each
If you phrase the resolution date "in (month name). (year)", ", then resolution of each question must remain definitive and applicable for the month of {month in (year).
Fnsure the question's 's outcome is verifiable and binary (YES or NO) during this period.
Context from Articles:
Use concrete events from the articles, providing cnough background to make the question understandable.
Ensure questions diverse, covering wide range of topics without bias triviality.
Goal: Generate diverse set of precise and objective forecasting questions that seamlessly align with the forecaster's assumed timeline without revealing the true creation date or source of the information.
Figure 20:
Prompt used to generate intermediate (rough) forecasting question consisting of just the title, body and resolution from article. Continued the next page.
53

---

Published as a conference paper at ICLR 2025
Rough FQ Generation Prompt (Continued)
User Prompt:
Task: Based the provided news article, generate multiple high quality forecasting questions that follow these structured guidelines. Each question must consist of title, body, and resolution. The generated forecasting questions must only be formed using information from the article and no other extrapolations or inferred information.
News Article: (source.article)
Title Guidelines:
YES/NO Clarity: Formulate each question that can be definitively answered with YES or NO, based on the article's 's content.
Avoid Sensitive Topics: Do not reference religion, polities, gender, race.
Direct and Precise: Titles must be straightforward and unambiguous, avoiding vague
terms.
Resolution Date: Include a resolution date using the format "by mnme), {year)?"
or "in (month.name). {year) ?", whichever is more suitable for the context.
Context for Clarity: Provide enough context if event names may not be clear as of forecaster's & present date (pose date)).
Named Entities: There is no limit on the number named entities from the article, but the question should avoid becoming overly specific.
Planned or Announced Events: Frame planned events as proposals or announcements
rather than completed facts, including sufficient context avoid ambiguity.
Body Guidelines:
Disambiguation: Stay focused the title's core question without introducing unrelated details.
No Extra Information: Only include relevant context to support the title.
Resolution Guidelines:
Binary Outcome: Resolutions must be clearly marked True for YES and False for NO.
Stable Outcome: Ensure the resolution remains consistent and unchangeable until the resolution date.
Definitiveness: The resolution must be verifiable based solely on the content of the article.
General Guidelines:
Avoid Specific Knowledge: Do not require specialized knowledge that could disadvantage forccasters unfamiliar with niche topics.
Base Questions Article Content: Ensure all forecasting questions directly derived from the article's ' content, avoiding speculative inferred details.
Examples included in the prompt have been skipped for brevity.
54

---

Published as a conference paper at ICLR 2025
Final FQ Validation and Rephrasing Prompt
System Prompt:
You expert in validating and rephrasing forecasting (prediction) questions based articles. forecasting question consists of title, body, and resolution.
Your task is to ensure that each question adheres to the established guidelines and to enhance formulating these forecaster assume they are answering them as of ipose_date ). The resolution date for the questions should be set as (month Jtame) ), {year).
the phrasing of valid questions. It important to that while questions after knowing the resolutions,
Guidelines to be followed are:
Forecaster's Context:
The forecaster's 's present date is set to {pose.date) so all questions must framed as if this the current date. Although the articles may reference future events, your questions must be phrased in a way that the forecaster cannot detect the actual date of question creation.
2. Clarity & Precision:
Each question must be clear, specific, and unambiguous.
Avoid subjective terms like "significant" or any similar ambiguity.
Do not reference sensitive topics from religion, politics, gender.
3. No Temporal Hints:
Do not include any information or context that implies the question was created after 1pose_date)
Ensure no indication that article is used to inform the question, keeping creation date fully hidden.
Resolution Period:
If you phrase the resolution date "by (month (year)", resolution each question must remain definitive and applicable from the current date until (month_name J. {year).
If you phrase the resolution date "in {month.name), {year)". then resolution each question must remain definitive and applicable the month (month_nand _aaee) in {year).
Ensure question's outcome verifiable binary (YES or NO) during this period.
Factual Basis:
Questions should be directly supported by the article content and not include fabricated information.
User Prompt:
You are tasked with the following steps:
1. Validation:
Check if the forecasting question adheres to the provided guidelines. A question is valid if it aligns with the guidelines.
2. Rejection:
Reject the question if it violates any guidelines. The rejected form should be: (example rejecppeerejeeeedffii
3. Rephrasing:
For valid questions, rephrase them to enhance clarity, specificity, and compliance with the guidelines while retaining the original intent. Do NOT add any new information that wasn't included in the original question.
HlihhQuality Forecasting Question Examples: Some Examples
Task:
-
Carefully validate and rephrase the following forecasting question: (source rough f4 data).
Figure 21: Prompt used validate the structure of NewsAPI generated forecasting questions and then rephrase them to enhance predictability.
55

---

Published as a conference paper at ICLR 2025
Final FQ Resolution Validation Prompt
System Prompt:
You Al agent tasked with verifying the resolution of forecasting questions based solely on the content of provided news article. Your crucial in ensuring that resolutions are definitive and accurately reflect the information available at the time the question was posed.
Factual Basis: The resolution should be based the factual information present in the news article.
Publication Perspective: Your assessment should he made from the perspective of the article's publication date, not any other date.
Inference Guidelines: Reasonable inferences are acceptable, speculate beyond what stated the article.
not fabricate details
Response Options: Use the 'None' option there absolutely no information article that allows for a reasonable inference either YES or NO. If the article provides any relevant context or information that can lead to definitive answer, choose either "True' 'False'.
User Prompt:
Consider the following news article:
Title: {article
Description: {article_descrpppll description)
Content: (article_conter
Date: farticle date)
Now, consider this forecasting question: (guestion title)
For additional context, the following information to disambiguate the question:
(question.
Your task is to determine the resolution of the question based solely on the factual information present in the news article, assuming the article's 's publication date the current date. Return:
'True' if the answer to the question can be reasonably inferred as YES.
'False" if the answer to question can reasonably inferred as NO.
'None' if there is absolutely no information the article that allows for reasonable
inference of either YES NO.
Please provide brief justification for your citing specific details from the article that support your reasoning.
Figure 22: Prompt used to verify whether forecasting question formed using NewsAPI has the correct reso- lution using the source article.
56

---

Published as a conference paper at ICLR 2025
Forecasting Question Generation Prompt
System Prompt:
Objective: Generate high-quality forecasting questions (FQs) by spanning the reference class of given source question. Your goal enhance diversity of dataset while minimizing bias.
probability theory. reference class refers group of similar events outcomes that share common features. Your task is to create new forecasting questions by varying key components (c.g., location, topic, action, or subject) of the source question, ensuring they stay within the same reference class.
Reference Class:
Key Requirements:
Consistency in structure and thematic integrity with the original source question.
Vary only one to two key elements while ensuring logical consistency. The new questions should remain unresolved.
Use the same resolution date the source question.
Question Structure:
YES/NO clarity, avoid sensitive topics, direct precise titles.
Context for clarity with clear binary outcome for resolutions.
Retain the resolution date the source forecasting question.
User Prompt:
The source forecasting question is: fooocsteoorecaatiiggggquu
Instructions:
Identify the components (event type, location, key subjects, outcomes) of the source question.
Replace significant elements with similar entity while maintaining logical structure.
Ensure balance neutrality, with diverse probability distribution possible outcomes.
Verify that the new questions remain realistic, relevant, and unresolved as now.
Create (num questions} forecasting questions by spanning the reference class of the provided source question.
Figure 23: Prompt used to generate high-quality forecasting questions by varying key elements of source question using reference class spanning. Examples have been omitted for brevity.
57

---

Published as a conference paper at ICLR 2025
L 2028 SYNTHETIC QUESTIONS CONSISTENCY CHECK DATASET
This section presents a set of questions with a resolution date 2028. These questions were created using a prompt similar the in Figure 14, with two key additions:
1. Target Resolution Date:
The prompt specifies target resolution date, in this case January 1, 2028. And asks the model to propose questions about events happening before the resolution date, or in the year of the resolution date. About half of the initial few shot examples are modified with the chosen resolution date.
2. Creation Date:
The prompt includes a creation date, in this case October 1, 2024. This is crucial prevent the generation of questions that could be trivially answered on the creation date, but are in the future from the perspective of the model knowledge cutoff.
Below are two example questions from this dataset:
Examples of Synthetic Questions with 2028 Resolution
Synthetic Question 1
2f2c7e08 5241-40ba 8ad1 -5037408388c
Title: Will Australia's GDP grow by least 3% annually for three consecutive years before January 2028?
Body: This question will be resolved as 'Yes' Australia's s GDP, reported by the Australian Bureau of Statisties, grows by at least 3% anmually for three consecutive years at any point between October 1, 2024, and January I, 2028. The growth rate must be based on official annual GDP growth figures released by the Australian Bureau of Statistics.
Additional Details:
Question Type: Binary
Resolution Date: 2028-01-01 00:00:00
Created Date: 2024-10-01 00:00:00
Data Source: Synthetic
Category: Economy & Business
Tags: Australia
Synthetic Question 2
ID: D
Title: Will Switzerland hold national referendum on joining the European Union before January 1, 2028?
Body: This question will be resolved as 'Yes' if. between the creation date of this question (October 1, 2024) and January 1, 2028, Switzerland holds national referendum on the issue of joining the European Union. The referendum must be officially sanctioned the Swiss government and the results must be publicly announced.
Additional Details:
Question Type: Binary
Resolution Date: 2028-01-01 00:00:00
Created Date: 2024-10-01 00:00:00
Data Source: Synthetic
Category: Geopolitics
Tags: Switzerland
We create 900 verified (see verification paragraph in Section 3.1) base forecasting questions resolv- ing in 2028. From these, we run the consistency check instantiation pipeline in Section 3.2, to create 300 tuples per check, for total of 3000 tuples. We then run a single forecaster on this benchmark to get sense of baseline performance on our dataset.
The consistency metrics on this dataset provide best proxy available comparing general long- term forecasting ability of LLM forecasters, but many caveats apply.
Future work may consider creating a similar benchmark with a secret subset, to prevent new fore- casters from being trained to cheat this benchmark. Note that, due to the lack of ground truth
58

---

Published as a conference paper at ICLR 2025
Table 10: Consistency metrics for CoT-GPT-40-08 the synthetic 2028 questions dataset.
Arbitrage
Arbitrage Scaled
Frequentist
Check
Avg
Frac
Avg
Frac
Avg
Frac
NEGATION
0.033
49%
0.016
49%
0.178
50%
PARAPHRASE
0.014
37%
0.007
37%
0.107
38%
CONDCOND
0.044
65%
0.011
54%
0.296
89%
EXPÉVIDENCE
0.031
35%
0.008
23%
0.186
50%
CONSEQURNCE
0.003
7%
0.001
7%
0.021
8%
AND
0.020
0.007
18%
0.080
25%
OR
0.016
36%
0.006
24%
0.105
37%
ANDOR
0.034
46%
0.008
36%
0.190
49%
BUT
0,050
58%
0.017
54%
0.317
81%
COND
0.042
66%
0.014
60%
0.242
71%
Aggregated
0.029
0.010
0.172
resolutions, accidental training on the dataset does not automatically invalidate any consistency met- ric, as opposed to what happens with standard benchmarks.
REPRODUCIBILITY STATEMENT
We include the questions, forecasting results, and consistency results necessary reproduce all tables and plots in the paper. The forecasts data 10 is organized by forecaster, with two directories for each forecaster:
Ground truth forecasting results:
JSONL file, where each entry has (question, boolean resolution, forecast, per-question
Brier score, metadata and reasoning traces)
JSON file: total Brier score, calibration, other metries
2. Consistency checks results:
JSONL file with raw results, where each entry (questions, forecasts, consistency vio- lations, metadata and reasoning traces)
JSON file:
summary statistics (c.g., average violation)
The consistency check results directories have a substring tuples in the directory name. For the 2028 synthetic dataset, have only the consistency check result directories.
/ tttp///ttthhttoo// http:////ttt..o.o//o///oo///oooa
b093e5 1 F4 a
59


Reviews:
[{"review_id": "ZMc0SWoKDK", "reviewer": "Reviewer_7grX", "summary": "The authors expand on forecasting consistency checks pioneered as part of Fluri et al. 2024 (Evaluating Superhuman Models with\nConsistency Checks). While that paper presented a proof-of-concept limited set of consistency checks on a single model, this work concretizes these checks into a set of usable metrics and demonstrates some of these metrics' high correlation with forecasting performance (as measured by Brier score).\n\nThese metrics include an arbitrage metric, incurring penalties according to how much guaranteed profit an arbitrageur can receive by exploiting the inconsistencies in the probabilities if they were market prices, and a frequentist metric, which uses a hypothesis testing setup to test whether a given forecaster could be sampling from a distribution based on adding noise to a consistent world model. The authors conduct experiments using a new already-resolved test dataset, and release another long-term forecasting dataset to resolve in several years (2028).", "questions": "1. Can you report confidence interval for the Brier score of a new model based on its consistency scores (so long as it is not Goodharted on consistency?) - e.g. with Fisher's z transformation?\n2. Can one report, in addition to the correlation between Brier and each consistency metric, a ranking metric on the models' Brier performance based on the consistency ranking? It seems that selecting the best model/few models based on consistency scores is a critical use case.\n3. Is there any way to ensure contamination-free or low-contamination consistency checking? Can we design an arbitrary search space for consistency checks that is large enough to not be saturated trivially by consistency training (see weakness 2).", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Strong communication of the problem and proposed resolution\n- Detailed data generation for both already-resolved experiments and an open-ended future forecasting benchmark\n- Extensive discussion of decisions & limitations\n- Principled theory and evaluation.\n- Correlation plots indicate that the consistency scoring criteria could be used to generally select better forecasters.\n- Theoretical and practical method to improve consistency at evaluation time.", "weaknesses": "1. Test-time ArbitrageForecaster does not gain consistency in an emergent set of checks; only those it is specifically designed to address. This reduces the impact of the paper slightly.\n2. New consistency metrics are rather easily Goodharted; one can imagine training a model on a vast set of synthetic forecasting problems respecting consistency checks in order to gain nearly perfect consistency scores with no improved knowledge of the future. This would limit the long-term efficacy of these metrics as a way to distinguish quality forecasters, unless novel checks are often added (which seems increasingly difficult without dramatically increasing the complexity of new checks).\n3. No comparison with system from Halawi et al. 2024: \"Unfortunately, we could not reproduce the system from Halawi et al. (2024) at the time of writing, due to deprecations in the Google News API (we could not obtain access to the alternative Newscatcher API).\" It is unclear if this is addressable in the longer term. It seems this comparison would be useful.\n\nNitpicks (grammar etc.):\n\n- 2536 \"with over the discrepancies\"\n- 2924 \" are modifies \"\n- 184 May want to use P, R here for the input set of questions to distinguish from the output tuple P, Q and the input set of questions P, Q.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2tIpBeAPGU", "reviewer": "Reviewer_N1zy", "summary": "The paper introduces a novel approach to evaluating language model (LLM) forecasters, given the challenge of evaluating predictions with unknown future outcomes. The authors propose a consistency check framework that uses arbitrage and hypothesis testing as a metric, measuring logical inconsistencies across different, related forecasting questions. They build an automated system for generating forecasting questions, applying consistency metrics, and demonstrate that consistency scores correlate well with forecasting accuracy as measured by Brier scores.", "questions": "* I wonder what's the performance for o1-mini for the negation and paraphrase experiment in Figure 3? Or is there any reason why the authors decided to evalute GPT4o instead of o1-mini?\n* Given that the long-horizon benchmark evaluates consistency without ground truth, how do authors see the questions without ground truth being indicative of forecasting ability? Are there specific applications where such consistency checks alone could be considered sufficient?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The manuscript is well-written, and the ideas are easy to follow.\n* The paper effectively addresses the challenge of evaluating LLM forecasters by proposing new metrics based on arbitrage and hypothesis testing.\n* It provides a thorough derivation and demonstrates a statistically significant correlation between the metrics proposed and Brier score.\n* The authors also curated a dataset of forecasting questions, enhancing the evaluation for the future works.", "weaknesses": "* The ArbitrageForecaster section would benefit from additional practical details, including parameter tuning, framework specifications, and methods for applying adjustments across various LLM models. Adding a framework visualization could further enhance clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4F9K0nXrJA", "reviewer": "Reviewer_YSMD", "summary": "This paper studies forecasting binary events using LLMs. They make numerous contributions involving consistency violations of binary forecasts. They introduce consistency metrics, propose an evaluation pipeline including the creation of two forecasting datasets, analyze correlation between consistency violations and the quality of the forecasts as gauged by the Brier scoring rule, a proposed forecaster to reduce violations, and a new benchmark where events are resolved in 2028.", "questions": "Here are some further questions and suggestions:\n\nHow large are the datasets created? Could the authors elaborate? I probably missed this information.\n\nHow can one be assured that the data contamination issue is not a problem for forecasting? I understand the authors have mentioned that they try to build a dataset that avoids this issue through time stamps – could they confirm if they are sure that there is no data contamination, in that the training data does not directly include information about the forecasted events? \n\nI noted several typos and related issues in the manuscript. Here are a few (most after resolution):\n\nLine 82: “analyses can extend to …”\n\nLine 107: should Forecast be in [0,1] rather than equal to the range? I’m not sure about this one.\n\nLine 161 (footnote 1): “Manifold Markets use …”\n\nLine 148-152: This choice seems arbitrary and merely pointing to an example with the sum of probabilities being 1.1 is not helping.\n\nLine 169: “individual step …”\n\nLine 242: what is “sensical”? Do you mean “sensible”?\n\nFigure 1: The description mentions 3 metrics. Should there be 3 panels? Also, there is a space before a period in the caption.\n\nFigure 2: What is COND? What is CONDCOND? It seems like some information is missing.\n\nLine 324: “a proxy …”\n\nNote that there are some citation style issues in Section 6.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper explores the important subject of using LLMs to forecast future events. Clearly a lot of effort has gone into the work, from the data generation aspects formalizing and running the evaluation pipeline. \n\nAnother strength of the paper, at least from what I understand, is that the authors have tried to avoid data contamination issues that are prevalent in the literature involving the use of LLMs. They have also created a new benchmark for events resolving in 2028, which could be beneficial to the community.", "weaknesses": "I’m a bit confused by the focus of the study itself, which is to use logical consistency to evaluate a forecaster before resolution of the events themselves; presumably, the point of this is to evaluate forecasts sooner. But what is the value of using consistency as an early marker? There are likely confounding factors at play here that determine a model's affect on both forecasting consistency and the performance on the task. For instance, maybe a certain class of models (perhaps larger or trained in certain ways) are better forecasters – and many of these features are observable anyway, so why not just use these instead of consistency? I don’t understand why the authors have gone down this path and would like to know more about what motivated this line of work. Also, the observed correlation in the two instances doesn’t provide too much evidence for the claims, in my view, so I wonder about the generalizability of the claims. Frankly, I do not fully see the merits yet but I’m open to changing my mind. \n\nIn my opinion, way too much (important) information is in the Appendix. I appreciate the space limitations, but I feel a lot of important material is missing from the main text, such as details about the frequentist metrics and details about the datasets. I did not get a chance to look closely at various parts of the Appendix.\n\nThe metrics themselves seem overly and possibly unnecessarily complicated. The rationale for these were also not provided clearly. What about something simpler, such as distances between scores that involve S? Perhaps I’m not understanding the complexity of scoring when arbitrary predicates are used for consistency.\n\nThe literature review seems too sparse. For instance, there are several papers on consistency in general, and probably many others on forecasting with LLMs. I will not point them out specifically but it is relatively easy to find more work. Perhaps the authors can consider expanding the related work section.\n\nSection 5 came across as somewhat rushed, perhaps as it is somewhat preliminary and the results are not too promising, from what I understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HYjCUQMQpw", "reviewer": "Reviewer_QioK", "summary": "The paper proposes to evaluate large language model (LLM) forecasters by assessing the consistency of their predictions through a \"consistency check\", since ground truth is often unavailable about the future – we have to live and see!). Thus, the paper proposes an arbitrage-based and a frequentist metrics to detect logical inconsistencies. It finds good correlations between these consistency metrics and forecasting performance metrics (such as Brier score) tested on events that have already happened from a datasets established by this paper (one from prediction markets and one synthetically generated from news articles).", "questions": "Since the correlation between consistency metrics and forecasting performance metrics is conducted on events that have already concluded, is it not possible that the LLMs already know that, thus they are more accurate and more consistent in their predictions? I would assume that you chose events that resolved “between May 1, 2024, and August 15, 2024.” (L203), because that would be after the knowledge cutoff, right. But are we absolutely 100 percent sure that we know the actual knowledge cutoff of these models (since OpenAI loves switching things behind the scenes in the API without letting users know?)\n\nL134, the phrase \"We set V(F(x1), . . . F(xn)) to be the maximum such 'minimum profit'\" could benefit from a rephrasing – \"We define V(F(x1), . . . F(xn)) as the maximum achievable minimum profit from the arbitrageur…\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "TLDR: This paper takes on a massive challenge and delivers as much as imaginable in such a hard setting.\n\n**S1. Novel evaluation approach with a lot of effort and care to do things right:** The authors cleverly circumvent the need for ground truth in long-term forecasts by developing consistency metrics, validated by correlation measures and by carefully gathering datasets with events that resolved “between May 1, 2024, and August 15, 2024.”, so after the knowledge cutoff of the models. The consistency checks are clever and make predictions more interpretable, since they appeal to logic. Also, the paper includes many and diverse LLMs into testing. Well done!\n\n**S2. Useful artefacts:** The paper creates a long-horizon consistency benchmark for forecasts resolving in 2028. This adds a massive a practical tool for future benchmarking, especially of more advanced LLMs that are yet to come.\n\n**S3. Lots of details for reproducibility:** The paper provides a solid appendix with comprehensive documentation, including forecasting question structures, pipeline details, and explicit consistency conditions. This enhances reproducibility, makes the paper experimental design and choices easy to follow, and can make the potential applicability across related forecasting tasks real, by enabling other researchers to implement this on their own domain.\n\n**S4. The paper’s writing is excellent.** Otherworldly.", "weaknesses": "**Minor:** Limited breadth of data domains: Dependency on NewsAPI and Manifold and Metaculus for some forecast questions, while practical, may limit the breadth of testing topics and introduce data biases. But this is a minor issue, since the challenge taken on by this paper is massive, and the experiments were conducted with a lot of care, and the methodology presented in this paper can be extended to as many domains as possible (S3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)