LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena
Authors: Qingchuan Yang
Simon Mahns
Sida Li
Anri Gu
Jibang Wu
Haifeng Xu
Venue: ICLR 2026 Poster (2026)
Keywords: Forecasting
LLM Benchmark
LLM-as-a-Prophet
LLM Evaluation
Primary area: datasets and benchmarks
TL;DR: We introduce Prophet Arena, a benchmark that evaluates LLM forecasting abilities, and systematically analyze strengths and gaps in AI predictive intelligence.

Abstract: With the rapid progress of large language models (LLMs) trained on every available piece of data, it becomes increasingly challenging to reliably evaluate their intelligence due to potential data contamination and benchmark overfitting. To overcome these challenges, we investigate a new angle of benchmarking LLMs' intelligence by evaluating their capability in forecasting real-world  future events, a paradigm we call "LLM-as-a-Prophet". Such forecasting tasks require combination of sophisticated capabilities while remaining free from data contamination or overfitting. To systematically evaluate such predictive intelligence of LLMs, we introduce $\texttt{Prophet Arena}$, a general evaluation benchmark that continuously collects live forecasting tasks and decomposes each task into distinct pipeline stages, supporting our controlled and large-scale experimentation. Our comprehensive evaluation reveals that many LLMs already exhibit impressive forecasting capabilities, reflected in, e.g., their small calibration errors, consistent prediction confidence and promising market returns. However, we also uncover key bottlenecks even in frontier models, such as inaccurate event recalls, misunderstanding of data sources and slower information aggregation compared to markets when resolution nears.

Published as a conference paper at ICLR 2026
UNDERSTANDING PREDICTIVE INTELLIGENCE WITH PROPHET ARENA
*
Qingchuan Yang ! Simon Mahns 2j Sida Li Anri Gu 3 Jibang Wu 4j Haifeng Xu 31
University of Southern California
2 Meta
3 The University of Chicago 4 <New York University
ABSTRACT
With the rapid progress of large language models (LLMs) trained on every avail- able piece of data, it becomes increasingly challenging to reliably evaluate their intelligence due to potential data contamination and benchmark overfitting. To overcome these challenges, we investigate a new angle of benchmarking L.MMs' intelligence by evaluating their capability in forecasting real- world future events, paradigm we call "LLM-as-a-Prophett" Such forecasting tasks require combina- tion of sophisticated capabilities while remaining free from data contamination or overfitting. To systematically evaluate such predictive intelligence of LLMs, we introduce P Prophet Arena, a general evaluation benchmark that continuously collects live forecasting tasks and decomposes each task into distinet pipeline stages, supporting our controlled and large-scale experimentation. Our compre- hensive evaluation reveals that many LLMs already exhibit impressive forecasting capabilities, reflected in, e.g., their small calibration errors, consistent prediction confidence and promising market returns. However, we also uncover key bottle- necks even in frontier models, such as inaccurate event recalls, misunderstanding of data sources and slower information aggregation compared to markets when resolution nears.
INTRODUCTION
Forecasting is fundamental intellectual pursuit that has shaped human progress from the earliest scientific inquiries to modern economics and finance. In machine learning, forecasting has also been central theme, with rich traditions ranging from time-series analysis (Chatfield, 2000) and online learning (Foster & Vohra, 1998) to conformal prediction (Barber et al., 2023). Yet, somewhat surprisingly, the challenge of open-domain forecasting, producing accurate predictions across wide range of topics without domain-speede specific tuning or specialized datasets, remains largely unexplored. Achieving reliable foresight in this setting would represent qualitative leap in Al capability, with far-reaching societal implications, from enhancing market efficiency to guiding high-stakes policy decisions (Arrow al., 2008).
At its core, forecasting is the process of connecting present knowledge to anticipate future outcomes. Large language models (LLMs)) seem natural candidates for this role. Trained on massive corpora of human knowledge through the seemingly narrow objective of next-word prediction, LLMs have developed emergent capabilities that extend far beyond their training objective (Bubeck ct al., 2023). This motivates the prospect that the ability predict the next word may also give rise the ability to predict the next event. Indeed, recent works by Zou et al. (2022); Halawi et al. (2024) have already made promising progress in employing language models for forecasting tasks. We envision that, with growing interest and continued progress, this would position LLMS not only as repositories of human knowledge but also as instruments of reliable foresight, leading the prospect of LLM- as-a as-a-Prophet: Can Al systems reliably predict the future by connecting the dots across existing real-world information?
In this paper, we seek to systematically examine the prospects and challenges of building general- purpose systems for open-domain forecasting. On one hand, forecasting is a natural next pursuit
"Correspondence to contact contact@proppptntocct aproppptareeeeete co.
Part of the work by Qingchuan Yang and Jibang Wu are done while they are at the University of Chicago. We thank Alex Gu, Chaplin Huang and Lucien Liu for help the early stage of this work.
Equal contribution.
I Equal advising.

---

Published as a conference paper at ICLR 2026
given the rapid progress of AI, as it draws on combination of advanced capabilities that cur- rent models are only beginning to demonstrate: information retrieval, complex reasoning and data analysis. Moreover, as many established benchmarks are approaching saturation and are increas- ingly prone to training-data contamination (Deng et al., 2024), open-domain forecasting provides forward-looking and contamination-free setting with objectively measurable outcomes, making it a rigorous testbed for evaluating advanced model intelligence. On the other hand, we observe that current LLMs often struggle with key requirements for reliable foresight, including calibrated un- certainty estimation (Geng et al., 2023) and robust reasoning (Zhou et al., 2024) in the presence of noisy or incomplete evidence. As a result, their forecasting results may at times resemble guess- work rather than deliberated prediction, raising the possibility that fundamental barriers must be addressed before such evaluations can serve as a meaningful benchmark at the present time (Paleka et al., 2025a). Toward this end, we introduce Prophet Arena, general framework for evalu- ating LLMs on live, real-world forecasting questions in a controlled and extensible way. Our goal is not only assess the current forecasting performance of LLMs, but also to use forecasting lens for studying core components of intelligence, including reasoning, calibration, evidence aggre- gation. By doing so, aim to identify which capabilities are emerging, which remain limited, and how forecasting evaluation can guide the development of more reliable predictive intelligence.
Our contributions. First, introduce the paradigm of LLM-as-aa aa-Po----- framing real- world open-domain forecasting as a rigorous and forward-looking measure of intelligence. This perspec- tive highlights forecasting as not only an application domain but also a unifying task that brings together many advanced capabilities, making it natural benchmark for the next phase of Al de- velopment. Second, we present Prophet Arena, a live and extensible benchmark that continu- ously collects real-world forecasting questions across diverse domains. The framework decomposes forecasting into distinct stages for controlled evaluation and incorporates multiple scoring metrics, including statistical accuracy, calibration, and economic value, to provide a comprehensive view of predictive quality. Third, we conduct large-scale experiments on leading LLMs using Prophet Arena. Our evaluation uncovers areas where models demonstrate non-trivial foresight, while also identifying systematic weaknesses particularly in interpretation of data sources, bottlenecks in reasoning traces, and conservativeness in predictions.
1.1 CONNECTION AND COMPARISON TO PREVIOUS WORKS
We briefly highlight our work's connection and comparison to previous works, while defer more in-depth discussions Appendix A.
Understanding and advancing LLMs' forecasting capabilities. A key goal of our work is to understand the
paradigm by measuring LLMs' current forecasting capabilities and analyzing how other capabilities like knowledge internalization and source usage shape predic- tive intelligence. This connects to recent studies examining specific aspects of LLM forecasting, such temporal generalization (Dai et al., 2025; Zhu et al., 2025) and forecast consistency (Paleka et al., 2025b). Like these works, we build a benchmark platform Prophet Arena, but ours is, to our knowledge, the first to diagnose the general predictive intelligence of LLMs. We also note recent progress on improving LLM forecasting capabilities (Zou et al., 2022; Halawi et al., 2024). While focus is benchmarking rather than model design, our insights about LLM strengths and limitations may inform future advances in forecasting.
Forecasting benchmarks. Forecasting has recently become widely used LLM benchmark since it challenges multiple capabilities while avoiding data contamination by testing models on future events (Dai et al., 2025; Karger et al., 2025). To our knowledge, one of the earliest effort is ForecastQA (Jin et al., 2021), which tests AI models on crowdsourced multiple-choiee forecasting questions. Subsequent work created increasingly difficult benchmarks by incorporating prediction- market events (Zou et al., 2022), extracting events from news (Zhang et al., 2024; Wang et al., 2025), curating future-oriented questions from websites (Wildman et al., 2025), using open-ended forecast- ing prompts (Guan et al., 2024), and constructing dynamic, live-updating benchmarks (Zeng et al., 2025; Karger et al., 2025; Bianchi et al., 2025).
We also develop a benchmark, Prophet Arena, which we intentionally ground on thousands of prediction markets events, leveraging their incentive- aliggnd participation and standardized resolu- tion criteria to obtain more rigorous testbed that mitigates selection bias from question design. Beyond ranking models, we are also interested in what conclusions we can draw from the analy- sis of the models' performance. Accordingly, we provide comprehensive evaluation of LLMS'
2

---

Published as a conference paper at ICLR 2026
predictive intelligence, including Brier scores, calibration and economic value and thoroughly discuss how they capture different aspects of a forecast in Section 3.1. In contrast, previous bench- marks mostly focuses on metric, such Brier score (Halawi et al., 2024; Karger et al., 2025), calibration (Zou et al., 2022) and accuracy (Dai et al., 2025; Zhang et al., 2024). Moreover, beyond evaluation metrics, we analyze how specific LLM capabilities (e.g., internalization, source usage) shape forecasting behavior.
2 PROPHET ARENA: A LIVE BENCHMARK FOR PREDICTIVE INTELLIGENCE
Prophet Arena is implemented a live, continuously updated pipeline for evaluating forecasts on real-world events. As illustrated in Fig. 1, it consists of three main stages, which together form an end-to-end workflow for assessing predictive intelligence at scale. Below we describe the design of each stage at a high level and defer the implementation details to Appendix B.
(1) Event/Norket Extraction
(2) Prediction Context
(3) Probabilistic Prediction Evaluation
Construction
Search AP1/Agent
Prediction Rodela
(L1Ms)
Zecaardin
Tell Use X
Pradtction Borkets
Neteral Ievalinge
Mari d. Souries
Human Insightd
aill
49% Janni Siener 52% Carlos Alcorns
Coopla 2i17
Iteedh Staner
Chers
Evaluction banking
The seratis fook Releed ullde nekat side ou 2our for
Aosunney Callentier
E
Corles Alceraz
Proept
Figure 1: Flowchart for the full Prophet Arena pipeline.
(1) Event and Market Extraction. Prophet Arena collects unresolved events from Kalshi, a live prediction platform with events spanning across categories such as finance, sports, politics and entertainment. Kalshi provides structured, real-time events with verifiable outcomes, enabling standardized and comparable evaluation across models. To ensure informativeness and compara- bility, we filter by Popularity (volmm//iquddit//ooaliity rence (repeated formats). Prophet Arena periodically retrieves a number of unresolved events, making it truly live benchmark, rather than a fixed, static dataset susceptible to training data con- tamination. The use of prediction markets guarantees that all questions pertain to genuine future outcomes grounded on real world importance and that their resolutions can be objectively verified once finalized.
Diversity (domain balance), and Recur-
(2) Prediction Context Construction. For each event, Prophet Arena constructs a unified prediction context that all models receive identically. This context contains: (1) Relevant informa- tion sources retrieved by an LLM-based search agent using web queries that collect titles, snippets, timestamps, and URLs of recent news reports; and (2) Market snapshots including the latest Yes/No contract prices and trading volumes, from which implied probabilities (based on market contract prices) derived. Providing identical contexts isolates differences in models reasoning and calibration rather than their retrieval capabilities. The search component in Prophet Arena fully searcher-agnostic: new search agents can be added or replaced without altering the fore- casting protocol or evaluation procedure. In all experiments presented in this paper, we use a single LLsed searcher instantiated with GPT-4o equipped with web access.
(3) Probabilistic Forecasting and Comprehensive Evaluation. Given an event and its con- structed context, each model produces a probabilistic forecast for every market within the event. The model outputs a predicted probability, interpreted as its belief that the market will resolve to Yes, together with a short natural- rationale. All forecasts are logged for later analysis but only the probabilities are used for quantitative evaluation. Once the event resolves and the true out- comes become available, Prophet Arena evaluates the forecasts along multiple complementary metrics which are detailed in Section 3. Together with the event extraction and context construction stages, this final step completes the Prophet Arena pipeline- turning live, real-world questions into a continuous, contamination--esistan benchmark for predictive intelligence.
3

---

Published as a conference paper at ICLR 2026
Market Baseline. To establish fair and interpretable anchor for comparing different LLM forecasters, we include Market Baseline - synthetic forecaster whose prediction is defined as the market-consensus probability that market will resolve to Yes. In practice, this probability is inferred from the normalized contract prices. For instance, if the Yes and No contract prices 0.8 and 0.2, respectively, then the market baseline assigns 80% probability to the Yes outcome." As we will demonstrate in Section 3, this market baseline serves as an informative benchmark for assessing forecast difficulty and contextualizing model performance. When LLMs outperform the market baseline, they demonstrate genuine predictive advantage over the aggregated human consensus in the real-world market.
(+) Bonus feature:
On the Design Choices of Prophet Arena. Despite the few recent benchmarks focused on forecasting, Prophet Arena distinguished by the following key design choices (see Table 4), all of which are tailored for our systematic evaluation of L.LM's forecasting capabilities. To our best knowledge, Prophet Arena is the first large-scale benchmark that continuously on real-market events under multi-horizon protocol and within a modularized forecasting pipeline.
evaluation
1. Probabilistic forecasts:
Future events intrinsically random, so we elicit each market's proba- bilistic forecast from each model, rather than a single choice of the most likely outcome. Notably, probabilistic forecasting also the standard in real-world forecasting platforms, such as Metac- ulus (2015) and Good- Sooood-Jdgemen-OOpee (2015), as well as in the design of forecasting systems (Halawi al., 2024).
2. Multi-horizon protocol:
We implement multi-horizon forecasting protocol that sets schedule for the models to make prediction across various timestamps before the event resolves; we defer details on the scheduling algorithm and multi-horizon aggregation Appendix B.3. This enables our temporal analyses of how models update their forecasts as market conditions and public information evolve across the full lifecycle of an event.
3. Modularized forecasting pipeline: The process of prediction is broken down from source col- lection to probability elicitation to market actionability. This enables a comprehensive and con- trolled study of LLMs' forecasting performance at different modules.
4. Market return metries: Prophet Arena allows the evaluation of market profitability of each forecast, measuring models' relative advantage over market consensus.
Benchmark
Live events
Probabilistic
Multi-horizon
Modularized
Return metrics
MIRAI
FORECASTBENCH
FUTUREBENCH
FUTUREX
Prophet Arena
Table I: Comparisons with related forecasting benchmarks, including MIRAI (Ye et al., 2024), ForecastBench (Karger et al., 2025), FutureBench (Bianchi et al., 2025), FutureX (Zeng et al., 2025).
EVALUATION METRICS AND EMPIRICAL RESULTS
To obtain a thorough evaluation, we consider a number of different evaluation metrics that measure various aspects of LLMs' forecasting. Specifically, we consider three metrics: (1) Brier scores (Brier, 1950; Gneiting & Katzfuss, 2014) which measures the absolute quality of probabilistic prediction; (2) calibration error (Murphy, 1973; DeGroot & Fienberg, 1983; Guo et al., 2017) which measures the statistical consistency of a probabilistic prediction; and (3) market return (Mallikarjuna & Rao, 2019) which measures the relative ad advantage over current market's s consensus." We note that recent works on forecast evaluations have considered Brier scores (e.g., (Halawi et al., 2024; Karger et al., 2025)) and calibration errors (e.g., (Zou et al., 2022)), though no work has systematically knowledge.
evaluated models' market returns
In practice, contract prices may not up 2 Markeet consensus about an event's forecast is generally difficult to obtain. However, a good proxy of such data is available for P Prophet Arena as it fetches forecasting events from prediction markets, which are widely helieved to offer good approximation of the market's 's consensus the event's probability (Arrow et al., 2008; Berg et al., 2008).
due exchange fees, requiring slight normalization.
4

---

Published as a conference paper at ICLR 2026
Let E, denote the i-th event (i 1,..., n), and My the j-th market within E, (j 1...., m,), where m, is the number of markets for event E, For each market, a model outputs a predicted probability Pij E [0. 1] of resolving Yes, and the realized outcome is Oij E {0. 1). A complete notation table with full definitions is provided in Appendix B.
3.1 THREE EVALUATION METRICS AND THEIR DIFFERENCES
In this subsection, we first describe the three evaluation metrics we employ, followed by an illustra- tion of their differences and what they are used for.
Proper scoring rules (Gneiting & Raftery, 2007) directly evaluate the quality of the predicted proba- bilities. Prophet Arena adopts a classic and strictly proper scoring rule, the Brier score (Brier, 1950), defined for event E, with markets {M,}} I as BS; = 1 mh (Pij osy)?. That is, the Brier score for an event E, is the squared distance between the forecasted probability and indicator of the event realization, averaged across the markets.
my دド=
Our second metric is the expected calibration error (ECE) (DeGroot & Fienberg, 1983; Guo et al., 2017: Kalai & Vempala, 2024), which captures the following discrepancy of a forecaster given that it predicts probability to Yes, how different the predicted ç is from Yes's real occurrence frequency. The formal definition of ECE and its empirical estimation are standard in the literature, hence we defer its detail Appendix C.2. ECE is also classically known as the reliability score of forecaster (Murphy, 1973; Guo et al., 2017). Lower ECE means the forecast is more reliable, in the sense that the predicted probabilities are closer to the (conditional) true probabilities.
Finally, since P Prophet Arena uses events from real-world prediction markets, it allows us to evaluate the economic value of forecast based current market prices. Our third metric is thus Average Return, capturing how profitable if would be to trade in the market using I.M forecasts? Prophet Arena allows the evaluation of the average return under optimal strategies targeting different risk levels", but in the main body we report the return of an optimal strategy under risk- neutrality, averaged across all markets with a unit budget allocated each market.
Differences among the Three Metries and What Each Metric is For. Differences among the three metrics are already reflected in our empirical evaluations below. However, is still conceptu- ally useful to understand how these metrics fundamentally differ and what each metric is for. First, the difference between the Brier score and ECE is well- (Murphy, 1973).* 4 A forecast with bet- ter/smaller ECE could have worse Brier score (see an example in Appendix C.4). In practice, ECE is crucial when decision makers have risk preferences since smaller ECE ensures that the risk encoded in the forecasted probabilities is more reliably captured, leading better risk- tekk--rrrrene-aiiieee decisions even when the forecasted probabilities have worse Brier scores. In Appendix C.4, we illustrate this insight with simple example in which forecaster with better ECE but much worse Brier score could lead to better risk-adjusted utility. Second, while both market return and Brier score assess forecast quality, they differ fundamentally. The Brier score is an absolute metric, mea- suring a forecasts closeness to the ground truth, independent of external factors like market prices. In contrast, market return is a relative metric, capturing how much a forecast outperforms the mar- kets current belief (interpreted as the contract price (Wolfers & Zitzewitz, 2006)). This distinction evident our evaluations, and Appendix C.5 provides an illustrative example where forecasts with worse Brier scores achieve higher market returns. Finally, while calibration is not related to the total market return, but is indicative about how balanced the market return is from betting on the Yes contracts and No contracts. To formalize this intuition, well-calibrated and symmetric forecaster - intuitively, one that is not systematically more aggressive or conservative than the market will have balanced expected returns from both contract types.
prove in Appendix C.6 that
3.2 PERFORMANCES ACROSS DIFFERENT DIMENSIONS AND DISCUSSIONS
Throughout the main body of the paper, we highlight (the same) five representative LLMs out of the 22 evaluated in total (the full results are available in Table 7 of Appendix D.I). As shown in
Appendix C.3 offers derivations such strategies using risk sensitvne utility functions from classic behay- ioral economics.
MMuphyy (1973) shows that the Brier score can be decomposed into three terms: the ECE (also coined it "reliability score" by Murphy (1973)), the uncertainty score that captures inherent randomness of the to-be- forecasted event, and the resolution score that captures forecasts' variance.
5 Models are chosen to span proprietary and open-source families, reasoning and non-reasoning variants, and a range of performance levels in the full ranking.
5

---

Published as a conference paper at ICLR 2026
Forecasting Loss
Calibration Error
Market Return
LLM
Brier (95% CT)
Rank
ECE
Rank
Average (95% CI)
Rank
GPT-5" A
0.184 (# 0.006)
0.042
0.943 0.042)
Grok 4" A
0.189 (= 0.005)
0.043
3
0.864 (1 0.052)
4
Claude Sonnet 4 A
0.194 (# 0.006)
0.041
0.909 (+ 0.101)
Gemini 2.5 Flash A
0.197 (= 0.007)
0.067
0.883 (# 0,053)
Llama 4 Scout A
0,219 (#= 0.008)
0.060
0.805 (1 0.040)
Market Bascline
0.187 (= 0.006)
N/A
0.069
N/A
0.899 (+ 0.043)
N/A
Table 2: Evaluation of five representative LLMs. For Brier and Average Return, bootstrapped 95% confidence intervals are reported. Superseript is used to denote reasoning model, with its reasoning configuration in Appendix B.6. The full results for all 23 LLMs are provided in Table 7. Although our benchmark updates in real time, for the purpose of writing this paper we need fix dataset. Our evaluation is conducted on 1,367 events that were resolved before October 11, 2025.
Table 2, frontier proprietary models consistently outperform the Market Baseline across all met- rics. Here, the Market Baseline refers to a "synthetic" forecaster that treats market prices its own predicted probabilities (Wolfers & Zitzewitz, 2006): specifically, for each market, it uses the normalized market price for Yes contract as its probabilistic forecast that the market will happen.
Notably, the relative rankings of models differ depending on which evaluation metric is used, il- lustrating the complementary perspectives offered by accuracy, calibration, and profitability. Con- cretely, Brier scores fall in narrow band [0.17, 0.24] (pure random guess has expected Brier score 0.25). By contrast, calibration differences are more pronounced: strong models typically achieve ECE < 0.05, whereas weaker ones fall in the [0.05, 0.2] range. For market performance, even GPT- 5", the top-ranked model, fails to reach break-even (Average Return < 1), and most models fall below 0.9. Since event-level payoff's depend heavily on market-implied probabilities, the resulting returns exhibit substantial variance, as evidenced by the wide confidence intervals. In Appendix D.2, further discuss the Sharpe ratio (Sharpe, 1998) of our betting strategy, which normalizes Average Return by volatility, providing a more stable comparison of models' economic performance.
Figure 2: Reliability diagrams for the best and worst L.LMs ranked by calibration score (ECE). The black histogram indicates fraction of predicted probabilities in each bin. The calibration error within each bin appears the height of the red rectangle, i.c., the gap be- tween accuracy and confidence. The reported ECE score (bottom right) corresponds
Reliability Diagrum GPT-5*
Rellubllity
Diaed Accaso pl
S.dpeedicass
of prabitaess
weighted these errors, weighted by the distri- bution. Superscript denotes a rea- soning model.
of
Probabliky
Trobu248b
Overall, our results suggest that absolute forecasting skill and relative profitability against prediction markets are still challenging for todays L.Mss. A more fine-grained investigation could help deepen our understanding about what could make an LLM good prophet. Fig. 2 draws for the best and worst models (Left & right) their reliability diagrams regarding calibration errors (Guo al., 2025), where predicted probability (æ-axis) is compared against realized frequency (y-axis) at different probability bins. While their calibration is similar in intermediate ranges, the stronger model - GPT- 5 performs much better in the extreme bins (0-0.1 and 0.9-1.0), where it almost always predicts correctly. Because such extreme forecasts occur frequently, this advantage helps explain the gap in both Brier score and market return.

---

Published as a conference paper at ICLR 2026
4 AN IN-DEPTH EVALUATION OF LLM-AS-A-PROPHET
In this section, we employ our customized design of Prophet Arena conduct a systematic suite of experiments that shed light on the prospects of LLMM-as-aProphhtt The events used for evaluation were collected via our real-time Kalshi data pipeline, and consequently the distribution of event categories reflects the underlying composition: 81% Sports, 5% Entertainment, 5% Poli- ties, and 9% Other. To ensure that the results remain robust to event composition, we additionally verify that these performance patterns robust to reweighting the dataset toward a more balanced distribution across categories. Full results from these balanced-subset evaluations are reported in Appendix C.8.
4.1
MECHANISTIR ANALYSIS OF LLM--AS-A-
From a mechanistic perspective, we seek to understand how forecasting is shaped by the interplay of distinct capabilities, and in doing so, pinpoint the bottlenecks and causal links that determine their effectiveness. To this end, we design experiments that examine the key factors behind strong forecasting performance, ranging from model's 's internal knowledge, to the quality and accessibility of external sources, its ability to integrate those sources effectively.
4.1.1 CAN INTERNALIZED KNOWLEDGE BECOME FORESIGHT??
Recall Rate by Category and Model
Recall rate (%)
Figure 3: Recall rate by event category and model. Computed using the knowledge-iterraaizaato recall prompt (Appendix F.3.1). The shaded region indicates the fraction of events the model claims to recog- nize but recalls incorrectly.
40-
Clande Soaset Oemaai Fus
Event Category & Notation
Climate and Weather
Entertalament
Polities
Financials
Mis-rocall partion
Predictive intelligence is complex decision-make making process which relies not only information retrieval, but also on internalized knowledge. One concrete question is to ask whether models merely reproducing surface-level details leveraging deeper knowledge of past outcomes to inform present forecasts. This requires us to study how models interpret and recall events. Hence, retrieve 100 past events from Kalshi before the models' knowledge cutoff dates, and evaluate models under recall prompt in Appendix D.8.2.
Event recall varies by topics and models. In Fig. 3, observe that models most reliably recall events in Entertainment. By contrast, Climate and Weather and Politics display low recall and frequent mis-recall. Two factors likely contribute. First, Weather prompts often require fine graiindd date-stamped facts (e.g., Highest temperature in Miami on Aug 29, 2023?), which are less likely to be memorized. Second, due to 2023 regulatory constraints our dataset skews toward politics-adjacent indicators (c.g., "Biden 538 approval rating on Aug 30, 2023?), which varies daily and may be sparsely represented in training corpora (Kalshi (2025)).
Kalshi's election markets, Politics in
Despite these broad patterns, models differ in recall accuracy. For Economics and Politics, GPT-5 (High) correctly answered all events it claimed recall. In contrast, models like Llama 4 Scout and Gemini 2.5 Flash (Reasoning) reported recognizing events in all categories. However, for categories like Climate and Weather well as Politics, almost all of their recalled events were false.
Event recall is approximate, not precise. Consider the event "Billhoard Hot 100 #1, Jul 13, 2023?". Gemini 2.5 Flash recalled (Appendix E.2) that Olivia Rodrigo's Vampire displaced Morgan
'uue the resource constraint, our experiment results this section uses a subset 100 events sampled uniformly from our full benchmark. This dataset is publicly available at ht tttss: whgginggfaceecoooooooooooooooooe co/d atasetss/ [redacted for- nor-anonymity])
7

---

Published as a conference paper at ICLR 2026
Influence by Prediction Context
Influence by Event Category
Data source
Note
Mean difference
0.00
0.472
0.003
Sources-
0,092
0,042
Market Data
0.04
6,014
Both
0.00
0.05
0.10
0.15
0.20
0.25
Palicaa
Ermormac
Warld
Crype
Sports
Enter tainment
Brier score
Figure 4: Prediction quality across different contexts and event categories. Left panel shows average Brier scores across evaluated LLMs for prompts with varying information availability (none, sources only, market data only, or both). Error bars represent the interquartile range (IQR) of model performance. Right panel displays mean Brier scores of events in each category.
Wallen's ' Last Night at number one, which is correct. However, it aligned the answer with the chart dated July 15, 2023, treating it as equivalent July 13, 2023. Thus, while the model demonstrated knowledge of the outcome, it failed to recall the precise alignment between dates and chart releases. This illustrates that event recall in LLMs is approximate: models often retain coarse associations (the correct song and transition) but lack fidelity on exact temporal details.
4.1.2 How Do CONTEXTS SHAPE FORECASTS?
Given that the P Prophet Arena prediction inputs consist of existing prediction market data along- side multiple relevant news sources, we analyze how these different information sources affect model performance. Fig. 4 shows the average Brier scores across all tested LLMSs under four conditions: access to both market data and news sources, market data only, news sources only, and none. The results demonstrate clear performance hierarchy; as expected, models with access to both market data and news sources output the best predictions, while those without access to either source of information exhibit the poorest performance.
Beyond performance differences, the variance patterns reveal deeper insights. Interestingly, models using only market data perform only slightly worse, average, than those with both market data and news sources; however, the key difference lies in the variability of predictions. Combining multiple high-quality sources substantially reduces variance in prediction quality, suggesting that sources still offer valuable signals and perspectives that enable more consistent forecasting. Thus, while market data is powerful precisely because it aggregates information from a plethora of sources and trends, adding few carefully chosen high-qualit, sources can still help stabilize and refine the signals the information provides. Appendix 1.9 includes a deeper analysis of LLMS abilities to find and utilize high-quality sources.
Sources can clarify or confound predictions. As shown Fig. 4, on average, adding sources improves mean Brier score. However, the effect is heterogeneous: not only does the magnitude of the effect vary based on event category, it also does not necessarily strictly improve prediction quality, shown in the case study in Appendix E.3. Regarding category differences, the benefit of adding sources is not uniform. In areas like politics, where events can be interpreted through multiple perspectives, incorporating information from varied outlets and institutions appears add useful context. In contrast, in domains such as entertainment or sports, the marginal value of ad- ditional sources seems smaller. This highlights that more information is not necessarily better; the effectiveness of sources depends on their relevance to the prediction task.
4.1.3 MODELS ARE MORE CONSERVATIVE THAN MARKETS
Fig. 5 compares model prediction probabilities to the market-immlice probabilities. Since market data included the LLM prompts - most models generally align closely with market predictions. However, across large majority of events, LLMS consistently output more conservative probabil- ities. The clearest example is Llama Scout, shown in Fig. 5d: even when markets assign near- certain probabilities (close to one), the model remains hesitant, rarely producing equally extreme predictions. This reflects a systematic conservatism, where the model underweights outcomes the market views almost certain. As illustrated in Fig. 5a & Fig, 5b, although both GPT-5 and Grok 4 adopt a cautious approach in their probability assignments, they generally track the market more closely across the full probability range and avoid the same degree of reluctance. Claude Sonnet
8

---

Published as a conference paper at ICLR 2026
exhibits slightly assertive pattern: in the mid mid-probability range (around 0.5-0.6), it occa- sionally assigns probabilities slightly above the market-implied probabilities. Nevertheless, across most events, it displays significant conservatism, and particularly the higher end of the probability spectrum, Claude Sonnet 4 also is reluctant to place extreme probability predictions. Overall, while conservatism is common trait across models, the extent of hesitation varies, with some models hibiting considerably stronger reluctance than others. Similar trends across a wider range of models are reported in Appendix D.6.
GiT s
Grok 4
Cloudee Somnet 4
Lisane 4 Sceuut
1.
2nft?
Mo-dl gash Lebael
1.
Putty
14
Moder you (Klior))
Stdtrr gou (0lo0w
Mokee g00 (oloo))
Figure 5: LLM prediction probabilities compared to market baselines. Scatter plots show the relationship between market-implied probabilities and predicted probabilities for various models. "Winner" refers the market which resolves to Yes. Points the diagonal line correspond to events where the model prediction matches the market-implied prediction probability for the Yes outcome markets.
4.1.4 WHICH CAPABILITIES ARE NEARLY MATURE?
In addition to the core experiments, we also examined two fundamental capabilities: (i) robusiness of probability elicitation, where calibration remains stable under prompt variations and alternative probability estimation methods, and (ii) logical consistency, where most LLMs correctly understand structures such as mutually exclusive or nested markets. For the majority of models, both capabilities appear already reliable and largely mature. Detailed results are provided in Appendices D.3 and D.4.
4.2 LONGITUDINAL ANALYSIS OF LLM-AS-A PROPHEET
Forecasting is fundamentally temporal: we observe both the market and source dynamics changing as events approach their resolution time. Prophet Arena schedules predictions across multiple forecast horizons, allowing us assess how predictive quality evolves with lead time (Fig. 6). As shown in Fig. 6, the average Brier score is plotted across lead-time intervals, where, for instance, the "0-3 h" bin represents predictions made within three hours before resolution. For both the market baseline and representative L.Mss, accuracy generally improves as the resolution time approaches reflecting that additional information becomes available and predictive signals strengthen.? In- terestingly, the market baseline lags behind several frontier LLMs when predictions are made far in advance, suggesting that LLMs can effectively synthesize broader prior knowledge and reason under noisy settings at long horizons. However, as resolution nears, markets incorporate information and news updates more rapidly than LLMs, quickly surpassing LLMs in short-term accuracy.
Brier Performance Over Prediction Time
0.20
0.18
Figure 6: Average Brier score across predic- tion lead times. Scores are binned by time-to- resolution, with each point representing the Brier score of all forecasts falling within that window.
Brier Score
0.16
0.12
0.104
Grek
8.08
Clasdo Sccoud
2-hd
6faa a-la
Prediction Time Before Market Resolution
In certain domains, such as live sports, markets may remain open during the event itself, allowing traders incorporate real-time developments outcomes unfold.
9

---

Published as a conference paper at ICLR 2026
4.3 GRANULAR ANALYSIS OF LLM-AS-A-PROPHER
The reported probabilities of LLMs compress a rich decision-making process into single number: two models may yield similar predicted probabilities and scores while relying on dramatically differ- ent reasoning processes. To address this, we open the black box and evaluate the process underlying each forecast. Specifically, we adopt an LLMMass-a-uudee framework (Zheng et al., 2023) to assess the soundness of reasoning along five critical dimensions: source selection, evidence extraction, reasoning synthesis, reasoning-te weaiinnggt--rddiiice prediction aligmment, and recognition of prediction uncertainty. Full prompts and additional details of the evaluation framework are provided in Appendix D.7. To assess the reliability of the L.LM-as-a-judge framework, we conducted a blinded human evaluation on random subset of 170 predictions made by the LLMs. Human raters scored the same five di- mensions using the identical rubric provided to the LLM judge. Overall, human and LLM ratings were closely aligned (mean difference < 0.5 out of a maximum of 4, with standard deviation s 0.3 across all dimensions). Full details of the study design and results are provided in Appendix D.7.1.
As shown in Table 3, the models demonstrate broadly comparable performance source utilization, evidence extraction, and uncertainty analysis. However, substantial disparities emerge in the reason- alignment categories. For example, when comparing GPT-5 with Gemini 2.5 Flash, the differences in reasoning synthesis (0.95) and reasoning-to- -o-peedcccno alignment (0.30) are markedly larger than the relatively minor gaps in source use (0.12), evidence extraction (0.00), and uncertainty analysis (0.20). Due to the significant deficiencies in those two key categories, the other models demonstrate a sigmificant gap in prediction quality relative to GPT- 5, supporting the findings in Table 2. These findings indicate a potential ceiling effect: once models attain proficiency in retrieval and evidence extraction, further performance gains depend primarily on advances in higher-order reasoning rather than incremental improvements information access.
ing synthesis and maasoning-predictin
LLM
Sources
Evidence
Reas. Synth.
Align.
Uncert.
Average Score
GPT-5"
3.69
3.66
4.14
3.97
3,94
3.88
Gemini 2.5 Flash"
3.57
3.66
3.19
3.67
3.74
3.57
Grok 4
3.40
3.51
3.33
3.48
3.66
3.48
Claude Sonnet 4
3.53
3.47
2.93
3.39
3.75
3.41
Llama Scout
2.97
2.88
2.29
2.37
2.87
2.68
Table 3: LLM performance reasoning evaluation criteria across dataset events. Each di- mension is scored on a standardized 5-point scale, where 1 and indicate poor and excellent perfor- mance, respectively. Average scores presented for each model, with bold values indicating the best-performing model for each criterion. Models are ordered by descending overall average score.
5 CONCLUSION
This paper systematically evaluates the prospects and challenges of using LLMs to forecast future events, paradigm coined LLM--a-a-Proppet.. Towards that end, we build Prophet Arena, benchmark that allows modularized analysis about various aspects of existing LLMs' predictive intelligence. Our thorough experiments demonstrates the promise of L.LM-as-a-Prophee reflected in the small forecasting loss, calibration error, strong reasoning synthesis and alignment of frontier models. However, we also identify key bottlenecks and highlight avenues for further progresses, such as better curation of context sources, more accurate internalization of knowledge and improving forecasts near events" resolution.
In addition, our findings raise deeper questions about the nature of predictability in forecasting tasks. Predictability depends not only on model capability, but also on the underlying information structure of eventswhat signals exist, how early they appear, how noisy they are, and how strongly they constrain outcomes. Our current metries alone do not reveal whether performance gains stem from better signal extraction, stronger reasoning under uncertainty, from tasks that are intrinsi- cally more predictable. Interpreting forecasting results therefore requires disentangling task-level predictability from model-level capability.
Finally, we view Prophet Arena not merely as leaderboard, but as a framework for probing pre- dictive intelligence in context. The benchmark implicitly evaluates intertwined components: sig- nal identification, evidence synthesis, uncertainty modeling, and probabilistic calibration. key direction for future work is to more clearly separate these factors, clarifying what makes events predictable in principle and which model behaviors enable more accurate forecasts.
10

---

Published as a conference paper at ICLR 2026
ETHIC STATEMENT
Our evaluation pipeline of LLMs involve publicly available prediction market questions concerning widely known real-world events (e.g., sports, politics, entertainments). While we open-source small subset of these events for transparency and reproducibility, no private, identifiable, or sensitive data is released and no human subjects were involved. The study adheres to the ICLR Code of Ethics by upholding standards of scientific integrity, transparency, and responsible research practice. We do not anticipate our methods findings to pose risks of harm, privacy violations, fairness concerns. While our findings may inform understanding of LLMs' probabilistic forecasting capabilities, we explicitly caution that they must not be construed as endorsements of specific applications nor as guarantees of real-world reliability.
REPRODUCIBILITY STATEMENT
We have taken steps to ensure the reproducibility of our benchmark and experiment results. In Sec- tion 4, we provided the link to curated subset of 100 evaluation events PPROPHET-ARENS SUBSET- 100) open-sourced on Huggingface. We also intend to expand this dataset in the future. All language models tested in our study are accessible through publicly available APIs and a complete list of the models and versions used are provided in Appendix B.6. The exact prompts used for all experiments are also included throughout Appendix D. Additional implementation and experimental details described in the main text and appendices to facilitate independent verification and extensions of our work.
REFERENCES
Anthropic. Introducing claude 4. /ttpppppwwwww.atttaao.a.cc////// 2025.
May
Kenneth Arrow, Robert Forsythe, Michael Gorham, Robert Hahn, Robin Hanson, John O Ledyard, Saul Levmore, Robert Litan, Paul Milgrom, Forrest D Nelson, al. The promise of prediction markets. Science, 320(5878):877-878, 2008.
Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics, 51 (1)))8116-845, 2023.
Joyce Berg, Robert Forsythe, Forrest Nelson, and Thomas Rietz. Results from dozen years of election futures markets research. Handbook of experimental economics results, 1:742-751, 2008.
Federico Bianchi, Junlin Wang. Zain Hasan, Shang Zhu, Roy Yuan, Clmentine Fourrier, and James Zou. Back to the future: Evaluating agents on predicting future events. https:/ htta://wwwwttt gether. ai fetoo//uuuuuebbnnhe 2025. Online; accessed 10 September 2025.
Glenn Brier. Verification forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2rX 2023.
Chris Chatfield. Time-series forecasting. Chapman and Hall/CRC, 2000.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Hui Dai, Ryan Teehan, and Mengye Ren. Are lims prescient? continuous evaluation using daily news as the oracle. In Forty-second International Conference on Machine Learning (ICML), 2025.
Morris H DeGroot and Stephen E Fienberg. The comparison and evaluation forecasters. Journal of the Royal Statistical Society: Series 1) (The Statistician), 32-1-2):12222, 1983.

---

Published as a conference paper at ICLR 2026
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 8698-8711, 2024.
Thomas DiCiccio and Bradley Efron. Bootstrap confidence intervals. Staristical science, 11(3): 189-228, 1996.
Dean P Foster and Rakesh V Vohra. Asymptotic calibration. Biometrika, 85(2):379 1998.
Gemini Team. Introducing gemini 2.0: our new ai model for the agentic era. https: //blog. /oogle deieee-depinndd ddepmmnd/gooogleeeeeeeeeee////ee -po.lee-eeeii-------u----------- r-2024, Dec 2024. Accessed: 2025-09-22.
Jiahui Geng, Fengyu Cai, Yuxia Wang, Heinz Koeppl, Preslav Nakov, and Iryna Gurevych. A survey of confidence estimation and calibration in large language models. arXiv preprint arXiv:2311.08298, 2023.
Tilmann Gneiting and Matthias Katzfuss. Probabilistic forecasting, Annual Review of Statistics and Its Application, 1(1): 151, 2014.
Tilmann Gneiting and Adrian E Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association, 102(477):359-378 2007.
Good-Judgement- Open. Good judgement open. https: tttp://wwwpppppooo////t//// com/ 2015. Online; accessed 10 September 2025.
Yong Guan, Hao Peng, Xiaozhi Wang, Lei Hou, and Juanzi Li. Openep: Open-ended future event prediction. arXiv preprint arXiv:2408.0657 065588 2024.
Chuan Guo, Gcoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948 2025.
Danny Halawi, Fred Zhang, Chen Yueh-Han, and Jacob Steinhardt. Approaching human-level fore- casting with language models. Advances in Neural Information Processing Systems, 37:50426- 50468, 2024.
Kalshi Inc. Kalshi prediction market for trading the future. https::/www.cc http::::/www..alsh..coom com/ 2025.
Woojeong Jin, Rahul Khanna, Suji Kim, Dong- Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. Forecastqa: A question answering challenge for event forecasting with temporal text data. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Iith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4636-4650, 2021.
Adam Tauman Kalai and Santosh S Vempala. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pp. 160-171, 2024.
Kalshi. Kalshi rulebook and contract specifications. https: /ttp///aashhi.. http://aaaaaat.cceeeeeataaoooo com/requlatooy/ rulebook, 2025. Accessed: 2025-09-23.
Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tetlock. Forecastbench: A dynamic benchmark of forecasting capabilities. In The Thirleenth International Conference on Learning Representations (ICLR), 2025.
Kimi Team. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.2053: 2025.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024.
12

---

Published as a conference paper at ICLR 2026
Mejari Mallikarjuna and R Prabhakara Rao. Evaluation of forecasting methods from selected stock market returns. Financial Innovation, 5(1):40, 2019.
Meta Team. The Llama 4 herd: The beginning of new era of natively multimodal AI innovation. Al at Meta, April 2025, URL ht https::/s/ //a:://aiie meta www.blog//lamaaeaaooooooooooooee 4-mult imoda 1- t-intelligencee Blog post.
Metaculus. Metaculus:
Forecasting for a complex world. https::/www.mmtaactt 2015. Online; accessed 10 September 2025.
Allan H Murphy. A new vector partition of the probability score. Journal of Applied Meteorology and Climatology, 12(4):595-600 1973.
OpenAl. Hello gpt4o. https: http://pppaaaannnneaee/e httppp/aaa.cmmmmeeeeee/ellllllop May 2024.
OpenAl. Introducing gpt4.1 in the api. httpps//ppeeaa..cooo////e////t/p Apr
2025a.
OpenAl. Introducing https:tt////
http://tpsseeeeeeaaacc///////aat - -evelopers/ Aug 2025b.
for
OpenAl. Introducing openai o3 and 04-mini. ng-o3 o0--o3-ae -3--add-o--iind Apr 2025c.
Daniel Paleka, Shashwat Goel, Jonas Geiping, and Florian Tramer. Evaluating forecasting is more difficult than other Ilm evaluations. In ICML. 2025 Workshop Assessing World Models, 2025a.
Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang, and Florian Tramèr. Consistency checks for language model forecasters. In The Thirteenth International Conference on Learning Representations, 2025b.
William F Sharpe. The sharpe ratio. Streetwise-the Best of the Journal of Portfolio Management, 3 (3):669855, 1998.
Amir Taubenfeld, Tom Sheffer, Eran Ofek, Amir Feder, Ariel Goldstein. Zorik Gekhman, and Gal Yona. Confidence improves self-consistency in LLMs. In Wanxiang Che, Joyce Nabende, Eka- terina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Compu- tational Linguistics: ACL 2025, pp. 20090-20111, Vienna, Austria, July 2025. Association for doi: 10. 10.18333/112005 htt.////w11222...nnnngn---a findings- 1030.
Computational Linguisties. ISBN URL https:tp////m http:////aaaaaahhhaca ogg.. org 2025. findings sealings--cl acl. 1030/
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. arXiv preprint arXiv:2305. 14975, 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou delf-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2003.1177. 2022.
Zhen Wang, Xi Zhou, Yating Yang, Bo Ma, Lei Wang, Rui Dong, and Azmat Anwar. Openforecast: A large-scale open-ended event forecasting dataset. In Proceedings of the 31st International Conference on Computational Linguistics. pp. 5273-5294, 2025.
Gwenyth Portillo Wightman, Alexandra Delucia, and Mark Dredze. Strength in numbers: Esti- mating confidence of large language models by prompt agreement. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pp. 326-362, 2023.
Jack Wildman, Nikos I Bosse, Daniel Hnyk, Peter Mühlbacher, Finn Hambly, Jon Evans, Dan Schwarz, Lawrence Phillips,, et al. Bench to the future: A pastcasting benchmark for forecasting agents. arXiv preprint arXiv:2506.21556 2025.
Adam Wilmot. Legal breakdown of kalshi and sports prediction markets, March 2025. URL http wttppppprr..ccc/////////aaaaaaaa
s: oddspport--preddicctoooomm--e
13

---

Published as a conference paper at ICLR 2026
Justin Wolfers and Eric Zitzewitz. Interpreting prediction market prices probabilities, 2006.
XAL. Grok 3. httpss///..a////e///grrr.-. Feb 2025a. Accessed: 2025-09-22.
XAI. Grok 4. https::t///
Jul 2025b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.003388 2025.
Chenchen Ye, Ziniu Hu, Yihe Deng, Zijie Huang, Mingyu Derek Ma, Yanqiao Zhu, and Wei Wang. Mirai: Evaluating Ilm agents for event forecasting. CoRR, abs/2407.012311 2024. URL https:
////oiiorg/////////////o// 100 48550 /a.////a//v...//. http//aawvvv.......... 21110001111...1
Zhiyuan Zeng, Jiashuo Liu, Siyuan Chen, Tianci He, Yali Liao, Jinpeng Wang, Zaiyuan Wang, Yang Yang, Lingyue Yin, Mingren Yin, al. Futurex: An advanced live benchmark for Ilm agents in future prediction. arXiv preprint arXiv:2508. /1987, 2025.
Zhihan Zhang, Yixin Cao, Chenchen Ye, Yunshan Ma, Lizi Liao, and Tat Seng Chua. Analyzing temporal complex events with large language models? benchmark towards temporal, long con- text understanding. In Long Papers, pp. 1588-1888 1606. Association for Computational Linguistics (ACL), 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595-46623, 2023.
Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales? Advances in Neural Information Processing Systems, 37:1 37:123866-123910 2024.
Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, and Benyou Wang. Is your IIm outdated? a deep look at temporal generalization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 7433-7457, 2025.
Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural net- works. Advances in Neural Information Processing Systems, 35:27293-27300 2022.
14

---

Published as a conference paper at ICLR 2026
APPENDIX: TABLE OF CONTENTS
A. Full Discussion of Related Work
Page 16
B. Prophet Arena Pipeline Details
Page 16
C. Technical Details
Page 20
D. Additional Experiment Results
Page 27
E. Case Studies
Page 35
F. Prompts
Page 43
G. LLM Usage Disclosure
Page 53
15

---

Published as a conference paper at ICLR 2026
A FULL DISCUSSION OF RELATED WORK
Understanding and Advancing LLMss' Forecasting Capabilitiiss A key goal of our work is to understand the novel paradigm of LLM-as-a-Prophet and analyze how different capabilities (e.g.. knowledge internalization, source usage, etc.) affect LLMs' predictive intelligence. This research goal shares similarities to few recent works understanding and diagnosing special aspects of LLMs' forecasting capabilities. For instance, Dai et al. (2025); Zhu et al. (2025) study LLMs' temporal generalization capability by challenging LLMs to forecast future events curated from news articles. Both works show that LLMMs forecasting accuracy degrades over time, even when they are armed with retrieval augmented generation (RAG). Paleka al. (2025b) studies whether LLMs can make consistent forecasts; for example, logical Al should not predict that both the Democratic and Republican parties have a 60% chance of winning the 2024 US presidential election. Towards that end, they build proper-scoring-ruke forecasting benchmark to measure the consistency of LLMs' predictions. Similar to both works' research methodology, we also build a benchmark platform Prophet Arena to study our research question. However, to our knowledge, our work is the first to investigate the general predictive intelligence of the L.LM-as-a-Prophe paradigm. Finally, we also note the recent line of works advancing the forecasting capabilities of Al systems using language models (Zou et al., 2022; Halawi et al., 2024). Though our goal of benchmarking is different, we envision that insights from our analysis about the strengths and limitations of current LLMs for forecasting could benefit future research for advancing Al's forecasting capabilities.
Forecasting Benchmarks. Forecasting has recently become popular challenge for benchmarking LLMs' capabilities. Besides being a real challenge to LLMs, it also avoids the thorny issue of benchmark contamination due to the evaluating LLMs "future" events (Dai et al., 2025; Karger et al., 2025). To our knowledge, (Jin et al., 2021) is perhaps one of the earliest to test such forecasting capabilities of language models. They introduced ForecastQA, an evaluation dataset consisting of 10,392 crowdsourced multiple-choiee questions, and the performance of language models at the time (mainly BERT models) still significantly lags behind human performance. Since Jin et al. (2021), there has been progressive line of work developing more and more challenging forecasting benchmarks for more advanced models, by integrating prediction market events (Zou et al., 2022), extracting events from news articles (Zhang et al., 2024; Wang et al., 2025), curating future-oriented questions from websites (Wildman et al., 2025), using open-ended queries (Guan et al., 2024), and lately developing dynamic benchmarks with live event and leaderboard updates (Zeng et al., 2025; Karger et al., 2025; Bianchi et al., 2025).
We also build benchmark Prophet Arena, but as mentioned above, the main goal of our work is to understand LLM-as-a-prophe and analyze how its different capabilities affect the forecasting. We built P Propht Arena primarily for this analysis purpose, similar to that of (Dai et al., 2025; Paleka et al., 2025b). Towards this end, our work departs from these benchmark researches in multiple ways. First, we focus on comprehensive evaluation about LLMs' predictive intelligence from various dimensions such as the Brier score. calibration and economic values of forecasts. In Section 3. we illustrate how these metrics differ and when each metric should be preferred. In contrast, most previously works mainly focus one metric, such as Brier score Halawi et al. (2024); Karger et al. (2025), calibration Zou et al. (2022) and accuracy Zeng et al. (2025); Dai et al. (2025), rendering them not full capture the quality of forecasting (as we also illustrate in Section 3). Second, because our goal is not to rank LLMs or agents as in the above benchmark papers but rather to understand frontier LLMs' forecasting capabilities, we need a natural baseline forecaster to know how good frontier LLMs are. Towards that end, we build P Prophet Arena based on events from prediction markets, and hence use the market's forecasts
cost-effective and justified bascline forecaster (Arrow et al., 2008). Finally, besides reporting different evaluation metrics, we further dive into the L.LM-as-a-Propha paradigm and offer insights about how the forecasts are affected by different capabilities such as knowledge internalization and source usage.
B DEFINITIONS AND PROPHET ARENA PIPELINE DETAILS
B.1 OVERVIEW
We summarize the core components of the Prophet Arena pipeline, which together distinguish it from prior forecasting benchmarks (see Table 4 for comparison with existing approaches).
(1) Event and Market Extraction. Prophet Arena collects unresolved events from Kalshi, live prediction platform with events spanning across categories such finance, sports, politics
16

---

Published as a conference paper at ICLR 2026
and entertainment. Kalshi provides structured, real-time events with verifiable outcomes, enabling standardized and comparable evaluation across models. To ensure informativeness and compara- biliiy, we filter by Popularity (volum//iguidityy/olaliiii rence (repeated formats). Prophet Arena periodically retrieves a number of unresolved events, making it truly live benchmark, rather than fixed, static dataset susceptible training data lamination. The use of prediction markets guarantees that all questions pertain to genuine future outcomes grounded on real world importance and that their resolutions can be objectively verified once finalized.
Diversity (domain balance), and Recur-
(2) Prediction Context Construction.
For each event, Prophet Arena constructs unified prediction context that all models receive identically. This context contains: (1) Relevant information sources retrieved by an LLM-based search agent using web queries that collect titles, snippets, timestamps, and URLs of recent news or reports; and (2) Market snapshots including the latest Yes/ No contract prices and trading volumes, from which implied probabilities (based on market contract prices) are derived. Providing identical contexts isolates differences in models reasoning and calibration rather than their retrieval capabil- ities. The search component in Prophet Arena is fully searcher-agnostie new search agents can be added or replaced without altering the forecasting protocol or evaluation procedure. In all experiments presented in this paper, we single LLM-based searcher instantiated with GPT-4o equipped with web access.
(3) Probabilistic Forecasting and Comprehensive Evaluation. Given an event and its con- structed context, each model produces a probabilistic forecast for every market within the event, outputting a predicted probability for a Yes resolution alongside natural-language rationale. This process is highly modularized, decomposing prediction into distinct stagesfrom source collection and context construction to probability elicitationenabling controlled experimentation and analysis of performance at each module. To capture the temporal evolution of forecasts, we implement multi-horizon protocol where models predict at various timestamps as market conditions and public information evolve. While all rationales are logged for qualitative analysis, only the probabilities are used for quantitative evaluation once the event resolves. By evaluating these forecasts against mul- riple complementary metrics detailed in Section 3, Prophet Arena transforms live, real-world questions into a continuous, contamination resistaan benchmark for predictive intelligence.
(*) Bonus feature: Market Baseline. To establish fair and interpretable anchor for comparing different LLM forecasters, we include a Market Baseline - a synthetic forecaster whose prediction is defined as the market-consensus probability that market will resolve Yes. In practice, this prob- ability is inferred from the normalized contract prices. 8 As we will demonstrate in Section 3, this market baseline serves an informative benchmark for assessing izing model performance. When LLMs outperform the market baseline, they demonstrate genuine predictive advantage over the aggregated human consensus in the real-world market.
difficulty and contextual-
Benchmark
Live events
Probabilistic
Multi-horizon
Modularized
Robust metrics
MIRAI
7
FORECASTBENCH
FUTUREBENCH
FUTUREX
Prophet Arena
Table 4: Comparisons with related forecasting benchmarks, including MIRAI (Ye al., 2024), ForecastBench (Karger et al., 2025), FutureBench (Bianchi et al., 2025), FutureX (Zeng et al., 2025).
B.2 EVENT EXTRACTION
Prophet Arena sources unresolved events from Kalshi, a live prediction platform with events spanning across finance, sports, politics, sports, and entertainment. To ensure informativeness and comparability, we filter by Popularity (volume/liqiidityyvvlaiii Recurrence (repeated formats). Prophet Arena periodically retrieves 20 unresolved events each day at 12 AM (UTC).
Diversity (domain balance), and
'In practice, contract prices may not sum up due exchange fees, requiring slight normalization.
17

---

Published as a conference paper at ICLR 2026
Event. Defined formally, let (ee) denote the set of evaluated events. An event is the overarch- ing question or subject concerning a future real-world occurrence. It serves as a high-level container for one more tradable markets. In many prediction markets, event itself is not tradable asset; rather, it sets the context, scope, and resolution criteria for the markets that fall under it.
Example 1: "Who will win the 2025-26 NBA Championship?"
Example 2: "Which individuals will President Trump officially meet in 2025?"
Market. Each event E, contains markets (Mi) JE(N.. A marker is specific, tradable proposition
under an event that resolves to a definitive Yes or No outcome. Each market represents a potential, verifiable answer to the event's overarching question. For given market, Yes contract is 0-1 random variable that achieves value if the Yes outcome is realized, and 0 otherwise. A "NO" contract is defined similarly, and always pays out in the opposite direction as the "YES" contract.
Under Event market could be:
"The Boston Celtics will win the 2025 NBA Championship."
Under Event 2, a market could be:
"President Trump will officially meet with Emmanuel Macron in 2025."
Event Resolution. An event E; resolves at time with realized outcome indicator Oi (0, where Oiy = 1 means the corresponding market May of the event realized (Yes). When the event index i is clear from context, we often drop the subscript i and write Mj Oj-
B.3 PREDICTION CONTEXT
Prediction scheduling. For each event E,, the benchmark specifies a finite set of pre-resolution forecast times (horizons) T C (-00, ri). We construct Tr by placing each subsequent forecast halfway between the current forecast time and the event close time Tú. Let the first forecast time be (0)
Ti and define the initial gap A :- Ti ( 0. For O.
(k)
A
(2)
+
T3
(k) T3 2- A (0)
A (++1)
2
To avoid excessive clustering near т;, we enforce minimum time gap Smin > between the last forecast and the close time.
For each event E; and forecasting time & T; we construct curated context Oi shared across models; this isolates forecasting ability in Pij.t from retrieval variability. The context C.. consists of two components: relevant news sources and market data. The relevant news sources are retrieved by LLM searchers. The prompt for search is shown below. The market snapshot fetched from Kalshi API, and contains three fields for each market: last price (price of the last transaction), yes ask (asking price for buying Yes), ask (asking price for buying No). From that snapshot we extract the implied probability 4j.t E [0, 1] for Yes contract at time (with No priced at 4.,,)..
Context construction.
At each forecasting time E Ti LLM searchers will be dispatched and live market snapshot will be retrieved. Together, relevant news sources from LLM searchers and market snapshots from Kalshi will serve as the prediction context.
Importantly, Prophet Arena searcher-agnostic: the search component is an independent, pluggable module and adding new searchers does not change the forecasting protocol or the scoring of Pij.t- Prophet Arena actively updates to include new LLM searchers. In this paper, we instantiate single LLM searcher using GPT-4o with web search enabled and all experiments that configuration (Appendix F.1.1).
B.4 PROBABILISTIC FORECASTING TO ACCOUNT FOR UNCERTAINTY
LLM-predieted Probabilities. Given (E,, Meys Ci (), model must output an LLM probability,
Pij.t [0, 1).
Implied probabilities are reverse-engineered from contract prices; transaction fees may cause YRS/ NO prices not sum 1.
18

---

Published as a conference paper at ICLR 2026
interpreted as its belief that My will realize Yes at time t, accompanied by natural-languaat rationale (logged for analysis but not used in scoring). The prediction prompts are documented in Appendix F.1.2.
Implied Probabilities. At each E Ti a Yes (resp. No) contract is valued Jij.t (resp. kz.t)) The implied probability 4tj represents the (human) market-consensus belief that the Yes outcome will come true.
B.5 RESOLUTION AND EVALUATION
Edge and Utility, We denote a (yes) edge Cij == Pi as the likelihood-ratt between the LLM- 9;
predicted and implied probabilities. A larger edge signals the LLM to be more confident (than the market) that market will be realized. Similarly, we define the (no) edge to be Egj
1-Pu, T-ax 1
We assume that the price of single Yes contract simply equals the implied probability daj- and the price of single contract is thus 1 qij- 10 In the sequel, strategy is limited to buying contracts (taking long position). But all contracts can be purchased in fractional amounts.
Our simulated trading policies (used only for relative metrics) considers the utility function with risk-aversion hyperparameter y E [0, 1]. It maps any wealth (i.e. payoff) to utility.
Prediction Evaluation. After the close time Tr will retrieve the outcomes of event E, from Kalshi. For each market Mij and each horizon E Tr we score (Pig.t: oij) with absolute proper scoring rules (e.g., Brier-based) and, separately, evaluate relative Average Return by simulating trades against dzj.t under Uy. Scores are then aggregated across j. i, with the specific calculations detailed in the subsequent evaluation section.
The below table summarizes the notations that will appear in the later math expressions:
E,
i [K], the i-th event in our evaluation set.
My (M₃)
j E [Na], the j-th market of the i-th event, we drop the subscript when the market is obvious (same for below).
Piy (p,)
the LLe-ddrddeted probability that Mi will realize.
4s (g))
the market implied probability that My will realize.
eg/Ess (es/Ez)
the (yes/no) edge of Mij-
og (oz)
indicator of whether Mij realized.
(w)
the utility function with rvsk-aversion hyperparameter y E [0, 1]. It maps any wealth (i.e. payoff) utility.
B.6 COMPREHENSIVE LIST OF EVALUATED LLMS
Remark. The column Default Reasoning specifies the reasoning configuration used when a model is referenced without qualifiers (e.g. GPT-5, when we write GPT-5* in Table 2). In the full evaluation table (Table 7). some models appear multiple times under different reasoning settings; each such variant is treated distinct model (e.g., GPT-5 (Minimal)).
Reasoning configuration is inherently model-dependent. For "hybrid reasoning" models that allow toggling between thinking and non-thinking modes, the configuration is specified either enabled or disabled. For models that expose explicit control over reasoning effort, the configuration is ex- pressed in levels (e.g., minimal, medium, high).
In practice, (1) actually "reverse-engineer" the implied probabilities from market contract prices, and (2) the prices of Yes/No contracts might not due transaction fees taken by the exchange.
19

---

Published as a conference paper at ICLR 2026
LLM
Citation
Open Weight?
Default Reasoning
GPT-5
OpenAI (2025b)
No
High
o3
OpenAl (2025c)
No
High
03-Mini
OpenAl (2025c)
No
Medium
04-Mini
OpenAl (2025c)
No
High
Gemini 2.5 Pro
Comanici et al. (2025)
No
Enabled
Gemini 2.5 Flash
Comanici et (2025)
No
Enabled
Grok-4
XAI (2025b)
No
Enabled
Grok-3-Mini
XAI (2025a)
No
Enabled
GPT-4.1
OpenAl (2025a)
No
N/A
Claude Sonnet 4
Anthropic (2025)
No
Enabled
Kimi-K2
Kimi Team (2025)
Yes
N/A
GPT-4o
OpenAl (2024)
No
N/A
Llama 4 Maverick
Meta Team (2025)
Yes
N/A
Llama 4 Scout
Meta Team (2025)
Yes
N/A
DeepSeck-Vi
Liu et al. (2024)
Yes
N/A
DeepSeek-RI
Guo et al. (2025)
Yes
Enabled
Qwen3-235B
Yang al. (2025)
Yes
N/A
Gemini 2.0 Flash
Gemini Team (2024)
No
N/A
Gemini 2.0 Flash (Lite)
Gemini Team (2024)
No
N/A
Table 5: Comprehensive list LLMs evaluated in Prophet Arena (as of submission time).
C TECHNICAL DETAILS
C.I FURTHER INTUITIONS BEHIND BRIER SCORE
In Prophet Arena, the set of markets {M under an event Eg need not be mutually exclu- sive. This has two implications: (1) the realized outcome vector o, = (Os1..... ... Osm, Y may contain multiple ones rather than being strictly one-hot, and (2) the predicted probabilities {py Jy may sum to more than one. Our Brier score formulation remains robust to these cases because the score is always evaluated at the market level, where each individual market binary resolving either to Yes or No, but not both. If we pool all markets across all events and relabel them MI...., Mr. the standard binary Brier score is simply
(e BS h: 1=1
Our event-level definition Section 3 can be viewed as weighted version Eq. (1). Since events vary greatly in the number of associated markets, directly pooling them would let large events domi- nate the metric. mitigate this, we assign each market a weight way 1/m;, inversely proportional to the number of markets in its event. This yields the final form:
N
N
m+
BS ΣE Wij
Wiy(Pip - -0,)
M
(p
N
m
j=1
This weighting ensures that each event contributes equally, regardless of its size, while still respect- ing the binary resolution of individual markets.
C.2 EXPECTED CALIBRATION ERROR(ECE) AND ITS EMPIRICAL ESTIMATION
In our binary market setting, let M {1. m) denote the set of markets. Suppose a forecaster provides predicted probabilities Pte for all markets in a set Mr. M (mz Mal denotes its cardinality). Then the expected calibration error (ECE) of this forecaster is formally defined as
ECE
Σ
Σ JEML
P(o; 1/p Pk) m.pk
(2)
m
20

---

Published as a conference paper at ICLR 2026
Intuitively, ECE captures how much probabilistic forecast differs from the real averaged probabil- ity, given the forecast. This formal definition of ECE is in terms of true conditional probabilities. In practice, this definition cannot be computed exactly, the conditional terms P(o, 1 Pj p;) are not directly observable. Instead, the standard approach in the applied literature is to approximate ECE via binned empirical estimate.
Formally, let {(pj. } denote a set of predicted probabilities and realized outcomes. We first partition the unit interval [0, 1] into B disjoint bins {4)]m and assign each prediction Pj to its corresponding bin. Let M, {j Pj E I,} be the index set of predictions falling into bin b and m, M& its size. Define
Σ JEM,
Σ Oj:
Pb
Pj:
m7
m&
JEM,
the average predicted probability and empirical frequency of Yes outcomes in bin respectively. The empirical expected calibration error is then given by
ECE
mb 1 -0ol
m
Intuitively, ECF measures the weighted average discrepancy between empirical accuracy and av- erage predicted probability across bins, with weights proportional to bin counts. Throughout experiments, all reported calibration results correspond to this empirical version (with B 10).
C.3 A UNIFIED FRAMEWORK FOR UTILITY-MAXIMIZA
BETTING STRATEGY
In Section 3, we introduced Average Return which captures expected reward under a risk-neural investor's optimal betting strategy. Here offer more details about how such betting strategy can be characterized. Formally, consider (binary) market with market price qy per share for the Yes contract and QN for the No contract. Given forecasted probability for Yes outcome, the following natural betting strategy turns out to be optimal for a risk-neural investor: allocate a unit budget ($1) to buy 1/qy shares of Yes contracts if P 2 1- or to buy 1/qN shares of No contracts otherwise. After the event resolution, the bought contracts match the event outcome, the return is simply the number of these contracts; otherwise, the return is O.
qN
To more systematically understand the optimal betting strategy, next we develop a unified frame- work for betting strategies in binary markets for risk-sensitive investors. This framework serves two purposes:
1. It formalizes betting as millit-mmaimizztiion problem, allowing to flexibly encode different risk preferences.
2. It shows that the simple strategy mentioned above is special case of this unified framework, corresponding to the risk-neutral setting.
Contracts and market prices. Each binary market resolves to either Yes or No. A share a Yes contract pays $1 if the outcome is Yes and $O otherwise; the price of this contract is denoted Symmetrically, a share of a No contract costs 1 and pays S1 if the outcome is No. These prices are often interpreted as market-implics probabilities (Wolfers & Zitzewitz, 2006), though our framework does not rely on this interpretation.
Utility function. Let U R20 + E denote utility function mapping payoff to its perceived value (satisfaction) U ((w). We focus on the class of constant relative risk aversion (CRRA) utilities:
w
057<1
U+(w)=
1-7" log w.
7=1
where E E [0, 1] indexes risk aversion. At 0, Un is linear (risk-neutral); at y it reduces to log w (log utility, risk-averse); and for 7 E (0, 1). it interpolates smoothly between the two. Fig. visualizes representative cases.
Budget allocation as optimization. Fix a unit budget of $1 for each market. Let ay and a denote the amounts allocated to Yes and No contracts, with ay ax 1. Purchasing ay dollars of Yes
21

---

Published as a conference paper at ICLR 2026
Constant Relative Risk Aversion (CRRA) Utility
Utility uy(()
Risk aversion parameter noutral)
(moce avens))
-2-
y- (logi
Figure 7: Visualization CRRA utility functions.
contracts yields ay /q shares (i.e., the pay off is ay if the outcome is Yes); similarly, ax g) shares of No pay off if the outcome is No. Given belief (predicted probability) p and risk preference 7, the optimal allocation is the solution to
max ON 20 +ay=1
p- Un
ay
(1 (1-
-) I [oN)
(5)
This objective is the expected utility of betting under p. Closed-form solutions. For > 0, the optimization admits a closed-form solution:
ay (7)
ai = xN(()) = 1 aÿ.
(6)
1 - p (1 1) (1 (1-p)
At 7 0, we take the limit y + 0+, yielding
[1, if pf>29.
a (0) lim ay
7-+0
10 if pS
which recovers the strategy described the beginning of this subsection. At the other extreme, when y = 1, the optimal strategy is to allocate ay = and a² = 1- p. i.e., to bet proportionally to ones own probabilities, independent of the market price. Overall, this unified framework highlights how different betting strategies arise from different risk preferences. The "all-in" rule in the main text is simply the risk-neutral optimum.
C.4 BRIER SCORE VS ECE.
As discussed in Section 3, calibration errors and Brier scores measure different aspects of forecasts.
We start with a simple example to illustrate that a better-calibrated forecast can have a worse Brier score. Suppose there are two markets M₁ M2 with ground- -trththhhhhhh probability pf 0.9, pt 0.1. Alice's 's prediction is p = 1 = 0 whereas Bob's 's prediction is p" = 0.5 for both markets. is difficult
that Alice has better/smaller Brier score, but she ECE. Specifically, since Alice has two different probability predictions, her ECE is {Ip p +
have a worse/larger
|p: - 15')) -3-pp'1 = 0.1 whereas Bob has single probability prediction with the ECE equaling ] [pt + 5s 2p-pp" 0. That is. Bob's prediction of 0.5 probability is perfectly calibrated and reliable as, conditioned on this probability, events' average probability is indeed 0.5.
A forecaster with smaller ECE but larger Brier score induces better risk-adjusted decision making. Consider an event with binary outcome o E {0. and a ground-truth probability p" 0.5 to have 1. Suppose forecaster A prediets probability PA = Pr(o 1) which
22

---

Published as a conference paper at ICLR 2026
correlates with the realized outcome in the following way
(1, 1), with joint probability 0.5(1 e)
(0,0), with joint probability 0.5(1 e)
(PA: 0))
(1,0), 0), with joint probability 0.5e
(0, 1), with joint probability 0.5c
where 6 is a small value. In other words, forecaster A predicts PA = or PA = 0, each with probability 0.5 and with fraction of errors. These are extremal and uncalibrated forecasts but is almost always correct. It is easy to verify that the Brier score of pA is 0.5c x (1- 0)2 x (1- 0)² C, whereas expected calibration error (ECE) is 0.5 x (1-c)) +0.5x 11 (1-c)) 0.5c².
Next consider forecaster B predicts probability ps = Pr(o = 1) which correlates with the realized outcome in the following way
(2/3, 1), with joint probability 0.5
(PB.O)) (2/3.0), 0),
with joint probability 0.25
(0,0), with joint probability 0.25
That is, forecaster B predicts pB = 2/3 for 75% the time, and predicts ps = 0 otherwise. is easy to see that p is perfectly calibrated with ECE- 0, but has Brier score equal 0.75 x (2/3
1)a + (2/3 0) = 1/6.
Clearly, pa is more well-calibrated than PA but has much worse Brier score, especially when is small. However, we show that pg is better for risereverse decision making, than PA- Suppose decision maker has 1 unit of fund and would like to buy contracts for event E's outcome. Suppose the current market price for the contract o 1 and 0 are both 0.5 per contract. Adopting a classic risk-averse utility function, we assume her utility U (x) = log(æ) (i.e., amount of return means (x)) utility to her).
If the decision maker adopts forecaster A, since A makes extremal forecasts, even under risk-averse utility log(r) her strategy is to spend all the unit of fund to buy two units of contract o = whenever PA = 1, and buy 2 units of 0 otherwise. With probability of losing all her fund, her expected risk-adjusted utility is thus (1 (2) + (O), which goes to oc when U (z) = O, regardless how small is.
If the decision maker adopts the perfectly calibrated forecaster B, she would spend all fund to buy 0 contract when pø 0, but split her unit of fund to buy 2B, of 2B, of o = contracts, so as to maximize her risk-adjusted utility as max Bo B,= =1 (2B + (2.Ba) Simple calculation shows that optimal B₁ 2/3, leading optimal risk-adjust utility 2/3U(4/3) (4/3) + 1/3U (2/3) = log( 27 )/3, which is strictly larger than her initial utility log 1 = 0.
contracts and
Remarks. A few remarks are worthwhile mention about this example. First, forecaster PA with small will be better for resk-neural decision maker. However, in this case, while the decision maker will have larger expected utility, she will have bear the possibility of losing all her unit of fund, though also enjoy the possibility of doubling her fund to 2 units. This is often not desirable in real-world applications, such as financial decisions in which risk control extremely important. In such cases, more well-calibrated forecaters are preferred. Second, our example has extremely negative utility- -o = (0). This helps us to cleanly highlight the underlying conceptual message, but simple tweak to the utility function can remove that extreme situation yet arrive at the same conclusion.
C.5 BRIER SCORES VS. MARKET RETURNS
We provide a simple example prove the following fact. Fact There exist a binary prediction market with two forecaster A, B such that A has strictly higher market return than B but has strictly worse/higher Brier score.
Proof. Suppose we bet on a single event with binary outcomes, with ground-truth probability of Yes being 0.6 (for the event to be realized), and prediction market price 0.5. Consider two different probabilistic predictions of this events Yes realization: model A predicts 0.45 and model predicts 0.9.
The expected Brier Score of A is:
0.66 (0.45- 1)² + 0.4 (0.45 0) 0.2625 0.7375
23

---

Published as a conference paper at ICLR 2026
The expected Brier Score of B is:
1 0.6 (0.9 1)2 + 0.4 (0.9 - 0)²] - 0.33 0.67
So A has higher Brier Score. However, because A predicts 0.45, lower than the 0.5 prediction market price, A will short Yes (or equivalenty, buy No) at 0.5. Meanwhile, B predicts 0.9, much higher than the prediction market price, so B will buy Yes. Respectively, A and Bs expected returns will be:
0.6 (-1+ + 0.4 (0.5) = 0011
0.6 (1-0.5) 0.4 0...0...0000 (-0.5) 0.1
Therefore, A has higher Brier score, but lower returns.
This example uncovers key difference between the two metrics. The Brier Score measures how close prediction is to the ground truth and, importantly, has nothing to do with market prices. Since As prediction above is closer to the ground truth, receives a higher Brier Score. However, returns on the market are not only driven by the true probability but also by the market price. Therefore, even though Bs prediction is exaggerated, it lies on the correct side of the market mispricing (buying "Yes" when the outcome is more likely than price suggests), thereby achieving higher returns.
C.6 MARKET RETURNS VS. CALIBRATION
In Section 3, we introduced the evaluation pipeline in Prophet Arena, including how market return is calculated based on market decisions (buying either the YES NO contract), which are derived from predicted and market-implied probabilities (pj. Qj) as well as the realized outcome of under the following decision rule: buy YES contract whenever Pj > qj. and buy NO otherwise.
Here we further make the connection between a (perfectly) calibrated predictor and market returns conditioning on the type of decision made. Before stating the main result, the following mild assumptions and definitions are needed.
Assumption C.1 (Data Generation). We assume that the triples (o,, qj) are drawn i.i.d. from some joint probability distribution D. The expectation E[|]] below is taken over D.
Assumption C.2 (Sufficiency of Calibrated Predictors). Perfect calibration implies that Vc E [0, 1].
E|o; P, = c] = PPoo,
P; = c| - Plo; =0 - Pj = c| =
We futher assume that a calibrated predictor is sufficient for the market outcome: the outcome o, is conditionally independent of the market price qj given the forecast pj. i.e. E[o; P/ 9,1 = Bio; p,]] Definition C.3 (Symmetric Disagreement). We say a predictor is symmetrie (against the market) if the probability distribution of dj P, - 9j is symmetric about zero. We let f(x) denote the p.d.f. of d, and the definition implies (1) f(z) = f(-x) œ) for all x E R. and (2) P(p; > 431 Pip, S g;]- In other words, dj represents the "disagreement" that the predictor perceives over the market, and symmetric predictor is not consistently more conservative or more aggressive.
Theorem C.4 (Symmetric Expected Returns for Calibrated Predictors). Let Ry = 03 qj be the return on a YES contract and RN -0-o,) - ((1 - 93) Oi - be the return on a NO contract. For a predictor that is both perfectly calibrated and symmetric against the market, the expected returns conditioned on the betting decision are equal, i.e.
E Pj > 93 = E( P3 < <ggl
Proof. Let Ey = Eio, 4j p2 > 91 and EN = =lg 0j pj < 4z] denote the average returns on YES and NO contracts, respectively. We begin by analyzing the expected outcome 0j under each condition. By the Law of Iterated Expectations and the sufficiency of the calibrated predictor (Assumption C.2), we can replace the outcome oj with the prediction Pj inside the expectation:
Eio, P3 > 41 E E oo Рソ 9;] Pj > 4,] = Elp, Pj > 9/l-
Eio, Рj < q = E [ p,. 9,] P, < 9,1 = Epp, Pj < 9,1-
Substituting these back into the expressions for Ey and Ex:
Ey Eip; P; > 9/1 Elg; p, > qj} = Elp; - 4 P, > 4/..
E, Egg
p3 4] Elp, P3 < gj Elg; P5 Pp < <))
24

---

Published as a conference paper at ICLR 2026
The theorem is proven if we show that Elp, qj Pj > 9/1 Elg, Pj
Pj
qzl. Let
dj = Pj q be the disagreement variable. The equality becomes:
E(d, d 0] EB|dd d3 < 0]
We show this using the integral definition of conditional expectation. The symmetric disagreement
assumption implies f(xx) = [(-x) and P(dj > 0) = P(dj < 0). Consider the right-hand side:
[ =x) dx
11-dyld < 0] P((d <0)
Applying the change variable u = I; we have x = u and dr - du. Therefore
- (-)))(-u)(ddu) -So
_0 us( (-u(du
P(dj < 0)
P(dj < 0)
P(dj < 0)
Using the symmetry properties -(-w) = (u)) and P(dj < <0) = PP(d > 0), this is equal to:
_0 uf(u)du
P(d, > B(d, [d > 0).
Thus, we have shown E| E[-djidj < 0] R[d 0j. which concludes the proof.
C.7 BOOTSTRAP CONFIDENCE INTERVALS
Most metrics in this paper can be summarized by the following two steps:
1. We first calculate collection of scores $1....,57 SN either the event level or the market level, where N denotes the total number of events markets.
2. We then aggregate these scores into a single statistic via a mapping T: RRN - IR (typically the arithmetic mean).
To assess the uncertainty of the resulting point estimate, we construct nonparametric bootstrap confidence interval (CI) level (1 a)% (DiCiccio Efron, 1996). Specifically, we form symmetric interval around the point estimate using bootstrap resampling. A pseudo-code for our implementation given below in Algorithm
Algorithm Symmetric Nonparametrie Bootstrap CI Centered at Point Estimate 0
Require: Data $1, sy; statistic T(-); number of bootstrap replicates B; target level 1 as Ensure: Symmetric CI (0 h, 0 + h] and achieved bootstrap mass k/B. B.
1: Compute point estimate - T($1....,,88
2: for b = to B do
(b) by sampling with replacement from {s 3: Draw a bootstrap sample {s1 (b) ..... SN
4: Compute bootstrap re-estimate (b) + T(s) (hh)
(h))
5: end for
6: Compute deviations do + (g-(b) - 0 for = I,..., B.
7: Sort {do} into order statisties d(1) S ... S d(m)
8: k +- [[1 a) B).
9: h+ d(R)-
return Symmetric CI 1 h, 0 h).
C.8 ROBUSTNESS TO EVENT-CATEGON REBALANCING
To examine whether our results are sensitive to the distribution event categories, we perform robustness analyses using rebalanced subsets of the evaluation data. The main dataset is heavily skewed toward Sports events; to assess the impact of this imbalance, we construct alternative evalu- ation sets in which the proportion of Sports events is reduced while still retaining mix of the other categories. One subset retains Sports as the largest category but reduces its dominance to roughly 50% (Sports: 0.5, Entertainment: 0.15, Polities: 0.15, Other: 0.2). Another subset ensures that all
25

---

Published as a conference paper at ICLR 2026
four categories equally represented (25% each). All analyses only submissions or before 2025-09-10
Table 6 reports the Brier scores across the original and rebalanced evaluation sets. The relative dering of the top five models remain unchanged across all distributions, confirming that their com- parative performance is robust event-category composition. As the proportion of Sports events decreases, overall Brier scores improve slightly, suggesting that Sports events are intrinsically more challenging predict.
Table Brier scores across the original and rebalanced evaluation subsets. Numbers in parentheses denote 95% confidence intervals. The "Moderate" subset contains 50% Sports events, while the "Full" subset has 25% Sports events.
Forecaster
Original (N=816)
Moderate (N=250)
Full (N=152)
GPT-5R
0.179
(=0.088)
0.146 (+0.013)
0.128
(+0.0155
Grok 4R
0.186
(=0.009)
0.156 (#0.014)
0.141
(#0.018)
Claude Sonnet 4 (Thinking)
0.190
(=0.009)
0.162 (#0.016)
0.148
(#0.020)
Gemini 2.5 Flash (Reasoning)
0.195
(=0.009)
0.165 (#0.016)
0.157
(+00021)
Llama 4 Scout
0.230
(=0.011)
0.202 (+0.0199
0.210
(+0.025)
Market Baseline
0.188
(=0.008)
0.164 (+0.010)
0.149
(+0.013)
As such, these results provide evidence that the conclusions drawn the main text regarding LLM performance are robust to variations in the event-category distribution.
26

---

Published as a conference paper at ICLR 2026
D ADDITIONAL EXPERIMENT RESULTS
D.1 FULL EVALUATION RESULTS
Forecasting Loss
Calibration Error
Market Return
LLMs
Brier (95% CI)
Rank
ECE
Rank
1 Average (95% CT)
Rank
GPT-5" (High) A
0.184 (# 0.006)
0.042
0.943 (# 0,042)
6
GPT-53
0.187 (# 0.005)
0.044
6
0.890 (+ 0.040)
13
Market Baseline
0.187 (= 0.006)
0.069
19
0.899 (# 0.043)
10
GPT-5* (Minimal)
0.188 (= 0.006)
0.036
2
0.869 (# 0.044)
19
03
0.188 (# 0.005)
0.030
1
0.959 (#: 0.109)
5
Grok-48 A
0.189 0.005)
0.043
6
0.864 (#: 0.052)
21
Grok-3-Mini
0.189 0.006)
0.046
7
0.869 (# 0.043)
20
GPT-4.1
0.192 0.007)
0.053
10
0.907 (# 0.035)
8
Gemini 2.5 Pro
0.193 (= 0.007)
0.061
14
0.876 (+ 0.050)
17
Claude Opus 4.1
0.193 (= 0.018)
10
0.054
11
0,982 (# 0.093)
1
Claude Sonnet 4
0.194 (= 0.006)
11
0.041
0,909 (# 0.101)
7
o3-Mini
0.195 (= 0.005)
12
0.046
8
0.897 (#: 0.046)
12
04-Mini (High)
0.196 (=: 0.006)
13
0.062
16
0.874 (#: 0.040)
18
Kimi-K2
0.197 (= 0.008)
14
0.048
9
0.966 (#: 0.124)
Gemini 2.5 Flash
0.197 (= 0.007)
0.067 15
17
0.883 (# 0.053)
15
GPT-4o
0.198 (= 0.007)
16
0.058
12
0.970 (+ 0.104)
DeepSeek-V3
0.201 (=: 0.008)
17
0.061
15
0.963 (#: 0.103)
4
Gemini 2.5 Flash
0.203 (= 0.006)
18
0.073
20
0.859 (# 0.042)
22
Llama-4-Mavrin
0.208 (= 0.008)
19
0.067
18
(1
9
Llama-4-Scout A
0.219 0.008)
20
0.060
13
0.805 (+: 0.040)
24
Gemini 2.0 Flash (Lite)
0.224 (= 0.013)
0.091
22
0.855 (# 0.074)
23
Gemini 2.0 Flash
0.224 (= 0.013)
22
0.079
21
0.876 0.078)
16
Qwen3-235B DeepSeek-R1*
0.234 (+ 0.007)
0.118
23
0.898 (+: 0.111)
11
0.303 (# 0.018)
24
0.165
24
0.884 (#: 0.058)
14
Table The full evaluation results for all 23 LLMs (including variants) and the market baseline. denotes models selected for presentation in the main text.
D.2 SHARPE RATIO FOR AVERAGE RETURN
As discussed Section 3.2 and Table 7, Average Return often exhibits wide confidence intervals, reflecting the high variance of event-level payoffs. most events, models earn nothing or only marginal gains, while in rare cases they achieve outsized returns. To account for this variability, we complement Average Return with the Sharpe ratio (Sharpe, 1998), a standard metric that normalizes expected returns by their volatility. Formally, for asset returns Rq the Sharpe ratio is defined as
S. ERR. - Ra]
(8)
Varl Rd]
where R2 denotes a reference risk-free return. In the Prophet Arena setting, Rt corresponds to the payoff from each event under the trading strategy of Appendix C.3, and we set Rs 1. the return from abstaining (i.c., keeping the budget unbet). The expectation and variance Eq. (8) are estimated by the sample mean and variance over events.
Sharpe ratios for all evaluated models are reported in Table 8, providing a volatility adjjsteed com- parison of economic performance.
27

---

Published as a conference paper at ICLR 2026
LLMs
Sharpe Ratio
Rank
LLMs
1 Sharpe Ratio
Rank
03-Mini
-0.0036
T
Gemini 2.5 Flash
-0.0747
12
GPT-5"
-0.0074
Grok-3-Mini'
-0.0772
13
Gemini 2.5 Flash A
-0.0100
o4-Mini
-0.0868
14
Gemini 2.5 Pro
+0.0139
DeepSeck-V2 V3
-0.0952
15
o3a
-0.0141
Qwen3-235B
-0.1080
16
GPT-5* (High) A
-0.0407
GPT-4o
-0. 114
17
Grok-43 A
-0.0467
DeepSeck-R1
-0.1133
18
GPT-4.1
-0.0515
8
GPT-5" (Minimal)
-0.1269
19
Llave---Maerrick
-0.0549
Gemini Flash
-0.1684
20
Claude Sonnet 4 A
-0.0574
10
Kimi-K2
-0.2456
21
L-Soot4-Scout A
-0.0599
11
Gemini 2.0 Flash (Lite)
-0.2600
22
Table Sharpe ratio performance for all 22 LLMs. A denotes models selected for presentation in the main text.
D.3 PROBABILITY ELICITATION METHODS
Section 2 introduced our default approach: directly prompting an LLM to verbalize the probability that market resolves to Yes. In this experiment, conduct ablation studies over alternative confidence estimation methods. The goal is twofold: (i) test the robustness of our default prompt against reasonable variations, and (ii) illustrate how Prophet Arena can comparing black-box confidence estimation methods under forecasting settings.
testbed for
Setup. We evaluate the five representative LLMs from main text on the PROPIIET-ARNA, SUBSET- 100 dataset. Metrics are Brier and ECE scores. We consider two families of confidence estimation methods (seven variants total):
1. Verbalized Probability (Tian et al., 2023)
Prompt variation: modify default prompt to make it: (A) more concise, (B) more verbose, and (C) rewritten by another LLM (Grok 4). Key logistics and formatting instructions preserved in all variations. These variation prompts
available in Appendix F.4.
al., 2023): for each market, we average the probabilities elicited from the default and all variation prompts (i.e. (A), (B), (C) above).
Prompt emsemble (Wightman
Bi-direction * (ours): in addition to eliciting Pij (probability of Yes) using default prompt, also elicit pi (probability of NO), and calibrate via (pij (1 pi2)).
2. Self-cousistency (Wang al., 2022)
Unweighted: instead of directly asking for probability, we repeatedly query the model 10 times for Yes /No decision; probability is the fraction of Yes answers.
Weighted (Taubenfeld et al., 2025): we further supplement each Yes/No with confidence score."! 11 Probabilities are formed by confidence-weightte aggregation.
Results (see Table 9). Among the verbalized probability methods, observe that all models exhibit strong robustness to prompt variations. For accuracy, Brier scores for all models vary by less than one standard deviation (ss 0.01). As result, GPT-5 ranks the highest under all methods, and Gemini 2.5 Flash (Thinking) consistently achieves second place in 5/6 cases. Prompt ensemble does not lead to substantial improvement, since elicited probabilities are already similar across vari- ations. For calibration, GPT-5 and Llama 4 Scout are consistently the best and the worst models, regardless of the prompting method. Despite slightly larger fluctuations among the ECE scores, no single method dominate the others for all models (i.e. achieves the best scores on all columns). Our original Bi-direction method improves calibration (over the default) on 4/5 models, supporting the view that LLMs tend to be overconfident toward Yes outcomes.
In contrast, self-consistency methods result in significantly lower accuracies and yield mixed calibration benefits. With 10 rollouts, the unweighted variant produces coarse-grained probabilities at a resolution of 0.1, limiting accuracy despite incurring higher compute cost. The weighted variant
This confidence reflects uncertainty about the chosen answer, not direct market probability (e.g., low confidence in Yes signals indecision, not belief in NO).
28

---

Published as a conference paper at ICLR 2026
Method Type/ Name
Grok
Gemini 2.5 Flash
Claude Sonnet
GPT-5
Llama 4 Scout
Verbalized Prob
Default
0.186/0. 18
0.166/0.036
0.173/0 373
0.165/0.020
0.
Variation A (Concise)
0.180/0.100
0.172/0.036
0.169/0.00 035
0.162/0.021
0.192/0. 134
Variation B (Verbose)
0.178/0. 124
0.166/0.039
0.172/0. 040
0.160/0.028
0.199/0. 164
Variation C (Rewrite)
0.176/0. 123
0,167/0.027
0.172/0 039
0.159/0.016
0.195/0. 167
Ensemble
0.177/0.117
0.165/0.032
0,170/0, 773/0,043
0.160/0.024
0.192/0. 142
Bi-direction
0.180/0.101
0.164/0.031
0. 0.165/0.028
0.158/0.023
0.203/0.144
Self-consistency
Unweighted
0.238/0. 115 0.205/0.091
0.231/0. 110
0.241/0.071 0.189/0.050
0.239/0.071
0.267/0. 129
Weighted
0.201/0.067
0.181/0.046
0.214/0.125
Table Evaluation Results for Different Probability Elicitation Methods Each cell contains pair of Brier\ /ECE scores. Bold denotes the best score for each row, underline for each column.
partially alleviates this by using finer- grained confidence signals, producing noticeable calibration gains but still trailing verbalized methods in accuracy.
To sum up, these results show that (i) LLMs are generally robust against prompt ablations, justifying our default prompt choice, and (ii) Prophet Arena provides natural benchmark for evaluating and contrasting future confidence estimation/caliivation methods.
D.4 REASONING CONSISTENCY OF LLMS
In addition the primary relative (Brier score) and absolute (Average return) metrics, certain types of forecasting events also enable us to evaluate the consistency an important component of reason- ing- of LLMs. These metries can be calculated solely by looking at the potential event outcomes and the LLM probabilities given to them. Below we give two concrete examples of such consis- tency metrics.
Logical chain score." 12 Consider the forecasting event "The bitcoin price by the end of 2026", and two of its outcomes are "(A) The bitcoin price is above $200,000* 0000 and "(B) The bitcoin price
above $220,000", respectively. No matter how good an LLM is at predicting the probabilities for (A) & (B), we know that anyone with consistent reasoning will give (P{ Z P((B) since the latter outcome logically implies the former. We denote such a relationship as logical chain, or
(B) + (A). Obviously, this logical chain can contain more than two outcomes, so we call a logical
chain S = (Si) + ... + (ST) maximal whenever it satisfies both:
1. For all 1 S 1 T, we have (S,) - (S++1),
2. No other outcome (K) satisfies (K) - (Si) or (Sr) (K)
A single event might contain multiple maximal logical chains Si,..., S, with lengths Ti,... Tn. For an LLM with probability PP[())) for outcome (A), its logical chain score for this event is given by score(S.), where
score (S, = (S,1 - >... Str.)) = T. - Σ ((P((S)) & P((S)+1]|}
with 1(.) being the indicator function. The final logical chain score is then averaged over all events with at least one chain. We adopt an LLM-judge (Gemini-22 5-Flash) automatically detect the maximal logical chains for all our events.
Mutually exclusive score. In certain forecasting events, the potential outcomes mutually clusive, meaning that exactly one outcome can occur. For example, in the event "Who will win the NBA championship in 2026?", the possible outcomes could be the teams, where only team win. A maximal set of mutually exclusive outcomes is a set where
1. each outcome is distinct and all outcomes are mutually exclusive,
2. the event will resolve and only outcome in the set.
This placeholder name. Feel free suggest better
29

---

Published as a conference paper at ICLR 2026
If such maximal set S {(S1). (Sm)} with size exists for forecasting event, calculate the mutually exclusive score at this event as:
SCOT Σ PI(S.) =1
(10)
(In practice, we allow the sum to deviate slightly from 1 with some tolerance level e).
We evaluate this score over all events where mutually exclusive outcomes are defined. The identifi- cation of maximal set is performed using the same LLM-judge (Gemini 1iaiiii2... 2-5Flash)
LLMs
Mutually Exclusive Consistency
Logical Chain Consistency
DeepScck-RI"
0.996
0.987
o4-Mini"
0.995
0.998
GPT-5" (High)
0.995
0.999
Gemini 2.5 Flash
0,995
0.997
Gemini 2.5 Pro
0.995
0.995
Claude Sonnet 4"
0.995
0.987
DeepScek-V3
0.994
0.973
Grok-4"
0.994
0.994
GPT-4.1
0.994
0.973
Llama Maverick
0.994
0.981
Llama Scout
0.994
0.979
Grok-3-Mini"
0.994
0.901
Qwen3-235B
0.994
0.988
GPT-4o
0.994
0,930
GPT-5" (Minimal)
0.994
0.962
GPT-5 (Medium)
0.994
0.990
o3-Mini
0.994
0.930
Gemini 2.0 Flash
0.994
0.990
Kimi-K2
0.993
0.984
o3
0.993
0.996
Gemini 2.0 Flash (Lite)
0.993
0.911
Table 10: Consistency scores for the 22 LLMs. Most LLMs evaluated have exhibited excellent performances in both consistency metries, indicating their mature logical reasoning skills.
D.5 VARIABILITY IN MODEL FORECASTS DESPITE IDENTICAL INPUTS
While all tested LLMs receive the same market data and news sources, they differ substantially in how they combine and weigh each piece of information, resulting diverse prediction patterns. Fig. 8 shows averaged pairwise differences in event predictions, and the generally large differences across models suggest that LLMs reason about events fundamentally different ways, even when given identical inputs.
Interestingly, even within model families, clustering patterns also vary notably. For example, GPT-5 variants produce relatively homogeneous predictions, reflecting similar training and reason- ing capabilities. contrast, Llama 4 Maverick and Scout, despite belonging to the same model family, exhibit the second largest average pairwise difference (0.43). This divergence illustrates that sharing model family does not guarantee similar predictions, nor does it imply identical reasoning processes.
D.6 CONSERVATISM OF MODEL PREDICTIONS RELATIVE TO MARKET-IMPLIED PREDICTIONS
Figure shows that all tested models exhibit conservative tendency, assigning significantly lower probabilities than markets on numerous events. The magnitude the conservative tendency varies by model, but the general trends Section 4.1.3 discussed in observed persist.
30

---

Published as a conference paper at ICLR 2026
300
8:1
631
9.2
9.0
oree GU
616
4G2
RM
521
Grdd 25 Yuat
A28
821
624
124
822
M4
137
639
023
A31
LC
E00
Gamot 00
Genaid Road
Figure 8: Pairwise divergence of model predictions across tasks. Heatmap shows the average L2 distance between probabilistic predictions of each model pair. GPT-5 H, M, labels represent high, medium, and minimal reasoning efforts, as explained by Appendix B.6.
GPT-5 (Medium)
GPT-5 (Minimal)
GPT-4o
Pasiy
Pasiy
Model prob: (Wimme)
1.
Parky
0.5
0.6
groh
04
04
804
0.2
0.2
0.
0.0
0.0
0.0
0.0
02
0.4
06
0.1
10
0.0
0.2
0.
06
10
0.0
0.7
64
05
10
Market grob (Wamner)
Market prob (winner)
Market poob (wimner)
Gemini
GPT-o3
1.0
Pasiy
Model peoh peod geiner))
Pasiy
Mo-del peoh (wine))
Pasky
0.4
0.
proh
04
Marcce (Wisnes)
Mutket
Market
Figure 9: LLM prediction probabilities compared to market baselines. Scatter plots show the relationship between market-implied probabilities and aasigned probabilities. Points on the diagonal line correspond to events where the model prediction matches the market probability.
D.7 EVALUATING REASONING USING AN LLM-AS-A-JUDOL FRAMEWORK
To directly evaluate LLMs reasoning processes, we employ an auddas-a-judge framework (Zheng et al., 2023) to assess the soundness of their reasoning methodologies. Our evaluation proceeds in two stages: (1) eliciting explicit reasoning from the prediction model, and (2) systematically evaluating this reasoning with an independent evaluator.
The assessment framework encompasses five critical dimensions, each scored on an 1-5 point scale (where and 5 indicate poor and excellent performance, respectively):
1. Source selection: Assessment of how effectively models incorporate provided sources and reliability of their source selection criteria.
2. Evidence extraction from selected sources: Evaluation of the model's ability to extract rel- evant evidence from sources and demonstrate sophisticated interpretation beyond surface-level analysis.
3. Reasoning synthesis: Analysis of how extracted evidence is integrated into coherent justifica- tions, including the model's approach combining and weighting disparate pieces of evidence.
31

---

Published as a conference paper at ICLR 2026
4. Reasoning-fo-prediccton Assessment of how effectively the reasoning process is translated into the final probabilistic prediction.
5. Recognition of prediction uncertainty: Examination the model's capacity to identify and appropriately account for uncertainties and potential counterarguments within their analysis.
To enhance evaluation reliability, incorporate human expert-assessed reference evaluation of an external event not included in the dataset, which serves as a grounding benchmark. Additionally, lower the temperature setting to 0 for the LLM judge (Claude Sonnet 4) to improve response consistency. We include the full prompt in Appendix F.2.
We include the full table of reasoning evaluations with wider range of LLMs below. The same trends observed in Section 4.3 are evident here: models reach pear-parity in source utilization, evi- dence extraction, and uncertainty analysis, while exhibiting substantial disparities in reasoning syn- thesis and reasoning-to-pue -rredcccionn alignment. These latter dimensions are the primary drivers of differences in overall predictive performance. As such, the findings further suggest that the de- velopment of future prediction agents should prioritize advances in higher-order reasoning and the alignment reasoning with probabilistic forecasts, rather than focusing on marginal improvements in retrieval evidence handling.
LLM
Sources
Evidence
Reas. Synth.
Align.
Uncert.
Average Score
GPT-5* (High)
3.69
3.66
4.14
3.97
3.94
3,88
03
3.71
3.74
3.93
3.78
3.87
3.81
Gemini 2.5 Pro
3.70
3.69
3.39
3.92
3.95
3.73
GPT-5 (Medium)
3.69
3.65
3.69
3.66
3.94
3.73
GPT-5* (Minimal)
3.69
3.64
3.26
3.58
3,90
3,61
Gemini 2.5 Flash
3.57
3.66
3.67
3.74
3,57
Grok 4
3,40
3.51
3.33
3.48
3.66
3,48
Claude Sonnet 4
3,53
3.47
2.93
3.39
3,75
3,41
Llama Maverick
3.14
3.29
2.43
2.14
3.14
2.83
GPT-4o
3.07
2.99
2.32
2.59
2,96
2.79
Llama Scout
2.97
2.88
2.29
2.37
2.87
2.68
Full table on LLM performance on reasoning evaluation criteria across dataset events. Each dimension is scored on a standardized 5-point scale, where and 5 indicate poor and excellent performance, respectively. Average scores presented for each model, with bold values indicating the best-performing model for each criterion. Models are ordered by descending overall average score.
Table 11:
D.7.1 VALIDATING LLM-AS-A-JUDGE VIA HUMAN RATINGS
To validate the LLM-as-a-judge system and check for alignment bias, we conducted a blinded human adjudication study. We randomly sampled 170 events with LLM-generated reasoning traces and hid both the model identity and LLM judge outputs from human raters. Each reasoning trace was independently scored by humans on 1-5 scale across the five rubric dimensions (Sources Used, Evidence Extracted, Combination & Weighting, Uncertainties Counterpoints, and Mapping to Final Probabilities). The rubric provided was identical to that given to the LLMss
Overall, we find a high degree of concordance between human ratings and L.L--a--ajuudee scores. As shown in Table 12, mean absolute differences between human and LLM ratings ranged from 0.37 to 0.44, with an average difference 0.42, and variances between 0.27 and 0.36. These values strikingly low given the 1-5 rating scale: all differences are, average, substantially smaller than 1, meaning that the LLM judges score deviates from human judgment by less than half point. Moreover, across all rubric categories, over 94% of human ratings were within I point of the LLMs score as seen in Fig. 10, underscoring the tight alignment between automated and human evaluation.
D.7.2 (LACK OF) EFFECT OF SCAFFOLDING PROMPT ON PREDICTION
Table 13 reports the differences in Brier score between the predictions with the reasoning scaf- folding prompt and the predictions with the default, non-scaffolding configurations. Overall. the differences are insignificant for all models, indicating that the probabilistic forecasts remain largely stable regardless of the prompt type. This suggests that the enhanced reasoning elicitation prompt
32

---

Published as a conference paper at ICLR 2026
Rubric Category
Mean Absolute Difference (0-4) (Variance)
Sources Used
0.43 (0.30)
Evidence Extracted
0.42 (0.27)
Combination Weighting
0.43 (0.36)
Uncertainties Counterpoints
0.44 (0.28)
Mapping To Final Probabilities
0.37 (0.29)
Table 12: Human ratings for rubric categories, with variance shown in parentheses. Mean absolute difference is calculated as the average human score ences are on a 0)4 scale.
L.M score across the 170 events. Differ-
Evidence Extracted
Dimension
Comtinatio Waighting
63.2%
94.2%
100.0%
Unoowrentt mentien & Crurtapartt
57.3%
98.2%
100.0%
Mapping to Final Probebli
64.9%
98.2%
99.4%
Cloas (41)
Modonate (87)
Agreement Threshold
Figure 10: Agreement matrix showing the percentage of human ratings that lie within exact, mod- erate (=1), and larger (+2) thresholds the LLM-judge ratings for each rubric dimension.
did meaningfully improve prediction performance, implying that the prompt primarily serves as a structured summary rather than enhancing the models underlying predictive capabilities.
Model
Brier Difference
GPT-5 (High)
-0.0112
GPT-55 (Medium)
0.0011
GPT-58 (Minimal) 03
0.0017
-0.0018
Gemini 2.5 Pro
-0.0080
Gemini 2.5 Flash
-0,0001
Llama 4 Maverick
0.0095
Llama 4 Scout
0.0138
Grok-4R
-0.0186
Claude Sonnet 4R
-0.0005
DeepSeek-RII
0.0057
GPT-4o
0.0053
Table 13: Brier differences between the reasoning scaffolding prompts and non-scaffolding config- urations.
D.8 KNOWLEDGE INTERNALIZATION
D.8.1 KNOWLEDGE INTERNALIZATION EVENTS
We sample 100 events from Kalshi (Inc., 2025), with market close time before October 2023 (i.e., before all model knowledge cutoff dates. Note that despite that many popular events on Kalshi Sports events of August 2025, sports betting not legal on Kalshi until 2024 (Wilmot,
33

---

Published as a conference paper at ICLR 2026
2025). The sampled past events span the categories available on the platform during that period and exhibit differing levels of temporal gramularity. Some events target a specific timestamp or date (e.g., NASDAQ price on August 20, 2023), while others coarset, period-level questions without a single focal timestamp (e.g., Will WTI crude oil prices decrease in Q2 2023?).
Event Category Distribution for Memorization Experiments
30
count
20
10
Financials
Economics
Politics
Entertainment
Climate and Weather
category
Figure 11: Event distribution for memorization experiments
D.8.2 KNOWLEDGE INTERNALIZATION PROMPTS
We three complementary prompts models' knowledge internalization.
1. Prediction prompt (no sources): the original forecasting-style prompt used in Prophet Arena without sources context (Appendix F.3.2). Although framed as forward-looking prediction, all of these past events are in fact already represented in the model's training data. Success in these cases indicates that the model can implicitly recognize the event as historical and draw on its internalized knowledge to answer correctly. Failure, in contrast, highlights gap between memorization and reasoning: the model may know the fact but still treat the prompt purely as a forecasting task, leading to mis-recall
2. Prediction prompt with sources: the same forecasting prompt with an additional block (Ap- pendix F.3.3), but augmented with event-specific sources. In this setting, the model is no longer reasoning only from internalized knowledge: must integrate retrieved evidence with what it already "remembers." This setup tests whether the model can align its internal recall with external evidence, and whether retrieval corrects, reinforces, or conflicts with its memorized knowledge.
3. Recall prompt: a specialized prompt that explicitly frames the task as recalling a past outcome (Appendix F.3 This isolates the models internalized knowledge, revealing whether it has retained coarse precise details about prior events.
D.9 How GOOD ARE LLMS AT FINDING SOURCES?
We evaluate models' nource-finding ability the set of past events (Appendix D.8.1), using the search prompt. Because these events have resolved, accurate information is publicly available; com- petent search should surface the correct evidence and hence, average, improve the quality of the models' recall. In Fig. 12b, we observe that the addition of sources improve LLM recall quality in certain events but not others. In fact, the figure shows that the events hardest to recall often becomes worse from the source-augee augmee-d variant. One likely explanation is that retrieved sources, while factual, introduce noise or extraneous details that interfere with the models internalized recall and reasoning. Events requiring precise, fine-grained recall (e.g., financial indicators or political ap- proval ratings) tend suffer when sources are added, likely because the retrieved evidence contains multiple overlapping numbers, dates, or conditions that confuse the model. By contrast, more salient and broadly covered events (e.g., major stock index movements or entertainment outcomes) gener- ally improve with sources, since the found sources likely have lower variance and higher accuracy.
This demonstrates that even on past events that have closed, LLM searchers not yet possess the ability to accurately pinpoint the most useful sources for fine-granularity events.
34

---

Published as a conference paper at ICLR 2026
Impact Adding Sources on Performance Events
Pefferanee Guange
Biner Scote by Calegory and Proeupl Varaal
(Dter cherrad
Variast
Recall procee
Prediction pronpe
Prodiction groape
Moun
Climade and Wettee
Reenomic
Betoniionnn
Fimar vil
Politca
Cregary
Figure 12: Impact of adding sources to context on past events. The low-performance events refer to past events that the models have worst recall on. The average Brier score change is computed by evaluating the difference between when models are provided with searched sources vs. when models are solely recalling based on memory.
E
CASE STUDIES
E.1 DIFFERENCES IN PREDICTION WITH THE SAME INFORMATION
In the following example, the LLMs predict on the event, the Club World Cup soccer game between Real Madrid and Al Hilal SFC, scheduled for Jun 18, 2025. The sources and market snapshots that LLMs receive are also the same, detailed in Table 14. The market prices were around (74% Madrid, 15% tie, 14% Al Hilal). Despite given the same market and source information, models showed differing rationales.
GPT-4o for example, explicitly adjusted its prediction to account for the possibility of a draw, pre- dicting (70% Madrid, 20% tie, 10% Al Hilal), noting:
"Sources universally favor Real Madrid, citing their superior quality, historical victory, and Al Hilals weakened state without Mitrovi. Market predictions strongly back Real Madrid, but I adjusted the probabilities slightly to account for Real Madrids lineup challenges and adaptation under a new coach. The uncertain defensive setup and potential absence of Mbapp slightly raise the probability of a tie.
Gemini 2.5 Flash, in contrast, acknowledged the sources that suggest higher chance of a tic, but nevertheless decides to place higher weight Madrid win (79% Madrid, 13% tie, 8% Al Hilal). reasons that:
"I heavily weighed the consistent expert analysis, betting odds, and market data, all strongly favoring Real Madrid. The probability distribution reflects Real Madrid's clear dominance and superior quality, further amplified by the critical injury to Hilal's top scorer. Although Real Madrid faces minor lineup chal- lenges and is adapting to a new tactical approach, these factors are signifi- cantly outweighed by Al Hilal's weakened offensive capabilities and the over- all disparity in team strength."
These fine-grained distinctions are not rarely found in collected data. They reflect how models reason under uncertainty, and often contributes to the differences in model performances.
E.2 EVENT RECALL IS APPROXIMATE, NOT PRECISE
Below is the model output of Gemini 2.5 Flash on the event Billboard Hot 100 #1, Jul 13, 2023?, from the Recall prompt in Appendix D.8.2.
35

---

Published as a conference paper at ICLR 2026
Table 14: Sources used for Real Madrid vs. Al Hilal SFC match.
Source
Summary
Title
Sporting News
Real Madrid is heavily favored to win against Al Hilal, with betting odds reflect- ing their dominance despite potential lineup challenges.
Real Madrid vs Hilal predic- tion, odds, betting tips and best bets for Club World Cup final
PokerStars Sports
Betting odds indicate a strong expectation for Real Madrid victory, with a significant likelihood of multiple goals being scored in the match.
Real Madrid Al-Hilal Betting Odds PooeerStars Sports
Sporting News (India)
Analysts predict 3-1 victory for Real Madrid, citing Hilal's 's reliance on penal- ties in previous matches and Madrid's supe- rior quality.
Real Madrid vs Al Hilal predic- tion, odds, hetting tips and hest bets for Club World Cup final
BetsLoaded
Real Madrid is predicted win against Al Hilal, with recent form and head-to-head statistics favoring the Spanish club.
Real Madrid Al Hilal Saudi Prediction, Betting Tips (18 June 2025)
El País
Under Xabi Alonso, Real Madrid is striving establish
El Real Madrid de Xabi busca nueva identidad en Miami: "Em- pieza rock and roll'
identity with focus high-pressure play, though the team still adapting this approach.
Sky Sports
Historical data shows Real Madrid's previ- ous victory over Al Hilal, suggesting a fa- vorable outcome for the Spanish team in the upcoming match.
Form and head to head stats Real Madrid vs Al- Al-Hilal
(Diario AS)
Al Hilal's top scorer, Aleksandar Mitrović, will miss the match against Real Madrid due muscle injury, significantly weakening their offensive capabilities.
Al Hilal pierde a Mitrovic
AS (Diario AS)
Real Madrid's potential lineup against Hilal may see Rodrygo replacing the Mbappé, with new signings Alexander- Amold and Huijsen expected to start in de- fense.
Alineación posible del Real Madrid contra Al Hilal en el Mundial de Clubes
AS (Diario AS)
Real Madrid, under new coach Xabi Alonso, faces defensive challenges due to injuries and is uncertain about its tactical formation ahead of the match against Al Hilal.
Nueva era y nueva defensa
Reuters
Kylian Mbappé is doubtful for Real Madrid's match against Al Hilal due to high fever, potentially leaving the team without recognized center forward.
Mbappe Madrid's Club World opener against Al-Hilal
doubtful Real Cup
Gemini 2.5 Flash Model Output
Event: TOPSOMG-23JUL.13
Q: Billboard Hot 100 #1, Jul 13, 2023?
Probabilities:
Vampire (olivia Rodrigo): 1.0
Last Night (Morgan Wallen) : 0.0
Fast Car (Luke Combs) 0.
Rationale:
I recall that Olivia Rodrigo's 'Vampire' debuted at number one on the Billboard Hot 100 for the chart dated July 15, 2023, which corresponds to the
36

---

Published as a conference paper at ICLR 2026
July 13, 2023, reference in the event. It displaced Morgan Wallen's s Last Night'.
Recall Assessment:
Recognized event: True
Evidence:
1. Olivia Rodrigo's 'Vampire' debuted at #1 on July 15, 2023.
2. The song accrued 191M streams and sold
25. 7k copies in its first week.
Recalled outcome: Vampire by Olivia Rodrigo
E.3 CASE STUDY: BITCOIN PRICE AT AM EDT
Two identical Bitcoin price prediction tasks, separated by five days (July 4 vs. July 9, 2025), il- lustrate the differences that source quality can have on an LLM's ability to form quality predic- tions. While the prediction task is essentially identical, for the July 4th event, including sources in the prompt worsened prediction accuracy, whereas for the July 9th event, source inclusion sub- stantially improved predictions (see App. E.3.1 for the detailed figure). Full source lists for both events are also available in App. E.3.2. Despite the July 4th source list have greater platform diver- sity (mainstream media, Al-generated predictions, algorithmic forecast bots), it suffers from qual- ity inconsistencies and methodological fragmentation. Numerous questionable forecast bots (e.g., produce extremely wide-ranged predictions ($82,822 $121,797), which works generate noise rather than useful signals. Similarly, dubious sources like ChatGPT-based predictions further dilute analytical credibility. contrast, the July dataset exhibits stronger coherence and practical utility, with predictions converging around $115,000 $125,000 from crypto- sppecillee platforms (CoinEdition, CoinDCX, Quickex.io, CoinCu) using consistent technical analysis.
priceforecasttot.com cryptopredictions.co
Thus, source competence and reliability are critical for enabling LLMs to generate accurate fore- casts. As such, these findings show that feeding in mass amounts of information L.MSs does necessarily enhance prediction quality.
E.3.1 BRIER SCORE COMPARISONS
Fig. 13 illustrates the differences in Brier score when sources were added to the prompt for both July 4th and July 9th events. While for the July 4th event, sources significantly improved the prediction quality, for the July 9th event, it worsened the prediction quality.
E.3.2 SOURCE LISTS
Below, we provide the complete source lists for both events. Each entry includes the title, URL, and brief summary generated by the searcher LLM.
July 4 Source List
ttttee; 'Bitcoin (BTC) Price Prediction 2025-2040', urr': /https///chaaggellyy.. htt.://wgwll...c.///oo////ttttoo prediction/',
summary': "Forecasts Bitcoin's price will increase by 18% reaching $130, 978.70 by July 5, 2025. Technical indi cators show bullish sentiment, with the Fear & Greed Index at 73 (Greed) 16 out of green days in the last month."
37

---

Published as a conference paper at ICLR 2026
Brier Score
0.3
Score
0.3
0.2
0.0
0.0
No Sourees
With Sources
No Sources
With Sourees
(a) July 4th
(b) July 9th
Figure 13: Brier Scores with and without sources in the prompt for July 4th (left) and July 9th (right)
Bitcoin events. Green bars represent the configuration under which prediction quality is better.
rtitle 'Bitcoin Price Prediction, Bitcoin Forecast by days: 2025',
urr': 'https: hatall/eiiieettro..oooooraaaa/ae prediction-data',
summary': : "Provides daily BTC price forecasts for July
2025. Example: July 15 prediction is $102, 206 (range $64,066 $110, 224) Based on historical data and market trends."
,,
'title': HHow High Can Bitcoin (BTC) Soar On July 4, 2025?', 1 'url': 'tttp:://thebtttiines http::/tteeettttts.....oooohhhh/ high can bit.iiiiin--bttooi----tto-t 4-20---s------------------ 42025
summary': "Suggests BTC could reach $125, 000, with downside risk to $90, 000 before resuming growth.
Based on 50-day EMA analysis.
"title': 'Bitcoin Price Prediction: 2025, 2030, 2040', 'url': prediction', 'summar': "Predicts BTC average of $107,531 on July 4, 2025 (range $100, 009 $115 065). Indicates steady upward trend from current market conditions."
1,
'title': 'Bitcoin (BTC) Price Prediction 2025, 2026 2030 CoinCodex', 'url': /https://// //tpn//o/////ccxxxx.////// com/crpptoobtttccsi/ppiicc prediction',
38

---

Published as a conference paper at ICLR 2026
s uumaary' : "Forecasts BIC to rise 12.518 to $118, 009 by July 4, 2025. 17/30 green days, 3.88 volatility. Fear Greed Index at heutral."
'title': 'ChatGPT Bitcoin Price Prediction for July 2025', 'url': attter///ccaaeddicco///cccceaasp
summary's "втС trading at 5107,024 (July 2, 2025) RSI and Bollinges Bands show BTC at critical inflection point. Potential breakout with volatility risk."
'title'; 'Bitcoin (BTC) Price Prediction For July 2025', 'url'; predicction-for-jyly-203//////
'summary: "BTC upward bias if $104k $106k holds support. Breakout above $110k could lead $114.5 & $125k. RSI and MACD support bullish view."
),
"title': BBtcooin on July 4, 2025 Know Today', 'url': htttss///rtttaarrr...ihhtooooooo
What Traders Should
'ummaay': "BIC testing $70k resistance. Breakout could spark rally; rejection may cause pullback to $68.2k.
Includes RSI, MA, and support/resistance zones."
),
'title' 'Bitcoin (BTC) Price Prediction 2025 & 2026-2029',
uur': 'https:/ " http://rpppppprecciiisoss.ss wwwp/o/w...coo/ooono.oooooooo//.
summary's "Predicts July 2025 BTC average 437,438 (range $82, 822 $1 .97). Suggests possible correction, 27% from prior months."
"title': 'Bitcoin (BTC) Price Prediction: $220145 Price Forecast Bot', ' 'url:: http:://ssseeeeeaaattacc.////ltt prediction.htin
"summary': "Forecasts 2025 BTC range: 578,588 $ 2.5 981. Average prediction $100,785 785. Based on historical data and market analysis."
'title' 'Cryptovers:: As markets question US exceptionalism, bitcoin starts to shine', 'url': htttpp://www.reeerrrr.ccc/////e/ xxppootteemnteeerrsseeeeereeorra
summary': "April 2025 втC rebounded 15s toward $100k amid skepticism of us markets. Analysts potential rally to $120k in Q2 2025 as investors hedge."
39

---

Published as a conference paper at ICLR 2026
July 2025 Source List
'title': "Bitcoin Price Prediction BTC Forecast 2025,
2026, 2030',
uurl: 'https:/ hatp:://altprceeet com/ bottt
summary's "Altpricer forecasts Bitcoin's s price to be $118,170 on July 12, 2025, reflecting a 0.258
increase from the previous day. The analysis suggests gradual upward trend, with prices reaching $120 229 by July 19, 2025. These predictions based on current market trends and technical analysis."
b,
'title': BBiccoin (BTC) Price Prediction 2025, 2026-2030 'url': CoinCodex',
prediction/
summary': "CoinCodex predicts Bitcoin's price to rise by 4.76% to $123, 274 by August 10, 2025. The analysis indicates a bullish sentiment with a Fear & Greed Index of 71 (Greed), It also reports that Bitcoin recorded 17 out of 30 green days with 2.14$ price volatility over the last 30 days."
b,
"title': 'Bitcoin price prediction for July 2025 | Quickex.io',
uurl; http:://tideee.l/oo/pppppeeepppp
price pulteerrrrdciiio-jjll------
summary' "Quickex.io reports that 'bullish flag' pattern is forming on Bitcoin's chart, suggesting potential for new highs around $115, 000 by mid-July. The analysis also warns of a possible dip to the $93 000 $ 000 range in late July early August. These projections ons are based on technical analysis and market sentiment."
b,
'title': 'Cryptoverse: As markets question US exceptionalism, bitcoin starts to shine', 'url': crrpooeesssnneeeessssstttoooosoe bitoih--tarree-shiee-222-------
s summary' "Reuters reports that Bitcoin has rebounded, gaining 15% in April 2025, nearing the $100, 000 mark. The article highlights increased investor interest due skepticism in U.S. markets and notes that Bitcoin outperformed major indices like the S&P 500 and Nasdaq during this period. Analysts suggest Bitcoin could reach $120, 000 in 02 2025."
b,
"title": 'Bitcoin (BTC) Price Prediction for July 12', frrl' htwpo//eediiiiioooooootttooooooo prediction for--ffo---jj-22-00000
40

---

Published as a conference paper at ICLR 2026
s sumnary' "This article reports that Bitcoin has broken through resistance to reach $118,000 its highest level since April, driven by ETF inflows and institutional interest. Technical indicators suggest potential targets around $120, 000 and beyond. The analysis highlights bullish market structure and increased on-chain activity."
b,
'title': 'BITCOIN FUTURE Price Prediction, BITCOIN FUTURE Forecast by days: 2025', 'url:: http:s///aleiiirreettooooorrrrre future peture-pprddltiiondddttt/
summary's "KalletInvestor provides daily price predictions for Bitcoin in July 2025, with prices ranging from 386,300 to $94,800. The forecast suggests moderate fluctuations, indicating stable market trend during this period. These projections are based on historical data and market analysis."
},
'title': 'Bitcoin (BTC) Price Prediction 2025-2040', 'url'; http:://ccaangllly.cco//////oooo prediction/',
ssummary' "Changelly's analysis forecasts Bitcoin's price to reach $139, 460.44 by July 9, 2025, indicating a 28.828 increase. The report notes a bullish market sentiment with Fear & Greed Index score of 73 (Greed). It also highlights Bitcoin's strong performance over the past 30 days, with 608 green days and 1.898 price volatility."
,,
"title': 'Bitcoin (BTC) Price Prediction Up To $1 1672,, 861.46 | BTC Forecast', http://///.......o..oop.oppeeeea
'url': 'https:/ httpp/////oonncccccooo////////// bitcoin',
summary': "CoinCu predicts Bitcoin's price to range between $131, and $150, 792.56 in July 2025. The analysis indicates potential for significant growth, with prices possibly reaching new highs. These projections ions are based on market. trends and investor sentiment."
"title': 'Bitcoin Price Prediction 2025, 2026- 2030: BTC Test Key Support $104K', urr': httt::////oondde...om///gg////// seroiin-price-weeklly/."
summary': "CoinDCX forecasts Bitcoin's price to trade within the $108, 500 to $111, 500 range over the next 24 hours, with average level near $110, 000. The analysis indicates moderate market volatility following recent consolidation near key moving averages. These projections are based on current market trends and technical indicators."
,
41

---

Published as a conference paper at ICLR 2026
'title': 'Bitcoin (BTC) Price Prediction 2025 2030 How Will It Perform?',
2 'url': hhttps://eryptoneee con////wss//ttccch-rrieee
prediction.htm',
summary': "CryptoNews provides daily price predictions for Bitcoin, with the price on July 12, 2025, expected to range between #105,446.55 and $108 77. The analysis suggests steady upward trend, with potential for continued growth. These forecasts are based on historical data and market analysis.
'title': 'Bitcoin Price prediction, Short/Long Forecast CoinLore', 'url': 'https:/ /wwp:///wwwww///// www.wocciooo.ccm... /ttp://caaareeccccccccaaaaaaaaaa
price-prediction
summary's "CoinLore predicts Bitcoin's s price to reach $130, 639 in July 2025, representing a significant increase from current levels. The analysis anticipates bull market in 2025, with potential for substantial growth. These projections are based on historical data and market trends."
42

---

Published as a conference paper at ICLR 2026
F PROMPTS
F.1 PROPHET ARENA PIPELINE PROMPTS
F.1.1 SEARCH PROMPT
Search Prompt
Find exactly 10 recent, relevant scurces that can help predict the outcome of the event: levent.title) This is a future event that you are collecting sources for. Note that the current time is [datetime now () .ttrftime (" Y-em-3d $H: (H:SM:8"") and the event closes at (event.
it ("sy-Em-Ed SH: $)
The event is from a prediction market with the following potential outcomes: levent.markett
The sources you find should be *recent, relevant and provide insights that can help predict the outcome of the event. Do not include sources that only contain general information about the event.
For each of the 10 relevant sources you will collect, return: 1. The full URL of the source.
2. The title of the source in *Englishe, If it is in another language, translate it to English. (if not available, provide a one sentence English summary as the title).
3. A 3 sentence summary in English of the *key* insights from that source.
Respond in the following JSON format:
{I "source": hhttps::/eexmmple..cmm/rrr "title": "Title of the source", "sumnary"; "This article discusses XYZ and is relevant because..."
F.1.2 PREDICTION PROMPT
Prediction Prompt
You are an AI assistant specialized in analyzing and predicting real-worle events.
You have deep expertise in predicting the outcome of the event: "[event_title)""
Note that this event occurs in the future. You will be given a list of sources with their summaries, rankings, and expert comments.
Based on these collected sources, your goal is to extract meaningful insights and provide well-reasoned predictions based on the given data.
43

---

Published as a conference paper at ICLR 2026
You will be predicting the probability (as float value from 0 to 1) of ONLY the following possible outcomes: (market_list_aet
IMPORTANT CONSTRAINTS:
1. You MUST ONLY provide probabilities for the exact possible outcomes listed above
2. Do NOT create or invent any additional outcomes
3. Use exactly the same outcome names as provided (case-
sensitive)
4. Ensure all probabilities are between 0 and 1
Your response MUST be in JSON format with the following structure:
'jjson
{f "rationale": "<text_explie "ccex__esplaainnnggoooorroooooo
"probabilities"; ((
(ison_example)
11
1)
In the rationale section of your response, please provide short, concise, sentence rationale that explains:
How you weighed different pieces of information
Your reasoning for the probability distribution you assigned
-nyy key factors or uncertainties you considered
Note: Market data can provide insights into the current consensus of the market influenced by traders of various beliefs and private information. However, you should not rely on market data alone to make your prediction. Please consider both the market data and the information sources to help you make a well-calibrated prediction.
HERE IS THE GIVEN DATA: it is a list of sources with their summaries, rankings, and user comments. The smaller the ranking number, the more you should weight the source in your prediction. (sources)
CURRENT ONLINE TRADING DATA:
You also have access to the predicted outcome probability
last trading price of each outcome turned out to be yes) from popular prediction market the moment of your prediction: (market_statisticss
44

---

Published as a conference paper at ICLR 2026
F.2 EVALUATING REASONING USING AN LLM-AS-A-JUDGE FRAMEWORK
Reasoning Evaluation Prompt
1. Sources Used (Citations, Attribution & Reliability)
**5 Exceptional:** Every single fact tied to +direct, authoritative+ source. Sources are reiigiyellabiltyy primary government data, central bank reports, peer- reviewed research, official statements) and pulled from the 1ist of sources provided to the predictor. Sources weighted by reliability with clear recognition that primary, authoritative sources 1ike Fed/Treasury/BLS data > news from reputable sources > provided market data >> articles blogs. Zero broken links, zero vague attributions. Connection between the rationale and the sources is very clear.
Good:** All major claims properly sourced with mostly hiabiieliability sources dominating, but **exactly one minor flaw** (e.g., one secondary source where primary was available, shows clear source quality discrimination. Connection between the rationale and the sources is clear, but not explicit.
minor formatting issue). Still
**3 - Adequate:*x Most important claims sourced, but ** multiple significant. weaknessesk*: broken links, 2-3 lower-quality sources treated as authoritative, source quality discrimination. Mix reliable and unreliable sources without proper weighting. Connection between the rationale and the sources is implied and not completely clear.
poor
**2 Poor:*** Sourcing is fundamentally inadequate.
Either most claims lack direct sources, OR heavy reliance weak sources (news summaries, blogs, non-specialist outlets),
OR no recognition of source quality differences. Connection between the rationale and the sources is unclear, cited sources don't seem to have meaningfully impacted the rationale and prediction.
**1 Terrible:** No meaningful citations, only unreliable sources, or completely broken/fabrical references. Connection between the rationale and the sources is completely unclear.
2. Evidence Extracted (Relevance Ranking)
**5 Exceptional::** Extracts *every* critical piece of evidence with surgical precision. Goes far beyond surface -level to uncover deeper insights. Perfect ranking importance. Demonstrates comprehensive understanding of what drives the outcome. Zero meaningful omissions.
45

---

Published as a conference paper at ICLR 2026
** Good:** Extracts most critical evidence with good depth but **issses exactly one important element** slightly misranks importance. Cenerally goes beyond surface-level with meaningful insights.
**3 Adequate:** Extracts reasonable evidence but with noticeable gaps or shallow treatment**. Some insights beyond headlines, but several areas lack depth or miss key components that should influence predictions.
**2 Poor:** Evidence is mostly superficial headline-level facts. Limited insight into underlying drivers. Significant omissions of relevant information.
*+1 Terrible:** No meaningful evidence extraction. Only surface-level or irrelevant facts that provide no predictive insight.
Combination Weighting (Reasoning Transparency)
Exceptional::* Crystal clear step-by-step reasoning with **epplccit numerical weights** and rock-solid justification for each weight. Complete transparency in how evidence combines. Machematical/logiicee rigor throughout.
**4 Good:** Reasoning mostly explicit with clear evidence combination, but **weights are somewhat implicit** justification could be slightly more rigorous.
**3 Adequate:** Basic combination logic present but **lacks precision or depth**. Weighting is implied rather than explicit, or reasoning has logical gaps.
**2 Poor:** Minimal attempt at systematic combination. Mostly just lists evidence without clear integration logic.
*+1 Terrible:*x No discernible combination methodology. Pure list of facts with no integration.
4. Uncertainties Counterpoints (Balance & Awareness)
**5 Exceptional: Identifies and **deeply explores multiple specific uncertainties.* with quantified impact on probabilities. Shows sophisticated understanding of how different types uncertainty (data, model, implementation, external factors) interact and compound.
**4 Good:** Identifies relevant uncertainties with reasonable depth, but *xexploration is somewhat surface- level** or impact probabilities not fully quantified.
**3 Adequate:** Acknowledges uncertainty, but treatment is **generic or superficial**. Limited exploration of how uncertainties affect the prediction.
46

---

Published as a conference paper at ICLR 2026
* Poor:** Minimal acknowledgment of uncertainty. Vague statements without substance.
*+1 Terrible:** No meaningful recognition of uncertainty completely one-sided analysis.
5. Mapping to Final Probabilities (Traceabilityy)
**5 Exceptional Every single probability is ** mathematically derivablexx from the evidence and weights. Complete audit trail from data -> logic -> numbers. No probability feels arbitrary or unjustified.
**4 Good:** Probabilities mostly well-justified, but *+1-2 numbers feel slightly under-explainedt or could use more explicit derivation.
**3 Adequate:** Partial traceability. Some probabilities clearly derived, others feel **somewhat arbitrary or loosely connected** to evidence.
++2 Poor:** Probabilities appear largely disconnected from evidence. Minimal justification for the numbers.
**1 Terrible:*x Completely arbitrary numbers with no connection to analysis.
Scoring Philosophy
**No participation trophies** weak work deserves low
scores regardless of effort
*xBe specific in justification behind scores** explain why (with examples if applicable) score was given.
47

---

Published as a conference paper at ICLR 2026
F.3 INTERNALIZATION PROMPTS
F.3.1 RECALL PROMPT
Recall Prompt
System Prompt
You are an AI assistant specialized in forecasting AND recalling previously seen real-world events.
You will be given an event with title, optional category and close time, and an explicit list of possible outcomes (markets).
Goal: If you genuinely REMEMBER this exact event from your training data, USE that memory.
Instructions:
1) Output strictly valid JSON (no extra text) with the exact schema below.
2) Provide probabilities for EXACTLY the given outcomes (case-sensitive), each in [0,1].
3) If you remember the event, include concrete verifying details (e.g., final score, date/time, venue, participants
key numbers, known from memory.
the resolution value) that would only be
4) If you do NOT truly remember the event, set perognized_event=-fee
and do NOT fabricate speci fics. Give best-effort probabilities from reasoning only.
Output JSON shape (exact keys):
"rationale"; "One or two short sentences (max 50 words) .", "probabilities";
{ "<outcomel>"; <float>, "<outcome2>";
float>, ,
"recall_assesmenn" {
"recognized : struelmidfalse.. srriddnceefactt""
"Concrete verifying details you recall (dates/scores/
"List at least 2 if secogmized_eventttneee otherwise
leave empty"
},
"recalled_outer outcone if known"; "<verbatim outcome name you remember the resolution, else null>"
Hard constraints:
JSON only. No text before/after.
Use only the provided outcome names.
Do not invent specifics unless you genuinely remember them.
User Prompt
This is the event: <event title>
Category: <category>
close Time (UTC): <close_time>
48

---

Published as a conference paper at ICLR 2026
Example market meaning (rules):
searket_name>; <rule text>
Possible outcomes (provide probabilities for exactly these)
<outcome_1>
<outcome 2>
<outcome_3 3>>
Your JSON must look like:
"rationale"; "<short 2-3 sentence rationale>",
"probabilities"; {
"<outcome_1>" sgrobabiliityyyaalee.fftoo....
"<outcome 2>"; <probability_value from t to 1>,
"recall aeeccll_assee aassel_sssssssttt.. {
"recognizeddeeentt" <true\midfalse>, selienne_ffactse
"<verifying detail 1>", "<verifying detail 2>"
,,
"recall
"<outcome name you remember the resolution, else null>"
F.3.2 INTERNALIZATION PREDICTION PROMPT
Internalization Prediction Prompt
System Prompt
You are an AI assistant specialized in analyzing and predicting real-world events.
Event: <event title>
Close Time (UTC): <close_time>
Example market rule: <market_namex <rule text>
Possible outcomes (provide probabilities for exactly these) <outcome_ls
<outcome__ 2>
coutcome_ 3>
Constraints:
1) Provide probabilities for exactly the listed outcomes case-sensitivel
2) Do not invent additional outcomes.
3) Each probability must be float in [0, 11.
4) Return JSON only; no extra text.
49

---

Published as a conference paper at ICLR 2026
output JSON:
"rationale": "<concise 2-3 sentence rationale>", "probabilities": "<outcome_l>": <float>, "<outcome_2>"" <float>,
User Prompt
Here is the given event:
Event title: <title>
Category: <category>
close time (UTC) <close_time>
Possible outcomes:
<outcome_ 1>
<outcome 2>
<outcome_33
Example rule excerpt: <rule text>
F.3.3 ADDITIONAL PROMPT FOR SOURCES
Prediction Prompt Sources
Additional Block (Sources)
Here are the given relevant data: it is list of sources with their summaries, rankings, and user comments. The smaller the ranking number, the more you should weight the source
in your prediction.
1. [Rank=1] <Source summary...>
2. [Rank=2] <Source cummary..."
50

---

Published as a conference paper at ICLR 2026
F.4 PROMPT VARIATIONS
Variation A
an AI specialized in real-world event analysis, your task is to predict the outcome of "[event_title]"" This future event requires detailed assessment based on provided sources, which include summaries, rankings, and expert comments.
Your objective is to leverage these insights to assign probabilities to the following specific outcomes: market _iiket__stt_tt)...
Crucially, your predictions must adhere these rules:
1. Only assign probabilities to the listed outcomes.
2. Do not introcuce new or alternative outcomes.
3. Use the exact srase-sensitive outcome names provided.
4. Ensure all probability values are between and
Your output must be a JSON object structured as follows: "'json {(
"rationale": seeex__eeppainnnggoourrrr. "probabilities"; {{
(json_ (]son_example)
1)
The "rationale" field should contain a concise, three- sentence explanation covering your information weighting methodology, the reasoning behind your probability assignments, and any significant factors uncertainties considered.
Variation B
You are an advanced AI system specialized in evaluating, interpreting, and forecasting real-world events.
Your assignment to thoroughly analyze and predict the outcome of the following event: (event_title)
This event has not yet occurred, and you will receive
curated set of informational sources. These sources may include, but are not limited to:
Concise summaries of the event and its context
Ouantitative qualitative rankings relevant to the event
Expert analysis, opinions, and commentary
User-generated discussions crowd-sourced predictions
Your responsibility is to systematically review these materials, extract key insights, and synthesize them into a reasoned probabilistic forecast. Your analysis must be both analytical and evidence-driven, using the provided
51

---

Published as a conference paper at ICLR 2026
information to support your conclusions rather than speculating beyond the given scope.
You must produce probability estimates (as floating-point
values between 0 and 1) for only the following possible outcomes: (market_list_str)
STRICT REQUIREMENTS MUST FOLLOW EXACTLY
1. Only assign probabilities to the listed outcomes. Do not create, modify, or introduce any additional outcomes.
2. Use the outcome names exactly as provided identical spelling, capitalization, and formatting.
maintain
3. Ensure all probabilities are valid sloating-poing numbers strictly within the range [0, 1].
4. The probability distribution must be internally consistent and make
in the context of the event.
OUTPUT FORMAT MUST THIS EXACT JSON STRUCTURE
Your final response must be single JSON object following the schema below:
'ijson
"rationale"; "kyour_reasoniing text>" "probabilities": {
[ison_example)
The "rationale" field should contain concise, three- sentence explanation covering your information weighting methodology,
the reasoning behind your probability assignments, and any significant factors or uncertainties considered.
Variation
You are an advanced forecasting model that evaluates real- world events.
Your sole task is to predict the event: flevent_titlen This event is still in the future. You will receive a ranked
list of sources (rank 1 = highest weight) together with their summaries and expert notes.
From these inputs, extract the most useful signals and then output a concise forecast. You must assign single probability (float from 0 to 1) to **each and only** the following outcomes: (market_list_str)
STRICT RULES:
1. Probabilities must be supplied only for the listed outcomes.
2. Do not add, rename, or rephrase any outcome.
52

---

Published as a conference paper at ICLR 2026
3. Preserve exact spelling and case of every outcome.
4. All probabilities must 1ie in the inclusive interval [0,1].
Return your answer in valid JSON, exactly as shown below: jjjson {
"rationale"; "<three_sentence, sarrressummmr"""
"probabilities": {{
(]son_example)
))
!)
Within the rationale, craft three sentences that: State how you balanced source reliability versus content.
Justify the resulting probability split.
Highlight the main uncertainties or decisive factors.
G LLM USAGE DISCLOSURE
Our study makes use of large language models (LLMs) in several ways: First, the benchmark and ex- periments themselves directly evaluate LLM performance; Second, we employed L.LMs as auxiliary judges for certain evaluation tasks (see Appendix D.7 for details). Beyond the benchmark, LLMs were occasionally used for language polishing (grammar, clarity, and style improvements) and for lightweight coding assistance (e.g.. boilerplate generation, debugging hints, syntax checks). All conceptual framing, theoretical analysis, experimental design, and substantive writing remain the authors own.
53


Reviews:
[{"review_id": "5CM4BQGBwh", "reviewer": "Reviewer_reZn", "summary": "The paper proposes evaluating LLM capabilities by forecasting live Kalshi questions. They setup an automated pipeline to extract, test, and resolve LLM forecasts. The evaluation results are summarized using both standard forecasting and calibration metrics like brier score and ECE, as well as (hypothetical) market returns and relative advantage between different LLMs. The paper has interesting analysis of LLM behaviours, ranging from: analysis across topics, effect of market data vs retrieved GPT 4o search results, different tendencies to assign extreme probabilities to forecasts, and analysis of reasoning traces and failure modes.", "questions": "1. I am slightly confused about the \"hypothesis\" / framing in the Introduction about predicting the next word leading to predicting the next event. Its definitely an interesting hypothesis, but one that can only be tested with base models without post-training which moves models away from \"next-token-prediction\" behavior. Could you either shift this hypothesis to be less prominent, or test it?\n\n2. It would be useful to shift the benchmark construction methodology to the main paper, as currently the main paper barely talks about it. To make space, section 3.1, 3.2, and 4.1.1 can be shifted to the appendix as they are not unique to this work.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a useful live benchmark for measuring LLM capabilities via forecasting open Kalshi events.\n\n2. I like the relative advantage based metrics used for comparing language models. This mitigates issues in existing benchmarks where questions can vary in difficulty (sometimes not even being \"forecasting\" questions as future information is leaked).\n\n3. The paper has detailed analysis of LLM forecasting behaviour and interesting insights across diverse ablations in Section 4 and the appendix.", "weaknesses": "1. The paper does not show awareness of existing literature in LLM forecasting. For example, probabilistic forecasting is mentioned as the first \"distinguishing feature\" of the benchmark in the introduction. However, this has been the standard used in existing papers in LLM forecasting [1]. I am also concerned about the supposed \"introduction\" of \"LLM-as-a-prophet paradigm\". LLM forecasting has been an active area of study for the last 3 years [2]. I do not see the value of adding a new term, especially given the connotations of the word \"prophet\". \n\n2. The use of GPT-4o based search as context across models could be a confounder. It is perhaps possible that the OpenAI models perform better on this benchmark because the retrieved context is more \"in distribution\" for them (as it comes from GPT 4o search + outputs) than other model families. Further, GPT 4o search is more likely to surface external context that bridges knowledge gaps in OpenAI models.\n\n[1] Approaching Human-Level Forecasting with Language Models. Danny Halawi, Fred Zhang, Chen Yueh-Han, Jacob Steinhardt\n\n[2] Forecasting Future World Events with Neural Networks.\nAndy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, Dan Hendrycks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "klqFJ62FC6", "reviewer": "Reviewer_kx9H", "summary": "1. The manuscript perfectly proposes = LLM-as-a-Prophet, testing LLMs’ ability. This actually helps forecast future real world events\n2. Introduces Prophet Arena, this is a live benchmark which is using prediction-market data and real-time web context.\n3. The manuscript also evaluates array of 22 LLMs. This represents diverse number of large language of models.\n4. The manuscript Measures forecasting loss (Brier), calibration error (ECE), and market return.\n5. GPT-5R shows best calibration and lowest loss; all models still lag human markets.", "questions": "1. Can you tell me how robust are results to prompt or search-engine variations along with some metrics to compare and evaluate?\n2. Could multi-agent or ensemble LLMs outperform single models and how this multi agentic workflow will be working together?\n3. How to improve knowledge precision and can you tell me what can be done to improve it?\n4. Can MCPs be used to do the same evaluation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript presents novel paradigm: which is real-world, contamination-free forecasting benchmark.\n2. The manuscript also presents comprehensive metrics: which is a good blend of accuracy and calibration.\n3. Good part is about the open-sourced subset for reproducibility.", "weaknesses": "1. The Event recall errors and approximate temporal memory which could be a potential weakness.\n2. The weakness Conservative probability estimates vs markets.\n3. Dependence on search/source quality; not all domains benefit equally.\n4. Limited profitability (returns < 1).\n5. Incomplete foresight near event resolution; calibration still imperfect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d91NwM5GSq", "reviewer": "Reviewer_Wxhc", "summary": "This paper proposes a live benchmark for evaluating LLMs’ ability to forecast future real-world events over time, addressing a core question with direct relevance to industrial decision-making. The results demonstrate both promising predictive intelligence and persistent bottlenecks in reasoning, evidence integration, and temporal information assimilation.", "questions": "1. Can the authors report performance differences across prediction horizons (short vs mid-term)? Since most events resolve within days or hours, current results may reflect late-stage market imitation rather than true foresight. Distinguishing performance across different lead times would reveal whether models are actually anticipating future outcomes, or simply reacting better as more information becomes available near resolution. \n\n2. Retrieval module lacks noise-robustness. News access relies solely on a search agent (GPT-4o) with minimal noise filtering or source-quality controls. How to improve evidence quality?\n\n3. Since LLMs function as judges without calibration against human scoring, there is a risk of alignment bias or shared model failure modes. Do the authors have plans for human adjudication or inter-rater agreement studies to validate reasoning evaluation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper proposes a live and realistic benchmark that directly measures LLMs’ forecasting ability on future real-world events, highly relevant to industrial applications.\n\n- Strong empirical design including probabilistic scoring, calibration, and market-based evaluation, providing a multi-angle understanding of model forecasting capability.\n\n- Clear analysis connecting reasoning quality and belief update dynamics.\n\n- Well-articulated motivation and clean methodology presentation.", "weaknesses": "- No measurement of alignment between human judges and model judges, raising concerns about rating validity.\n- Reliance on a single web-search agent without explicit evidence quality assessment may confound reasoning evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QG8zDCqbYf", "reviewer": "Reviewer_Uttz", "summary": "The paper aims to explore whether large language models (LLMs) possess genuine predictive intelligence. The authors argue that forecasting can serve as a rigorous and unified test of intelligence, as it integrates reasoning, calibration, and evidence aggregation.", "questions": "1) Do the authors view Prophet Arena primarily as a diagnostic tool for understanding LLM reasoning, or as a performance benchmark for future model comparison?\n\n2) How can the results from Prophet Arena inform or guide the next stage of model development? For example, can specific reasoning or calibration failures observed in forecasting tasks translate into actionable model improvements?\n\n3) Since the event selection, context construction, and preprocessing pipelines are not fully released, how do the authors ensure reproducibility and fairness across different models and time periods?\n\n4) Other questions are discussed in the Weakness section.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1) The paper is overall well written and easy to follow. \n\n2) The study is methodologically sound and supported by a comprehensive empirical evaluation. However, the benchmark’s reproducibility and transparency are limited, as key implementation details (event selection, context construction) are not fully released.\n\n3) The work offers a valuable new perspective on evaluating LLMs’ predictive and reasoning abilities. While the conceptual framing is interesting, the paper’s technical contribution remains limited. More importantly, a benchmark’s ultimate value lies in its ability to guide future model development by revealing actionable weaknesses or providing diagnostic insights. Prophet Arena, however, stops at evaluation and does not clearly demonstrate how its results can inform the improvement of model architectures, reasoning mechanisms, or calibration strategies. The work could be strengthened by explicitly linking benchmark findings to concrete modeling directions or by releasing more reproducible implementation details.", "weaknesses": "1) Although the paper claims to release part of the dataset, the full event selection, context construction, and preprocessing pipeline are not disclosed. No implementation details, scripts, or reproducible framework are provided for replicating Prophet Arena. As a result, the benchmark cannot be independently reproduced or audited, which weakens its scientific transparency.\n\n2) Prophet Arena primarily repackages existing components—prediction market data, web retrieval, and standard evaluation metrics (Brier score, ECE, return). The framework does not introduce new modeling techniques or algorithmic insights, serving mainly as an evaluation setting. Comparisons are limited to a single market-based baseline, lacking systematic benchmarks against structured forecasting models or classical probabilistic methods.\n\n3) The paper proposes using “real-world forecasting” as a proxy for intelligence, but does not justify why this task is representative, stable, or objectively measurable. It remains ambiguous whether Prophet Arena is intended as a benchmark, a task suite, or a conceptual lens for studying intelligence.\n\n4) (Important) The paper does not explain which cognitive abilities are actually being measured (e.g., reasoning, retrieval, knowledge recall, uncertainty estimation). Without disentangling these capabilities, the source of success or failure in forecasting is unclear. This makes the benchmark weak in diagnostic interpretability and limits its use for targeted model improvement.\n\n5)  Experimental results show that LLMs do not outperform the market baseline in prediction accuracy or expected return. The models’ predictions are often overly conservative or poorly calibrated, as acknowledged by the authors. Hence, the study fails to demonstrate real-world utility or unique advantages of LLMs in open-domain forecasting.\n\n6) (Minor Points) Figure 10 is blurry, and Figure 11 (right) has a formatting issue — the “Category” label is not fully visible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)