FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
Authors: Zhiyuan Zeng
Jiashuo Liu
Siyuan Chen
Tianci He
Yali Liao
Yixiao Tian
wangjinpeng.levi
Zaiyuan Wang
YangYang
Lingyue Yin
Mingren Yin
Zhu Zhenwei
Tianle Cai
Xinjie Chen
Zehui Chen
Jiecao Chen
Yantao Du
Xiang Gao
Jiacheng Guo
LIANG HU
Jianpeng Jiao
Xiangsheng Li
Jingkai Liu
nishuang
Zhoufutu Wen
Ge Zhang
Kaiyuan Zhang
周欣
Jose Blanchet
Xipeng Qiu
Mengdi Wang
Wenhao Huang
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmark
Future Prediction
Agent
Primary area: datasets and benchmarks
TL;DR: FutureX, a dynamic, live and contamination-free benchmark for evaluating LLM agents predicting the future.

Abstract: Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce FutureX, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents’ adaptive reasoning and performance in dynamic environments. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking.

Published as a conference paper at ICLR 2026
FUTUREX: AN ADVANCED LIVE BENCHMARK FOR LLM AGENTS IN FUTURE PREDICTION
Zhiyuan Zeng 22,,, Jiashuo Liu
Siyuan Chen 2- Tianci Hea-t, Yali Liao?.t, Yixiao Tian?.t, Jinpeng Wang2: Zaiyuan Wang Yang Yang?,, Lingyue Yin Mingren Yin Zhenwei Zhu
Tianle Cai', Zehui Chen'. Jiecao Chen', Yantao Du Xiang G Jiacheng Guo Liang " Jiao? Xiangsheng Li,, Jii,kinaaa Liu Shuang Niv, Zhoufutu Wen Ge Zhang?. Kaiyuan Zhang Xin Zhou
Wenhao Huang2 Jose Blanchet Xipeng Qiu Mengdi Wang'
ABSTRACT
Future prediction is a complex task for LLM agents, requiring a high level of an- alytical thinking, information gathering, contextual understanding, and decision- making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertain- ties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large- scale benchmark cxists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce FutureX, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/a- gent models, including those with reasoning, search capabilities, and integration of external tools such as the soen-source Deep Research Agent and closed-source
Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Our goal is to establish a dynamic, contamination-fiee evaluation standard that drives the development LLM agents capable of performing the level of professional human analysts in complex reasoning and predictive thinking.
INTRODUCTION
The rapid evolution of Large Language Models (LLMs) has catalyzed a fundamental shift in the landscape of artificial intelligence, moving from the generation of coherent text to the creation of autonomous agents capable of complex, goal-oriented behavior (Chase) 2022; Yao et al.) 2023 Sig. nificant Gravitas; Park al. 2023 Roucher al. 2025b). This transition from passive text gener- ators to active crollem-solvers necessitates corresponding evolution in evaluation methodologics. While foundational benchmarks like MMLJU (Hendrycks al., 2021) and SuperGL.UP (Wang al.) 2019) are instrumental in assessing the static knowledge of LLMMs, they are insufficient for measur- ing what a model can do when deployed as part of an interactive, goal-seeking system. An agent's performance is defined not just by its underlying model, but by its ability to plan, use external tools, and adapt to a dynamic environment.
Project Leaders & Equal Contribution (Junior First).
Corresponding authors.
i Core Contributors (a-3 order). Contributors (a-3 order).
Fudan University. "ByteDance Seed. 3 Stanford University.
Princeton University.
Email: {zhiyuanzeng.98, liujiashuo.77. henhoo @hytedance.com

---

it, to our knowledge, the largest and most diverse live benchmark for Published as a conference paper at ICLR 2026
In response, generation of agent-centric benchmarks has emerged, primarily focused on eval- uating search (Wei et al. 2025b), tool usage (Yao et al. 2025). and coding skills in controlled or simulated settings (Jimenez et al. 2024). While these benchmarks offer valuable insights into agent capabilities, they largely address static, well-defined problems whose solutions are already known. Further, they fail to address critical gap: the ability to synthesize dynamic, real-world informa- tion, process it, and perform complex analysis and reasoning, which human experts possess across different domains. Future prediction, in fact, directly addresses these two drawbacks. This task di- rectly tests an agent's 's ability to integrate dynamic, real-world information, process it in context, and generate complex analysis and reasoning about problems whose answers are not yet known to the world. Such tasks naturally involve a dynamic element, and their primary significance lies in prepar- ing agents to anticipate and navigate genuinely novel scenarios, mirroring the foresight applied by human experts across diverse domains.
In response this need, we introduce FutureX, a dynamic and live evaluation benchmark specif- ically designed for LLM agents performing future prediction tasks. FutureX is built upon semi- automated pipeline that continuously collects future-oriented questions from 195 diverse websites, curated from a pool of 2,008 sites covering areas such politics, economics, technology. sports, healthcare, and more. This curation process involves both LLM-based agents and human experts, a necessary combination ensure quality. Each event is associated with start date (several days prior to the resolution date) and a resolution date. The pipeline automatically collects and stores agents' predictions on the start date. After the resolution date passes, the system dynamically crawls the web to retrieve the ground-truth outcome and scores the agent's 's prior predictions. FutureX provides four key advantages that directly address the limitations of existing benchmarks:
Large-Scale and Broad Domain Coverage: Using semi-automated pipeline for question col- lection and filtering, we currently select 195 websites from a pool of 2,008 as our sources. These selected websites cover a wide range of inciuuinn politics, economics, finance, sports, and entertainment-
future prediction.
Real-Time Updates: FutureX continuously collects future-oriented questions from 128 websites, with daily updates ensure real-time relevance. By dynamically crawling questions and answers, the benchmark maintains both timeliness and diversity in the questions, presenting a unique chal- lenge for LLM agents to process and adapt to constantly evolving information.
No Data Contamination: Focusing exclusively on future events, FutureX inherently eliminates any risk of data contamination, preventing any exploitation of historical information to manipulate the results.
Building on FutureX, have developed a fully automated evaluation pipeline that updates future questions daily, runs various LLM agents for each event on its start date, collects event outcomes after the resolution date, and evaluates agents' performance. The models under evaluation include base LLMs, LLMs with reasoning and search capabilities, open-source Deep Research Agents, and closed-source Deep Research Agents, for a total of 25 models.
Comprehensive & Automated Assessment of LLM Agents:
In addition to the overall results, we conduct an in-depth analysis, including:
How do LLM agents perform on questions of varying difficulty levels and across different do- mains? (Section 4.2 and G.I)
What factors (such as the type of LLM model, agent framework, and question domain) have the most statistically significant impact on performance? (Appendix (.22
How do LLM agents perform when making predictions after the resolution date? (Appendix G.3
How planning and search capabilities affect the performance? (Appendix G.4 G.5)
To pave the way of the "Second Half of A we firmly believe FutureX has great potential to unlock new research directions for developing LLM agents capable of performing at the level of professional human analysts in real-world, high-stakes domains.
2 RELATED WORK
A significant gap in current LLM agent evaluations is their ability to handle dynamic, real-world information and reason about future events, which are core skills human professional analysts rou- tinely apply across finance, business, politics, and technology trend analysis. Future prediction
https::/yssnych..iiti io/Th Satee-Secooo-HHaaa
2

---

Published as a conference paper at ICLR 2026
Table I: Comparison with Previous Benchmarks for Future Prediction. A V in the Live Update column indicates that a benchmark supports this feature, though may not update regularly. A in the LLM Agents column for FutureBench reflects evaluation of only single contrast, VVV
open- open-source agent. In to denote regular updates and comprehensive coverage of multiple models.
Data
Evaluation
#Eyents
ADoonaia
Live Update
Time
Source
LLM Agents
Bav.
Froquency
Auto
PorecssoQQ
ci
3021
10302
Pist
21 News Websites
X
Sim.
One- Time
X
Astercat
7ou
1022
6707
Post & Funue
Prediction Markets
X
Sim.
Ono- Time
X
OpenFPPBecch Guan
2024
983
Fature
News Wabsites
X
Sim
One- Time
X
NavTomomo
Nako Janowt
2025
5000
Post
News APfs
X
Sim.
One- Time
X
ForecssBench
2025
V
Fature
Prediction Markets
5 Dataisuses
Sim.
Monchly
FunreBench
Together.
2025
Fature
Several Websites
Prediction Markets
Real
Weekly
FutureX
-00kwerk
VVV
Future
Websites
VVV
Real
Daily Weekly
serves as a critical test for these capabilities, demanding agents to gather up-to-date evidence and anticipate outcomes in open-ended enviromment. However, building reliable benchmark for forecasting presents unique challenges: unlike static question-anssering. future events are inher- ently uncertain and cannot be easily verified in advance. Previous efforts evaluate LLMs fore- casting, such as backtesting with historical data (Wildman et al., 2025), risk introducing temporal leakage and retrieval contamination due to retrospective information influencing internet searches on past events (Paleka et al. 2025). Furthermore, many existing benchmarks (Guan et al. 2024 Karger al., 2025; Nako & Jatowt, 2025) focus on evaluating LLMs without search capabilities, which is impractical for real-world future prediction. Benchmarks like those proposed in Guan et al. (2024) Nako & Jatowt 2025) are also one-time collected and lack the live updates necessary for dynamic events. While ForecastBench (Karger et al. (025) attempts to address the "future event" challenge by using only questions about future outcomes, it predominantly evaluates vanilla LLMs, and relies prediction market events, dominated by multiple-choiee questions. This limits both the diversity of events included and, critically, the assessment of an agent's 's ability to perform open- ended, real-world information gathering. Similarly, FutureBench (Together.ai (025) is restricted to events from Pol 1 PolyMarks yMarket and includes very small number of events (~30). This limited diversity in current future prediction benchmarks highlights the inherent difficulty in collecting and evaluating such events, particularly those with unknown answers or those that have not yet transpired.
Although challenging, we adopt this direction because ensures the absence of information contam- ination and directly evaluates how effectively an LLM-agent can synthesize real-time data, reason under uncertainty, and predict future events-capabilitie that represent the next frontier for expert- level Al agents, which is exactly what our proposed FutureX seeks to achieve. More related works on general LLM agent benchmarks are discussed Section
3 FUTUREX
The goal of FutureX is to provide dynamic, comprehensive, and contamination-ffes evaluation of L.M agents' advanced search and reasoning capabilities, aiming to match even surpass the expertise of human professionals. In this section, we will introduce the construction process of FutureX, as well as its core features.
3.1 DESIGN PRINCIPLES OF FUTUREX
To clearly articulate the design philosophy of FutureX and draw a sharp contrast with other bench- marks, we demonstrate four core design dimensions.
Eliminating Data Contamination. The design philosophy of FutureX inherently solves the data contamination problem. By defining its core task as "future prediction", FutureX guarantees that the ground-truth answers to all questions have not yet occurred at the time of the agent's prediction, making impossible for them exist in any model's training data.
Simulating RorldWorld Challenges and Evaluating Core Intelligence. FutureX places the agent directly into the real world's information flow (e.g., economic fluctuations, technological breakthroughs, political elections). This task is holistic, demanding suite of advanced cognitive skills, including information gathering, data synthesis, probability weighing, and causal reasoning.
3

---

Published as a conference paper at ICLR 2026
Event Dalebase Construction
Future Event Daily Curahon
Agent Daily Prediction
Antwer Daily Acquiring +
Weboite Collcttie
Webrre Ourclee
Over Masipuation >>>
Couniig
Ray camdic
hihering
8 L.55
Septooree Lasorr Agcis 15 tradi.
Dane fiterin ?>> Croding
e Sctraction
Daily Score
Agui
Lankeronse
Penebake
swars X6
unondall mort
X
Fully autometed Pipeline
Figure 1: The overall pipeline of FutureX, which consists of event database construction, future event daily curation, answer daily acquisition. The entire pipeline is fully automated and operates on daily basis.
Large-Scale and Cross-Domain Comprehensive Coverage. FutureX is currently the largest and most diverse live benchmark for future prediction. Through a semi-automated data pipeline, we curate and filter information from 195 high-quality sources, selected from pool of over 2,000 websites. These sources cover a wide array of domains, including politics, economics, finance, technology, sports, and entertainment.
Dynamic and Automated Evaluation Process. One of the core values of FutureX lies in its highly automated, dynamic, and closed-loop evaluation process. On each event's designated start date, it automatically runs the various agent models and stores their predictions. Once the event's resolution date has passed, the system again automatically crawls the web to obtain the ground- truth outcome and scores the agents' prior predictions. This entire process operates without man- ual intervention, ensuring the evaluation's timeliness, objectivity, and scalability.
Overview of FutureX. Following our design principles, FutureX is live-updating benchmark for future prediction that covers broad range of source websites and domains. With daily and weekly updates, it features an automated pipeline for event collection, curation, and agent evaluation all running smoothly and reliably. As shown in Table FutureX demonstrates clear advantages over previous benchmarks from both data and evaluation perspectives, supporting a much broader range of events and a more diverse set LLM agents for evaluation. Specifically, recent live bench- marks (Karger et al.] 2025: Together.ai 2025) primarily rely on prediction market websites for live updates. However, as demonstrated in Section 3.3, these events tend to be relatively simple, and many involve subjective questions that are not well-suited for rigorous evaluation. In sharp contrast, Future collects and curates events from a much broader range of sources to ensure a challenging and high-quality set of evaluation tasks. Moreover, FutureX evaluates 25 models across four differ- ent categories, which, to our knowledge, is the first comprehensive benchmark for LLM agents in the domain of future prediction. In comparison, FutureBench (Together.ai 2025 evaluates only single open-source agent with few LLMs.
3.2 CONSTRUCTION OF FUTUREX
As shown in Figure ! FutureX is an automated, live benchmark that operates on a daily cycle, encompassing four stages: event database construction, future event daily curation, agent daily pre- diction, and answer daily acquisition. Each stage is processed on a daily basis.
3.2.1 EVENT DATABASE CONSTRUCTION
This phase consists website collection and website curation. During website collection, we begin by using the AIME agent (Shi al. 2025) gather large number of website URLs relevant to domains such as politics, economics, finance, technology, and sports, with a total of 2,008 websites. Then for website curation, these URLs are then subjected to initial LLM-based check, utilizing a combination of Beeell.-Thining (Seed et al. (025) and DeepSeek-RI (Guo et al., (025). This process performs tasks such as deduplication, assessing the suitability of the websites for question generation, and evaluating their update frequency, reducing the number of websites to 636. The remaining URLs are then manually reviewed, focusing on identifying reliable sources, particularly ranking lists and websites with high update frequency, ensuring that only the most relevant and up- to-date information included. After this manual review, our initial version has 195 high-quality websites our current event database, including five types:
Prediction market websites: Websites that allow users to bet on or predict the outcomes of future events, including political events, sports outcomes, or financial market movements. Examples include jopen, Pol ymarkett rket, etc.
4

---

Published as a conference paper at ICLR 2026
Websites that provide up-to-date news, analysis, and market-moving events, such as earnings reports, economic policy changes, sports news, technology trends, and geopolitical developments. Examples include Reuters, ESPN, etc.
News websites:
Entertainment ranking websites: Websites that provide rankings related to music, movies, TV shows, and other entertainment forms. These rankings often based on user reviews, sales data, critical acclaim, or popularity metrics. Examples include Bi lboard, IMDb, etc.
Government websites: Official websites that provide economic data, regulations, and reports from governments. These include statistical data like GDP, unemployment rates, inflation, fiscal poli- cies, and other public economic indicators. Examples include U.S. Bureau of Economic cerriccuturaa and Rural Information Site, etc.
Anal Lysis,sss. s, China
Real-time data platforms: Platforms that provide real-time or near-real-time data various finan- cial markets. These platforms are used to monitor live stock prices, forex rates, cryptocurrency values, and other asset prices, offering instant updates to help with timely trading decisions. Ex- amples include Shenzhen Stock Exchange Site, Sina F Finance, etc.
In addition, as shown in Figure 1 this event database is updated daily to remove events with un- available outcomes and continuously add new events using current high-quality websites as "seeds". Based on these, we perform daily future event curation to generate prediction questions for each day, including event manipulation and event filtering, and details are shown in Appendix C
3.2.2 ANSWER DAILY ACQUISITION
Answer acquisition a critical phase in FutureX. While we can pose questions about wide range of future events, the success of evaluation ultimately depends on whether we can reliably obtain the corresponding answers. To this end, much of our effort constructing the event database (see Sec- tion 3.2.1) focuses ensuring answer availability. We carefully select high-quality websites that consistently provide verifiable outcomes on a daily or weekly basis. Building on this, our pipeline automatically retrieves answers each day by following the procedures outlined below.
1. Date Filtering. We filter events whose resolution date aligns with the current day.
2. Website Crawling. With the daily answer acquisition set, we then crawl the corresponding website and extract the core content.
3. Answer Extraction. Based on the core content retrieved from each website, we use the Seedl.5-Thinking model (Seed et al. 2025) to extract the precise answer.
In addition, expected during the initial stages, various types of failure cases arise. To systemati- cally address them, we categorize failures into two types: crawling errors and extraction errors. For crawling errors, such as those caused by anti-bot measures, we update our event database exclude websites that are no longer accessible. For extraction errors, such as incorrect or missing answers, manually review the cases and design customized prompts to improve answer extraction accu- racy. With these efforts, the answer acquisition success rate exceeds 97 % supporting an efficient and fully automated evaluation pipeline that runs on a daily basis.
3.3 DATA OF FUTUREX
Comprehensive Domain Coverage. As introduced in Section 3.2 FutureX achieves comprehen- sive domain coverage through the daily curation of future events from 195 high-quality websites. Between July 20th and August 3rd, our dataset comprises 1,272 events, systematically categorized into 11 main domains-including polities, sports, crypto, culture, finance, business, technology trends, weather, health, and space-eade further refined into several sub-categories. As depicted in Figure TO the distribution across these domains is notably well-balanced, which facilitates a robust and comprchensive evaluation of LLM agents across multitude of real-world scenarios. We show the examples in major domains in Table 4
Different Event Types. FutureX includes four different event types: single-choice, multi-choice open-ended ranking, and open-ended numerical prediction events. Single-choice events require se- lecting one correct answer from options fewer than 4. multi-choice events involve identifying multi- ple correct answers, making them inherently more challenging. In addition to events with predefined choices, FutureX features significant number of open-ended events, where no options are provided. There are two types of open-ended events: (1) ranking tasks, which typically involve forecasting the order of items prediction tasks, which require estimating a specific numeric value (e.g., a stock price or price in- dex). As detailed in Appendix C significantly downsampled binary yes-or-no events primarily
future leaderboard (e.g.. music or movie popularity rankings), and (2) numerical
5

---

Published as a conference paper at ICLR 2026
Nove
Sports
Politics
Single-Cheice
Nutti Choice
Crypto
Open-Errded lanking Low Velatilty
Culture
Ginanize
Open Ended Ranking HHgh Velacilty
Ended Numorical Volotilty
cond Pnformm
Open Numerical Vatetiltty
Single-Choice
Multi-Choice
Open-Ended Ranking
Open-Ended Endidd Numerical
Figure 2: Domains 1,272 events.
Figure 3: Event type distribution.
affecting single-choice questions increase the difficulty of the benchmark. As shown in Fig- ure IT] the distribution of the remaining three event types is relatively balanced.
Different Volatility. While all events in FutureX involve forecasting future outcomes, they vary significantly in how dynamic these outcomes are over time. We analyze the volatility of each event based on historical data, measuring how much the target outcome is expected fluctuate over time. We then tag open-ended events either "Low Volatility" or "High Volatility", which are visually indicated by different color depths in Figure
Difficulty Tiers. Guided by event type and expected volatility (see Figure we partition the benchmark into four progressively harder tiers, basic, wide search, deep search, and super agent, that correspond to the agent capabilities assessed in Levels through 4. An overview of the 4 tiers and the specific agent skills they assess is given Table The examples of events corresponding to these 4 tiers are shown in Table 3
The Basic tier (Level 1) contains single-choice events with options fewer than 4. The predefined options sharply limit the search space, information retrieval and reasoning are lightweight.
The Wide Search tier (Level 2) comprises multi-choice events with several correct answers. The agent must submit the full set of valid options and nothing more. This tier requires more complex reasoning, and therefore tests whether the agent can perform exhaustive yet precise discrimination across multiple plausible options.
The Deep Search tier (Level 3) contains open-ended events whose underlying facts are relatively stable. With no options provided, the agent must propose its answer, performing multi- step search and reasoning to gather evidence. Because volatility is low, exhaustive information collection should converge the correct response. This tier thus probes the agent's ability to navigate, integrate, and synthesize reliable information.
The Super Agent tier (Level 4) covers high nigh-volatility. open-ended events. Here the agent must cast a wide net for information and reason probabilistically under shifting signals and deep uncer- tainty. The task is taxing for human experts- -et alone machines-becnee because the scenarios complex, ambiguous, and resist simple fact retrieval. This tier therefore probes an agent's 's "super" capacity for nuanced, uncertainty-awarr forecasting in the most demanding real-world settings.
Notably, all Level 3 and Level events are generated through our automated pipeline (see Figure which supports scalable event creation while maintaining quality control. This marks a key dis- tinction from prior benchmarks (Karger et al. 2025; Together.ai 2025), where most events were relatively simple and collected directly from prediction market websites (see Table I
3.4 EVALUATION PROTOCOT. OF FUTUREX
Unlike traditional static benchmarks, where each query is associated with known answer, future prediction inherently lacks ground truth at the time of prediction, since the relevant events have not yet occurred. As a result, FutureX introduces an evaluation delay, referring to the time gap between when prediction is made and when it can be evaluated. For example, suppose agent makes predictions for several events on July 15th, with resolution dates ranging from July 16th to July 22nd. In this case, performance resulting in an evaluation delay of one week. FutureX adopts an one-week prediction window, which provides both sufficient event coverage and manageable evaluation latency.
only be evaluated after July n2nd once all outcomes are known,

---

Published as a conference paper at ICLR 2026
Table 2: Difficulty tiers and assessed agent's skills in FutureX.
Assessed Agent's Skills
Level
Tier
Event Type
Focus
Planning
Reasoning
Searching
Choose from fewer than options from a given list.
Basic
Few choices
Weak
Weak
Weak
Exhaustive discrimination and Return all correct futures.
Wide Seurch
Many Choices
Weak
Medium
Medium
Interactive search & synthesis Navigate sources (click, scroll. filter) Integrate evidence for answer.
Open-ended (low volatility)
Deep Search
Medium
Medium
Strong
Forecast high-volatility, open-ended events Conduct wide-scupe information seurch Reason and predict under deep uncertainty "Super-agent" tier
Open-ended (high volatility)
4
Super Agent
Strong
Strong
Strong
Table 3: Examples of different levels, where the specific date can be replaced with any future date.
Level
Example Events
1. Basic
Ethereum Up or Down on August 20, 2025?
2. Wide Search
Who will win the King of the Mountains / Polka-dot Jersey at the 2025 Tour de Francce A. Tadej Pogacar', B. Other', C. Jonas Vingeguard,
3. Deep Search
Which movies will be in the top 10 of Maoyan Movic Ticketing Rating List as Beijing Time August 20, 2025.
Super Agent
What the daily purchase transaction amount (in billion yuan) in the daily transaction information the Shanghai Hong Kong Stock Connect on August 20, 2025, Beijing Time?
We use the exact-match and Fi-score to evaluate the level-1 and level-2 questions, for the level-3 and level-4 questions, we adopt LLM-as- Judge (equation [T] and the clipped MSE score (equation 2 the evaluation metric. All details can be found in Appendix B
EXPERIMENTS
In this section, we present the main results for FutureX from July 20th to August 3rd The results in this section cover 25 models, including:
Open-source and closed-source LLMs without tool usage, including Gemini-225-pro, DeepSeek V3, GPT-4o-mini, GPT-4.1, DeepSeek RI, Qwen3-32B, Qwen3-235B, and Doubao-Seedl 6-hhniin..
Base LLMs (8 models):
SmolAgent for Deep Research (Roucher et al. 2025a) (6 models): SmolAgent is evaluated with various backbone L.Mss, including Gemini-2.5 2.5-in GPT-4.11 GPT-4o-mini, Qwen3-235B, Qwen3-32B, and DeepSeek V3. Among reasoning models, only Gemini-2.5 is included here, as others-such as Doubao-Seeeee Seeetdo--eeee... 6- 6-Thiinin GPT-o3/o4-mini, and DeepSeek Rl-incur signifi- cantly longer runtimes and are therefore currently excluded.
AgentOrchestra (Zhang et al. 2025h) (2 models): Tested with two backbone LLMS-Gemini- 2.5-pro and GPT-4.). Due to the complexity of this agent framework and limited compatibility, only these representative models are included for now.
LLMs (Searh)) models): Evaluation of advanced commercial LLM services with in- tegrated thinking and searching capabilities, including Doubao, DeepSeek R1, Hunyuan, Qwen3- 235B, GPT-40, GPT-o4-mint minii and Grok-4. Gemini-2.5-pre (Think& on the way.
Deep Research Models (2 models): Includes top-tier closed-source models tailored for deep research tasks: Doubao and Gemini Deep Research (with Gemini-2.5-flash),
Note that weekly results (till now) of FutureX online competition are shown in Appendix with much more advanced models like GPT5-pro, ChatGPT Agent, Claude 4.1 Opus, Gemini Deep Think, etc.
"The uedkly-updated results after August 3dd shown in Appendix

---

Published as a conference paper at ICLR 2026
Agwagg vole
Loffi speeari Desiqimnd
glonet Depfguect
Greddes -
Coed Gootee
the aodd
(a) Level I: Basic Tier
(b) Level 2: Wide Search Tier
Homen 0:24
ULM
denet ctodee
cande
(c) Level 3: Deep Search Tier
(d) Level 4: Super Agent Tier
Figure 5: Overall results of different difficulty tiers (hetween July 20th and August 3rd) Note that since AgentOrchestra is computationally intensive, we evaluate it with only two backbone models for only Level and 2 events.
4.1
OVERALL RESULTS
Bise LLM SmalAgerr-DR
LLM(Think&Seerch)
DespReseasch
Oeeat 129
Drypond
Depoit-f
Gheal-t1 (reil-211
Peppork 4t
(PK41
Gemal 2.t
Cemita 35
Jesdan Siud
2013
Herpee
Depelink
Seal & Deathe)
OP% deb
Seat Doattan (10
Geman 5-lao 132
@PTak miii
Oooke
Figure 4: Overall scores on Future between July 20th and August 3rd
As for the overall score, we combine scores from the 4 difficulty tiers (see Table 2 using weights of 10%, 20%, 30%, and 40%, respectively, with heavier weights assigned to the more challeng- ing tiers. The overall results sented using similar colors for clarity. As shown in Figure 4 across the four model types, Grok-4 achieves the highest overall performance, followed by Gemini-2.5-flash Deep Research, GPT--44 mini (Think &Sercc)) and Seed1.6 (DouBao). Generally, reasoning models equipped with search capabilities outperform the rest, underscoring the importance of advanced search and reasoning in FutureX. Moreover, SmolAgent-DR (Roucher et al. (025a) underperforms compared to LLM (Think&Search), likely reflecting differences their search API capabilities.
shown in Figure 4 where models of the same type repre-
4.2 RESULTS ACROSS DIFFICULTY TIERS
We show seperate results across 4 difficulty tiers in Figure 5 Our main findings are as follows:
Finding 1. Our difficulty tiers accurately reflect the complexity of the events. We observe clear, consistent decline in performance across the 4 defined difficulty tiers, which strongly supports the validity of our task stratification. Models achieve high accuracy on Levels I and 2, but their performance drops significantly on Level 3 and, in particular, on Level 4. This trend indicates that our difficulty labels effectively correspond to increasing levels of task complexity. Furthermore, by comparing Figure| 14 and Figure [5] we find that even within the same domain, model performance declines substantially.
In particular, Level 4 events, which are open-ended and highly volatile, pose significant challenges for current models. These tasks often demand multi-step reasoning, synthesis of ambiguous or incomplete information, long-hou long-horizon forecasting, and deeper understanding of world knowledge and strategic thinking. In our experiments, most models struggle to generate correct responses for these events, and even the strongest models often fail to score at all. In fact, these tasks are so complex that they not only test the limits of current models, but may also serve as a benchmark for measuring superhuman performance in future systems.
Finding 2. Base LLMs perform well on Level and Level 2 events. Level I and level 2 events designed relatively simple single/multi-choice questions. Our results show that even base
8

---

Published as a conference paper at ICLR 2026
LLMs (without tool usage advanced reasoning capabilities) consistently achieve high accuracy on these tasks. Notably, DouBao-Seeds Toadl.6-Thia 6-Thinking outperforms several agents equipped with web search tools, including the two Deep Research agents. These events may primarily rely on basic fac- tual recall or straightforward reasoning, which hase LLMs are already capable of handling without external tools.
This finding also suggests that Level and Level events are not sufficiently challenging to distin- guish between models of varying capabilities. While they are useful for establishing performance baseline, they offer limited insight when evaluating more advanced language models.
Finding 3. Search/tool usage becomes increasingly important for harder events. As the complexity of the events increases, particularly in Level 3, models that incorporate external tools such as web search, calculators, or code exccution tend to perform significandly better than those that rely solely on static knowledge. This highlights the critical role of tool tool-augmented reasoning in handling complex, multi-step problems that cannot be solved through pre-trained information alone.
This further demonstrates that FutureX is capable of analyzing advanced search capabilities that are closely linked to reasoning.
Finding 4. DouBao-Seed1.c 6- Thinking excels in knowledge retrieval (Level 1 and Level 2), and Grok-4 demonstrates exceptional performance on more difficult events (Level 3 and Level 4). Among base LLMs, we find that SouBao-Seedl .6- 6-Thinking performs the best on Level 1 and Level 2 events. Notably, it even outperforms agents equipped with search tools as well as deep research models. This suggests that, when provided with answer options, DouBao-Seed 1.6- 6-Thinking is highly effective at retrieving and applying its internal knowledge to make accurate predictions about future events, demonstrating strong inherent reasoning capabilities.
In contrast, among all evaluated models, Grok-4 stands out on the most challenging tasks. Re- markably, it surpasses even premium models such as Gemini Deep Research in both accuracy and efficiency. Despite operating with fewer searching and faster inference speeds, Grok-4 and GPT- 04-mini achieve top-tier results, highlighting an impressive balance between reasoning strength and runtime efficiency.
Finding 5: LLM agents still lag behind humans. In addition to automated model evaluations, we conducted a human annotation study with 40 industry experts. These participants comprised current or former employees from the Big Four accounting firms (c.g.. KPMG), top consulting firms (c.g., McKinsey), and nine leading investment banks (c.g., UBS). We randomly sampled 300 questions from our test bank and had these experts answer them independently, then computed their average scores on the same evaluation metrics (red dotted lines).
As shown in Figure 51 humans significantly outperform LLM agents on Level Level 3, and Level 4 events, indicating that LLMs still have considerable ground to cover before matching human ex- pertise. Conversely, for Level 2 events, some models actually surpass human performance; this may be because these multi-choice questions involve many options that people often cannot exhaus- tively compare every possibility. Overall, these results underscore the substantial potential for LLM agents to aid (and eventually rival) humans in forecasting future events.
4.3 MORE CASE STUDIES
In addition to our main results, we design and conduct multiple case studies, including:
1. Results across Different Domains. Differnt models have different strengths, for example, GPT models are good at Crypto. Details in Appendix G
2. Factor Analysis. We conduct a linear analysis to discover important factors with statistical significance. From it, the "difficulty level" and "event domain" factors are identified as the most critical factor that influence the performance of agents. Details in Appendix G..2
3. Past vs. Future Prediction. By comparing historical event searching and future prediction. we could directly characterize the search capability of different models, where Grok-4 exhibits excellent performance. Details in Appendix G.3
4. Analysis of Agent Trajectories. The number of tool-call, comprehensiveness of planning and the credibility of source websites all significantly influence the agents' performance in future prediction. Details in Appendix G.4

---

Published as a conference paper at ICLR 2026
5. Search Analysis. We count the average number of searches for each model, where find strong models typically conduct more searches, and Grok-4 searches even more than deep re- search models. Details in Appendix G.5
More recent results of our weekly competition are shown in Appendix and a comprehensive review of related works can be found in Appendix B
5 CONCLUSION
FutureX the first live benchmark that tests LLM agents on real-world future prediction tasks. Going forward, FutureX offers flexible platform for improving LLM agents. We are actively working on adding new domains and data sources to FutureX. By keeping the benchmark live and diverse, aim to push agents closer the level of human experts in making timely, strategic predictions across a wide range of fields.
ETHICS STATEMENT
This work did not involve animal subjects or personally identifiable data. All information used are publicly available and were accessed in accordance with their licenses and terms of use.
7 REPRODUCIBILITY STATEMENT
All information necessary to reproduce our results is included, covering benchmark construction procedures, the test date, evaluation metries, and the set of models under evaluation.
REFERENCES
Harrison Chase. LangChain, October 2022. URL httpp///gEhib...o///aghhhhaaaaap
Langchain
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li. Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating 1lms by human preference. In Forty-first International Conference on Machine Learning, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0620 2025.
Yong Guan, Hao Peng, Xiaozhi Wang, Lei Hou. and Juanzi Li. Openep: Open-ended future event prediction. arXiv preprint ar arXiv:2408.0657 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. arXiv preprint arXiv:2501.129.4 2025.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 0731v:2403.07974, 2024.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimban. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference Learning Representations, 2024.
10

---

Published as a conference paper at ICLR 2026
Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter Aram Galstyan, and Xiang Ren. Forecastqa: A question answering challenge for event forecasting with temporal text data. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 1 1ith International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4636 4650, 2021.
Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tetlock. Forecastbench: A dynamic benchmark of ai forecasting capabilities. In The Thirteenth International Conference on Learning Representations, 2025.
Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gon- zalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024.
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating Ilms as agents, In The Twelfth Inter- national Conference on Learning Representations, 2024.
Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: benchmark for general ai assistants. In The Twelfth International Conference Learning Representations, 2023.
Petraq Nako and Adam Jatowt. Navigating tomorrow: Reliably assessing large language models performance future event prediction. arXiv preprint arXiv:2501.05525, 2025.
OpenAl. Introducing SWE-Bench verified, 2024. URL https://openai: httpe/aaai..ommiiee//....a
Daniel Paleka, Shashwat Goel, Jonas Geiping, and Florian Tramèr. Pitfalls in evaluating language model forecasters. arXiv preprint arXiv:2506.00773, 2025.
Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang- Training software engineering agents and verifiers with swe-gym. In Forty-second International Conference on Machine Learning, 2025.
Joon Sung Park, Joseph C. O'Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023. URL
https:
Aymeric Roucher, Albert Villanova del Moral, Merve Noyan, Thomas Wolf, and Clémentine Fourrier. Open-source deepresearch freeing search agents, 2025a. URL https:// whwgingfacc.cco/
co reee Accessed: 2025-07-28.
Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kau- nismäki. 'smolagents': smol library build great agentic systems. httpp::///tthi thub. 2025b.
comm/hugginggface//ssmmn http://gffacee//maageaalooe
ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, al. Seedl. 5-thinking: Advancing superb rea- soning models with reinforcement learning. arXiv preprint arXiv:2504.1399 2025.
Yexuan Shi, Mingyu Wang, Yunxiang Cao, Hongjie Lai, Junjian Lan, Xin Han, Yu Wang, Jie Geng, Zhenan Li, Zihao Xia, al. Aime: Towards fully-autonomuus multi-agent framework. arXiv preprint 1xXX::2507.11988 2025.
Significant Gravitas. AutoGPT. URL htt.t//te.t////./......oooooaooa Gravitas/ AuT
wttp://twwwtccccoiiiiiiicaaaaaae 1/ //www. ww/.ll///er.abbog/////.o oio.ettteee../bloooooffuuuubbbrr futurebench Accessed: 2025-07-27.
Together.ai. Futurebench: Evaluating agents' future prediction capabilities, 2025. URL https
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpos language understanding systems. Advances in neural information processing systems, 32, 2019.

---

Published as a conference paper at ICLR 2026
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12511 2025a.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung. Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025b. URL https://arxv..or///D..
2254.12516
Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, et al. Livebench: A challenging. contamination-limite Ilm benchmark. In The Thirteenth International Conference on Learning Representations, 2025.
Jack Wildman, Nikos I Bosse, Daniel Hnyk, Peter Mühlbacher, Finn Hambly, Jon Evans, Dan Schwarz, Lawrence Phillips, al. Bench the future: A pastcasting benchmark for forecasting agents. arXiv preprint arXiv:2506.21558 2025.
John Yang, Kilian Leret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents. arXiv preprint arXiv:2504.21790 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URL https://arxiv.
hrg/aba/2210: 03629
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. tou-bench: A benchmark for underline (T) ool-) {A} gent- vent-\underline {U} ser interaction in real-world domains. In The Thirteenth International Conference on Learning Representationss 2025.
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench A multilingual benchmark for issue resolving. arXiv preprint arXiv:2504.02605 2025.
Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. Swe-bench goes live! arXiv preprint arXiv:2505.23419, 2025a.
Wentao Zhang, Ce Cui, Yilei Zhao, Yang Liu, and Bo An. Agentorchestra: A hierarchical multi- agent framework for general-purposs task solving. arXiv preprint arXiv:2506, /2508, 2025b.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. In The Twelfth International Conference on Learning Representaiinss 2024.
Andy Zou. Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song. Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural net- works. Advances in Neural Information Processing Systems, 35:27293-27305, 2022.
12

---

Published as a conference paper at ICLR 2026
A WEEKLY RESULTS OF THE FUTUREX ONLINE COMPETITION
Following the release of FutureX, we launched a weekly online competition that has attracted par- ticipants from both industry and academia, including MiroMind Foresigh Peking University, as well as many independent researchers and practitioners.
In the main body, we report results on 1,272 events conducted between July 20th and August 3rd In this appendix, we provide complementary results on weekly basis using more recent data. In order to lower the participation bar, we sample 100 events every week for the weekly test. For the weekly competition results. in addition to the models evaluated in the main body, we also include GPT-5 Pro (web), ChatGPT Agent (web), Gemini Deep Think (web), and Claude 4. 1 Opus.
P'aticipstor
Open ReepReoarch
Cloaa DegpReoarch
Oveeiil Svee Scote
hd
Herperdl (ranidamee
2318
cPy Rouin Paattamm)
grtal
reets
Cemmand dA play
Cuaid- Lita
Frenaic Sope foonsaan
Feali dislit
Ne (hatha)
Diveain fict)
Quaak-110 teet)
Chant chas
Andistal (Cpsutt LPB0
Deppels- hees
Oft-strte foos
indisiedd opppissa
Guati the
areit dwe)
fod cietl
Figure 6: Results of events between August 27th and September 2nd
Deppooond
Optr Deeplemserch ILM:TiiisdSSonnd Wil))
Indriittall Participater
Uresaall Scare
2
Unepuct vouulagnt)
(moad 2h0
ort aaalii (Sad))pe))) (pnn))
Gomba Are Fowdlann
desibu 5 (as)
am (Sáywal)
(Pea ant praun (and)
OPfatesia (ica)
Gmmind Cetakktf (90)
Clack Open (neo
Depesal-t #1 Deet
feetts italle Bio
Geniae 2. (owd)
aiinttikee 720 Poriio
Cheet Ageer twes
cacn sttrals daond
Hierrs
qustai Ayo
Figure 7: Results of events between September 3rd and September 9th
BeseLLM LLM
Closed DesyResearhh Individuall Pariiipune
LLNCThinkkSeechh Web)
5
Dopuak 11 Bandlooo
Deymot df
Dapood- by
ae (es))
Poedd shani (ens)
OPe-4 thin (geni)
cary saati Simddgrs
comi 20106610
the Damulpped
Gomer Sphee ano))
Shodio Lo
Cred Coddi
Fresigh
Doywook at fudi
Blsiied Sa (8))
corcctar (mdg)
Gemudd Saahu Pocciis
OPPic ofe spee 196)
derras
Borrr on
Figure 8: Results of events between September 1h and September t6
hettss://miromind.att https://foresight. Tightni ngrod. /
13

---

Published as a conference paper at ICLR 2026
As the competition progresses, observe that purpose-buil agents, such MiroFlow with GPT-5 and the Bbacktesting Agent with G consistently outperform general LLM web products and deep research agents. This finding indicates that the task of future prediction is, to some extent, both learnable and optimizable. Moreover, such results suggest the potential for agents to eventually surpass human experts, further underscoring the significance of our FutureX benchmark.
B COMPREHENSIVE RELATED WORK
In addition to recent benchmarks for future prediction (see Section 2) this section introduce relevant literature on L.M agent benchmarks, encompassing both established and recent live evaluations.
Agent Benchmark. A new wave of benchmarks is designed specifically to evaluate LLM-bbased agents on complex, interactive tasks rather than isolated questions. For example, AgentBench (Liu jet al. 2024) introduces 8 distinct simulated environments (from operating systems and databases to web interfaces and games) to assess agent's 's' planning, tool use, and decision-making abilities. WebArena (Zhou et al., 2024) provides
simulations of real websites (e-commerce, fo- rums, collaborative coding, content management) test long- monghooizoon web interaction tasks, where Agents must execute multi-step web browsing actions to accomplish user goals. Another bench- mark, GAIA (Mialon et al. 2023), focuses general-purpose assistant capabilities with 466 real- world questions that require reasoning, multi-modality. web search, and tool use. In the realm of information retrieval, BrowseComp (Wei al., 2025a) consists of 1,266 "inverted" questions de- signed to entangle information and thwart simple lookup. Agents must perform creative, multi-hop web searches to uncover hidden facts, testing their persistence and strategic search skills beyond basic fact retrieval. In software engineering, SWE-bench (Jimenez et al., (024) evaluates agents on resolving real GitHub issues by generating code patches and verifying them against project test suites, and based on this, multiple variants are proposed, such as SWF-bench-Verrine
(OpenAT 2024). SWE-gym (Pan al., 2025), Multi-SWEE-hench (Zan et al. (025). and SWE-smith (Yang al. 2025). Collectively, these benchmarks offer valuable insights into various aspects of agent performance from web navigation and tool use to coding- they primarily operate controlled environments with predefined task scopes and information. Additionally, these benchmarks do not integrate complex reasoning and information iiformationngaaherinng gathering capabilities, both of which are essential for our proposed FutureX.
Live Benchmark. Besides traditional benchmarks with static question sets, live benchmarks have recently emerged, such as LiveBench (White al. (025). LiveCodeBBench (Jain et al., (024) and SWE-bench-liv (Zhang et al.] 2025a). which automatically update questions to enable more reli- able, contamination- evaluations. Besides, Li al. (2024) introduce Arena-Hard that fre- quently updated from live data in Chatbot Arena (Chiang et al. 2024) to avoid potential over-fitting or test set leakage. Our proposed FutureX follows this trend, offering fully automated, live bench- mark for future prediction where even the events themselves are live. Compared to the benchmarks discussed, ours is more aligned with real-world professional analysis scenarios across multiple do- mains, making it highly practical.
TThese agents are given by the participators and currently not open-sourced.
14

---

Published as a conference paper at ICLR 2026
C DETAILS ABOUT DATA CURATION AND FILTERING
Q Fature Event Curation (Section 3.2.2)
(195 webelts in told))
1. Event Maripviation
2. Event Fibering
Coowting
Duy Hansfal
Desioasple
arsti-hhsie erents
erems
231
30t
24 -Jdy
New velssle
Terglaar Genomticn
200
Varobiz Runkmmattion
Liifert Sampinng for Templaze Daily and Waddy
4W0 spon-ended
Ceveerace vitsiio
Recline lre plaffrm
108 Okly A11)
11 weakl suaplase
dallyhanddy woees
Figure 9: The detailed future event daily curation process, which manipulates daily and weekly events from the event database. As an example, the number of events between July 24th and July 31 remaining after each step shown in red. Note that the number of events each week varies due to fluctuations in prediction market events (the first row in the figure).
Based the event templates, perform daily future event curation generate prediction questions for each day. As shown in Figure this process consists of two main steps: event manipulation and event filtering.
Event Manipulation format suitable for our pipeline, which varies depending on the type of website.
This phase involves transforming each website into a concrete future event
Prediction market websites There are already feature future prediction events on prediction mar- ket websites. Therefore, we crawl these events on daily basis, which are typically binary or multiple-choice. For each event, we employ the troduce set of unrelated (random) choices, thereby increasing the complexity and challenging the system's is search and reasoning capabilities at higher level. For instance, if LLM agents were to query each choice individually, would significantly reduce efficiency.
model (Seed et al. 2025) to in-
Other types of websites For other websites where future events cannot be directly obtained, follow these steps to make more challenging open-ended questions:
Template Generation: We create "event template" for each site, which can take variables (for example, target, date, etc.) as inputs to make the events adaptable over time, even for the same website. The process is follows: First, we use LLM design candidate question templates based on the content of each webpage. Next, we specify the input variables for answer scraping. After the answer is scraped, the LLM checks whether the answer can be successfully retrieved. To ensure reliability, human reviewer then verifies the results and selects the most appropriate question templates. Once an event template is established, it remains fixed within our pipeline, eliminating the need for recreation in subsequent iterations. Below are two examples.
Website 1: httts:://wwet httrps://ww..dcnncceei.. com/ (China's largest car review website) Variables: rank, date, target
Event Template: Which car will be ranked [rank] on the {target)} board on [date] } at Dongchedi?
Website 2: hhttts:://ww..eettt http://wwwwwgolcoo////o: com/ www.oofnanaa.o....oooccoce finance/
Variahles: stock, date
Event Template: What will be the highest point of {stock) on {date]?
Randomization: Based on the event templates, in order to prevent asking the same future event every day, we apply randomization to our templates by varying the input variables within the same event template (and for the same website). For example, for ranking web- sites, we may ask the LLM agents to predict different ranks each day or predict ranks within different sub-ranking lists. For govermment websites, we may request different statistics or
15

---

Published as a conference paper at ICLR 2026
metrics. For real-time data platforms, may focus different markets, indexes, or stocks. Below are some examples.
Template Which car will be ranked {rank} on the {target} board on {date) at Dongchedi?
Q1. Which car will he ranked /st on the SUV Popularity Ranking board on September 1st at Dongchedi?
Q2. Which car will be ranked 3rd on the MPV Sales Ranking board on September 15th at Dongchedi?
Template 2: What will be the highest point of {stock) {date)?
Q1. What will be the highest point of APPLE on September Ist?
Q2. What will be the highest point of NVIDIA September 7th?
After manipulating the events, we are able to generate ~500 daily and weekly future events as candidates out of 195 high-quality websites. Note that each future event is associated with an answer resolution date that will be used the Answer Daily Acquisition phase (see (octiion3.2.2)
Event Filtering For events crawled from the internet (for example, prediction market websites like Pol colakeeee and gjopen), we carefully filter the event set before testing the LLM agents to ensure the validity of the events, where mainly filter out easy trivial events, harmful events, and subjective events.
Harmful events: These events include content that may involve discrimination, hate speech, or other harmful factors. Such events can introduce bias propagate misleading informa- tion, undermining the integrity of the predictions. To mitigate this, we use combination of Seedl.5-Thinkiee Sced et al. (2025). DeepSeek-RI Guoo et al. (2025), and Gemini-2.55 flash (Comanici et al. 2025) to filter out harmful events from the set before testing to ensure that only appropriate and reliable data is used.
Subjective events: Events that rely on individual opinions subjective judgment are dif- ficult to predict reliably. These events introduce significant variability in responses, which can disrupt the testing process. To address this, we use LLM-as-a-judge to filter out such events with a combination of Seed1.5- 1.l...5Thhin Thinkiing [Seed et al. (2025), DeepSeek-RI Guo let al. (2025), and Gemini-2.5: 2.5-fllsh (Comanici al. (025) to ensure reliability. As shown in Figure 9 between July 15th and July 22nd. we drop 294 unsuitable events (both harm- ful and subjective events). Examples include: "Will win 100k tomorrow at the mara hackathon?" and "I finish Park's 's "Our Nation's 's Path" by EOM July?".
Yes-or-No events: Events with binary choices (such as yes or no, or the outcome of single match) are relatively easy to predict, with even random guessing achieving an accuracy of 50%. Since we cannot introduce additional choices for these events, we significantly down- sample these yes/no binary events. As shown in Figure 9 events collected from prediction market websites are reduced from 193 to 36 for one week.
Through event filtering, we significantly reduce the number of events from prediction market web- sites to make our benchmark more challenging. This ensures the high quality of FutureX and stands sharp contrast to previous benchmarks (Together.ai] 2025 Karger al. 2025), where prediction market events dominate.
Additionally, to maintain event diversity and prevent homogeneity across other types of websites, we randomly select only resulting in 61 daily events and 71 weekly events (61x7+71-498 open-ended events every week).
one question per template per website for inclusion in the daily prediction set,
D DETAILS ABOUT FUTUREX DATA
Comprehensive Domain Coverage As introduced in Section 3.2 FutureX achieves comprehen- sive domain coverage through the daily curation of future events from 195 high-quality websites. Between July 2hth and August 3rd our dataset comprises 1,272 events, systematically categorized into 11 main domains-incluun polities, sports, crypto, culture, finance, business, technology
16

---

Published as a conference paper at ICLR 2026
Cormanoon
Godec Ooleotes
Hapors Price
Miric
Polnics
Sports
Ellertinn Phice
Grypto
Culture
le
Sports
Politics
Vikurtio
Usnural Tress
Tooprraee
Cryplo
Finance
Earoin
Business
Technology
Weather
Finatro
At Modi Rohioss
Gomus Spreet
NASA Aleelonk
Pasle Statt Toeak
Health
Space
Speen Lanoch
Poreetic Eress
Figure 10: Detailed domains of all events included in FutureX, from July 20th August 3rd, total 1,272 events. These events are organized into 11 main categories politics, sports, crypto, culture, finance, business, technology, weather, health, and with each category containing several sub-categories.
trends, weather, health, and space each further refined into several sub-categories. As depicted Figure [O] the distribution across these domains is notably well-balanced, which facilitates a ro- bust and comprehensive evaluation of L.M agents across multitude of real-world scenarios. This comprehensive domain coverage provides two key advantages:
1. It enables holistic assessment of LLM agents' overall future prediction capabilities, as diverse fields often necessitate distinct analytical approaches and specialized reasoning strategies.
2. Coupled with FutureX's live updating, this breadth makes our benchmark significantly more robust against exploitation overfitting, and anticipate this will ensure it remains a challenging and relevant evaluation for the foreseeable future.
We show the examples in major domains in Table 4
812
201
Number
100
Level 1
Level
Level
Level
Figure 11: Difficulty level distribution (between July 20th and August 3rd).
17

---

Published as a conference paper at ICLR 2026
Table 4: Examples be Predicted by Domain. We take the date August 20, 2025 as an example, which can be replaced with any time in the future.
Domain
Event to be Predicted
Culture & Media
Please predict the Top 10 Gross in US dollars on Box Office Mojo's
Daily Box Office for August 20, 2025, Beijing Time.
Finance & Economy
Please predict the grain price index within the 'Agricultural Product Wholesale Price 200 Index' from the National Agricultural Product Wholesale Market Price Information System for August 20, 2025, Beijing Time.
Technology
Please predict the market share percentage in the operating system rankings published by 51.LA for August 20, 2025, Beijing Time.
Crypto
Please predict what the Fear and Greed Index on CoinMarketCap will be August 20, 2025, Beijing Time.
Business & Companies
Please predict which models will be in the top five of Dongchedi's s national popular sedan rankings for August 20, 2025, Beijing Time.
E DETAILS ABOUT EVALUATION METRIC
As introduced in Section 3.3 we have multiple types of events in the benchmark, each with a differ- evaluation metric. As for single-choice events, the metric is simply the 0-1 error:
score (Y, 8)= I(Y Y).
For multi-choice events, as the answer contains multiple right options (denoted by )), the metric is as follows:
score )) FI-Score()).
For open-ended ranking events, such as predicting the top-k ranked items, we treat the ground truth as an ordered list, denoted by (y1....,ye) To evaluate predictions . Us), we design the following metric:
1,
if y1 Di, for i I....,k otherwise,
k
score(fyn...... yec, (ú1..... Dnt) 0.8 Hu (D( U.H
(1) where partial credit (80%) is awarded based the overlap between the predicted and ground- truth sets, For open-ended numerical prediction events, where precisely forecasting the outcome is particularly challenging, we evaluate prediction accuracy relative the outcome's recent volatility. Specifically, we define the score as:
y (a11(--))) 1 o(Y)
score 8) max 0,
where o oYY denotes the standard deviation of the outcome over the past 7 days. Intuitively, predic- tions that fall within one standard deviation of the true outcome receive partial credit, while those beyond one standard deviation receive score of zero.
F DEALING WITH MISSING PREDICTIONS
One challenge is handling missing predictions. Since the pipeline runs a daily basis, if model fails to produce a prediction on a given day, it cannot retroactively provide it later. Given the large number of models evaluated, well occasional API instability cases where model may refuse to predict certain future events, missing predictions are inevitable. The ideal approach to handling missing predictions is to align the events across all models. However, because the events
18

---

Published as a conference paper at ICLR 2026
00130
0.023
Weekly Score Sid
0.1.1
8.010-
0.0000
00991
0.039
12.5
17.
Missing Rate %
01. Gemin 2.5 flast UR
07. GPI--o-ainn (Think&Search)
13. SmolAgent (Qwen)) (Qwen--215B)
19. SmolAgem (GPI-tommm)
Douilon-Soti 6
05. Owor 5-2359
14. DappSeck-V3
20. (PT44 I
Crok-4 (TorhideSorrc)
05. Tamyun (Think&Seanch)
18. SmelAgemi (GPT-4.)
21. SmwuAgmmmOenn)-000 (Qwen)-32D)
Sead i-DeoaRao
10. DespSeal-Rt
16. Ginnini-2.S-Paah
22. GP (P144ommini
DecsSack-RI (Tarsh&SSarcii
SmolAyent (Gominii-2 S-ltah)
17. GPT-4c (Think&&Search)
23. QwBr (Thini&Sardi)
SmelAgent (DeepSeck-V3)
12. Seedl á DeuBao DR
18. Qwen3-32B
Figure 12:
Standard deviation vs. missing rate. The missing rate x ranges from 1 1% 20%; for each model, we plot the standard deviation of its average score.
Addttionl Sid Ratio
0.1
0.10
0.6-
204-
0.02
2.5
58
7.5
1
12.5
15.0
17.5
200
Missing Rate %
01. GPI-4 (Think& Search)
07. SmolAgmt (GP m-4e-mini)
13. Huryson (Think&Serrch)
19. Seee--e
02. Qwen3-255B (Think&SSemm))
GPT-4
14. AmelAgent (Qwml-2355))
20 Qwen)-3358
03. Semini-2.5-fiaa DR
09. GPT-4o-mini
15. DospStek-Rt (Think&SSercc))
21. SmolA (Qwen)-320)
Sood1 6-Dmilbu
SmolAgei (DaupSec--3)
16. Dunilonn Seaak1
22. Gemini-2 5-Blsh
06. Grok-4 (Thik&&Srarch)
11. Qwen3-321
12. Amellgent (GPT-4.I)
23. Sock-V)
06. GPTiol-miii (Tannh&&Saarch)
SmolAgemt (Gamini2..5-laeh))
18. Saall Daultaa DR
Figure 13: Additional standard deviation ratio vs. missing rate.
with missing predictions typically differ between models, full alignment would substantially reduce the total number of events, which is undesired.
Based on our data, we first analyze the standard deviation (std) introduced by missing predictions using a Monte Carlo simulation. Since we have roughly 500 events each week, in each simulation trial we first sample 500 events (with both results and predictions) and calculate the "true" average score 8;. We then randomly sample (100- - k)% of these events, where k% denotes the missing ratio, varying from 1% to 20%, and compute the "pseudo" average score &G. This process repeated 20,000 times, after which we calculate the "true" standard deviation, Std(S), for (81,..., $20000). and the "pseudo" standard deviation, (td( for ( The "pseudo" standard deviation is plotted in Figure 12 against the missing rate. In addition, we quantify the relative increase in standard deviation (SStd(S) - Std(S))/Stdil St)) with respect to missing predictions in Figure 13
$20000)
The results indicate that the standard deviation remains relatively small. These values are computed from 500 total samples, which roughly corresponds to one week of data. As FutureX progresses and the test size grows, the standard deviation is expected decrease at a rate proportional to the inverse square root of the sample size. Therefore, given the complexity of the auto-evaluation pipeline and the relatively minor impact of missing predictions, we prioritize increasing the test sample size over strict ali alignment, allowing for slight misalignments across different models.
We plot the additional standard deviation ratio introduced by missing predictions in Figure [3 Note
Std 9) Sid(
that the ratio is calculated by
Std(S)
19

---

Published as a conference paper at ICLR 2026
Crypo
Culture Midaa
B
Coeparie
Thonee Rooneyy
Techooogg
Tlanace HenneRy
Techaslagy
Sprrt
Spert
Pikki
DepBook
GPE41
Owerk 255R
BeyBook VI
GPT
Ownrk 128
DerpSenk
GFT-4
Owent 2138
Denass
Thccking
Geski pm
Gurval ym
Oweet
(a) Base LLM
(b) SmolAgent
Miniis
seedaci Campatios
& Midia
SBatlaib Companias
Miedkkieess
Sperb
Spebb
Deptock-i: dt
Gri44
GP7-4
Hazque
Scalld (DoaBaa)
GPiiekesss
Ganzn Sankk
Secil (Dtu DI)
(e) LLM (Think&Search)
(d) Deep Research
Figure 14: Performance across different domains for Level (Basic Tier) and Level (Wide Search
Tier) events.
G MORE EXPERIMENTS AND ANALYSIS
Due to space limit of the main body, we put more experimental results and analyses here.
G.1 RESULTS ACROSS DIFFERENT DOMAINS
In addition to the overall performance, we present domain-specific results in Figure 14 and Fig- [3] which highlight the relative strengths of different models across various subject areas. Given the performance gap between Level 1&2 and Level 3&4, we draw two set of figures respectively. Several interesting observations include:
Different models have different strengths. GPT models-inclluing GPT-4.1 (Base LLM), GPT-4.1 (SmolAgent), and GPT-o4-mini (Think&Search) demeoosst))dddoosttttttte supe- rior performance in Crypto and Technology. DouBao-Seed 6 6-Thinking excels in Fi- nance&Econone and Business& Companies, while DeepSeek- V3 (SmolAgent) performs exceptionally well in Politics, even outperforming closed-source deep-research agents and Searh& LLMs.
Search-enhanced reasoning significantly improves performance information- driven domains. For domains like Culture & Media and Technology, performance in- creases notably as we move from basic to more advanced reasoning frameworks. This is likely because these domains benefit directly from timely information access and contex- tual reasoning grounded in real-world updates.
Tool using increases the performance differences. As shown in Figure 14 for Level and Level events, the performance differences among the base L.MSs not very large,
20

---

Published as a conference paper at ICLR 2026
Coype
Coype
Spsoo Comperio
Reitecs Caspasies
448
Culann Meda
Teckeokyy
Terkodogg
Pinnne Depanto
Experee
Klaickasnes
Depfick 31
GFF41
Deppicu- V3
Qvesi-320
DepSeck-V
Gri-
Qwent 3358
Destua-Sedd Soataan
GPT41
Gamal23-pa
Quad 328
(a) Base LLM
(b) SmolAgent
Orppe
Canpusics
Compaslo
Mada
Tachodogg
Timase& & Boquse
Hlanec Boueees
Duppock-it
Soudl (DodBlo))
Geman-- Baab DR
Saucil DDD DEI
(c) LLM (Shink&Searcc)
(d) Deep Research
Figure 15: Performance across different domains for Level (Deep Search Tier) and Level 4 (Super
Agent Tier) events.
but as tools are added, the gaps in the radar charts widen, likely reflecting each model's choice of search tools and reasoning style.
Advanced searching alone may not be sufficient in complex, abstract domains. In contrast, for Level 3 and Level 4 events (see Figure 15). even models equipped with strong search capabilities struggle to perform well. These open-ended tasks often demand deeper abstraction, multi-hop reasoning, and a synthesis of heterogeneous information, revealing a potential limitation in the current design of search- aagre-ee agents.
In addition to the overall results, we conduct several in-depth case studies to further understand the current models' limitations.
G.2 FACTOR ANALYSIS
To systematically investigate the impact of each factor, such as the choice of LLM, event domain, and difficulty tier, we perform a linear regression analysis on each model's 's score for each event. Note that the target variable is the score of each model on each event. solshoss the estimated coefficients for each factor, with *** indicating statistical significance (p < 0.005). From the results, we have the following observations:
Difficulty level really matters. Consistent with our earlier findings, difficulty level has a significant impact model performance. This also validates our overall scoring scheme, in which we assign 10% and 20% weights to Level 1 and Level 2 events, respectively, to place greater emphasis on more challenging cases.
21

---

Published as a conference paper at ICLR 2026
Domain also matters. We observe substantial variation in the coefficients of different domains. This highlights the importance of domain-specific challenges and the need for tailored evaluation.
Top models align with the overall leaderboard. The four (Grok-4, GPT-o4-mini, Gemini Deep Research, Seedl.6 (DouBao)) in our per-domain analyses are exactly the same as those in the overall score ranking, confirming the con- sistency and robustness of our benchmark.
models
Difficulty Tier
Domain
Model Name
Others
63-
Coeffiiient
Lovel
Lovel
Wester fortsonuu
Leves
Levela
Husl Medione
Bestaon & Compuciry
Plesce focorrey
Spatts
Mus dolovaaee
Calom Mella
Telonsogy
Space
Chokel
os midi
Cenani Ait
Decian deall Thonkiy
Deseteel Rt
(Aves car
GP 1
Horgan Hleptttd Loaatiiy
Capp
Figure 16: Coefficients of different factors in linear analysis. The R22 is 0.418.
G.3 PAST PREDICTION vs. FUTURE PREDICTION
Future SmolAgent
Future LLM(Think&Search)
Past SmolAgent
Past LLMrTinn&&Serrrr)
65-
Score
03-
64-
63-
02
Gpok-4
Gemtim-255 GP GrT-ot-mln
Qwen3-2358
GPT-4o
Hunycan
DeepSreb V23
GPT-Ao-coor DeepSrek-R Qwan3-2358 GPT-4.1
R1
Qwen8-328
Figure 17: Comparing Past and Future Predictions. We randomly select 30 events from Level and Level 2, then evaluate model performance on two tasks: predicting outcomes before they are known (future prediction) and searching outcomes after they have been resolved (past prediction).
To more precisely assess search capability, we introduce past-prediction task which models retrieve each event's 's outcome one week after its resolution date. Using the same set of 30 randomly selected events from Level and Level 19 we report performance scores for both past- past-prediction and future-prediction in Figure [7] In the figure, dark bars represent predicreiitiin results, while shallow bars represent nuture-prediction results. From the results, we find that:
Grok-4 leads search capability, followed by GPT models and Hunyuan. In the past-prediction task, Grok-4 significantly outperforms all other methods, underscoring its robust and timely information retrieval. GPT-o4-mini and GPT-4o also deliver strong past-prediction performance. Interestingly, although Hunyuan achieves impressive past- prediction results, the large gap between its future- and past-prediction scores suggests weaker reasoning ability-meaning that despite effective search, its overall performance suffers.
WWe focus Level and Level 2 events because their outcomes are readily retrievable.
22

---

Published as a conference paper at ICLR 2026
SmolAgent with Gemini-2.5-pro achieves significant gains, unlike with other base LLMs. Within the open-source SmolAgent framework, integrating Gemini-2.55 2.5mii-2.5-pr yields marked improvement in past-prediction mercial models (except Grok-4). Crucially, this demonstrates that SmolAgent's relatively modest overall performance cannot be blamed solely the quality of its search API. By contrast, when paired with other base LLMs, SmolAgent shows little to no improvement- and in some cases, even performance Seriine-Sugggsinng that the underlying search capabilities of those models play significant role in the agent's 's effectiveness.
most com-
G.4 PLANNING ANALYSIS OF SMOLAGENT
To understand the agent's performance, we examine the agent memory of SmolAgent when paired with different LLM backends. As cannot access the internal memory of closed-source models, our analysis focuses on SmolAgent.
SmolAgent's memory contains plan solve the problem, detailed log of tool usage, and the outcome of each tool call in every iteration, providing a rich foundation for our analysis. Examples of full memory shown in Section ! For each study plan generated by agent, first conduct an evaluation using Gemini-2.5-pra which assigns a score ranging from to 10 across three key dimensions: (1) comprehensiveness (assessing the extent which the plan covers all necessary components and relevant information), (2) source reliability (evaluating the credibility and validity of references or data cited in the plan), and (3) plan actionability (measuring how practical and executable the proposed steps are in real-world scenarios).
To ensure the fairness and objectivity of the evaluation process, we anonymize the model identity by replacing the original model name in the prompt with generic label "model-i" (where i is unique numerical identifier). This anonymity mechanism is designed to prevent Gemini-2.5-pp from exhibiting potential bias, specifically, avoiding any tendency to inflate scores for study plans generated by models within the Gemini series. By eliminating such identity-based influences, we aim to obtain more accurate and unbiased assessment results that truly reflect the quality of each study plan.
As shown in Table 5 GPT-4.1 and Gemini-2.5-pro obtained significantly higher planning scores compared to other models. This result consistent with their superior overall performance demon- strated in Figure which suggests strong relationship between an agent's planning capability and its prrurtopredictton performance.
A closer look at Table 5 reveals key differences in how models approach planning:
Comprehensiveness:
Powerful models like GPT-4.1 earn high scores for comprehensive- ness by generating plans that address a wide array of specific and sophisticated risk factors. In contrast, weaker models like Qwen3-32B produce plans that are often superficial, cov- cring only basic elements and lacking the necessary depth or specific guidance.
Source Reliability: We observed that strong models like GPT-4.1 consistently leverage authoritative and specialized sources for information. Conversely, weaker models such as Qwen3-32B frequently pull information from unvetted sources like Twitter, compromising the reliability of their plans.
Plan Actionability: Interestingly, certain models, including Deepseck-v3, sometimes reach conclusion within the planning phase itself. This behavior suggests that these models may not always rely on search tools to solve problems, instead leveraging their internal knowledge base form final plan.
23

---

Published as a conference paper at ICLR 2026
Table 5: Analysis of agent planning by scoring the memory in Comprchensiveness, Source Reliabil- ity and Plan Actionability. The predicted event the shown example is "What price will Ethereum hit July 21-27?"
Model
Criterion
Example
Score
Analysis
GPT-4.1
Comprehensiveness
Upcoming catalysts or risk factors affecting ETH from until July 2025 (e.g.,
Covers a wide range of specific and sophisticated risk factors.
expected upgrades, ETF approvals/rijections, major regulations, known hack/theft risks, etc.).
Source Reliability
Implied expectations from Ethereum derivatives markets... Source:
Identifies authoritative, specialized sources advanced metrics.
Deribit, CME, other derivatives market data/analysis.
Plan Actionability
5. Search for information from the derivatives markets (particularly options and futures pricing for ETH with expiry around July 2025) to infer market-implied price expectations and volatility.
9
The step is clear, specific, and executable instruction.
Gemini-2.5-pro
Comprehensiveness Ethereum's 's Technical Roadmap for 2024-2025: Major upgrades like the "Pectra" fork can act as significant price catalysts. Source: Use the search agent find reports and articles from sources like Bloomberg, CoinDesk, Messari, Goldman Sachs, JPMorgan,
10 Highly specific and knowledgeable, referencing key future network upgrade by name.
Source Reliability
10 Unmatched in its list of specific, top-tier financial and crypto-native sources.
Plan Actionability
Analyze all the gathered information (price history, expert forecasts, technical catalysts, macroeconomies, and regulation) synthesize coherent outlook.
Clearly defines complex but actionable goal synthesis.
Qwen3-32h
Comprchensiveness
Any relevant news, events, or macrocconomic indicators that may influence Ethereum's price during this timeframe. News outlets like Reuters, Bloomberg, Coindesk, or crypto- focused forums like Reddit Twitter/X.
generic statement that covers the basics but lacks depth or specific direction.
Source Reliability
2
Lowers reliability by mixing authoritative sources with unvetted social media for factual research.
Plan Actionability
Use the search agent team member to research any upcoming events, macroeconomic factors that could affect Ethereum's 's price...
The instruction is too broad and non-specific to be effectively executed.
(continued on next page)
24

---

Published as a conference paper at ICLR 2026
(Table 5
continued)
Model
Criterion
Example
Score
Analysis
Deepseck-v3
Comprchensiveness
Facts derive-
Demonstrates deep, specific, and relevant understanding of crypto market cycles.
Correlation between
Bitcoin halving cycles (April 2024) and
Ethereum's 's price 15
months later.
Source Reliability
Sources: Crypto market data platforms
Provides a strong list of specific and respected sources across different categories.
(CoinGecko, CainMarettaap)
Ethereum Foundation
announcements, financial news (Cointelegraph, Decrypt), and analyst reports (e.g., Ark Invest, Glassnode).
Plan Actionability
Finalize prediction: Select the most plausible options... and format the answer (boxed(A, B, ...}.
The plan's 's action is to provide conclusion, which it does immediately, negating the purpose the plan itself.
(boxod{B, D, E}
GPT-4o-mini
Comprehensiveness
Market conditions significant events that may affect Ethereum's price around that timeframe (e.g., regulatory changes, technological upgrades, macrosconomic factors). This information can found in articles or publications cryptocurrency websites or financial analysis reports.
5
Lists standard categories but remains generic, surface-level.
Source Reliability
Fails to name any specific sources, making the plan's quality entirely dependent on chance.
Plan Actionability
4. Review and compile the significant factors that could affect Ethereum's 's price between now and July 2025, including potential regulatory developments or technological adyancements.
4
A vague instruction to "review and compile" without guidance how weigh analyze these factors.
Qwen3-235b
Comprchensiveness
We can calculate potential price ranges using technical analysis tools like moving averages, Fibonacci retracements, etc.
Decent scope, and improves its quality by mentioning specific types of analysis tools.
Source Reliability
Cryptocurrency market forecasts for 2025: To understand expert opinions and analyses regarding the future of Ethereum.
A critical failure. It identifies the need for expert analysis but provides indication of where find it. clear, specific, and valuable step that adds quantitative layer the plan.
Plan Actionability
8. Calculate probabilities
7
for each option based on the analysi sis and select the most plausible options.
25

---

Published as a conference paper at ICLR 2026
Based on this, then perform linear regression analysis to assess the impact of several factors, including total tool calls, search text length, overall context length, and the three evaluation scores. We restrict study to Level 1 and Level events due to SmolAgents' poor performance Levels 3 and 4. As shown Figure 18 we find that:
Number of tool calls, source reliability, and plan comprehensiveness exert the strongest positive effects on the overall score: more frequent tool calling, higher trustworthiness of referenced information, and more thorough answer content all drive substantially higher user ratings.
In contrast, main agent think length carries the most negalive effect: longer accumulated dialogue history introduces noise and redundancy, which hurts the performance.
These results suggest that, for further improvements, SmolAgents could strategically invoke tools, rigorously check and cite reliable information, and maintain concise dialogue histories to improve the performance.
Costhiiee
the Cungeberiiezee
Sou Trrkk
Vea Vea. Phaik Longh
Figure 18: Coefficients of different factors the linear analysis SmolAgent's planning. The R 0.518.
G.5 SEARCH ANALYSIS
Figure 19 shows the average number of web queries performed by commercial LLMs with Think&Search. Notably, Grok-4 issues the largest number of searches among all evaluated mod- els This finding has two key implications: first, Grok-4's high query volume corresponds with its superior performance on the FutureX; second, achieves this result with remarkably low latency- completing searches in less than 5 minutes, compared approximately 30 minutes for Gemini deep research.
Furthermore, two deep research models conduct substantially searches than the rest of the & cohort, highlighting their aggressive retrieval strategies. By contrast, Qwen3-235B and Hunyuan perform significantly fewer web queries, which may help explain their relatively lower performance on the same tasks.
Search Number
30-
10-
Grok-4 (TbiokaSowri)
Geminni..5 4.5. (DouBao
DuepStck-Ri (HiinlaSpurt) Scedl (DudBao)
Qwesi-335B (aakesend)
Honeost (anl)))
Figure 19: The search number of different models.
We are unable measure the search counts for GPT-o4-mini and GPT-4o (Think&Search), these models excluded from the comparison.
26

---

Published as a conference paper at ICLR 2026
H EXAMPLES OF AGENT RESPONSE
This study analyzes the reasoning processes behind agent's response. To understand why their pre- dictions succeed or fail, we selected several lengthy agent responses as our subject of analysis. We utilized the Gemini-2.5-fae
model to anlyze the reason that the prediction is correct or incorrect.
We found that the correct predictions (Gemini-2.55 ...-flahh-Deep-Resccadd comprehensive and credible evidence, while the incorrect predictions come from insufficient data (GPT-4o), unreasonable assumption (GPT-o4-mini) or hallucination (Doubao-Seed 1.eed-1.6- 1.6-Deep- Research).
Grok-4) are based
Question
How many deaths will there be to claim during the riots California until the July 2025?
A. the outcome be 0
B. the outcome he 1 -
C. the outcome be 6 10 outcome be the outcome be 16 20
15
F. the outcome be 21 25
G. the outcome be Above
The answer
Geminn-225ffaash-DeeppRReerr
ANALYSIS
The prediction is correct due to exhaustive evidence showing no riot-related fatalities, proper distinction of unrelated deaths, and alignment with de-escalation trends.
PREDICTION (SUMMARIZED GEMINI-2.5-FLAN
This report provides an analytical assessment of fatalities directly attributable to the civil unrest and riots California from June 6 to July 31, 2025. Based on comprehensive review the available intelligence and public records for this period, there have been no reported deaths directly resulting from the civil unrest.
Key evidence supporting this conclusion includes:
1. Detailed accounts of the June 2025 Los Angeles riots consistently omit any mention of fa- talities.
2. Subsequent protests in July 2025 across the state were predominantly non-lethal in nature.
The tragic deaths three I Los Angeles County Sheriff's deputies an explosion were explic- itly identified as an isolated, non-riot-related incident, as they occurred during the handling of explosive devices as occupational hazard.
event's of fatalities stands in stark contrast to historical precedents like the 1965 Watts Riots (34 deaths) and the 1992 Los Angeles Riots (50-63 deaths), highlighting sig- nificant difference in outcomes.
Based on the exhaustive review of the provided data, the estimated number of deaths to claim during the riots in Californi until the end of July 2025 is O. Therefore, the correct option is A: the outcome be 0.
27

---

Published as a conference paper at ICLR 2026
Grok-4
ANALYSIS
The prediction is correct because it is supported by credible sources, accounts for the contained na- ture of the protests, and accurately reflects the absence of riot-related fatalities up to the prediction date.
PREDICTION
There have been ongoing protests and occasional riots in California, primarily in Angeles, since June 6, 2025, in response to ICE immigration raids. These events have involved clashes with law enforcement, vandalism, arson (e.g.. vehicles set on fire), and the use of less-lethal munitions like tear gas, flash-bang grenades, and rubber bullets. A state of emergency remains in effect, with federal forces (including National Guard and Marines) deployed but partially withdrawn mid-July. Protests continued into late July, including demonstration on July 23, but have been largely contained small areas of downtown LA and have not escalated to widespread destruction.
Multiple sources, including Wikipedia, BBC, LA Times, AP News, and NPR, explicitly state no deaths have been directly linked to the protests riots of July 25, 2025. Injuries total around 20-40 (affecting protesters, officers, journalists, others), with over 575 arrests. Two potential incidents were investigated:
A farmworker died on July 12 after falling during an ICE raid on July 10 (accidental, while flecing: occurred amid protest the site but not from riot violence). body was found unknown/unlinked to unrest.
looting site June 10, but showed trauma and the cause remains
Neither is attributed as riot-related fatality in official reports or media summaries. Given the protests' contained nature, reduced troop presence. and lack of fatalities over 1.5 months, it is unlikely significant escalation deaths will occur the remaining days through July 31. The total is predicted be O.
A
GPT-40
ANALYSIS
The prediction arises primarily due insufficient information collection and flawed logic. The prediction cites two deaths but concludes with a range (1-5) that not well-justified by provided evidence. Additionally, it fails to account for potential underreporting or delayed confirmation of fatalities, which are common in chaotic events like riots. The prediction also lacks clarity on whether the two deaths directly caused by the riots merely coincidental.
Summary: Insufficient and inconsistent logic in extrapolating death count range.
PREDICTION
Based on available information through late July 2025, here is the current situation regarding fatal- ities connected the California riots (primarily in Los Angeles):
According to multiple news reports, no deaths had been officially confirmed during the early days of the protests/riots in June in contrast the 1992 Rodney King riots, which saw over 60 fatalities (AP News).
However, media outlets have reported at least one body was discovered near a looted area in downtown Los Angeles during the unrest (New York Post).
Additionally, California farmworker named Jaime Alanis died from wounds sustained during a protest in early to mid-July 2025 (en. (en.wikipedia.org)
As of July 25, 2025, that brings the known death count to two attributed to events tied to the unrest. While reporting conditions remain complex-ane and it's possible further deaths could he confirmed later-based the best publicly available data now, the most plausible fatality total in Califomia riots through July 31, 2025 is in the 1-5 range. My prediction:
B
28

---

Published as a conference paper at ICLR 2026
GPT-o4-minii
ANALYSIS
The prediction is incorrect because it fails to account for the possibility of unforeseen escalations or new incidents that could occur hetween the time of the prediction (July 28, 2025) and the end of July 2025. The prediction relies limited data (only confirmed death) and assumes the situation will remain stable, which is an unrcasonable assumption given the volatile nature of riots and protests.
Summary: Unreasonable assumption of stability and insufficient consideration of potential escalations
PREDICTION
Based official reports and reputable news sources:
As of June 9, 2025, there (goggrids.com).
reported fatalitics in the Los Angeles protests and riots
On July 11-12, 2025, Jaime Alanis 57-year-old farmworker died from injuries sus- tained during chaotic ICE raid protest, marking the first confirmed riot-related death (peo- ple.com (theguardian.com)
No other deaths directly attributable to these disturbances have been reported through late July 2025.
Given the low fatality rate so far and the nature of these localized events, it is highly likely the total number of riot-related deaths by July 31, 2025 will remain in the 1-5 range.
B
ANALYSIS
The model's prediction completely wrong because its reasoning based entirely fabricated scenario, not real-world trend analysis. The model invented a narrative that riots in California began June 2025 due to a massive ICE raid, and established a false premise that approximately 13 dcaths had already occurred by mid-July. Based on this erroncous, self-created "fact," the model naturally ruled out the possibility of zero deaths and concluded that the final toll would fall within the 11-15 range. The entire analytical process is classic of "garbage in, garbage out," as was analyzing an event never happened.
The prediction failed not because of flawed analytical logic, but because its analysis was based complete "Al ballucination." " The model invented crisis and its casualties, and then predicted outcome based that fiction. In reality, the deseribed events did not occur, making the correct death toll zero.
Sumary:
PREDICTION
Current Situation and Historical Context The 2025 California riots began in early June, trig- gered by large-scale raids U.S. Immigration and Customs Enforcement (ICE) in Los Angeles's Latino-maoority communities. On June ICE arrested 44 undocumented immigrants seven loca- tions, including shopping malls and factories. This action quickly sparked intense community protests, which evolved into ongoing social unrest. By June 11, the incidents had spread to major Califor- nia cities like San Francisco and San Diego, with over 5,000 arrests statewide. 12 deaths, and direct economic losses exceeding $800 million.
situation escalated further in July. On July 3. Trump administration deployed additional 700 Marine Corps soldiers to California, reinforcing federal control over the state. These troops joined the 4,000 National Guard soldiers already the ground to maintain order in cities like Los Angeles. However, California Governor Gavin Newsom deemed the move unconstitutional, further intensifying the conflict between the sides.
29

---

Published as a conference paper at ICLR 2026
On July 10, federal agents clashed with protesters during an immigration raid on Southern Califor- nia farm. On-site video footage showed helmeted law enforvement officers throwing tear gas, forcing protesters and workers to retreat as thick smoke engulfed the area, affecting even children. This op- eration resulted in death and multiple injuries. On July 18, explosion the Biscaluz Training Center in Los Angeles County killed thrce veteran police officers instantly. The incident caused by an unexploded device discovered the day before in Santa Monica apartment garage, which det- onated accidentally during handling, brutally exposing fatal flaw in the law enforcement's high-risk materials handling protocol.
Compared historical Californi riots, the 2025 unrest shows new characteristics. The 1992 Los Angeles riots were triggered by the acquittal of white police officers who beat black motorist Rodney King. The riots lasted six days, caused 63 deaths, and resulted in $1 billion in damages. contrast, the 2025 riots are centered on immigration policy and have evolved into power struggle between the federal and state governments. It noteworthy that although the scale of the 2025 riots considerable, the current death toll (around 13) remains far below the 63 fatalities of 1992. This suggests that the current situation has not reached the same level of violence as historical highs, but given ongoing conflict and escalating confrontation, the developments over the next week require close attention.
Key Data and Influencing Factors According to statistics from reliable sources, as of July 24, 2025, the California riots have resulted in approximately 13 deaths:
1. By June 11, 12 deaths had been confirmed.
2. On July 10, the Southern California farm raid resulted in I death.
3. On July 18, the explosion at the Los Angeles County training center caused the deaths of three police officers, but whether this incident is directly related to the riots is yet to be confirmed.
It is important to emphasize that the above statistics may have omissions, especially considering other violent confficts or accidental events that may occur during the riots.
The power struggle between the federal and state governments is a key factor influencing the develop- ment of the riots. Since taking office in January 2025, the Trump administration has labeled undoc- umented immigrants a "national security threat", setting a goal 3,000 daily arrests and attempting to seal the -eeeio border. This policy directly impacts California's 's economy and social struc- ture, as the state's economy is highly dependent on immigrant labor, particularly in the agricultural and construction sectors. California Governor Newsom strongly opposes the federal government's intervention, publicly accusing Trump's actions of being unconstitutional and describing it as "con- stitutional war". He believes the White House's goal to weaken state power and set precedent for federal intervention in state affairs. This political antagonism not only exacerbates social divisions but also makes it difficult to resolve the conflict through conventional means.
The federal government has deployed a large number of military forces California. In addition the 700 Marine Corps soldiers, includes the 40th Infantry Division of the National Guard and rapid response forces from the Department of Homeland Security. These troops, equipped with riot gear, light armored vehicles, and armed helicopters, have established temporary posts at several strategic locations in Los Angeles. The escalation of law enforcement tactics is also significant factor in the increase of casualties. On-site videos show law enforcement officers using tear gas, rubber bullets, flashbang grenades suppress crowds. This high-intensity enforcement not only directly causes casualties but also easily escalates tensions, leading to further conflict.
Prediction for the Next Week Key factors influencing the death toll include:
Progress of negotiations between the federal and state governments: both sides can reach a compromise. may ease tensions and reduce violent clashes.
2. Scale and intensity of protest activities: During the Independence Day period, protests of varying scales were held many parts the United States, from Chicago to Los Angeles, from Roanoke, Virginia, to Sioux Falls, South Dakota, and Eugene, Oregon. Such large-scale protests carry a high risk of violence.
3. Changes in enforcement intensity: If law enforcement continues use high-intensity sup- pression tacties, it may lead to more casualties.
4. Economic factors: As the world's fourth-largest cconomy, Califomia's cconomic interests may prompt all parties to seek peaceful resolution.
Expert analysis points out that there are fundamental differences between these riots and the 1992 Los Angeles riots. The 1992 riots stemmed primarily from racial discrimination and judicial injustice,
30

---

Published as a conference paper at ICLR 2026
while 2025 conflict more the result political maneuvering. quittal of the four police officers involved sparked a six-day riot, causing 53 deaths, over two thousand injuries. and property damage estimated at over $1 billion. Currently the death toll from the 2025 riots (around 13) significantly lower than the 1992 level. This is partly due faster response from law enforeement and also reflects the differences the organization and intensity of the current protests. However, with the escalation of the confrontation between the foderal and state governments, the situation could still worsen.
1992 riots, the jury's ac-
Considering the fatality trend over the past month and the current situation, we can make the following predictions:
1. Most Likely Scenario: An increase of 0-2 deaths in the coming week, bringing the total to 13-15, 15, This prediction is based on the current relatively stable death toll trend and the likely scale of conflicts.
2. Optimistic Scenario:
the federal and state governments can reach a compromise and protests remain peaceful, there may be no new deaths in the coming week, keeping the total at around 13.
3. Pessimistic Scenario: the conflict escalates further, especially if large-scale violent events similar to 1992 occur. the death toll could increase significandy. However, given the current military deployment and law enforcement capahilities, the possibility of this happening is relatively low.
Final Prediction and Conclusion Taking into account historical data, current situation, future trends, my prediction for the total number of fatalities in the California riots by the end of July 2025 is as follows:
This option clearly underestimates the actual situation, the death toll has already reached around 13 of July 24.
B. 1-5 fatalities:
C. 6-10 fatalities: This option is also below the actual situation, as the current death toll is already
over 10.
D. 11-15 fatalities:
This is the most likely scenario. Based the current death toll of around 13 and the prediction of possible increase of 0-2 deaths in the coming week, the total number of fatalities most likely to fall within this range.
E. 16-20 fatalities: While this possibility cannot be completely ruled out, it would require multiple severe violent incidents to occur in the coming week to reach this level, making it relatively less probable.
21-25 fatalities and Over 25 fatalities: These scenarios would require significant lation in the scale intensity the conflict. Given the current military deployment and political environment, these are highly unlikely.
Final Prediction: The most probable death toll range is D. 11-15 people.
Risk Assessment and Uncertainties must be emphasized that this prediction contains degree of uncertainty, stemming primarily from the following factors:
1. Changes in political factors:
The relationship between the federal and state governments could change significantly in short time, affecting the situation's 's development.
Possibility of external intervention: Other political forces or international factors might intervene, altering the nature and intensity of the conflict.
3. Impact of sudden events: Unexpected incidents, such as the July 18 explosion, could sudden increase in the death toll.
4. Unpredictability of protest activities: The development of large-scale protests is often un- certain and escalate subside rapidly.
Despite these uncertainties, based on existing data and analysis, we have reason to helieve that the total number of fatalities from the California riots by the end of July 2025 is most likely to fall within the 11-15 range
I EXAMPLES OF SMOLAGENT'S PLANNING
Smolagent Roucher et al. (025b) is open-source agent framework where the planning of agents visible. The following examples are planning of SmolAgent's 's agents, which all follows the same structure:
31

---

Published as a conference paper at ICLR 2026
1. Facts Survey
(a) Facts to look up
(b) Facts to derive
2. Plan
Question
What price will Ethereum hit July 21-27?
A. The outcome be S4400
The outcome be $4300
The outcome be $4200
D. The outcome be S4100
E. The outcome be $4000
F. The outcome be $3700
The outcome be S3600
H. The outcome be $3500
I. The outcome be $3400
J. The outcome be $3300
GPT-4.1
1. FACTS SURVEY
1.1. FACTS GIVEN THE TASK
The event be predicted concerns the price of Ethereum (ETH) during the week of July 21-27, 2025, with reference time around 2025-07-28704:00000
Several options are specified, corresponding to price thresholds ETH might reach or dip to during that time:
A. $4400
$4300
C. $4200
D. $4100
E. $4000
F. $3700
G. $3600
H. $3500
$3400
J. $3300
The answer must use the boxed format and list all plausible options.
1.2. FACTS TO LOOK
The current (most recent) price of Ethereum (ETH) in USD.
Source: major crypto price aggregator (CoinGecko, CoinMarketCap. etc.)
Recent trends, volatility. and price movement data for Ethereum in the months leading up task date (currently June 2024).
Source: historical charts from crypto price sites.
Macroeconomic factors, crypto market sentiment, and news that could affect ETH price up to July 2025.
Source: news sites, market outlooks, analyst reports.
Historical Ethereum price behavior during similar periods (i.c., how ETH behaved in prior years between June/July and the following July).
Source: price charts historical returns data.
32

---

Published as a conference paper at ICLR 2026
Upeoming catalysts or risk factors affecting ETH now until July 2025 (c.g., expected upgrades, ETF approvalstrjections, major regulations, known hack/theft risks, etc.).
Source: cryplo news, Ethereum Foundation roadmaps, industry reports.
General market consensus or well-regarded forecasts for ETH price in mid-late 2025.
Source: analyst price targets, industry reports, prediction market odds for July 2025,
Implied expectations from Ethereum derivatives markets (options/futures) expiring around July 2025, if available.
Source: Deribit, CME, other derivatives market data/analysis.
1.3. FACTS TO DERIVE
A range of plausible ETH/USD price targets for the week of July 21-27, 2025, based on available data and trends.
Assessment which thresholds (A-J) are more likely to be reached or not based on funda- mental, technical, and market sentiment factors.
Probabilities relative likelihoods for each option (if possible), identify which boxed options are justified.
Reasoning process rule in/out thresholds based volatility, historical growth rates, trend continuation reversal scenarios.
Synthesis of risk factors or events that could cause extreme moves (upside or downside) by July 2025.
2. PLAN
1. Look up the current price of Ethereum (ETH) in USD, well as its price trend over the past year.
2. Gather historical ETH price data from prior years, especially from June/July to the following July, to understand typical annual volatility and trend direction.
3. Scarch for major events, upgrades, crypto-speeific and macrocconomic risks, and catalysts expected between now and July 2025 that might affect Ethereum's price.
4. Look up analyst price predictions, industry reports, and any well-regarded forecasts for ETH in mid-late 2025.
5. Search for information from the derivatives markets (particularly options and futures pric- ing for ETH with expiry around July 2025) to infer market-implied price expectations and volatility.
6. Synthesize all gathered information to construct reasoned, data-driven price range for ETH during the week of July 21-27, 2025.
Match the forecasted price range to specified answer options (A-J) identify plau- sible thresholds ETH could hit during the target week.
8. Format the prediction in the required boxed format, including all plausible letter options.
Gemini-2.5-pro
1. FACTS SURVEY
1.1. FACTS GIVEN THE TASK
Asset Predict: Ethereum (ETH).
Prediction Timeframe: The week of July 21-27, 2025.
Reference Timestamp: The prediction contextualized around 2025-07-28T04:00000
Prediction Options:
A. Ethereum reach $4400
33

---

Published as a conference paper at ICLR 2026
B. Ethereum reach $4300
Ethereum reach $4200
D. Ethereum reach $4100
Ethereum reach $4000
F. Ethereum dip to $3700
G. Ethereum dip to S3600
H. Ethereum dip to $3500
I. Ethereum dip to $3400
Ethereum dip $3300
Output Constraint: The final answer must be boxed list of plausible options, separated by commas (e.g., (boxed(A, B, C)).
1.2. FACTS TO LOOK UP
Current and Historical Price of Ethereum: need to know current price to have a baseline, and the all-time high and recent price action to understand its volatility and histor- ical cycles.
Source: Use 'search-agent" query financial websites or crypto news outlets.
Expert Price Predictions for 2025: We need to gather price targets and forecasts for Ethereum for mid-2025 from reputable financial institutions, crypto analysts, and investment firms.
Sourre: Use the 'search-agent" to find reports and articles from sources like Bloomberg, CoinDesk, Messari, Goldman Sachs, JPMorgan,
Ethereum's Technical Roadmap for 2024-2025: We need to identify any major network upgrades, forks, or Ethereum Improvement Proposals (EIPs) scheduled for the period leading up July 2025. Major upgrades like the "Pectra" fork can act significant price catalysts.
Source: Use the 'search-agent" to search the official Ethereum Foundation blog, crypto-focused news sites (c.g., The Block, CoinDesk), and developer forums.
Macroeconomic Forecasts mid-2025: need understand predicted global eco- nomic environment for mid-2025, particularly regarding interest rates (c.g., US Federal Re- serve's s stance), inflation, and overall market sentiment (risk-on vs. risk-off).
Use the 'scarch-agent' find reports from major financial (Reuters, Wall Street Journal) international economic bodics (IMF, World Bank).
Source:
outlets
We need to find information any anticipated reg- ulatory decisions changes in legislation concerning Ethereum and crypto assets in major jurisdictions (especially
Regulatory Climate for Ethereum:
U.S. SEC) for late 2024 and 2025. The status perfor- mance of spot Ether ETFs will be particularly relevant.
Source: Use the 'search-agent' to search for articles crypto regulation from rep- utable legal financial news sources.
1.3. FACTS TO DERIVE
Synthesized Market Sentiment: From the facts we look we need to derive consoli- dated view of the likely market sentiment for Ethereum in mid-2025. This involves weighing the bullish factors (e.g., successful upgrades, positive expert predictions, favorable macro environment) against bearish factors (e-g., regulatory crackdowns, economic recession, tech- nical delays).
A Probable Price Range: Based the synthesized sentiment and the specific price targets found, we will derive most likely price range for Ethereum during the target week. This will not be a single point but zone of probability.
Selection of Plausible Options: By comparing derived probable price range with the given options (A-J), we will identify all options that fall within or are very close our analytical conclusion.
34

---

Published as a conference paper at ICLR 2026
2. PLAN
1. Use the search agent to find the current price of Ethereum and its all-time high establish baseline for the prediction.
2. Use the search agent to gather a consensus of Ethereum price predictions for mid-2025 from multiple reputable financial analysts and crypto research firms.
Use the search agent research the Ethereum development roadmap, specifically looking information on major upgrades or updates planned for late 2024 and the first half of 2025 that could act as catalysts.
4. Use the search agent to find macrocconomic forecasts for mid-2025, focusing on projected interest rates and overall economic health, which will influence investment in risk assets like Ethereum.
5. Use the search agent investigate the current and expected regulatory environment for Ethereum in key markets, particularly concerning the status impact of spot Ether ETFs.
6. Analyze all the gathered information (price history, expert forecasts, technical catalysts, macroeconomics, and regulation) synthesize coherent outlook and derive probable price range Ethereum July 2025.
7. Compare the derived probable price range against the provided options (A-J) and select all options that are plausible based the analysis.
8. Provide the final answer by listing the letters all selected plausible options, separated by commas, in the specified [boxed[] format.
Qwen3-32B
1. FACTS SURVEY
1.1. FACTS GIVEN THE TASK
The event predict is the price of Ethereum (ETH) on July 21-27, around 2025-07- 28T04:00:00Z.
There are multiple potential outcomes listed as options A through J.
1.2. FACTS LOOK
Historical and recent price data for Ethereum from reliable financial sources cryptocur- rency market platforms.
Where to find:
Cryptocurrency tracking websites like CoinMarketCap. CoinGecko, financial data platforms such Yahoo Finance or Bloomberg.
Any relevant news, events, or macroeconomic indicators that may influence Ethereum's 's price during this timeframe.
Where to find: News outlets Reuters, Bloomberg, Coindesk, or crypto-focused forums like Reddit or Twitter/X.
Technical analysis reports forecasts about Ethereum from trusted analysts institutions.
Where to find: Reports from investment firms, analyst blogs, or social media posts by well-known crypto analysts.
1.3. FACTS DERIVE
Based on historical trends and current data, identify if Ethereum is likely risc dip in the specified time frame.
Compare the probability of cach option based on available data and expert predictions.
Determine a set of plausible outcomes (options A-J) that align with derived probabilities.
35

---

Published as a conference paper at ICLR 2026
2. PLAN
1. Use the 'search-agent" Team member to gather historical and recent Ethereum price data from reliable sources.
2. Use the 'scarch-agent agentt team member to research any upcoming events, news, or macrocco- nomic factors that could affect Ethereum's price during the specified period.
Use the 'search-agent' team member to collect technical analysis expert forecasts re- garding Ethereum's 's expected performance.
4. Analyze the gathered data determine whether Ethercum is more likely rise or dip be- tween July 27, 2025.
5. Evaluate the likelihood of each option (A-J) based on the compiled data and logical reason- ing.
6. Identify all plausible options based on derived probabilities.
7. Use the 'final-answer tool to present the final prediction in the required format.
Deepseek- V3
1. FACTS SURVEY
1.1. Facts given in the task
The event to predict the price Ethereum (ETH) between July 21-27, 2025.
The prediction options range from $4400 (highest) $3300 (lowest).
The prediction must formatted as \boxed(A, B, ...} for plausible options.
1.2. Facts look up
Historical price trends of Ethereum (especially around similar timeframes in previous years).
Current price of Ethereum and its recent volatility.
Upcoming Ethereum network upgrades events (e.g.. EIPs, hard forks) that could impact price.
Macroeconomic factors (e.g.. Bitcoin halving cycles, regulatory news) that could influence Ethereum's 's price.
Market sentiment indicators (e.g.. futures, options, institutional interest).
Expert predictions analyst reports for Ethereum's price mid-2025.
Sources: Crypto market data platforms (CoinGecko, CoinMarketCapp, Ethereum Founda- tion announcements, financial news (Cointelegraph, Decrypt), and analyst reports (c.g., Ark Invest, Glassnode).
1.3. Facts to derive
Likely price range based on historical growth rates and volatility.
Correlation between Bitcoin halving cycles (April 2024) and Ethereum's price months later.
Impact of known future Ethereum upgrades (e.g., Proto-Danksharding) price.
Probability distribution given options hased on historical data and expert consensus.
PLANN
Research Ethereum's historical price trends: Analyze ETH's price movements similar timeframes (c.g.. 15 months post- Bitcoin halving) to identify patterns.
2. Grab current market data: Retrieve ETH's current price, volatility, and institutional inter- short-term
36

---

Aggregate mid-2025 price forecasts from reputable analysts Published as a conference paper at ICLR 2026
Identify upcoming Ethereum events: Look for confirmed upgrades or events 2025 that could drive price action.
4. Analyze macrocconomie factors: Check for broader market trends (e.g., Fed rate cuts, regulatory clarity) that could impact ETH.
5. Compile expert predictions: and institutions.
6. Derive plausible price range: Combine historical trends, current data, and expert opinions to narrow down the most likely options.
7. Finalize prediction: Select the most plausible options (e.g., $4000 54000-44301 is the consensus range, choose B, C, D, E) and format the answer as \boxed(A, B, ...}.
B, B,C,D,E
GPT-4o-mini
1. FACTS SURVEY
1.1. FACTS GIVEN THE TASK
The event to predict is the price of Ethereum around July 21-27, 2025.
The potential price points provided are: $4400, $4300, $4200, $4100, $4000, $3700, S3600, $3500, $3400, and $3300.
1.2. FACTS TO LOOK UP
Historical price data of Ethereum, especially around July of previous years, to identify trends or patterns. This data can be found on cryptocurrency data platforms like CoinMarkerCap CoinGecko.
Predictions forecasts from cryptocurrency analysts for Ethereum prices in 2025. This information can be found in articles publications eryptocurrency news websites financial analysis reports.
Market conditions significant events that may affect Ethereum's 's price around that time- frame (e.g., regulatory changes, technological upgrades, macroeconomic factors). This also sourced from news outlets or industry reports that discuss future of cryptocurren- cies.
1.3. FACTS TO DERIVE
Trend analysis to determine how historical price movements correlate with specific months events in the crypto market.
An aggregate prediction based on gathered forecast data and price trends, leading to a set of plausible options based on market behavior leading to the specified date.
2. PLAN
1. Gather historical price data for Ethereum around July from previous years using CoinMar- ketCap or CoinGecko.
2. Analyze the historical price trends to identify patterns or significant events that could provide insight into Ethereum's price movements during July in years.
3. Search for cryptocurreney market predictions or analysis related to Ethercum for the year 2025 using news websites or financial analysis platforms.
4. Review and compile significant factors that could affect Ethereum's price between now and July 2025, including potential regulatory developments technological advancements.
5. Synthesize the information collected, relating historical trends to potential future outcomes.
6. Make a prediction based on the derived data, focusing on most plausible price point options.
37

---

Published as a conference paper at ICLR 2026
Format the final answer according to the specified format, ensuring to all plausible price options identified in the box format.
Qwen3-235B
1. FACTS SURVEY
1.1. FACTS GIVEN IN THE TASK
The task asks for prediction about Ethereum's price between July 21-27 around 2025-07- 28T04:00:00Z
* The possible options for prediction are:
A. $4400, B. $4300, C. $4200, D. $4100, $4000
$3700, G. $3600, $3500, I. S3400, S3300
1.2. FACTS TO LOOK
Current Ethereum price and recent price trends: Need to understand the current market situ-
ation and trajectory.
Historical Ethereum price patterns: Useful for identifying seasonal trends recurring pat- terns might help predicting future prices.
Cryptocurrency market forecasts for 2025: To understand expert opinions and analyses re- garding the future of Ethereum.
Macroeconomic factors that could influence cryptocurrency prices in 2025: Such as regula- tory changes, adoption by major institutions, technological developments, etc.
Any significant events developments in the Ethereum ecosystem expected around July 2025: Like protocol upgrades, major partnerships, or ecosystem growth.
1.3. FACTS TO DERIVE
Based on current trends and historical data, we can derive potential price trajectories for Ethereum leading to July 2025.
By analyzing market sentiment and macroeconomic indicators, we can derive likely market conditions for Ethereum in July 2025.
We calculate potential price ranges using technical analysis tools like moving averages, Fibonacci retracements, etc.
Considering all available information, we can derive probability distribution across the given price levels to identify the most plausible outcomes.
2. PLAN
Conduct comprehensive search on the current price Ethereum, its recent performance, and current market sentiment.
2. Investigate historical price patterns of Ethereum to identify any recurring trends or cycles that could help in predicting future prices.
3. Search for expert forecasts and market analyses specifically addressing Ethereum's 's price expectations for 2025.
4. Research macrocconomic factors, regulatory developments, and technological advancements that could impact Ethereum's 's price in 2025.
5. Look any specific information about events developments in the Ethereum ecosystem expected around July 2025.
6. Analyze all gathered information to determine the most likely price range for Ethereum during July 21-27, 2025.
7. Map the derived price range to the specific price levels provided in the options (A-J).
38

---

Published as a conference paper at ICLR 2026
8. Calculate probabilities for each option based on the analysis select the most plausible options.
9. Provide the final answer using the exact box format with the identified options.
J PROMPTS
J.1 PROMPT FOR PROPOSING NEW QUESTION FOR FUTURE PREDICTION
Please help me find some websites data/listi/rrankknnssssaiiiiiii tables.
Requirements:
The catd//isssraaniinnsttttiiiiil tables are real and released by authoritative and well-known institutions.
2. The dataa/isss/aankkiggsstttttttii tables have official websites.
3. The datt/liss/rmmiinessaaiiiiia tables are regularly updated, such as daily, weekly, or monthly. This point is very important.
4. Each aataa/ist/rankin/statiiss table always updated on fixed and unchanging website. This point very important.
5. The fata//istsrankiinss/ttttttt complete content of the data//istrrankinnssttstttt table when you open the wehsite, without any need for clicking, drop-down selection, page-turm page-turning, downloading, etc. This point very important.
tables directly visible, that is, you can directly the
The following
examples that meet the requirements and examples that do meet the requirements. Please visit these websites one by one to understand the above requirements, especially the three points of regular update, fixed and unchanging, and directly visible.
Example that meets the requirements: Top Ten in Terms of Market Capitalization of Stocks the Shanghai Stock Exchange
Website: https: www.. sse.. wwm. www...///aaaeeet/tttoeeeeeeeeeee The list is updated daily, always on this wehsite, and you can directly see the complete list when you open the website.
/et//maarktaaa lue/ma in/
Example 2 that meets the requirements: Billboard Hot 100
Website: htttps:/wwww bil bilaborrd. cem/ ctsm/cchartsss charts/ /hot 100/
The ranking is updated weekly, always this website, and you directly the complete ranking when you open website.
Example that does not meet the requirements: Central Parity Rate of RMB Exchange Rate Website:
fttt:://wet pbc gov. en/ wwwpgcchuoossi/ 12w.0 12111//21111111 1229925/
[ndex... , html
Reason for not meeting the requirements: The data of the central parity rate the RMB exchange rate is updated daily, hut it is updated on different websites every day, which does not meet the requirement of "fixed and unchanging".
Example 2 that does not meet the requirements: Added Value of Industrial Enterprises above Desig- nated Size
Website: https: www. stats gov. cn/s"/2xtx /x//
Reason for meeting requirements: The data of the added value industrial enterprises above designated size is updated monthly, but is updated on different websites every month, which docs not meet the requirement of "fixed and unchanging".
39

---

Published as a conference paper at ICLR 2026
J.2
PROMPT FOR FUTURE PREDICTION
For the multi-choice questions
You agent that can predict future events. The he predicted: "[title} (around {time {options]"
IMPORTANT: listing all plausible options you have identified, separated by commas, within the box. For example: [A) for single option or [Boxed[B, C. DJ for multiple options.
Do not any other format Do not refuse make prediction. Do say "I cannot predict the futuree. You must make a clear prediction based on the best data currently available, using box format specified above.
For the other questions
You agent that can predict future events. The event to be predicted: "Please Predict Beijing Time (time), [title]"
IMPORTANT: Your final answer MUST end with this exact format: PREDICTION Do not use any other format. Do not refuse to make prediction. Do not say "I cannot predict the future", You must make a clear prediction based on the best data currently available, using the box format specified above.
K HUMAN ANNOTATION DETAILS
In order compare recent advanced LLMSs with human experts, conduct human annotation experiments. In this section, we provide more details.
K.1 NUMBER OF QUESTIONS PER CATEGORY
We involve 31 human experts and collect their predictions prior to the resolution date each task.
Since FutureX is a live benchmark, it is challenging to gather human predictions on a daily basis, which makes nearly impossible align the tasks for L.LMs and human experts consistently. As result, we acknowledge that some mismatch between human and LLM tasks is inevitable. The number questions in each difficulty tier is as follows:
Table Scores and Number of Questions by Level
Level-1
Level-2
Level-3
Level-4
Num Questions
106
44
85
Human Score
79%
39%
48%
24%
K.2 QUALITY CONTROL
Information access for human experts. As will be demonstrated below, regarding the informa- tion sources for human experts, our annotation guidelines explicitly require experts to use web search to find relevant information before making their final prediction. This ensures that human experts, like the agents, are tasked with synthesizing information from external sources rather than relying solely on their existing knowledge.
Consistency of answers. We acknowledge that having multiple annotators for each question would enhance the statistical robustness of our results. However, future prediction is a significantly more intteninteesive task than typical annotation. Our protocol requires each annotator spend at least 5 minutes per question, document their thought process and action steps, perform at least one web search, and record their screen for verification. To ensure high-quality annotations under
40

---

Published as a conference paper at ICLR 2026
these demanding conditions, conduct rigorous verification process, including manual checks of the recorded logs and screen recordings for each annotation. Given these expensive annotation and verification costs, prioritized in-depth quality control for single, well-vetted annotations over col- lecting multiple annotations for the same question. We believe this could maintain the high fidelity of our human performance baseline.
K.3 HUMAN ANNOTATION GUIDELINES
We ask human annotators to predict future events by following specific set of guidelines. The core principle of this project is to rely solely on human reasoning and publicly available information. The use of any AT tools strictly forbidden. critical component of the task is for each annotator to provide detailed and logical thinking process that leads to their final prediction. This ensures that every prediction based on verifiable information and sound human judgment, rather than on an Al's output.
Human Annotation Guidelines
ANNOTATION BACKGROUND
We need to collect predictions for future events. You will predict the outcomes of events that will occur within the next days, such the result of sports match company's stock price change. You must gather information from the internet to predict the outcomes of these events that have not yet happened.
ANNOTATION RULES
Do not use large language models (LLMs) or Al software for predictions. If screencast shows the use of Al software, all tasks for that day will be void.
Time Requirements for Solutions (All expert-level tasks are considered difficult):
Each task must take minimum of 5 minutes to solve.
Each task must include minimum of 3 steps.
You must consult a minimum 1 web page.
If screencast shows prolonged pauses other time-wastinn behaviors, the task will be void.
The entire solution process for each task must be recorded in a screencast.
THOUGHT PROCESS PREDICTION RATIONALE
You must briefly write down your thought process and the reasoning behind your prediction.
TEMPLATE FOR THOUGIIT PROCESS & PREDICTION RATIONALE
1. Search Keywords: (keywords). Accessed Webpage: (webpage)
2. Observations:
(observations)
3. My Reasoning:
{reasoning) (The reasoning must clearly explain how you reached the pre- diction from your observations).
4. If a correct prediction can be made. stop. Otherwise, repeat the above steps.
ANNOTATION BONUS
If the screencast and solution process meet the requirements, you will receive the basic reward, if the prediction is incorrect.
If the final prediction is correct and the thought process and reasoning are sound, you will receive an additional bonus (the bonus for difficult tasks is higher than for simple ones).
If recorded thought process and reasoning are unreasonable or perfunctory, you will receive no reward, regardless of whether the prediction is correct.
IMPORTANT NOTES
Before submitting the required screencast videos for this project, please ensure they do not contain your private personal information. they do, please redact anonymize the information before submission.
41

---

Published as a conference paper at ICLR 2026
ANNOTATION TASK OUTPUT
Solution Process
Screencast
Prediction Result
L
WEBSITES
Regarding the list of websites from which we collect data, we have made sure to only include publicly accessible sites that explicitly allow data scraping, or have no restrictions outlined in their terms of service. To ensure compliance, we have adhered to ethical data collection practices and ensured that our actions align with the site's usage policies.
In this section, we list only some of the publicly accessible websites that do not have restrictions on data access, and each website can generate various questions.
http://wwww2occccceaaacaaa22//22 index.html
httes://plaaaaa...aaaaa....///or
http://www.bllllaaaaddddaattttt2
attes://www.3aasssooroo////222/
http::///aaaaaf.faarrrrrrraaaaar 1d14597
h.trss//wetherrccaaao//atttt
https://ww..dnnccceii..co
3ttps://pfsc.egri..cc/
attps:://ueequu.com
attps://ww.hulu.com
httpps:/wweb.1..1bb
http://ww..uff..cn//raaiiiii
stDS://ww.kklraxk.comm
https::/vor.dathheeticcooo
M THE USE OF LARGE LANGUAGE MODELS
In this work, LLMS only used general-purpose language polishing tool improve the clarity, fluency, and grammatical accuracy of the manuscript text, without involving any contribution research content. All contents checked and edited by the authors to ensure alignment with the technical content. The authors take full responsibility for the final version of the paper.
42


Reviews:
[{"review_id": "obUT1RaF7u", "reviewer": "Reviewer_diMG", "summary": "This paper presents FutureX, a large-scale, live benchmark for evaluating base LLMs and LLM agents on real-world future prediction tasks. It features a four-stage pipeline for event database construction, future event curation, agent prediction, and answer acquisition, with all stages executed daily to ensure dynamic, contamination-free evaluation across diverse domains and difficulty levels. Experiments with 25 models show that search- and tool-augmented agents outperform base LLMs but still lag behind humans on complex tasks, establishing FutureX as a scalable framework for assessing LLM reasoning in uncertain, real-world environments.", "questions": "1. In Line 252, it is mentioned that \"the answer acquisition success rate exceeds 97%\". How is this success rate calculated?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Future prediction is a good testbed for evaluating the capabilities of LLMs and agents in information gathering, reasoning, and predictive analysis, while naturally mitigating data contamination since ground-truth answers are not available yet at prediction time.\n\n2. FutureX provides an automated and scalable pipeline for data construction, future event curation, and answer verification, offering good practical value with minimal manual effort.\n\n3. The tiered event categorization in FutureX (Basic, Wide Search, Deep Search, Super Agent) enables systematic assessment of reasoning depth and tool usage in LLM agents.\n\n4. The evaluation spans 25 models covering base, search-augmented, and tool-augmented agents, which offers comprehensive and comparative insights across different model classes.\n\n5. The paper is clearly written and well-organized, with informative figures and tables.", "weaknesses": "1. Some details on human annotation are unclear. For example, the number of questions per category, whether human experts had access to all information sources or relied solely on their own knowledge, and the consistency of their answers on the same question. Such information would clarify the human–agent performance comparison and also serve as an indicator of the robustness and quality of the questions in the benchmark.\n\n2. While the performance analysis across 25 models is comprehensive, the paper offers limited discussion on the diagnostic utility of FutureX. It is not clear whether the benchmark can help identify specific weaknesses in agents or provide actionable insights for improving their reasoning and prediction capabilities. Such discussion will further benefit the benchmark users and the development of new agents.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "luCTEWkr5N", "reviewer": "Reviewer_Kgce", "summary": "This paper introduces FutureX, a large-scale, dynamic benchmark for evaluating the future prediction capabilities of LLM agents. The authors argue that existing benchmarks fail to test complex reasoning and decision-making under real-world uncertainty.\n\nFutureX uses a fully automated pipeline that continuously gathers future-oriented questions from 195 diverse websites. It runs 25 different LLM agent models to get their predictions on a \"start date.\" After the event's \"resolution date,\" the system automatically finds the ground-truth outcome and scores the predictions. This live-updating design inherently eliminates data contamination, as the answers do not exist at the time of prediction.\n\nThe benchmark includes four difficulty tiers, ranging from simple single-choice questions (Level 1) to high-volatility, open-ended numerical and ranking tasks (Level 4). The paper's findings show that while agents with search tools (like Grok-4) perform best, all models struggle significantly with the harder tiers. Furthermore, all evaluated agents still perform substantially worse than a baseline of 40 human experts.", "questions": "What are the computational and financial costs associated with maintaining this live benchmark? Its complexity might make it difficult for other researchers to reproduce or build upon.\n\n\n\nHow does the benchmark disentangle the performance of the base LLM from the performance of the agent framework? For example, are the failures of open-source agents due to poor planning logic or the base model's inability to follow the plan?\n\nAre there plans to expand the benchmark's scope beyond the one-week prediction window to include long-term forecasting tasks, which require different reasoning skills than short-term information synthesis?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The primary strength of FutureX is its novel \"live\" evaluation design. By focusing on future events whose outcomes are not yet known, it provides a robust and scalable solution to the critical problem of data contamination that plagues static benchmarks.\n\nThe automated pipeline for question collection and answer verification is a significant technical achievement, ensuring the benchmark remains current and challenging.\n\nAnother strength is the logical stratification of tasks into four difficulty tiers. The results validate this structure, showing a clear performance drop as tasks move from simple retrieval (Level 1/2) to complex, open-ended reasoning under uncertainty (Level 3/4).\n\nI like the the inclusion of a human expert baseline. It provides crucial context, grounding the model scores and highlighting the significant gap that still exists between current AI agents and human-level analytical reasoning.", "weaknesses": "As far as I understand, the benchmark's \"prediction window.\" is only \"one-week\". This restricts the evaluation to short-term predictions, not long-term forecasting. It fails to test an agent's ability to reason about events months or years in the future, which is a different and critical skill for human analysts. I think this heavily limited this benchmark.\n\nThe evaluation metrics, while appropriate for correctness, do not capture the probabilistic nature of forecasting. The benchmark does not assess an agent's ability to express calibrated confidence (e.g., providing a probability or a confidence interval). It only scores the accuracy of a single-point answer unlike some recent works that assess the confidence too.\n\nThe comparison to human experts is a bit ambiguous. It is not specified whether the 40 human experts and the 25 LLM agents had access to the exact same information retrieval tools (I think this is very important as the author pointed out about potential leakage of data to models; such leakage could happen to human experts too), making the performance gap difficult to interpret fairly.\n\nthe benchmark tests the entire agent system at once. This makes it difficult to isolate the point of failure. When an agent performs poorly, it's unclear whether the fault lies with the underlying LLM's reasoning or with the agent's planning and tool-use framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rJivP363gu", "reviewer": "Reviewer_S6CR", "summary": "The paper introduces FutureX, a large-scale live benchmark designed to evaluate LLM agents on future-prediction tasks that require reasoning, information gathering, and decision-making. Unlike prior static benchmarks, FutureX uses a semi-automated pipeline that continuously crawls, curates, and resolves questions from 195 websites across 11 domains. The system automatically collects future-oriented questions, gathers LLM predictions at event start dates, and scores them after resolution, enabling a contamination-free, real-time evaluation. The benchmark assesses 25 models, from base LLMs to research agents (e.g., Grok-4, Gemini Deep Research), with human-expert comparisons.", "questions": "- What are the daily compute and maintenance costs for running FutureX, given the live crawling and 25-model evaluation pipeline?\n- Could future versions compare LLM predictions with crowd forecasts (e.g. prediction-market aggregates) to test alignment with collective prediction?\n- Will the full list of websites, question templates, and answer-extraction code be released for transparency and reproducibility?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "- Contamination-free design: The design of the closed environment to prevent information contamination is well-executed and addresses a critical challenge in LLM evaluation.\n- Comprehensive model evaluation: The study includes 25 models spanning reasoning, search, and deep-research agents with human expert baseline, providing a comprehensive evaluation landscape.", "weaknesses": "- The pipeline's automated expansion might favor scale over fidelity. No quantitative validation is provided to ensure the \"extra\" questions truly add value.\n-  The paper lists 195 websites but will be good to include the full list of the selected websitesl. More details on domain balance, filtering reliability would help.\n- The authors may consider including Brier score as most prior works in forecasting adopt this metric, which would facilitate comparison with existing literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)