FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
Authors: name: Zhiyuan Zeng, name: Jiashuo Liu, name: Siyuan Chen, name: Tianci He, name: Yali Liao, name: Jinpeng Wang, name: Zaiyuan Wang, name: Yang Yang, name: Lingyue Yin, name: Mingren Yin, name: Zhenwei Zhu, name: Tianle Cai, name: Zehui Chen, name: Jiecao Chen, name: Yantao Du, name: Xiang Gao, name: Jiacheng Guo, name: Liang Hu, name: Jianpeng Jiao, name: Xiangsheng Li, name: Jingkai Liu, name: Shuang Ni, name: Zhoufutu Wen, name: Ge Zhang, name: Kaiyuan Zhang, name: Xin Zhou, name: Jose Blanchet, name: Xipeng Qiu, name: Mengdi Wang, name: Wenhao Huang
Organization: ByteDance Seed

Future prediction is a complex task for LLM agents, requiring a high level of
analytical thinking, information gathering, contextual understanding, and
decision-making under uncertainty. Agents must not only gather and interpret
vast amounts of dynamic information but also integrate diverse data sources,
weigh uncertainties, and adapt predictions based on emerging trends, just as
human experts do in fields like politics, economics, and finance. Despite its
importance, no large-scale benchmark exists for evaluating agents on future
prediction, largely due to challenges in handling real-time updates and
retrieving timely, accurate answers. To address this, we introduce
FutureX, a dynamic and live evaluation benchmark specifically
designed for LLM agents performing future prediction tasks. FutureX is the
largest and most diverse live benchmark for future prediction, supporting
real-time daily updates and eliminating data contamination through an automated
pipeline for question gathering and answer collection. We evaluate 25 LLM/agent
models, including those with reasoning, search capabilities, and integration of
external tools such as the open-source Deep Research Agent and closed-source
Deep Research models. This comprehensive evaluation assesses agents' adaptive
reasoning and performance in dynamic environments. Additionally, we provide
in-depth analyses of agents' failure modes and performance pitfalls in
future-oriented tasks, including the vulnerability to fake web pages and the
temporal validity. Our goal is to establish a dynamic, contamination-free
evaluation standard that drives the development of LLM agents capable of
performing at the level of professional human analysts in complex reasoning and
predictive thinking.