UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon
  Scenarios
Authors: name: Haotian Luo; name: Huaisong Zhang; name: Xuelin Zhang; name: Haoyu Wang; name: Zeyu Qin; name: Wenjie Lu; name: Guozheng Ma; name: Haiying He; name: Yingsha Xie; name: Qiyang Zhou; name: Zixuan Hu; name: Hongze Mi; name: Yibo Wang; name: Naiqiang Tan; name: Hong Chen; name: Yi R. Fung; name: Chun Yuan; name: Li Shen
Published: 2025-09-26 02:04:00

Abstract: Autonomous agents have recently achieved remarkable progress across diverse
domains, yet most evaluations focus on short-horizon, fully observable tasks.
In contrast, many critical real-world tasks, such as large-scale software
development, commercial investment, and scientific discovery, unfold in
long-horizon and partially observable scenarios where success hinges on
sustained reasoning, planning, memory management, and tool use. Existing
benchmarks rarely capture these long-horizon challenges, leaving a gap in
systematic evaluation. To bridge this gap, we introduce UltraHorizon a
novel benchmark that measures the foundational capabilities essential for
complex real-world challenges. We use exploration as a unifying task across
three distinct environments to validate these core competencies. Agents are
designed in long-horizon discovery tasks where they must iteratively uncover
hidden rules through sustained reasoning, planning, memory and tools
management, and interaction with environments. Under the heaviest scale
setting, trajectories average 200k+ tokens and 400+ tool
calls, whereas in standard configurations they still exceed 35k tokens
and involve more than 60 tool calls on average. Our extensive
experiments reveal that LLM-agents consistently underperform in these settings,
whereas human participants achieve higher scores, underscoring a persistent gap
in agents' long-horizon abilities. We also observe that simple scaling fails in
our task. To better illustrate the failure of agents, we conduct an in-depth
analysis of collected trajectories. We identify eight types of errors and
attribute them to two primary causes: in-context locking and functional
fundamental capability gaps.
https://github.com/StarDewXXX/UltraHorizon{Our code will be available
here.}