UltraHorizon: Benchmarking Agent Capabilities in Ultra Long-Horizon
  Scenarios
Authors: name: Haotian Luo, name: Huaisong Zhang, name: Xuelin Zhang, name: Haoyu Wang, name: Zeyu Qin, name: Wenjie Lu, name: Guozheng Ma, name: Haiying He, name: Yingsha Xie, name: Qiyang Zhou, name: Zixuan Hu, name: Hongze Mi, name: Yibo Wang, name: Naiqiang Tan, name: Hong Chen, name: Yi R. Fung, name: Chun Yuan, name: Li Shen

Autonomous agents have recently achieved remarkable progress across diverse
domains, yet most evaluations focus on short-horizon, fully observable tasks.
In contrast, many critical real-world tasks, such as large-scale software
development, commercial investment, and scientific discovery, unfold in
long-horizon and partially observable scenarios where success hinges on
sustained reasoning, planning, memory management, and tool use. Existing
benchmarks rarely capture these long-horizon challenges, leaving a gap in
systematic evaluation. To bridge this gap, we introduce UltraHorizon a
novel benchmark that measures the foundational capabilities essential for
complex real-world challenges. We use exploration as a unifying task across
three distinct environments to validate these core competencies. Agents are
designed in long-horizon discovery tasks where they must iteratively uncover
hidden rules through sustained reasoning, planning, memory and tools
management, and interaction with environments. Under the heaviest scale
setting, trajectories average 200k+ tokens and 400+ tool
calls, whereas in standard configurations they still exceed 35k tokens
and involve more than 60 tool calls on average. Our extensive
experiments reveal that LLM-agents consistently underperform in these settings,
whereas human participants achieve higher scores, underscoring a persistent gap
in agents' long-horizon abilities. We also observe that simple scaling fails in
our task. To better illustrate the failure of agents, we conduct an in-depth
analysis of collected trajectories. We identify eight types of errors and
attribute them to two primary causes: in-context locking and functional
fundamental capability gaps.
https://github.com/StarDewXXX/UltraHorizon{Our code will be available
here.}