The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic,
  and Long-Horizon Task Execution
Authors: name: Junlong Li; name: Wenshuo Zhao; name: Jian Zhao; name: Weihao Zeng; name: Haoze Wu; name: Xiaochen Wang; name: Rui Ge; name: Yuxuan Cao; name: Yuzhen Huang; name: Wei Liu; name: Junteng Liu; name: Zhaochen Su; name: Yiyang Guo; name: Fan Zhou; name: Lueyang Zhang; name: Juan Michelini; name: Xingyao Wang; name: Xiang Yue; name: Shuyan Zhou; name: Graham Neubig; name: Junxian He
Published: 2025-10-29 17:32:49

Abstract: Real-world language agents must handle complex, multi-step workflows across
diverse Apps. For instance, an agent may manage emails by coordinating with
calendars and file systems, or monitor a production database to detect
anomalies and generate reports following an operating manual. However, existing
language agent benchmarks often focus on narrow domains or simplified tasks
that lack the diversity, realism, and long-horizon complexity required to
evaluate agents' real-world performance. To address this gap, we introduce the
Tool Decathlon (dubbed as Toolathlon), a benchmark for language agents offering
diverse Apps and tools, realistic environment setup, and reliable
execution-based evaluation. Toolathlon spans 32 software applications and 604
tools, ranging from everyday platforms such as Google Calendar and Notion to
professional ones like WooCommerce, Kubernetes, and BigQuery. Most of the tools
are based on a high-quality set of Model Context Protocol (MCP) servers that we
may have revised or implemented ourselves. Unlike prior works, which primarily
ensure functional realism but offer limited environment state diversity, we
provide realistic initial environment states from real software, such as Canvas
courses with dozens of students or real financial spreadsheets. This benchmark
includes 108 manually sourced or crafted tasks in total, requiring interacting
with multiple Apps over around 20 turns on average to complete. Each task is
strictly verifiable through dedicated evaluation scripts. Comprehensive
evaluation of SOTA models highlights their significant shortcomings: the
best-performing model, Claude-4.5-Sonnet, achieves only a 38.6% success rate
with 20.2 tool calling turns on average, while the top open-weights model
DeepSeek-V3.2-Exp reaches 20.1%. We expect Toolathlon to drive the development
of more capable language agents for real-world, long-horizon task execution.