The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
Authors: Junlong Li
Wenshuo Zhao
Jian Zhao
Weihao Zeng
Haoze Wu
Xiaochen Wang
Rui Ge
Yuxuan Cao
Yuzhen Huang
Wei Liu
Junteng Liu
Zhaochen Su
Yiyang Guo
Fan Zhou
Lueyang Zhang
Juan Michelini
Xingyao Wang
Xiang Yue
Shuyan Zhou
Graham Neubig
Junxian He
Venue: ICLR 2026 Poster (2026)
Keywords: language agents
tool use
benchmark
Primary area: datasets and benchmarks

Abstract: Real-world language agents must handle complex, multi-step workflows across diverse applications. For instance, an agent may manage emails by coordinating with calendars and file systems, or monitor a production database like BigQuery to detect anomalies and generate reports following a standard operating manual. However, existing language agent benchmarks often focus on narrow domains or simplified tasks that lack the diversity, realism, and long-horizon complexity required to evaluate agents' real-world performance. To address this gap, we introduce the Tool Decathlon (dubbed as Toolathlon), a benchmark for language agents offering diverse applications and tools, realistic environment setup, and reliable execution-based evaluation. Toolathlon spans 32 software applications and 604 tools, ranging from everyday platforms such as Google Calendar and Notion to professional applications like WooCommerce, Kubernetes, and BigQuery. Most of the tools are based on a high-quality set of Model Context Protocol (MCP) servers that we may have revised or implemented ourselves. Unlike prior works, which primarily ensure functional realism but offer limited environment state diversity, we provide realistic initial environment states from real software, such as Canvas courses with dozens of students or real-world financial spreadsheets. The Toolathlon benchmark includes 108 manually sourced or crafted tasks in total, requiring interacting with multiple applications over around 20 turns on average to complete. Each task is strictly verifiable through dedicated evaluation scripts. Comprehensive evaluation of state-of-the-art models highlights their significant shortcomings in performing real-world, long-horizon tasks: the best-performing model, Claude-4.5-Sonnet, achieves only a 38.6% success rate with 20.2 tool calling turns on average, while the top open-weights model DeepSeek-V3.2-Exp reaches 20.1%. We expect Toolathlon to drive the development of more capable language agents for real-world, long-horizon task execution.

Published as a conference paper at ICLR 2026
THE TOOL DECATHLON: BENCHMARKING LANGUAGE AGENTS FOR DIVERSE, REALISTIC, AND LONG- HORIZON TASK EXECUTION
Junlong Li""
Wenshuo Zhao
Jian Zhao
Weihao Zeng"*
Haoze Wu" Wei Liu' Juan Michelini
Xiaochen Wang!
Rui Ge
Yuxuan Cao
Yuzhen Huang
Junteng Liu!
Zhaochen Su'
Yiyang Guo!
Fan Zhou'
Lueyang Zhang'
Xingyao Wang*
Xiang Yue
Shuyan Zhou
Graham Neubig
Junxian Hel!
'The Hong Kong University of Science and Technology
All Hands AI *Carnegie Mellon University *Duke University
Website: toolathlon. xyz
ABSTRACT
Real-world language agents must handle complex, multi-step workflows across diverse applications. For instance, an agent may manage emails by coordinating with calendars and file systems, or monitor a production database like BigQuery detect anomalies and generate reports following standard operating manual. However, existing language agent benchmarks often focus on narrow domains simplified tasks that lack the diversity, realism, and long-horizon complexity required to evaluate agents' real-world performance. To address this gap, we introduce the Tool Decathlon (dubbed as TOOLATHLON), a benchmark for language agents offering diverse applications and tools, realistic environment setup, and evaluation. TOOLATHILON spans software applications and 604 tools, ranging from everyday platforms such as Google Calendar and Notion to professional applications like WooCommerce, Kubernetes, and BigQuery. Most of the tools are based high-quality set of Model Context Protocol (MCP) servers that we may have revised or implemented ourselves. Unlike prior works, which primarily ensure functional realism but offer limited environment state diversity, we provide realistic initial environment states from real software, such as Canvas courses with dozens of students real-world financial spreadsheets. The TOoLATHlON benchmark includes 108 manually sourced or crafted tasks in total, requiring interacting with multiple applications over around 20 turns on average to complete. Each task is strictly verifiable through dedicated evaluation scripts. Comprehensive evaluation of ttate-of-the-art models highlights their significant shortcomings in performing real-world, Long-horizon tasks: the best-performing model, Claude-4.5 4.-dde--55-Sonet, achieves only a 38.6% success rate with 20.2 tool calling turns average, while the top open-weights model DeepSeek-Va -3. V reaches 20.1% %. We expect TOOLATHILON to drive the development of more capable language agents for real-world, long-houd oonghhoiioon task execution.
reliable execution-base
INTRODUCTION
Tool-based language agents have already demonstrated their impact in real-world domains such as software engineering (Jimenez et al.) 2024: The Terminal- -ench Team, 2025). deep research (OpenAI 2024), and web browsing (Zhou al.] (024). To further expand the reach of language agents across diverse domains and applications, the Model Context Protocol (MCP) has been proposed to establish a standard for connecting language agents to tens of thousands of applications (Anthropic) 2024).
Existing benchmarks for language agents, however, are restricted to limited domains and tools MMialon
ct al. 2023 Liu al. 2024 Ma et al. 2024, Jimenez et 2024 Zhou et al. 2024 Yao et al. 2025
Equal Contribution. CCorresponding author.

---

Published as a conference paper at ICLR 2026
Task Prompt #I: Your task to check your email for homework2 subrisssooss and grade on Canvas. Please download Python files from email attachments to local workspace and execute each Python file in terminal to check for errors. if the Pythan file correct, give it score of 10 in Camvas; ocherwise, give homework2 in seiigments/homewworre md'mmdd a'and students' "student carvas_ ids.
score could check requirements students who zubmittted multiple times, use the latest submission.
assigtments
mn
Aef iedatrrm teroe))
tomewor
s..dent.ikt es
exlation
Jeacher_Jst.cow
Email (Poste.io)
Filesystem
Terminal
Lanvas-LMSS
Task Prompt #2: Idenaffy the tickes in the dacabase that have exceeded the inicial response time according to the relevant documentation, and send manual, the respective responsible mansgers, well apology involved
reminder erails, based tumplates
The
face
Snowflake
Filesystem
PDF
Email (Poste.io)
Figure 1: Two examples and the initial environment states in TOOLATHLON. We showcase real-world environ- and realiste state initialization ($2.3) here.
ment interaction ( 2.2
Wei et al. 2025 Xu et al. 2025). By contrast, real-world tasks often require switching across various applications. For example, as demonstrated in Figure (Example #2), a company's 's administrative agent may need to monitor real Snowflake database for customer tickets, locate the appropriate PDF operation manual containing instructions on how to identify and handle overdue tickets, and then send the required emails to managers and customers accordance with the manual. Importantly, this diversity gap extends far beyond differences in tool names or descriptions. The diversity and complexity of environment states across applications, compounded by interaction with them in long trajectories, present substantial challenges for generalization.
To address these challenges, we introduce the Tool Decathlon (TOOLATHLON), benchmark for evaluating language agents on diverse, realistic, and long-horizon tasks. TOOL.ATHION spans 32 real-world applications and 604 tools across 108 tasks, covering a wide spectrum of domains ranging from daily affair and education to technology and finance. Tasks are grounded in realistic scenarios and mostly require coordinating multiple applications. Each task is fully verifiable with a dedicated, deterministic evaluation script, comparing outcomes against either static or dynamically generated ground-truth states (e.g., tasks involving the latest NVIDIA shareholder information or real-time train schedules). All tools in TOOLATHLON sourced from the real world, with the majority obtained from MCP servers.
To faithfully capture the realism and complexity of practical environment states, we tried to adopt the most representative applications such as Google Sheet, Gmail, and Snowflake. However, some remote environment states are difficult to set up to mimic real scenarios. For example, simulating a Canvas course with tens of students would require registering a real account for each student and resetting the states each evaluation run. Therefore, while adopt the commonly used applications most of the time, we also incorporate several open-source software deployed locally via containers for convenient and complex environment simulation, such as poste.io for email management to replace Gmail and WooCommerce for online ecommerce platform to replace Shopify. These services provide complex observations while allowing us to set up the states in a scalable way. This stands in stark contrast with simplified or artificial environment states as in prior benchmarks (Patil et al.) 2025). In addition, task prompts in TOOLATHLON are crafted to mirror authentic user queries, which are often concise and fuzzy. Models must therefore infer user intent and autonomously devise plans to accomplish tasks, an example is shown in Figure S
Coneurrent with this work, several MCP-based tool-use benchmarks have emerged (Luu et al. 2025
Mo et al., 2025) Yan et al., 2025 Yin et al., 2025) The MCPMark Team, (025), but they do not match TOOLATHI.ON in its reflection of real-world complexity. Some rely on LLM judges without
2

---

Published as a conference paper at ICLR 2026
Table Comparison of Tool-Based Language Agent Benchmarks. where BFCLv3-MT represents the multi-tum subset released in the 3rd version of BFCL. "# Apps" denotes the number of MCP servers, which we not annotate for benchmarks without clear application definition. "Avg # Turns" denotes the number of tool calling turns made by Claude-4-Soneet, which we use as proxy for task complexity. "Real Eny" ($2.) means the environment states and observations are from real-world software rather than artificial databases. "States Init" (2..) indicates the evaluation begins with state initialization. "Verifiable Execution" ($2.4) denotes that models need to execute tools and final results are evaluated based on stales. "Realistic Fuzzy Prompt" represents that the task instructions are often fuzzy and ambiguous to mimic real user input ($3.1). 'For MCPUniverse, only 10% of the tasks are cross-App. For ACEBench and LiveMCPBench, only < <10% of the tasks contain simple
states initialization. dn-depth discussion of these related works is in $6
Avg # Turns
Real Env
States Init
Verifiable
Cross-App Task
Realistic Fuzzy Prompt
Benchmark
Tasks
# Apps
Execution
7 7-Bench
165
BFCLv3-MT
800
Partial 3.8
ACEBench
2000
1.7
X
AppWorld
750
V
X
MCPWorld
201
10
X
X
MCP-RADAR
507
6
X
X
MCPEval
676
19
x
V
X
LiveMCPBench
95
70
5.6
Partial*
V
X
MCP-AgenBBench
600
33
V
X
LiveMCP-101
101
41
5.4
X
MCPAilas
1000
40+
3-6
V
X
MCPUniverse
231
11
7.5
Partial
MCPMark
127
18.5
X
GAIA2
800
12
22.5
TOOLATHLON
108
32
26.8
verifiable tasks (Mo et al. 2025 Yin al. 2025). while others cover few domains or mostly single- application tasks. For instance, MCPUniverse (Luu et 2025) spans only six domains, with 90% of tasks involving one app and synthetic initial states, yielding simplified, short interactions (<8 turns). Similarly, MCPMark (The MCPMark Team 2025) includes only five apps and overly detailed prompts (Figure 3p. GAIA2 (Andrews et al. (2025) covers merely the mobile domain on mostly daily tasks with simplified synthetic environments. A full comparison is shown in Table T
TOOLATHLON includes a lightweight framework for automated, safe, and scalable evaluation. Each task comes with initial states setup if needed as well an evaluation script (Figure 2 Executing and evaluating each task is isolated in separate containers to prevent interference. This enables fast parallel evaluation- for example, rumning Mlaude-455-oonme on all 108 tasks takes only 70 minutes using 10 parallel processes. With extensive experiments on TOOLATHLON, the best-performing models, Claude-4.5-Sonee achieve only 38.6% accuracy, highlighting the unique challenges posed (DeepSeek-Aa Al (2025) achieves 20.1 10.1% % success rate as the best performer among open-source models. Further analysis reveals that weaknesses in long-context modeling and robust tool calling error tracking are major challenges for all evaluated models. We have fully open-sourced the benchmark and the TOoLATHLON environment, aiming for TOOLATHLON to accelerate the development of practical language agents.
by TOOLATHLON. DeepSeck-
THE TOOLATHLOO ENVIRONMENT AND EVALUATION FRAMEWORK
2.1 TASK DEFINITION
Each task in TOOLATHLON can be formulated as a partially observable Markov decision process (POMDP) (S, A, NOT, R,2) with state space S, action space A, observation space O transition function T S x environment states 2.2, 2..3) can be the status in the current email inbox and the observations are the sequential input to the model. The action space A is the available tools for the respective task and the tool implementation directly defines the transition function. The reward function R. ($2.4) represents our execution- -aaedd evaluation which directly evaluates the environment state. Intuitively, real-world tools and environments will yield significantly more complex and diverse environment states and observations than the synthetic ones, and in the following sections, we will detail our designs of these variables in ATH.O..
x O, reward function R : S - [0, 1), and instruction space 44 The
3

---

Published as a conference paper at ICLR 2026
2.2 TOOLS, ENVIRONMENTS, AND FRAMEWORK
MCP Servers: In TOOLATHLON, we source our tools through variety of MCP servers. Specifi- cally, we first decide list of valuable and common real applications that we aim benchmark on, then we see if we can find the corresponding open-souree MCP servers for them. If not, we implement the MCP servers by ourselves. Notably, many open-source MCP server implementations contain bugs or exhibit certain limitations, for example, without the tools needed to complete our tasks. We further refine and improve these implementations ourselves. This way, we obtain a high-quality set of 32 MCP servers in total, where we include complete list and their sources in Appendix [A] The applications span diverse domains, extending well beyond common daily-use applications such as Google Maps, Notion, and Google Calendar, and we also incorporate number of professional and domain-specific applications to evaluate language agents in high-value productivity scenarios, such as Snowflake for enterprise data management and Kubernetes for cluster management. Although the majority of tools are sourced from MCP servers, the benchmark usage itself is not tied to MCP employment from the model developer side. For examples, these tasks can also been solved via pure GUI or CLI workflow, as long as certain account information like usernames, passwords, tokens or credentials are explicitly given to the agents.
While tools provide interface for inter- acting with environments, they do not directly constitute the environments. Many real-world tools interact directly with existing, remote environments, such Google Sheets, Google Calendar, No- tion, and Gmail. Although remote environments require no implementation effort, they introduce significant challenges when benchmarking tasks that involve modifying environment states. For instance, simulating realistic Gmail inbox with hundreds of emails from diverse senders would require registering hundreds of Google accounts for every benchmark user, and this inbox would need to be reset prior to each evaluation run. Previous works have attempted to bypass this issue by only supporting read operation to the states (Mialon et al., (023), or implementing simplified synthetic data structures to mimic environment states (Patil et al. 2025 Yao et al. 2025), but such approaches drastically reduce realism and fail to reflect the complexity of real software environments. In contrast, in TOOLATHLOON we leverage both remote environments and locally containerized, open-source applications. Specifically, we deploy the open-source Poste.io for email management, Canvas for course administration, Kubernetes for cluster orchestration, and WooCommerce for e-commerce management. By hosting these realistic applications locally within containers, we can efficiently set up dozens of accounts and initialize complex environment states during evaluation. Compared with existing dedicated agent sandboxes such as SWE-Bench (Jimenez et al. 2024), our environments are more diverse and encompass a wider range of software.
Remote and Locally Containerized Environments:
Agent Framework: We implement simple agent framework based on the OpenAl Agents SDK (v0.0, 0. 15) to conduct the agent action loop- - at each turn, the model is expected to (optionally) reason explicitly and make tool calls. We make several enhancements to improve its basic setup for more robust workaround to evaluate language agents, including tool error handling, overlong tool response handling and context history management. We also equip this framework with some basic yet common local tools like python execution, web search, claim done and sleep. The details can be found in Appendix B
2.3 INITIAL STATE SETUP
In real world, tasks are rarely executed from empty environment state (c.g., an empty inbox). Instead, agents are typically required to operate based on pre-existing environment states. In agentic scenarios, task diffieulty is determined not only by the task instructions but also by the underlying environment states. For example, operating on a folder with only one file to be used is easier than working with ten mixed useful and unrelated files (Figure example #2), even if the task descriptions are nearly identical. To capture this, for tasks in TOOLATHLON that starts with an initial state ? each of these tasks is equipped with a state initialization script to set up the states at running time. or (and) an ini ttal. directory containing pet files. Figure [ and Figure 3 showease such initial states. When constructing these initial environment states, we design them to closely
hhttp:///ithubb.co httpps//gghuuu..oggggaa/ag/rg//e htwp://nnnaiooonaaaggttttsasoeee
As shown in Table 67% of the tasks fall into this category.
4

---

Published as a conference paper at ICLR 2026
Realistic State
Initialization
Tool Outputs
Real- World Software
Task Instruction
Language Agents
MCP & Local Tools
31
State-based Evaluation
Tool Call
Execute
N
WOO
Figure 2:
Overview of the TOOLATHLON evaluation framework.
reflect realistic scenarios. Notably, only very few previous benchmarks have incorporated realistic initial state construction before entering the agent loop, as summarized in Table By contrast, most existing benchmarks start from empty state or overly simplified environment states, thus failing to capture the full complexity of real-world task execution.
2.4 RELIABLE EXECUTION --ASED EVALUATION
First, unlike some traditional tool-calling benchmarks that measure single-step tool call accuracy given fixed context without actual execution (Patil et al., 2024). think that execution-basse evaluation is essential for reliably assessing language agents in realistic scenarios. Second, while
many existing benchmarks rely on LLMs judges to score agent trajectories (Gao et al. 2025 Yin
et al. 2025). we contend that verifying the final environment states using deterministic rules offers a far more reliable and reproducible evaluation framework, demonstrated in several widely adopted agent benchmarks (Zhou et al., 2024 Xie et al. [024 Jimenez et al. (2024). To achieve this, each task in TOOLATIILON is equipped with unique, manually crafted evaluation script that ensures precise and consistent measurement of task success. The script may perform robust matching against a static snapshot of the ground-truth enviromment or follow reference execution workflow dynamically retrieve and match real-time information (e.g.. NVIDIA shareholders). During evaluation, each task associated with configuration file that specifies only the necessary MCP servers (< 10) and tools available for use rather than all 32 MCP servers. For each selected MCP server, we do not further filter or retrieve a subset of tools but load all tools within it, which is consistent with the current state of mainstream agent frameworks. Intuitively, providing the model with a larger set of unrelated tools increases task difficulty, as the agent must identify the relevant tools while ignoring distracting ones.
Safe and Efficient Parallel Evaluation in Containers: Our TOOLATHLON evaluation framework supports parallel execution to enable efficient model evaluation. Our framework launches each task inside a separate container in parallel, providing strict workspace isolation. On a standard Ubuntu 24.04 Linux cluster with 16 CPUs and 64 GB of memory, we are able to evaluate Claude-4.5-Sonnet on 108 tasks in just about 70 minutes of wall time using only 10 parallel processes. This demonstrates that TOOLATHL is both convenient and efficient for practical use by model developers to get instant feedback on how their models perform in realistic scenarios and requirements.
3 THE TOOLATHLON TASKS
3.1 TASK SOURCING AND FUZZY TASK INSTRUCTION
The authors of this work, who are researchers and senior undergraduate students in computer science, source and implement the tasks. We carefully design and adhere several principles when collecting tasks: (1) Real User Demands: All tasks are either directly soureed from real-world websites or crafted to reflect genuine user demands. (2) Multi-App Orchestration: We intentionally source tasks that require interaction with multiple MCP servers, as this reflects authentic human workflows and increases task complexity. (3) Diversity: To ensure broad task diversity, adopt two-stage sourcing process. In the first stage. we start with an initial MCP server list covering more than 50 applications and freely source tasks without restricting to specific servers. In the second stage, analyze the distribution of the sourced tasks and identify Apps that are important but underrepresented. We then conduct additional round of targeted task sourcing specifically for them.
5

---

Published as a conference paper at ICLR 2026
Please update the candidate information on HR Record sappage Notion according the redumes my wor lapace infor mation must be Med out iull a sybrding to the concert in the rosumes, without making any usaythorized modiication of stdkggheeoopg any works Alio, please celete the nitting cimple seriss die recrrd the position applicd for the applicant pierently open for recastment, please seed
Cresse comprehensie weekerd advenmure planer thet aralane Toronto Guide
stnurtured stoerry page. need "Prrfett Wewlend Adveruur'' at3 child the Toronno Guide
desitsue
page cilled
Task Re Requiremente:
ublc
saled WerkersWasknnd Adventuri a child page of the main Toronco Guide page
Crease
ggrreppoding sppicaut using the
2. Query the Activiree es donoore to idee fy all activitie than have the "Beochee" 1ag
tolnwinng pamblre informotin f d liwe brooks Do nor seed my frxst by mamber (A
3. Quuery the Food duclose to find all respurante with "Turkas" or "rosko" rags
TEMPLATE HERE]
Inter fromy of rfadd Count (tter from yresumes Find info by agent
4. Querr the Cofee deniosse in retr rerrawe all cales eneries
Clear by Step Guides
May Sean existing oxamples
Serueture the page with the isl olowing speciti forman
(SOME SPECIFIC REQUIRMMENTT]
6. After the summary parag aph, add diiccr blods
HR Record
firally, add callurt block with the amoj commaising the cear "Pro ep. Check
Seceons rbratose the beir time exjoy nundoor actimtte"
Ersure headings use exact aniji
speeifed above
in osxc for specifed (bulktted for beaches nambered
resouraroe cafe)
N
Cordides
Teronto Gulde
Figure 3: Example task instructions from our benchmark (Ieef) and MCPMark (The MCPMark Team (Right). Ours contain more fuzzy intent that the model need to infer from the environment states.
202.5
Realistic Fuzzy Task Instruction: We de- sign task instructions to resemble authentic user input, which is often fuzzy or ambigu- ous, but whose actual intent can be deter- ministically inferred from the environment's states (e.g., existing data examples docu- ment templates). This requires the agent to infer the user's intent from the environment state, formulate plans, execute them, and in- tellectually handle unexpected events such as tool call errors. For example, as shown in Fig- ure Left, real user may simply say "Please update the candidate information on HR Record subpage according to all the resumes..... if the position applied for by the applicant is currently not open,
Manuscngt Editigg Research
DTC Managemment
pnweerory Control
Schelarly Activitles
Product Aete peotmment
Onliiie Purchase e
Sports Analynt yis Persenal Arfair Pansportation
Maragemen
Profiin
commmnn Pooue (20.1%)
Coursie
(23.99))
Socia Media
Daiy (25 79)
Travel Helper
School Application
Campue
f2.6.
Health Guidance
Academic Affairs
Teaching
Finance
(9. (9.3%))
Busin (16. 7%)
Rnancial Admin
Quansitative Srategy
Data Auditing
Tech (1.
stock Trading
Tatent Mana Management gemment
Harketing
Data Wrangingg
Analytics
Deployment
DevOps Sweap
peaeee sso
Expevimentt Analysis Anaaysis Project sync
Data Curettio
This is fuzzy instruction without specifying in which format the agent should fill in the information, but the Notion database has provided some examples that the agent needs to know to check itself. Also, the instruction does not mention where to find the status of the posted job and the agent needs to check Notion to find that by itself. In contrast, task instructions in some existing benchmarks (Figure Right) explicitly include detailed step-by-step plans, which reduce the role agents for planning. More examples of this kind are shown in Figure 10 and
Figure 4: Task topic distribution of TOOLATHLON.
All the sourced tasks experience multiple rounds of rigorous quality check, filtering and refinement which last for several weeks before we implement them into our benchmark, and finally we obtain 108 tasks in total. The topic distribution of all tasks is shown in Figure 4 and Table shsoow some key statistics of the complete benchmark.
3.2 TASK IMPLEMENTATION
As described in 2.4 each task in our bench- mark is fully implemented with a correspond- ing evaluation script and potential initial states setup. This process involves collecting ground- truth states statically or dynamically, and design scripts to automatically clear and re-fill new ini- tial states. To ensure realistic setups and reli- able evaluation, implementing a single task in
Table 2: Key statistics of TOOLATHLON.
Statistics
Value
# MCP servers (# tools) # Local toolkits (# tools)
32 (604)
7 (16)
Avg/Min/Max tools per task Tasks with state initialization
69.9/28/128
72/108 (67%)
6

---

Published as a conference paper at ICLR 2026
Table 3: Main results for all the models. P@ P.33, P"3 and # Turns represents Pass Pass@ Pass and average numbers of turns, respectively. make bold the highest score.
Model
Research Campus Finance Tech Business Daily E-com]
P@
P@3
P3 Turns
Proprietary Models
Claude-4.5 -.5-Snnnt
31.1
42.6
33.3
42.1
42.6
35.3
39.4
38.6
51.9
20.4
20.2
GPT-5
20.0
33.3
13.3
40.4
38.9
39.2
12.1
30.611.5
43.5
16.7
18.7
Claude-4-Sonnet
33.3
33.3
30.0
26.3
25.9
33.3
27.3
29.9
41.7
17.6
27.3
GPT-5-high
15.6
31.5
23.3
29.8
44.4
33.3
15.2
29.0+3.1
16.7
19.0
Grok-4
24.4
22.2
13.3
43.9
24.1
27.5
30.3
27.5+1.7
38.9
16.7
20.3
Claude-4.5-HB
11.1
22.2
26.7
29.8
27.8
37.3
27.3
26.2+1.9
39.8
13.0
21.9
Grok-code-Fate
20.0
14,8
16.7
19.3
14.8
21.6
24,2
18,5+2.0
30.6
9.3
20.2
Grok-4-Fast
15.6
22.2
16.7
24.6
14.8
13.7
21.2
18.512.0
32.4
5.6
15.9
03
15.6
14.8
10.0
22.8
13.0
29.4
6.1
17.0+0.0
25.0
9.3
19.4
04-mini
13.3
11.1
20.0
17.5
11.1
21.6
9.1
14.8
26.9
3.7
16.6
GPT-5-mini
11.1
16.7
20.0
15.8
9.3
21.6
6.1
14.5+1.2
23.1
5.6
19.7
Gemini-2.5-Pro
4.4
3.7
3.3
15.8
5.6
27.5
9.1
10.5+1.9
21.3
2.8
26.5
Gemini-2.55Flash
4.4
3.7
6.7
3.5
1.9
3.9
3.0
3.721.5
8.3
0.0
8,3
Open-Source Models
DeepSeek-v3.. vo.ckkv3..--Ep-
11.1
16.7
23.3
19.3
14.8
29.4
30.3
20.1
27.8
12.0
26.0
GLM-4.6
22.2
18.5
20.0
17.5
16.7
11.8
30.3
18.81122
29.6
9.3
27.9
Qwen-3-Coder
4.4
16.7
10.0
19.3
14.8
17.6
15.2
14.5+1.9
21.3
6.5
28.5
Kimi-k2-0905
8.9
22.2
16.7
14.0
5.6
9.8
15.2
13.012.0
22.2
5.6
26.6
TOOLATHLON requires, average, 4-6 hours of work by research graduate student majoring in computer science.
Finalizing Tasks and Quality Check: After crowd-soureing task implementations from multiple contributors, we perform intensive quality checks conducted by 5-6 experienced authors. In this stage, each task carefully reviewed and revised to unify standards across all tasks and ensure correctness, solvability, and unambiguity, which requires approximately 5 hours of labor per task per round of checking. Once all tasks are finalized, we perform an additional round comprehensive cross-checking and bug fixing of the entire benchmark before running the final experiments.
EXPERIMENT
In this section, we present the configuration details and experimental settings for several leading commercial and open models on TOOLATHLOON, as well as their performance.
4.1 SETUP
Models and Configuration: Our evaluation includes the leading commercial model series in
terms of agentic abilities, such as GPT-5(-mini) (OpenAl 2025b), 03&o4-mini (OpenAl 2025a).
Claude-4-Sonnet (Anthropic 2025a). Claude-4.5l-Sonnet,, Haiku) (Anthropic 2023c b), Gemini 2.5
Pro,-Flash) (Comanici al. 2025). Grok-4(-Fast) (xAl 2025a b). Grok-Code-Fast (xAi 2025c
We also benchmark the best- pert-performing open-weight models including Qwen-3-Coder (Qwen Team
2025). DeepSeek-V2 V3 Exp (DeepSeek-Al 2025), Kimi-K2-0905 (Kimi Team
2225 and
GLM-4.6 (Zipp AI 2025). As described in $2.4 each task is precontigured with a list of MCP servers and common tools access. During evaluation, we the maximum allowable number of turns as 100 for all models. In our main evaluation setup, we only provide the models with the MCP servers and common tools that are useful for executing the task. We note that as each MCP server is equipped with multiple related tools, so the models will still see many unnecessary tools during evaluation.
We evaluate each model three times and report the average pass @ success rate well as the standard deviation. We also include the pass the fraction of tasks with at least one correct trajectory, and pass"3 3 (Yao al. 2025) the fraction of tasks where all three trajectories correct, to measure the model's potential capability coverage and its ability to complete tasks reliably. We also report the average number of turns used.
Metrics:
7

---

Published as a conference paper at ICLR 2026
Whong Thai Name
Tool Erlrrrrr
Pass
-30
Potrig (94)
Kimik eans
Q344.6
Gaude Saaii
GPTS
Baude 4 Sonner
Genisiz 5.Pm
GPrSSmi
Gratt Grat.Cade fesh
Oneen Gwen- Caatter
Claude S-5nnnt
Gemmiz 5 5Paah
Figure 5: Two kinds of tool calling error presence ratios calling tools for different models.
4.2
MAIN RESULTS
Results in Table 3 show that Claude-4.5-Someet ranks first, but still achieves a success rate of less than 40%. GPT-5, Claude-4-Sonne,, Grok-4, and Claude-4.5-Haikee each with Pass scores above 26% but below 30%, clearly fall into the second tier. All other models remain 20% or below. This indicates that our benchmark remains challenging for state-of-the-ee models and effectively distinguishes their capabilities. For open-source models, scores less than or equal to 20%, with the best, DeepSeek-V3. V3.2-Exp. achieving 20.1 %, revealing a clear gap compared with proprietary models. Interestingly, increased reasoning effort for thinking-orienter models (e.g., GPT-5 vs. GPT-5-high) shows no benefit, suggesting that exploring new observations matters more than extended internal reasoning in agentic tasks. We also find that Gemini-2.5's ability to understand requirements and proactively explore is insufficient-Di may neglect certain requirements or give up prematurely during execution, resulting in poor performance on complex tasks.
Looking at performance across task categories, Claude-4.5 --nnntt excels in almost all domains, especially in Campus and E-Commerce tasks, demonstrating strong general capabilities across diverse tools. GPT-5 performs exceptionally well in Daily tasks, showcasing its effectiveness in everyday scenarios, while Grok-4 stands out in Tech, indicating a specialized strength in technology-rell related development operations. We also observe significant differences between Pass @3 and 3 success rates. This indicates that while many models have certain capability coverage, they lack consistency tasks, building agents with both high success rates and
in producing reliable results. For robust consistency remains a critical challenge.
ANALYSIS
In this section, we conduct analysis in depth to better understand model performance in TOOLATIILON, focusing on tool-call errors, well as long-context and overlong-output challenges. More analysis, including how tool eтror and the involvement of unrelated MCP servers impact model performance, and qualitative analysis case studies, can be found in Appendix Cland
5.1 THE FAILURE CALLING TOOLS
We mainly focus two major tool-calling cr-
rors: hallucinating non-existing tools (e.g., in- correct tool names) and errors raised during tool execution. Statistics for different models on these two types of errors shown in Figure 5) It can be seen that all models produce tool execution errors varying degrees, possibly due to incorrect parameter passing or attempts to access non-existent resources. However, we found no significant correlation between overall success rate and the frequency of such errors. In fact, error messages from tools may help models understand the tool implementation or structure, allowing adjustments in subsequent turns. The other type of error- incorrecttreett tool names- more
Ohite- Sonnut
Surcess Rate (95)
40
45%
k11%
WS 2.
Ower dicader
Grak-4Far
57%
52%
300
28%
28%
20:
Tume
215%
21%
23%
28%
23%
tome
1.%
30
Jume
Turm
Fumm
11%
Hard
[4.2,
Figure 6: Model performance on three groups of tasks divided by average turns. The x-axis represents different task difficulty groups determined by different avg turns range [Min Turns, Max Turns] likely affects final scores. Leading models produce few tool name errors. In Appendix C Figure
8

---

Published as a conference paper at ICLR 2026
9 further analyze the success rate difference between trajectories containing tool-calling errors versus error-free trajectories, showing that most models do suffer from tool call errors.
5.2 THE LONG-CONTEXT CHALLENGES FOR LANGUAGE AGENTS
Since our benchmark built on real tools and environments, it naturally generates many long-horizon trajectories. To quan- titatively describe the differences between tasks, we calculate the average number of execution turns for each task across all models, and use this as a proxy divide all tasks into three
Success Rate With Overlong Toal Output (36)
Bue Dadk
equally sized groups: Easy, Medium, and Hard, with execu- tion turns increasing with difficulty. In Figure [6] we show the performance of five representative models on different groups,
Gamend
Calden S-dima
Me 2 Ann----oss
20-
GUM-N
along with their average turns in each group. The results indi-
Owee Scoeer
cate that groups with higher average turns generally have lower success rates across models, and leading models like Claude- 4.5-Sonnet maintain clear advantages in all groups. We also find that there is no significant difficulty difference between the Medium and Hard groups, and that even Claude-4.5-Soneet and GPT-5 achieve higher scores on the Hard subset then in Medium ones. This suggests that our benchmark's difficulty does not entirely stem from standard multi-step long- long-horizon execution, but possibly from models ending tasks prematurely without sufficiently exploring available observations, leading to failure.
100
30
40
Success Overlong Output (%6
Figure 7: Avg. Success Rate Trajec- tories w/wo overlong tool outputs.
Another concern is whether models can successfully complete tasks when encountering overlong tool outputs, like fetching lengthy HTML source code directly listing all data from database (we refer the readers to Appendix for handling overlong outputs in our framework). We calculate the proportion of trajectories containing overlong tool outputs encountered by all models during evaluation, as well as each model's 's success rates with and without overlong tool outputs. Results show that the proportion of overlong tool outputs varies from approximately 15% to 35% across different models. Additionally, Figure(Z shows that most models experience decline in success rate when encountering overlong tool outputs, with only few models maintaining nearly unchanged performance. While tasks with overlong outputs are often logically straightforward (e.g., price comparison, data extraction), most models get trapped trying to process these lengthy outputs.
5.3
THE RELATIONSHIP BETWEEN PERFORMANCE AND EXPENSES
Since evaluate the models in realistic settings, both cost and token usage important factors, as they determine how a model should be selected for different budget constraints. Therefore, during the evaluation period, measure the actual number of output tokens and the associated cost with prompt caching enabled. For costs, Figure 8 Left shows that Claude-4-Sonned and Grok-4 incur relatively high expenses, whereas most other models remain under $1 per task. Slaude-4.5-Sonme achieves the highest performance but ranks third in cost. Several models, such as Grok-4-Fas Fast, Grok-Code Code-Fast-LL 1, and DeepSeek DrepSeek--3..-EEE V3.2-Exp. incur only a small cost, suggesting that they can serve as strong alternatives under limited budgets without an extreme pursuit of maximum performance.
We also plot the output token count distribution Figure Right, which illustrates the success rate varies with different average output token counts. Most models cluster between 5K and 10K output tokens. Some reasoning-foeee focused models, such as o4-mini and GPT-5(-high), generate more tokens, whereas the Claude series and Grok-4 achieve strong results with fewer tokens, suggesting they rely more on environment observation rather than extensive internal reasoning. Models like Gemini-2.5 Fla-h have the lowest output token counts and correspondingly lower accuracy, while others exhibit a similarly concentrated distribution.
6 RELATED WORK
Benchmarks for tool-based language agents differ substantially in the realism their tools, environ- ments, and task configurations, and can be viewed along spectrum from fully simulated settings

---

Published as a conference paper at ICLR 2026
40
Cluude-43-Someet
Caude-4D-SSrrett
pass pass@11 (%6
30
GPTS 4-arne-
Claude-4 S-onnet
GPTS
GPI-Sshighh
5P13-H1lPgoo44
"Grok4
Cloude 4.5 Halku
Claude-4.-Haalle
20
Grok-Codle
-4-Fast
Grak-
DeepSeeeksS5 2-Exp
GPT.mmini
GLM-4.6
o4-mini
dedFas--1 .4.6
Grok-4-
Fast
BLM-
DeepSeek-
Xm--2-090pp CoPT-S-mini
63
o4-mini
10
Kmi-K2-09ps
25
-Coder
Comini
"cemini-2 Seflish
GGemin-225--lash
10
10a
5K
10K
15K
20K
25K 30K
average cost per task ($, log scaled)
average output tokens per task
Figure 8: The relationship between average task success rate and average cost (Left) and output tokens (Right).
to those grounded in real-world applications. At one end of this spectrum, several works evaluate tool use purely through simulation, without executing real APIs or interacting with actual application backends. Representative examples include r-Bench Yao al. 2025). BFCL. (Patil al. 2025 and ACEBench (Chen et al.) 2025). which assess function calling accuracy or multi-turn tool selection in emulation. While such designs enable efficiency and reproducibility, they omit many of the challenges that arise from executing real tools unpredictable environments.
controlled scenarios, but rely on mock implementations or language-model-
Moving beyond simulated tools, other benchmarks connect agents to real APIs yet operate in synthetic or constrained environments where initial states are artificially constructed. For exam- ple, AppWorld (Trivedi al., 2024) offers high-fidelity simulation of multiple apps, and MCP- World Yan et al., al.] (202). MCP-RADAR (Gao et al.) 2025), MCPEval (Liu et al., 2025). and MCP- AgentBench (Guo al. 2025) grant access to real Apps via Model Context Protocol (MCP) (An thropic 2024) but often begin from zero or artificially designed states or center on single-application tasks. These setups capture tool execution more faithfully than pure simulation, yet still fall short of representing the complexity of authentic, multi- culti-application workflows.
Closer to realistic settings, number of recent benchmarks combine real tools with more authentic environment conditions. LiveMCPBench (Mo et al. 2025). LiveMCP-101 (Yin et al. 2025). MCPAI- las (Scale AI, 2025), MCPUniverse (Luo et al. 2025), and MCPMark (The MCPMark Team 2025)
introduce production-gran grade MCP servers, multi-step workflows, and realistic tool outputs. Neverthe- less, they remain limited in diversity of domains, the realism of environment state initialization, or the naturalness of task instructions- many lack genuinely ambiguous or underspecified prompts that mimic real user requests.
Our work, TOOLATHI.ON, advances this trajectory by combining real tools with genuinely realistic environments across 32 applications and 604 tools, spanning a broad range of domains. Initial states grounded in authentic usage scenarios rather than synthetic constructs, and tasks often require long-horizon, cross-application orchestration. Moreover, prompts are intentionally concise and fuzzy, compelling agents infer intent and autonomously plan, while deterministie, script-based evaluation ensures correctness in evaluation.
7 CONCLUSION
We introduce TOOLATHLON, comprehensive benchmark for evaluating language agents on real- world, long-horizon tasks spanning 32 applications and 604 tools. Our evaluation reveals significant limitations in current models, with the best- best-performing Claude-4.5-Sonne achieving only 38.6 suc- cess rate, highlighting substantial room for improvement in handling complex multi-step workflows. Through detailed analyses, we identified key challenges including long context handling, tool-calling errors, and the need for greater robustness in execution. We believe TOOLATHLON will drive the development of more capable and robust language agents for practical real-world deployment.
ACKNOWLEDGMENT
We thank Pengcheng Yin for helpful discussion this project.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Pierre Andrews, Amine Benhalloum, Gerard Moreno-Torres Bertran, Mattco Bettini, Amar Budhiraja. Ricardo Silveira Cabral, Virginie Do, Romain Froger, Emilien Garreau, fean-Baptiste Gaya, Hugo Laurençon, Maxime Lecanu, Kunal Malkan, Dheeraj Mekala, Pierre Ménard, Grégoire Mialon, Ulyana Piterbarg, Mikhail Plekhanov, Mathieu Rita, Andrey Rusakov, Thomas Scialom, Vladislav Vorotilov, Mengjue Wang, and Ian Yu. ARE: Scaling up agent environments and evaluations, 2025.
URL htRLLttts://arxivoorr// attgg//b//25500 2509.17158
Anthropic. Introducing the model context protocol. https://ww.antirrpiccccc///e/e/e
model-context-prooocoll 2024.
Anthropic. Introducing claude
2025a.
Anthropic.
httos:/www.anthropiccccmm
(ewss/Llude--haiku--45 2025b.
Anthropic. Introducing claude sonnet 4.5.
https://ww.arrtroppcc.ccm////
claude-sonnet-44 2025c.
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Yuefeng Huang. Wulong Liu, Xinzhi Wang, Defu Lian, Baoqun Yin, Yasheng Wang, and Wu Liu. ACEBench: Who wins the match point tool usage?, 2025. URL https 7/arxiv. .rgg 2501 12851.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
DeepSeek-AL. Introducing deepseek-v3.2-exp. news250929 2025.
http:///api-doossdeepeeeeee news
Xuanqi Gao, Siyi Xie, Juan Zhai, Shqing Ma, and Chao Shen. MCP-RADAR: A multi-dimennional benchmark for evaluating tool use capabilities in large language models, 2025. URL https
//arkiv.org/abs/2555.111 16700
Zikang Guo, Benfeng Xu, Chiwei Zhu, Wentao Hong, Xiaorui Wang, and Zhendong Mao. Mcp- agentbench: Evaluating real-world language agent performance with mep-mediated tools, 2025.
URL htk.:/ttpss//axxv.orr///////// 09734
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. htttt:://pp//e wwt.mmidddFFFyyyNOM
net/
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534. 2025.
Xiao Liu, Hao Yu. Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding. Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng. Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang. Yuxiao Dong, and Jie Tang. Agentbench: Evaluating LLMs agents. In The Twelfth International Conference on Learning Representations, 2024. URL htttttts///ppeeeeeeeeee
net mett/frrumidd=zAAUBBCCTTT
Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Shelby Heinecke, Silvio Savarese, Huan Wang, and Caiming Xiong. MCPEval: Automatic basedased deep evaluation for ai agent models. 2025. URL hhttps://arxivoor//aapa 12806.
Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. MnP--niv:rse: Benchmarking large language models with real-world model context protucol servers, 2025. URL lhttps://are org/abs/2508.11 14704
11

---

Published as a conference paper at ICLR 2026
Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang. Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. Agentboard: An analytical evaluation board of multi-tura LLM agents. In The Tiiht Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL. htt.hhttss//ppeeeeeeee wtwpeewwwwweeeeooooiiiiee////eee net/foorum2dd444BBBgk
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general ai assistants, 2023. URL
12983
Guozhao Mo, Wenliang Zhong, Jiawei Chen, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. Livemepbench: Can agents navigate an ocean of mep tools?, 2025. URL
https://arxiv.org: wwwg/bs/////...ooooooooogo 000022008800010 01780
OpenAl.
Introducing deep
research.
https///openal..cmmnnnd//
reseaii 2024.
OpenAl. Introducing openai o3 and 04-mini.
http:://openai.comm/ndee
introducing-o3-and- -3d-----------miii/ 2025a
OpenAl. Introducing
2025b.
Shishir G Patil, Tianjun Zhang, Xin Wang. and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. /tt.tttss//oppeervviii wttp://wwwe/eeeoooooo/oooo/oCCo.
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. URL /
Qwen Team. Qwen3-coder: Agentic coding in the world. https://qwenlmm http://wwinn...ithu..oo////////
qwen3-coder/] 2025.
Scale AI. Mcp atlas.
com/ http://aaslec.mmmmmeeeeeerrrr//e atlas 2025.
The MCPMark Team. MCPMark: Stress-testing comprehensive mep use. https://gi thub. com/
eeall symammms/mm 2025.
The Terminal-Bench Team. Terminal-bench: A benchmark for ai agents in terminal environments, Apr 2025. URL
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16022- 16076, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18655//11/0224..allloooggg.. URL wtw.hhtss//claaathollooo..o org/ 2024 acl long 850/
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URL httl:/ss///aaiv..og/a////////// 12516
XAI. Grok-4. httpp:://..al/news/rrokkk
2025a.
xAI. Grok-4-fast.
2025b.
xAI. Grok code fast
2025c.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In The eiiht Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https!
12

---

Published as a conference paper at ICLR 2026
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang. Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany: Benchmarking Ilm agents on consequential real world tasks, 2025. URL http:tss://aattt.orr////////22// 14161
Yunhe Yan, Shihe Wang. Jiajun Du, Yexuan Yang, Yuxuan Shan. Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, Mao Qin, YInxiao Chen, Chen Peng, Shangguang Wang, and Mengwei Xu. MCPWorld: A unified benchmarking testbed for api, gui, and hybrid computer use agents, 2025. Utt.htts:://arat wwwpx/ivvoooggooeeee abs//526..22.. 07672
Shunyu Yao, Noah Shinn. Pedram Razavi, and Karthik R Narasimhan. r-bench: A benchmark for Tool-Agent- interaction in real-world domains. In The Thirteenth International Conference Learning Representations, 2025. URL https://openreeieww nwt nett/ForummmddrroosXxzp
Ming Yin, Dinghan Shen, Silei Xu, Jianbing Han, Sixun Dong, Mian Zhang. Yebowen Hu, Shujian Liu, Simin Ma. Song Wang, Sathish Reddy Indurthi, Xun Wang, Yiran Chen, and Kaiqiang Song. Livemcp-101: Stress testing and diagmosing mep-enabled agents on challenging queries, 2025.
httphhtsss//axiiv wrw.lio.rrr//bbaa abs/ 2508. 15760
Zhipu AL. Glm-4.6: Advanced agentic, reasoning and coding capabilities.
glm-4.6. 2025.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo. Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. In The Twelfth International Conference on Learning Representations, 2024. URL
13

---

Published as a conference paper at ICLR 2026
A MCP SERVER LIST AND SOURCE
We show all the MCP servers used in the TOOLATHLON benchmark in Table The MCP servers we have selected span multiple domains, ranging from everyday entertainment to education, and even to productivity-level business, software development, and beyond. Most of these MCP servers are sourced from existing ceanmnnittddeveloppe projects, and for a substantial proportion of them, we have made further functional enhancements - including but not limited to optimizing tool output, improving robustness in error handling, and adding new tools. Moreover, we recognize that the current coverage of available MCP servers is still insufficient. Therefore, we have also developed new MCP servers for certain application software ourselves, enabling to extend the supported task scope into more domains. We will make these MCP servers publicly available to the community as well, in order promote the building and usage of agents.
Table 4: Complete list of MCP servers used in TOOLATHLON and their sources. Remote/Local means whether server can access local or remote resources like files databases, and Writable means whether server has tools to createfupdate/dellte these resources or just read them.
MCP Server
Remote/L.ccal
Writable
Acxiv Latex
Remote
X
bitps hts.shhiihssaaaaassiiiii
Arxiv
Remote
X
hirps thin cony maziaaz (parxiv- mop server
Canvas-LMe LMS
Local
Kevised Kised on hilpss// wtt.///gghhhh.oo.... com/ Moni gomery440mcp canvas Ims
(Poste.io)
Custom Implemeni
Exeel
Local
V
Bitgs//gtno com harss masssssmuu//eoooo wwwps///ooo-mmp-seeeee -epe-errver
Fetch
Remote
Ditps (000
Filesystem
Local
hirpsw/ghunn.oo mode connle p proloce moloooowervee wervers wnw.veeeeeeeeeeennneee htee/////m/i//////eeeeeett
Git
Local
Bilpss/gohuubco com moneeleceeeeeeeee ppooocc
Gitbub
Remote
Keysee besed on wtt.://gghhubbbooo... wan.u-u-yauuuu------aaa
Google Cloud
Remole
Custom Implementaaion
Google Calendar
Remote
Bllps GoagRKoee Calendar-ennnnannn Mutouub--CP-SSee
Google Forms
Remote
hitps//rmmpp cont mareonntt Veoorie torms-niep
Google Maps
Remote
hippspptiin conn modelconeet contecppooosesoooooooooo Veeeeve www.ee//eeee......ieeeeeeee....e wtt.://www.....teeeeoeooo/a/oooe
Google Sheets
Remole
hilpss/galhub com/ cottuu/..om////mmyymmmm ympp gougle sheels
How ToCokk
X
Kevsed bused httpps//stttab..coo htwp://ww....o..oyyyyyyy//o/oo/o
Hugging Face
Remote
X
Ditps
Migssnigginggaccc.o cornec
Kuhemnetes
Local
on grphs//gthh com/Flux 559mmmp sereeeeueeeeeeeeeess
Memory
Local
hilps://thub.coom mode www.elcooeex..peeeee. www.:n/wwwwppppoppppmooom
Notion
Remote
Revised based https://g/thhbbbb http:///whhh..ccoommmmmmmmoooooo mep-server
PDF Toals
Local
C Custom Implementalion
Playwright
Remote
Kevised bused on
mep
PowerPoiat
Local
hmtpswithubb.oo Btttub..oo//GaggK2SSS (0222200mm Uoc.e- Power SaRt-MC P-Server
12306
Remote
Kevisea NSSS on impps /tmpp//gtttinnn Joconook/ 1. 30-mmpp
Scholarly
Remote
Keyise based on hillps//gttuhh convantyak 147m, schilarly
SnowBlake
Remole
Kevised based on htllps:/gtthhb..c. isann asserman mepp Snowllake server
Terminal
Local&&Remotee
Kevised based on httpss//ttttbbbbb.tttt com Miadeenn sci-mpp-seive
Weights & Binses
Remote
X
Kevised besed on
www.awa wando-me---ooee
WanCommerce
Local
Custom
Implemetitation
Word
Local
V
bitps https:/erhuu..ooog httthhibosom//noneeeeh000iiiiiie Word-MCP-Serw
Yahoo Finance
Remote
finance-epp
YouTube
Remote
X
Kevised bused om htlpp////hhh wtt.://.oohhoooooooooooooom / lemancks emwneckyyouuueee mep server
YouTube Transcript
Remote
http://ghubb.omnjaaaaaaat
traneccppp
B IMPLEMENTATION DETAILS OF AGENT FRAMEWORK
Our framework is built and developed based on OpenAI-Agent--Se (Version 0.0.15), and we make the following main enhancements to mke it more robust and capable for our complex evaluation:
(1) Tool Error Handling: When models call non-existing tool or the tool call returns errors, the agent loop breaks and exits by default. We improve this by giving the errors as observations to the agent without breaking the loop, so that the agent can continue the trajectory to proceed further. This way mimics the realistic, noisy environments where tool calling sometimes does not work and the agent needs to deal with such scenarios;
(2) Overlong Tool Response Handling: Overlong tool outputs (like huge HTML) can easily exhaust models' context, therefore we truncate them to preset threshold (100K characters) instead of placing the entire response into the context. To prevent information loss, toolkit is implemented to enable the agent to search and navigate through the cached raw lengthy tool outputs via paging. The page size is set to 10K characters by default. This toolkit is available for all task evaluations.
14

---

Published as a conference paper at ICLR 2026
a. Surcos hate (k0)
-20-
Wroong Test Narmw 8Sit Tool
GP75
Grat Grdl-Code Codl Neets
63
GP7.5-min
Giis-iign
Despliet Giuste SHalt
Qeren Cader
od-ming
Gnde-4
fimi-22-990 -9t5
Gemoniz Saiah
Claate Someer
Caudee Sonnel
Grad Grak---aat Haat
Geminl-
Figure 9: The success rate difference between trajectories with certain kind of tool calling errors and without errors.
(3) Context history management: To further prevent model context overflow, design context management mechanism with supporting tools. Models can check accumulated token counts and turn numbers current context, and drop historical turns reduce context pressure via these tools. All history, whether dropped or not, remains searchable through these tools as well. When context exceeds limits without model intervention, our framework automatically clears everything except the last 10 turns' preview and initial user input, ensuring continuous agent operation as a final safeguard.
We implement and include the following tools alongside the existing MCP sersers: (a) Python, which executes arbitrary Python code; (b) Web Search, which searches content on the Internet driven by Google Search. (c) Done, which the model can call to explicitly indicate the completion of tool calling for task. (d) Sleep, which the model can call to wait for some time before proceeding.
(4) Extra local tools:
C EXTRA ANALYSIS FOR TOOLATHLON
C.I THE IMPACT OF TOOL CALL ERRORS ON FINAL SUCCESS RATES
During our experiments, we observe that models exhibit tool call errors. These included invoking incorrect tool names caused by hallucination forgetting, and errors raised in tool execution. As shown in Figure 9 we analyze the impact of tool call errors on success rate. For most models, the frequency of tool call errors was negatively correlated with successful task execution, indicating that model's misunderstanding of tool adversely affects its performance. This negative impact was most pronounced in GPT-5.
C.2 CASE STUDY: TYPICAL CHALLENGES MODELS MET IN TOOLATHLON
Unable to fully utilize the given tools resources in fuzzy instructions We sometimes find that the model fails to take full advantage of the resources we provide if we do not explicitly tell the model in detail how and when to use them. It may fail to use indirect methods with existing information and available tools to reach its goal if its first attempt fails. We show an example as follows:
Task dataset-license--ssut by DeepSeek-V3. V
Prompt::.. asking about the license information. please update the corresponding huggingface dataset pages If you need the huggingface token, you can find it in the ".hf token" file.
I notice that the dataset doesn't have a detailed README that I can directly update through the API. However, based my analysis, I can provide the license information response as requested. Based on my research, the datasets are derived from HuggingFaceTBB 's oducational datasets which use the ODC-BY (Open Data Commons Attribution License). This the most permissive license from the original data sources
Last (28th) Turn:
As per the prompt, the model has already been reminded to use the given ".hf.token" file since noticed that the Hugging Face MCP does not allow update operations. However, find that DeepSeek- V tried several times to access this on the server or use web search instead of writing terminal commands Python code to directly access the needed dataset with the given token.
15

---

Published as a conference paper at ICLR 2026
Therefore, even though it has identified the correct license information (ODC-BY), it still fails to update the datasets pages and finish this task.
Fail to handle real complex environment states One of our design principles for this benchmark to setup realistic environments, such as a large project collaboration with tens of developers or an online shop with hundreds of products. We find that models often get lost in such scenarios, c.g. by omitting a large number of items that should be examined. We provide example below:
Task task-tracker by Claude-4-Sonnet
Prompt: Please find developers' branches the "BenchColl" project for most recent commits Update our Notion page: "Task Tracker" based on the new tasks, and create a new branch in GitHub
Evaluation: ... Comparing Notion Page with local groundtruth Excel Issues found:
Row count mismatch. Expected: 116, Got: 91
Excel task "amalytiss-aashoordd" not found Notion database
In this task, we ask the model check a project repository involving more than 10 developers, determine the progress of each person, and add new records to Notion. This requires the model to examine all the different branches and compare them in detail against an example task template. In that the model does attempt to follow this process; however, it does
the observed trajectory, not actually go through all possible files and folders to produce a complete update. As a result, the model updates the Notion pages with far fewer task items than expected, which leads to failure.
Laziness in long horizon working Certain tasks require the model to complete sequence of subtasks over an extended time horizon, such as processing data spanning multiple years. Such workloads are common in real-world applications. While expect the models execute these complex and repetitive tasks in a fully end-to-end manner, our observations indicate that they often terminate prematurely and delegate the remaining work back to the user. An example is shown below:
Task music-analysis by Grok-Code-Fast-
Prompt: music streaming platform wants accurately identify the 1940s pop singles with the most sustained popularity Follow the format of the amusic.analysee result.xlxx" anwlassi...uuuuu.lss'' in the workspace, create one sheet for one year and fill them....
Last (66th) Turn: The analysis has been completed for 1940. used the original Gaogle Sheet data to calculate the longest consecutive weeks each song stayed in the top sorted the leaderboard by longest top 3 streaks descending, then by song name ascending, then by artist ascending. For other 1940s years (1941-1949), the same steps can be applied using their respective sheet IDs. The file now contains the analysis for 1940 as requested. [Claim Done]
In this task, the model is asked to analyze popular singles from the 1940s, year by year. The model carries out this process for the first year (1940), but then stops, claiming it has completed the task after 66 turns of work. This happens even though we have set the system prompt as: ... you can either call the "claim done" tool indicate completion. This will immediately terminate the task, and you will have further opportunity work on which is intended to enforce the model finishes everything before exiting. Nevertheless, this kind of premature termination still occurs, causing an early exit and failing even the first completeness check in the corresponding Excel sheet.
D PROMPT
We use a very simple system prompt (except the tool schemas) in our evaluation, where the (workspace dir) will be replaced with actual agent workspace directory in execution.
16

---

Published as a conference paper at ICLR 2026
Agent System Prompt
Accessible workspace directory: (workspace dir}
When processing tasks, if you need to read/write local files and the user provides relative path, you need to combine it with the above workspace directory to get the complete path. If you believe the task is completed, you either call the "claim_done" tool respond without calling any tool to indicate completion. This will immediately terminate the task, and you will have no further opportunity to work it.
Please complete the given task independently. Do not seek confirmation or additional feedback from the user. You should handle all situations on your own, as the will provide any further information.
preduction
moniortt Ted ternnio
re5e2
hurkl socruaiiie wil
Ado:
de vMtkoo
13
992
stading
clgg
Edds
2025-8844 325-48-1772-477
reced
MidBer
text
nel tupper stppe--tt77e/tteeee www.ht/tteeeeeeeeee acclem
tapper
Fabe
bxntect. 36
1825
06-13 12:999772
1620-2
Medium
Weekd
heverpere
Red Nare
Loclateer Serve
Phiiogel Mede
Iraae
Lseatien lime
Moole
Elh Serce
Level
tee oyee iila prtie
Fobe
Oig 5.70
2025
rede:1
3pl
Egiec.22
Highng
servec tedkcccc%
2023
Weeks
Narraspaca
Ped Name
Corboner flem
Friillged Mede
Imags
Creation Tira
Noce
Bae Score
Bit Lenal
Figure 10: example of format inference task &&s DSs-safety-aaaee where the agent needs to sheets Week1 and Week2 to understand the format should use when filling in Week3 in this Google Sheet with the safety auditing results on a given kubernates cluster.
2010
Figure 11: An example of file edit inference in task muil--ppper-hhomppage where the agent is only given the instruction to update an example personal page on Github needs to explore the structures by itself and determine which files to edit.
E MORE EXAMPLES OF QUALITATIVE ANALYSIS
E EXAMPLES FOR FUZZY USER INSTRUCTIONS
We present two examples of fuzzy user instructions real-world scenarios in this subsection. The first example (Figure 10) comes from the task k8s-safety-audii, where the agent needs to conduct a security audit of deployed cluster based on predefined security audit rules and synchronize the results Google Sheet. However, the user instruction only mentions "update to Week3 sheet," which requires the agent to independently read the existing Week1 and Week2 sheets and infer the required format for filling in the information.
17

---

Published as a conference paper at ICLR 2026
The second example (Figure comes from the task mmatl--pppr-homeepeae where the agent needs to update the relevant content on a personal GitHub homepage based on paper acceptance emails in the inbox. The user instruction only mentions "update my personal page," which requires the agent to independently find the corresponding repository, explore the file structure, and decide which files and which parts of them should be modified.
In both examples, we examine whether the model can, given concise and fuzzy instructions, use tool calls explore and determine the actual actions that need to be performed in the real environment.
E.2 COMPLETE EXAMPLE TASK TRAJECTORIES
We present the trajectories of Claude-4-Sonnet (Anthropic 2025a) on two different tasks. Given that some tool-call results are excessively long, we have simplified certain tool outputs and removed all personally identifiable information.
Traj #1: HuggingFace Upload. This task requires the model to identify, among the 10 training checkpoints, the one with the highest average score across all benchmarks, and write the result to README. md. Subsequently, the model must create a Hugging Face repository and upload all relevant files. We do not directly provide the evaluation scores; after inspecting the folders, Claude-4-Sonnet realized needed to run the evaluation script itself. To increase the difficulty of this task, we also introduce some easily fixable syntax errors into the test code, which the model must identify and correct on its own. Although Claude-4-Sonee eventually identified the correct checkpoint, failed to complete the task because it omitted the images required in README, md when uploading the files. We do not explicitly prompt the model to upload these images, but it should have inferred the dependency on them from the content of README. md.
Traj #2: Notion HR. requires the model to retrieve open positions from Notion and assign roles to candidates based on their resumes. For candidates without suitable position, the model needs send rejection emails. Claude-4-Somee perfectly completed this task after making 45 tool calls. successfully read all the resumes, and fill in correct information accordingly on the Notion table and send rejection emails to all candidates taht apply for positions with no head counts.
Traj #1: FaggingFFcc Upload (Claude-4-Sonnc) failed after 47 tool callings)
User Please scan workspace folder, pick the model checkpoint with the highest eval_accuracy, then push the best model's folder Hugging Face Hub as a model repo named SyAwesomeModel-Teteee
Finalize the repo's README... md with the detailed evaluation results for all 15 benehmarks (keep three decimal places), you refer to the current README. md under workspace.
L Agent I'll help you scan the workspace, find the model checkpoint with the highest avcluacy push it to Hugging Face Hub, and update the README..d with detailed evaluation results. Iet me start by exploring the workspace structure.
filesystem list directory
"path": "workspace"
Tool Response:
[FILE]
tokens
[FILE]
README. md
[DIR]
checkpoints
[DIR] evaluation
[DIR]
figures
[FILE]
hf token. txt
Agent Let me explore the checkpoints directory to see the available model checkpoints:
18

---

Published as a conference paper at ICLR 2026
filesystem list.directory
"path": "workkspace/cheecppointtt"
Tool Response:
[DIR] step.100
[DIR] step.200 200
[DIR] step. 300
...
[DIR] step_1000
Agent Now let me check the evaluation directory to find the evaluation results:
filesystem list.directory
"path": "vorkspaceeevvauuttionn
Tool Response:
[FILE] setup.py..eoe swp
[DIR] benchmarks
[DIR] build
[FILE] eval.py
[FILE] setup.py
[DIR] utils
Agent Let me check the benchmarks directory for evaluation results:
filesystem list.directon
"path": "verk<ppccr/eeallucii/o//eeenrrs
Tool Response:
[DIR] code.gener
[DIR] common_sense
[DIR] creative wr wirting
[DIR]
dialogue.generatin
[DIR] text classification
[DIR] translation
e Agent can see there are 15 benchmark directorics. Let me search for evaluation result files that might contain accuracy metrics:
filesystem search files
"path": "workspace", "pattern": "eval"
19

---

Published as a conference paper at ICLR 2026
Tool Response:
vorkspace/eealluation
oar.apaceeelllaaaoeeeeeeereeeeer vorksspace/evvaluatiimmnvaaaa_ py
Agent Let me search for result files or JSON files that might contain the evaluation results:
filesystem search_filed
"path": "workspace", "pattern": "result"
Tool Response:
matches found
filesystem search_file,
"path":
"workspace", "pattern": "json"
Tool Response:
wor.gpeee/dcekkpoinr//////////// json
horkkppcce/chackoonnts/ssttp 1000/config.
Agent Let me check there are any log files or training logs:
filesystem search files
"path": "workspace", "pattern": "log"
Tool Response:
generation
Agent Let me look for any files with accuracy evaluation metrics in their names:
filesystem search.files
"path": "workspace", "pattern":
"accuracy"
Tool Response:
matches found
20

---

Published as a conference paper at ICLR 2026
Agent Let check the current README.md to understand the structure and see it contains any evaluation information:
filesystem read file
"path": "workspace/README.cc
Tool Response:
Benchmark | Model1 |
Model2
Model 1-v2
MyAwesomeModel
1-----1-------11111
**Core Reasoning Tasks** (RESULT)
Math Reasoning
0.510
0.535
0.521
**Language Understanding** Reading Comprehension [RESULT)
6.1
0.685
0.690
**Generation Tasks**
Code Generation
|
0.615
0. 631
0.648
{RESULT)
**Specialized Capabilities*x| [RESULT]
Translation
0.
0.799
0.801
1 Agent I can see the README.nd has placeholders {RESULT) for the evaluation results. Now need to find the actual evaluation results. Let me check each checkpoint directory to see if they contain evaluation results:
filesystem list.directory
"path": "wookpppaee/hhccooonnnsstttppp""
Tool Response:
[FILE] config.json (FILE) wytorch.model.co
filesystem 1ist dist.directory
"path": "wokkpppcee/ceekpppppntttttttttt
Tool Response:
[FILE] config.
[FILE] pytorch.model.bin
Agent Let me check there are any evaluation results the main evaluation directory:
filesystem directory.tree
"path": "Norkspace/evaluattion"
21

---

Published as a conference paper at ICLR 2026
4 Tool Response:
"name": w.settp.pp.yst.. swp"
"type": "file"
"name": "benchmarks", "type": "directory", "children":
"name": "code generation",
"type": "directory", "children": [
"name": "eval.py",
"type": "file"
},
Agent
Let me check the main evaluation script to understand how evaluation results are generated:
filesystem read file
"path": "eval.py"
Tool Response:
def run benchmark evaluation (benchmark name, model_path)
e Agent Now need to run the evaluation script for each checkpoint to find the one with the highest eval eval_accuracy, Let me run the evaluation for each chockpoint:
terminal command
"command": : "ed workspace & python evaluation/eval..p checkpointss/step.188"
22

---

Published as a conference paper at ICLR 2026
Tool Response:
("type": "text", "text": "0.493 "annotations": null),
("type": "text", "text": "math_reasoning: 0.345 code_gemeration: Error
text.classification: Error sentiment sentiment.analysis: 0.617 question.answering answering:
0.475 logical reasoning: 0.319 Common_sense, 0.53 reading comprehension:
8.475 dialogue.gent generation: Error summarization: 0.517 translation: 0.64
knowledge_retrieval: 0.529 creative.writing:
328 instruction.following
0.55 safety.evaluation: 0.628 " "annotations": null, "error": true}
Processing other model checkpoints ( (Step_200 Step_900)
terminal run.command
"command": "cd workspace & python evaluation/eval.ov checkpointss/seep.000
Tool Response:
("type": "text", "text": 712 ", "annotations": null),
( Agent Perfect! I that step 1000 has the highest eval accuracy of 0.712. Let also extract individual benchmark scores for step 1000. notice there are some errors with code generation, text classification, and dialogue generation benchmarks, but have scores for the other benchmarks. Let me fix those missing benchmarks first to get complete results. Let me check what's wrong with the code ged_eeenettioon benchmark:
filesystem read file
"path": wwt.:p/pwweeeaaaoooeeeeeeeee//ee
Tool Response:
not os. path.
isfile(argsssssss os.path.isfile(arg
model path):
e Agent see the issue the code-generation script is checking if the model_path is a file, but we'r 're passing directory. Let check the text.cassiffication script:
filesystem read. file
"path": hww.gheweeeesltooooeeeeeeeeeeeee
23

---

Published as a conference paper at ICLR 2026
Tool Response:
from wtw.s. benchmark.ttllll import get.benchmark.score
Agent Let check the dialogue generation script:
filesystem read_file
"path": www.sp/wwecllalallooooooooc/a/a.
Tool Response:
from bencharrktaas import get.benchmark.score
Agent Let me check the benchmark utils understand how the scoring works:
terminal run.command
"command": "cd workspace/evallutioo &
python
-c "from utils. www.beark..tttiiii import
get tet.benchmark.score;
(gin (wweeeebbencaarr.scorreeee code gonerrgenertioon 1000)) ""
Tool Response:
Security violation: sells.bbencmarkkutiiil import get_benchmark_scom Invalid command format:
Invalid
command
part
'python
"from
closing quotation
eethon-execute
import sys
import sys. sys.path. insert(0, from utils. wwwcheark.utill.. import get pet.benchmark.scor
# Get scores for step_1000 for the missing benchmarks code.gen sedeee bet.behhaaarkkcoeeee wwaeccoreeeccdeeeeeeeeeeeee "eeeeration., 1000)
text.class score get. benchmarkkckkkkkk score (eerrr(("eeee teext classificatiion 1800)
dialogue score get wtt.benchmark.cco score ("dialogue "'diallguegemerrationn,, 1000)
(fantt""cdeeggeeeraaiiii code.gen.score")
print print (f" generation: dialogue gen score")
text_class_score
24

---

Published as a conference paper at ICLR 2026
Tool Response:
=== STDOUT ===
sedeegeneration: 0.65
fext.clasifficatin:: 828
dialogue_generation 0. 644
L Agent Great! Now have all the scores for step 1000. Let me compile the complete results for step 1000:Based the evaluation results, step 1000 has the highest overall eval accuracy of 0.712. Here are complete results for step 1000:
Math Reasoning: 0.550
Code Generation: 0.650
Now need
Get the HF token
Push the step. 1000 model Hugging Face Hub
Update the README. md with these results
Let first get the HF token:
filesystem read file
"path": "workspace/hfftoon token. txt"
Tool Response:
hf XXXXXXXXXXXXX
Agent Now let me authenticate with Hugging Face:
terminal run command
"cd workspace && echo "hf "xf_xx..wwx""""".""""""........" --token"
"command":
huggingface-cli login
Tool Response:
Security violation:
Invalid command part "huggingface-cll login --token' Command huggingface-cll is not allowed
Agent Let me try using the Python API instead:
25

---

Published as a conference paper at ICLR 2026
python-execute
import
from huggingface.hul import HfApi, login
# Set the token
token "hf XXXXXXXXXXXXXX os. environ[ [OVETrO TOKEN""
# Login
login (token=token)
Initialize API api HfApi()
(rint((Suucessfulll authenticated with Hugging Face Hub")
Tool Response:
STDOUT
Successfully authenticated with Hugging Face Hub STDERR ===
Note: Environment variable Dariable'HF_Tone "OKEN" is set and is the current active token independently from the token you' ve just configured.
Agent Now me check are in step. 1000 checkpoint:
filesystem list.directory
"path": twerksspcce/hhhcchooonntttttptep
Tool Response:
[FILE] config.json
[FILE] pytorch model.bin
e Agent Let me also check there are any other necessary files (like tokenizer files) in the workspace:
filesystem search files
"path": "workspace", "pattern": "tokenizer"
Tool Response:
No matches found
Agent Let me check there are any vocab or other related files:
filesystem tearch.files
"path": "workspace", "pattern":
"vocab"
26

---

Published as a conference paper at ICLR 2026
Tool Response:
No matches found
Agent Let me check the config.json to understand the model better:
filesystem read,file
"path": www.k/pppe//ccccpoon///////s/scp
json""
Tool Response:
"model_type": "bert",
"architectures": ["BertModel"]
L Agent Now let the repository and push the model. First, let copy the best model to
temporary directory and add updated README:
filesystem create directory
"path": "workspace//moelluupll upplaa
Tool Response:
Successfully created directory workspace/mode upload
teeriiall run.command
"command": "cd workspace cp checkpoints/step.1009// model.upload/"
Tool Response:
Command completed with return code: 0
@ Agent Now let me update the README.nd with the evaluation results. need to replace the (RESULT) placeholders with the actual scores:
filesystem read.file
"path": "workspace/README.cc
27

---

Published as a conference paper at ICLR 2026
Tool Response:
Benchmark | Model1 |
Model2 Modell-y
MyAwesomeMode
1-------1----- 1-------------------------------
x*Core Reasoning Tasks** (RESULT)
Math Reasoning
0.510
0.535
0.521
*ALanguage Understandingex Reading Comprehension [RESULT)
0.671
0.685
0.
**Generation Tasks**
Code Generation
|
0.615
0.631
0.648
(RESULT)
**Specialized Capabilitiess4 Translation
0.782
0.799
0.801
[RESULT]
Agent Now let me create the updated README.nd with the actual evaluation results:
filesystem edit file
"path": "README. mmnd
"edits": [
"oldText":") "I **Core Reasoning Tasks** Math Reasoning 0.511
0.535
0. 521 [RESULT) I",
"newText":" =| **Core Reasoning Tasks** Math Reasoning 0.510 0.535
0.521
0.550 1",
4 Tool Response:
Benchmark | Model1
Model2 Mode11-v2
MyAwesomeModel
--- 1-------|-~-- 1--
-1 Core Reasoning Tasks
Math Reasoning
0.510
0.535
0. 521
RESULT
Core Reasoning Tasks
Math Reasoning
0.510 0. 535
0.521
0.555
Agent Now let me copy the updated README to the model upload directory:
terminal run command
"command": "cd workspace & cp README md model upload/"
Tool Response:
Command completed with return code: 0
Agent Now let me push the model to Hugging Face Hub:
28

---

Published as a conference paper at ICLR 2026
python-execute
import
from huggingface.hul import HfApi, login
# Set the token and login
token "hf_xxxe -аао os. environn"H__iiiiieee [OKEN"] login (okee)
# Initialize API api HfApi()
# Create the repository repo_id Syydessomeodel--estRRee.
try:
api.create rewo(repo..ddreeeeeeeeee id=repo id,
private=False, exist.ok=True)
print (f""(ffssiitoo tory repo_id created successfully!")
except Exception as e:
pfint(""Error creating repository: e")
Upload all files from the model.upload directory model_path het.k//pcee/dmmss//orrss////////
try:
upload_follee
folder path#model path,
repo id=repo id,
repo.type="model"
pfintefdM uploaded successfully to repo_id!")
except Exception as e:
print (f"Eror uploading model: e")
Tool Response:
STDOUT
Repository hyywessomeooel-TestReee created successfully!
Error uploading model: 404 Client Error.
Repository Not Found.
== STDERR =
Note:
Environment variable'HF. TOKEN is set and is the current active token independently from the token you' just configured.
Agent need to include my username the repository ID. Let me check who am:
huggingface hf whoami
None
Tool Response:
You are authenticated as anonymized.user.
Agent Now let me create the repository with the correct format:
29

---

Published as a conference paper at ICLR 2026
exthon-execute
repo_id
Tool Response:
STDOUT torository hwo.yiiiee.ssss/////eeeeeeeeeeel
created successfully!
Model uploaded successfully to
STDERR Note: Environment variable HF_TE is set and the
active token independentlyy from the token you' ve just configured.
Agent Perfect! Let verify what files were uploaded:
filesystem list.directon
"path": "vorksppace/modeeluuplaadd"
Tool Response:
[FILE]
README. md
[FILE]
config.json pytorch.model..e
[FILE]
Agent Excellent! Let me also verify README. md content upload directory to make sure it has the correct evaluation results:
filesystem read file
"path": "workspace/mooell hitsspaceemodee.uulloo///E/////
"head": 50
Tool Response:
license: mit
library name: transformers
MyAwesomeModel
Agent Perfect! The task has been completed successfully.
clne
None
Tool Response:
you have claimed the task is done!
30

---

Published as a conference paper at ICLR 2026
Traj #2: Notion HR (Clande-4-Sonnet, succeed after 45 tool callings)
User Please update the candidate information the HR Record subpage of the Notion page according the resumes in my workspace. All information must be filled out strictly according the content the resumes, without making any unauthorized modifications or adding/removing any words. Also, please delete the existing sample entries in the record table. At the same time, if the position applied for by the applicant is currently not open for recruitment, please send email to the corresponding applicant using the following template information (including line breaks). Do not send the email by mistake:
Dear Applicant.
Thank you very much for your interest in joining MCP Inc and for taking the time sub- mit your application. We truly appreciate the effort you put into applying for this position.
After careful review, we regret to inform that the position you applied for is currently filled. However, file for future opportunities that may better match your skills and experience.
impressed with your qualifications and would like to keep your
We encourage you to check our careers page regularly for new openings, and please feel free to apply for other positions that align with your background and interests.
Thank you again for considering MCP Inc as potential employer. wish you the very best in your job search and future career endeavors.
Best regards, HR Team, MCP Inc
M Agent help you update the HR Record subpage in Notion with resume information and handle any necessary email communications, Let me start by exploring your workspace to find the resumes and then locate the Notion page.
filesystem 1ist directory
"path": "workspace"
Tool Response:
[DIR] tod..tools..empfiiee [DIR] cvs
filesystem list.directory
"path": "workspace/cvs"
Tool Response:
[FILE] alice_jackson_reseeeee resuue. .pdf
[FILE] angela_moore wttp/mooorrrrueeeeee
[FILE] mharles.castillo.rees pdf
[FILE] nancy.robinson.resiie pdff
Agent Great! found resume files. me search the Notion page HR Record information.
31

---

Published as a conference paper at ICLR 2026
notion API-post-search
"query": "HR Record"
Tool Response:
("object": "rrsuusl""" "results":.... [....]>
n Agent found the Record page. Let me retrieve contents the structure and any existing database.
notion API-get-block-chilede
"block.id"; "xxxxx"
Tool Response:
("object":
"list", "lbist" "results":
Agent can see there are three databases in the HR Record page:
Positions (block xx-yy-zz)
2. Job Application Timeline (block ID: yy-zz-xx)
Candidates (block ID: zz-xx-yy)
Let first examine the Job Positions database understand what positions available, and then Candidates database see its structure.
notion API-retrieve-a-dattbass
"database.id": "xx-yy-zz"
Tool Response:
("object": ""object"""databaat"" "database", "xx"""xx""""""""""""""""" "xx-xx=z",x"....
notioo API-post-databass-query
"database_id"; "xx-yy-zz"
Tool Response:
("object": "list", "list"" "results":....
Agent Now me examine Candidates database structure:
notion API-retrieve-a-dattbbe
"database id": "zz-xx-yy"
32

---

Published as a conference paper at ICLR 2026
+ Tool Response:
{ ("object": "database", "id": "zzxx=xxxx--"",,...
notion API-post-databass-qqerr
"database.id": "zz-xx-vy"
'ooo Response:
("object": "list", "resuu""""su "results":....
Agent Now I have clear understanding of the situation:Job Positions with available headcount
(Head Counts 0):
Advanced Investment Banking Analyst: 3 positions
Market Risk Specialist: positions
Existing sample candidates and their applied positions:
Alice Jackson Financial Risk Analyst positions available) v
Debra Smith + Securities Trader (0 positions available)
Michael Diaz Investment Analyst (2 positions available) v
need to:
1. Delete all existing sample entries
Read resume PDFs extract candidate information
3. Add new entries hased on actual resumes
Send rejection emails to candidates applying positions with headcount
Let me start by reading all the resume PDFs:
pdf-tools read_pdff ppges
_pages
"pdf_file.pan -itthhhhehh "end_page": 10
vvrrksppceeoos//llccceaaooeneeee pdf", "star t_page"; 1,
Tool Response:
PDF total pages:
Extracted page range: 1-3
Page ALICE JACKSON
**Applied Position: Financial Risk Analyst ** CONTACT INFORMATION
Read the resumes of the other candidates in sequence
33

---

Published as a conference paper at ICLR 2026
L Agent Now I have all the resume information. Let analyze what found:
Candidates from resumes their applied positions:
Alice Jackson Applied: Financial Risk Analyst (2 positions available) v
Education: Master Finance, Columbia Business School; Bachelor Economics, University of Pennsylvania
Email: alice Jackson38@mcp.. com
Angela Moore Applied: Securities Trader positions available) x Education: Master of Science in Computational Finance, Carnegie Mellon University
Email: angela_moon mcc.eemmccor...cp..cmmmm com
Now need
1. Delete all existing sample candidates
2. Add the real candidates from resumes
3. Send rejection emails to candidates applying for positions with 0 headcount
Let start by deleting the existing sample candidates:
notion API-update-a-bloct
"block.id"; "xy-yz-zx", "archived"; true
Tool Response:
"status" :400,
"object: :loject"""ernee "error", "code": "validattonnerror"...
Agent need to the correct API delete pages from the database. Let me delete them by one:
notion API-patch-page
"page_id": id": "xy-yz-zx", "archived": true
Tool Response:
("object": "ppgee"""""age" "id": "xy-yz-zx",...
Delete others one by one
Agent Great! I've deleted the existing sample candidates. Now need to create new pages for actual candidates based their resumes. Let me start creating them one by one:
notion API-post-page
"parent": ("database.id"""""z=xxx
"properties":
"title" [["teet[(ttt ("content"; "Alice Jackson"))], "ttte"""
"title"
Tool Response:
("status" :000, "object"; "error", "code": "val validation error",...
34

---

Published as a conference paper at ICLR 2026
L Agent need the correct format for creating pages in database. Let check the database
structure again and create the pages properly:
notion API-post-page
"parent": ("database id": "zzxxxxxxx-y"";
"properties": { ("Name": ("title" :[[ [["text": ("content"; "Allice Jackson }]],
(Email":("riee" [["xt""[("xxtt"" content"" "alice jackson38@mcp.. com't ]]]], "Applied Position": ("rich"(ttttttt [e[[tt""": ("content": "Financial Risk Analyst"]]]},
Crrichhtextt:(("t"eeereee [""extt:: ("content") "Columbia Business School }}]}
"Highest Degree": ("select"; ; ""name" "mmater")))
Tool Response:
("object": "page", "id": "yx==""yxxxxx,....
/
Agent Great! Now let me add the remaining candidates:
Add the remaining candidates
Agent Perfect! I've successfully added all 10 candidates from the resumes to the Notion database. Now I need send rejection emails candidates who applied for positions that currently not open for recruitment (positions with 0 headcount). Based my analysis, the candidates who need rejection emails are:
Angela Moore (Securities Trader 0 positions)
Debra Smith (Securitics Trader positions)
Emily James (Securities Trader 0 positions)
Martha Morales (Portfolio Manager 0 positions)
Let send the rejection emails:
emails send_email
"to"; "subject": "Application Status Update Securitic Trader Position", "body":..
com",
Tool Response:
Email sent successfully to angela..moree9@mppcco
Send other rejection emails
Agent Excellent! have successfully completed all the required tasks:
_laim_done
None
35

---

Published as a conference paper at ICLR 2026
< Tool Response:
you have claimed the task is done!
36


Reviews:
[{"review_id": "LjSClrwTsK", "reviewer": "Reviewer_VZsp", "summary": "This paper introduces TOOLATHLON (Tool Decathlon) — a benchmark designed to evaluate the performance of language agents on complex, long-horizon, and cross-application tasks. The benchmark covers 108 tasks, 32 applications and 604 tools, ranging from common productivity software (e.g., Notion, Google Sheets, WooCommerce) to domain-specific systems (e.g., Kubernetes, Snowflake). Each of the 108 tasks requires multi-step reasoning, realistic initial environment states, and verifiable execution via deterministic scripts.", "questions": "Are the locally containerized applications (e.g., Poste.io for Gmail) treated as “real tools,” and if so, how do they differ philosophically from “simulated” counterparts in other benchmarks?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "+ TOOLATHLON spans a 32 real-world applications demonstrating tool and environment diversity.\n\n+ Execution-Based Evaluation: TOOLATHLON uses deterministic evaluation scripts that compare environment states, ensuring objectivity and reproducibility.", "weaknesses": "Benchmark Labeling and Definitions: The categorization of existing benchmarks in Table 1 as “Real Tools” or “Not-Real Tools” appears inconsistent and potentially misleading. For example, τ-Bench is flagged as not-real despite subsets (“airline,” “retail”) interacting with actual databases. Similarly, BFCL is marked not-real, although it supports real execution in the “Execute” category,  with \"Crowd Sourced\" being community-contributed tools. LiveMCPBench is labeled not-real despite its claim to run live MCP tools, while TOOLATHLON’s own use of simulated components (e.g., local Poste.io for Gmail) blurs the same boundary. \n\nNovelty Ambiguity: While TOOLATHLON integrates many existing elements (real MCP servers, realistic states, cross-app tasks), the conceptual novelty beyond combining these components is modest. The work would benefit from a clearer articulation of methodological advances versus engineering scale-up.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "frFezwLssc", "reviewer": "Reviewer_aKrQ", "summary": "This paper introduces the Tool Decathlon (TOOLATHLON), a new benchmark designed to evaluate tool-using language agents on tasks that are diverse, realistic, and long-horizon. The authors argue that existing benchmarks are often limited in scope, focusing on narrow domains, single applications, or tasks that lack realistic complexity and environment states. To address this, TOOLATHLON spans 32 software applications (from everyday tools like Google Calendar to professional ones like Kubernetes and BigQuery) and 604 tools, mostly sourced from real-world Model Context Protocol (MCP) servers.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A strong benchmark, heavy engineering behind \n- will be good use for the community", "weaknesses": "- Not really a weakness but the paper uses the average number of turns as a proxy for task difficulty. While reasonable, this is an outcome-based metric that can be influenced by the agent's (in)efficiency. A more intrinsic, task-defined complexity metric (e.g., based on the number of required applications, minimum number of steps in a ground-truth trajectory) could provide a slightly more objective measure of difficulty when analyzing performance across Easy/Medium/Hard tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5JyqRHdvcL", "reviewer": "Reviewer_EmBX", "summary": "This paper introduces TOOLATHLON, a benchmark designed to evaluate LLMs on complex, real-world tasks using MCP tools. It extends existing MCP benchmarks by incorporating 108 real-world tasks and 604 tools across 32 different software applications. The authors benchmark current LLMs on this dataset and find that state-of-the-art models still perform poorly: the best model, Claude-4-Sonnet, achieves only a 29.9% success rate. These results highlight that long-horizon, multi-application task execution remains a significant challenge for current AI agents.", "questions": "see above", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. The benchmark contains a substantially larger number of tasks than prior MCP benchmarks and evaluates models across realistic settings, including real-world state initialization and fuzzy instructions.\n\n2. Rigorous task validation and design make the benchmark reliable and valuable for MCP-agent research.\n\n3. The authors benchmark a wide range of state-of-the-art LLMs and provide detailed analysis, including case studies of failure modes and in-depth discussion of experimental results.", "weaknesses": "The current evaluation setup for LLM agents with tools is relatively simple. The authors specify all tools for a task and provide the full tool list to the agent, which can significantly inflate the input context. More sophisticated tool-selection strategies could be explored—for example, using retrieval methods to surface relevant tools dynamically rather than supplying the entire tool inventory upfront.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)