MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
Authors: Zhenting Wang
Qi Chang
Hemani Patel
Shashank Biju
Cheng-En Wu
Quan Liu
Aolin Ding
Alireza Rezazadeh
Ankit Shah
Yujia Bao
Eugene Siow
Venue: ICLR 2026 Poster (2026)
Keywords: Tool-using Agent; Real-World Tasks; Model Context Protocol
Primary area: datasets and benchmarks

Abstract: We introduce MCP-Bench, a benchmark for evaluating large language models (LLMs) on realistic, multi-step tasks that demand tool use, cross-tool coordination, precise parameter control, and planning/reasoning for solving tasks. Built on the Model Context Protocol (MCP), MCP-Bench connects LLMs to 28 representative live MCP servers spanning 250 tools across domains such as finance, traveling, scientific computing, and academic search. Unlike prior API-based benchmarks, each MCP server provides a set of complementary tools designed to work together, enabling the construction of authentic, multi-step tasks with rich input–output coupling. Also, tasks in MCP-Bench test agents’ ability to retrieve relevant tools from fuzzy instructions without explicit tool names, plan multi-hop execution trajectories for complex objectives, ground responses in intermediate tool outputs, and orchestrate cross-domain workflows—capabilities not adequately evaluated by existing benchmarks that rely on explicit tool specifications, shallow few-step workflows, and isolated domain operations. We propose a multi-faceted evaluation framework covering tool-level schema understanding and usage, trajectorylevel planning and task completion. Experiments on 20 advanced LLMs reveal persistent challenges in MCP-Bench.

Published as a conference paper at ICLR 2026
MCP-BENCH: BENCHMARKING TOOL-USING LLM AGENTS WITH COMPLEX REAL-WORLD TASKS VIA MCP SERVERS
Zhenting Wang', Qi Chang', Hemani Patel!., Shashank Biju"?, Cheng-En Wu', Quan Liu', Aolin Ding', Alireza Rezazadeh', Ankit Shah', Yujia Bao', Eugene Siow! 'Center for Advanced AI, Accenture, UC Berkeley
(zhenting. wang, chang / @accenture. com (hemani,, abi ju @berrely..... com (cheng-en. wu, quan.b. bil., a.ding, alireza. rezazadeh @accenture, lankit.parag.c parag. yujia.bac, eugene. siow) Baccenture..... com
ABSTRACT
We introduce MCP-BENCH, benchmark for evaluating large language model (LLM) agent on realistic, multi-step tasks that demand tool use, cross-tool co- ordination, precise parameter control, and planning/reasoning for solving tasks. Built on the Model Context Protocol (MCP), MCCP-BENCHE connects LLMs to 28 representative live MCP servers spanning 250 tools across domains such as finance, traveling, scientific computing, and academic search. Unlike prior API- based benchmarks, each MCP server provides set of complementary tools de- signed to work together, enabling the construction of authentic, multi-step tasks with rich input-output coupling. Also, tasks in MCP-BENCII test agents' abil- ity to retrieve relevant tools from fuzzy instructions without explicit tool names or execution step, plan multi-hop execution trajectories for complex objectives, ground responses in intermediate tool outputs, and orchestrate cross-domale workflows workflows-cappablittes not adequately evaluated by existing benchmarks that rely explicit tool specifications, shallow few-step workflows, and isolated do- main operations. We propose a multi-faceted evaluation framework covering tool- level schema understanding and usage, trajectory-level planning and task comple- tion. Code and data: https: ///gttgttt///////////////// com/ Accent mmp- bench.
INTRODUCTION
Recent advances in large language models (LLMs) have enabled new generation of tool-using agents that can interpret natural language instructions, plan multi-step workflows, and interact with external tools to solve complex tasks (OpenAI, 2025c; Comanici et al., 2025; Anthropic, 2025; Yang et al., 2025; Kimi et al., 2025; Zeng et al., 2025; Chen et al., 2025). Such agents are increasingly deployed in real-world domains such travel (Xie et al., 2024), healthcare (Saab al., 2024; Mehandru et al., 2024), and finance (Xiao et al., 2024), where solving user queries requires chaining multiple tools, reasoning over structured outputs, and coordinating interdependent operations.
Despite rapid progress in LLM agents, existing benchmarks for tool use remain fundamentally lim- ited. Early efforts such as ToolBench (Qin et al., 2024) and BFCL v3 (Patil al., 2025a) aggregate large collections of APIs, but these APIs are designed for isolated functionality. As a result, tasks often reduce to few-step tool calls or rely on artificially stitched pipelines, since tool inputs and outputs rarely align naturally across APls. 7 7-Bench (Yao et al., 2025) moves a step further by selecting small set of APIs whose interfaces are relatively compatible, enabling cleaner composi- tions. However, its coverage is limited to only handful of domains and tools, making it difficult to scale task diversity or capture the complexity realistic multi-domain workflows. Together, these benchmarks fall short in modeling realistic dependency chains and stress-testing long-horizon plan- ning. More recent benchmarks such as MCP-RADER (Gao et al., 2025) and MCPEval (Liu et al., 2025a) begin to leverage the Model Context Protocol (MCP) (Anthropic al., 2024), which pro- vides a standardized invocation schema across servers. However, these benchmarks remain narrow in scope. For example, MCP-RADER (Gao et al., 2025) and MCPEval (Liu et al., 2025a) cover only

---

Published as a conference paper at ICLR 2026
arxiv
STEAM
Google Maps
Hugging Face
LLM-based Task Synthesis
NIH
NATIONA
M
THE MOVIE DB
THE MTT
Query
Tasks
reddit
Perform Tasks
Rule-based and Judge
Numpy
Agent
Performance
LLM Agent
Exccution Routts
and Trgecticy
Real-world
Connect to Severs
Figure 1: MCP-Bench connects LLM agents to real-world MCP servers exposing 250 structured tools across domains such as finance, science, and research. Tasks are generated via LLM-basee synthesis, then executed by the agent through multi-turn tool invocations. Each execution trajectory is evaluated using a combination of rulc-based checks and LLM-as-a-Judge scoring, assessing agent performance in tool schema understanding, multi-hop planning, and real-world adaptability.
a few servers with at most several dozen tools, which limits task diversity and makes most work- flows relatively short (e.g., single retrieval followed by summary). Also, both existing API-based and MCP- based loil-using benchmarks lack testing of planning capability under fuzzy instructions: tasks typically specify detailed execution step explicitly, so agents not challenged infer which tools are appropriate when the instructions are underspecified. Furthermore, they omit evaluation of more complex scenarios such as multi-goal objectives (c.g.. booking travel that requires coordinat- ing flights, hotels, and local transport), pridence-based reasoning with information grounding (c.g., generating answers that cite intermediate tool results rather than hallucinating), and cross-domain orchestration (e.g., combining financial tools with news sources to explain stock movements). As summarized in Table 1, none of the existing benchmarks adequately reflect the complexity, fuzzy, and diversity inherent in real-world tool use.
To overcome these limitations, we introduce MCP-Bench, a large-scale benchmark that evaluates LLM agents in realistic, ecosystem-basee tool-use scenarios. As illustrated in Figure 1, (CCP-Benc) connects agents to a diverse ecosystem of production-grade MCP servers exposing 250 structured tools across domains such as finance, science, and research. Each server provides complementary tools designed to work together (e.g., a scientific computing server integrating data loading, matrix operations, and visualization), while the MCP protocol ensures consistent invocation schemas across servers. This combination enables both realistic intra-server dependency chains and complex cross- server, multi-hop workflows. Tasks in MCP-Bench synthesis pipeline. Dependency chains are first discovered from tool I/O signatures, then translated into natural language instructions. A quality filtering mechanism ensures solvability and realism. To assess agent in realistic scenarios, each task is rewritten into fuzzy and instruction-minima. variant that retains the core objective but omits explicit tool references and execution steps. The example of the tasks in MCP-BENC can be found in Table and Table 10. Each task is executed by the agent through multi-turn interactions with MCP servers, and the resulting trajectories are evaluated with two-tier framework: (1) rule-based checks for tool validity, schema compliance, runtime success, and dependency order, and (2) rubric-driven 1.M-as-a-Judge scoring of task completion, tool usage, and planning effecttivnness. To ensure stability, prompt shuffling and score averaging are applied.
generated automatically via L..d-based
Our contributions can be summarized as follows: A realistic tool-using benchmark that leverages MCP servers expose 250 tools across 28 servers, enabling both intra-server dependency chains and cross-server orchestration. structured task synthesis pipeline that generates both fuzzy
Table I: Comparisons to existing tool-using benchmarks.
MCP Ecosystem
Information Grounding
Fuzzy Task Description
Complex Tasks with Cross-domain domain Massive Goals
Beuchmark
Domains
Tools
Orcbestration
ToolBench (Qin al. 2024)
49
3451
X
X
BFCL (Palil al., 2025a)
8
24
X
Bench (Yao al., 2025)
2
28
X
MCP-RADER (Gan et al,, 2025)
9
42
X
X
MCPEval (Lio all, 2025a)
5
X
MCC-Bench(Ours)
28
250
/
2

---

Published as a conference paper at ICLR 2026
Table 2: Example of task in MCP-BENCH.
Servers Tools
Task Description
Servers: Paper Search, BioMCP Useful Tools: gene gene_getter, variant searcher, variant getter, article_scarcher, article_gctter, scarch_ ocarch_pubmed, scarch_arxie arxiv, download_ame _
I'm working on a project about why melanoma patients with the BRAF V600E mutation so often become resistant to treatment, and I'm a bit stuck piecing everything together. I'd love to know:
What we know about how common V600E is in the general popu- lation, and what Vai says about its pathogenicity The five most influential research papers from the past year specifically on V600E- positive melanoma and resistance to vemurafenib or dabrafenib Any Phase 2 or Phase 3 trials that are actively recruiting patients with V600E melanoma and testing new combinations or approaches to beat resistance The key molecular mechanisms behind why V600E tu- mors stop responding to treatment Serious adverse events from the FDA database for vemurafenib in melanoma (say. the 10 most recent reports) Any functional annotations for V600E that explain how it affects BRAF protein activity. Could you pull all that together with real paper IDs, trial numbers, and data sources? can't 't present vague information to my team need concrete evidence.
read arxiv paper,
search biorxiv,
download biorxiv. read_ Les__biorxv__pppp__
rial_sce trial_getter
trial Rial_Jocations___ Lolllocaatinn_ettee. _eetter
trial maal_refrences_gettee drug_getter,
nei_ iii_oraaizaatoo_scareee
paper_search, fetch, think
instructions of complex, multi-hop tasks grounded in real tool semantics. @ A robust evaluation framework combining rule-based execution checks with rubric-based LLLM-as-a-Judge scoring, en- abling comprehensive assessment of execution correctness and strategic reasoning. @ large-scale empirical study evaluating 20 state-of-the-ar LLMs on 104 challenging tasks, revealing persistent weaknesses in agentic capabilities in realistic and complex tool-using scenarios. By bridging the gap between isolated API benchmarks and real-world ecosystems, MCP-Bench provides the standard- ized and scalable platform for evaluating the agentic reasoning and tool-use capabilities of LLMs.
2 RELATED WORK
Benchmarking LLMs. Recent benchmarks have steadily progressed from static evaluations to more interactive, real-world tasks. Early efforts such as MMLU (Hendrycks et al., 2021) and BIG- bench (Srivastava et al., 2023) focused on single-turn fixed-format evaluations, testing broad factual knowledge and reasoning via multiple-choice or free-form responses. HELM (Liang et al., 2023) introduced a multi-metric evaluation framework over static-text tasks to compare LLMs holis- tically across accuracy, calibration, fairness, and robustness. More recently, the focus has shifted to reasoning and agentic capabilities (Koh al., 2024; Kokane et al., 2024; Zhang et al., 2025; Du et al., 2025; Wei et al., 2025). MMLU-Pro (Wang et al., 2024) increases difficulty via LLM-generated, ceasoning-intensivv items to reduce contamination. MT-Bench (Zheng et al., 2023) evaluates multi- turn dialogue quality, measuring consistency and contextual coherence. AgentBench (Liu et al., 2024) assesses tool-based decision making in simulated environments. WebArena (Zhou et al., 2024) explores open-ended web navigation, while REALM M-nnhh (Geng & Chang, 2025) focuses on goal planning under dynamic disruptions. Despite these advances, most benchmarks still fall short of modeling realistic complex workflows where diverse tools should be composed, and inter- mediate outputs integrated across steps.
Evaluating Tool-using Capability. As tasks grow more complex, evaluation now targets reasoning. planning, and execution across tool interfaces. Mind2Web (Deng et al., 2023) used fixed browser- action APIs for think-to-act planning, and WebArena (Zhou et al., 2024) added self-hosted domains with embedded tools, yet both depend on hand-crafted toolsets. To broaden tool selection and coor- dination, subsequent benchmarks pursue broader tool coordination in distinct ways: r-Bench (Yao et al., 2025) adds simulated users and pass end-state checks; BFCL v3 (Patil et al., 2025b) vali- dates multi-tume API workflows via AST analysis; _ (Yu et al., 2025) stresses inter-tool de- (Zhong et al., 2025) adopts rubric-based, execution-
pendency reasoning; and ComplexFuncceeee verified scoring. Yet all still depend on bespoke toolsets, limiting realism. This gap motivates MCP- based benchmarks, which standardize LLM-too interaction and exttoe domain-aligne tools. MCP-RADAR (Gao et al., 2025) and MCPWorld (Yan et al., 2025) test tool selection, parameteri- zation, and execution within MCP servers yet need manual setup. MCPEval (Liu et al., 2025b) also automates MCP-using task generation and evaluation with five MCP servers. Scalable, cross-server evaluation in MCP ecosystems with complex tasks remains open, motivating our direction.
3

---

Published as a conference paper at ICLR 2026
Algorithm 1 Multi-turn Planning and Observation
1: Input: Task instruction maximum steps maxx
2: Output: Final answer answer, execution trajectory t jectory
3: function MULTITURNEXECUTER T mmax)
4: trajectory + 0
> Initialize execution trajectory
5: 8o f- Update(u)
> Get initial state
6: for t= 0 to max do
7:
D Generate current tool plan >> Execute tools in the plan > Generate a compressed summary of the observation trajectory + trajectory U U((a,, o,}
(continuet, ax) <- Tlpln (8t)
8:
0: f Texec (at)
9:
0: - Feompress (0t)
10:
b Log plan and observation > Update agent internal state
9t+1 t Update(S), 0()
12:
if continue; = False then break
13:
> Stop if agent signals termination
14:
answer t- (u, trajectory) return (answer,
Produce final answer from trajectory
15:
3
MCP-BENCH FORMALIZATION AND DESIGN PRINCIPLES
Following Yao al. (2023), we formalize our benchmark structured extension of the classical Partially Observable Markov Decision Process (POMDP), tailored to tool-using agents that oper- ate across multiple external servers and tools. Our formulation includes execution paradigms: (1) one-shot global planning, and (2) multi-turn planning and observation. Each benchmark task represented as a POMDP tuple AS,A, O, T, R,,, 2, 2), where: S is the global state space; A is the action space including both planning steps and tool invocations; O is the observation space containing tool execution results and internal signals; T S x A
O the transition and observation function; R S + [0, 1] is the reward function; l denotes the task instruction space; and Σ = (01, 02 on is the set of available MCP servers. To stress-test the agent's reasoning and tool-selection capabilities, we attach set of distractor servers (10 in this paper) to each task, in addition to the MCP servers required for completion. This setup exposes the agent to over 100 extra tools per task. Each server 0; Σ exposes set of tools T, defining the complete tool set T T= U, Te. A structured tool invocation is written as agoo = (o., tool_ name, par parameters). The full action space is A Apluming V Awulss and the observation space is O = O U Osate- For the workflow of the agent, we adopt a multi-round decision process (Yao et al., 2023). At each round I, the agent generates plan o, conditioned on all previously observed outputs, then executes the tools in az and updates its internal state. Note that each a, could includes the plan for execut- ing multiple tools parallel. This continues for up to Tmax (20 in this paper) rounds or until the agent signals to stop. Final reasoning is performed after observing the complete trajectory. The full routine detailed in Algorithm 1. In line 4-5, we initialize the exccution trajectory trajectory, and the initial agent state 8o from the task instruction u. In line 6-11, we iteratively plan and ex- ecute actions: the planning policy plann produces the current tool plan, the execution policy Texec performs the planned actions, and the compression policy Teompress generates concise summary of the observation. This compression step is crucial because some tools return very long outputs, and summarizing them prevents excessive context windows. The compressed observation is logged into tra trajectory, and the agent state is updated. In line 12-13, we check the termination signal continue and stop early if is False: In line 14-15, the final answer is generated from the com- plete execution trajectory via Fimal- The prompt used for the agent execution can be found in Section A.4. Detailed design principles and how MCP-BENCE reflects them can be found in Section A.
4 MCP-BENCH CONSTRUCTION
4.1 MCP SERVER COLLECTION
Our benchmark covers 28 representative MCP servers spanning eleven functional domains (Fig- ure 2a). The largest categories are Media & Entertainment and Research & Knowledge (each 14.3%), followed by Finance, Science, and Software Development (each 10.7%) Smaller shares include Geography & Travel, Social & Intelligence, Mathematics, and Health (7.1% each), with niche domains such Weather, Time, and Divination (3.6% each). In total, these servers provide 250 tools. Tool counts vary widely (Figure 2b), from single-tool servers (e.g., Call for Papers, Vice, Movie Recommender) to large multi-tool platforms such BioMCP (35 tools), Scien-
4

---

Published as a conference paper at ICLR 2026
Divirstinn
Wenther
Goegraiiyy Thae
16-3%
7.%%
MaJamurto
10L70
7.2%
TO
19.75
HEP Servers
(a) Category distribution MCP servers.
(b) Tool distribution servers. Figure 2: Overview of MCP server ecosystem used in the MCP-BENCHL
tific Computing (26 tools), and Medical Calculator (22 tools). This diverse ecosystem spans scien- tific computation, finance, content discovery, geospatial services, and specialized analytical utilities, ensuring broad capability coverage in MCP-BENCII. Details of the involved MCP servers and the descriptions of all tools can be found in Table 7.
4.2 TASK SYNTHESIS
A challenge in building tool-using agent benchmarks lies in transforming collection of real-world MCP servers into high-quality tasks with realistic natural language descriptions. Given tools spread across different servers, how can we construct meaningful, solvable, structurally grounded, but challenging tasks at scale? We decompose this challenge into three key stages: dependency chain discovery, automatic quality filtering, and task description fuzzing. Examples of synthesized tasks can be found in Table and Table 10. Besides the task synthesis pipeline, the tasks in also undergo human inspection to ensure their realism, executability, and the reasonability of the de- pendency chain analysis. We use 04-mini (OpenAI, 2025c) as the task synthesis LLM. All prompts used can be found in Scction A.5, In total, we synthesized 56 tasks with a single server, 30 with 2 servers, and 18 with 3 servers. The single-server tasks span all servers in benchmark. The two-server and three-server combinations for multi-server setting are listed in Table 11.
Dependency Chain Discovery and Task Generation. We start the task synthesis by analyzing dependency chains among the provided tools: sequences where each tool's outputs naturally flow into the next tool's inputs. These chains both inherent dependencies arising from natural tool relationships and scenario- based dependencies constructed for meaningful workflows. For multi-server configurations, dependencies to ensure genuine tool coordination across different data sources. This yields diverse structural patterns including linear workflows, parallel execution groups, and hybrid compositions. The task synthesis LLM are then asked to generate tasks based on the analysis results for dependency chains (see prompts Section A.5). Also, the analysis results for the dependency chains are used in the evaluation phase as the reference for the LLM judge (see Section A.6).
structural scaffolds for task generation. We analyze
emphasize cross-server
Automatic Quality Filtering, Each generated task undergoes rigorous vro-dimensional quality evaluation: Solvability: Whether the task can be completed using available tools. Practical utility: Whether the task addresses genuine user needs rather than contrived scenarios. Tasks failing the quality threshold (solvability: 9,0/10, utility: 5.0/10) are disgarded (see details in Section A.5). This ensures only high-quality tasks that meet our standards enter the final benchmark, maintaining benchmark integrity the cost of reduced quantity.
Task Description Fuzzing, For tasks that pass quality filtering. the algorithm generates fuzzy task variants that state high-level goals without explicit operational details. These fuzzy descriptions transform structured instructions into natural business requests, requiring agents to infer appropriate tool sequences and execution strategies from the available dependency structures. For domains requiring precise inputs (e.g., scientific computation, unit conversion), the fuzzy variants critically preserve all numerical values and concrete parameters while adopting conversational language. This ensures tasks remain mathematically solvable while testing the agent's 's ability to bridge the gap between user intent and technical execution. Detailed prompt used for task description fuzzing can be found in Section A.5.
5

---

Published as a conference paper at ICLR 2026
5 EVALUATION METHOD AND METRICS
We use comprehensive evaluation framework combining rule-based metrics and LLM judge scor- ing. The rule-based component measures tool usage robustness across four dimensions- -ame validity, schema adherence, and runtime component assesses strategic quality in task completion, tool selection, and planning efficiency and effectiveness, using structured rubrics with prompt shuffling and score averaging to ensure fairness.
execution traces. The LLM-as-a-Judee
5.1 RULE-BASED EVALUATION
To assess the schema understanding and execution robustness of agent's behavior, we evaluate its tool usage along dimensions of name validity, input schema adherence, and runtime success. Let E (ei..... ] be the set of all tool invocations during execution.
Tool Name Validity Rate. This metric assesses whether the agent selects tools that exist within the [[eE E:toolee) T where lool(e) returns the identifier of the tool [E]
allowed set Iivailable: Realid
invoked in event e. This metric penalizes hallucinations invalid tool references and reflects the agent's grounding in tool availability.
Schema Compliance Rate. This metric measures whether each tool invocation provides correctly structured parameters that match the tool's expected input schema, e.g., the type of the parameters:
Oschema (eeE:vvald teal()) schesssme))iidsssceeeeeee where valid_ [eil___o(((e) is Boolean function returning Eivaid [
True if tool(e) E Tavailable. and valid schema(e) returns True if the parameters in event match the expected input schema of the tool. This ensures the agent understands the expected API argument formats and avoids malformed requests.
Execution Success Rate. This metric quantifies the proportion of tool invocations that successfully
return results without runtime failure: Raaceess [feE (ssccres((e)]] where success(e) returns True if the tool call in event is executed without runtime errors and produces a valid result. high success rate indicates robust interaction with external systems and proper error handling.
[E]
5.2 LLM-AS-A-JUDGE EVALUATION
To further assess the strategic quality of agent behavior beyond raw execution correctness, we em- ploy an LLM-as-a-Judge framework. The evaluator is prompted to score agent performance across three core axes: task completion quality, tool selection/usage rationale, and planning effectiveness. Evaluations are grounded solely in observable evidence from the task definition, final solution, and exccution trace. By default, the judge model used here is o4 04-mini (OpenAI, 2025c).
Rubrics- based Judge Prompt. The LLM judge is provided with the fuzzy task description given to the execution agent, the concrete task description before fuzzing (not provided to the agent being evaluated; see Section 4.2), the dependency analysis (not provided to the agent being evaluated; see Section 4.2), the agent's final solution, the total number of execution rounds, summarized execution trace, and the list of available tools. It is explicitly instructed to remain impartial and evidence-driven, and assign scores strictly based on proportional success. Scoring follows structured rubric that decomposes each evaluation axis into multiple sub-dimensions (detailed in Section A.6). It assigns scores based on structured rubric that breaks down each evaluation axis into multiple sub-dimensions (detailed in Section A.6). Each sub-dimension is rated on scale from 1 to 10. The average score across the sub-dimensions yields the overall score for that axis, which is then normalized to the [0, 1] range for benchmarking.
Task Completion Quality assesses whether the agent delivers a correct, complete, and evidence- based solution. This includes evaluating how well the task goal is fulfilled (task fulfillment), whether all necessary sub-tasks are covered and supported by evidence (information grounding), and whether the response remains relevant and focused.
Tool Usage Quality evaluates the agent's is effectiveness in employing tools. Sub-dimensions include suitability of tool selections for each sub-task (tool appropriateness) and the correctness and com- pleteness of parameter values provided to these tools (parameter accuracy).
Planning Effectiveness assesses the coherence and efficiency of multi-round execution. This in- cludes whether inter-tool constraints are respected (dependency awareness) and whether the agent minimizes redundancy and exploits opportunities for parallel execution (parallelism and efficiency).

---

Published as a conference paper at ICLR 2026
Table 3: Leaderboard on MCP-BENCH, ie., results of different models, averaged across settings
with single server and multiple servers.
Reloshaed
LLM Julge
Schemo Understanding
Task Completion
Tool Usage
Planning Effectiveness
Modid
Overall Scure
Valld
Scherna Compliance
Exccution
Tosk Pultilinntt
Information Grounding
Tral Appropriateness
Porameter
Dependency Awarcness
Parullelis and Eifhclency
Nome Rate
Accuracy
0.361
0.395
0,353
0.428
Bona 5 906 visicn
59.65
55
0.295
0.464
0.115
0.267
0,495
micro-11
56
95. 1%
0.338
0.415
0.544
0.315
0,212
0.516
59
50 53
0.318
04533
0.51
01367
0.191
0.30
stioial amall 2503
96.4
95.6%
85.2%
0.313
0.445
0.53
0.446
0.319
0.232
0,530
spr 4o zsiei
97.5%
98.1%
03.9%
0.374
0,500
0.555
0,544
0,352
0.201
0.57
99.99 9/.%%
95.8%
91.63
0.349
0491
0.581
0.525
0.355
0.262
0.588
garmita 3-27h-i1
98.8%
97.6%
94.4%
0.378
0.500
0.08
0.572
0.363
0.249
0.382
spi-4o
98.9%
98.3%
92.55
0.384
0.542
0627
0.587
0.426
0.272
0,595
gemmi-2.5 2.5-foeb-
59.65
97
94.35
01412
0.777
0627
0.997
0.804
0.226
0596
quen
93.115
0.411
0,90
0663
0168
0.413
0.305
D672
98.5%
0.502
0.577
0631
0.448
0629
sps-oss-22
98.5%
93.65
0.547
0.62
3661
0.68
0854
gim-
0661
0868
99.3
0.549
39.5
0.354
0.676
0.611
gomin- 2.5pmm
59.65
99 6/
0.52
0.23
0.620
0.329
0.40
2015
07.7%
96.55 83
94.09
0.16
0.001
0661
0.576
0.339
0,622
99 19.93
99 9%
97.194
0.61
0.106
0734
0.326
0.562
0.359
0.715
spo.5
100.0%
99.3%
99.19
0,777
0.52
0.707
0.749
0.649
0,339
0,749
Prompt Shuffling and Score Averaging. Li et al. (2025) has shown that LLM judge can exhibit sensitivity to the ordering of rubric dimensions. To mitigate this issue, we adopt prompt shuffling strategy that randomly permutes the order of major evaluation axes (c.g., Task Completion, Tool Selection, Planning Efficiency) as well as the sub-dimensions within each axis. Importantly, while the ordering is shuffled, the semantic content and phrasing of the rubrics remain unchanged to ensure fairness and consistency. By default, we perform five independent shufflings of the rubric prompt for each task instance. Each shuffled prompt is submitted separately to the LLM judge, resulting in five sets of rubric-based scores. For each scoring run, we first average the sub-dimension scores within each axis and normalize them to the [0. 1] range. The final judgment score for the task is then computed as the average of the four independently obtained axis-level scores. The final benchmark overall score is the average task overall score under the 104 tasks (including both single-server tasks and multi-server tasks). More discussion on calculating the final overall score can be found in Section A. This randomized multi-pass evaluation strategy substantially reduces the likelihood that evaluation outcomes are biased by prompt structure, and enhances the robustness and fairness judgment process. Empirical results (Section A.9) show that this method lowers score variance, leading to more reliable and stable assessments.
of the
6 BENCHMARK RESULTS
In this section, We present the experiment results and discussion for MCP-BENCH, Due to the page limitation, we put the discussion about the quality of our LLM Judge pipeline and the ablation studies for the prompt shuffling and score averaging strategy in Section A.9.
6.1 MAIN RESULTS
We evaluate 20 representative LLMs in our experiments:
(Meta, 2024a), (Meta, 2024a), mistral-small-
Mcta (2024b), 2503 (Mistral, 2025), nova-micro-vl (Amazon, 2024), llama-3-3-70b-instruce (Meta, 2024c), gpt- 4o-mini (OpenAI, 2024), gemma-3-27b-it (Google, 2025), gpt-4o (Hurst et al., 2024), gemini-2.5- flash-lite (Comanici et al., 2025), kimi-k2 Kimi et al. (2025), gpt-oss-200 (OpenAI, 2025b), qwen3- 30b-a3b-instrrut-2507 (Yang et al., 2025), gpt-oss-1200 10b (OpenAI, 2025b), glm-4.5 (Zeng et al., 2025), (wen3-235b-a222-22555
(Yang et al., 2025), claude-sonnet-4 (Anthropic, 2025), gemini-2.5- pro Comanici et al. (2025), o3 (OpenAI, 2025c), and gpt-5 (OpenAI, 2025a). Table 3 reports results averaged across settings with single server and multiple servers. We find that schema understanding capabilities remain consistently high for strong models, with o3, gpt-5, gpt- 120b, (wen3-233b. a22b-2507, and got-4o all surpassing 98% in schema compliance and valid tool naming. However, substantial differences emerge in leeeer-level reasoning. The strongest models gpt-5 (0.749), o3 (0.715), and gpt-oss-1200 120b (0.692)acchieve the highest overall scores, reflecting both accurate tool use and robust planning effectiveness. By contrast, smaller models such (0.428) lag behind, showing weaker performance in dependency awareness and parallelism despite adequate execution success. These results highlight that while basic execution has largely converged, planning and reasoning capabilities remain the key differentiators among models. Table 4 and Ta- ble 5 provide detailed comparison between single- and multi-server settings. We that weaker
7

---

Published as a conference paper at ICLR 2026
Table 4: Detailed results with different models on single-server setting.
Julp
Schonaa Undertanding
Tank Completion
Tedl Fsage
Marning Effectioane
Provider
Madel
Vald Tial
Schens Corglianc
Excottion
Tesk Fultiturnt
Iefirmation Grozading
Tedl Approristemss
Pacameter Accurery
Deperdeses Awareses
Puniliiinn and Iiickrey
Overall
Nams Rate
Stccen
ZAI
pa--5
99.05%
0.51
0.01
0.543
0:11
0.855
chank
0,42
Amnons
V6.1
92.9%
91.175
0:51
0.421
0350
0.300
0.20
0.530
Mateal
95.74
95.1%
$7.25
0.390
0.450
0.514
0.484
0,358
0.238
0.544
Allvala
cvaad 2355 2226-222
creas 30t 2000ad-2500
18. 8%.8%
95.5%
97.3%
97.1%
92.99
6.60
0,550
0311
0.541
0.81
0151
0:511
0.578
0:11
0.585
0.647
Google
greves Bidv-ite
09 65
48.33
0.394
0.566
0.69
0.699
0.200
gresee-3. 2.5 pm
10300%
09.8%
48.55
0.551
0.736
0.70
0.700
0.551
0.341
Baax- 3-1 Aliisttent
96 84
93.45
92.05
0.03
0.308
0.577
0.37
0.534
0.142
0.458
91.79
66
93.8%
0.355
0.440
0.570
Blara 3-3 706-isinet
09.55 90.5%
51.9%
95.15
0.358
0.518
0.638
0.566
0.579
0 0.280
0.500
97.85
04.95
05.35
0.81
01551
0.566
0.59%
0:771
0:211
0.576
[p-
w.05
07.9%
91.55
6:96
01518
0.501
01200
0-3M
0.278
QueA1
12th
00.7%
92.15
99.8%
6631
0331
0.221
03500
0.944
0.502
11:10
65
09.2%
92395
07.17
6A32
01712
0.31
03333
0.8M
0.390
12720
zpe-3
000075
92.15
99.25
0.54
01388
0.381
03881
0/627
0.209
0.349
Table 5: Detailed results with different models on multi-server setting.
Jnnel
LLM Judas
Complettion
Promider
Medel
Yald Tha!
Schems
Esecttiin
Taat Fxttiinent
Infeemation Grardiing
Trol Appropraimmmx
Parsmeter Accurary
Deperdency Avrrssse
Paraliation and 12tikincy
Overall Score
Name Hate
Complonne
Stacess
7.A1
99 9t
49.6%
46.2%
0:917
0.672
0.611
0.13
0.6
Kimi
kimn-kk2
M6.4%
91.25
91.79
6311
0.556
0.58
0.58
0.06
0.24
0.10
90.7%
96.8%
92.7%
0.405
Marual
97.2%
95.05
$5.1%
0352
0.438
0.412
01401
0.399
0.255
0.122
Allbaba
(werch 235) 222h 2507
(we12 30h 2lt inttadt 5507
09 26
84
91.25
99.36
91 $1.95
92.1%
0.544
0.71
0.50
0.003
0.025
0.5%
0.664
0.53
0.480
0.09
0.316
0.2%6
0.602
0.60
0379
0.355
01562
Google
96.7%
92.75
95.15
0571
0171
0366
13534
0.515
Hbrra- 3-1 Kh-te-mmr
95.75
15.15
$3.5%
0251
02335
0331
0237
07717
0.101
01115
Mdta
Harra- 2-0-40t--icre-eeeee
11.15
99.9%
0:911
01:30
0.47
01611
0.711
0.176
81111
Haraa 1-3- 11--ttttme
54-15
43.55
0.574
0.463
0517
0.425
05500
135- 4o-iiii a24-40
07.3%
97.2%
91.0%
6.99
0.463
0.504
0.419
0.300
0.212
0.534
08.8%6
03.88
91.0%
0.990
0.355
0,744
0.547
0.444
0.265
0.581
spt 1af-voo-22 206
03.798
91.2%
6.59
0.625
0.566
0.656
97.8%
0.354
0.525
00.54
93.0%
97.0%
6351
0.66
0.991
0.586
0.5%
0.592
0.710
314-5
000.09
93.5%
98.7%
0.701
0.517
0.509
0.734
0.676
0.388
0.730
models degrade noticeably once the number of servers increases. For example, -la-i-t---
falls from an overall score of 0.438 in the single-server case to 0.415 with multiple servers, while nova-micro-vl drops from 0.520 0.471. The main sources of decline lie in dependency aware- ness and parallelism, which become harder to sustain in distributed workflows. Interestingly, the drop is not always smooth- peer----errmmaceeee fluctuates across different server counts, suggesting that the mix of sequential dependencies and parallel orchestration stresses models in different ways. In contrast, strong systems such as gpt-5, o3, and gwen3-235b--22bb
remain much more stable. gpt-5 holds the highest overall score around 0.75 across both settings, while o3 and qwen3-235b- a22b-2507 consistently stay competitive above 0.70. These results underline that execution quality alone is no longer the bottleneck-the real differentiator is robustness to scaling, where top-tier models demonstrate clear advantages in handling long-horizon, cross-server tasks. Together, these results show that while modern LLMs have mastered execution fidelity, their ability to generalize to complex, adaptive, cross-server workflows is still limited. CCC--BENCH exposes this gap system- atically, providing rigorous benchmark for advancing agentic LLM capabilities.
6.2 DISCUSSION ON THE AGENT PERFORMANCE ON DIFFERENT CAPABILITIES AND INSIGHTS FROM MCP-BENCH
Score on Different Capabilities. Table and Table 5 also provide a fine-grained breakdown of performance across six evaluation axes: task fulfillment, information grounding, tool appropriate- ness, parameter accuracy, dependency awareness, and parallelism efficiency. On task completion, frontier models such as gpt-5, o3, and opt-oss-120b achieve the strongest results, exceeding 0.63 in fulfillment and 0.70 in grounding. whereas smaller systems like miero-vI remain below 0.35 and 0.45 respectively, reflecting weaker semantic consistency. In tool
and nova-
8

---

Published as a conference paper at ICLR 2026
Table 6: Average rounds and tool calls per task on different models.
Model
Single
Multiple Calls
Average
Rounds
Tool
Rounds
Rounds
Calls
glm-4.5
10.7
50.0
87
42.9
Kimi
kimi K2
3.8
20.2
4,0
21.1
39
20.6
Anthooie
cludde-onnet-4
7.8
39.2
10.5
49.2
9.2
44.2
Amazon
nova nova-mizro-vi
9.0
48.7
12.7
07.4
103
58.1
Misral
mistral-small- 2503
6.4
66.9
6.6
67.2
6.5
67.0
Alibaba
ywen 30b-33 insoinnoneee 3507
gwen3-235b- aaa--a22b--00 2507
3.7
22.7
4,4
25.4
18.0
4.0
4,0
24.1
gemma- gemini-23 lite
44.5
7.8
Google
129
101.7
86.8
gemin 2.5 pcu
31.3
10.0
43.5
8.2
37.4
Tamm- 1h---innst
16.4
137.6
18.2
173.6
17.3
153.6
Meta
Uaama 3-2 90b vision-instrrct
Hamm--31-- 1i-ma-3---70--nssiii -icinsi-nstt
10.9
12.1
58.4
63.9
13.7
11.4
67.6
47.9
11.8
12.3
55,9
63.0
Llama instruct
30.3
5
26,9
2pt-4o-mini
56.9
15.4
64.4
14.2
gpt-4o
5.8
21.8
OpenAT
gpt-oss- 120b o3
gpt-oss-20h
37.7
8.3
36.9
18.3
7.0
31.7
43.0
4.5
23.0
8.0
33.7
6.3
28.3
gpt-5
8.1
76.5
10.5
81.
9.2
78.9
selection, top-tier models again dominate: gpt-5, o3, and gemini-2.55pro maintain appropriateness and parameter accuracy around or above 0.70, while weaker baselines plateau closer to 0.30-0.50. The sharpest disparities appear in planning effectiveness. gpt-5 sustains the highest dependency
awareness (0.76) with competitive parallelism efficiency (0.34), closely followed by o3 (0.69 and 0.37) and gwen3-235b-a226-225
(0.54 and 0.31). By contrast, smaller models rarely exceed 0.30 on either dimension, underscoring planning the most significant frontier capability that separates state-of-the-aas agents from weaker baselines.
Insights from MCP-BENCHA The combined evidence from Table 3. Table 4, and Table 5 yields several insights into the strengths and weaknesses of current L.LM agents:
Schema understanding convergence. Low-level capabilities such schema compliance and valid tool naming have largely converged across models. Even mid-scale systems achieve accuracy above 95%, suggesting that basic execution fidelity no longer the primary bottleneck. More discussion about the schema understanding convergence can be found in Section A.12.
Scalability under multi-server settings. As the number of servers increases, task complexity rises, but the performance curves are not strictly monotonic. Strong models (e.g.. o3, gpt-5) maintain relatively stable scores across single- and multi-server settings, while weaker/small models (c.g., -lama-3-1770-innstruee show clear degradation with occasional fluctuations. This indicates that adaptation in multi- server scenario is a differentiating capability. Note that single-server vs. multi- server settings can be also viewed as a task difficulty axis, as multi-server tasks are substantially more complex. They involve cross-server dependencies, and empirically require more interaction rounds and tool calls. As shown in Table 6, even for strong models like o3, the average number of rounds and tool calls increases notably when moving from single-server to multi-server tasks (c.g., 4.5 rounds/ / 23.0 tool calls to 8.0 rounds/ / 33.7 7 tool calls).
Gaps in higher-order reasoning. The largest separations appear in planning effectiveness. Top models demonstrate coherent structural reasoning, dependency awareness, and adaptive reflection, reaching around 0.72 on these aub-dimensions, whereas weaker models rarely exceed 0.30. This highlights that morizooi reasoning and multi-hop coordination remain open challenges.
6.3 NUMBER OF ROUNDS AND TOOL CALLS FOR DIFFERENT MODELS EXECUTING TASKS
Table 6 reports the average number of interaction rounds and tool calls required for different models to complete tasks in MCP-BENCE The results highlight both the complexity of the benchmark and the efficiency differences across models. Tasks involve chaining heterogeneous tools across servers, requiring both sequential reasoning and paral- lel orchestration. As a result, even strong models typically require several rounds of interaction and multiple tool calls, reflecting the non-trivial nature of the task distribution. Model-level differences are nevertheless clear. Smaller systems such as Ilama-3-1 3-1-Sb-inttruct consume the most resources, averaging 17.3 rounds and over 155 calls per task, while models like gemini-2.55 2------ff---tteee also
CCC-BENNC are inherently multi-step and often

---

Published as a conference paper at ICLR 2026
hibit heavy reliance repeated tool usage (86.8 calls average). In contrast, stronger models such as gpt-4o, 03, and gwve33-23bb-a22b222 achieve comparable or higher success rates with much leaner execution, typically under 30-40 calls and 6-8 rounds. Frontier systems like gpt-5 and gpt-oss- strike a middle ground: they engage deeper multi-step reasoning (7-9 rounds) but with more controlled call budgets (48-79 calls).
CONCLUSION
In this paper, we introduced MCP-BENCH, a large-scale benchmark for evaluating LLM agents in realistic, ecosystem-based tool-use scenarios. Built on MCP, MCP-BENCH connects agents to 28 production servers with 250 tools, enabling complex multi-hop workflows and cross-domain orchestration. Our automated task synthesis pipeline generates 104 challenging tasks with fuzzy instructions that require strong agentic capabilities to solve. Through our evaluation framework combining rule-based checks and LLM Judge scoring, we revealed that even state-of-the models struggle with different capabilities such as dependency chain compliance, tool selection under noisy environment, and long-horize planning.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Amazon.
Amazon nova foundation models, 2024. URL http://///wsammroocooo///a/aa/ gererative- ai /nvva/.
Anthropic.
Introducing claude 4, 2025. URL claude-4.
Anthropic al. Model context protocol. GitHub repository, 2024.
Aili Chen, Aonian Li, Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang, Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, et al. Minimax-ml: Scaling test-time compute efficiently with lightning attention. arXiv preprini 2506. /3585, 2025.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprin! arXiv:2507.0626/ 2025.
Xiang Deng, Yu Gu, Boyuan Zheng. Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. Advances in Neu- ral Information Processing Systems (NeurIPS), 36:28091-28114, Sept. 2023. URL
ppppeeeeeeecsc.r.rrrrrpeeepppeee 3550022011taa00001e62222222peeee taterratttttttt_ee__aaaaaarrrrr pdf.
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench:
A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506. //63,, 2025.
Xuanqi Gao, Siyi Xie, Juan Zhai, Shqing Ma, and Chao Shen. MCP-RADAR: A multi-dimennsional benchmark for evaluating tool use capabilities large language models. arXiv preprint, 2025. URL htwps: /ttass::/aee //arxi//aiiii wwg.abs.////.oooooooo.o......... 2505. 6700.
Longling Geng and Edward Y Chang. REALM-Bencht A real-world planning benchmark for LLMs and multi-agent systems. arXiv preprint, 2025. URL httpp:://a..i..orgg////p///// 2502.
18836.
Google.
Introducing gemma 3: The most capable model you can run on a single gpu or tpu, 2025. URL htttt:htt//ooott google httt/hhchhcogyyeeeeeeeooeeeoooaa developers //ges///emmmmm///
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding. In International Confer- ence on Learning Representations (ICLR), Jan. 2021. URL https://cpenreviee n
forum? d=d7dd 3mmm2.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21226, 2024.
Team Kimi, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.205344 2025.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yo Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint, 2024. URL https:/// //arxiv. org/ abs//b////// 2 13649.
Shirley Kokane, Ming Zhu, Tulika Awalgaonkar, Jianguo Zhang, Thai Hoang, Akshara Prabhakar, Zuxin Liu, Tian Lan, Liangwei Yang, Juntao Tan, et al. Spectool: A benchmark for characteriz- ing errors in tool-use lms.. arXiv preprint, 2024. URL https: : arxiv. www.obs.///oo..o.ooooo....o... 2411. 13447.
11

---

Published as a conference paper at ICLR 2026
Qingquan Li, Shaoyu Dou, Kailai Shao, Chao Chen, and Haixiang Hu. Evaluating scoring bias in llm-as-a-judge. arXiv preprint wiXi::2006.2316,, 2025.
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga,
Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning. Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Pe- ter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language mod- els. Transactions on Machine Learning Research (TMLR), Aug. 2023. ISSN 2835-885555 58568866. URL https: wwpppeeeeeeeeveeee/ net/ /
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang. Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as agents. International Conference on Learning Representa- tions (ICI.)), Jan. 2024. URL https: // //tpp/opprr/////ee peppenevvieee .et.e/eweeeeeeeeeeeeeeeee.eeeeee forum? d=zAdD30aCT.
Zhiwei Liu, Jielin Qiu, Shiyu Wang. and et al. MCPEval: Automatic MCP-based deep evaluation for Al agent models. arXiv preprint arXiv:2507.12806, 2025a.
Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang. Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao, Huan Wang, Shelby Heinecke, Silvio Savarese, and Caiming Xiong. MCPE- val: Automatic MCP-based deep evaluation for ai agent models. arXiv preprint, 2025b. URL https: //arxiv org/ abs 20bs22507 12806.
Nikita Mehandru, Brenda Y Miao, Eduardo Rodriguez Almaraz, Madhumita Sushil, Atul J Butte, and Ahmed Alaa. Evaluating large language models as agents in the clinic. NPJ digital medicine, 7(1):84, 2024.
Meta. Introducing llama 3.1: Our most capable models to date, 2024a. URL http::/p metta. com/bl og //eta-tttttm a1ama 11am-33111
Meta. Llama 3.2: Revolutionizing
and vision with open, cus- URL https://a http://a.meaaa...eeeeme com/blog/ /e
edge ai
tomizable models, 2024b.
11ama- 1-2--conntt-2024-viiiioo
Meta. Llama 3.3, 2024c.
URL https: http://wwwwalaaaaacmm
Mistral. Mistral small 3.1, 2025.
URL.
attps/taaa.aaaaaaa///aae nnwss/
mistral-small-
OpenAl. Gpt-4o mini: Advancing cost-efficient intelligence, 2024. URL https::///pnnaa
OpenAl. Introducing gpt-5, 2025a.
URL https::/openal..com/nnnt
introducing-gpt-55
OpenAl. Introducing gpt-oss, 2025b.
URL httips://openal.com/inn
introducing- spttnggggpttss/////
OpenAl. Introducing o3 and 04-mini, 2025c.
URL https::/openai..cm//
introducing-oo o3- -nd--4--innnn
Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agen- tie evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025a.
12

---

Published as a conference paper at ICLR 2026
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, lon Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL); From tool use to agentic evaluation of large language models. In International Conference Machine Learning (ICML), May 2025b. URL https: openreviivvv ew net/ / forum? dddgddddddmmdmm
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. In International Conference on Learning Representations (ICLR), 2024.
Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, et al. Capabilittes of gemini models medicine. arXiv preprint, 2024. URL /aapps///areee org/ abs 2404.44444.00 18416.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adri Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transac- fions on machine learning research (TMLR), May 2023. ISSN 2835-8856. // //opeeeeview..eeeeeeeeee. net/ forum? wttpummmiidyyyyttytoo.ooo
URL https:
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro:
more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (NeurIPS), 2024. URL https: //proceee ngs neurips wtwp./nerrrssss....oppeee____ www.innn..nninneeeneen
Berchmarks pdf.
Zhenting Wang, Shuming Hu, Shiyu Zhao, Xiaowen Lin, Felix Juefei-Xu, Zhuowei Li, Ligong Han, Harihar Subramanyam, Li Chen, Jianfa Chen, et al. Milm-as-a-uudge for image safety without human labeling. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 14657-14666. 2025.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504. 12516, 2025.
Yijia Xiao, Edward Sun, Di Luo, and Wei Wang. Trading/geens: Multi-agents Ilm financial trading framework. arXiv preprint, 2024. URL https:://aaxaiaaaee org/ abs wwwpaa/s...... 22112220128.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. TravelPlanner: benchmark for real-world planning with language agents. Interna- tional Conference Machine Learning (ICML), Jan. 2024. URL https: /ttp:///ppeeeeeeeeeeee pdf.
net pst/pol met/pdf/ wtpf///eed/e 22e/aadd877 of 6c21 8af2dee382222ddd833353
Yunhe Yan, Shihe Wang, Jiajun Du Yexuan Yang, Yuxuan Shan. Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, et al. MCPWorld: A unified benchmarking testhed for API, GUI, and hybrid computer use agents. arXiv preprint, 2025. http::t 07672.
org/ abs/25066
An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 08X. 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Represenlations (ICLR), 2023.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan. 7-Bench: Evaluating tool- augmented language agents through human in in-the-loop collaboration. In International Confer- ence on Learning Representations (ICLR), Jan. 2025. URL https: / /ppppevieeeeeeettt... forum? forumm?.d=roNsx2guun
13

---

Published as a conference paper at ICLR 2026
Peijie Yu, Yifan Yang, Jinjian Li, Zelong Zhang, Haorui Wang, Xiao Feng, and Feng Zhang. ca- Bench: The things real disturbing Ilm based agent in nulti-tasking. arXiv preprint, 2025. URL https:/ /arxiv.. org/ htg/bb//// /t0p/2505...2 2505.18746.
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang, et al. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint arXiv:2508.0644 2025.
Jianguo Zhang, Thai Hoang, Ming Zhu, Zuxin Liu, Shiyu Wang, Tulika Awalgaonkar, Akshara Prabhakar, Haolin Chen, Weiran Yao, Zhiwei Liu, et al. ActionStudio: A lightweight framework for data and training of large action models. arXiv preprint, 2025. URL https://arxiv.
org/ arg/abs// 2503. 22673.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neu- ral Information Processing Systems (NeurIPS), volume 36, pp. /ppp/////// URL https: htttpss/pproceeeiigg. neurips cc/ wwwpepaaprraeee fi www.s/eeppppaaeee 0111612888899557822200555-222222 Datar--aatttttet Bar_eetss_____eeenharaaaeee
Dec. 2023.
pdf.
Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, and Jie Tang. ComplexFuncBench: Ex- ploring multi-step and constrained function calling under long-context scenario. arXiv preprint, 2025. URL https:::p.........tt....tt..tts. : arxiv. org/ abs 22501.1132.
Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realis- tic web environment for building autonomous agents. In International Conference on Learn- ing Representations (ICLR), Jan. 2024. URL https:/ http///ppeeeeeeeeeeeeeeeeeeeeeee ew net ferumformm? oK. 3c6ytx.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
In this Appendix, we first discuss the key capabilities for tool-using LLM agents and how MCP- Bench reflects them (Section A. We demonstrate more details of used MCP servers and LLM in Section A.2 and Section A.3. We then demonstrate the detailed prompts used in task execution, task synthesis, and evaluation in Section A.4, Section A.5, Section A.6, respectively. We then display examples of the input schema for tools involved and more details of the tasks in Section A.7 and Section A.8. We also have the discussion about the quality of our LLM Judge pipeline and the abla- tion studies for the prompt shuffling and score averaging strategy in Section A.9. We then provide the qualitative analysis of planning failures in Section A. 10, the discussion about overall score com- putation in Section A. 11, discussion about frontier models' saturation on schema understanding in Section 1. discussion about long-teu term maintainability of the benchmark in Section A.13, more discussion about the novelty and contributions in Section A.14, and the discussion about the limi- tation in Section A.15. Finally, we discuss the disclosure of LLM usages in this paper Section A.16.
A.1 KEY CAPABILITIES FOR TOOL-USING LLM AGENTS AND How MCP-BENCII REFLECTS THEM
To perform effectively in tool-augmented environments, LLM agents should demonstrate several critical capabilities beyond standard language modeling.
Tool Schema Understanding and Compliance: Agents must faithfully interpret and satisfy com- plex invocation schemas that involve nested JSON structures, enumerated types, constrained value ranges, and mixtures of required and optional arguments. Success requires aligning natural language reasoning with precise formal specifications. 250 tools of varying complexity from simple scalar inputs to deeply nested hierarchical struc- sures-ensure that even subtle schema violations are detected. IIlustrative examples of diverse input schemas are provided in Section A.7. Tool Retrieval and Selection under Fuzzy Instruc- tions: Agents must identify the correct tools from large, heterogeneous tool spaces when confronted with ambiguous or underspecified task descriptions. This requires disambiguating semantic variants, coping with naming inconsistencies, and avoiding traps posed by superficially plausible but irrele- vant tools. MCP- -Cnc-Bench stress-tests retrieval precision by attaching 10 distractor servers to every task, introducing 100+ additional tools per instance. Moreover. fuzzy task variants (Section 4.2) de- liberately omit explicil tool names and detailed step descriptions, forcing agents to infer appropriate tools purely from contextual cues. Long- Horizon Planning and Cross-Server Orchestration with Long-Horizon Massive Goals: Realistic applications demand multi- multi-round workflows that span domains, maintain interdependent states across rounds, and sometimes pursue multiple goals simultaneously. Agents must manage sequential and parallel dependencies, coordinate heterogeneous outputs, and optimize efficiency through judicious orchestration. MMCP-Bencl includes both single-serve and -ertir tasks with up to 20 execution rounds. Its evaluation framework explicitly measures structural co- herence, dependency awareness, parallelism efficiency, and reflective adaptation (Section 5). Tasks include not only linear workflows but also complex compositions requiring concurrent interactions across multiple servers with multiple objectives. Information Grounding and Evidence-Basee Reasoning: To avoid hallucination, agents must ground responses in actual tool outputs, maintain factual consistency across calls, and provide traceable evidence for their claims. MCP-Bend evalu- ates grounding by coupling execution history with rubric-based LLM judgments, rewarding answers that correctly cite tool outputs and penalizing unsupported reasoning (Section 5). Real- World Adaptability: Finally, agents must leverage broad world knowledge to interpret domain-specinn semantics, robustly handle diverse tool behaviors, and synthesize heterogeneous outputs into coher- ent solutions. and healthcare to scientific computation and cultural heritage, ensuring that tasks reflect the diver- sity and unpredictability of real-world tool use.
-eechBecch enforces strict schema validation across
spans 28 production graadd MCP servers covering domains from finance
A.2 DETAILS OF USED MCP SERVERS
In Table 7, we show the detailed descriptions for the involved MCP servers and the associated tools. Detailed information about rate limits is provided in Table 8, which lists those MCP servers with
15

---

Published as a conference paper at ICLR 2026
official API documentation explicitly specifies rate limits, All rate limits are sufficient for running one model at a time in our experiments.
Table Details of tools and descriptions in used MCP servers.
Server Name
GitHub Repository
Tools
Description & Tools
Bibliomantic
htops:/ /e//ggtttun tuub zon/ /
Description: Ching divination service provid- ing traditional Chinese divination methods with enhanced hesagran interpretaton and statist- cal tracking. Tools: i_ching divination (Per- forms enhanced Ching divination using tradi- three method changing lines ysis), bibliomantic. consultation (Provides compre- hensive bibliomantic consultation with tra- ditional Ching elements and interpretations). get, hexagram details (Retrieves detailed hexa- gram information including traditional Chinese names, Unicode symbols, and rich commentary), server statistics (Displays enhanced server usage statistics and perfornance metries)
bibl Lomant pep-seewer
Commit Hash:
0tce:533:667e44666 117700h5
Math MCP
Etss httpps///ght http///ihhhha thub, con/
13 Description: Mathematical computation service providing essential arithmetic operations and tistical analysis functions for numerical data pro- cessing and analysis. Tools: add (Performs ad- dition of two numbers with precision handling). subtract (Executes subtraction second number from first with numerical accuracy) multiply (Cal- culates multiplication flow protection), division (Performs division with zcro-division error handling and precision con- trol). sum (Cumpules sum of any number ues in list or array), mean (Calculates arith- metic mean average of numerical data sets), me- dian (Determines middle value of sorted numerical datasets), mode (Finds most frequently occurring value numerical datascts), min (Identifies min- imum value from lists of numbers), max (Deter- maximum
Blsandlen Loksun/ /na
Commit Hash:
433739 757.3557577702222000020066666
two numbers with over-
from numerical datasets), floor (Roands numbers down to nearest integer us- ing floor function). ceiling (Rounds numbers up integer using ceiling function). round (Rounds numbers to ncarest integer with standard rounding rules)
BioMCP
ps //ss///p thbbb
cotuubbcoo con/
14
Description: Comprehensive biomedical research platform integrating literature search, clinical trial data, and genetic variant analysis with Al- powered research planning and Google Decp- Mind's AlphaGenome predictions. Tools: search (Mulii datahuse biomedical literature and clini- search with tion), fetch (Retrieves comprehensive details for specific biomedical recerds using unique identi- liers), think (Required structured sequential think- ing tool for research strategy planning). arti- cle searcberrr (Searches PubMed/Pubi Tatur3 for search articles and prepeints about genes and variants), article petter (Fetches detailed article information including abstracts and full text) trial searcher (Comprehensive "linicalTrrals.gor search with multiple filtering criteria), trial, getter (Retrieves all available clinical ural informa- by NCT
ceaoronzclogy bocg/bioncp
Commit Hash:
9848ece
thinking integra-
protocol gelter (Fetches core protocol details including study design and sponsor information), trial references, getter (Rc- trieves all linked publications and background lit- erature for trials), trial, pulcomes, geller (Fetches detailed outcome measures and results data), trial locations getter (Retrieves study locations contact details
investigators),
ant searcher (Searches MyVariant for go- netic variant database records with pepulation frequencies). variant_getier (Fetches comprehen- sive genetic variant details including consequences and annotations), alphagenome predictor (Predicts variant effects on gene regulation using Google DeepMind's state-of:the-ar AlphaGenome model)
16

---

Published as a conference paper at ICLR 2026
Table 7 continued from previous page
Server Name
GilHHub Repository
Toools Description Tools
Call Papers
Ettpss/ cithub. com/ /oonsst///
Description: Academic
dis- covery service for researchers seeking publication presentation opportunities. Tools: get events (Searches for academic conferences and events matching specitic keywords with detailed submis- information)
call- tor- papezo-nag
Commit Hash:
(.cba277160999991111 wwwp://www1ssscccccc22c2eo/////
Car Price Evaluater
ELuper whub. cun/yussaallt
Description: Brazilian automative market anal- ysis service providing current vehicle pric- ing data through FIPE (Fundaçãoo Instituto de Pesquisas Econónicas) API integration. Tools: get brands (Retrieves comprehensive list al available car brands from FIPE database with brand names), search, price (Seurches specific car midelss and their curren market prices by brand name with detailed pricing informa- tion), get_ vehieles, by_typ (Fetches vehicles car- egorized by type including cars, motorcycles, and trucks with specifications)
Commit non-noaanann
Context7
Htepss
Description: Programming library documentation service providing up-to-date documentation access through Context7's encrypted and secure library system. 'Hool: resolve library (Resolves pack- to Context? computible brary IDs and returns matching librarics list,, get- library-dccs (Fetches current documentation for libraries using exact -omextt--oomaaibbee library IDs with comprehensive API reference)
t/github thub luper
$5906656037000 (oR.uuood744775leeee Eefieal 2876600
age product eibrary
DEX Paprika
pttpps//gtt
inppprr p
Comprehensive decentralized change analytics platform providing real-time DeFi data, liquidity analysis. and trading insights across mulliple blockchain networks. Thols: getNet- works (Required first step to retrieve all sup- ported blockchain networks with network IDs like ethereum and solana), getNetwork (Fetches available decentralized exchanges specific works). getNetworkPools (Primary function to get top liquidity pools specilic comprehensive pool data), geiDexPools (Retrieves pools from specific
Description:
despaeprike map
Commit Hash: Rec2b9657eh5n 140000044211 14dele
platforms on networks), getPoolDetails (Provides detailed pool information including liquidity, volume, and trading metrics), DetaokenDtaills (Fetches comprehensive token formation including price. market cap. and con- tract details). get getTokenPowe
liquidity pools containing specific tokens for trading analy- sis), getPoolOHLCC (Retrieves historical OHLCV price data essential for backtesting and technical analysis), getPool Transaciions (Fetches recent pool transactions including swaps, additions, movals), search (Cross-netwoek. search functional- ity for tokens, pools, DEXes name, symbal, address), getStars (Provides high-level DexPa- prika ecosystem statistics including cotal networks. DEXes, pools, and tokens)
Fruity Vice
E.Lipas / Lhub,, oun/
Description: Nutritionl information service viding comprehensive fruit nutrition includ. ing vitamins, minerals, calories, and dietary infor- mation. Tools: get fruit notrition (Retrieves de- tailed nutritional information for specilied fruits in- cluding calorics, carbohydrates, protein, fat, sugar, liber, content)
salalhhalloow fuultyvion
Commit
abohskacHbbel www.hh wiw.aa/eeeeaaaaeeee..ee
17

---

Published as a conference paper at ICLR 2026
Table 7 continued from previous page
Server Name
GilHlub Repository
Toools
Description Tools
Game Trends
Etepas/ chubb. 200/
Description: Gaming industry analytics plat- form providing real lime data on game ularity, trends, ties across major gaming platforms. Tools: get ste_sttannreeddnng trendingggamant gamess (Fetches real-time trending games from Steam with live data from multiplc sources), get stcammmmm sellers (Re- top-selling games mlss_playedd form with live sales data), get steum, most, (Gets real-time most played games with live player statisties from SteamCharts) geLepic free troe_games games (Fetches current and up- coming free games from Epic Games Store with prometion details). geLepic geLoopi__tendding games (Retrieves trending games
tall snort ktrt
ktt///no----eeeeeeooorr game- spondo-no RCP
real-time
Commit Hash:
promotionaa
Epic Games Store platform), get all trending games (Provides comprehensive real-time gaming data aggre- gated from all platforms including Steam and Epic). get health (Checks bealth status and availability the Gaming Trend Analyties API) Description: Comprehensive location services platform intograting Google Maps API for geospo- tial queries, place discovery. navigation. and ge- ographic data unalysis. Tools: search nearby (Searches for nearhy places hased location with optional filtering by keywords, distance, rating, and operating hours), get place details (Retrieves detailed information about specific places including contact details, reviews, rar- ings, and operating hours), maps_gescode (Con- verts addresses or place names to precise geo- graphic coordinates with latitude and longitude), maps_reverse rewerseeeeeeeeeeeeee goocode (Converts geographic co- ordinates
Google Maps
Fit Purppss / shgt muub.eoe con/ccaii wtte?
seeg google mag
Commit Hash:
rendable addresses with cation context), maps, distance matrix (Calculates travel distances and durations between multiple origins and desinations for different trunsporta- tian modes), maps directions (Provides detailed turn-by-tum navigation directions between lo- cations with comprehensive route information). maps, elevation (Retrieves elevation showing height above sea level for specific geographic loca- tions)
Huge Icons
Ettpss//et huub com/ ctt/hugcccon//
libeary vice providing access to thousands of high-quality icons with capubilities and platform specific implementation guidance. Tools: list icons (Retrieves complete list of all available Huge- icons with metadata and categories), search, icons (Searches for icons by name or tags using commaa-ecaarted queries for multiple icon dis- covery). get pla_footoom Luaggnn_saaeeea specific usage instructions
Description: Comprehensive
mep- server
Commit Hash: cef68i www.sh/seeeekkaaa000000aaaaaaaa
usage (Provides platform-
implementation de- tails for different development environments)
Hugging Face
Etopo:
10
Description: Al model hub integration service providing comprebensive access machine ing models, datasets, interactivee spaces, reseurch papers, and curated collections. models (Searches Hugging Face Hub for Al mod- els with filtering by task. library. and popular- ity), get-model-info (Retrieves detailed informa- tion about spocific models including architecture, usage, and performunce metrics), seurch-datasets (Searches for machine learning datasets with fil. tering by task type and size), get-dataset-infe (Fetches comprchensive dataset information in- cluding structure, licensing. and usage examples). seareb-spaces (Searches for interactive Spaces ap- plications and demos), get-space-info (Retrieves detailed information ing functionality and source code), get paper-infs (Fetches information about specific research pa- linked to models), get-daily-papers (Retrieves list daily curated research pupers from Hug- ging Face), scacc--collstions (Scarches for cu- rated collections of related models and datasets), get collection (Feiches detailed information about specific collections including contents and curation details)
//glt h/g.t wtw.thooo/ooottttt www/s/oeesssaareeeeeeeeeeee
Fugg sigfwee neg
Commit Hash:
search-
14:00 6302406 Ids440e88f1 fell55
specilic Spaces includ-
18

---

Published as a conference paper at ICLR 2026
Table 7 continued from previous page
Server Name
GilfHub Repository
Toools Description Tools
OSINT Intellgencc
htopss
Description: Open Source Intelligence (OSINT) platform providing comprehensive cybersecurity reconmiissance tools for domain analysis, work scanning. and intelligence gathering, Tools: whois Jookup (Performs domain registration in- formation queries including
7/githue
www//oo/////aeeeeee com/himencchuaaeecn..
nep on cait--erver
Commit Hash:
registrar, and DNS details), amap_ scan (Exccutes network scan- ning port discovery security assessment), dusrrcon, lookup (Conducts DNS reconnaisaance gather
and record information),
dustwist_ Jookup (Anallyzes domain similarity and potential typosquatiing threats). Jookup (Per- forms detailed DNS queries and record analysis), host Jookup (Gathers compeehensiee bost infor- network details), usint, overview (Pro- vides comprehensive intelligence overview and analysis summary)
Medical Calculator
ktapes
22 Description: Comprehensive medical calculation platform providing evidence-hased clinical deci- sion support tools kidney
//gitnub, con/vitt l /nedcc
Commit Hash:
cardiovas cular risk assessment, drig dosing, and special- ized medical scoring systems. Tools: _efi_epi (Caleulates estimated glomerular filtration ing 2021 EPI l'ormula without race adjustment), egft epii_epp.c _r__errre_c.eeeeeeeeeeeeeeeeee (Computes eGFR using combined creatinine-cystee cystttttin C equation for enhanced accu- racy), bp children (Calculates pediatric hlard pres- sure percentiles based on age, height, and der). bmi bsa, calculator (Computes body mass index and body surface mulas), crcl cackeroft gault (Determines creati- nine clearance using Cockeroft Gault formula for drug dosing). map_ cal_ucalc (Caloulates mcan arterial pressure from systolic and
with multiple for-
ues), chads2_ vase. score (Assesses stroke risk in atrial fibeillation patients using validated scor- system), prevent_evd_rin cardiovaseular disense risk in patients aged 30. corrected calcium (Adjusts caleium levels for aboornal albumin conceutrations). qte_ calculator (Corrects QT
(Predicts 10-year
heart rate using multi- ple validated foruulas), wells_pe eriteria (Objec- tilies pulmenary embolism risk using clinical cri- teria). ibw abb calculator (Calculates ideal and adjusted hody weights using Devine formula), pregnancy calculator (Determincs pregnancy dates from last menstrual period or gestational age), revised cardiac risk index (Estimates periopera- tive cardiac complications neucardiac surgery). child, pugh_ score (Assesses cintosis severily and mortality risk), stervid ponversion (Converts he- tween different corticosteroid equivalencies), cal- (Computes
daily morphine mil- ligram equivalents for epioid prescription)) main- tenance, fluids (Calculates pediatric fluid rates using 4-2-1 rule), corrected_ sodium (Adjusts sodium levels in hyperglycemic patients using correction formulas), meld 3 (Calculates MELD 3.0 score for liver transplant priority). framing- ham, score (Estimates 10-year coronary disease risk), homa, ir (Calculates insulin resis- tance using homeostatic model assessment)
Metropolitan Museum
ptepss// httts//gtttti
Deseription: Metropolitan Museum of dig- ital collection access service providing compre- hensive search detailed information about works, artifacts, and cultural objects. Tools: list- departments (Retrieves complete list museum departments with organizational structure). search- museum- objects (Searches museum collection oh- jects with filtering options and total counts), oet-museum-objet (Fetches de- miled informatio by ID inchuding images, provemance, and cultural context)
men muserun- mop
Commit Hash:
63:298800107/77106 777.45/287777777660002ooo
object IDs
specific museum objects
19

---

Published as a conference paper at ICLR 2026
Table 7 continued from previous page
Server Name
GilfHub Repository
TTools
Description Tools
Movie Recommender
bt:pas/ chubb. 200/ con/Lronctt/
Description: Intellieent
movic- novio-Eocom rocommendez-mtp
service providing personalized film suggestions matching content analysis algorithms. Tools: get mov_mss (Generates movie suggestions and recommendations based on uscr- provided keywords with relevance scoring and de- tailed film information)
Commit Hash:
National Parks
Etspss
Service data integration providing comprehensive infor- mation about parks, facilities, alerts, and recre- ational opportunities
Description: National
7/g1 thub con//yr Kaen//yyelleeee eerileTTagg #TangShwngg
server-nd ponttiinnllppp... lpacese
Commit Hash:
the national park sys- tem. Tools: findParks (Searches for national parks based on state,, name, activities, teria with detailed filterin), getaarkDetails (Re- trieves comprehensive information about specific national parks including descriptions. contact info, and amenities). getAlerts (Fetches current park alerts including closures, hazards, and important visitor information), getVisitor (Gets infor- mation visiter centers operating and services), perCampgrounds (Retrieves camp- ground information incloding availability, ameni- ties, and reservation details), getEvents (Finds up- comnng events at parks including programs, tours, and special activities)
OpenAPl Explorer
httpss// /otthub. con/ janwl inake? ogenaapl -mop
Description: Universal API integration platform providing dynamic OpenAPI spacification explo- ration and interaction with various cloud services. social media platforms, developer tools, and en- terprise APIs. Tools: getApiOverview (Get overview of an OpenAPI specification for services including OpenAl, GitHub, Twitte/X, Cloudilare, npm, Slack, Stripe, and many others should be the first stop when working with any APD) callApi (Execute API calls dynumically based OpenAPI specifications with automatic parameter validation and response handling)
Hash: Ro1354999uui000:5777570d::lllld0
20

---

Published as a conference paper at ICLR 2026
Table 7 continued from previous page
Server Name
GilfHub Repository
Toolls Description Tools
NASA
Description: Comprehensive
integra- tion platiorn providing access to astrunomy im- agery, space
7/github.
Commit Hash:
data, planetury information, and satellite observations through official NASA APIs. Tools: aettastrn pictuurppppreeee of_day (Retrieves NASA's daily astronomy picture with explanations and metadara). get_asteros asteroids feed (Fetches
5990c3212220551 728806009966PPP9555555
based dates Earth,, get asteroid lookup (Looks specific asteroids using NASA JPI. system IDs), browss_ asteroids (Browses compre- hensive asteroid dataset with liltering capabilities) get coronal
approach
body
ojection (Retrieves coronal mass ejection data with date range filtering.. get LeL_gennagnetic storm (Fetches geomagnetic storm data with temporal analysis), get, solar, flare (Gets solar
with intensity ceolar_carrgetic__.. (Retrieves solar energetic particle event data), get mwg.mmaaauuuseeccmmm crossing (Fetches magnelopuuse crossing event informa- tion), get radiation beli, enelncenennm (Gets radiation helr enhancement event data), get bieehhsppeeddd___eoooooeee setespppeeeeeeeeee stream (Retrieves high-speed solar wind stream datal, BoLwsa_call evsa_eall_simmllae (Feiches WSA+Enlil solar wind simulation results). get potifficttoo
activity
classifications.
DONKI space weather notifications and alerts), get earth imagery (Retrieves lite imagery for specific coordinates and dates). get carth_assetss (Gets information about available Earth imagery cations). get epic imagery (Fetches images from Farth Polychromatic Imaging Camera), get get_epic_ imagery. by date (Retrieves EPIC images for specific dates). get epic dates (Gets available dates for EPIC image colles- tions). get cel_exxplanet_cccc data (Queries NASA Exoplanet Archive with custom search pa- rameters), get, mars rover, photns (Fetches photos from Mars get mars, rover manifest (Retrieves mission manifests with rover and photo statistics)
satel.
for lo-
get_epic
sol Earth date),
21

---

Published as a conference paper at ICLR 2026
Table 7 continued from previous page
Server Name
GilfHub Repository
TTools Description Tools
NixOS
Description: Compreheusve NixOS ecosystem integration providing package manggement, con- Manager support, macOS nix-darwin compatibility, and community flakes discovery. Tools: nixos aaaecch (Searches NixOS packages, options, programs, flakes with figurable result limits), nixos info (Retrieves de- specilic
//github. com/attnniii nix--nioos
Commit Hash:
figuration options,
39289 19286176 1c421 12
tailed information
packages options with chamnel selection), nixos, channels available
channels tus information), nixos_stee stats (Gets comprehen- sive statistics for NixOS channels including pack- and option counts),
manager search (Searches Home Manager configuration options by name description).
manuger. info (Fetches detailed information ahout specific Home Manager options
exact name matching), home manager stats (Retrieves Home Manager statistics including total options and calegory breakdown), home manager. list, options (Lists Home Manager option categories with counts). munager managerr_ptioons options. by prefix
Home Manager options matching specific prefixes for category browsing), darwin_scarde search (Searches nix- darwin macOS configuration options by name and description), darwin, info (Retrieves de- tailed information about specific nix-darwin op- tions). darwin stats (Getts nix nix-darwin statistics including option counts win lt__list_ppiii options (Lists all nix-darwin option car- egories with counts). darwin options by_ prefixx (Gets nix nix-darwin options matching specilic fixes), nixos, flakes, stats (Retrieves statistics about available NixOS flakes including repositories and contributos), nixes, nixos_faakes search (Searches com- munity NixOS flakes name, description, owner, repository). nixhub_ package versions (Gets version history and nixpkes commit hashes for spe- cific packages). nixhub [ind, version (Finds cific package versions with smart search and in- creasing limits)
categories),
OKX Exchange
Description: OKX eryptocurency exchange gration providing real-time trading data and his- torical analysis for digital assets and trading pairs. Tooks: get price (Retrieves latest price in- formation for OKX trading instruments with time market data), get_ candlesticks (Fetches his- torical candlestick data for technical analysis and price charting)
//github,
Commit
47a0
22

---

Published as a conference paper at ICLR 2026
Table 7 continued from previous page
Server Name
GilHHub Repository
TTools
Description Tools
Paper Scarch
Et-pas/ chubb 200/ oponags/
research platform inlegrating mulliple scholarly databases retrieval, text analysis across diverse scientific disciplines. Tools: search_arite (Searches arXiv prepeint repository with metadata and abstract retrieval), scarch_ pubmed (Searches PubMed biomedical lit- crature database with detailed paper tion), search, biorxiv (Searches bioRsiv biol- egy preprint server with recent research ings), maacch_medrxiv (Searches medRxiv med- ical proprint repository search goagle scholar (Searches Google Scholar across all academic disciplines with citation met- ries), search (Searches
Description: Comprehansive
paper-Searche seasch-mop
Commit Hash:
paper discovery.
clinical research),
ePrint Archive for cryptography and security rescarch), down- load, arxiv (Downloads PDP files from arXiv pa- with local storage), download, pubmed (Ar- templs PDF download from PubMed with ac- limitations notice), download bioxiv (Down- loads bioRxiv paper PDLs with DOI-based re- trieval), downlead medrxiv (Downlouds paper PDFs with automated file manugement), download jacr (Downloads IACR cPrint paper PDFs with paper ID validation), read _aper (Extracts and processes full text content from paper PDEs). read puperd_ppper paper PubMed puper content with direct database ac- cess limitations), read full text from bioRxiv papers with strictured content analysis), read medrxiv medrxiv_paper paper (Processes medRxiv poper text with medical content purs- ing). read iacr paper (Extracts text from IACR papers with cryptography search, semantic (Searches Semantic Scholar with advanced filtering year field), load_ semantic (Downloads papers from Seman- tie Scholar using multiple identifier formats), read semantic puper (Reads and processes Se- mantic Scholar papers with comprehensive text ex- traction)
paper
formatting)
Roddit
Description: Reddit social media platform integration providing acces to community discussions, trending content, and detailed post analysis with comment threading. Tools: fetch reddit_ threads (Fetches trending bot threads from specified subreddits with coutigurable limits), fetch reddit post content (Retrieves detailed post content including comments with traversable comment tree structure and depih contro))
Commit wtwp////t//////ppoooo////// rediis
//github,
23

---

Published as a conference paper at ICLR 2026
Table 7 continued from previous page
Server Name
GilHHub Repository
TTools Description Tools
Scientific Computing
Ettpas/ citbub. 200/
Description: Advanced scientific computing plat- form providing comprehensive lineur algebra op- erations, vector calculus computations, and math- ematical visualization tools with in in-memory ten- sor storage. Tools: create leasor (Creates NumPy armys with specified shapes and val- ues in memory store), view_tensor tensor (Returns im- memory). delete_tensor lensor (Removes lensors from in memory stornge), add matrices (Performs element- ad- dition of two stored matrices), subtract_ matrices (Performs element-wiss wise subtraction of stored ma- trices), multiply matrices (Executes matrix mul- uplication between stered tensors), scale_matrix matrix (Scales tensor by scalar tional in in-place operation), matrix inverse (Com- putes inverse stored square matrices singularity checks), transposc (Computes trans- pose of stored tensors), determinant (Calculates determinant of stored square matrices), rank (Computes matrix rank of stered tensors). com- eigenvec- pule pule_eigen eigen (Calculates eigenvalues tors of square matrices), decompose (Performs QR decomposition into orthogonal and upper tri- angular matrices.. svd decompose (Executes Sin- gular Value Decomposition into S. V compo- nents), find orthonormal basis (Finds orthenorual basis for column space using QR decompositio). change, (Transforms matrix nate basis), vector project (Projects stored vector onto specified target voctor), voctor_dok product (Computes det product between stored vee- tors), vector cross, product (Calculates cross prod net of stored 3D vectors), gradient (Computes sym- holic gradient of scalur functions), curl (Calcu symbolic
Imar- 1m1 ta--taattttpy wtw.s//scici.ic.....
computat Lopputattooon.. Lon_on
Commit Hash:
mutable view stored lemsors
op-
coordi-
vector fields optional point evaluation), divergence (Computes symbolic divergence of vector fields), laplacian (Calculates Laplacian operater for scalar vector fields), di- rectional deriv (Computes directional derivative along specified vector direction), plot_ _ield (Visuulizes 3D vector fields with customizable bounds), plot function (Plots 2D/DD mathematical functions from symbolic expressions)
Time MCP
Ptepes
Description: Time zone conversion and world clock service providing
//gl t/gb. Uhub- calh--oo//iti www../////////////mmm liw mep
informa- tion and conversions actoss different time zones globally. Tools: get current time (Get current time specific timezones using IANA timezone names), convert, time (Convert time between lime- with and target timezane specifica- tions)
Commit Hash:
91D07681 Tecesllccckcooes44777771s
Converter
Et htspps//sg thuls, cuw/s/aeenttiiene
Description: Comprehensive conversion ser- vice supporting multiple measurement categories including lemperature, angle, length, energy, force, pressure, power, speed,
unit- converter-nig Commit Hash: SSludikDa 1a550
volume, data storago, density, time, and batch quantities. Tools: convert lemperature (Temperature conversion be- tween Celsius, Fahrenheit, and Kelvin), con- angle (Angle conversion hetween degrees, ra- dians, and gradians), convert conver_leenth (Length con- version across metric and imperial units), con- energy (Energy conversion including joules, calories, and BTU), convert foree (Force conver- more),
newtons, pounds-force, convert, pressure (Pressure conversion across mul- tiple units), convert power (Power conversion cluding watts and borsepower), convert_speee spoed (Speed conversion between various velocity units), convert_ar
square units), convert mass (Mass and weight conversion), con- vert, volume (Volume comversion for liquids solids). convert computer data (Digital stonge conversion), convert density (Density conversion across different units), convert time (Time dura- tion conversion), convert batch (Barch processing for multiple conversions), convert weight (Legacy weight conversion function)
conversion
24

---

Published as a conference paper at ICLR 2026
Table 7 continued from previous page
Server Name
GilfHub Repository
Toools Description Tools
Weather
Et-pas/7 chubb. 200/ cendeclu/
Description: Comprehensive weather infor- mation service providing current conditions, forecasting,
woather, mcp
Commit Hash:
UbeR (0002220e0000000000000 i8
teorological data for global locations. Tools: gerrentent weather_ jtoteer_toolllloooo (Rettieves current weather information including temperature, condi- tions, humidity, and wind data for specific cities), get weatber forecast tool (Provides weather fore- casts for 1-10 days with detailed meteorological predictions), search, locations locations by name with detailed geographic infor- mation), gev_ (Legucy tool for current temperature retrieval with backward compatibility support)
(Searches for
Wikipedia
htsps://githib zom/ com//udra-ccen pavi/
Description: Comprehensive Wikipodin content sccess and analysis service providing advanced article search, content extraction, and knowledge discovery with stractured data analysis capabili- ties. Tools: search, wikipedin (Searches Wikipedin matching specific queries vance ranking and metadata); get articlle (Retrieves content Wikipedia articles with complete and formatting.. geLsummary (Generates concise article summariees with key information extraction), summarize article for query (Creates query-tailored summaries focusing on specific as- pects articles), summarize article section (Pro- vides focused summaries of specific article sections with contextual information), extract _ facts (Extracts structured key lacts and data points from articles with calegorization) get get_ related, Jopics (Discovers related topics and articles through link analysis and category exploration). get sot_soctions and subsections articles hicrarchical structure), get_linke links (Retrieves all in- temal and external links with link context and rele- scoring)
(i-c-edd----- peedia-mee mep
Commit Hash:
2ade (2w.646kk02222777774433333
Table MCP server rate limitations.
Server Name
Rate Limitation
BioMCP
5000 API calls per hour for part of tools
Car Price Evaluator
500 API calls per day
DEX Paprika
60 API calls per minute
Game Trends
100000 API calls per day
Hugging Face
1000 API calls per minutes
Google Maps
6000 API calls per minute
Reddit
100 API calls per 10 minutes
National Parks
1000 API calls per hour
Metropolitan Museum
80 API calls per second
Movie Recommender
40 API calls per 10 seconds
NASA Data
1000 API calls per hour
OKX Exchange
60 API calls per seconds
Paper Search
20 API calls per 10 seconds for part of tools
Weather Data
Million API calls per month
Wikipedia
500 API calls per day
25

---

Published as a conference paper at ICLR 2026
A.3 DETAILS OF EVALUATED MODELS
In this section, we show the details of the inference parameters (i.e., Temperature and Top_p) for the evaluated LLM models in our benchmark. Detailed parameters be found in Table 9. For OpenAl models, we use them via Azure OpenAl' endpoints. For non-OpenAl models, we access them via OpenRouter' popular model endpoint provider), using its official authentication mechanism and API endpoints. All models are queried through hosted cloud APIs (Azure OpenAl or OpenRouter), so the local computational resources are not required.
Table 9: Inference parameters of models.
Model
Temperature
Top_P
glm-4.5
0.70
0.95
kimi-k2
0.60
1.00
claude-sonnet-4
1.00
1.00
nova-micro-vl
0.70
0,90
mistral-small-2503
0.15
0.90
0.60
0,95
gwen3-235b-:222-2507
0.60
0.95
gemma-3-27b-it
0.70
0.90
gemini-2.5-flash--i
1.00
0.95
gemini-2..-prr
1.00
0.95
Summa-3---b-inntrrcc
0.60
0,90
1iama---2-0b--viioo---i-e-
0.60
0.90
1amma-31--7b-insse
0.60
0.90
llmma-3---70-instruu
0.60
0.90
gpt-4o-mini
1.00
1,00
gpt-4o
1.00
1.00
opt-oss-20b
1.00
1.00
gpt-oss-
1.00
1.00
03
1.00
1.00
gpt-5
1.00
1.00
A.4 DETAILS OF THE USED PROMPT FOR THE TASK EXECUTION AGENT
In this section, we show the detailed prompt used for the task execution agent in MCP-BENCE
Strategic Planning Prompt
Purpose: Strategic preision-makingg and tool planning for multi-round execution
You are strategic decision-making expert for multi-tool Al agent using the provided tools to perform the task.
TASK:
"[task)"
CURRENT ROUND: (round num)
AVAILABLE TOOLS ACROSS SERVERS:
[ool_list]
DECISION AND PLANNING:
1. Assess if the original task is fully completed
not complete, decide if another round would provide significant value
3. If continuing, plan PARALLEL tool executions for this round
http:///ww..ltlttlccoo///////o/r hhtpps//opeenrouter.ai/
26

---

Published as a conference paper at ICLR 2026
PARALLEL EXECUTION PLANNING (IF CONTINUING):
Plan ALL tool calls for this round to execute PARALLEL
ALL tools in this round will rum simultaneously without dependencies
EARLY EXECUTION PRINCIPLE: Plan all necessary tool calls that don't 't require de-
pendencies
AVOID REDUNDANT CALLS: Don't repeat successful tools unless specifically needed
BUILD ON PREVIOUS RESULTS: Use information from previous rounds
FOCUS ON INDEPENDENT TASKS: Plan tools that work with currently available information
REQUIRED JSON RESPONSE FORMAT:
"<Detailed explanaticn for your decision and parallel execution plan>", "planned_tools";
"reasoning";
true/false,
"tcol": "server:tool_nmme" "parameters": "param": "value value"
27

---

Published as a conference paper at ICLR 2026
Final Solution Generation Prompt
Purpose: Generate multi-round execution results into final comprehensive answer
You are an expert solution synthesizer for multi-tool Al agent execution.
ORIGINAL TASK: "(task)"
A multi-round execution process has completed with across multiple MCP servers.
total tool calls
ACCUMULATED INFORMATION AND RESULTS: fnccumulated_inoormaaioooo
TASK REQUIREMENTS:
Based on the original task and all the information gathered from multiple servers, provide a final, comprehensive, and well-structured answer that directly addresses the user's request.
Synthesize the key findings and present them clear, organized manner that shows how the different server capabilities were combined.
SYNTHESIS GUIDELINES:
Extract and consolidate key information from all execution rounds
Highlight how different tools and servers contributed to the solution
Present findings in logical, structured format
Address all aspects of the original task
Provide clear, actionable conclusions where appropriate
Content Summarization Prompt
Purpose:
Compress large execution results to reduce token usage
You are a helpful assistant. need your help to extract key information from content.
Summarize the following content to less than (threshold) tokens while preserving all impor- lant information:
CONTENT: (content)
SUMMARIZED CONTENT:
SUMMARIZATION REQUIREMENTS:
Preserve all critical findings and results
Maintain factual accuracy
Keep important numerical data and specific details
Remove redundant or verbose explanations
Focus on actionable information
28

---

Published as a conference paper at ICLR 2026
A.5 DETAILS OF THE USED PROMPT FOR TASK SYNTHESIS
In this section, we show the detailed prompt used for task synthesis in ACC-BBENC.
Task Generation Prompt
Purpose: Generate complex tasks with deep tool dependencies
You are a task designer for testing Al agents with MCP tools.
STEP ANALYZE AND CREATE TOOL DEPENDENCIES
Analyze these available tools and CREATE meaningful dependencies for your task scenario:
(tool_ (tool_ [eoo__eecripions
Consider both:
A) IHHERENT dependencies (tool's natural input/output relationships)
.hhich tools naturally produce data others consume
Standard workflow patterns (search - fetch analyze)
B) SCENARIO-BASEL dependencies (create logical connections for your task), for ex-
ample:
Tool A's result determines WHICH tool to next
Tool B's output sets PARAMETERS for Tool C
Tool D validates or contradicts Tool E's findings
Parallel tools whose results must COMBINED
Iterative loops where results trigger RE-ANALYSIS
Record your dependency analysis Key tool chains and data flow
arpendency_analysiie field that describes:
Critical decision points
Parallel vs sequential requirements
Cross-server dependencies (for multi-server tasks)
For multi-server tasks [[server_name) )), create CROSS-SERVS dependencies:
Server data influences Server queries
Cross-validation between different data sources
One server's limits trigger fallback to another
STEP 2: DESIGN ONE COMPLEX TASK
Based on your dependency analysis, create ONE task that:
Create MAXIMUM complexity requiring massive tool calls
Must use tools from all available servers
Must consider inter-servers dependency if more than
server available
You may create the tasks with the following properties if suitable:
Deep dependency chains where Tool B needs Tool A's output, Tool C. needs B's output, etc.
Multiple decision branches based on intermediate results
Iterative refinement: initial findings lead to deeper investigation
Cross-validation: use multiple tools to verify critical findings
Data transformation: output from one tool needs processing before next
Conditional workflows: condition then workflow Y, else workflow Z
29

---

Published as a conference paper at ICLR 2026
CRITICAL DATA REQUIREMENTS:
ALL tasks MUST be self-contained and executable WITHOUT any external depen- dencies
2. NEVER reference external resources like:
URLs (like wttpps://pp..c.mppl.... or any external API)
Local files (like "user-managemmett.yy
"config.ison")
Databases or external systems
"Our API", "our system", "our database"
3.
ALL data must come from either:
The provided tools themselves (what they can gencrateefiteehcaalllla))
Concrete values you specify in the task (numbers, names, parameters)
4. NEVER use vague references:
'user-provided parameters" or "user-specified"
"fetched from database" or "retrieved from external source"
"based on user preferences" or "according input"
"specified location/value" or "to be determined"
5. ALWAYS provide concrete values:
Specific numbers (e.g.. "analyze heat exchanger with inlet temp 80CC outlet 60CC, flow rate 0.5 kg/s")
Named entities (e.g., "analyze weather in San Francisco" not "specified city")
For locations: Use city names, landmark names, or general areas, NOT specific street addresses
"San Francisco", "Times Square", "Central Park area", "downtown Seattle" BAD: "123 Main Street", "456 Park Avenue", specific house numbers or street ad- dresses
GOOD:
Exact thresholds (e.g., "alert efficiency drops below 85%" not "desired threshold")
ALWAYS USE relative dates/times (c.g.. "next days", "past 3 months", "upcoming week" not "January 2024" or "2024-01-15")
6. If the task involves analysis, provide ALL input data in the task description:
For calculations: provide all numbers, formulas, and units needed
For searches: provide specific search terms and criteria
For comparisons: provide specific items with their properties
For optimization: provide current values and target metrics
REQUIREMENTS:
MUST require multiple tools in specific sequence
Tool B should need output from Tool A (dependency chain)
3. Include decision points based intermediate results
4. Be realistic and valuable for business/research purposes
5. Define expected analysis and output format
6. Task must be immediately executable agent should never need to ask for more informa- tion
7. Task should be executable and solvable by using the provided tools. You need pay attention to the function and the output of the provided tools.
30

---

Published as a conference paper at ICLR 2026
OUTPUT FORMAT:
Output ONLY a JSON object (not an array). ALWAYS USE relative dates/times:
"task_id"; "task_ xXx",
"task_aescription": "detailed task that Leverages the identifien tool dependencies", "dépendeency_aallssis "Your analysis from STEP 1 describe the key dependencies, tool chains, decision points, and data flow patterns that this task zequires"
Focus on creating task that CANNOT be completed without understanding tool dependen- cies.
Task Quality Assessment Prompt
Purpose: Evaluate task quality on solvability and utility dimensions
Evaluate this task's quality on two dimensions:
Task Description: [task_ |
Fuzzy Description (what the agent sees):
Available Tools: )
EVALUATION CRITERIA:
SOLVABILITY (1-10):
10: All required data is provided, tools perfectly match needs, clear success criteria
8-9: Task is clearly solvable with the given tools, minor ambiguities acceptable
6-7: Mostly solvable but some steps may be challenging or unclear
4-5: Significant gaps in tool coverage or data requirements
1-3: Task cannot be meaningfully completed with available tools Consider:
Are all necessary tools available?
Is all required data provided (no external dependencies)?
Can the agent achieve the stated goal with these tools based on the function and output of the tools?
Are success criteria clear and measurable?
2. UTILITY (1-10):
10: Critical business/research value, addresses real-world problem perfectly
8-9: Strong practical value, useful for decision-makee making operations
6-7:
Moderate value, interesting but not critical
4-5:
Limited practical value, mostly academic exercise
1-3: Trivial artificial task with real-world application Consider:
Does this address a real business or research need?
Would the results be actionable and valuable?
Is the complexity justified by the outcome?
31

---

Published as a conference paper at ICLR 2026
Does it test meaningful agent capabilities?
OUTPUT FORMAT:
Provide scores and brief feedback in JSON format:
"solvability sccreeeittyycrr...oooooooo <number 1-10>,
"utility "utility_scored score": <number 1-10>, "solvability_feedbeet "Brief explanation of solvability assessment", "utility_feedback" "Brief explanation utility assessment"
Task Description Fuzzing Prompt
Purpose: Convert detailed tasks into natural, conversational user requests
Convert this detailed task into a NATURAL, CONVERSATIONAL USER REQUEST that truly tests the agent's reasoning ability.
Original detailed task:
[detailed_tase
Available tools: (len(tools)) tools (but don't mention them in the fuzzy version)
CRITICAL: CREATE A GENUINELY NATURAL REQUEST
ABSOLUTELY FORBIDDEN:
ANY structured language that looks like task description
Phrases like "I need to analyze", "I want to compare", "Please evaluate"
ANY specific server/platform names (arXiv, PubMed, Yahoo Finance, Google Maps, etc.)
ANY tool names technical implementation details
Lists, enumerations, or step-by-step instructions
Formal task language ("perform". "conduct", "execute", "implement")
INSTEAD, CREATE A NATURAL CONVERSATION:
Start with context or problem the user is facing
Use conversational openers: "I'm m trying to figure out...", "Been wondering about...", "Got situation here..."
Include uncertainty: "not sure if", "maybe", "possibly", "might be"
Add personal context: "for my project", "my boss asked", "I'm curious about"
Express the need through story or scenario, not task list
HIDE THE TASK STRUCTURE COMPLETELY:
Don't say: "I need analyze financial metrics for AAPL,, GOOGL, and MSFT"
Say instead: "I've been thinking about rebalancing my portfolio and I'm curious how tech giants like AAPL, GOOGL., and MSFT have been doing lately. Which one would you say looks strongest right now?"
Don't say: "Search for recent papers CRISPR and summarize the key findings" Say instead: "I'm giving a presentation next week about gene editing. What's 's the latest buzz around CRISPR? Any breakthrough discoveries should know about?"
Don't say: "Calculate the thermal efficiency and optimize the heat exchanger parameters" Say instead: "We've got this heat exchanger running at 80 inlet, 60 outlet with 0.5 kg/s
32

---

Published as a conference paper at ICLR 2026
flow. It doesn't seem very efficient to me. Can you help me figure out what's going on and maybe how to improve it?"
PRESERVE CRITICAL DATA NATURALLY:
Embed specific values conversationally: "around 150 or so", "somewhere near San Fran- cisco"
Use approximate language when appropriate: "roughly", "about", "somewhere between"
Keep exact values only when truly necessary (calculations, IDs, etc.) [calculation [caleulation_requirrennn requirements
MAKE IT SOUND LIKE A REAL PERSON:
Use contractions: "I'",, "don't", "can't", "what's"
Include filler words sparingly: "actually", "basically", "you know"
Show emotion urgency when appropriate: "really need to know", "been bugging me"
Ask questions naturally: "What do you think?", "Does that make sense?". "Am over- thinking this?"
EXAMPLES OF NATURAL FUZZY DESCRIPTIONS:
Example 1 (Finance):
"So I've been watching my tech stocks lately and honestly, I'm not sure if should hold or sell. AAPL, GOOGL, and MSFT make up most of my portfolio. With everything going on in the market, which one do you think has the best outlook? I'm particularly worried about their debt levels and cash flow situation. Need some real data to back up any decision here, not just gut feelings."
Example 2 (Research):
"I'm preparing for a journal club presentation and everyone's been talking about these new CRISPR developments. What's actually new in the past few months? keep hearing about off-target effects being solved but can't 't find solid evidence. Would really appreciate concrete findings from recent studies, not just hype."
Example 3 (Technical):
"We're re having issues with our heat exchanger setup running at 80CC in, 60CC out, flow rate's about 0.5 kg/s. My manager thinks we're wasting energy but I need prove it with actual numbers. Can you work out what our current efficiency is and maybe suggest what parameters we should tweak? Need solid calculations to convince them to approve changes."
CRITICAL: END NATURALLY WITH EVIDENCE REQUIREMENTS WOVEN INTO THE
CONVERSATION:
Instead of: "Please provide evidence-based analysis with concrete data" Say: "I really need actual data on this can't go to my boss with just opinions. Whatever you find, make sure it's backed up by real numbers or solid sources, okay?"
ALWAYS USE relative dates/times (e.g., "next days", "past 3 months", "upcoming week" not "January 2024" "2024-01-15")
Return ONLY the natural, conversational fuzzy description make it sound like a real person asking for help, not robot executing task.
A.6 DETAILS OF LLM JUDGE
In this section, show the detailed prompt used for the LLM judge in our benchmark.
33

---

Published as a conference paper at ICLR 2026
LLM Judge Prompt
System Role:
You are an impartial evaluator judging the quality of an Al agent's multi-server tool-based task execution.
User:
You must assign scores only based on evidence from the task, solution, and tool usage. Your evaluation should be:
Objective (avoid being influenced by language fluency or formatting)
Justified (include specific reasons tied to each score)
Robust against bias (ignore narrative style, verbosity, or formatting polish)
TASK PRESENTED TO AGENT: " "(task)"
CONCRETE TASK REFERENCE (For evaluation context only):
Note: The agent did NOT see this concrete version. It only saw the task above. The task visible for the agent is the fuzzy version of the concrete task. The agent's interpretation of the fuzzy task may differ but still be valid. "(concrete task description)'
DEPENDENCY ANALYSIS:
Note: This analysis was generated during task creation to help understand tool dependencies. The agent did NOT see this analysis. is provided as reference for evaluation purposes. dkepeddncy_annalyee
FINAL SOLUTION: ((final_solution)
TOTAL ROUNDS: Kcoaal_rounde
EXECUTION SUMMARY:
[execution_ summmorrymmmr
AVAILABLE TOOLS ([num_tools tools):
ciiilable_toolss
TASK COMPLETION RUBRIC (1-10 PER SUBDIMENSION)
1. Task Fulfillment
1-3: Perfectly completes 10-30% of requirements.
4-6 Perfectly completes 40-60% of requirements.
7-8: Perfectly completes 70-80% of requirements.
9-10: Perfectly completes 90-100% of requirements.
2. Grounding
1-3: 10-30% of claims are perfectly grounded tool outputs.
4-6 40-60% of claims are perfectly grounded in tool outputs.
7-8: 70-80% of claims are perfectly grounded tool outputs.
9-10: 90-100% of claims are perfectly grounded in tool outputs.
TooL. USAGE RUBRIC (1-10 PER SURDIMENSION)
1. Tool Appropriateness
1-3: 10-30% of tools were perfectly selected for their subtasks.
4-6: 40-60% of tools were perfectly selected for their subtasks.
7-8: 70-80% of tools
perfectly selected for their subtasks. 9-10: 90-100% of tools were perfectly selected for their subtasks.
2. Parameter Accuracy
1-3: 10-30% of tool calls have perfectly accurate and complete parameters.
4-6: 40-60% of tool calls have perfectly accurate and complete parameters.
34

---

Published as a conference paper at ICLR 2026
7-8: 70-80% of tool calls have perfectly accurate and complete parameters.
9-10: 90-100% of tool calls have perfectly accurate and complete parameters.
PLANNING EFFECTIVENESS AND EFFICIENCY RUBRIC (1-10 PER SUBDIMENSION)
Dependency Awareness
1-3: 10-30% of dependency chains are perfectly executed.
6-6 40-60% of dependency chains are perfectly executed.
7-8: 70-80% of dependency chains are perfectly executed.
9-100 90-100% of dependency chains are perfectly executed.
2. Parallelism and Efficiency
1-3: More than 70% redundant calls OR less than 30% of parallelizable tasks were executed in parallel.
6-: 40-60% redundant calls OR 40-60% of parallelizable tasks were executed in parallel.
7-8: 20-30% redundant calls AND 70-80% of parallelizable tasks were executed in parallel.
9-10: Less than 10% redundant calls AND 90-100% of parallelizable tasks were exe- cuted parallel.
PERCENTAGE-BASED SCORING SYSTEM:
How Calculate Scores:
For each dimension, calculate the DEFECT RATE:
Defect Rate = (Number Issues Total Opportunities) x 100%
Then map defect rate
0-10% defects + Score 9-10 (Excellent to Perfect)
10-30% defects - Score 7-9 (Good performance)
30-50% defects Score 5-7 (Average performance)
50-70% defects + Score 3-5 (Poor performance)
70-100% defects + Score 0-3 (Failed)
How to Score:
1. When evaluating percentages, be EXTREMELY STRICT about what counts as "perfectly executed"
2. "Perfectly" means ALL of the following must be true:
Correct tool selection (not just "works" but OPTIMAL choice)
Complete and accurate parameters (not just valid, but IDEAL)
Zero redundancy (no repeated or unnecessary calls)
Proper eтror handling (graceful recovery from ANY failure)
Efficient execution (parallel when possible, minimal rounds)
Concise output (no verbose explanations unless requested)
3. If ANY of the above is missing, that portion is NOT perfectly executed (counts as 0%)
4. Example: Task completed correctly but with redundant call = that portion is 0% perfect
KEY PRINCIPLES:
1. ALWAYS calculate as percentage, NOT absolute numbers
10 errors in 100 calls (10%) = same score as error in 10 calls (10%)
3. Consider the OPPORTUNITY COUNT for each dimension:
Tool calls: How many total calls were made?
Parallelization: How many tasks COULD have been parallel?
35

---

Published as a conference paper at ICLR 2026
Parameters; How many total parameters across all calls?
Claims: How many factual statements were made?
Dependencies: How many dependency relationships exist?
4. NORMALIZE by complexity don't punish complex tasks:
Simple task: error/5 steps (20% defect) Score 7
Complex task: 4 errors/20 steps (20% defect) Score
CRITICAL: Apply the STRICTEST interpretation of "perfectly executed". there's ' ANY doubt, score lower.
CONCRETE SCORING EXAMPLES WITH PROPORTIONS:
Task Fulfillment:
Completed 19/20 requirements (5% defect rate) = Score
Completed 16/20 requirements (20% defect rate) Score
Completed 1/22 requirements (40% defect rate) Score
Completed 8/20 requirements (60% defect rate) Score 4
Tool Appropriateness:
19/20 tools optimal (5% defect rate) Score 9
16/20 tools optimal (20% defect rate) Score 8
12/20 tools optimal (40% defect rate) = Scorre
8/20 tools optimal (60% defect rate) = Score 4
Parallelism & Efficiency:
9/10 parallelizable tasks done in parallel (10% missed) = Score 9
8/10 parallelizable tasks done in parallel (20% missed) Score 8
6/10 parallelizable tasks done in parallel (40% missed) Score 6
4/10 parallelizable tasks done in parallel (60% missed) Score 4
Grounding:
19/20 claims supported by evidence (5% unsupported) Score
16/20 claims supported by evidence (20% unsupported) Score 8
12/20 claims supported by evidence (40% unsupported) Score 6
8/20 claims supported by evidence (60% unsupported) = Score
Parameter Accuracy:
95/100 parameters perfect (5% defect rate) Score
80/100 parameters perfect (20% defect rate) Score 8
60/100 parameters perfect (40% defect rate) Score 6
40/100 parameters perfect (60% defect rate) Score 4
FORMAT NOTE: Text output when JSON not required = NO PENALTY (0% defect) Missing JSON when explicitly required Count failed requirement Remember: Most real-world executions should score 4-6. Scores of 8+ should be EXCEP-
FORMAT NOTE:
TIONAL.
FINAL REMINDER BEFORE SCORING:
Default to 4-5 unless you have strong evidence for higher
Count ONLY truly perfect executions toward the percentage
Be your most critical self find flaws first, then acknowledge successes
If you's re considering
above 7, re-examine for ANY imperfection Server count is IRRELEVANT using more servers is NOT better
36

---

Published as a conference paper at ICLR 2026
CRITICAL EVALUATION REQUIREMENTS:
You MUST map each score to the exact percentage ranges the rubrics.
2. Task Completion and Tool Selection MUST be evaluated against the CONCRETE TASK REFERENCE, not the fuzzy task.
3. Planning Effectiveness should be evaluated based on the PROPORTION of dependencies correctly handled, not the absolute number of steps executed exact conformance the dependency analysis.
First calculate the actual percentage of completion/suuceess, then assign the corresponding score range.
5.
IMPORTANT: Focus on completion RATIOS not absolute numbers completing 7/10 steps (70%) should score similarly to completing 14/20 steps (70%), regardless of task complexity.
Please score based on COMPLETION PERCENTAGES and PROPORTIONAL SUCCESS, not absolute numbers of tools called or steps executed. Return your evaluation scoring and reasoning in this exact JSON format:
"task fulfillment reasoning": : "Explain how well the agent fulfilled the detailed task objectives, referencing specific content from the CONCRETE TASK DESCRIPTION and what percentage was completed.", "groundinggreascnii
"Explain hew well the agent's outputs were grounded in actual tool results versus unsupported claims.", "tool_ape aapp__apprprrieeeess...... reasoning"; "Explain whether the tools "erriiateeessrrssniinnng selected were appropriate for each subtask requirement.", "Explain the accuracy and completeness of paraneters used in tool calls, noting any missing required paraneters or incorrect values.",
(lapeedency_Lavrreees___. reasoning":: "Explain how well the agent understood and respected task dependencies (what percentage of dependencies were hand" ed correctly), refer to the provided dependency analysis section.",
efficiency reasoning": "Explain the efficiency execution, including use of parallelism and avoiding redundancy, refer to the provided dependency analysis section
"task_ "task_fulfill fu X, "grounding": X,
"tsol_ppproriittnness" "paraneter__ccuraay"" X,
"dépendency__werrnness": X, "parallelism innrallllllmmmmdeeeffieeeeen
Return only the JSON object.
37

---

Published as a conference paper at ICLR 2026
A.7 EXAMPLES OF THE INPUT SCHEMA FOR TOOLS INVOLVED.
Input Schema Example 1: Blood Pressure Percentiles in Medical Calculator
Tool: wt_pbpp_hhloeeeee children
Input Schema:
"type": "objec=", "properties": "years"; "type": "integer",
"minimum"; 1, "maximum": 17,
"description": "Age in years"
"months":
"type": "integer", "minimum"; 0,
"maximum": 11, "éescription"; "Additional months of age"
"height":
"type": "integer", "minimum": 50,
"maximum"; 250, "description"; "Height in centineters"
"sex": {
"type": "string", "enum": ["male", "female"),
"description": "Patient gender"
"systolic"; "type": "integer", "minimum": 50, "maximum": 250, "description": "Systolic blood pressure in mmHgg
}
"diastolic":
"type": "integer", "minimum"; 30, "maximum"; 150, "dcscription": "Diastolic blood pressure in mmHg"
}
reeuui red": ["years", "months", "height", "sex", "systolic", diastolic"]
Input Schema Example 2: Murti-parameter eGFR Kidney Function Calculator
Tool: egfr_epi_cr_oo Input Schema:
"type": "object", "properties";
"scr": {
"type": "number", "minimum"; 0.1,
38

---

Published as a conference paper at ICLR 2026
"maximum": 50.0,
mmultiple0""; 0.01,
"description": "Serum creatinine level in mg/dl, (0. (0.1-50.0)
sccss: "type": "number",
"minimum": 0.1,
"maximum": 10.0,
"multiplcof"; 0.01, "description"; "Serum cystatin c level in mg/L (
"age": { "type": "integer", "mi "minimum" 18, "maximum": 120, "description": "Patient age in years (18-120)*
mmaee" "tpee" "boolean",
description": "True if patient is male, False if female"
"required": ["scr", "scys", "age", "male"), Poopttiiealrroertieess false
Input Schema Example 3: Tensor Creation Scientific Computing
Tool: create_tensor Input Schema:
"type": "object", "properties"; "shape": "type": "array",
"items": { "type": "intsger",
"minimum": 1,
"maximum": 10000
"minItems": maxItems" 10, ddscrippionn "Tensor shape as list of integers (max 10 dimensions)
vvaluss"
"type": "array", "items": { "type": "number"
"minItems"; 1,
maax tems": 1000000, "éescription"; "Flat list of floats to fill the tensor"
"name":
"type": "string", "pattern": "^[a-zA-2] [a-zA-20-9_1ss "minLength"; 1, "maxLength"; 50,
39

---

Published as a conference paper at ICLR 2026
"description"; "Variable name (alphanumeric with underscores, starts with letter)
},
"required": ["shape", "values", "name"], "adistiooalPPppprrii"" false
Input Schema Example 4: Matrix Basis Change in Linear Algebra
Tool: beange_basing
Input Schema:
"type": "objec=", "properties";
"name": {
"type": "string",
"pattern": "^ "M[a-ZA-2] (a-zA-20-9_144
"description": "Name of matrix in tensor store"
"new_basis""
"type": "array",
"items": {
"type": "array", "items":
"type": "number"
"minItems": 1, "maxItems"; 1000
"minItems": 1,
maxItems": 1000,
description": "2D array where columns basis vectors"
"required"; ["name", "new_basis" "nex_basis"), ooddtiinnlProperteees false
Input Schema Example 5: Multi-Domai Search in Biomedical Research
Tool: srricle_searcher
Input Schema:
"type": "object", "properties"; "chemicals";
anyoo":
("type": "array", "items": ("type": "string", "minlength
": ))),
("type": "string", "minlength"; 2),
("type": "null"]
1,
"description": "Chemical/drug nanes to search for"
"genes":
40

---

Published as a conference paper at ICLR 2026
"anyoi": 1
"type": "array", "items": {
"type": "string", "pattern": [A[2] (A-20-9145",
"minLength": 2, "maxLength": 20
("type"; "string", "pattern"; "A [A-2] [A-20-9] ), ("type": "null")
1,
"description"; "Gene symbols (uppercase a phannumeric)
variants"; "anyOf":
"type": "array", "items": { "type": "string", "pattern"; htt.aahhh.......mmm....aaaaaa--. A-2x]?s"
b ("type": "string"), ("type": "null")
1, "descriptiin feescription: "Genetic variants (HGVS notation)
"include__a srieludd_preprintss::
"type": "boolean",
"default"; true, *dcscription"; "Include preprints from bioRxlv/medRRiv"
"page": "type": "integer", "minimum": 1,
"maximum": 1000,
*default"; 1,
"description"; "Page number (1-based)
"page_size";
"type": "integer", "minimum": 1, "maximum": 100,
"default"; 10, "description"; "Results per page"
"additionalPrppprties"" false
A.8 DETAILS OF THE TASKS
In this section, we demonstrate more examples of the tasks in MCP- (Table 10) and list the combinations of the servers to construct the tasks (Table 11).
41

---

Published as a conference paper at ICLR 2026
Table 10: More examples of task in ACP-BENCH.
Servers & Tools
Task Description
Servers: Google Maps, Weather Data, National Parks
Hey there-I'm gearing up for a quick three-day camping get- away to Yosemite from San Jose and, to be honest, I'm feeling bit swamped by all the options and details. I'd love zero in on the three best campgrounds that actually have real comforts think showers, drinking water, maybe even Wi-Fi are definitely open on my dates and aren't under any alerts or closures right now. Once I've got that shortlist, can you help me figure out roughly how far and how long it takes drive from San Jose each af those spots? I'm planning to settle into one as my "base camp," so for that primary site it'd be great to know the nearest visitor center's hours and exactly how to get there-like turn-by-turn directions, plus the distance and travel time. Also, what's 's the elevation at that main campground? Since want to pack smart, really need a solid three-day weather outlook for Yosemite- saaemite-ntthing vague, just the highs, lows and general conditions for the next few days. And, just in case I run out of snacks cooking supplies, is there grocery convenience store within about five kilometers of that first camp- ground?
Useful Tools: findParks, getParkDetails, getAlerts,
getCampgrounds, getVisitorCenters saas__eocodee
maps_ lii_ddittaeemmmae
maps_diress pip__direc,ions, maps_ eelevatii, search nearby.
get weather forecast tool, getEvents, reverse geocode,
maps_reverse
get_ppacee_dese details, get__errent serrenttwwather_too
't just wing this trip, so any real numbers or solid reference points you can dig up would be awesome no vague guesses, please. Thanks! Please ensure all findings are supported by concrete data and verifiable sources. I need specific numbers and evidence, not generalizations.
Servers: Hugging Face, Paper Search, Wikipedia
I' I'm working on project where need to pick the very best news- artiele classifier out there right www-spccffically the one built for that 4-category news dataset (world, sports, business, tech). My boss wants me to find publicly available, open-source model that has the highest Fl score, and then see if any fresh paper from the last three months has pushed the bar another 5 percentage points higher. If recent research write-up really beats the community model by at least 5 points in F1, I'd like to know what architectural tweak or training trick they used so can apply it to the top model we found. If not, we 'Illl just roll with that open-source champion Also, need quick, Jagg-SEnglish refresher what micro- averaged Fl score actually means how it's calculated- got to explain it clearly to stakcholders. Could you dig into this for me, pull together the model ID and its reported FI, track down any pa- per from roughly the past three months with its own F1, compare them, and then recommend next steps? Really need solid num- bers and clear references so I'm not just guessing. Thanks! Plcase ensure all findings are supported by concrete data and verifiable sources. I need specific numbers and evidence, not generalizations.
Useful Tools: search-models, set-model-info, search-datasets, search, arxiv, download arxiv, read arxiv paper, search pubmed, search wikipedia, get article, get_ summary,,, extract_ke berect_ke_fffoee facts, search-spaces, get-space-info search_biorxiv, download biorxiv, read_ biorxiv_paper, get_sections, get_ links
42

---

Published as a conference paper at ICLR 2026
Table 10 continued from previous page
Servers & Tools
Task Description
Servers:
Google Maps,
National
I've been itching to head of Denver for 5-day camping trip sometime in the next week, but I'm kind of torn on which national park makes the most sense. Ideally it's no more than about a 200 km drive, offers solid hiking and camping, and has visitor center where I can catch any talks events going that week. I'm also really curious about spending nights at camp spots that vary in clevation- maybe one high ridge, one mid-level meadow and one lower valley-just to see how the landscape and weather change. On top of that, I don't want to be stuck cooking at every stop, it'd be awesome and where can grab a good meal-not just any greasy spoon, but something rated at least four stars, and need to know how long the drive is and exactly how to get there. In the middle of the trip I'd like to base myself to break things up dive into any ranger-led programs. Could you put together a day-by-day itinerary for the upcoming week that docs all of that-picks the hest park within reasonable drive from Denver, highlights three campsites that maximize elevation differences, flags any alerts events happening. finds the nearest town restaurants with ratings and drive times, and then lays out morring//fternoon//eeeerr
Parks
Useful Tools: findParks,
getParkDetails, getAlerts, getEvents, gelCampgrounds, gettitsoorentsrss, maps geocode,
geoscdde
get_place_dctaie search_nearby, maps_ caps_directtons maps_ hipps_dittnne_____ matrix, maps_ peps_eleettion
know what town is nearest each campsite
visitor center for couple of nights
plans for each of the five days? really need actual data on this this-can't go wandering off with just vague advice. Whatever you find, please back up with real numbers solid sources, okay?? Please ensure all findings are supported by concrete data and verifiable sources. need specific numbers and evidence, not generalizations.
Servers: NixOS, Context7 Useful Tools: nixos_ nixos_scarch,
I've been banging my head trying to get a Flask-based web app running in totally reproducible way across our team's 's setups. We need Python 3.10, Flask itself, Redis, and Docker all coming from the same Nix channel (we're we'ree 25.05), plus config snippets that play nicely with Home Manager on Linux laptops and nix- darwin on macOS. On top of that, my lead wants a tiny excerpt- like 500 words or on how Flask routing works to stick in our README. What would really help is if you could pull to- gether: quick snapshot
nixos_info, nixos_ ceos__channels
nixos_stats, home_ aamme_maaager_scrreee
home manager info, home manager stats, home manager list options, -aaaanrr_pptions br_pfef, darwin_scaren search, sarwin_iiffo darwin_ _tatts,t,sssssstsstsssstss darwin ooption,, darwin options_ by by_prx,, nixos flakessstall stats,
the 25.05 channel (how big broadly speaking) e The exact Nix package names and versions for python3, flask, redis, and docker, ideally with the commit or revi- sion that pins them The main Home Manager options we should for Python and Docker, with their descriptions The equivalent nix- nix-darwin settings so my mac-using teammate can just drop them in a Whether there's a Poetry flake out there we can lean on (or a note if none exist) * And finally, about 500 tokens' worth of official Flask routing does so can paste it straight into our project guide If you could wrap all of that up as single JSON can hand off to my team, that'd save me hours of guesswork- and give me the hard data I need to prove this setup will actually work everywhere. Thanks! Please ensure all findings are supported by concrete data and verifiable sources. I need specific numbers and evidence, not generalizations.
nixos_ flakes search,
nixhub_ package_versionss, versions, nixhub find version, search context, context_eent
43

---

Published as a conference paper at ICLR 2026
Table 10 continued from previous page
Servers & Tools
Task Description
Servers:
Metropolitan
Museum,
I'm putting together small art-history spotlight on seating in New Kingdom
Wikipedia
what's on view the Met-and I'm a bit stuck on how to pull everything together. My professor wants me to pick out an example piece from the Met's Egyptian section, but I'm not find chairs with pictures their collection. Once have a few candidates, need to know their dates (make sure they're 're really New Kingdom) and exactly what they re made of. If there aren't 't enough chairs, I might have slip in stool footrest to hit at least examples, and then choose the one with the most elabo- rate materials as my main focus. Afler that, have to see what Wikipedia says about Ancient Egyptian furniture-grab the arti- cle summary, pull out the top five insights specifically about New Kingdom pieces, and boil down the "Construction and materials" bit into a quick blurb. It'd also help know handful of related topics could mention for extra context. Finally, need to check if my chosen Met object uses any materials that don't show in those Wikipedia facts- could be neat anomalies to point out. I really need actual Met IDs, image links, periods, materi- als lists, the Wikipedia summary, key New Kingdom facts, that short construction/nattrial paragraph, related topics, and a note on any unmatched materials. Can you help me track it all down? can't go to my professor with guesses gotta have real data solid sources. Please ensure all findings are supported by concrete data and verifiable sources. need specific numbers and evidence, not generalizations.
Useful Tools:
search- museum-objeets,
get-museum-ooject, dist-eppartments, search wikipedia, get article,
sure what they call that department
how
_o__summayy, get_sections, get_links, cet_rreated__opeee toopics extract_key key_fts, cummarize_artichee fitticlessccton, summarize, article for query
Servers: Scientific Computing, Math MCP
I'm working on a mini portfolio analysis for a class project and could use some help untangling the math. I've got three assets with expected returns of 0.08, 0.12 and 0.10, and estimated their covariance matrix as:
Useful Tools: compute_cigest eigen, svd_ decompose, determinant, rank, matrix inverse, create tensor, multiply matrices, gradient, add, multiply, sum, mean,
[0.04 0.006 0.014 0.006 0.09 0.02 0.014 0.02 0.16]
When peeked at the determinant, I worried it might be zero or really small, so I thought I might gently bump the whole matrix by 0.1% % until it's safely nonzero. After that, I'd like to get its eigenvalues and eigenvectors, figure out the largest and smallest eigenvalue, and compute the condition number. If it turns out to be over 100, I'll need to go the SVD route and build a pseudoinverse; otherwise regular inverse should do. Once I've got whichever inverse appropriate, I want multiply it by the return vector [0.08, 0.12, 0.10] to see what portfolio weights pop out. I'm also curious to project the return vector onto the principal eigenvector (the one tied to the biggest eigenvalue) and then verify weights sum to by dotting them with [1,1,1] Could you walk me through The
vector_dot dood_ptoduuttttooooooooooeeeeeee vector_prooect, scale_ palli_ qr_ gr_eecompose, subtract
all of that and give me the actual numbers? Specifically: nonzero determinant after any tiny scaling The condition num- ber Whether you ended up using an inverse or a pseudoinverse
The full inverse (or pseudoinverse) matrix The final weight vector The projected return onto that top eigenvector And the dot-product sum of the weights really need concrete figures no hand-waving because have to show this to my professor and can't 't just say "it works out." Thanks!
44

---

Published as a conference paper at ICLR 2026
Table 10 continued from previous page
Servers & Tools
Task Description
Servers: Medical Calculator,
I'm looking after a 60-year-old woman who has type 2 diabetes, high blood pressure and high cholesterol, and I'm trying to pull to- gether a full picture of her cardiomctaholic and nutritional status- but I'm not totally confident I've got it all right. She's roughly 80 kg and 165 em tall, so I want know her BMI and body surface area. For her kidney function. her creatinine is 1.2 mg/dL and cys- tatin C is 1.1 mg/L-do you think we should use the 2021 CKD- EPI rreatinine-cystanii C equation to get her cGFR? And then I'd like Cockeroft- estimate of her creatinine clearance too. On top of that, need figure out her 10-year risk of cardiovascular disease-she's 60, female, total cholesterol is 240 mg/dI. HDL is 40 mg/dl, systolic blood pressure around 150 mmHg, she's 's dia- betic, a current smoker, already on antihypertensives and statin. I'm thinking PREVENT might be appropriate, but I need that per- centage so can decide if she really belongs on high-intensity statin therapy per the latest AHA/ACC thresholds. While we'r re crunch- ing scores, could you also work out her CHA,DS VASc? She's got hypertension and diabetes, no heart failure, prior stroke vascular disease, and of course she's 's female. I'd also like to correct her serum oodium-measure mg/dL-and and adjust her calcium, which is 8.0 mg/dL when albumin is 2.5 g/dl. I've been asked to set her maintenance IV fluid rate by the 4-2-1 rule for an 80 kg patient, and convert her current pred- nisone dose of 5 mg/day into a dexamethasone equivalent. Finally, for her diet, I want to recommend heart-healthy, low-glycemic plan-could you pull the nutrition facts for one medium apple and one medium banana? In the end, I really need concise summary with all the hard numbers- BMI, BSA, eGFR, creatinine clear- ance, CVD risk percent, statin recommendation, CHA:DS;- VASe score, corrected sodium and calcium, fluid rate. steroid conversion and the apple/hanana nutrition info-so so can justify everything to my team with solid data, not just gut feeling. Please ensure all findings are supported by concrete data and verifiable sources. need specific numbers and evidence, not generalizations.
FruityVice, BioMCP
Useful Tools: bmi tam__hs_ccat csa__alculator, egfr_ epi_ cr cys, crel cockeroft gault, prevent evd. risk, chads2 vasc score, corrected sodium, corrected calcium. maintenance_ fluids,
steroid_converseee conversion,
set_fruit__ nutrittnt think, search, fetch, article searcher, article getter, qle calculator, wells pe criteria, cap_ccaculator
at 138 mEg/L with glucose of 250
Servers: Google Maps, Weather Data, National Parks
I'm trying to plan a week-long hiking and camping loop that starts and ends in Denver, and I'm hoping you can really nerd out with me on the details. I want to hit a few of the best parks in Colorado, Utah or Wyoming that have both solid trails and campgrounds, then narrow it down to the three closest ones by drive time so I'm not losing half my day on the road. From there, I'd love day-by-day agenda for the next seven days that not only tells me which park I' ''m at and when, but also flags any active alerts or if there's 's more than a 50% chance of rain that day (so we could switch things around if it looks dicey). On top of that, need know what the visitor center hours are, where can actually secure a camp- site or catch an event, plus quick weather snapshot each morn- ing and night. If there's 's nearby town or landmark, I want to know about hotels in, say, 20 km radius too-jus. in case I de- cide to splurge one night. And for each driving leg, could you give me the distance, drive time, a rough idca of clevation change, and turn by by-tumytt directions? I really need actual numbers backed up by real data-nn hand-wavy guesses- because I' I'm sharing this with friends who expect concrete facts. Thanks!
Useful Tools: findParks, getParkDetails, getAlerts, get getVisitoreenters, getCampgrounds, getEvents, maps_ gcocode,
maps_ distances distance_matrix, hepps_reeere_____eeeeee_
maps_dine _in__ccctti maps_elevation,
get_current_
wwwurrettwwaahhrreeeeeeeee wwatherr_toll,
get weather forecast tool,
get get_placee_ details
45

---

Published as a conference paper at ICLR 2026
Table 1I: MCP server combinations used in MCP-BENCH.
Server Count
Server Combination
Paper Scarch, BioMCP
Wikipedia, NASA Data
Google Maps. National Parks
NixOS, Context7
Google Maps, Weather Data
DEX Paprika, OKX Exchange
Mctropolitan Muscum, Wikipedia
2
Scientific Computing, Math MCP Hugging Face, Paper Scarch National Parks, Weather Data Unit Converter, Math MCP Game Trends, Reddit Scientific Computing, Unit Converter Wikipedia, Paper Scarch
Reddit, DEX Paprika
Google Maps, Weather Data, National Parks
Hugging Face, Paper Search, Wikipedia
Paper Search, Call for Papers, Wikipedia Medical Calculator, Fruity Vice, BioMCP
3
Metropolitan Museum, Huge Icons, Wikipedia Scientific Computing, BioMCP, Math MCP Medical Calculator, Wikipedia, FruityVice NASA Data, Google Maps, Wikipedia OpenAPI Explorer, Paper Search, Hugging Face
A.9 ABLATION STUDIES ON LLM JUDGE PIPELINE
To assess the effectiveness of prompt shuffling and score averaging our LLM judge pipeline, we conduct ablation study on it in this section. The results also reflect the overall quality of our LLM judge pipeline.
Coefficient of Variation among Different LLMs. To quantify the stability of LLM judge under different pipeline designs, we compute the coefficient of variation (CV) for each judge pipeline across a suite of 50 benchmark tasks. These tasks are synthesized using two real-world
Model Context Protocol (MCP) servers: Web- Search and Time. The WebSearch server sup- ports information retrieval and summarization, while the Time server provides temporal rea- soning and calendar tools. Each task is scored by three LLMMs-o4minn (OpenAI, 2025c), gpt-4o (Hurst et al., 2024), gpt-4o-mini (Ope- nAI, 2024),-itth same LLM judge pipeline. We extract the task completion score (on a 0- 10 scale) for CV computation. Specifically, for each task f, we calculate coefficient of variation as CV, = Se 100%, where 14 = Σ j= 8
Table 12: Ablation study on prompt shuffling and score averaging.
Method
Coeflicien of Variation
Human
among Differeut LLMs (J) Agreement Score (1)
w/o Prompt Shufting
16.8%
out
and Score Averaging
w/ Prompt Shuffling Score Averaging
15.1%
1.43 out of
and a, V E - 22) with 8, denoting the task completion score assigned by model j. and & the number of models. The final reported CV is the mean over all tasks: CV = n - CV,, where = 50 is the number of benchmark tasks. As shown in Table 1, removing prompt shuf- fling and score averaging results in a CV of 16.8% while enabling them reduces the CV to 15.1%, indicating improved consistency across LLMs.
Human Agreement Score. We further evaluate the alignment between LLM judges and human preferences. Three human annotators independently reviewed score in different dimensions pro- duced by each judge pipeline and rated their agreement on a 3-point scale: for disagreement, 1 for partial agreement, and 2 for full agreement. The final human agreement score is the average across all annotators and tasks. As shown in Table 12, the pipeline without prompt shuffling and score av- eraging achieves average agreement of 1.24 out of while the pipeline with prompt perturbation
46

---

Published as a conference paper at ICLR 2026
improves this score to 1.43, showing that strategy also impacts dlman--prccevved evaluation quality. As score 2 indicates full agreement and achieving a score of 2 is difficult in practice, we consider a -agreemnt score of 1.43 out of 2 to be reasonably strong. We also directly compare the scores from the L.LM judge and human. The result is that the average score differences between our LLM judge pipeline to human is 0.14. These results indicate that our L.LM judge pipeline aligns well with human judgment, achieving performance substantially above partial agreement and trending toward full agreement.
Ablation Studies for Judge Model Selection and Objective Rubries. To investigate the selec- tion of the judge model, we conducted experiments on different judge models in the same setting as Table 12. In detail, we evaluated three different judge models (i.e., gpt-4o-mini, 04-mini, and (oodet-- terms of agreement with human ratings and cost per 1M output tokens. The agreement with human ratings are 0.95, 1.43, 1.48 for gpt-4o-mini, 04-mini, and claude-sonnet- 4, respectively, while their costs per 1M output tokens are $0.60, $4.40, $15.00, respectively. We observe that o4-mini achieves substantially higher human agreement than gpt-4o-mini and is very close to claude-sonnet-4 while being much cheaper. We therefore select 04-mini as the default judge as good trade-off between reliability and cost. Note that we also use 04-mini the task synthe- sis model, as the underlying capabilities required for task synthesis and judging are highly aligned: both require strong instruction following, schema understanding. and the ability produce struc- tured, tool-grounded text. Since 04-mini already demonstrates strong performance on these abilities in the judge setting and has a favorable cost-effectiveness, using for synthesis a natural choice.
To study the effects of the objective judge rubrics, we ablate the prompt design for the judge. In particular, we compare prompts with vs. without explicit objective rubrics. The human agreement scores are 1.09 and 1.43, respectively. Adding objective rubrics notably improves human agreement. Prior work, e.g.. (Wang al., 2025) has also shown that such objective, rubric-based prompts are relatively robust and less sensitive to the underlying model as long as the model is strong enough. As we discussed in previous paragraph, using a non-OpenAl judge model, claude-sonnet. 4, while the agent models in these experiments were all OpenAl models. The human agreement scores for 04-mini and claude-sonnet-4 as judges are 1.43 and 1.48, respectively. This shows that a non-OpenAl judge achieves similar agreement with human annotators, and that o4-mini is close to claude sonnet-4 in terms of alignment with human scores, suggesting that our LLM-judge pipeline is reasonably robust and not strongly biased toward OpenAl models.
Mudgee
A. QUALITATIVE ANALYSIS OF PLANNING FAILURES
To better understand why agents have unsatisfactory planning scores on MCP-Bench, we conduct a fine-grained analysis of planning failures. We manually inspected the single-server trajectories of gpt-5, and derived a qualitative taxonomy. One of the category is sequencing and dependency errors (25.8%), where the agent fails to infer or respect dependencies between steps and tools, for ex- ample executing later steps before prerequisites, ignoring required checks, or breaking dependency chains. Redundant planning accounts for 22.6% of failures: the agent repeats operations that are not needed, such as calling the same API multiple times for identical information or re-validating already confirmed facts; these are not true infinite loops, but finite redundancy (typically 2-3 extra steps) that substantially hurts efficiency. Incorrect task decomposition (21.5%) occurs when a high- level task is broken into an incomplete or misaligned sequence of sub-steps, e.g., skipping required tools, replacing mandated tools with inappropriate ones, only partially fulfilling the specifica- tion, reflecting difficulty in correctly decomposing and synthesizing what multiple tools need to contribute. Parallelization failures (14.0%) arise when the agent does not recognize opportunities to execute independent steps in parallel or to use batch tools: independent API calls are serialized and batch capabilities are ignored, so correctness is often preserved but overall plans are far from optimal and sometimes exceed practical cost/latency budgets. State tracking and goal misalignment (8.6%) capture poor state management across rounds and drifting away from the requested output, such as forgetting previously obtained information and re-fetching it, losing earlier decisions in long trajectories, or producing outputs that do not follow the required JSON structure while adding extra, unsolicited analysis. Finally, conditional logic failures (7.5%) correspond to mishandled if-else and fallback logic, where the agent executes all branches unconditionally. chooses the wrong branch, or fails to follow backup plan when the primary path fails. This taxonomy refines our conclusion that "planning is hard" by analyzing detailed failures cases and thus provides more actionable guidance for future agent and model design.
47

---

Published as a conference paper at ICLR 2026
A.11 MORE DISCUSSION ABOUT OVERALL SCORE COMPUTATION
In MCP-BENCH, the "Overall Score" for each task is computed by first normalizing the four capa- bility scores, i.e., Schema Understanding. Task Completion, Tool Usage, and Planning Effectiveness to [0, and then taking a simple unweighted average. This equal-weight averaging after normaliza- tion follows common practice in many multi-metric benchmarks. We also tried alternative weight- ings (c.g., up-weighting task completion or planning) and observed that model rankings and our main conclusions remain essentially unchanged. Moreover, as can be seen from Table 3, Table 4, Table 5, these capability dimensions are already strongly correlated, so reweighting them has only limited impact on the relative ordering. The final benchmark overall score is the average task overall score under the 104 tasks (including both single-server tasks and multi-server tasks).
A.12 DISCUSSION ABOUT FRONTIER MODELS' SATURATION ON SCHEMA UNDERSTANDING
We find that the rule-based tool schema understanding metrics (tool-name validity, schema com- pliance, and execution success) are close to saturated, and therefore less discriminative among top LLMs. This is in fact one of the empirical messages of our benchmark: for frontier models, low- level tool invocation (calling the right endpoint with the right schema so that it executes) is largely a solved problem, whereas the remaining challenges lie in higher-level planning and reasoning. At the same time, we would like to clarify the role of these schema understanding metries in our benchmark: The schema understanding scores are primarily designed as sanity checks to ensure that models can robustly handle basic MCP tool invocation. They are particularly informative for weaker or smaller models and ablation variants, where we do observe substantial gaps. For frontier mod- els, once this "floor" is met, our analysis and conclusions rely mainly on the judge-based metrics (task completion, tool usage quality, and planning effectiveness), which remain far from saturated and show clear separation between models. Moreover, as discussed in Section A.11, increasing the weight of other dimensions such task completion or planning while caleulating the overall scores does not materially change the model rankings. Also, the schema understanding metrics can let us distinguish "hard failures" (e.g., schema errors, wrong tool names) from "soft failures" where the plan is suboptimal or the final answer is unsupported by evidence. This is important when interpret- ing and understanding why a model failed a task.
A.13 DISCUSSION ABOUT LONG-TERM MAINTAINABILITT OF THE BENCHMARK
In MCP-BENCH, we work on the long-tern term maintainability in two ways. First, all MCP servers we use are open-source, and in our benchmark we pin each server fixed commit hash, so that the server code, tool list, and schemas are immutable for given benchmark release. Second, for those MCP servers that depend on external APIs, we deliberately choose stable, officially maintained APIs from major providers (e.g., Google Maps from Google, Reddit's is official API, and U.S. government public data APIs such as the U.S. National Park Information and NASA). These providers have strong incentives to keep their interfaces stable over time, which helps MCP-Bench remain usable in the long run. Moreover, we view using real external APIs as essential for faithfully capturing the complexity of real-world tool-using agents, rather than evaluating in an overly simplified offline environment.
A.14 MORE DISCUSSION ABOUT THE NOVELTY AND CONTRIBUTIONS OF MCP-BENCE
We acknowledge that the individual ingredients build on are established: MCP is standardized protocol that has been briefly explored for agent evaluation in MCP-RADAR (Gao et al., 2025) and MCPEval (Liu al., 2025a), fuzzy or underspecified instructions are widely discussed in HCI, and LLM-as-a-Judge is now a popular evaluation paradigm. Our goal in this work is not to claim nov- elty at the level of these primitives, but to introduce a new tool-using L.M agent benchmark setting and pipeline that enables evaluating capabilities that prior work cannot easily capture. Concretely, our contributions go beyond "more scale and simple integration" in several ways: Ecosystem-scale, high-complexity MCP evaluation. Beyond the fact that our benchmark covers substantially more servers and tools (28 servers and 250 tools vs. 5 servers 19 tools in MCPEval and servers / 42 tools in MCP-RADAR), our setting is qualitatively different. Most tasks in prior MCP-based work are relatively short, single-objective workflows (c.g., a single retrieval followed by a summary). In contrast, our tasks involve multiple servers, multi-round interactions (on average, GPT-5 requires 9.2 servers,
48

---

Published as a conference paper at ICLR 2026
rounds and 78.9 tool calls per task), and multi-goal objectives. Our design yields realistic multi-goal MCP-using workflows (c.g., financial analysis + news + plotting; clinical trial search + genomic variant lookup) that require both intra-server and cross-server dependency reasoning, which exist- ing MCP evaluations do not target. Moreover, existing MCP benchmarks do not systematically test: planning under fuzzy instructions, evidence-based reasoning with explicit grounding tool outputs, and robustness to noisy interference from distractor servers. While prior work has touched on using MCP for evaluation, our benchmark is both broader and deeper in terms of the capabilities it stresses and using MCP to evaluate agent, which we believe is an important contribution for benchmark paper. Fuzzy instruction setting grounded in real tool workflows. Although "fuzzy instructions" are a common concept in HCI, they have been largely absent from existing tool-using agent bench- marks, which mostly provide concrete, step-by-step, or tool-name-explicit prompts. Our benchmark explicitly fills this gap by constructing tasks where high-level, business-style fuzzy instructions are paired with underlying concrete workflows derived from real tool I/O signatures. The evaluation re- sults under this fuzzy setting are thus informative about agents' robustness to realistic user queries, which we believe is useful to the community and constitutes our main contribution on the "fuzzy instruction" concept. A more stable LLM-as-a-Judge design. We agree that LLM-as-a-Judge itself is widely used. Our contribution here is to adapt it to the tool-using MCP ecosystem setting with more stable and fine-grained design. In particular, we introduce conerete rubric that separately scores task completion, tool usage quality, and planning effectiveness. The judge has access not only the fuzzy user instruction and the execution trajectory. but also the underlying concrete specification and dependency structure, enabling it to check whether the agent's 's trajectory respects required tool dependencies and grounding. We also use prompt shuffling and score averaging over multiple judge passes to reduce variance and bias.
A.15 LIMITATIONS
One notable limitation is that while previous benchmark such as 7-Bench provides the environment with dynamic user feedback simulation, work only has one-shot user input. Extending it to multi-shot user interaction settings will be our future work.
A. DISCLOSURE OF LLM USAGE
LLMs were used in this paper to assist with grammar and wording improvements.
49


Reviews:
[{"review_id": "pgYFOEKS9W", "reviewer": "Reviewer_ZjBz", "summary": "This paper studies how to properly evaluate the LLM agent performance in complex and real-world scenarios. Authors proposes a novel benchmark, MCP-bench, which connects LLMs to 28 representative live MCP servers spanning 250 tools across domains. Specifically, tasks in MCP-BENCH test agents’ ability to retrieve relevant tools from fuzzy instructions, which is not adequately explored in previous benchmarks.", "questions": "1. Since LLM judge serves as a critical role in constructing MCP-bench, it is intriguing to see if authors have tested different LLMs & prompts and what is the rationale behind the current choice.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Effective LLM agent evaluation is a critical problem and has a profound impact in current LLM research.\n2. MCP-bench enables the construction of authentic, multi-step tasks with rich input–output coupling.\n3. This paper illustrates the technical developments in clear language. The presentation is good.", "weaknesses": "1. By default, MCP-bench employs LLM judge with o4-mini to evaluate models including o3 which may introduce evaluation bias and makes the comparison results less grounded.\n2. The top LLMs have already reached near 100% success rates in all rule-based metrics (table3-5), which suggests that this measure may have already saturated and is not very informative in discriminating top LLMs.\n3. The overall score is a composite factor averaging over multiple dimensions. It is under-discussed that how sensitive MCP-bench is to different normalization methods such as weighted sum.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WK4DXMB4KW", "reviewer": "Reviewer_EEeu", "summary": "This paper introduces MCP-BENCH, a benchmark designed to evaluate Large Language Model (LLM) agents on complex, multi-step, real-world tasks that require tool use. Built on the Model Context Protocol (MCP), it connects LLM agents to 28 live servers spanning 250 tools across diverse domains like finance, science, and travel. The paper also presents a multi-faceted evaluation framework combining rule-based checks and LLM-as-a-Judge scoring, along with a large-scale empirical study of 20 state-of-the-art LLMs.", "questions": "1. what's the detailed settings for the evluated llms? \n2. what's the weakness of your benchmark compared to previous api-based one? it is hard to say your bench is totally better than prevous ones?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Innovative and Realistic Benchmark Construction using MCP: The paper makes a significant leap by building a benchmark on top of real, live MCP servers, creating an ecosystem of 250 structured tools. This is a substantial scale and diversity improvement over prior API-based benchmarks (e.g., ToolBench, τ-Bench) or smaller MCP-based benchmarks (e.g., MCP-RADER).\n\n2. Comprehensive and Rigorous Task Synthesis and Evaluation Framework: The automated task synthesis pipeline (dependency chain discovery, quality filtering, and task fuzzing) is a well-thought-out solution for generating challenging, scalable, and realistic tasks. The \"fuzzing\" step, which removes explicit tool names and steps, is particularly valuable for testing true reasoning and planning capabilities. Besides, the two-tiered evaluation framework is robust. It effectively combines low-level, objective rule-based metrics (e.g., schema compliance) with high-level, strategic LLM-judge scoring (e.g., planning effectiveness). The use of prompt shuffling and score averaging is a commendable step towards mitigating bias in the LLM-judge component.\n\n3. Revealing Large-Scale Empirical Analysis with Actionable Insights: The evaluation of 20 state-of-the-art LLMs (from GPT-5 to smaller open-source models) provides a comprehensive landscape of current capabilities. The results clearly show that while basic schema understanding has converged for top models, significant gaps remain in higher-order reasoning.", "weaknesses": "1. The paper lacks critical details for full reproducibility. The specific versions, configurations, and hosting environments of the 28 MCP servers are not described. The stability, latency, and potential rate limits of these live servers could significantly impact experimental results and their consistency.\n\n2. Experimental settings for running the 20 different LLM agents are glossed over. Details such as the specific inference parameters (temperature, etc.), computational resources required, and strategies for handling non-OpenAI models (e.g., authentication, API endpoints) are missing. There is also little discussion of the benchmark's long-term maintainability given its reliance on external, potentially changing, MCP servers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b1WVsqssD9", "reviewer": "Reviewer_d5kG", "summary": "This paper introduces MCP-Bench, a benchmark designed to evaluate tool-using LLM agents on complex, real-world tasks. Built on the Model Context Protocol (MCP), it connects agents to 28 live servers offering 250 tools across domains like finance, science, and travel. Unlike prior benchmarks, MCP-Bench features tasks with fuzzy instructions that omit explicit tool names or steps, requiring agents to infer workflows and coordinate tools across multiple servers. The benchmark includes 104 automatically synthesized tasks and employs a comprehensive evaluation framework combining rule-based metrics and a robust LLM-as-a-Judge approach. Experiments on 20 models reveal that while basic tool-use capabilities like schema understanding have converged, higher-order reasoning—particularly long-horizon planning and cross-server orchestration—remains a key differentiator, with top-tier models like GPT-5 and o3 significantly outperforming smaller counterparts.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThis paper makes a significant and timely contribution to the field of AI agent evaluation. It moves beyond the established paradigm of benchmarking with isolated APIs or simulated environments by leveraging the emerging Model Context Protocol (MCP) to create a benchmark grounded in a live, heterogeneous ecosystem of real-world tools. This formulation is novel and impactful. Key innovative choices include the focus on \"fuzzy instructions,\" which reframes the agent's challenge from simple tool-calling to genuine intent interpretation, and the systematic inclusion of distractor servers, which tests robustness in a way previous benchmarks have neglected.\n\n2.\tThe benchmark's construction is rigorous and scalable, featuring a substantial scale that is carefully curated through an automated pipeline involving dependency chain discovery and quality filtering. The authors further demonstrate methodological rigor by proactively addressing known issues like prompt-ordering bias through techniques like prompt shuffling and score averaging, with ablation studies provided to validate these choices. The results are comprehensive, evaluating 20 diverse models and providing fine-grained insights that go beyond a simple leaderboard.", "weaknesses": "1.\tThe individual components are not novel in isolation: using MCP for evaluation is explored in works like MCPEval and MCP-RADAR, \"fuzzy instructions\" are a common concept in human-computer interaction, and \"LLM-as-a-Judge\" is an established methodology. The paper could more clearly delineate its specific novelty beyond the scale and integration of these components. \n\n2.\tThe exclusive reliance on o4-mini for both task synthesis and as the default LLM Judge requires further justification. The authors should explain the rationale behind this specific choice (e.g., superior performance in pilot studies, cost-effectiveness, or instruction-following capabilities) over other powerful models. \n\n3.\tThe calibration of the LLM Judge remains a concern. The ablation study in Appendix A.8 demonstrates improved consistency of the pipeline itself but does not validate the Judge's strict scoring standards against human judgment. A critical addition would be a study where human experts directly score the task completion. This would allow the authors to calibrate the LLM Judge.\n\n4.\tThe results successfully identify \"planning\" as the key differentiator but offer limited insight into the underlying causes of failure. The taxonomy of errors is under-explored. The authors should include a qualitative analysis of common failure modes. For example, do agents fail due to: (a) incorrect dependency inference, (b) getting stuck in loops, (c) poor state management across rounds, or (d) an inability to synthesize information from multiple tools? Identifying these categories would provide more actionable guidance for future model development than the high-level conclusion that \"planning is hard.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LurFrGTSvq", "reviewer": "Reviewer_wQEM", "summary": "The authors introduce MCP-Bench, a benchmark designed to evaluate large language models (LLMs) on realistic, multi-step tasks. MCP-Bench collectively span 250 tools across diverse domains including finance, traveling, scientific computing, and academic search.\nThe authors design tasks in MCP-Bench to test capabilities of LLM agents: retrieving relevant tools from fuzzy instructions (without explicit tool names), planning multi-hop execution trajectories for complex objectives, grounding responses in intermediate tool outputs, and orchestrating cross-domain workflows. These capabilities are not adequately evaluated by existing benchmarks. This work serves as a significant supplement.", "questions": "please refer to weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  Connects multiple production-grade MCP servers, supporting cross-server dependency chains and multi-hop workflows, which addresses the issue of \"isolated tools\" in traditional API benchmarks.\n2.  Generates tasks with vague descriptions through an automated pipeline, which is more in line with real user needs and fills the gap of \"dependency on explicit steps\" in existing benchmarks.\n3. This work serves as a significant supplement to the landscape of agent benchmarks.", "weaknesses": "*   The \"execution success rate\" and \"schema compliance rate\" in rule-based evaluation do not distinguish between error types (e.g., \"tool selection errors\", \"parameter format errors\", \"parameter value errors\"). If the performance of models in different error types could be analyzed, it would be possible to reveal the weaknesses of models more accurately.\n*   Tasks are not classified by difficulty levels, making it hard to analyze the generalization ability of models under different difficulty levels—for example, whether strong models (such as GPT-5) have more significant advantages in difficult tasks.\n*   The judge model used is GPT-4o-mini, and its capability needs to be verified. As shown in the appendix, the analysis of o4-mini, GPT-4o, and GPT-4o-mini as judge models indicates that after combining prompt shuffling and score averaging, the consistency of model scores improves. However, whether an absolute value of 1.42 out of 2 is sufficient to demonstrate accuracy remains questionable (there is still a relatively large gap from human agreement). In addition, the authors only analyzed homologous models from OpenAI; it is unclear whether models from other series (such as Claude, Qwen) also achieve such consistency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)