MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use
Authors: Zijian Wu
Xiangyan Liu
xinyuan zhang
Lingjun Chen
Fanqing Meng
Lingxiao Du
Yiran Zhao
Fanshi Zhang
Yaoqi Ye
Jiawei Wang
Zirui Wang
Jinjie Ni
Yufan Yang
Arvin Xu
Michael Qizhe Shieh
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Agent
Tool Use
Benchmark
Model Context Protocol
Primary area: datasets and benchmarks
TL;DR: MCPMark is a comprehensive benchmark for stress-testing agents and models in realistic MCP-based scenarios, with 127 tasks across Notion, GitHub, Filesystem, PostgreSQL, and Playwright.

Abstract: The MCP standardizes how LLMs interact with external systems, forming the foundation for general agents.
However, existing MCP benchmarks remain narrow in scope: they focus on read-heavy tasks or tasks with limited interaction depth, and fail to capture the complexity and realism of real-world workflows.
To address this gap, we propose \texttt{MCPMark}, a benchmark designed to evaluate MCP use in a more realistic and comprehensive manner. It consists of $127$ high-quality tasks collaboratively created by domain experts and AI agents, each with a curated initial state and programmatic verification script. These tasks demand diverse CRUD operations and richer environmental interactions.
We evaluate cutting-edge LLMs using a minimal agent framework. The best-performing model, \texttt{gpt-5-medium}, reaches only $52.56$\% pass@1 and $33.86$\% pass\textasciicircum{}4, while other strong models including \texttt{claude-sonnet-4} and \texttt{o3} fall below $30$\% pass@1 and $15$\% pass\textasciicircum{}4. On average, LLMs require $16.2$ turns and $17.4$ tool calls per task, highlighting the stress-testing nature of \texttt{MCPMark}.

Published as a conference paper at ICLR 2026
3 MCPMARK: A BENCHMARK FOR REALISTIC AND COMPREHENSIVE MCP USE
Zijian Wu Xiangyan Liu*1,4, Xinyuan Zhang Lingjun Chen" Fanqing Meng" 1,4 Lingxiao Du 1,4, Yiran Zhao"! Fanshi Zhang* 2.3, Yaoqi Ye', Jiawei Wang", Zirui Wang". Jinjie Ni,, Yufan Yang? Arvin Xu Michael Qizhe Shieh
National University of Singapore
EvalSSs
LobbeHub
aabsolute Al
ABSTRACT
The MCP standardizes how LLMs interact with external systems, forming the foundation for general agents. However, existing MCP benchmarks remain narrow in scope: they focus on read-heavy tasks or tasks with limited interaction depth, and fail to capture the complexity and realism of real-world workflows. To address this gap, we propose MCPMark, a benchmark designed to evaluate MCP use more realistic and comprehensive manner. It consists of 127 high-quality tasks collaboratively created by domain experts and AI agents, each with a curated initial state and programmatic verification script. These tasks demand diverse CRUD operations and richer environmental interactions. We evaluate cutting-edge LLMs using minimal agent framework. The best-performing model, gpt --meii um, reaches only 52.56% pass 1 and 33.86% pass*4, while other strong models including claude sonnet-6 and o3 fall below 30% pass @ and 15% pass*4, On average, LLMs require 16.2 turns and 17.4 tool calls per task, highlighting the stress-testing nature of MCPMark.
s mepmark.ai
eval-sys/mepmark
Initialization
Agent
Verification
Initial State
Framework
Final State
ONLOSE RESUPE
BILDIC RESINE
Flesystem
Agent Trajectory
CINTACT
Delug
coothhrrcom
post-search
Debug
footShascom
127-201-404
127-001-404
PRODECTS
-Plogn Secuee"
Git-lub
Vilbe Coding
2013
PROJOCTS
N
Silpal
MccM
Boogle
June
Dlaya]
Unlearing Engineer
2023 2023
Verffcation
PestgreGQL
3.588 part-prejec: bicekt:
datate outilated
HoMark project
patck Stack-chhllllo Ldtron ("0lack
project
hooaiin
"Settt" section betow "Intereat"
5.58i exille cectiie with leact
All ist. Hend's what dii updett your "Online page reoepania the Project section
Past-prazect blocks not foune
Sely axe now 5X111 Tound
Figure I: MCPMark evaluation pipeline with full state tracking. Each task begins from curated initial state with specific task instruction. The MCPMark-Ageer then executes tool-calling loop, followed by programmatic verifier that evaluates whether all required checks are satisfied.
Student leads; listed in random order. Equal contribution. Correspond (zijian.wo wu, liu. liuxxaangyan @u.nus.edu
Equal advising.

---

Published as a conference paper at ICLR 2026
Cloudilare Turnstile Authertication
N
Weekend Adventure Planner
Employee Project Tracking
Chslagy
Playeright HCP tools comp Clcudflard temettle
Tast: Duild trecking system with bebris for projects, assigrment rés, cotorics, por sorfrmancc inobies, 1: project tradking
1: Naxiqute mtiee///mmmmmari auchitu
2: toreign key redetteennpp,
Nol
Linting Ci Workflow
Contact Information
ancl
Issues
8'
Actiona
Document
Downloads/
Archives!
P budget.cov
P
back.cr
giignos
G log-cev
calostatorr
sserigis
Partonal!
price.cov
2 READHNG and
Templi
Teak: Sei esai ntt workitun for codle quality
Task: Create comp raheratvv wescend asventarr planteer thit pererates structured Binerary page
various fiie formats on deectro and parform anelyde on the collected rele- data.
Tasks Extrect
enformmment on al Pis with propeer EI integration.
Create leriing cunfiganetion brunch;
configuratiun;
2: Quunry lcentilyy activiters on 'Deaches' sag:
05 child page of main page:
2: cotte a CSV file, 31
Exbract contact information;
Figure 2: Representative task instances, showing initial states (Top) and task instructions (Bottom). Examples include: Login with Cloudflare Tursstle in Playwrighr; CI/CD setup with ESLint in GitHuu: weekend planner using tagged queries in Notion; schema design for project tracking in PQl: and contact extraction to CSV in Filesystem. All tasks show complex, multi-step workflows typical of real-world usage.
INTRODUCTION
The Model Context Protocol (MCP) (Anthropic, 2024) is standardized interface that connects large language models (LLMs) (Comanici al., 2025; OpenAI, 2025c; Team, 2025) with external systems such as tools, APIs, databases, and contextual resources. By standardizing the way LLMs access and operate on these systems, MCP allows agents to function more effectively with "eyes and hands" in real environments, and many see it as a foundational layer for AI in the agentic era (Hou et al., 2025).
Despite growing use in practice, existing MCP benchmarks remain limited: tasks often involve shallow or read-heavy interactions (Liu et al., 2025; Yin et al., 2025; Mo et al., 2025; Luo et al., 2025), leading narrow range of task patterns. As result, they fail to capture the complex, multi-step workflows typical of real-world usage. This makes it difficult to probe the performance boundaries especially in assessing whether current models and agents possess the necessary capabilities, such as reasoning, planning, long-context processing, and tool use, to tackle realistic and demanding agent tasks.
To address these gaps, we introduce MCPMark, a benchmark designed to simulate realistic user scenarios within mirrored or isolated container environments, accompanied by reliable automated evaluation. Specifically, MCPMark spans five representative MCP environments: Notion, GitHub, Filesystem, PostgreSQL and Playwright. Figure presents overview of the evaluation pipeline, where each task comprises three components: task instruction, initial state, and programmatic verification script. Figure 2 provides examples of task instructions and initial states.
For task creation, after selecting or designing an initial state, a task instruction and verification script are developed through a human-Al collaborative pipeline, where domain experts and language agents iteratively co-design and refine each task. After this pipeline, we apply expert cross-review and community-level validation to ensure clarity, realism, and quality. Compared existing MCP benchmarks, MCPMark offers significantly broader coverage of create, read, update, and delete (CRUD) operations across diverse workflows. In total, MCPMark comprises total of 127 tasks and 38 unique initial states, with 20 to 30 tasks in each MCP environment.
To fairly evaluate model performance on these tasks, we introduce MCPMark-Agent, a mini- mal and general agent framework that executes models through a standardized tool-calling loop. MCPMark -Agent integrates with a variety of MCP servers and model providers, enabling
2

---

Published as a conference paper at ICLR 2026
Table 1: Benchmark Comparison.
consistent and automated evaluation grounded the programmatic infrastructure defined by MCPMark. Comprehensive experiments on state-of-the-art models demonstrate the benchmark's diffieulty. Specifically, the best-performing model, gpt 5-me-iumm achieves only 52.56% pass 1 and 33.86%
Task Pattern
Average Turns
Benchmark
Verification
MCPEval
Synthetic
Hyyrid
N/A
LiveMCPBench
CRUD-diverse
LL--as-judge
3.2
(OpenAI, 2025c),
MCP-Universe
Read-heavy
Programmatic
6.8
LiveMCP-101
N/A
LLM-as-judge
5.4
pass*4, while other leading models such as (Anthropic, 2025a)
MCPMаrk
CRUD-diverse Programmatic
16.2
and o3 (OpenAI, 2025d) fall below 30%
pass@1 I and 15% pass^4. On average, each task requires 16.2 execution turns and 1 17.4 tool calls, with some models such as kimi k2 instruct (Team et al., 2025) averaging over 20 turns per task. Overall, shown in Table 1, prior MCP benchmarks are limited by task patterns or verification rigor. In contrast, MCPMar rk integrates CRUD-diverse tasks, programmatic verification, and longer workflows, offering closer alignment with real-world MCP and workflow complexity.
In addition, our evaluation reveals several consistent patterns that underscore the distinctive properties of the benchmark. First, the benchmark demonstrates its intrinsic difficulty through consistently low performance on the 4 which more convincingly reflects real-world conditions than commonly used metrics like pass@ pass@ (Yao al., 2024), emphasizing the challenge of solving tasks re- liably and consistently across multiple runs. Second, performance varies substantially across differ- ent MCP environments, suggesting a notable environment gap. This variation arises from differences in data availability and simulation fidelity: tasks involving local services such as the Filesystem are generally easier emulate and more commonly represented in training data, whereas remote services like Notion require more complex, underrepresented interaction patterns that are harder to reproduce. Finally, the benchmark emphasizes efficient tool use: successful completions tend to involve fewer, more targeted tool calls, while failure cases often exhibit repetitive exploratory interactions that fail to make meaningful progress. Collectively, these patterns show that MCPMark effectively surfaces key challenges in stability, generalization, and planning across diverse multi-component environments.
2 MCPMARK: STRESS-TESTING COMPREHENSIVE MCP USE
In this section, we provide a detailed introduction to MCPMark rk, including the benchmark construction process, the associated evaluation framework, and an overview of the benchmark.
2.1 BENCHMARK CONSTRUCTION
MCP services and initial states. MCPMark integrates 5 MCP servers that span diverse and practical application environments. A partial overview of each MCP tool set shown in Figure 3 (right). Moreover, unlike prior work that uses generic or minimally initialized environments task starting states (Liu et al., 2025; Luo et al., 2025; Yin et al., 2025), we carefully design initial states that reflect realistic and comprehensive usage scenarios, serving as the starting points for the tasks. Specifically:
Notion connects to the official remote API for creating, editing, and querying both documents and databases. Initial states are instantiated from widely adopted templates.
Q GitHub leverages the official remote API to support project management and Git operations, including CI/CD, issues, branches, pull requests, and commits. Initial states derived from repositories with realistic development histories and configurations.
Filesystem supports file I/O, directory organization, metadata inspection, and search. Initial states are curated folder structures that mirror everyday user scenarios.
PostgreSQL provides access to a relational database, with tools for schema exploration and SQL. query execution. Initial states are representative template databases with realistic schemas.
Playwright enables browser automation, offering commands for navigation, form completion, data extraction, and generating screenshots or PDF exports. Initial states come from two sources: self-authored webpages designed to test specific functionalities (c.g., login through Cloudflare) and localhost webpages adapted from WebArena (Zhou et al., 2023).
3

---

Published as a conference paper at ICLR 2026
Eid oh Juan
couee pC. www.ppi.nn..... Fis
conerr 241 Jeppa
crear wep.cppyy
25, Jo.oooos
grt ponees
get popec
g5. kg.
Raced
Bosal recis
Rnvel ppl fice
swite 1k
Roll Sk
mooo Jivesry Beweees
D11
Bsorth Siks
Egg.t
Bld alowad dinctories
Natton todd)
-Aft-gr-cur
Alf-grtadt
po quayy
MCPWkrkk
APlipplan Alf-crales-ommnn
APSerzzioee acremses:
Plasniggt (2) toob))
bosoe clckk
boswee: clise
broweer prnser, mewsge
boowet 1985
borest endeue
brsear: fir sphod
browser from
beonast Junda, disko
Junz
broweet optice
peos
Jeasl
Pesggeeggg toabl Fanees of Laaaanncceenn Maar iskkoo
bbaaateca sperie
copplie qury
aaadlle worbbed Indrac
Figure 3: Task distribution and tool set overview of MCPMark. Left: 127 tasks distributed across 5 MCP servers and 38 curated initial states. Right: toolset per server, covering commonly used functionalities, with full support for CRUD operations in each corresponding MCP environment.
Task creation pipeline. Each task in MCPMark is grounded in an initial state of the respective environment (e.g., a template Notion page or a designated website) and consists of natural language instruction paired with an automatic verification script. Constructing tasks of this form is difficult if we rely solely on humans or solely on agents. To address this, we design a human-A Al collaborative pipeline that pairs human experts with two agents: task creation agent and task execution agent. The pipeline proceeds in four steps:
L. Exploration: Given an initial environment state, human expert and the task creation agent jointly explore the environment, guided by a high-level instruction or topic informed by expertise and real-world experience. This stage aims to capture both wide overview of the environment and deep, specific context that will later support realistic and well-grounded task creation.
II. Evolvement: The task creation agent proposes new task instruction or refines an existing one by introducing additional complexity. This may include removing unnecessary instructions, increasing the difficulty of information seeking, raising the processing burden (e.g., through longer input content), or requiring more interaction steps. The human expert ensures that the task remains practical, verifiable, and sufficiently challenging.
III. Verification:
The task creation agent drafts programmatic verification script. The human expert then completes the task with assistance from the task execution agent. Afterward, the verification script is executed and iteratively refined until it is fully consistent with the task instruction. To ensure reliability, the human expert also adjusts the final environment state to validate whether the script correctly detects both successful and unsuccessful outcomes.
IV. Iteration: Steps @ and @ are repeated to progressively increase task difficulty, while preserving automatic verifiability and maintaining realism through authentic user scenarios.
Overall, even with agent assistance, constructing each sample remains labor-intensive. Involving 10 experts with diverse backgrounds-inclleee full-stack & Al infra engineers, and AI investors eeachh task takes 3 ~ 5 hours of focused expert effort. While most tasks are built through the standard pipeline, experts occasionally leverage their accumu- lated experience or domain knowledge to directly write natural language instructions. In these cases, the task creation agent is bypassed. but the verification scripts are still generated and refined within the same pipeline. We defer the prompts and guidelines used in the task creation pipeline to Appendix B.
computer science PhD students, front-end designers,
Quality control. All tasks underwent cross-review by human experts and month-long community check to ensure clarity, consistency, and alignment with real-world application scenarios. In particular, for tasks that model solved correctly, we conducted additional verification to ensure their validity. This process ensures that the benchmark remains challenging yet practical, and that evaluation outcomes unambiguous.
4

---

Published as a conference paper at ICLR 2026
2.2 BENCHMARK OVERVIEW
Dataset statistics. We create a total of 127 tasks across 5 MCP servers-30 for Filesystem. 28 for Notion, 25 for Playwright, 23 for GitHub, and 21 for PostgreSQL- on 38 curated initial states. On average, the task instructions contain 288.6 words, and the corresponding verification scripts consist of 209.8 lines of code. The detailed task distribution is presented in Figure 3 (left), while the corresponding toolsets for each MCP are shown in Figure 3 (right).
Task characteristics. The tasks span a wide range of realistic workflows, including updating nested properties in Notion, managing commits and pull requests in GitHub, automating interactive forms in Playwright, organizing complex directory structures in the Filesystem, and executing transactional updates in PostgreSQL. Five representative tasks, one from each MCP, are shown in Figure 2. Collectively, these tasks provide diverse CRUD coverage and reflect the challenges of authentic multi-step workflows across varied application scenarios.
Necessity of MCPMark. Existing tool-use agent benchmarks such as AppWorld (Trivedi et al., 2024), WebArena (Zhou al., 2023), and SWE-Bench (Jimenez et al., 2023) provide valuable testbeds, but they introduce trade-offs that limit their suitability for evaluating multi-step agent workflows in real-world use. SWE-Bench offers strong realism but is restricted a single domain, whereas AppWorld and similar environments support diverse tasks but rely on custom wrappers or research-oriented APIs that do not reflect the stateful behavior of production systems.
In contrast, MCPMark builds directly on official MCP servers and real environments, enabling agents interact with the same APIS and conditions they encounter in deployed settings. This design allows us to capture workflow complexity that cannot be reproduced in simulated environments, including CI/CD operations on live repositories, transactional database behavior, and multi-file organization in realistic file systems. As result, MCPMark complements existing tool-use benchmarks by providing a high-fidelity evaluation setting tailored to modern MCP-based agent workflows.
2.3 EVALUATION FRAMEWORK
State tracking and management. MCPMark executes all tasks within sandboxed environments that enforce explicit state tracking, a design choice that ensures safety, reproducibility, and fair comparison across models. Each evaluation follows a consistent lifecycle: tasks begin from a well-defined initial state that mirrors realistic application scenarios, @ proceed with agent execution based on task instruc- tions, and conclude with an automatic verification script that programmatically checks whether the final environment satisfies the task requirements. After verification, @ the environment is reset to its original state, preventing side effects and enabling repeated evaluation under identical conditions.
Evaluation Agent. To standardize evaluation, provide MCPMark-Agent lightweight and peneral-purpose agent framework. It is built on LiteLLM together with the Model Context Protocol Python SDK support compatibility and extensibility. Specifically, MCP servers configured through the SDK, and their tools are exposed to the agent. LiteLLM then (1) converts the tools into the OpenAl function-call format and (2) routes requests to the official APIs of different providers, thereby ensuring execution that reflects each model's 's native capabilities.
During task evaluation, the agent follows a tool-calling loop in which the model iteratively invokes MCP tools, interprets responses from MCP servers, and adjusts its actions. The loop terminates once the model produces a final response without further tool calls. Although this agent framework deliberately basic and omits optimizations that may be desirable in production systems (which we leave for future work), this design avoids task-specific heuristics and model-specific biases, thereby providing clearer measure of model's intrinsic agentic capabilities in MCP environments.
3 EXPERIMENTS
In this section, we describe the experimental setup, introduce the evaluated models and metrics, and present results and analyses on different environment, reasoning efforts, and failure patterns.
We also provide subset of MCFMark comprising 50 tasks (10 per MCP server), derived from the standard tasks by relaxing subtask requirements or providing hints. The results are reported in Table 14.
https: / // thub. com wttpuu///.ee/rrrettttee.... https: / /gi i thub. httpbb.ccmmoooeeecoootttrrrrrooo
wwwp:/oooo/pppttooooooooo adk
5

---

Published as a conference paper at ICLR 2026
3.1 EXPERIMENTAL SETUP
Models. We test range of state-of-the-ar proprictary and open-source models, primarily accessed through LiteLLM. Proprictary models include gpt-5 (OpenAI, 2025c) with different reasoning effort levels (low, med i um, high) and smaller variants (mini and nano), well as earlier 9pt-4.1 (OpenAI, 2025b) variants. We also evaluate claude-opus-4: claude-scnnet-4, grok-4, grok-code-fast-i,
o3, o4-mini, qwen3-max, and gemini 2.5 pro (Anthropic, 2025b;a; xAI, 2025; OpenAI, 2025d; Comanici et al., 2025). On the open-source side, we evaluate qwen3-code coder--pus, deepseek-v3... 1, glm-4. 4. 5, and qpt-oss-120b (Team, 2025; Team al., 2025: Liu et al., 2024; Zai, 2025; OpenAI, 2025a). We do not test small open-soures models (s 100B) due to the difficulty of the benchmark.
5-po
Metrics. We use three complementary metrics to measure agent performance: pass@1 1, pass@4, and pass*4. Pass@1, 11 captures the single-run success rate, i.e., the proportion of tasks successfully in one single attempt. Pass@4 measures success when allowing up to independent runs, indicating whether repeated attempts improve coverage of difficult cases. Pass"4 is stricter measure: task is counted as correct only if all four independent runs succeed, making it a strong indicator of model consistency and stability under stochastic generation (Yao et al., 2024).
Implementation Details. We use MCPMark AAgett as the unified framework to benchmark MCP use across models. While specialized agent designs could further improve performance, we leave such optimizations as important future work. Each run is limited to a maximum of 100 turns with 3600-second timeout. Unless otherwise specified, all models inference settings (e.g., temperature, top-p, reasoning effort). The agent supports two execution paths: general path via LiteLLM with function-calling tools and native path with direct tool support for certain models (e.g., Anthropic API for extended thinking mode). For MCP server selection, we generally choose the most commonly used ones (see Appendix C for details).
evaluated under their default
3.2 MAIN RESULTS
We evaluate all 127 tasks using MCPMark-Agent reporting pass@1. 1, pass@4, and pass"4 metrics. Unless otherwise specified, pass@1 I scores are averaged over four independent runs and reported as mean = std. Detailed results on each MCP service are provided in Appendix D, and representative trajectories appear in Appendix E.
MCPMark remains challenging for frontier models. Table shows that the best cest-performing model, gpt-5-med i um, reaches only 52.56% pass@1 1, while source model, achieves 24.80%. Most proprictary models fall within the 15% to 30% range on pass@1 and several open-source models perform below 10%. Moreover, Table 9 highlights the high interaction demands of the benchmark: for example, g-en3-max and kimi -k2--kk-inss- constructt average 23.85/20 23.85/66.95 turns with 23.02/26. 2. tool calls, respectively. These results underscore that MCPMark remains highly challenging benchmark for current frontier models.
the strongest open-
Models generally perform better on local service tasks. We observe from Table that perfor- mance varies significantly across MCP services, showing a clear divide between local and remote environments. Local services such as PostgreSQL, Filesystem, and Playwright achieve substantially higher success rates, with gpt 5ont-5-mead 5-medi i um reaching 76.19% 57. 57.50% and 43.00% pass@ @ respec- tively. Remote services like Notion and GitHub remain challenging, with most models achieving below 25% pass@ 1. This gap likely stems from data availability: local services are easier to simulate and collect training data for, while remote service APIs require authentic interaction traces that are expensive to curate at scale. These results suggest that data remains key to enabling better MCP use.
Robustness lags far behind. Table demonstrates that p provides substantial gains, with gpt-5-med um and c claude-sonnet-4 achieving 68.50% and 44.88% compared to just 52.56% and 28. 15% for pass@1. 1. However, the performance at pass^4 drops sharply to 33.86% and 12.60%, respectively, underscoring the model's inconsistency and instability across runs. Similar discrepancies are observed across other models, with pass@4 often exceeding 30% while pass^4 remains in the 5% to 15% range, suggesting that while repeated attempts improve success, robustness under multi-tum tool use in MCP contexts remains a common challenge- a shortcoming that poses significant risks for real-world deployment where reliability across runs is essential.

---

Published as a conference paper at ICLR 2026
Table 2: Model comparison across MCPs. Pass@1 is computed as the average over four independent runs, with the superscript showing the standard deviation; each MCP service value is also averaged over four runs. Within each model group (Proprietary Open-Source), the best result marked in bold and the second best result is underlined. For GPT-S series models, explicit suffixes (e.g., "-medium") indicate the reasoning effort setting; for all models, results correspond their default reasoning effort if supported. Abbreviations of MCP services are: FS Filesystem, GH GitHub, NT Notion, PW Playwright, PG PostgreSOL.
MCP Services
Metries
Model
FS
GH
NT
PW
PG
pass 6
pass@4
pass 44
Proprietary Models
gpt-5-medi
57.50
47.83
41.96
43.00
76.19
52.56 1:229
68.50
33.86
grok-4
50.83
14.13
2.68
35.00
58.33
31.69 #2.91
44.88
18.11
claude-opus-4.1
33.33
21.74
35.71
24.00
33.33
29.92 10.00
claude-sonnet-4
27.50
16.30
21.43
26.00
53.57
28, 25
44.88
12.60
opt-5-mini-nedi
33.33
18.48
16.07
12.00
61.90
27.36 1.312
45.67
9.45
c3
35,83
14.13
24.11
15.00
36.90
25,39 #2004
43.31
12.60
grok-code-fast-
23.33
8.70
2,68
25.00
47.62
20.47 #3.39
30.71
9.45
ewen3-max
10.83
14.13
16.96
8.00
44.05
17.72 1:13
22.83
11.02
04-mini
25.00
14.13
20.54
12.00
11.90
17.32 #2330
31.50
6.30
gemini-2.5-pro
24.17
9.78
4.46
15.00
26.19
15.75 1:.56
29.92
4.72
remini-2.5-flaas
8.33
15.22
6.25
6.00
10.71
9.06 1.0.88
18.11
3.94
gpt-4.1
12.50
7.61
6,25
8.00
4.76
8.07 1:066
12.60
3.15
spt-S-nane-nedium
6.67
7.61
3.57
0.00
15.48
6.30
42201
11.81
1.57
spt-4.1-mini
3.33
6.52
1.79
0.00
9.52
3.94 #:1.6
7.09
1.57
spt-4.1-nano
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
Open-Source Models
cwen3-coder-plus
13.33
19.57
19.64
30.00
47.62
24.80 12.05
40.94
12.60
kimi-k2-instruct
14.17
16.30
8.04
30.00
47.62
21.85 #1.16
31.50
12.60
deepseek-v3.1
15.83
9.78
12.50
7.00
42.86
16.73 1:441
28.35
7.87
glm-4.5
7.50
22.83
21.43
13.00
14.29
15.55
24.41
6.30
gpt-oss-12s -20b
5.83
4.35
3.57
3.00
7.14
4.72 1:0.6
13.39
0.00
More turns do not necessarily yield better performance. Figure 4 highlights distinct tool-calling behaviors across models. In particular, the aeleiinnc-aacuuraae correlation shows that stronger models succeed through better decision making and targeted exploration, not blind trial-and-error. Notably, kimi-k2 instruct often enters an overcalling mode, exceeding 30 turns with diminishing success rates indies-iddiaainng the model might get stuck or loop without effective information retrieval. In contrast, gpt-5 50:5---miiiiu um um achieves the highest pass@ while maintaining reasonable turn budgets, demonstrating that success arises from efficient decision fecision-making rather than exhaustive tool calls. Turn counts also vary significantly across MCP services (see Appendix G for details).
Cost is not a proxy for performance. Figure 21 shows that higher cost does not lead to higher accuracy. Some of the most expensive runs achieve lower pass @ 1, while several lower-cost runs reach stronger results. Table 9 reports per-task averages and further shows that costs vary widely even when the number of turns is similar. Higher cost alone does not imply better results.
3.3 BENCHMARKING MCP SERVERS AND AGENT FRAMEWORKS
Beyond measuring model performance, MCPMark can also be used to evaluate differences across MCP server implementations and agent frameworks. These aspects are important practical MCP use, because the same model can behave very differently depending server design or scaffold logic.
7

---

Published as a conference paper at ICLR 2026
100
0.5
80
60
50
0.4
40
Turns
0.1
0
gpt-oss-120b
gemini-2.5-pro
deepseek-v3.1
kümi-kz-inntrucc
qwen3 coder-plus
daude-sonnet-4
03
claude claude-opus-ee -pus-4.1
grok-4
gpt-5 upt-S-medium
Figure 4: Turns distribution. Each point is run (gray fail). Plots show the turn distribution of successes; color encodes pass@1. Stronger models finish with fewer, better-targeted calls.
MCP server variation. As shown in Table 11, the same model can show large performance gaps across MCP servers. For GitHub tasks with cl pouudd--onnet- 4, the KlavisAI server reaches 31.5% pass@1 compared 16.3% for the official server. A similar gap appears in Notion (34.8% vs. 21. 4)). PostgreSQL shows the same trend on claude-sonnet: Supabase (52.4%) both exceed the official server (48.8%). These gaps also align with differences in token usage, where higher-performing servers frequently require fewer tokens. Together, the results indicate that choices in schema exposure, error messaging, and other engineering details on the server side can materially influence agent success.
5: InsForge (54.8% and
Agent scaffold comparison. We also benchmark MCPMark -aggett against ReAct (Yao et al., 2022) and Codex (OpenAI, 2025e) (Table 12). Surprisingly, naive iterative tool-calling provides the strongest baseline (gpt- -5-medi 52.6% pass 1), outperforming both ReAct (37.8%) and Codex (36.2%), suggesting that structured scaffolds may introduce redundant constraints in MCP settings.
ANALYSIS
In this section, we investigate two aspects that shape model performance on MCPMark: the role of reasoning effort in agent generalization, and the types of failures that prevent successful execution.
4.1 REASONING MODE AND EFFORT
We study how models benefit from different levels of reasoning effort, which are typically reflected in the number of consumed thinking tokens before issuing tool calls. Table 3 reports results for the gpt-5 series and -laude-soneet. across different effort settings.
Model perspective. The gpt-5 series benefits from increased reasoning effort moderate and large scales, though effects diverge by size. For g medium effort raises pass to 52.56% from 46.85% at low effort. gpt-5-m n shows even stronger relative gains, improving from 8.27% to 30.32% between low and high. By contrast, gpt --nnno shows only marginal changes around 4% to 6%, suggesting models of this scale lack the capacity to exploit additional reasoning tokens. claude-scnnet- is similarly insensitive, remaining stable around 27% 28%. These results indicate that translating additional reasoning steps into better MCP use is non-trivial and likely depends model's base capacity and training approach.
MCP perspective. Reasoning effort selectively improves generalization in agentic tasks. Remote services benefit most: GitHub performance nearly doubles from 27.17% 50.00% between low and high effort for gpt-5, while Notion rises from 36.61% % to 44.64%. Local services remain stable, with PostgreSQL at 72% 76% and Filesystem varying under percentage points. We interpret this discrepancy as stemming from differences in training coverage. Remote services typically have limited exposure due rate limits and access restrictions, making the tasks harder and
We also include results other agent scaffolds Table 13 for reference.
8

---

Published as a conference paper at ICLR 2026
Table 3: Reasoning effort. Comparison of gpt- -5 series models and claude-sonnet- under different reasoning effort settings. Pass@ is reported mean with standard deviation (4 runs). Each model expands into its supported reasoning effort settings. Best values in each column are bolded.
Model
Reasoning
Overall
GH
N
NT
PW
Low
46.85
54.17 27.88
27.17 -2:17
36,61 48.3
45.00
42000
73.81 #:4.6
gp=-5
Medium
52.56
57.50 14.9
47.83
19.39
41.96
43.00
76,19 1:66
Iligh
51.57 #2291
52.50 14.19
50.00
#2.51
44.64
42.00 4:536
72.62 #:456
Low
8.27 11.11
12.50
15.69
8.70
13.55
5.36 16.19
1.00 42,00
14,29 13389
gpt-5-mini
Medium
27.36
33.33
27:20
18.48 29.99
16.07 26.84
12,00 $7.30
61.90 22675
High
30.32 11.98
35.00 1882
19.57 12251
20.54 1:500
15.00 16.00
66.67 13889
Low
4.33 1::36
12.50 14419
0.00 1.000
0.00 2:000
0.00 1.0000
8.33 1-4.56
9pt-S-nano
Medium
6.30 22.92
6.67 12600
7.61
#2 17
3.57 =0.00
0.00 20000
15.48 $5599
High
5.12 12266
5.83 15.69
8.70 13355
0.89 1.799
2.00 12,31
9.52 13389
N/A
28.15
27.50 2:19
16.30 16680
21.43
26.00
53.57
LLaude-sonned
Low
27.36 +1.97
23.33
25.00 25.00 14
22.32 1:3.2
22.00 1:4000
48.81
#:813
High
28.35 2.773
23.33 1471
28.26
13.51
19.64 1:945
26.00
1:2.1
50.00 1:825
gemini-2.5-flash
gpt-4.1
gpt-5-high
kimi-k2-instruct
14%
16%
13%
52%
20%
17%
66%
10%
84%
84%
context window overflow
turn limit
abandoned
premature stop malformed calls
implicit
Figure 5: Failure breakdown across models. Failures are categorized as either implicit (task completes normally but fails verification) or explicit (e.g.. context window overflow, turn limit exceeded, abandoned, premature stop, or malformed tool calls).
requiring stronger generalization at test-time. Reasoning helps bridge this gap by enabling models to extrapolate to unseen cases, aligning with recent discussions (Yao et al., 2023b; Yao, 2025) that highlight "language generalizes through reasoning in agents".
4.2 FAILURE BREAKDOWN
Introduction. We classify failures into two categories to ease presentation: implicit and explicit. Implicit failures oceur when the task completes successfully but the output does meet the required specifications. These often stem from issues such as reasoning errors, suboptimal planning, ineffective tool usage, or difficulty handling long contexts, which may interact cause complex failures that difficult to attribute to single factor. In contrast, explicit failures can be directly linked to specific issues. These include context window overflow (input exceeding the model's processing length), turn limit exceeded (the model exhausts its allowed interaction steps), abandoned tasks (model decides the task infeasible), premature stop (model halts without completing or making necessary tool calls), and malformed tool calls (invalid parameters or improperly structured payloads).
Observations. As seen in Figure 5, implicit failures account for the majority of errors across all models, often exceeding 50%. Models like 9pt-5-high and k kimi-k2- i ruct show over 80% implicit failures, indicating they generally complete tasks without obvious breakdowns, with errors being more subtle and capability-driven. In contrast, gemini 5-flash and gpt-4.1 have lower implicit failure rates (52% and 66%, respectively). suggesting more explicit causes. For explicit failures, gemini-2 5- flash and gpt-4 1 mainly experience abandoned premature stop errors, reflecting weaker reasoning and planning. gemini- 2. 5 Flash also shows a higher in- cidence of malformed tool calls (around 10%), possibly due to mismatches in tool-call conventions

---

Published as a conference paper at ICLR 2026
insufficient training. gpt-5-high has more context window overftow errors, indicating difficulties with cong-context handling, while -imi-kk-instruct faces frequent turn limit exceeded errors, often due to repetitive tool-calling loops. These results suggest that explicit errors are model-specific, highlighting the need for targeted improvements in reasoning, context management, and tool use.
5 RELATED WORK
LLM Agents. With the development of large language models (LLMs) (Team, 2025; Anthropic, 2025a; Team et al., 2025; OpenAl, 2025c; Comanici et al., 2025), LLM agents have progressed from early prompting methods such as ReAct (Yao et al., 2023b), which integrated reasoning traces with tool actions, to more structured designs like MetaGPT (Hong al., 2024) that coordinate multi-agent collaboration through explicit role assignment. This evolution has been supported by research on tool use (Schick et al., 2023; Qin et al., 2023; Patil et al., 2024), which explore when and how models should call APIs, as well as planning and reflection methods (Yao et al., 2023a; Shinn et al., 2023; Wang et al., 2024a) that improve robustness in multi-step workflows. Multi-agent frameworks (Wu et al., 2024; Li et al., 2023; Chen et al., 2023) further demonstrate the benefits of coordinated division of labor. In applied domains, coding agents (Yang al., 2024; Wang et al., 2024b) enable real repository interaction; GUI and computer-use agents are advanced by benchmarks (Zhou et al., 2023; Deng et al., 2023; Xie et al., 2024); and deep research efforts are represented by initiatives (Wei et al., 2025; Starace et al., 2025; Du et al., 2025). Together, these developments illustrate the trend toward general agents that can operate across heterogeneous systems and contexts, naturally pointing to the need for standardized protocols such as the Model Context Protocol (MCP) (Anthropic, 2024) that provide a unifying interface for tool and environment integration.
Benchmarks for evaluating MCP use. Recent work has begun to systematically benchmark agent performance in MCP-enabled settings (Yan et al., 2025; Liu et al., 2025; Mo et al., 2025; Gao et al., 2025). MCP-Universe (Luo al., 2025) constructed tasks across multiple domains and evaluators, revealing the difficulty models face with long and dynamic workflows. LiveMCP-10) (Yin et al., 2025) focused on multi-tool interaction and execution-plan validation, while MCP-AgentBench (Guo et al., 2025) scaled up evaluation with hundreds of tasks spanning diverse servers and tools. These efforts primarily emphasize broad tool coverage or easier execution but leave gaps in assessing high-fidelity workflows tied to realistic application environments. Our proposed MCPMark addresses this by designing tasks with diverse CRUD operations in containerized settings to ensure safety and reproducibility. Each task is paired with programmatic verification scripts and full environment state tracking, enabling reliable and fine- mme-grained evaluation.
6 DISCUSSION ON LIMITATIONS AND FUTURE DIRECTIONS
Our task creation pipeline, while ensuring task quality, is difficult scale. This creates bottleneck for producing the large-scale training data needed to advance the field. Furthermore, the steep difficulty of many tasks in MCPMark limits its utility for evaluating and guiding the development of smaller, more efficient models. Future work on the benchmark should therefore focus on introducing more fine-grained difficulty gradient, potentially through remi-automated task generation and reduced task execution chain. Additionally, to better reflect real-world complexity, the benchmark could be expanded to include tasks with ambiguous user intent. This would test an agent's ability to ask clarifying questions or infer the user's actual intent. Finally, incorporating wider variety of MCP servers could also help challenge agents across more diverse set of digital tools.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This section outlines how we address the ethical considerations involved in the construction of our benchmark, which includes several key components that could raise ethical concerns:
Initial State of MCP Environment: Each initial state and environment used in the benchmark is provided with the appropriate license information (see Appendix for details). A few environments were self-curated, and for these, we have ensured transparency and compliance with relevant licensing requirements, promoting ethical usage.
All tasks included in the benchmark were collaboratively annotated by both experts and AI agents. The experts involved in the curation process have been properly recognized as co-authors in the author list, ensuring that their contributions are duly acknowledged. Addi- tionally, the licenses for the agents used, including Claude Code (License) and Cursor (License), are provided to ensure that all resources are used responsibly and in accordance with the relevant licensing terms for research purposes.
Task Curation:
The licenses for each specific MCP server used the benchmark are provided in Appendix C. This ensures that all external systems and tools are properly licensed for research and evaluation purposes.
MCP Servers:
By adhering to these practices, we ensure that high ethical standards are maintained throughout the construction of the benchmark, and that all resources are used responsibly and in accordance with relevant regulations.
REPRODUCIBILITY STATEMENT
In order ensure the reproducibility of our experiments, we have made available the evaluation code, task data, and corresponding run instructions in the supplementary materials. The evaluation code has been modified to remove any identifiable personal information to protect privacy. Additionally, the tasks and data used for evaluation included in the supplementary materials along with detailed instructions on how to execute the experiments. This ensures that other researchers can replicate our results under the same conditions while adhering privacy standards.
ACKNOWLEDGEMENTS
This project was partially supported by the Singapore Ministry of Education Academic Research Fund Tier (Award Number: T1 25 251RES2514).
REFERENCES
Anthropic. Introducing the model context protocol. http://wwwwatttrrooo.oooo////eee model -ontexxtt cenntexx-prooooool November 2024. Accessed: 2025-06-30.
Anthropic. Claude opus 4.1. https: /ttp://wwwww//..// August 2025a. Accessed: 2025-08-06.
Anthropic. Introducing claude 2025b. Accessed: 2025-07-28.
May
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors in agents. arXiv preprint arXiv:2308. 10848, 2(4):6, 2023.
Gheorghe Comanici. Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 2025.
11

---

Published as a conference paper at ICLR 2026
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-281 14, 2023.
Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506. 11t.:.6666.1163,, 2025.
Xuanqi Gao, Siyi Xie, Juan Zhai, Shqing Ma, and Chao Shen. Mep-radar: A multi-dimensional benchmark for evaluating tool use capabilities in large language models. arXiv preprini arXiv:2505. 16700, 2025.
Zikang Guo, Benfeng Xu, Chiwei Zhu, Wentao Hong, Xiaorui Wang, and Zhendong Mao. Mep- agentbench: Evaluating real-world language agent performance with mep-mediated tools. arXiv preprint MXX1:22509.09334. 2025.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for multi-agent collaborative framework. International Conference on Learning Representations, ICLR, 2024.
Xinyi Hou, Yanjie Zhao, Shenao Wang, and Haoyu Wang. Model context protocol (mcp): Landscape, security threats, and future research directions. arXiv preprint arXiv:2503. 23278, 2025.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770 2023.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 11451991-52000 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Zhiwei Liu, Jielin Qiu, Shiyu Wang, Jianguo Zhang, Zuxin Liu, Roshan Ram, Haolin Chen, Weiran Yao. Huan Wang. Shelby Heinecke, et al. Mcpeval: Automatic mep-based deep evaluation for ai agent models. arXiv preprint arXiv:2507. 2t..225077.226.,. 2025.
Ziyang Luo, Zhiqi Shen, Wenzhuo Yang, Zirui Zhao, Prathyusha Jwalapuram, Amrita Saha, Doyen Sahoo, Silvio Savarese, Caiming Xiong, and Junnan Li. Mcp-universe: Benchmarking large language models with real-world model context protocol servers. arXiv preprint arXiv:2508.14700 14704 2025.
Guozhao Mo. Wenliang Zhong, Jiawei Chen, Xuanang Chen, Yaojie Lu, Hongyu Lin, Ben He, Xianpei Han, and Le Sun. Livemepbench: Can agents navigate an ocean mep tools? arXiv preprint arXiv:2508.0/77 0/780, 2025.
OpenAl. Introducing gst-oss. https: August 2025a. Accessed: 2025-08-14.
OpenAl. Introducing gpt-4.1 in the api. httphh/w.w.......ggg..ttaaaaaaaa 2025b. Accessed: 2025-07-28.
April
OpenAl. Gpt-5 system card. https: /tttp://cch.oppeaa.. http://///cc.ogaaaaa.c.c////g/ge
August 2025c. Accessed: 2025-08-13.
OpenAl. Introducing openai o3 and 04-mini. introducing--3 -44-334-------------------
April 2025d. Accessed: 2025-07-28.
OpenAl. Codex: Lightweight coding agent that runs in your terminal. https://githin: httpgggttthhooooo/
opena /odexx 2025e. Commit snapshot as of 2025-11-27.
12

---

Published as a conference paper at ICLR 2026
Shishir G Patil, Tianjun Zhang. Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. Advances in Neural Information Processing Systems, 37: 126544-126565, 2024.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, al. Toolllm: Facilitating large language models to master 16000+ real- world apis. arXiv preprint arXiv:2307. 16789, 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zet,lemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to tools. Advances in Neural Information Processing Systems, 36:68539-68551, 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, (6626634-665,, 2023.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, al. Paperhench: Evaluating ai's ability to replicate ai research. arXiv preprint ar arXiv:2504.0/848, 2025.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint 1rXiv:2507.20534 2025.
Qwen Team. Qwen3 technical report, 2025. URL
Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. Appworld: controllable world of apps and people for benchmarking interactive coding agents. arXiv preprint arXiv:240.118011, 2024.
Xingyao Wang. Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. In Forty-first International Conference on Machine Learning, 2024a.
Xingyao Wang. Boxuan Li, Yufan Song, Frank Xu, Xiangru Tang. Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint 24X.:2407.11741 2024b.
Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504. /2516, 2025.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen Ilm applications via multi-agem conversations. In First Conference on Language Modeling, 2024.
/ xAI. Grok 4. http::////.aaai///////e/ee
July 2025. Accessed: 2025-07-28.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040-52094, 2024.
Yunhe Yan, Shihe Wang, Jiajun Du, Yexuan Yang, Yuxuan Shan, Qichen Qiu, Xianqing Jia, Xinge Wang, Xin Yuan, Xu Han, et al. Mepworld: A unified benchmarking testbed for api, gui, and hybrid computer use agents. arXiv preprint 2500:2500.00672. 2025.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Advances in Neural Information Processing Systems, 37:50528-50652 2024.
interfaces enable automated software engineering.
13

---

Published as a conference paper at ICLR 2026
Shunyu Yao. The second half. 2025.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances neural information processing systems, 36:1 1809-11822, 2023a.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models. International Conference on Learning Representations (ICLR), 2023b.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.1200 2024.
Ming Yin, Dinghan Shen, Silei Xu, Jianbing Han, Sixun Dong, Mian Zhang, Yebowen Hu, Shujian Liu, Simin Ma, Song Wang, et Livemep- 101: Stress testing and diagnosing mep-enabled agents challenging queries. arXiv preprint arXiv:2508. 15760, 2025.
Zai. Glm-4.5:
Reasoning, coding, and agentic abililties. https://2.ai//looee
July
2025. Accessed: 2025-07-28.
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: realistic web environment for building autonomous agents. arXiv preprint 2rXhv:2307.33844 2023.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
TABLE OF CONTENTS
A Use of LLMs
15
Details of the Task Creation Pipeline
16
C MCP servers
18
D Detailed MCP Benchmark Results
19
Case Studies by MCP
24
Cost and Turn Distribution
39
G Turn Distributions across MCP Services
40
H Initial States Selection and Licenses
42
H.1
Notion Templates
42
H.2
GitHub Repositories
42
H.3
Playwright Usage
42
H.4
Filesystem Components
42
H.5
PostgreSQL Databases
43
I Additional Experimental Results
43
A USE OF LLMS
Large language models were used as the generation of some initial environment states. Specifically, claude-code and codex prompted to produce structured directory layouts. starter scripts, or database schemas that served as initial configurations in selected tasks. These outputs were carefully reviewed and refined by human experts to ensure correctness, realism, and alignment with benchmark goals.
assistants to support writing, debugging, and
LLMs also helped improve the grammar, flow of writing, and occasionally assisted in resolving minor coding issues such as syntax errors or implementation quirks. All core ideas, benchmark designs, implementations, experiments, and analyses were independently developed by authors. No part of the scientific contribution or methodological reasoning was generated by an LLM.
15

---

Published as a conference paper at ICLR 2026
B DETAILS OF THE TASK CREATION PIPELINE
We use Playwright example to illustrate the guideline for human experts and the initial instruction/prompt for the task creation agent. These are simplified for reference.
Guideline (Playwright)
Step 1. Select the starting environment
Pick a website or web app as the initial state. Prefer a staging or test instance to avoid side effects. Examples: Reddit-like forum Shopping Admin dashboard.
Step 2. Configure the agent environment
In Cursor or Claude Code, set up the MCP server stack and include the Playwright MCP server the agent can control a browser.
Step 3. Define an initial question or topic
Write a seed quest on or topic that will guide agent exploration and task creation. It can be broad moderately specific.
Step 4. Create and refine the task
Step 4.1. Exploration with the agent
Have the agent read the initial instruction (which includes the seed question), then explore the target site together with the agent. Based on the collected con- text, ask the agent to propose task that fits the objectives and requirements.
Step 4.2. Provide feedback to improve the task
Guide the agent revise the task as needed. Examples:
If verification is weak: "This task is not sufficiently verifiable. Please revise make verification clearer and more reliable."
If exploration lacks coverage: "You can explore deeper collect more diverse and detailed information."
If subtasks feel disconnected: "Make the subtasks integrated rather than unrelated."
Step 4.3. Save the task
Store the task description and the verification script as separate files. Use consistent folder structure based on category and name. Follow well-structured prior examples for formatting.
Step 4.4. Human-in-tian in-te---o- adjustments
Iterate between the agent and the reviewer until both the task description and the verification script meet quality standards.
Step 5. Execute and verify Run the task with Playwright MCP reach the final state, then run the verification script. Stress-test the checker to confirm:
Step 5.1. The task executable end to end.
Step 5.2. Pass or fail is clear and objective.
Step 5.3. The script flags both correct and incorrect outcomes, including edge cases.
Step 6. Assess difficulty (optional)
If the task and checker pass, consider whether difficulty is high enough test the model. Adjust scope or constraints if needed.
Notes. These steps target experts working with Cursor or Claude Code. They are guidelines. issues appear, collaborate with colleagues to debug efficiently.
16

---

Published as a conference paper at ICLR 2026
Initial Instruction for Task Creation Agent (Playwright)
Your job is to:
1. First explore the web environment to understand available MCP tools and capabilities.
Generate one challenging, verifiable, and realistic task based on your collected information.
Focus your exploration and task generation on the following specific topic or question:
Use this guiding theme for creating more targeted and relevant tasks.
Ensure the task addresses different aspects or components related to this requirement.
Playwright MCP Tools Reference:
splaywright_mcp_r [contents of docs docs/playue http//paaaaiiaa/mam mepttt-mep-aeee camm-ppppitttote introduction. md go here] </playwright__mp__oo
Output Format:
"tasks":
"task_id"; "task_1",
"description", "Clear, conversational task description", "difficulty": "hard",
pverification_criteriie l'criterion 1", "critericn 2"I,
"expected mep_ calls": "bbrowser_navigate" "browser_ snapssott, "browser_click").
restimateddcoompleiii "high"
Based on the given web application environment, write one challenging, verifiable, and realistic browser automation task that aligns with users' actual web interaction workflows. The goal is evaluate an Agent's ' ability to use Playwright MCP tools effectively. Requirements:
Difficulty: The task should be really hard (omitted)
Verifiability: Avoid open-ended outcomes (omitted)
Authenticity: Describe the task in a natural, conversational tone ... (omitted)
Context Awareness: Leverage page structure, form elements, navigation patterns, ... (omitted)
Start by exploring the web application environment using MCP tools to understand the current structure, interactive elements, and user workflows, then generate a task that combines:
1. Your real-time MCP exploration findings.
The specific website structure and interactive elements you discover.
focus on browser automation operations that require multiple Playwright MCP tools rather than only content reading.
4. The specific focus area: _seed_topics
Please explore thoroughly before creating the task. Consider:
Form elements and input fields.
Navigation patterns and menu structures.
Dynamic content and interactive features.
User workflow patterns.
Authentication and session management.
Data submission and validation processes.
17

---

Published as a conference paper at ICLR 2026
C MCP SERVERS
We relied on five Model Context Protocol (MCP) servers in our setup. Below we summarize their functionality, invocation, repository, and license.
Filesystem. The filesystem server provides local read, write, and directory operations over the host file system. It is invoked as attoce/llcottettpprrooooo///eeee Eeeeooooooeeeeeeeeeeeeeeeeeeeeee The implementation is hosted at gritbb..coommoeeccooeeeeooooooee under the MIT License.
GitHub. The GitHub server integrates with the GitHub API to manage repositories, issues, and pull requests. The endpoint used is https: /api. githubcopilos com/ mcp/. The code is available at httthuub.oomggihuugghiiiee mee-ssee,,,,ereerr, released under the MIT License.
Notion. The Notion server allows interaction with Notion databases and pages. It is invoked as @notionhq/ /ooooonggoo//o///ooooooooooooooo itt//ooiiinnn inntiionnn -on-inp------- cnn-mp-seeeer. server. The official repository github.commmalennttoonnooiinn mep-server, licensed under the MIT License.
Playwright. The Playwright server enables browser automation and scripted web work- flows. It is started using httpaaywiiggt//p@@aaeeeee The source code is provided at suthub.coomnicrooo///ayyyynnnnno distributed under the Apache License 2.0.
PostgreSQL. The PostgreSQL server provides access to relational database through SQL queries. It is launched with postgres-mcy -access mode=unrestri The implementation is maintained at github. hatp.//c////taaab//ooooggoo mop, and released under the MIT License.
18

---

Published as a conference paper at ICLR 2026
D DETAILED MCP BENCHMARK RESULTS
Tables and presented the overall success rates and usage statistics, aggregated across all MCPs. Here we provide the corresponding breakdown by individual MCP from Table to Table 8. #Input and #Output are measured in thousands of tokens (K), and Cost is reported in USD. For success metrics, bold and underline indicate the best and second-best results, respectively. For usage statisties, bold and underline denote the largest and second-largess values, without implying better performance.
Table
Filesystem MCP benchmark results.
Metries
Per-TTask Avg Usage
Model
Pass@1
Pass@4
Pass
Input
# Output
Cost
Turns
Tool Calls
Proprietary Models
gpt-5-mest ppt----mdium
57.50 2:163
76.67
36.67
215.96
17.38
01.4
10.06
21.07
grok-4
50.83 20:40
73.33
26.67
247.33
10.70
0.90
10.80
16.87
#2.76
26.67
17.79
1.52
28.79
27.80
gpt-5 1ot=5-mini--i mnini-medium
33.33 #6,24
53.33
10.00
398.34
12.58
0.12
14.84
36.93
claude-opua-
33.33 #0.00
272.17
4.37
4.41
16.37
15.40
claude-sonnet-1
27.50 #2.76
50.00
6.67
302.21
4.00
0.97
16.02
15.08
04-mini
25.00
36.67
13.33
293.34
15.89
0.39
20.88
19.88
gemini-2.5-pro
24.17
43.33
10.00
214.97
7.75
0.65
14.35
14.72
grok-tode-fast-1
23.33 #7.45
40.00
10.00
276.40
2.36
0.06
16.38
16.77
9pt-4.
14
20.00
3.33
143,95
1.81
0.30
9.28
18.48
gemini-2.5-ff.e 5-.f-
8.33 11.00
13.33
6.67
67.64
7.57
0.04
6.50
11.15
ept-B-nano-mediicm
6.67 #5.27
0.00
19.53
20.75
27.76
9pt-4.1-mini
3.33 $:000
gpt-4.1-nano
0.00 8:000
0.00
0.00
116.98
1.32
0.01
12.17
15.32
Open-Source Models
decpseek-v3.2
15.83
26.67
6.67
421.33
3.38
0.24
23.83
23.12
imi--22-instruet--000
14.17
23.33
6.67
696.79
4.47
0.43
26.27
25.70
qpen3-coder-plus
13.33
26.67
3.33
972.41
4.15
0.20
28.23
27.32
qwen3-max
10.83 11.44
13.33
10.00
389.56
2.87
0.48
19.27
18.39
glm-4.5
7,50 1:144
13.33
3,33
193.95
3.92
0.07
16.39
17.09
ept-oss-1200
5.83 1:433
16.67
0.00
19.75
1.08
0.01
4.62
3.62
19

---

Published as a conference paper at ICLR 2026
Table 5:
GitHub MCP benchmark results.
Metrics
Per-Task Avg Usage
Model
Pass@1
Pass@4
Pass
Input
# Output
Cost
Turns
Tool Calls
Proprictary Models
gpt pp:-S-medium
47.83 1::11
65.22
17.39
659.73
20.57
1.03
14.33
21.23
cl claude-opus44,
21.74 4:000
620.63
5.84
9.75
10.78
10.13
5oz-5-mini-medd
18.48 47776
34.78
4.35
614.68
7.71
0.17
13.92
17.28
claude-sonnet-4
16.30 #5.05
696.81
4.44
2.16
11.16
10.50
gemind-2.s-fleah
15.22 12217
21.74
8.70
1107.04
12.70
0.36
10.46
17.71
grok-4
14,13 #3.6
21.74
8.70
804.50
1.93
2.44
12.98
16.76
04-mini
14.13
26.09
4.35
510.13
8.74
0.60
10.92
10.08
o3
14.13 1:36
21.74
4,35
451.18
3.56
0.93
9.20
8.24
gemini-2.5-p:o
9.78
21.74
0.00
173.43
5.75
0.52
5.45
6.29
grok-code-fast-
8.70 #5.32
17.39
4.35
751.41
6.50
0.16
17.85
17.28
up--S-nano-medium
18.88
13.04
0.00
751.62
26.77
15.15
17.63
9pz-4.1
#1.88
4.35
445.88
2.49
0.91
9.95
14.97
9pa-4.1-mini
6.52 #6.52
0.00
466.70
1.51
0.19
12.00
14.63
9p:-4.1-nano
0.00 16:00
0.00
312.86
2.59
0.03
9.27
11.04
Open-Souree Models
glm-4.5
22.83 #6.43
34.78
13.04
482.00
3.65
0.16
11.92
11.04
qwen3-coder
19.57
34.78
13.04
1987.14
3.36
0.40
19.12
18.13
iimi-k2-inst-rutt-0999
16,30 1:188
26.09
8.70
995.65
8.25
0.62
23.68
23.23
gwen3-max
14.13 #:361
17.39
4.35
1348.13
2.55
1.63
26.70
25.78
despsesk-v3.1
9.78 #1.88
13.04
8.70
362,36
2.24
0.21
9.46
9.22
9pc-uss-120b
4.35 13.00
8.70
0.00
76.30
1.41
0.01
4.62
3.62
20

---

Published as a conference paper at ICLR 2026
Table 6:
Notion MCP benchmark results.
Metrics
Per-Task Avg Usage
Model
Pass
Pass@4
Pass
Input
# Output
Cost
Turns
Tool Calls
Proprictary Models
gpt - 5-medium
41.96 1:2.9
50.00
32.14
375.04
31.62
0.79
12.94
21.60
al aldde-opus-4,1
35.71 #:000
638.06
3.93
9.87
17.04
16.04
24.11 2:300
7.14
224.93
9.47
0.53
13.72
12.72
21.43 #5.05
7.14
4.24
2.00
19.71
18.71
04-mini
20.54 8:585
42.86
7.14
267.63
25.97
0.41
15.29
14.29
pps-S-miin-modiimm
16.07 #5592
32.14
3.57
705.09
12.34
0.20
14,60
17.28
gemini-2.S-flem
6.25
21.43
0.00
201.00
6.58
0.08
6.11
9.61
9pt-4.1
6.25
14.29
0.00
135.55
1.37
0.28
8.58
11.82
gemini-2.5-pe
4.46 17.96
7.14
0.00
212.92
7.13
0.64
7.12
8.67
gp:-G-nnano-mediuu
3.57 50000
3.57
3.57
204.32
32.08
0.02
7.46
8.74
grok-4
#1.55
0.00
678.64
13.04
2.23
20.14
24.80
grok-code fost---cde-fat-1
#1.55
0.00
561.49
7.26
0.12
20.27
20.09
spa-4.1-mini
1.79 #1.79
0.00
262.75
1.35
0.11
12.57
14.56
8p:-4.1-nan9
0.00
0.00
93.38
1.40
9.64
10.93
Open-Souree Models
glm-4.5
21.43
32.14
10.71
625.97
5.04
0.21
22.15
21.17
qwen3-coder
19.64 #6.44
39.29
7.14
796.73
2.75
0.16
21.07
20.23
qwen3-max
16,96 14.64
25.00
3.57
973.92
3.66
1.19
26.57
25.63
deepseek-v3.1
12.50 1:300
28.57
0.00
503.35
2.20
0.29
17.94
17.40
kimi-k2-innstrctt-099
8.04 #2.96
10.71
3.57
1117.21
5.20
0.68
33.55
32,72
ep:-uss-120b
3.57 11.55
14.29
0.00
68.31
1.72
0.01
5.49
4.49
21

---

Published as a conference paper at ICLR 2026
Table
Playwright MCP benchmark results.
Metries
Per-Task Avg Usage
Model
Pass
Pass@4
Pass
Input
Output
Cost
Turns
Tool Calls
Proprictary Models
gpt-5-medium
43,00 1:500
56.00
36.00
1807.17
21.79
2.48
23.78
22.96
grok-4
35.00 67
1264.91
3.89
23.02
claudd-sonnes-4
26.00 1.000
1241.92
3.78
19.80
19.12
9Eok-oode-East-1
25.00 2:175
36.00
8.00
1157.72
7.17
0.24
18.23
18.18
c
24.00 20.00
1146.05
2.88
17.41
19.04
18.40
gemini-2.5-pro
15.00
32.00
4,00
1696.44
5.58
4.32
19.15
18.33
03
15,00 1:530
32,00
8.00
556.30
4,46
1.15
16.30
15,40
04-mini
12.00
28.00
0.00
862.51
18.07
1.03
17.70
16.93
gpt-5-mini-mett -in-miiiimmddimm
12.00
24.00
4.00
1814.94
8.55
0.47
22.75
22.04
gpt-4.1
8.00 #2.53
12.00
4.00
859.77
0.86
1.73
13,80
15.21
Remini-2.5-flant
6.00 3:200
12.00
0.00
38388.93
8.21
1.17
26.33
38.78
2ot-5-nano-med.
0.00 30000)
0.00
71..95
17.71
0.04
18.52
17.55
gpt-4.1-mini
0.00 $0000
4959.14
1.99
31.33
31.52
9pt-4.1-nano
0.00 3000
389.80
13.61
Open-Source Models
gwed3=coder-piuu
30.00 1:4.47
48.00
8.00
2851.57
2.39
0.57
21.21
20.40
kimi-k2-Lnstruct-O0
30.00
40.00
20.00
1358.02
2.17
0.82
20.64
19.79
g1m-4.5
13,00 #3332
20.00
4,00
582.73
2.76
0.20
15.36
14.61
qwen3-max
8.00 1:000
12.00
4.00
2297.67
1.16
2.76
27.83
27.41
deepseek-v3.1
7.00 +3332
16.00
0.00
836.01
1.77
0.47
19.09
20.78
gpt-oss-120b
3.00 4:173
4.00
0.00
139.33
1.27
0.01
7.21
6.26
22

---

Published as a conference paper at ICLR 2026
Table 8:
PostgreSQL MCP benchmark results.
Metries
Per-TTask Avg Usuge
Model
Pass@1
Pass@4
Pass
Input
# Output
Cost
Turns
Tool Calls
Proprietary Models
ept- S-memium
76.19 #7.53
100.00
47.62
113.35
17.04
0.31
13.37
12.45
gpt-S-mini --medimm
61.90 25:3
90.48
28.57
115.40
9.27
11.77
10.77
grok-4
#7.81
80,95
38.10
186.07
8.23
17.89
17.08
-laude-sooneetd
53.57 #6.39
71.43
38.10
331.10
7.54
26.80
25.81
cLoK-code-tast-1
47.62 2:.76
61.90
28.57
226.41
5,46
0.05
19.70
18.70
36.90 #3.95
66.67
14.29
63.56
4.72
0.16
10.71
9.71
t clauie-opus-4.1
33.33
260.68
9.80
4.64
24.86
23,86
gemin -
26.19 $7.00
47.62
9.52
39.74
8.91
0.23
7.45
6.45
opt-S-meno-medii.m
15.48
28.57
4,76
105.02
23.04
0.01
9.46
10.15
04-mini
11.90 #4412
19.05
4.76
15.92
5,76
0.04
5.06
4.06
gemini-2.5 -2
10.71 $6.19
23.81
4.76
46.08
9.93
0.04
8.76
11.38
9pt-4.1-mini
9.52 $3.37
14.29
4.76
1.78
0.02
9.77
11.61
gpt-4.1
4.76 $0,00
4.76
55.11
1.20
8.12
10.54
gpt-4.1-nano
0.00 1:000
0.00
0.00
71.06
2.43
0.01
8.73
10.18
Open-Source Models
qwnn3-ooeer-plus
47.62
61,90
38.10
$73.90
5.13
0.12
29.00
28.00
k
imii-t2-instrrctt-99933
47.62
66.67
28.57
441.16
5.38
0.28
30.21
29.25
gwen3-max
44.05
52.38
38.10
192.13
4.91
0.26
18.88
17.92
deepseek-v3.
42.86
61.90
28.57
316.60
4,65
0.19
26.48
25.49
g1m-4.5
14,29 #7.53
23.81
0.00
204.61
5.14
0.07
25,39
24.40
opt-oss--20b
7.14 #2338
23.81
0.00
21.36
1.42
0.01
5.07
4.07
23

---

Published as a conference paper at ICLR 2026
E CASE STUDIES BY MCP
Filesystem Contact Information
Task Description
Your suic rpll connact affrrmtt or from all the the 1012 single cable You reed estrcc al people contret formaric 1 orpariie systomeciolly
Task Objectives
strwtary
Fint oak une Nome (mgganed)
delondl ireaa (ocoure))
Theckcium thone fimes roa))
contact information found
information by marping typm into singie
Mank spaciic elommut avalianle parsonio gemast
Expected Output
Inntect Lefe.
Furmet with hedder ditartw
Reasoning Task
oveiing contact, ane yux to srwerr What Charbe Devti's jebppreesniioe
for contec infernttinne contect infocis
Important Notes
De rood cry sting
Only uneit Inc Mile centact. (eft: and umwer
Initial State
book list.txt
epesseu.ov .on
bookmark eeppott.tt
Downbouh
caleu tat
pióo www.gaatjjarssssscoo .ao
Persozal
info 222220222
correspondence 2twp:ooddeneee222..xxe 2023.txt
Projects
bulget tracker.car
Rool
draft lettentxt
Documeuts
Wick
dicut listesv
budget.ss
iunsbort car
emergency contacts. txt
example.txt.
Destsoo
contacto cop
baciap
Aechiivss
experiment results.txt
lox, vw__dccanetts_oon 22tpmntt___2222oo .oo
Figure 6: Task sheet and initial directory tree for the Filesystem case; trajectories are in Figures 7-8.
24

---

Published as a conference paper at ICLR 2026
Model Trajectory claude-sonnet-4
daude-sonnet-4
18 turns
I'll help you compile all contact informati from files into CSV Let start by exploring the directory structure undarstand what files are available.
filesystem List Llwwed dirrcctoress
output result
Allowed cirrettoiess: fapp/mcpmark, happ:/mmppssskbcccuuusssssuuuup httpp//ww.e.....e..oo.ooooooooee
filesystem directory_tres _tree
by reading most likely
to contain contact information:
filesystem read, ruttiple. Files
Let also read some other files that might contain contact information:
filesystem read, ip le_ Files
remaining
sure don't miss any contact information:
filesystem pea_rrultiple. pettttppeerreeeeeeeeeee_____ Lles
Let me check few more files to be thorough:
filesystem read rultiple tiles
check remaining text files:
filesystem read_rultiple. files
Verifier Result
Contact info CSV Exists
Answer TXT Exists
PASS
Files in Correct Locations
Correct CSV Structure
Answer Content
Figure 7: Successful run by claude Claude-sonnett 4: extracts contacts, writes CSV and answer file.
verifier passes.
25

---

Published as a conference paper at ICLR 2026
Model Trajectory gemini-2.5-pro
gemini-2-5-pro
turns
filesystem search_"lles
Okay, understand, will start by listing all the files the current directory to identiff the files containing contact information. Then, will read each these files extract the necessary data. After collecting the contact details, compile
consolidated and formatted correctly with the required headers. Finally, will aza the compiled data to determine Charlie Davis's joh/profsooooiiii and save this answer in an answer. txt file.
single contact, info..cvv ensuring
filesystem List directory
need compile contact information from all the files in the current directory, create CSV file with it, and then question based the compiled dete. However, search_files and List_directory commands failed. start by listing the allowed directories to understand the file system's constraints.
filesystem Ltst at Lowed di rectories
filesystem Hist_dErectory
filesystem directory_tees
filesystem read ruitipte Tiles
filesystem write_file
filesystem writa_file
filesystem wrlte_file
Done.
have created the contact_info... info.csvv file with all the contact information and the answer.txt file with the answer your question.
Verifier Result
Contact info CSV Exists
Answer TXT Exists
FAIL
x
Files in Correct Locations
Correct CSV Structure
Answer Content
Figure 8: Failed run by gemini-2 -2. 22.5 22.5prr 5-pro: files are created but CSV/answer content is incorrect, verifier fails.
26

---

Published as a conference paper at ICLR 2026
GitHub- Linting CI Workflow
wordfow
standarde anferced repuerz Hertt what nead
Step Create Lindnng Corfigaation branch Cruns row branch calked cledo workfow foom branch.
Step 2: Create Corfigaaaionn new brendh creite the file Leatt
not with
"noce"
festerde::
the tre-anundiveett: "arrer,
cosatee "here'
it Create dithes Actione Livting muricfiew Coutte the toteettt wotTiet wiy
Wardfoww "Coda
Teguine paalttorain
plobelly
Step ae Create File That will tnd Linking Create the lire Srt/eeerstee 15 with ineezional hes eg volatorn that will cause the creck feil.
Create
tök.in conig, workfiww fils,
coldd
Thde Sae ESki norddion for code cuality bxtormmen"
Body must dude:
heading
Step & Fie Linting Brirr and Update FR the the lnnding emarn in Snicteueette.. coment the change in singls commit to spdete the mitz thet the clicedk puovess
Initial State
Seerty
MOCE
MCPMark CI/CD
light veight Vork Faprr suaiiater
Figure 9: Task sheet and initial repository snapshot for the GitHub case; trajectories are in Figures 10- 11.
27

---

Published as a conference paper at ICLR 2026
Model Trajectory gpt-5-medium
gpt-5 5-mciium
tums
thub creatie, creath_branch branch
arguments
owwner: "mcpmark-evaa''
repo": "ncpnark-cicd"
"branch": "CL/add-es( Lnt-wworkloww,
'fron_branch branch"; "na
autputtresut
"ntaa: null, content"; ["
"tyee": "text"
"[V"refv"s "(""ref":\\"effs///"""""" 'annotations", null,
"t null,
"sError" false
github push. files
github greate_pull_requess
github fush_files
All set. compeete all steps and created a pull request, then fixed the linting violations and updated the PR so should pass CI now. Here's what did:
Step Created branch glladdeslint-workfffe Added esl eslintrc.json the repo root with the specifled configuration.
2:
3: Added GitHub Actions workflow at github/worrlloossiiitttttt:
Linting
Triggers: push pull. request targeting
Verifier Result
CI branch exists
.eslintrc.json config
giithub/worfoow//intt.ttl
PASS
Linting PR exists
Workflow 1 fail pass
Linting error fixed
Figure 10: Successful run by gettgppteeet 5-mediumm um: branch, ESLint config, workflow, and PR are created; CI run fixes lint errors; verifier passes.
28

---

Published as a conference paper at ICLR 2026
Model Trajectory qwen3-coder-pluu
awen-3-coder-pluue
help you set up proper linting workflow your CI pipeline. Let start by creating necessary branch files.
First, fill create the new branch 'c//add-esiil fehadd-eslin--vvee -slknt-workffoww from the main branch.
github createe branch
Now fil create the ESLint configuration file
Lestintre. the new branch.
github create, update. file
Next, Illl the GitHub Actions workflow file
githhn//werkloowsiiitt.tw
for linting workflow.
github create.ppr pr wtu.udd___fill
Now create the example JavaScript with intentional linting violations src/example..j file.
github create_or peeatt_r__uddtte... _wp_ste__fllleee
Now f'll create a pull request with all the changes we've 'vve made so far. This will includo the ESLint configuration, the workflow file, and the example file with liiting errors.
github create putl, request
Now need to fix the linting errors in the src/example.) file and update the pull request that the CI check passes. me update the with proper formetting according ESLint rules.
github create. update.
need to get the SHA the existing file before can update Let me retrieve the information first.
github get_ Ltt,, contents
Let me try different approach to get the file information.
Verifier Result
CI branch exists
.eslintrc.json config
.github/worfoow//int.yy
FAIL
X Linting PR exists
x Workflow 1 fail pass
x Linting error fixed
Figure 11: Failed run by qwen3-coder-plus. partial setup leaves artifacts or CI incomplete, verifier fails.
29

---

Published as a conference paper at ICLR 2026
Notion Toronto Guide
Navigate page change calored (tagr colors) different
Requirements
Find and access the Toronto Guide page in Notion
Identify and change elements Including:
Pink tags databases
Pink calleue backgrounds
3. Change all pink colors to any other color your choice
Initial State
MCPMark Source Hub Toronto Guide
Share
*
Food
Toranto's food scene a delicious ref laction of hts cultural diversity. From sizzling street food to high end dining, the city serves up an endless array of giobal cuisines. Hore'' collection of must-try
city.
tng
Bybles Downtown
Emimas Country K tohen
Amal
Lebanese (nldal Eastem
Brunch
Leba
map..es..gregttiheu..u Wyyuha
mapoasp app goo AARPSI
mapp goo XXDTWWe
Pizzeria Badial
Taeeost
maps
Maha's Brunch
Bar Isabel
Patois
Fevnttnn (Dunchh)
Spernit
Carltteen
glai), lPoud
Tung
Cance
Hesks
Canadian
mappuaso-ggo.. Pi/...
rapx.sp gon slobo.. DoAiviss
mape apo goo glO...
Swan
Loga's Corner
Bosphonus
alfidt. To00ke
diQuel A2ND56
Campechano
Khao San Road
Sunny's Chinese
Medcan
Thai
Chinese
IT a50 goo www.:ottll...ttttteee.ee vigroe
TapA upo uoo wwt.//...........
Mimi Chinese
Crumbs Patties
Locla Indian Food Bar
North of Brooklyn
Dailo
(Nzza
Jalan
New
mapp--ao--goe goolll Alt J04V13
mops-app glWe Fiu/4us
Figure 12: Task sheet and initial Notion page/databases for the Notion case; trajectories are in Figures 13-14.
30

---

Published as a conference paper at ICLR 2026
Model Trajectory claude-opus-4.1
14--
I'll help you navigate to the Toronto Guide page and change all pink-colored elements to different colors. Let me start searching for the Toronto Guide page Notion.
not API t-search
Great! found the Toronto Guidee page. Let
identify any pink-colored elements.
-et--bl-llleeeee lock-chhldreen
Now let me check the chilcren of the column_ list block to find any callouts with pinkcolors: colors:
notion API-get-bloc k-chhtdrren
notion API-get-bloc Eldre
not on API-get-bi Ldren
check the toggle block's children:
API-get-bloc k-ch Eldren
Now, let me check the databases for any pink tags. Let me start with the Activities detabase:
not on MPL-retrieve-a-cie -aatt---e-
can see there are two pink tags the Activities database:
"Parks" with color "pink'
"Neighbourhood" with color "pink"
other
First, database:
notion
Verifier Result
Callout Block
Activities Database Tags
PASS
Food Database Tags
Cafes Database Tags
Additional Check
Figure 13: Successful by claude--pus- 1: updates callout and retags database items consistently, verifier passes.
31

---

Published as a conference paper at ICLR 2026
Model Trajectory -depssee-vv..1nnnn-hinnn
chaat
35 turns
I'll help you find the Toronto Guide page change pink-colored elements. by searching for page.
notion oeI-post-search
Great found the Toronto Guide page. Let retrieve the full page content to see its structure and icentify any pink
API-retr Leve-a-page
Now me get the block children to see the content of the Toronto Guide page:
notion API-get-be D2I-get-blackkeee block-cht Ldren
can see the Toronto Guide page has several child databases. Let me check each database to look for pink elements.
start with the Activities database:
API-retr Leve-a-catabase
Perfect! found pink elements in the Activities database. There are two tags with pink color:
1. 'Parks' (pink)
2. "Neighbourhood" (pink)
Let check the other databases well to there pink olaments:
notion API-retrieve-s aacababase
notion API-ritr API-retrieve-a--oe Leve-a-catabesee
notion API-retr Leve-e-cataaeeeee
Great! found several pink elements across the databeses:
Activities
'Parks' - pink
2. 'Neighbourhood" - pink
Verifier Result
Callout Block
Activities Database Tags
FAIL
Food Database Tags
Cafe Database Tags
Additional Check
Figure 14: Failed run by deepseek-v3 performs partial edits but misses required tag updates, verifier fails.
32

---

Published as a conference paper at ICLR 2026
Playwright Cloudflare Turnstile Challenge
Cloudflare Turnstile Authentication Challenge Task
Meywoghh comp furrat auchentcattor challange
Requirements:
pepmort siuuthhummtle
Liwmame: "wamuee"
Possadd 1soumer122
Dluutirr fumarta chalaang vidget completnly
Inocutt with Turstlle challnge wilget to on plane the ohthe cotios wooded)
War for aucoeerd eallenee comslecion (vidget an scues zome with checemeni)
So bent form cliclig gn tuit
Notes:
Use the provices tre credandats senuter pessvoret22
doea redined veppantt
silua care before proceeding
Initial State
Secure Sign In
Complete the security challenge to proceed with authentication
Username
Enter username
Password
Enter password
Security Challenge
Verify you are human
FLOUD
Prawyy Trmms
Sign In
Test credentials: testuser password123
Using test sitekey (interactive challenge required)
Figure 15: Task sheet and initial login page for the Playwright case; trajectories are in Figures 16-17.
33

---

Published as a conference paper at ICLR 2026
Model Trajectory - -3
03
27 turns
playwright prowsrr_nvvigee
playwright browser evaluate
playwright browser_Halt_fo
playwright brovser_evaluate
playwright browser_evatuae
playwright browser_evaluate
playwright browser_type
playwright brovser_evaluate
playwright browser_type
playwright browser_type
playwright snaapsrs
playwright brovser_evaluate
playwright browser evaluate
playwright browser_evaluate
playwright browser click
playwright browser_evatuate
playwright browser_evaluate
playwright browser_evaluate
playwright brovser_naviçate
playwright browser_netwerk_reuuee
playwright brovser_evaluate
playwright browser_evatuate
playwright browser_evaluate
playwright browser_type
playwright brovser_evaluate
playwright browser evaluate
Verifier Result
Credential Correct
PASS
Cloudflare Turnstile Passed
Figure 16: Successful run by o3: navigates login, fills credentials, passes Turnstile, reaches authenti- cated state, verifier passes.
34

---

Published as a conference paper at ICLR 2026
Model Trajectory -grok-4
grok-4
playwright browser_navigate
playwright browser_snapshot
playwright browser_type
playwright browser_type
playwright browser_snapshat
playwright browser_evaluate
playwright browser_.onslleeeesssaee
playwright browser_vatt_fe
playwright browser_evaluate
playwright browser_evaluste
playwright browser_evaluate
playwright browser_evatune
playwright browser_evaluate
playwright brovser_evaluste
playwright browser evaluate
playwright prosser_vvaluaee
playwright browser_evaluste
playwright browser. evaluate
playwright browser, evaluatee
playwright browser_evetuate
playwright browser_evaluate
playwright brouser evaluate
playwright browser_evaluse evaluate
playwright browser_evaluste
playwright browser. click
playwright browser network,
Verifier Result
Credential Correct
FAIL
Cloudflare Turnstile Failed
Figure 17: Failed run by grok-4: credentials entered but Turnstile not solved, verifier fails.
35

---

Published as a conference paper at ICLR 2026
PostgreSQL Employee Project Tracking
sture popur spec
support ong menagavent
Your Tasks:
1. Creete the project tredding tables
build cheee new dudiea the empleyess achame:
tettage cemeery outer oner
start cets teeacrotinn
or 2013 Hinsl
Just Ided DOZ!
CTST44 bare 1a0 defutt actwe)
project aeeignsees
snoig mant....... 13 Emsiges primary Auy.atto inveeed)
aeployes. Digint,net
age Lrange rit null
braeges check contrrl behwees
project
willedeenn Ortager prinsry ate-hoomeed
Entage ni
Idea,
Dookee debuit
forigg reladioneippe
cratect insigmetts molovee
enploven
analoyer, protects
Create performance Indeus
Indox
eartty iotbal
Into
50 lines not shown
Project Eutobeue Modemiasioor, start dete- 2024
Audyet-25000500
Initial State
employees department
employees www.rtteeetccrpooooo...oooo
employees dww.rterarrrooo manager
id
bpchar(4) NN
employos..k
sets RP
employoacjd
NN
dept
varchar(4t) NN
deppartment,p
(pchar(4)
cleparomen,
bpchar(4)
from,
from
NN
tn
date
dote
dece NN
amployeess.ud satary
Teck NW
3m DuTu
employess.co ride
barth
date MN
po,datt
cate NN
loyee,
first
vancher(1-0
185m
warchar (S0)
KN
last
vancher(10) NN
gender
engloyeen smployes_perder
dane
6ate
NR
Figure 18: Task sheet and initial schema for the PostgreSQI. case; trajectories are in Figs. 19-20.
36

---

Published as a conference paper at ICLR 2026
Model Trajectory grok-code-fast-1
grok-code-fast-1
postgres List_schomas
arguments
{
ouspuc result
nnta": null, content": ['
"type": "text"
"text": "[[ "[['schena nane": 'information_sen "schema_owwe _wrrmr:: 'postgres', 'sche "annotations null, "meta": null
StruccuredContennt' "result":
"tyse": 'text" "text": sschena nanc": iinformation inntiatttionccohomoe "schema_ownet owner': 'postgres'; 'sc "annotations" nult, meta": nutl
istrror": false
postgres list_objects
postgres det_alsect,eetatls
postgres detatccct_detttls
postgres set_object_detatts
postgres execute_sql.
postgres execute_sql
postgres execute_sql
postgres execute_sql
Verifier Result
Table Structures
Required indexes Found
PASS
Project Data Exists
Assignment Data Exists
Milestones Data Exists
Figure 19: Successful run by grok-code-fast 1: creates/updates tracking tables, adds indexes and seed rows, verifier passes.
37

---

Published as a conference paper at ICLR 2026
Model Trajectory grok-4
gpt-oss-
postgres List, schenas
postgres List_objects
postgres get, www.cotjecccco dedattls
ana ysisWW need to create three new tables employees schema, constraints, Indexes, Insert data, etc. create tables with sequences auto-increment (serial or generated). Use integer primary key auto increment: SERIAL? In Postgres, SERIAL creates sequence int4, Use "GENERATED DEFAULT AS IDENTITY". We'l write CREATE TABLE statoments.
Also need foreign key to employees.emploge employee.ld That's bigint id. So prgget_esigmmentteemmleeejee bigint references amployees.empoyyeellee project id references employee, projects.
Need to insert projects (three rows). Then assign employees based department. get employees per department: likely there is a department_enge eemplooyeee table linking employee to department. Let's inspect that table.
postgres
arguments
"object_ _amee:
"departnent, "ddeartteet__emttoyee""" eemployee", "object "ypct_type": "table" Scchema_namett "employées"
output result
"meta": null, "content"; ['
"type": "text"
"text"; "{ "{'basic': ("schena': 'employees', 'name" "department, enpeeyeee 'type': "annotations" null,
StructuredCantent":
Verifier Result
Table Structures
Required Indexes Found
FAIL
Project Data not Exists
X
Assignment Data not Exists
Milestones Data not Exists
Figure 20: Failed run by grok-4: schema work incomplete and required rows/indexes missing, verifier fails.
38

---

Published as a conference paper at ICLR 2026
F COST AND TURN DISTRIBUTION
60%
High cost-performance zone
50%
spl-S S-lce
40%
120
Avg total token per
pass@1
30%
gak-S-miinnggh
100
gel--min-madl.mm
dludeeernaad
cluckhopuaa44
80
Gaude sonne----lww
Mrni Instuct
60
20%
max
400
deopsockev9.1 4m.445
gemp-22.-pr
20
10%
gpi-S-mini-iow
120b
40
120
160
200
240 280
400 440 480 520
1080 1120 1160 1200
Average cost per run
Figure 21: Cost-performance map per run. The shaded area highlights runs with higher performance lower cost.
Table 9: Usage stats. Per-task averages: input/output tokens (K), cost (USD), turns, tool calls,
Per-Task Avg Usage
Model
Input
Output
Cost
Turns
Proprietary Models
claude-opue ropusoppssss. 4
16.57
grok-4
633.51
8.42
2.03
16.25
19.84
claude-sonnet-
17.62
gemini-2.5-pro
469.65
7,02
1.28
10.95
11.20
qwen3-max
23.02
gpt-S-medium
627.66
21.91
1.00
14.71
20.16
15.50
cpt-4.1
323.00
1.55
0.66
9.94
14.42
mini
393.10
14.60
13.68
spt 4.1 mini
1172.70
1.90
0.47
16.39
18.61
Bamind-2.5-flamt
1024.09
8.80
0.33
11.41
17.47
spt-S-mini -modium
737.22
10.31
0.20
15.67
21.78
grok=coda-fost-1
590.50
5.65
0.13
18.42
18.19
opt-4. 1-naro
193.37
0.02
10.78
12.39
Snt-S-nano-median
447.99
23.83
0.03
14.50
16.81
Open-Source Models
kimi -2 instruct
931 55
5.01
0.57
26.95
26.22
wwen3-codez-oee
3.51
0.29
23.75
22.84
deepaeek-v3.1
493,05
2,81
0.28
19.43
19.27
glm-4.5
cpt-oss-120h
64.50
1.37
0.01
5.40
4.41
39

---

Published as a conference paper at ICLR 2026
G TURN DISTRIBUTIONS ACROSS MCP SERVICES
In this section, we provide per-service turn distributions for the five MCPs in MCPMark from Figure 22 Figure 26. These plots complement the overall analysis in Figure 4 and illustrate how turn requirements differ by service.
100
0.5
60
50
0.4
Turns
0.2
0.1
gpt- 120b
qwen33-cderrpplue
kiml-k2 2-instrucc
deepsock-3.11
geminl-2.5-pr
claude-sonnet-6
cladde-ops-4.11
grok-4
gpt- mpe-i-medium
Figure 22: Turn distribution per task on the
Filesystem MCP.
100
0.40
80
60
50
40
Turns
0.25
passe
0.20
20
0.15
0.05
grok-4
gpt-oss-120b
gemini-2.5-prro
kimi kimi-k2-insttinn -2-nkk-rritt
deepseek-v3.1
qwen3 coder-pplus
claude claude-sonnet-4
claude-opus- ouude--ppus4.1
63
gpt-5-medium
Figure 23: Turn distribution per task the
N Notion MCP.
100
80
60
0.40
0.35
40
Turns
0.30
30
0.20
20
0.15
10
0.10
gpt-oss-120b
1
gemini-2.5-pro
grok-4
cladde-sonet-d
o3
kimi- kimi-t2-Instruut Instruct
qwen3-code pwen3-coder-plus
claude--pus-4.1
spt-5 mpt-5-medlum
Figure 24: Turn distribution per task the
GitHub MCP.
40

---

Published as a conference paper at ICLR 2026
100
80
60
40
Turns
pass
30
20
gpt-oss-1205 120b
gemini-2.5-pro
claude-opuse 1pus-4.1
doepseek-v3.1
e3
kimi-k2-instruct
qwen3 cwodr-coder-plus
caudoo-oonet-4
grok-4
gpt-5- medduum
Figure 25: Turn distribution per task the PostgreSQL MCP.
100
80
0.40
60
0.35
0.30
40
Turns
30
20
0.15
0.10
0.05
gpt-oss-120b
doepseek.v3.1
gemini-2.5-pro
-laude-opus-..1
chuude-sonnt-d
kimi kiml-k2-inte -i-nt-tuittttt
qwen3 pwen3-coder-plus
grok-4
gpt 5-delumm
Figure 26: Turn distribution per task the Playwright MCP.
41

---

Published as a conference paper at ICLR 2026
H INITIAL STATES SELECTION AND LICENSES
This section provides an overview of the initial states selection, including Notion templates, GitHub repositories, PostgreSQL databases, Playwright websites, and Filesystem components, along with their corresponding licenses.
H.1 NOTION TEMPLATES
We utilized publicly available Notion templates from the Notion Template Marketplace for bench- marking purposes. According Notion's Marketplace Guidelines Terms, templates provided under non-exclusive license for use within the user's workspace long as an active Notion sub- scription maintained. Redistribution or resale prohibited. Our use these templates was limited to internal research and benchmarking. in compliance with the licensing conditions.
#
Template
1
Online Resume
2
Japan Travel Planner
3
Company in-a-Box
4
Computer Science Student Dashboard
5
Standard Operating Procedure
6
Team Projects
7
Python Roadmap
8
Toronto Guide
IT Trouble Shooting Hub
Table 10: Notion templates used in this research benchmark.
H.2 GITHUB REPOSITORIES
Several GitHub repositories were utilized during the research. Below is a summary of the repositories and their respective licenses:
anthropicssclaudecooe:: ④ Anthropic PBC. All rights reserved. Use is subject An- thropic's Commercial Terms of Service.
openai/harmony: Apache License 2.0.
missing-enmstte//tiiinnggeeeete CC BY-NC-SA 4.0.
coderaatter--oobuiidyyyr-rooo- CodeCrafters, Inc. has waived all copyright and related neighboring rights this work.
hiyouga/EasyRl: Apache License 2.0.
mepmark-cied: Written by authors and hosted via GitHub.
H.3 PLAYWRIGHT USAGE
We utilized environments "reddit", "shopping", and "shopping_admi admin" from the web-arena- x/webarena repository, which licensed under the Apache License 2.0. These modules were incorporated for testing and evaluation purposes within the benchmarking setup. Other websites were written by authors and hosted via Vercel.
H.4 FILESYSTEM COMPONENTS
The following filesystem components were used as part of our research environment: (1) desktop. desktop template, file. context, file property. folder structure, papers, and student database were collected from the authors' own local environment or files synthesized using LLMs. (2) legal document refers legal document NVCA financing, which can be accessed at CooleyGo.
42

---

Published as a conference paper at ICLR 2026
(3) threestudio and votenet ooen-s projects utilized from GitHub repositories. Specifically, votenet (MIT License), and threestudio (Apache License 2.0).
H.5 POSTGRESQL DATABASES
We utilized the following PostgreSQL databases, which publicly available with their corresponding licenses:
chinook: MIT License, and Apache License 2.0.
employees: CC BY-SA 3.0, and Apache License 2.0.
lego: CCO 1.0 Universal (Public Domain Dedication), and Apache License 2.0.
sports: Apache License 2.0.
dydrental: MIT License.
I ADDITIONAL EXPERIMENTAL RESULTS
Table 11: Comparison of MCP server implementations using pass @ 1 and token usage.
Service
MCP Server
Model
Pass @ 1 (avg std)
Avg. Tokens
KlavisAT
claude-sonnet-4
31.5% + 3.6
533,385
GitHub
GitHub Official
claude-sonnet-4
16.3% + 5.7
701,252
KlavisAI
claude-sonnet-
34.8% + 6.4
424,474
N
Notion
Notion Official
claude-sonnet-9
21.4% +: 5.1
650,879
InsForge
claude-sonnet-4411
54.8% + 5.3
391,019
PostgreSQL
Supabase
claude-sonnet-4-5
52.4% + 5.8
554,427
Postgres Official
claude-sonnet-4--
48.8% 1: 4.0
492,931
Table 12 compares MCPMark-Agent against ReAct and Codex agent scaffolds across multiple models and MCP services.
Table 12: Performance comparison of MCPMark SaAPPMark-Agennnn Agent, ReAct, and Codex across MCP tasks.
Model
Agent
Overall
PW
MCPWark-Agent
52.6%
57.5%
47.8%
42.0%
43.0%
76.2%
opt-5-medin
ReAct
37.8%
49.2%
39.1%
21.4%
19.0%
64.3%
Codex
36.2%
33.3%
30.4%
25.0%
20.0%
81.0%
MCPMark-Agect
27.4%
33.3%
18.5%
16.1%
12.0%
61.9%
gpt-5- 1et-5-mind -medlun
ReAct
26.6%
35.0%
22.8%
14.3%
15.0%
48.8%
MCFMark-Agent
24.0%
29.2%
13.0%
3.6%
27.0%
27.0%
grok-4-fo fastt
ReAct
26.0%
31.7%
21.7%
28.6%
11.0%
36.9%
MCPMark MCPMark-Agent
23.6%
10.0%
19.6%
25.0%
19.0%
51.2%
g1m-4.6
ReAct
22.1%
10.8%
21.7%
25.0%
14.0%
44.0%
Table 13 further reports ReAct and Codex results under different reasoning effort settings.
43

---

Published as a conference paper at ICLR 2026
Table 13: Performance comparison of ReAct and Codex across MCP tasks.
Agent
Model
Reasoning Effort
Overall
FS
GH
NT
PW
PG
gpt-5
High
35.4%
40.0%
34.8%
14.3%
20.0%
76.2%
gpt-5
Medium
40.2%
53.3%
34.8%
28.6%
16.0%
71.4%
gpt-5
Low
29.9%
36.7%
34.8%
17.9%
12.0%
52.4%
ReAct
spt-5-mini
High
33.9%
40.0%
34.8%
25.0%
12.0%
61.9%
9pt-5-mini
Medium
21.3%
33.3%
8.7%
10.7%
12.0%
42.9%
gpt-5-mm miminn
Low
18.9%
26.7%
17.4%
7.1%
12.0%
33.3%
gpt-5
High
37.0%
46.7%
26.1% 1%
28.6%
16.0%
71.4%
Codex
gpt-5
Medium
36.2%
33.3%
30.4%
25.0%
20.0%
81.0%
gpt
Low
34.6%
36.7%
26.1%%
32.1%
20.0%
61.9%
Table 14: Performance comparison the easier 50-task subset for small and open-source models.
Model
Overall Passi1
Overall Pass
Overall Pass
FS
C GH
P NT
PW
$ PG
03
68.5 2.6
78.0%
51.0%
85.0 1: 5.0%
57.5 1 8.3%
80.0
25.0 L 5.0%
95.0 8.7%
58.5 = 3.0%
70.0%
46.0%
82.5 a: 4.2%
45.0 5.0%
50.0 2: 1228
22.5 4.3%%
92.5 4.3%
gpt-oes-_200
30.0 +: 3.2%
42.0%
20.0%
82.5 4.3%
300 10.0%
2.32
00::
35.0 508
44


Reviews:
[{"review_id": "BJzfLltOzz", "reviewer": "Reviewer_ZaUC", "summary": "The paper adds to the collection of MCP benchmarks. The key distinguishing aspects of this benchmark relative to existing MCP benchmarks are the following:\n- It introduces more complex tasks requiring larger number of turns\n- It covers more operations (read/update/delete) relative to other benchmarks that are only read-heavy.\n\nThe benchmark ensures repeatable and reliable evaluation by setting up resettable initial states and the tasks are co-desgined by a human-LLM agent team, with a significant effort in reviewing and cross-reviewing the tasks to ensure quality control. \n\nThe evaluations illustrate the key challenges of todays LLM-based agents in addressing the tasks. The analysis provide useful categorization of the failure modes.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper improves the existing MCP benchmarking space by contributing a more complex collection of tasks with rigorous programmatic evaluation. \n- The benchmarking and analyses provide a useful empirical assessment of the state-of-the-art models (both closed and open) in tackling complex tasks using MCP servers.", "weaknesses": "I see two areas that can be improved:\n\n- While I fully appreciate that creating these benchmarks take significant time and effort, the number of instances created overall and the pathway for creating more future instances dont appear to scale. Some justification of why 127 tasks are adequate (in a statistical sense) and representative (I do think has been argued for reasonably in the paper) needs to emphasized more clearly in the paper. \n\n- The choice of a simple agent for comparative model evaluation is a valid argument. However, it would still be useful to know what is the best performance that at least one single model could have achieved given more sophisticated agent formulations would be useful to know. In practice, this is how agents are going to be deployed. In some sense, one would like to what is the best performance that can be achieved on this benchmark with the best known techniques today.\n\n- The arguments for why MCP specific benchmarks are necessary can be stronger. There are multiple multi-tool use benchmarks that exist (e.g. AppWorld, WebArena, and SWEBench). For example, benchmarks like AppWorld provide a rich set of problems defined over multiple applications covering nine different applications and have tasks that are much longer in terms of turns than proposed in this paper. the Why shouldn't we consider MCP wrapping around these multi-tool and multi-turn agentic benchmarks? Why invest effort into creating new benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WBIjL45mye", "reviewer": "Reviewer_VEYJ", "summary": "The paper proposes a new benchmark MCPMark for evaluating more relaistic use of Model Context Protocol (MCP) in context of LLM agents. Model Context protocol (MCP) standarizes how LLMs connect with external systems and tools. However, the paper argues that existing MCP bencmarks are often too narrow or not realistic. The proposed benchmark proposes to stress-test MCP use across 127 tasks in more realistic environments and larger interaction depth. The paper provides some interesting analysis on implicit vs explicit errors for state-of-the-art LLMs on MCPMark. Finally, they also show that best LLMs often require much more interactions (tool calls and execution turns) than those required by previous benchmarks.", "questions": "* The authors mention in Sec. 3, that more turns or cost does not equal better performance. This is actually consistent with findings in SWE agents or coding tasks. Have the authors verified if the trend also holds with test-time scaling and parallel rollouts?\n\nPlease see the weaknesses section for some additional questions.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper highlights an interesting problem of evaluating realistic use of MCP in context of LLM agents.\n\n* The analysis on implicit vs explicit errors and failure cases in Sec. 4.2 is interesting.\n\n* The paper shows results on state-of-the-art LLMs on MCPMark (Tab. 3), showing the need for better stress testing for MCP use.", "weaknesses": "* Can the authors please provide details on the scalability of the current data curation pipeline? \n       *  As mentioned in Sec. 2.1 and L164, the data curation pipeline requires a human expert. Thus, it will be important to also discuss the scalability of the data curation pipeline.\n\n* Also in Sec. 2.1, the authors mention that for each task \"including computer science PhD students, front-end designers, full-stack & AI infra engineers, and AI investors—each task takes 3~5 hours of focused expert effort\". While not a major concern, providing more details on impact of human expertise on the task quality will be interesting.\n\n* Which scaffold is used for MCPMark-Agent?\n\n* Also can the authors have some intuition for effect of the used agent scaffold on the numbers reported in Tab. 3?\n\n* Finally, in Sec. 4, in the discussion on failure cases it is noted that most errors are implicit. Could this be fixed by better prompting or more precise problem statements or does this point to a more fundamental limitation? If possible it will be interesting to see some qualitative examples of these implicit errors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "POV1knTiGq", "reviewer": "Reviewer_NsiE", "summary": "This paper introduces MCPMark, a benchmark designed to evaluate the realistic and comprehensive use of MCP. It includes 127 tasks across five MCP environments: Notion, GitHub, Filesystem, PostgreSQL, and Playwright. It uses a minimal agent that has a tool-calling/llm-calling loop to evaluate LLMs against these tasks. The authors evaluated popular models and found even the best performing model only achieves 52% at pass@1 and 33% at pass^4.", "questions": "None. The paper is clearly written.", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The proposed benchmark contains realistic and diverse tools and tasks\n- Each task includes automatic verification scripts.\n- It uses a simple but standardized agent framework to just evaluate LLM tool calling capabilities\n- It evaluates many different models and draw some interesting conclusions", "weaknesses": "None. I like the paper quite a lot.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)