LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?

## Abstract

Abstract With the rapid development of Model Context Protocol (MCP), the number of MCP servers has surpassed 10,000.
However, existing MCP benchmarks are limited to single-server settings with only a few tools, hindering effective evaluation of agent capabilities in large-scale, real-world scenarios.
To address this limitation, we present LiveMCPBench , the first comprehensive benchmark comprising 95 real-world tasks grounded in the MCP ecosystem, designed to evaluate LLM agents at scale across diverse servers. To support a scalable and reproducible evaluation pipeline in large-scale MCP environments, we curate LiveMCPTool , a diverse and readily deployable collection of 70 MCP servers and 527 tools. Furthermore, we introduce LiveMCPEval , an LLM-as-a-Judge framework that enables automated and adaptive evaluation in dynamic, time-varying task environments, achieving 81% agreement with human reviewers. Finally, we propose the MCP Copilot Agent , a multi-step agent that routes tools for dynamic planning and executes tools for API interaction across the entire LiveMCPTool suite. Our evaluation covers 10 leading models, with the best-performing model (Claude-Sonnet-4) reaching a 78.95% success rate. However, we observe large performance variance across models, and several widely-used models perform poorly in LiveMCPBench’s complex, tool-rich environments. Overall, LiveMCPBench offers the first unified framework for benchmarking LLM agents in realistic, tool-rich, and dynamic MCP environments, laying a solid foundation for scalable and reproducible research on agent capabilities. Our code and data will be publicly available at https://icip-cas.github.io/LiveMCPBench.

## Introduction

Figure: (a) Tool-Use Benchmarks: Simple Simulator.
Refer to caption: x1.png

Figure: Figure 2: LiveMCPBench comprises four parts: (1) Diverse Daily Task, a collection of everyday tasks; (2) LiveMCPTool, a large-scale, out-of-the-box MCP toolset; (3) MCP Copilot Agent, a ReACT-based agent system capable of retrieval and multi-turn tool invocation across the MCP toolset; (4) LiveMCPEval, an automated LLM-as-a-Judge evaluation system for accurately assessing online, time-varying tasks.
Refer to caption: x4.png

Recent years have witnessed remarkable progress in tool-use agents powered by large language models (LLMs), demonstrating promising potential as a pathway towards artificial general intelligence (Qu et al. 2025; Wang et al. 2024).
As model capabilities advance and application scenarios expand, enabling LLMs to effectively invoke external tools has emerged as a critical research direction for enhancing their generalization and real-world task execution abilities.
Notably, with the widespread adoption of the Model Context Protocol (Anthropic 2024, MCP), an increasing number of real-world tools now expose their functionalities through standardized contextual interfaces, forming a vast ecosystem encompassing more than 10,000 MCP servers (Hou et al. 2025). Concurrently, certain pretrained models have begun to directly learn interaction patterns with MCP (Qwen 2025), further accelerating the evolution of tool-use agents.

However, existing tool-use benchmarks predominantly rely on simulate API interfaces (see Figure [1](https://arxiv.org/html/2508.01780v1#Sx1.F1)), which suffer from a fundamental limitation. API interfaces exhibit high instability—For instance, 55.6% of APIs in ToolBench (Qin et al. 2024) have become unavailable (Guo et al. 2024), forcing evaluation frameworks (e.g., API-Bank (Li et al. 2023)) to resort to simplified simulated tools, significantly compromising task authenticity and challenge. And the emergence of MCP provides a stable tool call interface. For the few MCP benchmarks (e.g., MCPBench (Luo et al. 2025) and MCPEval (Liu et al. 2025)), their experimental scales remain severely limited, typically involving only a small number of MCP servers (about 10), failing to reflect agents’ generalization and decision-making capabilities in a large-scale toolset.

Consequently, two pivotal questions remain underexplored: (1) How can optimal planning and retrieval be achieved in large-scale MCP toolset to accomplish real-world tasks? (2) Can an LLM-driven agent, trained for tool invocation, exhibit meta-tool-learning capabilities—i.e., autonomously explore and compose tools from real-world toolset to complete tasks?

To address these questions, we present LiveMCPBench, a multi-domain, multi-step MCP tool-use benchmark. LiveMCPBench focuses on practical, everyday tasks across six domains, comprising 95 high-quality tasks solvable via MCP tools. To lower the deployment barrier and avoid complex multi-API-key dependencies, we introduce LiveMCPTool, a ready-to-use, fully functional MCP toolset encompassing 70 MCP servers and 527 tools, significantly improving reproducibility and reducing research overhead.

Furthermore, evaluating agent systems in real-world MCP environments presents two unique challenges: (1) Task dynamism (e.g., time-sensitive answers for news summarization) and (2) Solution diversity, multiple tool combinations can solve the same task. Consequently, traditional metrics based on tool-matching accuracy in API-Bank become inadequate. We thus design LiveMCPEval, an LLM-as-a-Judge (Zheng et al. 2023) evaluation system capable of automatically assessing multi-turn tool invocation trajectories.

Finally, the server-tool architecture of MCP fundamentally differs from traditional parallel API structures, rendering existing tool-use agent methods inapplicable in MCP environments. To evaluate planning and retrieval strategies in large-scale MCP toolset and verify whether LLMs possess meta-tool-learning capabilities, we propose MCP Copilot Agent—an agent integrating reasoning and acting (Yao et al. 2023, ReACT) strategy with tool retrieval and multi-step invocation abilities, capable of dynamically responding to environmental changes. Our evaluation of 10 frontier models reveals that the Claude-Sonnet-4 agent achieves a 78.95% task success rate, demonstrating strong meta-tool-learning capabilities. We further conduct human annotation on the agent trajectory and measure the human agreement rate across different evaluator models, with DeepSeek-V3 achieving an agreement rate of 81.05%, validating the reliability of our evaluation method.

In summary, our key contributions are:

- •
We propose LiveMCPBench, the first evaluation framework leveraging large-scale MCP toolset for everyday tasks.
- •
We evaluate the frontier models using LiveMCPBench, revealing their limitations in meta-tool-learning.
- •
We analyze the agent trajectories, identifying key bottlenecks in multi-tool collaboration and providing insights for future improvements.

## LiveMCPBench

**Table 1: Comparison with existing MCP and Tool Benchmarks. Plug & Play indicates whether the toolset can be used directly without additional API keys. Time-Varying denotes whether the task output changes over time. Note that MCP-Zero is not a benchmark, but contains a toolset. And, it is not directly usable, as many of its tools require API keys. Syn. means synthetic.**
|  | Tools | Tasks |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Type | Stable | Servers | Tools | Plug & Play | Type | Time-Varying | Evaluation |
| MCP Benchmarks |  |  |  |  |  |  |  |  |
| MCPBench (Luo et al. 2025) | MCP | ✓ | 10 | 10 | ✗ | Real | ✗ | Rule |
| MCP-RADAR (Gao et al. 2025) | MCP | ✓ | 9 | 42 | ✓ | Real | ✗ | Rule |
| MCP-Zero (Fei, Zheng, and Feng 2025) | MCP | ✓ | 308 | 2,797 | ✗ | - | - | - |
| MCPEval (Liu et al. 2025) | MCP | ✓ | 12 | 77 | ✗ | Syn. | ✓ | LLM |
| LiveMCPBench (ours) | MCP | ✓ | 70 | 527 | ✓ | Real | ✓ | LLM |
| Tool Benchmarks |  |  |  |  |  |  |  |  |
| ToolAlpaca (Tang et al. 2023) | Simulate | ✓ | 50 | 426 | ✗ | Syn. | ✗ | Rule |
| ToolBench (Qin et al. 2024) | API | ✗ | 49 | 16,464 | ✓ | Syn. | ✓ | LLM |
| API-Bank (Li et al. 2023) | Simulate | ✓ | 1,000 | 2,138 | ✓ | Real | ✗ | Rule |

We present LiveMCPBench (see Figure [2](https://arxiv.org/html/2508.01780v1#Sx1.F2)), a novel benchmark designed to evaluate the capability of agent systems in retrieving appropriate tools from a large-scale MCP toolset to accomplish general-purpose everyday tasks. The construction of such a benchmark necessitates addressing three fundamental challenges:

- •
How to construct representative daily tasks requiring multi-step tool use?
- •
How to collect a large, redundant yet functionally complete MCP toolset?
- •
How to automatically evaluate performance on evolving online tasks?

In this section, we first elaborate on our task construction process. Then, we outline the collection process of the LiveMCPTool. Next, we introduce LiveMCPEval, an LLM-as-a-Judge evaluation framework designed for robust and scalable assessment. Finally, we describe MCP Copilot Agent as our baseline approach.

### Task Construction

To advance the development of practical agents for real-world applications, we create a diverse set of tasks grounded in everyday scenarios, spanning six key domains: Office (e.g., spreadsheet analysis), Lifestyle (e.g., news retrieval), Leisure (e.g., video game inquiries), Finance (e.g., stock price monitoring), Travel (e.g., ticket search), and Shopping (e.g., product recommendations). These scenarios were carefully selected to embody three critical characteristics. (1) Time-varying: Tasks exhibit time-sensitive outcomes; (2) Long-horizon: Tasks require multiple tools to complete; (3) Genuine utility: Tasks address authentic user needs.

The annotation process employed a rigorous two-stage methodology involving two groups of computer science students serving as task proposers and validators. Proposers first generated scenario-specific tasks based on personal experience, with LLM-assisted ideation permitted but strictly vetted for authenticity. Each proposer then interacted with our toolset to complete their proposed task, meticulously annotating key points to preserve the task’s compositional depth. Validators subsequently scrutinized both the task design and corresponding toolchain invocations, eliminating duplicates while enforcing quality standards. This iterative pipeline yielded 95 high-fidelity daily tasks (see Appendix C for annotation principles and distribution statistics).

**Table 2: Task success rate results for the frontier models. Evaluation using Deepseek-V3.**
| Model | Office | Leisure | Travel | Lifestyle | Finance | Shopping | Overall (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Claude-Sonnet-4-20250514 | 90.32 | 64.29 | 75.00 | 80.00 | 78.57 | 66.67 | 78.95 |
| Claude-Opus-4-20250514 | 80.65 | 64.29 | 66.67 | 86.67 | 64.29 | 33.33 | 70.53 |
| DeepSeek-R1-0528 | 41.94 | 50.00 | 58.33 | 46.67 | 50.00 | 55.56 | 48.42 |
| Qwen3-235B-A22B | 54.84 | 35.71 | 41.67 | 53.33 | 50.00 | 44.44 | 48.42 |
| GPT-4.1-Mini | 45.16 | 50.00 | 50.00 | 46.67 | 42.86 | 22.22 | 44.21 |
| Qwen2.5-72B-Instruct | 35.48 | 35.71 | 50.00 | 40.00 | 57.14 | 55.56 | 43.16 |
| DeepSeek-V3-0324 | 41.94 | 42.86 | 50.00 | 40.00 | 28.57 | 55.56 | 42.11 |
| Gemini-2.5-Pro | 48.39 | 28.57 | 16.67 | 60.00 | 57.14 | 11.11 | 41.05 |
| GPT-4.1 | 51.61 | 28.57 | 25.00 | 46.67 | 35.71 | 22.22 | 38.95 |
| Qwen3-32B | 29.03 | 14.29 | 25.00 | 53.33 | 28.57 | 33.33 | 30.53 |

### LiveMCPTool Collection

While prior study (Hou et al. 2025) suggests the existence of over 10,000 MCP servers, curating a practical and accessible toolset remains nontrivial due to critical usability constraints. The predominant challenge stems from dependency fragmentation: the majority of MCP servers necessitate proprietary API keys or integrations with third-party services, rendering them impractical for a standardized toolset. To address this, we introduce a rigorously validated methodology for constructing a high-quality, dependency-free MCP toolset—prioritizing reproducibility and broad applicability. Our approach first aggregates 5,588 server configurations from mcp.so, then systematically filters out key-dependent servers to eliminate access barriers.

Beyond accessibility, we ensure the toolset’s representativeness through structured curation and expert annotation. Tools are taxonomically organized into five functional categories (Discovery, Visualization, File Access, Location, and Miscellaneous), followed by manual vetting to exclude low-quality implementations. This two-stage pipeline yields 70 MCP servers providing 527 tools, each verified for standalone functionality and categorical relevance. By decoupling the toolset from external dependencies, our collection establishes the first reproducible toolset for large-scale MCP performance analysis (see Appendix D for distribution details).

### LiveMCPEval

Automated evaluation of trajectories generated by the agent is essential for benchmarking task performance. However, achieving robust automated evaluation remains challenging due to several factors: (1) Time-varying nature of daily tasks, (2) Inherent instability of MCP tool outputs caused by its online dynamics, and (3) Diversity of trajectories resulting from different tool combinations that can accomplish the same task.

To address these challenges, we employ an LLM-as-a-Judge system, leveraging the adaptability of LLMs to dynamically assess task completion based on tool usage patterns and feedback. While dynamic tasks may exhibit variability, they often share a set of critical subtasks or key points that must be fulfilled. Incorporating these key points—whether manually annotated or automatically extracted by LLM—improves the accuracy of the LLM-as-a-Judge system. In our framework, all tasks are annotated with a verified set of key points to ensure a reliable evaluation. Specifically, given a task $T$, a set of key points $P$, agent’s execution trajectory $A$ with retrieval and tool-call sequences and descriptions of all tools used $D$, the evaluator performs binary classification to determine the outcome $\mathcal{O}$ as either “Success” or “Failure”:

$$ $\mathcal{O}=Evaluator(T,P,A,D)$ (1) $$

Therefore, we use the success rate as the primary metric. Finally, we systematically compare LiveMCPBench with existing benchmarks in Table [1](https://arxiv.org/html/2508.01780v1#Sx2.T1), highlighting their fundamental differences.

### MCP Copilot Agent

Due to the dynamic nature of daily tasks and the inherent uncertainty in retrieval systems, a fixed pipeline cannot be effectively employed for tool retrieval and invocation in LiveMCPTool. Instead, we require agents to dynamically adapt to environmental changes. To model this dynamic tool retrieval and invocation process, we formulate it as a Partially Observable Markov Decision Process (Silver and Veness 2010, POMDP), as the agent can only make decisions based on the textual descriptions of retrieved tools and feedback from tool execution.

We characterize the toolset environment using the following components: (1) Hidden state space $\mathcal{S}$; (2) Observation space $\mathcal{O}$ containing the descriptions of retrieved tools and feedback from tools; (3) Language action space $\mathcal{A}$, including three key actions—route, execute, and response—along with their associated descriptions; (4) State transition $\mathcal{T}:\mathcal{S}_{t}\times\mathcal{A}\rightarrow\mathcal{S}_{t+1}$; (5) Terminal reward $\mathcal{R}:\mathcal{S}\rightarrow\mathbb{R}$ quantifying task completion.

Our agent implementation is based on the ReACT framework. For the route tool, we adopt a retrieval strategy inspired by MCP-Zero (Fei, Zheng, and Feng 2025), where tool prioritization is determined by a weighted combination of server description similarity and tool description similarity.

## Experiments and Results

### Setup

We evaluate 10 frontier models: Claude-Opus-4 and Claude-Sonnet-4 (Anthropic 2025), GPT-4.1 and GPT-4.1-Mini (Openai 2025), Gemini-2.5-Pro (Google 2025), Deepseek-V3 and Deepseek-R1 (DeepSeek-AI 2025), Qwen3-235B-A22B and Qwen3-32B (Qwen 2025), and Qwen2.5-72B-Instruct (Qwen et al. 2025). For assessment, we employ Deepseek-V3 as our primary evaluation model. Detailed implementation and computational resources are documented in Appendix E to ensure reproducibility.

### Main Results

We show the task success rates for different models in Table [2](https://arxiv.org/html/2508.01780v1#Sx2.T2). We can see that:

- 1.
Meta-Tool-Learning Capabilities in Claude Models. The Claude series demonstrates remarkable meta-tool-learning proficiency, with Claude-Sonnet-4 and Claude-Opus-4 achieving success rates of 78.95% and 70.53% respectively. These results indicate their superior ability to effectively explore and combine tools from a large-scale toolset to accomplish complex real-world tasks.
- 2.
Performance Variance Among Models. We observe substantial performance gap across frontier models. While most contemporary models achieve only 30%–50% task success rates, the Claude series shows significantly superior performance. This performance gap suggests fundamental limitations in the meta-tool-learning capabilities of other models.
- 3.
Domain-Specific Superiority of Claude Models. The Claude series exhibits particularly dominant performance in Office and Lifestyle scenarios, outperforming other models by more than 30%. This substantial advantage highlights Claude models’ unique strengths and adaptability in these specific domains.

Figure: Figure 3: Correlation between model evaluation performance and human agreement rates across multiple evaluators: The analysis of annotated agent trajectories from Claude-Opus-4 and Claude-Sonnet-4 (color indicates evaluator variant).
Refer to caption: x5.png

To validate the reliability of LiveMCPEval’s automatic evaluation, we conducted human annotation of the execution trajectories for the top-performing models (Claude-Sonnet-4 and Claude-Opus-4). We systematically tested all models used in the baselines and calculated human agreement rates, with results presented in Figure [3](https://arxiv.org/html/2508.01780v1#Sx3.F3).

- 1.
LiveMCPEval demonstrates high accuracy under appropriate model conditions. Our experiments show that Deepseek-V3 achieves an average human agreement rate of 78.95%, validating the reliability of our evaluation framework. Additionally, GPT-4.1 Mini and Qwen2.5-72B-Instruct exhibit comparable performance, with human agreement rates around 75%, making them viable alternative models for accurate assessment.
- 2.
Certain models prove less suitable for evaluation tasks. Notably, advanced reasoning models such as Deepseek-R1, Claude-Opus-4, and Qwen3-32B exhibit lower human agreement rates (60%-70%). We hypothesize that this limitation stems from their reduced ability to process long trajectory inputs.

Figure: Figure 4: Comparison of human agreement rates across different models when evaluated using human-annotated versus LLM-generated key points.
Refer to caption: x6.png

To investigate the generalizability of LiveMCPEval, we conducted experiments on Claude-Sonnet-4’s trajectories by evaluating how LLM-generated key points affect human agreement rates, with results visualized in Figure [4](https://arxiv.org/html/2508.01780v1#Sx3.F4) (Detaied examples in Appendix F). Our key findings reveal:

- 1.
Generalizability of LiveMCPEval. The framework demonstrates strong generalizability across models. Notably, even without human-annotated references, the majority of evaluated models achieved improved human agreement rates through automatically generated key points.
- 2.
Optimal Utilization of Human-Annotated Key Points. Deepseek-V3 exhibits superior capability in leveraging human-annotated key points compared to other models. This observation provides important implications for model selection in scenarios where human-annotated references are available.

## Analysis

**Table 3: Performance efficiency metrics: Steps denotes average dialogue turns, Tools indicates average used tools, execute represents average tool executions, route refers to average retrievals, and Overall shows average task success rate.**
| Model | Steps | Tools | excute | route | Overall (%) |
| --- | --- | --- | --- | --- | --- |
| Claude-Sonnet-4 | 20.09 | 2.71 | 5.59 | 2.98 | 78.95 |
| Claude-Opus-4 | 25.53 | 3.40 | 6.93 | 4.35 | 70.53 |
| Qwen3-235B | 16.76 | 1.59 | 5.12 | 1.77 | 48.42 |
| DeepSeek-R1 | 10.33 | 1.24 | 2.11 | 2.00 | 48.42 |
| GPT-4.1-Mini | 10.89 | 1.37 | 2.71 | 1.65 | 44.21 |
| Qwen2.5-72B | 11.22 | 1.31 | 2.80 | 1.38 | 43.16 |
| DeepSeek-V3 | 8.33 | 1.01 | 1.29 | 1.41 | 42.11 |
| Gemini-2.5-Pro | 8.08 | 0.99 | 1.46 | 1.35 | 41.05 |
| GPT-4.1 | 9.03 | 1.31 | 1.72 | 1.64 | 38.95 |
| Qwen3-32B | 9.99 | 1.16 | 2.31 | 1.19 | 30.53 |

Figure: Figure 5: Log-Price vs. Performance scatter plot with pareto frontier representation. Different colors represent different model families.
Refer to caption: x7.png

### Efficiency Analysis

To compare the behavioral characteristics of different models, we present the average number of dialogue turns, used tools, tool execution attempts, and retrieval calls in Table [3](https://arxiv.org/html/2508.01780v1#Sx4.T3). Based on these metrics, we draw the following conclusions:

- 1.
Claude series models exhibit more proactive exploration and utilization behavior. Their retrieval and execution frequencies are significantly higher than other models, accompanied by a greater number of used tools. This suggests that Claude models actively engage with and adapt to the tool-augmented environment, demonstrating a stronger tendency to explore and exploit available tools.
- 2.
Most models suffer from severe underutilization of tools. The average number of tools used by these models remains close to 1, indicating that once a model identifies and adopts a single tool, it tends to rely exclusively on it while neglecting other available tools. This behavior highlights a critical limitation in their ability to dynamically leverage multiple tools during task execution.

In practical applications, a trade-off between model performance and cost must be carefully considered. To provide actionable insights for model selection, we plotted the relationship between logarithmic cost and performance, along with the corresponding pareto frontier (Lotov, Bushenkov, and Kamenev 2004). As illustrated in Figure [5](https://arxiv.org/html/2508.01780v1#Sx4.F5), our analysis reveals two key findings:

- 1.
Near-Linear Trade-off on the Pareto Frontier. The performance and logarithmic cost of models along the pareto frontier exhibit an approximately linear relationship. This observation presents a valuable opportunity for optimizing cost-performance balance in real-world tool-calling agent.
- 2.
Optimal Cost-Performance Models. The models positioned on the pareto frontier represent the most cost-effective choices for tool calling. These include Qwen3-32B, Qwen2.5-72B-Instruct, Deepseek-R1-0528, and Claude-Sonnet-4, each demonstrating distinct advantages in terms of cost-performance efficiency.

### Error Analysis

Figure: Figure 6: Distribution of errors for four types. We counted two agent trajectories for the advanced claude series models.
Refer to caption: x8.png

We conducted a detailed error analysis on the trajectories of current retrieval and invocation agents to provide insights for future development. Human annotators were employed to classify error types in the trajectories of Claude-Opus-4 and Claude-Sonnet-4. Based on the modules defined in the MCP Copilot Agent framework, we identified four distinct and easily distinguishable error categories (Figure [6](https://arxiv.org/html/2508.01780v1#Sx4.F6)). Each erroneous trajectory was uniquely classified into one error type without overlap. Detailed error examples are provided in Appendix G.

#### Query Error.

Query errors occur when the generated query either lacks semantic relevance to the required tools or exhibits a granularity mismatch with tool capabilities. For instance, in the task “summarize today’s news and save as PDF,” the agent might request a single omnipotent tool despite the availability of specialized tools for news retrieval and PDF generation. Such granularity mismatches prevent the retrieval system from providing appropriate tools, and agents often fail to refine queries based on retrieval feedback. Hallucinated queries for irrelevant tools further exacerbate this issue. These errors stem from limitations in LLMs’ task decomposition and planning capabilities, suggesting room for improvement despite their generally competent performance.

#### Retrieve Error.

Retrieve errors arise when semantically appropriate queries fail to match available tools due to retrieval system shortcomings. For example, in the task “Convert the YouTube video to MP3 format,” the retrieval system may overlook the youtube downloader tool (which supports format conversion) due to unrecognized semantic equivalence between “convert to MP3” and the tool’s documented “extract audio tracks” functionality. These errors highlight challenges in hierarchical retrieval (e.g., MCP server-tool structures) and semantic similarity computation. Dominating the error distribution, retrieve errors underscore the critical need for enhanced retrieval architectures and more robust similarity metrics.

#### Tool Error.

Tool errors occur when the agent retrieves the correct tool but invokes it incorrectly—e.g., via error parameters or incomplete server/tool names. In the task “summarize news and save to specified path,” the agent might supply “path name” instead of the required “path” parameter to the save tool. Such inaccuracies reflect limitations in contextual precision and memory retention. While modern LLMs exhibit strong contextual understanding, these errors indicate a need for more sophisticated memory mechanisms to ensure reliable tool usage.

#### Other Error.

This category encompasses sporadic failures beyond the above types, including network timeouts or model invocation errors. For example, in “summarize today’s news,” a network timeout during news retrieval may cause the agent to abandon the task without retries or alternative solutions. Such behavior reveals deficiencies in framework design, particularly the absence of robust error-handling mechanisms (e.g., failure recovery, adaptive tool exploration). The prevalence of these errors suggests that while current frameworks support basic exploration, significant improvements in fault tolerance and proactive problem-solving are needed.

## Related Work

### Tool-Use Benchmarks

Most existing benchmarks for tool utilization rely on simulated APIs due to the inherent instability of real-world APIs. For instance, API-Bank (Li et al. 2023) and StableToolBench (Guo et al. 2024) employ artificially constructed toolsets to ensure API stability. Other works, such as ToolAlpaca (Tang et al. 2023) and Seal-Tools (Wu et al. 2025), collect real-world API interfaces but are unable to execute actual calls. A third category of tool-use benchmarks, including ToolBench (Qin et al. 2024) and ShortcutsBench (SHEN et al. 2025), attempts to integrate real-world APIs but faces challenges due to rapid API changes, leading to frequent tool unavailability (Guo et al. 2024).

Recent efforts, like StableToolBench-MirrorAPI (Guo et al. 2025), leverage fine-tuned LLMs to simulate API interfaces and calls. However, these prior tool-use benchmarks predominantly focus on API-based tools, which introduces instability and limits functionality—particularly in directly manipulating local user files or enabling complex operations (e.g., interacting with local software).

The emergence of MCP has shifted this paradigm by providing a stable, unified interface, enabling the development of general-purpose toolsets. In this work, we construct a practical MCP toolset, addressing both the instability of APIs and their functional constraints, thereby delivering a comprehensive and reliable real-world tool-use benchmark.

### MCP Benchmarks

The evaluation of MCP systems remains an emerging and rapidly evolving field. Among existing benchmarks, MCPBench (Luo et al. 2025) stands as one of the earliest efforts, primarily focusing on comparative analyses between MCP tools and traditional API-based tools. Building upon this, MCP-RADAR (Gao et al. 2025) extends the evaluation scope by introducing a multi-dimensional assessment framework that examines critical aspects such as efficiency, accuracy, and robustness. More recently, MCPEval (Liu et al. 2025) has been advanced the field further by proposing a fine-grained evaluation framework capable of automatically generating queries to assess the performance of MCP servers.

Despite these advancements, existing benchmarks suffer from a key limitation: their evaluations are predominantly conducted on small-scale MCP servers (typically around 10 servers), which inadequately reflects real-world scenarios where agents must operate in large-scale, dynamic environments. To bridge this gap, our work introduces a large-scale MCP toolset and systematically investigates agent capabilities in accomplishing everyday tasks through extensive tool utilization.

Recent efforts, such as RAG-MCP (Gan and Sun 2025), MCPZero (Fei, Zheng, and Feng 2025), and ScaleMCP (Lumer et al. 2025), have explored retrieval methods over large-scale MCP toolsets. However, these approaches are constrained by rigid pipelines that lack dynamic adaptability in tool invocation and error feedback. Furthermore, ScaleMCP relies on a manually constructed toolset with limited functional diversity, restricting its applicability to broader, real-world use cases.

## Conclusion

In this paper, we present LiveMCPBench, a benchmark designed to evaluate the capability of agents in accomplishing daily tasks using large-scale MCP toolset. We introduce LiveMCPTool, a comprehensive and readily deployable collection of MCP tools. We propose LiveMCPEval, an automated evaluation framework based on the LLM-as-a-Judge, which effectively assesses complex tool-usage tasks characterized by time-varying dynamics and diverse completion paths. Human evaluations confirm the reliability of LiveMCPEval. Furthermore, we develop MCP Copilot Agent, an agent framework capable of autonomous exploration and dynamic decision-making in large-scale MCP environments. We conduct extensive evaluations on ten frontier models, revealing limitations in widely-used LLMs when applied to large-scale tool invocation tasks. Our in-depth analysis uncovers distinct behavioral patterns across different models and identifies the most cost-effective solutions. Finally, through detailed error analysis, we highlight two critical shortcomings in current models: (1) deficiencies in task decomposition and planning, and (2) inadequate adaptation of tool retrieval systems in MCP environments. These findings provide clear directions for future improvements in the field.

## References

- Anthropic (2024)
Anthropic. 2024.
Introducing the Model Context Protocol.
https://www.anthropic.com/news/model-context-protocol.
Accessed: 2025-07-24.
- Anthropic (2025)
Anthropic. 2025.
Introducing Claude 4.
https://www.anthropic.com/news/claude-4.
Accessed: 2025-07-24.
- DeepSeek-AI (2025)
DeepSeek-AI. 2025.
DeepSeek-V3 Technical Report.
arXiv:2412.19437.
- Fei, Zheng, and Feng (2025)
Fei, X.; Zheng, X.; and Feng, H. 2025.
MCP-Zero: Active Tool Discovery for Autonomous LLM Agents.
arXiv:2506.01056.
- Gan and Sun (2025)
Gan, T.; and Sun, Q. 2025.
RAG-MCP: Mitigating Prompt Bloat in LLM Tool Selection via Retrieval-Augmented Generation.
arXiv:2505.03275.
- Gao et al. (2025)
Gao, X.; Xie, S.; Zhai, J.; Ma, S.; and Shen, C. 2025.
MCP-RADAR: A Multi-Dimensional Benchmark for Evaluating Tool Use Capabilities in Large Language Models.
arXiv:2505.16700.
- Google (2025)
Google. 2025.
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities.
arXiv:2507.06261.
- Guo et al. (2025)
Guo, Z.; Cheng, S.; Niu, Y.; Wang, H.; Zhou, S.; Huang, W.; and Liu, Y. 2025.
StableToolBench-MirrorAPI: Modeling Tool Environments as Mirrors of 7,000+ Real-World APIs.
arXiv:2503.20527.
- Guo et al. (2024)
Guo, Z.; Cheng, S.; Wang, H.; Liang, S.; Qin, Y.; Li, P.; Liu, Z.; Sun, M.; and Liu, Y. 2024.
StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Models.
In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., *Findings of the Association for Computational Linguistics: ACL 2024*, 11143–11156. Bangkok, Thailand: Association for Computational Linguistics.
- Hou et al. (2025)
Hou, X.; Zhao, Y.; Wang, S.; and Wang, H. 2025.
Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions.
arXiv:2503.23278.
- Li et al. (2023)
Li, M.; Zhao, Y.; Yu, B.; Song, F.; Li, H.; Yu, H.; Li, Z.; Huang, F.; and Li, Y. 2023.
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs.
In *The 2023 Conference on Empirical Methods in Natural Language Processing*.
- Liu et al. (2025)
Liu, Z.; Qiu, J.; Wang, S.; Zhang, J.; Liu, Z.; Ram, R.; Chen, H.; Yao, W.; Wang, H.; Heinecke, S.; Savarese, S.; and Xiong, C. 2025.
MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models.
arXiv:2507.12806.
- Lotov, Bushenkov, and Kamenev (2004)
Lotov, A. V.; Bushenkov, V. A.; and Kamenev, G. K. 2004.
*Interactive decision maps: Approximation and visualization of Pareto frontier*, volume 89.
Springer Science & Business Media.
- Lumer et al. (2025)
Lumer, E.; Gulati, A.; Subbiah, V. K.; Basavaraju, P. H.; and Burke, J. A. 2025.
ScaleMCP: Dynamic and Auto-Synchronizing Model Context Protocol Tools for LLM Agents.
arXiv:2505.06416.
- Luo et al. (2025)
Luo, Z.; Shi, X.; Lin, X.; and Gao, J. 2025.
Evaluation Report on MCP Servers.
arXiv:2504.11094.
- Openai (2025)
Openai. 2025.
Introducing GPT-4.1 in the API.
https://openai.com/index/gpt-4-1/.
Accessed: 2025-07-24.
- Qin et al. (2024)
Qin, Y.; Liang, S.; Ye, Y.; Zhu, K.; Yan, L.; Lu, Y.; Lin, Y.; Cong, X.; Tang, X.; Qian, B.; Zhao, S.; Hong, L.; Tian, R.; Xie, R.; Zhou, J.; Gerstein, M.; dahai li; Liu, Z.; and Sun, M. 2024.
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs.
In *The Twelfth International Conference on Learning Representations*.
- Qu et al. (2025)
Qu, C.; Dai, S.; Wei, X.; Cai, H.; Wang, S.; Yin, D.; Xu, J.; and Wen, J.-R. 2025.
Tool learning with large language models: A survey.
*Frontiers of Computer Science*, 19(8): 198343.
- Qwen (2025)
Qwen. 2025.
Qwen3 Technical Report.
arXiv:2505.09388.
- Qwen et al. (2025)
Qwen; Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao, K.; Yang, K.; Yu, L.; Li, M.; Xue, M.; Zhang, P.; Zhu, Q.; Men, R.; Lin, R.; Li, T.; Tang, T.; Xia, T.; Ren, X.; Ren, X.; Fan, Y.; Su, Y.; Zhang, Y.; Wan, Y.; Liu, Y.; Cui, Z.; Zhang, Z.; and Qiu, Z. 2025.
Qwen2.5 Technical Report.
arXiv:2412.15115.
- SHEN et al. (2025)
SHEN, H.; Li, Y.; Meng, D.; Cai, D.; Qi, S.; Zhang, L.; Xu, M.; and Ma, Y. 2025.
ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents.
In *The Thirteenth International Conference on Learning Representations*.
- Silver and Veness (2010)
Silver, D.; and Veness, J. 2010.
Monte-Carlo Planning in Large POMDPs.
In Lafferty, J.; Williams, C.; Shawe-Taylor, J.; Zemel, R.; and Culotta, A., eds., *Advances in Neural Information Processing Systems*, volume 23. Curran Associates, Inc.
- Tang et al. (2023)
Tang, Q.; Deng, Z.; Lin, H.; Han, X.; Liang, Q.; Cao, B.; and Sun, L. 2023.
ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases.
arXiv:2306.05301.
- Wang et al. (2024)
Wang, L.; Ma, C.; Feng, X.; Zhang, Z.; Yang, H.; Zhang, J.; Chen, Z.; Tang, J.; Chen, X.; Lin, Y.; et al. 2024.
A survey on large language model based autonomous agents.
*Frontiers of Computer Science*, 18(6): 186345.
- Wu et al. (2025)
Wu, M.; Zhu, T.; Han, H.; Tan, C.; Zhang, X.; and Chen, W. 2025.
Seal-Tools: Self-instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark.
In Wong, D. F.; Wei, Z.; and Yang, M., eds., *Natural Language Processing and Chinese Computing*, 372–384. Singapore: Springer Nature Singapore.
- Yao et al. (2023)
Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K. R.; and Cao, Y. 2023.
ReAct: Synergizing Reasoning and Acting in Language Models.
In *The Eleventh International Conference on Learning Representations*.
- Zheng et al. (2023)
Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023.
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.
In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., *Advances in Neural Information Processing Systems*, volume 36, 46595–46623. Curran Associates, Inc.

## Appendix A Limitations

While LiveMCPBench represents a comprehensive benchmarking framework, we acknowledge several limitations in its design and evaluation methodology:

#### Dependence on LLM evaluation.

The LiveMCPEval component relies heavily on LLM-based assessment. Although we have validated the evaluation accuracy through human experiments, potential model biases may still influence the results. To mitigate this concern, we conducted extensive case studies analyzing model judgment failure cases, which helps improve the robustness of our evaluation framework.

#### Evaluation Assumptions.

Our assessment framework operates under the assumption that agent behavior trajectories and tool descriptions sufficiently reflect task performance, without explicitly verifying the final environmental impact. While this assumption holds in most cases, expanding the toolset could introduce inconsistencies between actual tool effects and their descriptions, potentially compromising evaluation reliability. To address this, we rigorously inspect the quality of LiveMCPTool to minimize such discrepancies.

## Appendix B Ethical Considerations

The advent of large-scale multi-tool retrieving and calling agents promises to revolutionize traditional UI-based interaction paradigms by shifting from complex message retrieval or manual UI operations to automated tool invocation. This transition holds significant potential to reduce usability barriers, enhance operational efficiency, and accelerate progress toward Artificial General Intelligence (AGI). Furthermore, such systems can augment the capabilities of smaller models through automated tool construction by larger models. For instance, when faced with tasks beyond their native competence (e.g., complex code generation), smaller models can leverage tools dynamically encapsulated by larger models through MCP interfaces.

However, alongside these benefits, our framework introduces potential risks that warrant careful consideration. Malicious actors could exploit the system by disguising harmful or unsafe tools through misleading descriptions, potentially inducing models to execute dangerous operations. Such misuse may lead to information security breaches or financial losses. Additionally, erroneous tool invocation by the model—such as unintended deletion of local files—could cause significant losses, underscoring the need for robust safeguards in tool validation and execution monitoring.

## Appendix C Details of Task Construction

### C.1 Task Statistics

Figure: Figure 7: Task distribution in LiveMCPBench: a comprehensive benchmark comprising 95 tasks Across 6 distinct domains.
Refer to caption: x9.png

The task statistics of LiveMCPBench are illustrated in Figure [7](https://arxiv.org/html/2508.01780v1#A3.F7). LiveMCPBench comprises six categories of tasks, each designed to reflect common real-life scenarios:

- 1.
Office. This category represents typical office-related tasks, primarily involving reading and writing documents in Word, Excel, and PowerPoint.
- 2.
Lifestyle. These tasks pertain to daily routines, such as retrieving news updates or querying the latest arXiv papers.
- 3.
Leisure. This category encompasses entertainment-oriented tasks, including fetching gaming news, obtaining specific game-related information, or retrieving details about museums.
- 4.
Finance. Tasks in this category focus on personal financial management, such as checking stock prices, analyzing market trends, or obtaining cryptocurrency valuations.
- 5.
Travel. This category includes tasks related to personal travel, such as route planning, hotel searches, and ticket inquiries.
- 6.
Shopping. These tasks revolve around personal shopping activities, including product information retrieval and recommendations.

### C.2 Annotation Principles

LiveMCPBench focuses on leveraging a large-scale MCP toolset to accomplish complex tasks. To ensure high-quality task construction, we employ two groups of annotators involving proposers and verifiers. All annotators first freely explore the MCP toolset, including tool descriptions and real-world calls, to gain familiarity with its functionalities.

First, Proposers are randomly assigned a scenario and instructed to formulate tasks adhering to the following principles:

- 1.
Real-World Relevance. Tasks must reflect realistic needs within the given scenario.
- 2.
Temporal Dynamics. Tasks should be time-sensitive, requiring real-time information retrieval from tools rather than relying solely on static internal knowledge.
- 3.
Tool Diversity. Tasks should necessitate the integration of multiple tools, avoiding cases where a single tool suffices for completion.

After proposing a task, the proposers try to complete it using the MCP toolset, documenting the required tools and key points.

Once all tasks are collected, verifiers manually consolidate similar tasks to prevent redundancy. Additionally, they rigorously assess task feasibility and execution quality to maintain high standards in the benchmark.

## Appendix D Details of LiveMCPTool Collection

### D.1 Toolset Statistics

Figure: Figure 8: Distribution of tools in LiveMCPTool, categorized into 8 distinct types (total: 527 tools).
Refer to caption: x10.png

Figure: Figure 9: Distribution of servers in LiveMCPTool, categorized into 8 distinct types (total: 70 servers).
Refer to caption: x11.png

The statistics of LiveMCPTool’s tools and servers are illustrated in Figures [8](https://arxiv.org/html/2508.01780v1#A4.F8)-[9](https://arxiv.org/html/2508.01780v1#A4.F9). The collected toolset is categorized into eight distinct classes:

- 1.
Discovery. This category encompasses tools for information gathering and retrieval, such as search engines and news aggregators.
- 2.
Visualization. Tools in this category facilitate data or concept visualization, including bar chart plotting and mind map creation.
- 3.
File Access. This class comprises tools for local file operations, such as reading Word, Excel, or PowerPoint files, as well as executing command-line instructions.
- 4.
Code. These are programming-related tools, such as those providing the latest AntV documentation and sample code.
- 5.
Entertainment. This category includes recreational tools, such as those for retrieving Yu-Gi-Oh card information or League of Legends game data.
- 6.
Finance. Financial tools fall under this class, including those for fetching real-time stock prices or cryptocurrency market data.
- 7.
Location. This category consists of map-based services, such as navigation systems and points-of-interest discovery tools.
- 8.
Miscellaneous. This catch-all category accommodates tools not fitting the above classifications, such as calculators and local memory utilities.

## Appendix E Implementation Details

Figure: Figure 10: An illustrative case of evaluation failure in LiveMCPEval. The assessment was conducted using DeepSeek-V3 on the completion trajectory from Claude-Sonnet-4. The evaluator model erroneously concluded the task was successful based solely on the agent’s file creation action, while failing to recognize that the agent did not actually acquire the required information.
Refer to caption: x12.png

### E.1 Computing Resources and Private Models

In our experiments, we deployed two models: Qwen2.5-72B-Instruct and Qwen3-Embedding-0.6B. The computational infrastructure consisted of a Linux server (Ubuntu 22.04) with 4 NVIDIA A800-80G GPUs and 1TB of memory.

We accessed the following proprietary models through their respective platforms:

- •
OpenRouter: GPT-4.1, GPT-4.1-Mini, DeepSeek-R1-0528, DeepSeek-V3-0324, Qwen3-235B-A22B and Qwen3-32B.
- •
Anthropic Console: Claude-Opus-4-20250514, Claude-Sonnet-4-20250514.
- •
Google AI Studio: Gemini-2.5-Pro.

To address suboptimal greedy decoding in certain reasoning models, we implemented a uniform temperature parameter of 0.7 across all experiments. This configuration introduces controlled stochasticity while maintaining result reliability for long-horizon tasks, as we observed that sporadic randomness has negligible cumulative impact on aggregate performance.

### E.2 Tool Retrieval Configuration

For consistent retrieval performance, we established a standardized framework using Qwen2.5-72B-Instruct for tool summarization and Qwen3-Embedding-0.6B for embedding generation. To control for potential variance in this component, we maintained identical retrieval module parameters across all experimental conditions. Given the inherent temporal variability in tool outputs, we conducted all experiments within a tightly controlled window (July 20–27, 2025) to minimize fluctuations attributable to temporal factors.

Figure: Figure 11: Comparison of key points in DeepSeek-V3 and human annotations: Similar content despite different ordering.
Refer to caption: x13.png

## Appendix F Evaluation Analysis

### F.1 Case Study: Error Evaluation Examples

To illustrate cases where evaluator judgments diverge from human assessments, we conducted a case study on Claude-Sonnet-4 trajectories evaluated by DeepSeek-V3, presenting a representative example in Figure [10](https://arxiv.org/html/2508.01780v1#A5.F10).

In this instance, the evaluator failed to recognize that the agent did not actually acquire the correct information, despite successfully creating the required file. The evaluator erroneously concluded task completion based solely on the file creation process. This case highlights a potential limitation of LiveMCPEval: the system’s tendency to overlook critical details when processing excessively lengthy and complex trajectories.

We propose that this long-range evaluation challenge could be addressed by modifying existing evaluation frameworks to incorporate dynamic agent-based assessment of each trajectory step. However, such an approach would significantly compromise evaluation efficiency. While our current evaluation method achieves satisfactory human agreement rates, this particular issue warrants further in-depth investigation.

### F.2 Human and LLM Key Points Examples

To analyze the differences between LLM-generated key points and human-annotated key points, we conducted a comparative study between key points generated by Deepseek-V3 and those manually annotated by humans. The comparison results are presented in Figure [11](https://arxiv.org/html/2508.01780v1#A5.F11).

Our analysis reveals that while the ordering of key points differs between human and LLM-generated outputs, both consistently capture similar critical steps. This observation suggests the practical applicability of LLM-generated key points in evaluation tasks. Importantly, our findings indicate that LLM-generated key points can serve as a reliable alternative for robust evaluation in scenarios where human annotations are unavailable.

## Appendix G Case Study: Error Examples

Figures [12](https://arxiv.org/html/2508.01780v1#A8.F12)-[15](https://arxiv.org/html/2508.01780v1#A8.F15) present concrete examples of four distinct error types: Query Error, Retrieve Error, Tool Error, and Other Error.

Broadly speaking, Query and Other errors primarily highlight design flaws in the agent’s architecture—specifically, whether the agent incorporates sufficient mechanisms to ensure task completion. In contrast, Tool errors are more closely tied to the capabilities of the LLM itself, particularly its ability to accurately process tool parameters and descriptions while maintaining nuanced contextual understanding. Retrieve errors, on the other hand, largely reflect the limitations of the tool retrieval system, testing its effectiveness in identifying relevant tools based on the server-tool description.

## Appendix H Prompts

### H.1 MCP Copilot Agent Prompt

### H.2 LiveMCPEval Prompt

Figure: Figure 12: An illustration of Query Error: Discrepancy between the agent-generated query and the task’s required competencies.
Refer to caption: x14.png

Figure: Figure 13: An illustration of Retrieve Error: The retrieve system incorrectly identifies and returns an inappropriate tool.
Refer to caption: x15.png

Figure: Figure 14: An illustration of Tool Error: The LLM misapplication with improper tool parameters.
Refer to caption: x16.png

Figure: Figure 15: An illustration of Other Error: The agent’s inadequate response to tool timeout.
Refer to caption: x17.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: 1. How were the ten evaluated models selected, and are there plans to include more recent systems such as GPT-5, Grok-4, or GLM-4.5 to improve completeness and ensure broader coverage across model families?
2. Why did the evaluation rely solely on single-run success rates instead of including reliability metrics such as pass@k or pass^k that better capture model stability under repeated execution? Could such metrics be incorporated in future updates of the benchmark?
3. Can you provide a concrete annotation example illustrating how the annotation principles are applied in practice, along with a detailed summary of API token usage of LiveMCPBench?
4. What criteria were used to define task difficulty across the 95 daily tasks, and how well do these categories align with real-world MCP usage complexity?
5. Could the benchmark be extended to support multi-agent collaboration scenarios, where multiple LLMs coordinate across MCP servers? If so, what challenges or modifications would be required in the current setup?
soundness: 2
strengths: 1. Realism and scale: LiveMCPBench captures the complexity of real MCP ecosystems with 527 tools across 70 servers, moving beyond prior single-server or simulated tool-use settings.
2. Well-rounded design: The integration of LiveMCPTool, LiveMCPEval, and MCP Copilot Agent provides a coherent, end-to-end evaluation framework that is both scalable and reproducible.
3. Empirical insight: The benchmark yields meaningful diagnostic findings—retrieval errors dominate failure modes, and tool composition correlates strongly with success.
4. Promised reproducibility: The work offers a ready-to-deploy tool suite and automated evaluation protocol, enabling consistent, and the authors intend to release code and data.
summary: This paper introduces LiveMCPBench, a benchmark for evaluating LLM agents in large-scale MCP environments. It comprises (1) 95 real-world tasks, (2) LiveMCPTool with 70 servers and 527 tools, (3) LiveMCPEval achieving 81% human agreement for automated evaluation, and (4) MCP Copilot Agent as baseline. Testing 10 frontier models reveals Claude-Sonnet-4 reaches 78.95% success rate, while most models struggle with retrieval errors (50%) and meta-tool-learning in multi-tool scenarios.
weaknesses: 1. Limited model coverage: Only ten models are tested, excluding several widely used or newly released systems (e.g., GPT-5, Grok-4, GLM-4.5). This undermines the generalizability of the reported findings.
2. Volatile metric design: The experiment employs a single-pass success metric. Realistic deployments often depend on reliability across multiple attempts. Using metrics such as pass@k (success in k runs) and pass^k (all k runs successful) would better capture stability and practical robustness.
3. Evaluation methodology limitations: While LiveMCPEval enables scalable automatic assessment via the LLM-as-a-Judge paradigm, up to a 10-point discrepancy from human annotations indicates that purely LLM-based evaluation may be insufficient for structured MCP interactions. Integrating rule-based or code-verification modules might improve transparency, interpretability, and consistency with MCP's deterministic design.
4. Missing annotation and cost detailed demonstration: As a benchmark paper, it lacks concrete examples demonstrating the annotation process and a detailed report of API token usage and computational cost, which weakens transparency and reproducibility.
5. Typo in paper writing: In Figure 11, "Evaulation Error" should be corrected to "Evaluation Error."

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Could the authors elaborate on the implementation of the LiveMCPTool servers? Specifically, are the 70 servers accessing live, real-time public APIs (e.g., a real stock market ticker, a real weather service), or do they simulate dynamism using static but complex datasets? 

2. Given that Retrieve Error is the dominant bottleneck ($\sim 50\%$), the ReACT agent used only has a single retrieval step before attempting execution. What minimal architectural modification (e.g., a Retrieval Reflection step, where the agent critiques the $k=5$ tools before acting) could be incorporated into the MCP Copilot Agent to address this weakness, and why was this approach not chosen for the baseline?

3. Cost of Composition: Table 3 shows Claude-Sonnet-4 uses $2.71$ tools and $5.59$ executions on average, demonstrating active composition. Could the authors provide a small analysis (e.g., in the Appendix) breaking down the token cost or dialogue turns associated with this active composition versus simple single-tool calls?
soundness: 3
strengths: 1. The paper correctly identifies and addresses the fundamental shift from small, unstable API sets to large-scale, hierarchical Model Context Protocol (MCP) ecosystems. The focus on retrieval over $500+$ tools and multi-tool composition in daily, dynamic tasks makes this benchmark uniquely challenging and representative of real-world use.
2. The curated, dependency-free toolset (LiveMCPTool) is a major engineering strength. By eliminating the reliance on scattered, proprietary API keys, the paper ensures that the benchmark is reliable and accessible, overcoming the known instability issues of prior works.
3. The empirical results are sharp and actionable. The finding that retrieval errors account for nearly 50% of failures and the positive correlation between active tool composition (measured by tool count and execution attempts) and success are clear diagnostic insights for future research on the planning and execution modules of agents.
summary: This paper introduces LiveMCPBench, a novel and challenging benchmark for evaluating LLM-powered tool-use agents within a large-scale, real-world Model Context Protocol (MCP) ecosystem.1 The authors argue that existing benchmarks are insufficient because they rely on simulated or small-scale APIs and often bypass the critical challenges of large-scale retrieval and multi-tool composition.

The key contributions include:
1. LiveMCPBench: A benchmark of 95 multi-step, dynamic "daily tasks" across six practical domains (Office, Travel, Finance, etc.).
2. LiveMCPTool: A reproducible, dependency-free tool suite comprising 70 MCP servers and 527 tools, packaged for immediate use.
3. LiveMCPEval: An automated LLM-as-a-Judge framework utilizing human-aligned key points to verify success, even with dynamic data and multiple valid solution paths.

Empirical evaluation of 10 frontier LLMs reveals a substantial performance gap (Claude-Sonnet-4 achieves 78.95% success, while most others are 30–50%). Error analysis identifies tool retrieval as the dominant bottleneck (accounting for $\sim 50\%$ of failures), and confirms a strong correlation between active tool composition and task success.
weaknesses: 1. Limited Agent Architecture Analysis: The paper uses a single baseline, the MCP Copilot Agent (a ReACT agent). While the diagnosis is excellent, the paper doesn't explore how the retrieval error issue might be mitigated by a more advanced planning or memory architecture (e.g., Tree-of-Thought, specialized memory/reflection, or workflow memory). 

2. While the dependency-free nature of LiveMCPTool is a strength, the underlying data source for the 70 servers (e.g., news, weather, stock data) is only broadly mentioned as being derived from the "MCP Marketplace" and "daily scenarios." Clarifying whether these servers utilize pre-cached static data or truly access live/dynamic public APIs (even if locally hosted) would solidify the claim of dynamism.

3. The observation that the LLM-as-a-Judge effectiveness depends on the model (e.g., DeepSeek-V3 performs well, but Claude-Opus-4 performs poorly) is important. This causes a slight concern about the evaluation stability if the chosen judge model were to change or degrade. A brief discussion on judge model robustness or alternative verification protocols would strengthen this section.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. Why did you opt for a binary success metric instead of more granular measures like retrieval recall, parameter correctness, or stability across reruns?
2. How do you ensure that the fixed retrieval pipeline (embedding model, summarizer, top-k=5) does not bias the benchmark results?
3. Have you conducted ablations with alternative retrievers, varying k-values, or hybrid retrieval methods to confirm that retrieval errors are not artifacts of your chosen pipeline?
4. Given that LiveMCPEval achieves only ~81% agreement with human judgments, what steps can you take to improve evaluator reliability?
soundness: 3
strengths: 1. The paper addresses a real gap in MCP evaluation. Unlike prior MCP benchmarks (e.g., MCPBench, MCP‑RADAR, MCPEval), LiveMCPBench evaluates large‑scale retrieval and multi‑tool composition over 70 servers / 527 tools and 95 tasks.
2. The authors curate a key‑free, ready‑to‑deploy MCP tool suite that removes scattered API configuration barriers, improving reproducibility relative to API‑centric benchmarks where many APIs require access keys or have become unavailable.
3. Breaks down failure root causes into tool composition and retrieval errors.
summary: The paper introduced LiveMCPBench, a benchmark consisting of a “plug‑and‑play” tool suite of 70 MCP servers with 527 tools and 95 multi‑step daily tasks, plus an LLM‑as‑a‑Judge evaluator (LiveMCPEval) to handle dynamic data and multiple valid solution paths. It aims evaluating LLM agents’ ability to retrieve and compose tools across a realistic, large‑scale MCP servers with low effort (e.g., no API access keys).
weaknesses: I appreciate the authors’ effort in creating a curated collection of MCP tools that work plug-and-play. Their main motivation is clear: existing MCP benchmarks are either too small or too difficult to use. Many tools in those benchmarks require access keys or are no longer valid. In contrast, LiveMCPBench includes key-less tools, making them easy to use. However, the benchmark itself is not large. It covers only 95 tasks, which is modest for evaluating general-purpose agentic reasoning. The number of MCP servers and tools is also limited—just 70 servers and 527 tools. This makes it unsuitable for testing agents designed for much larger MCP ecosystems with thousands of servers (something that the authors mention in the beginning of the paper).

There is another concern. Like other large benchmarks, LiveMCPBench tools may become unusable if their owners stop maintaining them. Although the toolset is packaged for reproducibility, the benchmark depends on dynamic online sources. It does not address long-term sustainability issues such as server versioning, security, or maintenance (see Limitations §A). Over time, servers may change, become unavailable, or behave differently. This can harm the reproducibility and reliability of benchmark results. The paper does not propose concrete solutions, such as Dockerized server snapshots, health checks, or semantic versioning. Without these, future evaluations may not be consistent or valid. This technical gap puts the benchmark’s long-term value for the research community at risk.

A key requirement of dealing with an 'ocean of MCP tools' is a good retrieval system that can reliably select a small superset of  tools relevant to a given task. Authors use a embedding-based retrieval subsystem which is technically rigid: it uses a single embedding model (Qwen3-Embedding-0.6B) and a single summary generator (Qwen2.5-72B-Instruct) to process server and tool descriptions, with a fixed top-k=5 retrieval (§G.2). This design choice means that all agentic models are evaluated with the same retrieval backbone, potentially conflating agent reasoning ability with retriever recall. If the retriever fails to surface relevant tools, even a strong agent may be unable to succeed. The paper’s own analysis shows that retrieval errors account for ~50% of failures (Fig. 6), but does not explore how varying the retriever, k-value, or embedding model might affect outcomes. This limits the generalizability of the findings and may overstate the impact of “retrieval errors” attributable to the chosen pipeline.

As mentioned before, LiveMCPBench includes only 95 tasks, which is modest for a benchmark intended to evaluate general-purpose agentic reasoning. Furthermore, the task distribution is skewed: 32.6% are Office tasks (Fig. 7), and annotators ultimately used only 150 out of 527 tools (28.46%; §E.1). This means that a large portion of the curated toolset remains untested, and the benchmark may over-represent office-suite workflows while under-testing other domains such as location, code, or finance. The limited and imbalanced task set restricts the breadth of agentic behaviors that can be meaningfully evaluated and may bias results toward models that excel in office-related scenarios.

The paper depends on an LLM-based automatic evaluator (LiveMCPEval) to judge task success. While this approach enables scalable, open-ended evaluation, the reported agreement with human annotators is low (see Fig. 10 and Appx. H). This means a significant fraction of judgments can diverge from human consensus, especially for long or complex trajectories. Such residual disagreement undermines the reliability of leaderboard rankings and may allow models to “game” the judge or be unfairly penalized for nuanced outputs. The paper acknowledges these limitations but does not propose robust adjudication mechanisms (e.g., dual judges, confidence calibration, or selective human audits) to mitigate the risk.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Although the team validated scoring accuracy with human labels (e.g., DeepSeek-V3 achieved 78.95% agreement with humans), LLM judges can still have systematic biases — for instance, misjudging an empty PDF as a valid “completed” output. While the authors analyzed some error cases, they didn’t fully address or mitigate these inherent biases, which may cause certain models’ true capabilities to be over- or underestimated.

2. Tool-use hallucinations may produce seemingly correct but actually invalid results (e.g., fabricated tool responses or outdated data). The paper doesn’t discuss how such hallucinations are detected or penalized in the evaluation, which could limit the reliability of the benchmark results.
soundness: 3
strengths: The dataset is highly practical and valuable for real-world LLM evaluation.
summary: This paper introduces LiveMCPBench, a benchmark made up of 95 real-life multi-tool tasks such as “Check next Monday’s Beijing–Shanghai high-speed train tickets,” “Summarize today’s news into a PDF,” and “Find a 3-day stay in Paris.” These cover six domains — office work, daily life, entertainment, finance, travel, and shopping — and each task requires combining multiple tools and accessing real-time information (e.g., tickets, news). This design effectively tests a model’s real abilities in tool discovery and tool orchestration.

The team also developed LiveMCPTool, which includes 70 servers and 527 tools. It works right out of the box — no complex API keys or permission setups required — ensuring reproducibility across different users (so results aren’t skewed by unavailable tools).

Finally, they proposed LiveMCPEval, an automatic evaluation system using another LLM (e.g., DeepSeek-V3) as a referee to directly judge task completion rather than strict tool usage. For example, in the “generate news PDF” task, the model passes as long as it produces a correct PDF — regardless of the specific tool order. The system also handles dynamic information (like changing weather) automatically, removing the need for manual checking.

I really like this paper — the dataset is valuable and fills an important gap. I’d recommend acceptance.
weaknesses: 1. Relying on LLM-based evaluation may introduce bias.

2. The framework doesn’t account for hallucinated tool calls or factually wrong but plausible outputs.


Decision: Reject