LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?
Authors: name: Guozhao Mo; name: Wenliang Zhong; name: Jiawei Chen; name: Xuanang Chen; name: Yaojie Lu; name: Hongyu Lin; name: Ben He; name: Xianpei Han; name: Le Sun
Published: 2025-08-03 14:36:42

Abstract: With the rapid development of Model Context Protocol (MCP), the number of MCP
servers has surpassed 10,000. However, existing MCP benchmarks are limited to
single-server settings with only a few tools, hindering effective evaluation of
agent capabilities in large-scale, real-world scenarios. To address this
limitation, we present LiveMCPBench, the first comprehensive benchmark
comprising 95 real-world tasks grounded in the MCP ecosystem, designed to
evaluate LLM agents at scale across diverse servers. To support a scalable and
reproducible evaluation pipeline in large-scale MCP environments, we curate
LiveMCPTool, a diverse and readily deployable collection of 70 MCP servers and
527 tools. Furthermore, we introduce LiveMCPEval, an LLM-as-a-Judge framework
that enables automated and adaptive evaluation in dynamic, time-varying task
environments, achieving 81% agreement with human reviewers. Finally, we propose
the MCP Copilot Agent, a multi-step agent that routes tools for dynamic
planning and executes tools for API interaction across the entire LiveMCPTool
suite. Our evaluation covers 10 leading models, with the best-performing model
(Claude-Sonnet-4) reaching a 78.95% success rate. However, we observe large
performance variance across models, and several widely-used models perform
poorly in LiveMCPBench's complex, tool-rich environments. Overall, LiveMCPBench
offers the first unified framework for benchmarking LLM agents in realistic,
tool-rich, and dynamic MCP environments, laying a solid foundation for scalable
and reproducible research on agent capabilities. Our code and data will be
publicly available at https://icip-cas.github.io/LiveMCPBench.