MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models
Authors: name: Zhiwei Liu, name: Jielin Qiu, name: Shiyu Wang, name: Jianguo Zhang, name: Zuxin Liu, name: Roshan Ram, name: Haolin Chen, name: Weiran Yao, name: Huan Wang, name: Shelby Heinecke, name: Silvio Savarese, name: Caiming Xiong

The rapid rise of Large Language Models (LLMs)-based intelligent agents
underscores the need for robust, scalable evaluation frameworks. Existing
methods rely on static benchmarks and labor-intensive data collection, limiting
practical assessment. We introduce \oursystemname, an open-source Model Context
Protocol (MCP)-based framework that automates end-to-end task generation and
deep evaluation of LLM agents across diverse domains. MCPEval standardizes
metrics, seamlessly integrates with native agent tools, and eliminates manual
effort in building evaluation pipelines. Empirical results across five
real-world domains show its effectiveness in revealing nuanced, domain-specific
performance. We publicly release MCPEval
https://github.com/SalesforceAIResearch/MCPEval to promote reproducible and
standardized LLM agent evaluation.