OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents
Authors: name: Hongrui Jia; name: Jitong Liao; name: Xi Zhang; name: Haiyang Xu; name: Tianbao Xie; name: Chaoya Jiang; name: Ming Yan; name: Si Liu; name: Wei Ye; name: Fei Huang
Published: 2025-10-28 15:56:36

Abstract: With advances in decision-making and reasoning capabilities, multimodal
agents show strong potential in computer application scenarios. Past
evaluations have mainly assessed GUI interaction skills, while tool invocation
abilities, such as those enabled by the Model Context Protocol (MCP), have been
largely overlooked. Comparing agents with integrated tool invocation to those
evaluated only on GUI interaction is inherently unfair. We present OSWorld-MCP,
the first comprehensive and fair benchmark for assessing computer-use agents'
tool invocation, GUI operation, and decision-making abilities in a real-world
environment. We design a novel automated code-generation pipeline to create
tools and combine them with a curated selection from existing tools. Rigorous
manual validation yields 158 high-quality tools (covering 7 common
applications), each verified for correct functionality, practical
applicability, and versatility. Extensive evaluations of state-of-the-art
multimodal agents on OSWorld-MCP show that MCP tools generally improve task
success rates (e.g., from 8.3% to 20.4% for OpenAI o3 at 15 steps, from 40.1%
to 43.3% for Claude 4 Sonnet at 50 steps), underscoring the importance of
assessing tool invocation capabilities. However, even the strongest models have
relatively low tool invocation rates, Only 36.3%, indicating room for
improvement and highlighting the benchmark's challenge. By explicitly measuring
MCP tool usage skills, OSWorld-MCP deepens understanding of multimodal agents
and sets a new standard for evaluating performance in complex, tool-assisted
environments. Our code, environment, and data are publicly available at
https://osworld-mcp.github.io.